Deep Reinforcement Learning for Stability Enhancement of a Variable Wind Speed DFIG System ^†

Abstract

Low-frequency oscillations are a primary issue for integrating a renewable source into the grid. The objective of this study was to find sensitive parameters that cause low-frequency oscillations and design a Twin Delayed Deep Deterministic Policy Gradient (TD3) agent controller to damp the oscillations without requiring an accurate system model. In this work, a Q-learning (QL)-based model-free wind speed DFIG was designed on the rotor-side converter (RSC), and a QL-based model-free DC-link voltage regulator was designed on the grid-side converter (GSC) to enhance the stability of the system. In the next step, the TD3 agent was trained to learn the system dynamics by replacing the inner current controllers of the RSC, which replaced the QL-based model. In the first stage, the conventional PSS and Proportional–Integral (PI) controllers were introduced to both the RSC and GSC. Then, the system was trained to become model-free by replacing the PSS and the PI controller with a QL algorithm under very small wind speed variations. In the second stage, the QL algorithm was replaced with the TD3 agent by introducing large variations in wind speed. The results reveal that the TD3 agent can sustain the stability of the DFIG system under large variations in wind speed without assuming a detailed control structure beforehand, while QL-based controllers can stabilize the doubly fed induction generator (DFIG)-equipped wind energy conversion system (WECS) under small variations in wind speed.

1. Introduction

Based on the type of generator and the grid interface, wind energy systems can be categorized into four types [1]:

(a)

Squirrel-cage induction generator (SCIG) or fixed speed system.

(b)

Wound-rotor induction generator (WRIG) with variable rotor resistance.

(c)

Doubly fed induction generator.

(d)

Full-power converter generator.

SCIGs are mainly used for smaller wind turbines, as they are simple and economical compared to other generators. In a squirrel-cage induction generator, the rotor bars are permanently short-circuited; therefore, the rotor voltage is zero. The stator is connected to a soft starter and is connected to the grid through a transformer. A capacitor bank is employed to compensate for the reactive power, and a soft starter is employed to mitigate high starting currents and to produce a smooth grid connection [2]. In a WRIG, the stator is directly connected to the grid, and the wound rotor winding is connected to a variable resistor via slip rings. In a WRIG, it is possible for the rotor to have configurations such as slip power recovery, the use of cyclo-converters, and rotor resistance chopper control [3]. In both WRIGs and SCIGs, by controlling the rotor resistance, the slip of the machine can be changed to 2–10% [1], through which the generator output can be controlled. The third type of generator used is the DFIG. The constructional features of DFIGs are like those of WRIGs, except that in a DFIG, the rotor winding of the WRIG is connected to the grid through an AC-DC-AC converter. In a fully converted synchronous generator or fully converted squirrel-cage induction generator (SCIG), the total power is interchanged between the wind system and the grid through a power electronic converter system, whereas in some systems, it can be transmitted to the grid directly [1]. Figure 1a,b show the configuration of a wind system with a synchronous generator and with a permanent-magnet synchronous generator (PMSG) [1]. It can be recognized that in Figure 1a, the synchronous generator is excited by using the power electronic converter externally, while in Figure 1b, it is excited by a permanent magnet, as the generator is a permanent-magnet synchronous generator. For large wind farms, both DFIGs and PMSGs are preferred due to increased power control.

There are many advantages and superior characteristics of using a PMSG machine over a DFIG. A PMSG machine has better performance, higher reliability, and wider speed control [4]. Due to the PMSG’s better performance, most of the current research work focus has shifted to topologies utilizing synchronous generators with permanent-magnet excitation. However, DFIGs are still dominant in modern wind power generation systems, as DFIGs can operate under variable speeds, regulate active and reactive power independently, and have a low converter cost [5]. The reason for the lower converter cost in DFIG wind turbines is that the power electronic devices are fed with power generated only by the rotor (25–30% of rated power), and due to this, the lower converter rating compromise offers significant cost savings over a PMSG [6].

The doubly fed induction generator (DFIG) model is considered one of the best solutions for wind energy conversion systems (WECSs). The two main reasons for using a DFIG in a WECS are its asynchronous characteristics and the flexibility of utilizing power electronic converters, which results in cost savings due to a lower converter rating. In DFIG-based grid-connected wind systems, the stator is directly connected to the grid, whereas the rotor is connected to the grid by using a back-to-back power electronic converter. Figure 2 depicts the schematic model of a grid-connected DFIG-based WECS [7].

Every power network (grid) experiences a lot of disturbances, as there may be a great number of variations, such as voltage, frequency, active power, and reactive power, at the load end or at power generating stations (renewable or non-renewable). For any system to operate without disturbances, power system stability and voltage regulation are critical control issues that need to be considered [8]. The importance of power system stability can be illustrated clearly by investigating [9], which presents a clear description of power system instability as the primary grounds for any major blackout. The main objective of any power system stabilizer is to provide stable power to the grid and to improve the damping of oscillations. Especially when connecting any renewable energy conversion system to the grid, supplying stable power is always challenging due to fluctuating frequencies and voltages.

2. Literature Review

In a DFIG-equipped WECS, decoupled controllers are used to control the active and reactive power on both the rotor and grid sides, the rotor speed, and DC voltage and to track the maximum power. Low-frequency oscillations were observed for a WECS with weak grid interconnections; this oscillation mode caused torsional interconnections with a remote synchronous generator and led to the shutdown of the power plant [10]. To overcome the problem of low-frequency oscillations, a power system stabilizer can be applied at the output or the input of the controller. As per [11], a PSS can be employed for any DFIG variable that is influenced by network oscillations, such as rotor speed, stator electrical power, and voltage or network frequency. In [11,12,13,14,15,16,17,18], the importance of employing a PSS and improving the damping of oscillations in a grid network fed by a DFIG-WES using slip signal and rotor speed deviations are discussed. Various oscillations that are damped using a PSS are: electromechanical oscillations [19], inter-area power system oscillations [20,21], network damping capability [22], and oscillations caused by DFIG when integrated into a network that is already fed by a synchronous generator [17,18]. These papers clearly illustrate the need for a PSS to damp the oscillations and show that a PSS can be used to improve the stability of the grid-integrated wind system.

Most of the conventional PSSs (CPSSs) employed for wind energy systems are classical designs in which the system is linearized around an operating point. However, PSSs developed using Artificial Intelligence (AI) techniques have been used for design and stability studies of non-renewable-sourced power systems. Some of the frequently used techniques for a PSS design are ANN [23,24] and Fuzzy Logic [10,25,26], and support vector regression was used to design an adaptive PSS in [27]. The use of AI to solve stability issues has been categorized into three separate methodologies based on the techniques used: supervised learning, unsupervised learning, and reinforcement learning (RL) [28]. RL is completely different from supervised learning and unsupervised learning. Supervised learning alone is not adequate for learning from interactions, and in interactive problems, it is often impractical for supervised learning to obtain the desired behavior for all situations in which the agent must act [28]. In contrast, in an RL territory, an agent must be able to learn from its own experience. Unsupervised learning is typically focused on finding structures hidden in collections of unlabeled data, whereas the RL paradigm aims to maximize a reward signal instead of trying to find hidden structures [28]. Thus, an RL is a third machine learning paradigm, alongside supervised and unsupervised learning.

The current research used reinforcement learning to control the entire system, as there is a need for interaction between the mechanical system (wind turbine) and electrical system (DFIG and controllers) to supply stable power to the grid for wind speed variations. One of the main reasons for using the RL control method is its capability of adapting itself to evolving generation levels, load levels, and operating uncertainties and responding to arbitrary disturbances [29]. In [29], the RL method was used in online mode and applied to control a thyristor controller series capacitor (TCSC), aiming to damp power system oscillations. RL controllers were designed to stabilize the closed loop system after severe disturbances in [30]. A specific RL algorithm called Q-learning was utilized to control and adjust the gain of a conventional PSS in [31]. The RL algorithm was also used in generation control and voltage and reactive power control [32,33,34]. In [35], a control strategy was developed for a PSS using the Q-learning method to suppress low-frequency oscillations. In [36], a proportional resonance PSS (PR-PSS) was proposed using an actor-critic agent, one of the RL techniques for adaptive adjustment of parameters to suppress ultra-low-frequency oscillations. The RL techniques discussed in [35,36] were used for comparing the PSS results obtained with the Q-learning algorithm discussed in this paper. The TD3 method was implemented in [37,38] for continuous power disturbances to overcome low-frequency oscillations. In [38], the TD3 method was used to perform parameter estimation and the fine-tuning of PID controllers and to overcome the problem of low-frequency oscillations caused by load generation variations. Deep reinforcement learning methods were implemented for load frequency control of a multi-area power system and battery energy management. In [39], a multi-agent deep reinforcement learning method was proposed, which utilized the DDPG method to optimize load frequency control performance. In this study, in addition to the QL method, the TD3 method was explored and implemented to solve the low-frequency oscillations caused by huge variations in wind speed. In this research, the TD3 method and QL method were implemented by replacing the existing PI controller and PSS. This paper is an extension of [40], where a Q-learning algorithm was implemented on the rotor-side converter for a small range of changes in wind speeds. The major contributions of the current work are discussed in Section 3.

3. Contributions

In this work, a CPSS was first designed, and control strategies were applied at a fixed wind speed. The optimal PI gain values for both the rotor-side converter and the grid-side converter were obtained by using eigenvectors for the PSS, and the stability of the system was proved mathematically. In the next step, a Q-learning algorithm was implemented on the designed PSS on the RSC and PI controllers on the GSC for variable wind speeds. The objective of the Q-learning algorithm implemented on the RSC and GSC is to suppress the low-frequency oscillations of the DFIG-based WECS when variable speeds are applied to the system. Since the control objective is to stabilize the system with active power (P) and reactive power (Q) without low-frequency oscillations, this paper uses the active power change as the state of the agent and the control output of the RSC as the action of the agent to train the model. On the grid-side converter, the grid-side active power is controlled, which is a function of DC-link voltage ($V_{dc}$), which acts as the state of the agent, and the reward function generates the input signal to the grid-side current ($i_{dg}^{*}$). Simulation results verify that the designed Q-learning-based model-free controllers can quickly stabilize the DFIG wind system under a small range of wind speeds. The terms action, state, and agent used in this section are explained in detail in Section 3 of this paper. With the Q-learning agent, the system becomes unstable under huge variations in wind speed. One of the solutions is to implement the system with an actor-critic method. In this research, the TD3 agent was trained to learn the system dynamics under huge variations in wind speed and control the active and reactive power. Real-time wind speed variations in the Ottawa region were used as the system input. The PI controller in the inner current control loop of the RSC was replaced with the TD3 agent. From the results discussed in Section 5, it is observed that the TD3 agent can mitigate the frequency changes under huge variations in wind speed and provide stable power to the grid. In both QL and TD3 implementations, the PSS and PI controllers were removed, and the agent was trained to learn the system dynamics and suppress low-frequency oscillations.

4. Wind Turbine System Structure and Model

Figure 2 shows the physical model of the grid-connected DFIG-based wind turbine system. This benchmark power system model was first introduced in [7] to study the effect of shaft systems and low-frequency oscillations by comparing the switching-level (SL) and Fundamental Frequency (FF) models. As illustrated in Figure 2, the system consists of mechanical and electrical models. The mechanical model has 3 different components: (1) wind turbine, (2) gearbox, and (3) pitch controller; all 3 of these components together provide a complete drivetrain model, which provides the required mechanical energy to the generator. The electrical model consists of a DFIG generator, of which the stator is directly connected to the grid, and the rotor is connected through back-to-back IGBT-based pulse-width modulation converters.

4.1. Design of Drivetrain Model

The mathematical representation of the drivetrain is formed by the turbine rotating mass, low-speed shaft, gearbox, high-speed shaft, and generator rotating mass. The model of the drivetrain is developed by neglecting the mechanical twisting and stresses, as these are more related to mechanical design and studies. Moreover, for power system stability studies, it is suggested in [41] to consider the turbine, gearbox, and generator as rigid disks and shafts as mass-less torsional springs. A two-mass model is recommended for power system stability studies; this is because there are more possibilities for the representation of shaft stiffness and inertia constants [42]. Figure 3 shows the physical representation of the two-mass model [7]. The dynamics of the two-mass model can be obtained by applying Newton’s equation of motion for each mass. The equations obtained are [7,43]:

$$2H_{t}p{\omega }_t=T_m-D_t{\omega }_t-T_{sh}$$

(1)

$$2H_{g}p{\omega }_r=T_{sh}-D_g{\omega }_r-T_e$$

(2)

where $p=d/dt$, and $T_{sh}$ is the shaft torque, which is given as:

$$p{T}_{tg}=K_{tg}\left({\omega }_t-{\omega }_r\right)$$

(3)

$$T_m=\frac{P_t}{{\omega }_t}$$

(4)

In Figure 3, ${\omega }_t$ and ${\omega }_r$ are the turbine and generator rotor speeds, respectively. $T_m$ and $T_e$ are the mechanical torque applied to the turbine and electrical torque, respectively; $H_t$ and $H_g$ are the turbine inertia constant and generator inertia constant, respectively; $D_t$ and $D_g$ are the damping coefficients of the turbine and generator, respectively; $T_{tg}$ is the internal torque of the model; $D_{tg}$ is the damping coefficient of the shaft between the two masses; $K_{tg}$ is the spring constant or shaft stiffness. Finally, $N_t/N_g$ is the gear ratio of the gearbox.

4.2. DFIG Model

For power system stability studies, the DFIG machine is modeled by neglecting stator transients, as they do not affect the electromechanical oscillations [44], and by neglecting stator transients, it is easier to solve stator and grid equations [45]. Similarly, the rotor electrical transients are also neglected, as rotor winding is controlled by fast-acting converters [46]. In addition, other assumptions that are made for modeling the DFIG are: the skin effect, the saturation effect, and iron losses (hysteresis and eddy currents) are neglected, as these phenomena contribute more to conducting loss performance in transient fault analysis. For designing the DFIG, a synchronous reference frame is used. The reason for representing the DFIG model in a synchronous rotating reference frame is because the $qd$-axis model is convenient for conducting steady-state analysis and deriving a small-signal model [47]. The equivalent circuit of a DFIG in a synchronously rotating reference frame is given in Figure 4.

In Figure 5, the $q$- and $d$-axes are orthogonal to each other, rotating at an angular velocity of $\omega$, whereas $a_s$, $b_s$, and $c_s$ represent stator variables displaced by ${120}^{°}$. To analyze the induction machine variable associated with the rotor, the reference frame also needs to be transformed to $qd$-axis reference frame. In Figure 5, $a_r$, $b_r$, and $c_r$ represent rotor variables displaced by ${120}^{°}$, rotating at an angular velocity of ${\omega }_r$.

The stator model is represented in the $qd$-axis reference frame.

$$v_{ds}=R_s{i}_{ds}-\frac{{\omega }_s}{{\omega }_b}{\psi }_{qs}+p{\psi }_{ds}$$

(5)

$$v_{qs}=R_s{i}_{qs}+\frac{{\omega }_s}{{\omega }_b}{\psi }_{ds}+p{\psi }_{qs}$$

(6)

The rotor model is represented in the $qd$-axis reference frame.

$$v_{dr}=R_r{i}_{dr}-\left(\frac{{\omega }_s-{\omega }_r}{{\omega }_b}\right){\psi }_{qr}+p{\psi }_{dr}$$

(7)

$$v_{qr}=R_r{i}_{qr}+\left(\frac{{\omega }_s-{\omega }_r}{{\omega }_b}\right){\psi }_{dr}+p{\psi }_{qr}$$

(8)

The mathematical model including the saturation effect is provided below [48].

$$i_{md}=i_{ds}+i_{dr}$$

(9)

$$i_{mq}=i_{qs}+i_{qr}$$

(10)

$$i_m=\sqrt{i_{md}^2+i_{mq}^2}$$

(11)

$$L_{ms}=\raisebox{1ex}{${\psi }_m$}\!\left/ \!\raisebox{-1ex}{$i_m$}\right. \& L_{md}=\frac{d}{d{i}_m}\left({\psi }_m\right)$$

(12)

The voltage equations of the DFIG with flux saturation are:

$$v_{ds}=R_s{i}_{ds}-\frac{{\omega }_s}{{\omega }_b}\left(L_s{i}_{qs}+L_{ms}{i}_{mq}\right)+L_{s}p{i}_{ds}+L_{md}p{i}_{md}$$

(13)

$$v_{qs}=R_s{i}_{qs}+\frac{{\omega }_s}{{\omega }_b}\left(L_s{i}_{ds}+L_{ms}{i}_{md}\right)+L_{s}p{i}_{qs}+L_{md}p{i}_{mq}$$

(14)

$$v_{dr}=R_r{i}_{dr}-\left(\frac{{\omega }_s-{\omega }_r}{{\omega }_b}\right)\left(L_r{i}_{qr}+L_{ms}{i}_{mq}\right)+L_{r}p{i}_{dr}+L_{md}p{i}_{md}$$

(15)

$$v_{qr}=R_r{i}_{qr}+\left(\frac{{\omega }_s-{\omega }_r}{{\omega }_b}\right)\left(L_s{i}_{dr}+L_{ms}{i}_{md}\right)+L_{r}p{i}_{qr}+L_{md}p{i}_{md}$$

(16)

where $p=\frac{d}{dt}$.

4.3. Control Strategies

To develop control strategies for the converters, any one of the axes in two reference frames $\left(qd\right)$, rotating with synchronous speed, is aligned with either the flux or the voltages of the stator. The two commonly used control strategies are stator flux-oriented control and stator voltage-oriented control. In stator flux-oriented control, the $d$-axis is aligned with the stator flux linkage vector [7]. This results in ${\psi }_{ds}={\psi }_s$ and ${\psi }_{qs}=0$. In contrast, in stator voltage-oriented control, the $d$-axis is aligned with the stator voltage linkage vector, resulting in $V_{ds}=V_s$ and $V_{qs}=0$. For rotor-side converter control, stator flux-oriented control is used, and for grid-side converter control, stator voltage-oriented control is adopted [7]. For this work, stator voltage-oriented control is implemented for both the rotor-side converter controller and grid-side converter controller, as discussed in [49]. Compared to flux-oriented control, voltage-oriented control has the advantage of deriving the model and aligning with the stator voltage space vector using the measured phase voltages. Another advantage is that, typically, the grid-side converter is controlled using the stator voltage orientation, so by choosing voltage-oriented control for the rotor-side controller, the model will be simpler to implement. The usage of voltage-oriented control on both the RSC and GSC can be observed in [49]. The objective of these control strategies is to decouple the control of active and reactive power [49].

4.4. Rotor-Side Controllers

The main objective of the rotor-side converter controller is to control both the active and reactive power of the stator. It consists of two control loops: the inner control loop regulates the $d$- and $q$-axis rotor currents, whereas the outer loop controls the active $P_s$ and reactive $Q_s$ power of the generator stator. By neglecting the transients in the stator flux linkages in the stator voltage orientation and under the assumption that resistive drops are negligible, it is observed that active power $P_s$ is independent of the $q$-axis rotor current. Under the same assumptions for the rotor model, the stator reactive power $Q_s$ is totally dependent on the $q$-axis rotor current, and it is independent of the $d$-axis rotor current. It can be concluded that, in the RSC, the stator active and reactive power can be controlled independently by rotor $d$-axis and $q$-axis currents, respectively, which generates stator active and reactive power reference current outputs $i_{dr}^{*}$ and $i_{qr}^{*}$, respectively.

The generated reference currents are fed as inputs to the inner current loops of the rotor current controller. The final outputs of the rotor-side converter controller are $v_{dr}$ and $v_{qr}$. The controller is designed based on the rotor voltage models, which can be expanded from Equations $\left(17\right)$ and $\left(18\right)$ and expressed as [49]:

$$v_{dr}=R_r{i}_{dr}-\left({\omega }_s-{\omega }_r\right)\left(L_r{i}_{qr}+L_m{i}_{qs}\right)+p\left(L_r{i}_{dr}+L_m{i}_{ds}\right)$$

(17)

$$v_{qr}=R_r{i}_{qr}+\left({\omega }_s-{\omega }_r\right)\left(L_r{i}_{dr}+L_m{i}_{ds}\right)+p\left(L_r{i}_{qr}+L_m{i}_{qs}\right)$$

(18)

When designing the rotor-side controller with decoupling elements, if the transients of the machine are neglected, the derivative terms become zero. With this condition, the $d$-axis rotor voltage will be in terms of $i_{dr}$; similarly, the $d$-axis rotor voltage will be in terms of $i_{qr}$. With these elements, a transfer function can be developed for the inner current control loops, and from this, the PI gains are obtained. From the mathematical models developed, it is observed that the design is the same for both control loops, so the $PI$ control design is also the same for both inner current control loops. In these control loops, $v_{dr1}$ and $v_{qr1}$ are the outputs of the rotor current controller loops, and they are fed to the pulse-width modulator of the converter along with the decoupled elements. Finally, these obtained signals from the modulator are fed to the converter circuit, and this is connected to the DC link of the model.

In Figure 6, $i_{dr}$ and $i_{qr}$ are the rotor currents, and they are transformed from rotor three-phase currents $i_{rabc}$ to $i_{rdq}$ by applying a transforming angle of $\left({\theta }_s-{\theta }_r\right)$, where ${\theta }_s$ is the angle obtained from $v_{sabc}$ at the grid frequency, and ${\theta }_r$ is the rotor angle. $P_s$ is the stator active power, and $Q_s$ is the stator reactive power, and these are obtained from $v_{sabc}$ and $i_{sabc}$. Reactive power reference $Q_{sref}$ is given as $0$, whereas stator active power reference $P_{sref}$ is generated using the maximum power tracking method. In this work, a simple lookup table is used, which tries to obtain the reference stator power by plotting power and speed. The other reference at $v_{dr}$ control is the generator reference value, which is set by the speed.

4.5. Grid-Side Controllers

To design the grid-side converter controller, first, the grid model in the $abc$ reference frame is transformed into two reference frames by using a phase-locked loop (PLL), which provides the required transformation angle and the frequency for synchronizing the model with the grid. The design and operation of the PLL are derived from [50], and it is shown in Figure 7 [50]. The grid model in the $abc$ reference frame is transformed to the $dq$ reference frame based on the transformation matrix from Equation $\left(19\right)$ [50].

The transformation angle $\theta$ is the grid-side converter terminal voltage angle, and it is expressed as $\theta ={\omega }_{0}t+\delta$, where $\delta$ is given as $p\delta =\omega -{\omega }_0$, where $\omega$ is the grid-side converter terminal frequency. The phase angle measured by the PLL is given as ${\theta }_{pll}$, and the measured frequency is given as ${\omega }_{pll}$; this provides the transformation angle and the frequency to the grid-side converter controller design. The obtained angle and frequency information is used for both $abc-dq$ and $dq-abc$ transformation at the grid end.

$$\left[\begin{array}{c}{v}_d\\ v_q\end{array}\right]=\frac{2}{3}\left[\begin{array}{ccc}\cos \theta & \cos \left(\theta -\frac{2\pi }{3}\right)& \cos \left(\theta +\frac{2\pi }{3}\right)\\ -\sin \theta & -\sin \left(\theta -\frac{2\pi }{3}\right)& -\sin \left(\theta +\frac{2\pi }{3}\right)\end{array}\right]\left[\begin{array}{c}{v}_a\\ v_b\\ v_c\end{array}\right]$$

(19)

Initially, for operating the PLL, the $q$-axis voltage of the stator $v_{qs}$ is obtained by the $abc-dq$ transformation technique. Once the grid-side converter terminal voltage is transformed and the transformation angle is ${\theta }_{pll}$, then the obtained equation is $v_{qs}=V\sin \left(\theta -{\theta }_{pll}\right)=V\sin \left(\delta -{\delta }_{pll}\right)$, where ${\delta }_{pll}$ is the measured phase angle and is expressed as ${\delta }_{pll}={\theta }_{pll}-{\omega }_{0}t$. The grid-side voltage frequency ${\omega }_{pll}$ is obtained by adding the error signal processed in the PI controller and ${\omega }_0$, and it is given as:

$${\omega }_{pll}=\left(k_{ppll}+\frac{k_{ipll}}{s}\right)v_{qs}+{\omega }_0$$

(20)

$$p{\delta }_{pll}={\omega }_{pll}-{\omega }_0$$

(21)

By using this three-phase PLL, the grid equations can be transformed into two reference frames. The $dq$-transformed equations of the grid model are as follows:

$$v_{ds}=r{i}_d+Lp{i}_d-{\omega }_{pll}L{i}_q+e_d$$

(22)

$$v_{qs}=r{i}_q+Lp{i}_q-{\omega }_{pll}L{i}_d+e_q$$

(23)

In Equations $\left(22\right)$ and $\left(23\right)$, $v_{ds}$ and $v_{qs}$ are the stator voltages; $r$ and $L$ are the grid filter resistance and inductance, respectively; and $i_d$ and $i_q$ are the total currents supplied to the grid in the $dq$ reference frame. Finally, $e_d$ and $e_q$ are the transformed voltages of the grid terminal. The model for the total current supplied to the grid, i.e., $i_d$ and $i_q$, are given as the sum of the stator currents $i_{dqs}$ and grid-side converter currents $i_{dqg}$. The grid-side converter voltages in the $dq$ reference frame can be expressed as:

$$v_{dg}=R_g{i}_{dg}+L_{g}p{i}_{dg}-{\omega }_{pll}{L}_g{i}_{qg}+v_{ds}$$

(24)

$$v_{qg}=R_g{i}_{qg}+L_{g}p{i}_{qg}+{\omega }_{pll}{L}_g{i}_{dg}+v_{qs}$$

(25)

Finally, the grid voltages are expressed as:

$$e_d=E\cos \left({\delta }_{pll}\right)$$

(26)

$$e_q=-E\sin \left({\delta }_{pll}\right)$$

(27)

From the above-derived grid-side model, the grid-side converter controller can be designed. The main objective of the grid-side converter controller is to regulate the DC-link voltage and exchange power between the rotor-side converter and the grid. The other objective of this controller is to control the reactive power that is delivered to the grid at the grid-side converter. Like the rotor-side converter controller, the grid-side converter controller also consists of two cascaded control loops. The DC-link voltage and the reactive power are controlled by the outer control loop, whereas the inner current control loop regulates the current components in the grid-side converter. From the above-discussed grid model, the active power and the reactive power at the grid can be expressed as:

$$P_{gc}=\frac{3}{2}\left(v_{ds}{i}_{dg}+v_{qs}{i}_{qg}\right)$$

(28)

$$Q_g=\frac{3}{2}\left(v_{qs}{i}_{dg}-v_{ds}{i}_{qg}\right)$$

(29)

For grid-side active power in these equations, by applying the synchronously rotating reference frame and aligning the $d$-axis on the grid voltage vector, the obtained results are $v_{ds}=v_s$ and $v_{qs}=0$. Applying this to Equations $\left(24\right)$ and $\left(25\right)$ yields:

$$v_{dg}=R_g{i}_{dg}+L_{g}p{i}_{dg}-{\omega }_{pll}{L}_g{i}_{qg}+v_{ds}$$

(30)

$$v_{qg}=R_g{i}_{qg}+L_{g}p{i}_{qg}+{\omega }_{pll}{L}_g{i}_{dg}$$

(31)

From the grid-side active $\left(P_{gc}\right)$ and reactive power $\left(Q_g\right),$ the outer control loops can be designed; the obtained result will be a function of DC-link voltage $v_{dc}$ and the grid-side converter current $i_{dg}$ [7,49]. By using this, an independent control loop is developed for DC-link voltage $v_{dc}$, with $i_{dg}^{*}$ as its output. Similarly, the grid-side reactive power can also be controlled individually. From this outer loop, $i_{qg}^{*}$ is obtained. Similarly, for the inner control loop design, the same reference frame is applied. The inner control loops are mainly designed as current control loops, which use grid-side currents as inputs. The GSC control block is given in Figure 8 [7].

4.6. Grid-Side Inner Current Control Loop

With Equations (30) and (31), the inner current control loop can be designed. In Equation (30), ${\omega }_{pll}{L}_g{i}_{qg}$ and $v_{ds}$ are fed as inputs at the outer end of the controller design, and in Equation (31), ${\omega }_{pll}{L}_g{i}_{dg}$ is fed at the outer end. ${\omega }_{pll}{L}_g{i}_{qg}$ and ${\omega }_{pll}{L}_g{i}_{dg}$ act as decoupling elements for the grid-side converter controller. The final obtained equations are:

$$v_{dg}=R_g{i}_{dg}+L_{g}p{i}_{dg}$$

(32)

$$v_{qg}=R_g{i}_{qg}+L_{g}p{i}_{qg}$$

(33)

From Equations (32) and (33), the PI controller can be designed, and it can be observed that both controllers have similar structures. Due to the very short sampling period and as the system requires a fast response rate, the controller is limited to a PI controller. Moreover, with the usage of power electronic devices and with huge variations in wind speed, which generate a lot of noise in the system, adding a derivative controller would result in an undesirable simulation result.

5. Q-Learning (QL) and Twin Delayed Deep Deterministic Policy Gradient (TD3)

Reinforcement learning (RL) algorithms are focused on goal-directed learning from iterations, which are mainly used to solve closed-loop problems. RL uses actions from learning systems that influence the later inputs [28]. An RL algorithm consists of a discrete set of environment states $S$, a discrete set of agent actions $A$, and a set of scalar reinforcement signals $R$. Here, the agent interacts with the environment through action, and the agent receives the current state as input; then, the agent chooses an action to generate an output. The final goal of any RL algorithm is to increase the long-run sum of values of the reinforcement signals, which can learn over time by the trial-and-error method and solve the problem [51].

The environment in reinforcement learning is fully observable and can be described as a Markov decision process (MDP), and most RL problems are formalized as MDPs. In an MDP, the action taken in the current state also affects the next state and not just the current state itself, so action plays a dominant role. Due to the action in the current state, a return reward will be assigned to the corresponding state–action pair [28].

Of the various available RL algorithms, the Q-learning algorithm is considered simple and easy to implement due to the simple way in which agents can learn and act optimally in controlled Markovian domains. The other main advantage of the Q-learning algorithm is that it is exploration-insensitive: Q values will converge to the optimal values, independently of how the agent behaves while the data are being collected [51,52]. With these advantages, this paper uses a Q-learning algorithm to train the agent to suppress oscillations and provide stable power to the grid under variable wind speeds. Assuming that the best action is taken initially, the Q-learning optimal value function is taken from [51].

$$Q\left(s,a\right)=R\left(s,a\right)+\gamma {\displaystyle \sum }_{s^{\prime }\epsilon S}T\left(s,a,s^{\prime }\right)\max Q\left(s^{\prime },a^{\prime }\right)$$

(34)

In the above equation, $Q\left(s,a\right)$ is the expected discounted reinforcement of choosing action $a$ in state $s$. Once the action is taken, the agent will be given a reward $R$ for the effectiveness of the action by observing the resulting state, $s,$ of the environment. Here, $T$ is the probability of action $a$ applied to state $s$ that changes the state from $s$ to $s^{\prime }$. For each action executed, the $Q$ values will converge with a probability of 1 to $Q^{*}$, and when the $Q$ values are nearly converged to their optimal values, the agent will act greedily by taking the action with the highest $Q$ value; from this greedy policy, the optimal action is determined. At any time step, there is at least one action whose estimated value is greatest or optimal; choosing the greatest value is called greedy action [28]. $\gamma \left(0\le \gamma \le 1\right)$ is the discount factor, which discounts the rewards exponentially in the future [51]. Typically, an agent will look up the Q-memory lookup table, which has state $s$ and action $a$ and is updated as per [34,51,52]. The parameter $\alpha \left(0\le \alpha \le 1\right)$ in Equation (35) updates the Q-memory and affects the number of iterations [34].

$$Q\left(s,a\right)=\left(1-\alpha \right).Q\left(s,a\right)+\alpha (r+\gamma \max \left(Q\left(s^{\prime },a^{\prime }\right)\right)$$

(35)

Even though the Q-learning method is simple to implement, the agent cannot learn under conditions with huge variations, which is one of the observations from the current research work and a limitation of the QL agent. The other problem with the QL method is that the Q-table must be limited to certain states and actions, and the Q-table cannot be updated for a large state–action space. Therefore, the current research work explored and implemented the TD3 method so that the Q-table limitation can be overcome, and the agent can adapt to large variations in conditions.

The TD3 method is one of the model-free policy-based deep reinforcement learning algorithms built on the DDPG method [53]. The objective of the TD3 method is to increase the stability and performance by considering the function approximation error [53]. In an actor-critic setting, the learning target is:

$$y=r+\gamma Q_{{\theta }^{\prime }}\left(s^{\prime },{\pi }_{\varnothing }\left(s^{\prime }\right)\right)$$

(36)

where $y$ is the learning target, $r$ is the reward received for every action, $s^{\prime }$ is the new state of the environment, $\gamma$ is the discount factor, ${\pi }_{\varnothing }$ is the optimal policy, and $Q_{\theta }\left(s,a\right)$ is the function approximator with parameter $\theta$.

The stability and performance are increased by applying three modifications [53]:

a.

Clipped double Q-learning:

In this update, the value target cannot introduce any additional overestimation using the standard Q-learning target. With a pair of actors $\left({\pi }_{{\varnothing }_1},{\pi }_{{\varnothing }_2}\right)$ and critics $\left(Q_{{\theta }_1},Q_{{\theta }_2}\right)$, the target update is

$$y_1=r+\gamma mi{n}_{i=1,2}{Q}_{{\theta }^{\prime }}\left(s^{\prime },{\pi }_{\varnothing }\left(s^{\prime }\right)\right)$$

(37)

In this update, the computational costs are reduced, as a single actor is optimized with respect to $Q_{{\theta }_1}$, and the same target is used to update $Q_{{\theta }_2}$.

b.

Target networks and delayed policy updates:

In this update, the target network is used to reduce the error over multiple updates. The policy network is updated at a lower frequency than the value network to minimize the error. The modification is made to update the policy and target networks after a fixed number of updates to the critic [53]. Thus, very few policy updates are made with this modification, and policy updates are not repeated for an unchanged critic.

c.

Target policy smoothing regularization:

In this approach, the relationship between similar actions is forced explicitly by modifying the training procedure, which is carried out by fitting the value of a small area around the target action. This will have the benefit of smoothing the value estimate by bootstrapping off a similar state-action value estimate [53]. The expectation over actions is approximated by adding a small amount of random noise to the target policy, where the noise is kept close to the original action.

The modified target update is:

$$y=r+\gamma Q_{{\theta }^{\prime }}\left(s^{\prime },{\pi }_{\varnothing }\left(s^{\prime }\right)\right)+ϵ$$

(38)

$$ϵ~clip\left(N\left(0,\sigma \right),-C,C\right)$$

(39)

6. Deep Reinforcement Learning-Based WECS

6.1. Design of PSS

In this paper, the PSS is designed based on the transformation technique. The block diagram for the PSS with the transformation technique is shown in Figure 9.

The signal ${\theta }_f$ is obtained from the PLL frequency (${\omega }_{pll}$):

$${\theta }_f=\left(K_t{\omega }_{pll}\right)\left(\frac{s{T}_w}{1+s{T}_w}\right)\left(\frac{1}{1+s{T}_1}\right)$$

(40)

The transformation is defined as:

$$i_{dr}^{*}=\cos \left({\theta }_f\right)i_{drref}-\sin \left({\theta }_f\right)i_{qrref}$$

(41)

$$i_{qr}^{*}=\sin \left({\theta }_f\right)i_{drref}+\cos \left({\theta }_f\right)i_{qrref}$$

(42)

The PSS developed with the transformation technique is implemented on the inner current controller of the RSC on both $d$- and $q$-axis control loops. The block diagram with the transformation technique is given in Figure 10.

6.2. Q-Learning Algorithm on RSC

The main motivation for using the Q-learning algorithm is to suppress low-frequency oscillations generated by variable wind speeds by searching for the best action in each state. To ensure continuous exploration and avoid tending to the local optimum, a pursuit algorithm based on the learning automata algorithm is utilized for the action policy. Initially, actions are determined with a uniform probability distribution. When the Q-table is updated, the probabilities of actions are updated as follows [54].

$$\{\begin{array}{c}{T}_s^{k+1}\left(a_g\right)=T_s^k\left(a_g\right)+\beta \left(1-T_s^k\left(a_g\right)\right)\\ T_s^{k+1}\left(a\right)=\left(1-\beta \right)T_s^k\left(a_g\right) \forall a\in A, a\ne a_g\\ T_{\tilde{s}}^{k+1}\left(a\right)=T_{\tilde{s}}^k\left(a\right) \forall a\in A, \forall \tilde{s}\in S, \tilde{s}\ne s \end{array}$$

(43)

where $T_s^k\left(a_g\right)$ denotes the probability that an action–state pair $\left(a_g,s\right)$ is selected in iteration k, and $\beta$ represents the action exploration rate. After the effect of this algorithm, $Q^k$ tends to $Q^{*}$ for a sufficiently large $k$, and an optimal policy is obtained. The specific Q-learning-based adaptive parameter algorithm proposed in this paper is summarized as follows:

An episode in Algorithm 1 is defined as a notion of the final time step when the agent–environment interaction breaks naturally into subsequences [28].

Algorithm 1 Q-learning-Based Adaptive Parameter in Rotor-Side Algorithm

For each episode do      Initialize $Q^0\left(x, a\right)=0, \forall a\in A, \forall s\in S$      Initialize $T_x^0\left(a\right)=\frac{1}{\left|A\right|}, \forall a\in A, \forall s\in S$   For each step of episode do        Choose a from s based on the current distribution $T_x\left(a\right)$        Take action a, observe r, $s^{\prime }$        Update $Q\left(x, a\right)$ according to Equation (40)     Update $T_x\left(a\right)$ according to Equation (43)      $s\leftarrow s^{\prime }, a\leftarrow a^{\prime }$   End for End for

The damping of frequency oscillations discussed in this paper can be formulated as an MDP problem because the future state of the closed-loop controller is always dependent on the current state. Since the problems considered by reinforcement learning can be modeled as MDP models, we thus transform the adaptive parameter problem into an MDP model by designing a five-tuple $\left(S,A,P,\gamma ,r\right)$ as follows:

Design of state space $S$:$S$ is a set of states that represent configurations of the system. It is assumed that all possible states are finite. To damp the low-frequency oscillations of the power system and obtain stable power output, we use active power $\Delta P$ as the state information.

The states of the MDP are described as follows: $\Delta P$ state-space (in per-unit value) is discretized into 11 spaces: $\left(-\infty ,-0.5\right]$, $\left(-0.5,-0.3\right]$, $\left(-0.3,-0.1\right]$, $\left(-0.1,-0.05\right]$, $\left(-0.05,-0.02\right]$, $\left(-0.02,-0.01\right]$, $\left(-0.01,-0.005\right]$, $\left(-0.005,-0.002\right]$, $\left(-0.002,0.002\right]$, $\left(0.002,0.01\right]$, $\left(0.01,\infty \right)$. We can see that the state distribution is unbalanced around zero, because the deviation of the active power at the rated value is also unbalanced with wind fluctuation. The QL agent is trained with 10 states and 10 action pairs. The state at iteration zero is chosen at random, and at each incremental step, the state will be in one of the intervals. Therefore, the states are divided into various sections based on the simulation results using a PSS with a PI controller, where the controlled variable is monitored. In this case, it is the active power that is a feedback parameter dependent on various other system parameters. The state space is divided into ten interval sates between an interval of -∞ to ∞. Each state consists of an interval; as the system is dynamic and continuous, the states are chosen between the intervals. The observation from the small-signal model and the simulation model with the PSS is that the change in power is within the range of −0.5 to 0.01. The intervals for each state are chosen randomly; however, the number of states is chosen based on the action.

Design of action space $A$: $A$ is a set of actions that are executed by the agent to influence the environment. As mentioned before, the output of the agent should be the controller parameter at the rotor side. In this paper, we obtain the discrete action as follows: A = [−0.025 −0.02 −0.015 −0.01 −0.005 0.005 0.01 0.015 0.02 0.025]. The action space is chosen based on the output signals from the PSS controller, which are replaced by the QL agent. These are the list of actions that are provided to the agent to determine which action to choose at its corresponding state. In state 1, the action chosen can be any action and is not necessarily action 1. The Q-table, which is updated as per Algorithm 1, will choose either the action with max Q or a random Q, which is updated as per Equation (40). Even though any action can be chosen, the trained agent will use the Q values from the trained model. Therefore, under varying wind speeds, the agent will know which action to choose based on the chosen state from the updated Q values.

$P:S\times A \to \theta \left(S\right)$ is the state transition function that shows the distribution of the next state $s_{k+1}$ after executing an action $a_k$ in the environment with the current state $s_k$. Since the parameter variation model is unknown, we can use the temporal-difference (TD) method to train the adaptive parameter policy. TD learning is a combination of Monte Carlo and dynamic programming ideas. Q-learning is an off-policy TD control algorithm [28]. In this paper, we apply a Q-learning algorithm to optimize the parameter of the rotor-side controller.

Design of reward function $r\left(s,a\right)$: $r\left(s,a\right)$ is the function that maps the state–action pair $\left(s,a\right)$ to a scalar, which represents the immediate reward after applying an action $a$ to the environment with state $s$. In this paper, $r_k=-k\times \left|P-P_{ref}\right|$. This means that the more the active power deviates from the reference value, the smaller the immediate reward, which prompts the adjustment of the controller parameters so that the active power reaches the reference value. The parameter β used in Equation (43) is used to update the probability distribution. The value of β is 0.1. The value of α is 0.2.

The control application of QL on the RSC is employed as per Figure 11.

In Figure 11, it can be observed that the PI controller, which is developed for the PSS with the transformation technique, is completely replaced with the Q-learning algorithm.

6.3. Q-Learning Algorithm for DC-Link Voltage Control on GSC

As discussed in the above section, the primary objective of the Q-learning algorithm used is to suppress the low-frequency oscillations generated by variable wind speeds by searching for the best action in each state. On the GSC, the sensitive parameters are the DC-link voltage and grid-side current, which are observed from the small-signal model. For the action policy, a pursuit algorithm based on the learning automata algorithm is utilized. Initially, actions are determined with a uniform probability distribution. When the Q-table is updated, the probabilities of actions are updated using Equation (43). The Q-learning algorithm discussed for the RSC (Algorithm 1) is used to update the episodes on the GSC as well; the main difference is in the action and reward that are chosen for each controller. For the GSC, the action space is designed for control parameters $K_{p5}$ and $K_{i5}$, and the reward function is defined from the DC-link voltage. The new controller design with QL on the grid-side converter is shown in Figure 12.

It can be observed that the PI controllers on the GSC are replaced with the QL algorithm. The adaptive parameter problem can be transformed into an MDP model by designing a five-tuple $\left(S,A,P,\gamma ,r\right)$ as follows:

Design of state space $S$: To damp low-frequency oscillations, the sensitive parameters observed from the GSC are the DC-link voltage and input reference current to the current control loop, so DC-link voltage $\Delta V_{dc}$ is used as state information.

The states of the MDP are described as follows: On the GSC, the same state spaces defined for the RSC can be used, as the objective is to provide stable power to the grid, which is a function of active power from the RSC and GSC. In addition, the deviation of the active power at the grid-side converter is a function of $V_{dc}$, which is also unbalanced with wind fluctuation. However, the action space will be completely different from the RSC action spaces, as the control parameters differ from each other.

Design of action space $A$: The output of the agent is a controller parameter $K_{p5}$ and $K_{i5}$ at the grid side. In this paper, we obtain the discrete action as follows:

A = [−0.19 −0.186 −0.18 −0.176 −0.172 −0.17 −0.168 −0.164 −0.16 −0.15].

The action space on the GSC is chosen based on the PI controllers, which are replaced with the QL agent. However, the state space will remain the same, as the end output is the active power, which is monitored and controlled by the agent.

$P:S\times A \to \theta \left(S\right)$ is the state transition function that shows the distribution of the next state $s_{k+1}$ after executing an action ak in the environment with the current state.

Design of reward function $r\left(s,a\right)$: In this paper, $r_k=-k\times \left|V_{dc}-V_{dc}^{ref}\right|$. This means that the more the grid-side active power deviates from the reference value, the smaller the immediate reward, which prompts the adjustment of the controller parameters so that the voltage reaches the reference value.

A complete block diagram with the QL-controller units discussed above can be observed in Figure 13.

In Figure 13, $V_w$ is the wind speed, ${\omega }_r$ is the generator rotor speed, and ${\theta }_r$ is the rotor angle, which is used for transformation from the $abc-dq$ reference frame for rotor currents. $V_{sabc}$ and $I_{sabc}$ are stator voltage and currents, respectively, whereas $V_{abc}$ and $I_{abc}$ are grid voltage and currents, respectively. Grid voltage $V_{abc}$ is provided as input to the phase-locked loop, which extracts the angle ${\theta }_{pll}$. ${\theta }_{pll}$ is used for transformation from the $abc-dq$ reference frame for stator voltage, stator current, grid voltage, and grid currents. In Figure 13, it can be observed that voltages fed to the rotor-side converter $\left(V_{r\_abc}\right)$ and grid-side converter $\left(V_{gc\_abc}\right)$ are provided by the QL controllers, which are discussed in this section.

6.4. TD3 Method

As discussed in Section 5, the TD3 method is derived from the DDPG method with a few modifications. In this section, the TD3 algorithm and its implementation are discussed. One of the objectives of this research work is to control the frequency under huge variations in speed by replacing the PI controllers, as the PI controllers cannot provide or produce satisfying tracking performance with huge variations in wind speed. Thus, the TD3 method is implemented, which provides nonlinear control. In this paper, the inner current PI controllers are completely replaced with the TD3 algorithm. The TD3 agent is implemented in an environment where the observation space is continuous or discrete, and the action space is continuous. In this experience, both the observation and action spaces are continuous. When it comes to the actor and critic, the actor is a deterministic policy actor, and the critic can be one or more Q-value function critics $Q\left(S,A\right)$. The training process consists of three steps. In step 1, the actor and critic properties are updated at each iteration by the agent. In step 2, the experience buffer is used to store past experiences. From the experience buffer, the actor and critic use a mini batch of experiences randomly. In step 3, noise is applied to the action chosen by the policy using the stochastic noise model at each episode.

TD3 implementation has four stages. In stage 1, the actor and critic functions are chosen. The actor is a deterministic actor that returns the action that maximizes the long-term reward with input with parameters $\theta$ and state $S$. Apart from the actor, to choose the best action, a target actor is also developed to improve the stability; the target actor parameter ${\theta }_t$ is updated using the latest actor parameters. There are two critics: one is the value critic, and the second critic is the target critic. The value critic takes state $S$ and action $A$ as inputs and provides the expectation of the long-term reward. The target critic is responsible for improving the stability of the optimization. Target critic parameters are updated using the latest critic parameter values. In stage 2, the agent is created with the state and action specifications of the environment. As the agent has both actor and critic networks in the environment, in stage 3, the agent is now trained to learn and update the actor and critic models at each episode. Training is implemented as per Algorithm 2 [53]. In stage 4, the target actor and the target critic parameters are updated using one of the target update methods, such as periodic, smoothing, and periodic smoothing. However, for this research work, the smoothing update is chosen.

Algorithm 2 TD3 [53]

Initialize critic networks $Q_{{\varnothing }_1},Q_{{\varnothing }_2}$ with random parameters ${\varnothing }_1$ Initialize target critic with same random parameters ${\varnothing }_2$; so ${\varnothing }_1={\varnothing }_2$ Initialize actor network ${\pi }_{\theta }$ with random parameters ${\theta }_1,{\theta }_2$ Initialize target actor network ${\pi }_{\theta }$ with same random parameters ${\theta }_2$ So, for target networks ${\theta }_1^{\prime }\leftarrow {\theta }_1, {\theta }_2^{\prime }\leftarrow {\theta }_2,{\varnothing }^{\prime }\leftarrow \varnothing$ Initialize replay buffer $\beta$ For t = 1 to T do     For current state of observation $S,$ select action with exploration noise $a~{\pi }_{\varnothing }\left(s\right)+\epsilon ,$ $\in ~N\left(0,\sigma \right)$. Here, $\in$ is the stochastic noise from the noise model Execute action and observe reward $r$ and new state $s^{\prime }$. Store the experience $\left(s,a,r,s^{\prime }\right)$ in $\beta$ (experience buffer) Sample a random mini batch of $N$ transitions $\left(s,a,r,s^{\prime }\right)$ from $\beta$ If $s^{\prime }$ is a terminal state, set the value function target $y_i=r;$ else      $\tilde{a}\leftarrow {\pi }_{{\varnothing }^{\prime }}\left(s^{\prime }\right)+ϵ, ϵ~ clip\left(N\left(0,\sigma \right), -C,C\right)$        $y \leftarrow r+\gamma mi{n}_{i=1,2}{Q}_{{\theta }^{\prime }}\left(s^{\prime },\tilde{a}\right)$ Update critics ${\theta }_i\leftarrow argmi{n}_{{\theta }_i}{N}^{-1}{{\displaystyle \sum }}^{ }{\left(y-Q_{{\theta }_i}\left(s,a\right)\right)}^2$ If $t$ mod $d$ then     Update $\varnothing$ by deterministic policy gradient:         ${\nabla }_{\varnothing }J(\varnothing )=N^{-1}{\displaystyle \sum }{\nabla }_a{Q}_{{\theta }_1}\left(s,a\right){|}_{a={\pi }_{\varnothing }\left(s\right)}{\nabla }_{\varnothing }{\pi }_{\varnothing }\left(s\right)$    Update target networks (smoothing):    ${\theta }_i^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$    ${\varnothing }^{\prime }\leftarrow \tau \varnothing +\left(1-\tau \right){\varnothing }^{\prime }$ end if end for

In the current work, the inner current controller on the rotor-side controller is replaced with the TD3 agent. Therefore, the inputs for the TD3 agent are $f\left(i_{dr},i_{qr},i_{dr\_ref},i_{qr\_ref},{\omega }_r,{\omega }_{r\_ref}\right)$. Here, $i_{dr},i_{qr}$ are rotor-side currents, and $i_{dr\_ref},i_{qr\_ref}$ are the outputs of the outer current control loop on the RSC. ${\omega }_r$ and ${\omega }_{r\_ref}$ are the rotor speed and reference speed, respectively. Both the current and speed with their references are provided as inputs or are used as the parameters for observation. The reward is computed by taking the error between the reference values and the actual values of currents. The reward $R$ is computed as shown in Equation (44).

$$R_t=-\left(Q_1\ast {\left(i_{dr}-i_{d_{r_{ref}}}\right)}^2+Q_2\ast {\left(i_{qr}-i_{q{r}_{ref}}\right)}^2+R\ast {\displaystyle \sum }{\left(a_{t-1}^j\right)}^2\right)-100d$$

(44)

Therefore, the number of observations is six, which are the inputs to the TD3 agent, and the number of actions is two, $v_{dr}$ and $v_{qr}$. As we have the observations and action space available, the next step is to create the agent block, as shown in Figure 14.

For the TD3 agent, two critic networks are created with a deep neural network with state and action as inputs and one output. The neural network developed is a fully connected layer for both state and action paths. In the same way, for both actor networks, fully connected layers are used. The TD3 agent will determine the action that needs to be taken for the given state. To train the agent, the TD3 agent is provided with its discount factor, the buffer length, mini-batch size, target smoothing factor, and finally, the target update frequency. For training, the agent is allowed to run each training for 1000 episodes and stop training when the agent receives a cumulative average reward ≥ −200 over 100 consecutive episodes. Training progress with the TD3 agent is shown in Figure 15, which shows that the average reward of each episode increases, and the policy becomes stable after 40 episodes. The training process was executed for a duration of 32 h, as it reached the specified maximum episode numbers. In Figure 15, the thick blue line represents the average reward, and the light blue line indicates the episode reward and the yellow line shows the episode $Q_0$.

7. Results and Discussion

7.1. Simulation Results with the Newly Designed PSS

The model discussed in this paper was built in MATLAB/Simulink. The topology is shown in Figure 13. There have been extensive works published based on simulations [4,10,35,55]. A small-signal model was developed to identify the state variables that affect the stability of the overall system. From the small-signal model, the sensitive variables observed are $i_{dr}$, $i_{qr}$, $i_{qs}$, $x_7$, $v_{ds}$, and ${\theta }_{pll}$. The input given to the power system stabilizer is the frequency ${\omega }_{pll}$. One of the main observations from small-signal stability analysis is that the state variables associated with the inner current control loop tend to move faster towards the real axis, making the system unstable. This is observed with the large-signal model as well when applying small faults. The performance evaluation of the system with a PSS and without a PSS can be observed from the small-signal model. The low-frequency mode of the system can be observed from the same small-signal model. Usage of the eigenvalue distribution of systems with and without a PSS to identify the ultra-low-frequency oscillation mode can be observed in [36]. The shift of eigenvalues with the new PSS implemented is noted in

Table 1. Small-signal model eigenvalues and state variables.

	$\mathit{\lambda }\mathbf{=}\mathit{\sigma }\mathbf{\pm }\mathit{j}\mathit{\omega }$	State Variables $\mathbf{\Delta }\mathit{X}$
${\lambda }_4,{\lambda }_5$	$-5.02\pm 389.67i$	$i_{dr} \mathrm{and} i_{qr}$
${\lambda }_6,{\lambda }_7$	$-73.54\pm 376.36i$	$i_{qs} \mathrm{and} x_7$
${\lambda }_8,{\lambda }_9$	$-64\pm 37.08i$	$V_{dc} \mathrm{and} x_5$
${\lambda }_{12},{\lambda }_{13}$	$-10.34\pm 8.47i$	$v_{qs} \mathrm{and} {\theta }_{pll}$
${\lambda }_{19},{\lambda }_{20}$	$-0.0019\pm 0.93i$	$x_2, x_4$
${\lambda }_{11},{\lambda }_{21}$	$-16.16\pm 1.79i$	$x_1 \mathrm{and} {\theta }_f$

below. The eigenvalues of $\left(E,A\right)$ $x_1 \mathrm{and} {\theta }_f$ are newly added signal inputs from the PSS. The time constants for the washout filter and compensator are determined from the small-signal model by observing the stability of the system. $V_{dc}$ and $x_5$ are the signal inputs of the outer current control loop of the grid-side converter. $x_2$ and $x_4$ are the input signals to the PI controllers $\left(P{I}_2 \mathrm{and} P{I}_4\right)$ at the inner current control loop of the RSC. $x_7$ is the input signal to the PI controller $\left(P{I}_7\right)$ at the inner current control loop of the GSC. Controller data, DFIG, and the complete system parameters are presented in Appendix A.

The results for the power system stabilizer with the transformation technique are shown below in Figure 16.

From Figure 16 and Figure 17, it can be observed that there is an increase in the damping of the oscillations for both active power and generator speed with the PSS implemented with the transformation technique.

7.2. Fault Analysis with Transformation PSS

The plot of the transformation PSS during the fault condition is shown below. It can be observed that the oscillations are damped even during the fault condition. This can be more clearly observed from the power plot, in which the oscillations are damped, and the settling time is also much less. From this discussion, it can be concluded that the PSS is working effectively during the fault condition.

For fault analysis, a three-phase fault was applied to the system for a time of $\tau$ = 2 s between times $\tau$ = 30 s and $\tau$ = 32 s. The results obtained for active power and reactive power are shown in Figure 18a,b, respectively. In the above plot, it can be observed that the PSS with voltage as input is able to damp the oscillations during the fault, and the settling time after the fault is also much less.

Figure 19 provides the simulation results for the system during a single-phase short circuit fault. The fault was applied to the system for $\tau$ = 5 s between times $\tau$ = 60 s and $\tau$ = 65 s. It can be observed that the PSS is able to damp the oscillations.

7.3. Results with Q-Learning Algorithm

Next, a Q-learning algorithm was implemented on the developed PSS at the RSC and the outer current control loop of the GSC with a variable wind speed. The algorithm model was applied at $P_{sref}$ and $V_{dc}^{ref}$. Since the control objective is to stabilize the output of the power system with active and reactive power without low-frequency oscillations, this paper uses parameters that are directly related to the active power change in the power system as the state of the agent and the control output of the controller as the action of the agent to train the appropriate control strategy. After testing, when the wind speed fluctuates in a small range, it is easier to stabilize the system through the reinforcement learning controller. Therefore, the wind speed of this experiment was designed to be between 14 m/s and 15 m/s.

The following is the output waveform of the active power of the power system after several iterations of the reinforcement learning controller. The active power waveform is observed at iteration 18, which can be observed in Figure 20.

In Figure 20, it can be observed that the generated power tends to be smooth without any oscillations, which indicates that the QL algorithms implemented were able to learn from the system and provide the desired output.

However, the final active power is still not smooth. Next, the system was iterated until the desired active power was generated without any oscillations, which was observed at the 21st iteration. Figure 21 shows the response of the active power at the 25th iteration. After this point, the system response will not differ, as the Q-learning agent is able to learn about the system in various states and provide the required action at different states. In Figure 20, Figure 21, Figure 22, Figure 23 and Figure 24, it is observed that under the action of the trained reinforcement learning controller, the system can output stable active power without low-frequency oscillations. The total time taken for 25 episodes or iterations is 90 min.

Figure 22 shows the control of reactive power at the 25th iteration. It can be observed that the QL agent is able to learn from the environment and control reactive power as well.

The output speed and power as the wind speed changes are plotted in Figure 23. As can be seen in Figure 23a, when the wind speed changes, the speed and active power of the power system are stabilized.

7.4. Comparing Q-Learning Algorithm with PI Controllers

The Q-learning algorithm developed completely replaces the PI controllers both on the RSC and GSC and the power system stabilizer at the RSC. From the results, it is observed that the model-free algorithm can learn from the system environment and generate similar results to those of the PI controller.

In Figure 24, the dotted red line indicates the curve for active power with the QL algorithm, and the continuous curve shows the active power with PI controllers. It can be observed that the Q-learning algorithm controls the active power similarly to the PI controller. The advantage of the developed Q-learning model over the PI controller design is that the Q-learning agent provides the controller parameters dynamically, whereas the gains in the PI controller are fixed; moreover, the Q-learning algorithm is implemented with varying wind speeds, whereas the PI controller is implemented with a constant wind speed. Figure 25 shows a comparison of the reactive power for the developed PSS with the PI controller and Q-learning algorithm.

7.5. Comparing Q-Learning Algorithm with PI Controllers under Fault Conditions

To evaluate the control of the Q-learning algorithm under fault conditions, a three-phase fault was employed at the grid end before the transformer. The clearing time was 2 s, starting at 30 s and clearing at 32 s. Figure 26 and Figure 27 show the active power and reactive power plots, respectively, under fault conditions. In Figure 26, the red dotted line is the active power with the Q-learning algorithm, and the continuous line shows the curve for the PI controller. It can be observed that the damping of oscillations is more effective with the Q-learning algorithm when compared to the PSS with PI controllers.

In Figure 27, the red dotted curve indicates the reactive power with the Q-learning algorithm, and the continuous curve is with the PI controller. In the figure, it can be observed that the damping of oscillations is more effective with the Q-learning algorithm when compared with the PI controllers.

The proposed Q-learning algorithm can be compared with the PSS using the Q-learning algorithm from [35]. In [35], the case study used is a four-machine two-area system where a Q-learning-based PSS is employed to damp the frequency oscillations. The same can be observed in the current paper; however, the systems used are completely different. Moreover, in the current paper, the performance of Q-learning is observed under fault conditions as well. In Figure 24, Figure 25, Figure 26 and Figure 27, a clear comparison can be observed between the PSS with the PI controller and Q-learning. Here, the PSS with the PI controller can be treated as a test case to evaluate the performance of the Q-learning-based model. In Figure 24, Figure 25, Figure 26 and Figure 27, the proposed Q-learning-based model reaches the steady state in a shorter time compared with the classical PI controller-based PSS.

7.6. Comparing TD3 Agent with Q-Learning Algorithm

Simulations were carried out with large variations in wind speed using PI, Q-learning, and the TD3 agent. The wind speeds used are from the Ottawa region for the whole month of June 2019.

The wind speed variation shown in Figure 28 was used as an input to the DFIG WECS. With varying wind speeds, it is hard for the regular PI controller to maintain the frequency and, at the same time, provide a table output to the grid. The average wind speed variations can be observed in Figure 29. The active power and reactive power with the PSS with QL vs. the PSS with the TD3 agent can be seen in Figure 30 and Figure 31, respectively. It can be observed that both active power and reactive power are not stable with such huge wind speed variations in either the PSS with the PI controller or the Q-learning agent. However, once the TD3 agent implemented on the inner current control loop of the RSC learns the system dynamics, the agent can control the generator speed and inner currents with respect to their reference values. Therefore, with large variations in wind speed and without using an inner PI controller, with the TD3 agent, the system can produce stable active and reactive power with varying wind speeds by mitigating lower frequencies. The system with the PI controller does not even respond to large variations in wind speed, as it can work only for a constant wind speed.

Figure 32 and Figure 33 show the DC-link voltage and generator speed comparison between the QL agent and TD3 agent. In the figures, it can be observed that the TD3 agent can learn and control the system under large variations in wind speed and performs better than the QL agent.

8. Limitation and Future Work

The wind DFIG system with the PSS and PI controllers works fine for a constant speed, which is observed under normal and fault conditions. If a varying wind speed is provided, then the PSS cannot suppress any oscillations, and the system is very unstable. The PSS with the PI controller cannot be adopted or used for large variations in wind speed. To overcome the limitation of varying wind speeds, reinforcement learning is used to learn the system dynamics and mitigate oscillations. For this, first, the PSS and the PI controllers from both the RSC and GSC are replaced with the QL agent. The Q-learning algorithm developed in this paper is implemented for a very small range of varying wind speeds and does not respond accurately for large variations in wind speed. The observation is that the QL agent is not successful in learning the system dynamics with huge variations in wind speed. To overcome the limitation of the application of an RL to large variations in wind speed, advanced RL techniques such as actor-critic (A2C or A3C) or policy gradient methods such as Deep Deterministic Policy Gradient (DDPG) will be helpful, as these methods are robust in learning about the environment. To handle large variations in wind speed, the TD3 method is introduced in this research. Usage of the A3C-based strategy can be observed in [36], where the PSS parameters were tuned to suppress low-frequency oscillations for a 10-machine 39-bus transmission network. Thus, A2C or A3C can be adopted to suppress low-frequency oscillations for large variations in wind speed in DFIG-based WECS. As discussed at the beginning of Section 7, the complete model was built in the MATLAB/Simulink environment. For this research work, there was no experimental hardware involved. The complete results and comparisons are discussed in Section 7.

9. Conclusions

In this work, a DFIG-equipped WECS was developed, and closed-loop controllers were designed to damp the oscillations by controlling both active and reactive power. First, a small-signal model was developed to identify the sensitive parameters that affect the stability of the system, and from the SS model, the proportional and integral gains were derived. The derived PI gains were used in the inner current control loops of the large-signal model. Next, the PI controllers were completely replaced with the Q-learning-based RL technique. The Q-learning-based model-free power system stabilizer and DC-link voltage regulator were developed on both the rotor side controller and grid side controller. From the results, it is observed that the designed model-free PSS can damp the oscillations under small wind speed variations and faulty conditions. The conclusion is that the QL algorithm agent can learn from the environment and control the active power by helping the system to operate under normal conditions with variable wind speeds by making the system model-free. However, the limitation of the QL method is that it cannot control under large variations in wind speed. To overcome this limitation, an actor-critic method called the TD3 method is introduced. From the results, it can be observed that the TD3 agent can learn the system dynamics under large variations in wind speed, and the agent can deliver the desired active and reactive power to the grid.