Deep and Reinforcement Learning in Virtual Synchronous Generator: A Comprehensive Review

Abstract

The virtual synchronous generator (VSG) is an important concept and primary control method in modern power systems. The penetration of power-electronics-based distributed generators in the power grid provides uncertainty and reduces the inertia of the system, thus increasing the risk of instability when disturbance occurs. The VSG produces virtual inertia by introducing the dynamic characteristics of the synchronous generator, which provides inertia and becomes a grid-forming control method. The disadvantages of the VSG are that there are many parameters to be adjusted and its operation process is complicated. However, with the rapid development of artificial intelligence (AI) technology, the powerful adaptive learning capability of AI algorithms provides potential solutions to this issue. Two research hotspots are deep learning (DL) and reinforcement learning (RL). This paper presents a comprehensive review of these two techniques combined with VSG control in the energy internet (EI). Firstly, the basic principle and classification of the VSG are introduced. Next, the development of DL and RL algorithms is briefly reviewed. Then, recent research on VSG control based on DL and RL algorithms are summarized. Finally, some main challenges and study trends are discussed.

1. Introduction

At present, the power system is experiencing ongoing transformation and reconstruction in order to be more flexible, sustainable, distributed, environmentally friendly, and intelligent. Traditional large thermal power plants are gradually replaced by distributed generators (DGs) with renewable energy sources (RESs), such as wind and solar. This means a shift from centralized power generation and long-distance power transportation to distributed generation and regional energy autonomy. The energy storage system (ESS), which allows power to flow in both directions, is applied to deal with the volatility of RESs and the randomness of local loads. The concept of the energy internet (EI) was first proposed by American economist Jeremy Rifkin in his book “The Third Industrial Revolution: How Lateral Power Is Transforming Energy, the Economy, and the World” [1]. The principle of the EI combines the technologies of internet, power electronics, large data processing, intelligent monitoring, and management with power systems [2]. The use of power electronic converters to connect the RES generators and ESS to the power grid brings a low inertia to the system and the operation conditions are more complicated. Inertia plays an important role in system stability since a large inertia means a strong ability to stay in place when disturbance occurs. To deal with this problem of the lack of inertia, the research pays much attention to the control strategies for inverter-based distributed generators (IBDGs) to participate in frequency and voltage regulation [3,4,5].

To provide inertia and improve system stability, virtual synchronous generator (VSG) control technology has been proposed and widely studied. As the name VSG suggests, this control strategy is brought up by imitating the behavior of traditional synchronous generators (SGs) [6].

In order to better apply the VSG in the energy internet, many works in the literature take different methods and optimize different aspects for the existing VSG framework. As the current artificial intelligence (AI) methods are widely used in various traditional fields, they bring new breakthroughs to these traditional industries. This paper will focus on the most popular deep learning (DL) and reinforcement learning (RL) methods in the current AI methods combined with VSG application. To the best of our knowledge, there are no review papers on the subject combining VSG control with DL and RL methods. This paper will fill the vacancy in this field.

The structure of this paper is as follows:

• Section 2 provides an introduction of VSG principles. The main classifications of the VSG are introduced in detail;
• Section 3 gives an overall review of the development of deep learning and reinforcement learning;
• Section 4 focuses on the dynamic process of a single VSG that is optimized by DL and RL algorithms;
• Section 5 reviews the DL and RL algorithms used in multi-VSG systems;
• Section 6 concludes this paper and provides some future research trends according to all the research reviewed in this paper.

2. Principle and Classification of VSG

In general, a simplified VSG control system is shown in Figure 1, where $L_r$ and $C_r$ are, respectively, the filter inductance and filter capacitance of the inverter, $Z_{line}$ is the line impedance of the inverter output to the AC bus, and $P$ and $Q$ are, respectively, the active power and reactive power output. The power supply on the DC side represents distributed power supplies such as the photovoltaic and energy storage systems. And the AC side can either operate in the off-grid mode to support local loads, or provide inertia to the grid in the grid-connected mode.

In this kind of imitation, the transient process of the SG is described mathematically, and then the mathematical equations are introduced into the control process. Since the transient process is complex and it is unnecessary to fully integrate the characteristics of the SG into the control process, various models with slight differences have been proposed. These models and principles are discussed in this section.

2.1. Virtual Synchronous Machine (VISMA)

The VISMA is the first VSG algorithm proposed by Hans-Peter Beck in [7]. This scheme simulates the rotor motion equation and stator electrical equation of the SG, and obtains the reference value of the stator current or the reference value of the motor port voltage according to the volt–ampere characteristic of the stator electrical equation. The external characteristics of the synchronous generator are simulated by controlling the output current or output voltage of the inverter.

The swing equation for the rotor can be expressed as Equation (1):

$$T_m-T_e=J\frac{\mathrm{d}\omega }{\mathrm{d}t}+k_{d}f\left(s\right)\frac{\mathrm{d}\omega }{\mathrm{d}t}$$

(1)

where $k_d$ is the mechanical damping coefficient and $f\left(s\right)$ is the phase compensation term [8].

The stator electrical equation can be expressed as Equation (2):

$$\mathit{e}=\mathit{u}+R_s{\mathit{i}}_L+L_s\frac{\mathrm{d}{\mathit{i}}_L}{\mathrm{d}t}$$

(2)

where $\mathit{e}=\left[e_a,e_b,e_c\right]$ is the three-phase stator-induced electromotive force, $\mathit{u}=\left[u_a,u_b,u_c\right]$ is the three-phase voltage at the motor port, ${\mathit{i}}_L=\left[i_{La},i_{Lb},i_{Lc}\right]$ is the three-phase stator current, and $R_s$ and $L_s$ are stator resistance and inductance values, respectively.

According to the different implementations of the Equation (1), the VISMA can be divided into the current-control type [7] and the voltage-control type [8]. The current-type VISMA scheme is based on Equation (1), obtaining the reference value of the three-phase stator current ${\mathit{i}}_{Lref}=\left[i_{Laref},i_{Lbref},i_{Lcref}\right]$:

$${\mathit{i}}_{Lref}=\frac{1}{L_s}\int \left(\mathit{e}-\mathit{u}-R_s{\mathit{i}}_{Lref}\right)\mathrm{d}t$$

(3)

While the voltage-type VISMA scheme obtains the reference value of the three-phase motor port voltage ${\mathit{u}}_{ref}=\left[u_{aref},u_{bref},u_{cref}\right]$:

$${\mathit{u}}_{ref}=\mathit{e}-R_s{\mathit{i}}_L-L_s\frac{\mathrm{d}{\mathit{i}}_L}{\mathrm{d}t}$$

(4)

Figure 2 shows the block diagrams of these two types of VISMAs.

2.2. Synchronverter

The synchronverter [9] simulates the operation mechanism of the synchronous generator more precisely, which not only considers the rotor motion equation of the SG in the control process, but also mimics the electromechanical coupling characteristics of the synchronous generator. The synchronverter fully takes into account the electromechanical and electromagnetic transient characteristics of the SG and is obtained according to the following physical and mathematical models:

$$\mathit{e}=\omega M_f{i}_f\left[\sin \theta ,\sin \left(\theta -\frac{2}{3\pi }\right),\sin \left(\theta +\frac{2}{3\pi }\right)\right]$$

(5)

$$Q=-\omega M_f{i}_f\left[i_{La}\cos \theta +i_{Lb}\cos \left(\theta -\frac{2}{3\pi }\right)+i_{Lc}\cos \left(\theta +\frac{2}{3\pi }\right)\right]$$

(6)

$$T_e=M_f{i}_f\left[i_{La}\sin \theta +i_{Lb}\sin \left(\theta -\frac{2}{3\pi }\right)+i_{Lc}\sin \left(\theta +\frac{2}{3\pi }\right)\right]$$

(7)

where $M_f$ is the mutual inductance between excitation windings and stator windings, and $i_f$ is the exciting current.

The synchronverter strategy can be applied both in the power-side grid-connected inverter control, as shown in Figure 3a, and the load-side grid-connected rectifier control, as shown in Figure 3b. This makes the DGs and loads have the same operating mechanism as the synchronous units, allowing them to participate in the operation and management of the EI. When the power grid, DGs, or load disturbance occurs, the DGs and loads can be adjusted bi-directionally to improve the ability to resist external disturbance.

The synchronverter applied on the power side and the load side as mentioned above can be interpreted as a virtual synchronous generator and a virtual synchronous motor (VSM). The VSG type calculates the reactive power $Q$ and electromagnetic torque $T_e$ according to Equations (6) and (7). It realizes the function of primary frequency regulation and damping with the damping coefficient $D$, and simulates the rotor inertia of the synchronous motor by the rotor motion equation. In order to realize the reactive power-voltage droop control, the difference between the grid voltage rating and the actual voltage is multiplied by a factor $D_q$ and added to the reactive power rating $Q_0$.

2.3. Other Virtual Synchronous Models

From 2008, several different VSG models are proposed worldwide. In 2008, K. VISSCHER et al. proposed the concept of the VSG in the European VSYNC joint project [10]. The first-order sag characteristic curve and dynamic mechanical equation of the generator are simulated, and the moment of inertia and damping characteristics are simulated by adding short-term energy relief. The basic principle is shown in Figure 4.

Moreover, the Institute of Electrical Power Engineering (IEPE) at Clausthal University of Technology in Germany, the VSG research team at Kawasaki Heavy Industries (KHIs), and the ISE laboratory at Osaka University in Japan have proposed several different models with which to provide inertia.

In the remaining part of this paper, the models mentioned above are collectively referred to as VSGs and no specific distinction is made.

3. Deep Learning and Reinforcement Learning

DL and RL are both popular branches of machine learning (ML). There are some differences in the problems they focus on and the tasks they solve.

3.1. Deep Learning

Deep learning is a subset of machine learning that emphasizes the use of deep neural networks (deep-learning models) to learn and represent data [11]. DL models are able to learn complex features and representations from large amounts of data, enabling them to achieve remarkable achievements in the fields of vision, speech, natural language processing, etc.

In this section, some mainstream types of DL algorithms are briefly introduced. DL typically refers to a deep neural network which contains multiple hidden layers, as well as input and output layers. Generally, DL algorithms deal with tasks such as the following: (i) supervised: the input data are labeled; (ii) unsupervised: the input data are not labeled or without a known result; and (iii) semi-supervised: the input data are a mixture of labeled and unlabeled examples. For different learning tasks, neural networks with different structures are used.

3.1.1. Multi-Layer Perceptron (MLP)

The multi-layer perceptron is a supervised learning approach and a fundamental structure of DL. There is an input layer to receive data, an output layer to make a decision, and several hidden layers to compute and process data in an MLP. The front and back layers in the MLP are typically fully connected.

3.1.2. Convolutional Neural Network (CNN)

The convolutional neural network was first proposed by LeCun et al. in 1998 [12]. The CNN is suitable for processing spatial data. Its design ideas are inspired by visual neuroscience. A CNN mainly consists of a convolutional layer and a pooling layer. The convolution layer can maintain the spatial continuity of the data structure and extract the local features of the data. The pooling layer can be max-pooling or mean-pooling, which can reduce the dimension of the middle hidden layer, reduce the calculation of the next layer, and provide rotation invariance. CNN is widely used in computer vision; the ImageNet large scale visual recognition competition (ILSVRC) has greatly promoted the development of convolutional neural networks, and many classic neural networks were born in this competition.

AlexNet [13], the winner in 2012, brought about a range of technologies that are still widely used today. It adds training data by means of translation, flipping, and capturing part of the image, uses dropout to prevent overfitting, trains the model with batch gradient descent with momentum and weight decay, and uses a Rectified Linear Unit (ReLU) as an activation function to shorten the training time. Zeiler and Fergus used deconvolution networks to visualize the CNN as a way to understand the role of each layer of the CNN, which led to the 2013 winner, ZF Net [14]. VGGNet, proposed by Simonyan et al. in 2014, showed that deeper networks can significantly improve the effectiveness of deep learning [15]. The champion of the same year, GoogleNet, placed the convolution layer and the pooling layer in parallel, breaking the pattern of the convolution-layer–pooling-layer stacking for the first time [16]. Deep Residual Network (ResNet) [17] is the ILSVRC 2015 champion. The same network won the three tasks of picture classification, object location, and object detection. ResNet directly adds a linear connection path between two or more layers to form a residual module, which effectively solves the problem of gradient disappearance in deep networks. R-CNN [18] and its many improved methods [19,20,21] are the mainstream methods applied to object detection.

3.1.3. Recurrent Neural Networks

Recurrent neural networks (RNNs) take the output of the hidden layer of the previous moment as the input of the hidden layer of this moment, and can use the information of the previous moment [22]. This property of memory is well-suited for processing time-series data. The basic RNN principle can be illustrated as follows: RNN shares the weight at each time, and, when using information from a long time in the past, there will be a gradient disappearance or gradient explosion problem [23].

The long short-term memory (LSTM) network replaces neurons in the RNN with LSTM units, adding input gates, output gates, and forget gates to input, output, and forget information to control how much information is allowed to pass through [24]. The LSTM model can remember things that need to be remembered for a long time and forget unimportant ones, which can alleviate the problems of gradient disappearance and gradient explosion, and have a better performance than RNN on longer sequences. The gated recurrent unit (GRU) is a lightweight variant of the LSTM model with only two doors: an update door and a reset door [25]. The update gate determines how much past information is retained and how much information is entered from the input layer; The reset door is similar to the forget door in LSTM. With less connection and fewer parameters, the training is faster compared with LSTM. Some other improvements like the hierarchical RNN [26], bidirectional RNN [27] and multi-dimensional RNN [28] have been proposed to deal with different problems arising in the RNN.

3.1.4. Attention Mechanism

When people observe things through the visual system, the human eye focuses on certain parts (of interest). Similarly, the attention mechanism enables deep-learning models to assign different weights to different input data, giving more attention to more important data, which is realized by incorporating mechanisms to focus on the moving, scaling, and rotating of the areas into the network [29]. The attention mechanism can be combined with the CNN [30], RNN [31], and LSTM [32].

The classification of supervised DL mentioned above can be classified as shown in Figure 5.

3.2. Reinforcement Learning

Reinforcement learning is a machine-learning method in which agents interact with the environment to learn optimal behavioral strategies. Agents adjust their behavioral strategies based on feedback from the environment (reward or punishment). RL focuses on developing the best decision-making strategy by maximizing long-term rewards through trial and error and learning. It is widely used in decision problems, control systems, and automation.

The development of reinforcement learning can be traced back to the 1950s and 1960s, and the founding work can be traced to Richard Bellman’s dynamic programming theory. Bellman first proposed the Bellman equation for optimal control problems, which laid the foundation for the theory of reinforcement learning [33]. In the 1960s, Ronald Howard et al. introduced the Monte Carlo method into optimal control problems [34]. Later, Ronald Howard and Andrey Kolmogorov et al. formally described the Markov decision process (MDP), which became an important mathematical model in reinforcement learning. The MDP provides a theoretical framework for describing the interaction between agents and the environment and making decisions. Christopher Watkins proposed the Q-learning algorithm in [35] in 1989. RL combined with deep learning, which is referred to as deep reinforcement learning (DRL), has developed rapidly in the past decade. New algorithms have been constantly proposed and the application fields have become more and more extensive. To be more specific, RL in this paper mainly refers to the deep reinforcement method.

There are two main components in an RL algorithm. The agent can interact with the environment and obtain a reward from it. The agent explores the action space and finds out which action under the current state can bring the greatest cumulative reward [36]. Different from the immediate rewards of supervised and unsupervised learning, the rewards in RL are long-term and emphasize the process of accumulation.

A reinforcement-learning problem can be described as a Markov decision process (MDP), which consists of a set of interacting objects, the agent, and environment [37]. A reinforcement-learning agent can make decisions by sensing the state of the external environment, take actions, and adjust decisions through the feedback of the environment. The environment refers to all things outside the agent in the MDP model, the state of whom is changed by the actions of the agent. The changes can be fully or partially perceived by the agent. A positive change may give a reward to the agent while a negative one can bring a punishment, causing the agent to adjust accordingly.

MDP can be described as a tuple $\langle S,A,P,R\rangle$, where:

• $S=\left\{s_1,s_2,\dots ,s_t\right\}$ stands for the state space, which can be either discrete or continuous. It refers to the state of the environment observed by the agent.
• $A=\left\{a_1,a_2,\dots ,a_t\right\}$ stands for the action space taken by the agent, which can be either discrete or continuous. $a_t$ refers to the action produced by the agent decision at each time step $t$. After each action, the environment will enter the next state.
• $P$ refers to the transfer probability function, which represents the possibility that the system moves to the next state after the agent takes a certain action in the current state.
• $R$ refers to the reward function. By interacting with the environment, the agent will receive a reward for its action. A positive reward indicates the action is effective, while a negative reward, also known as a punishment, indicates a wrong action. The goal of an agent is to maximize the expected future rewards by optimizing the policy.

Besides the elements above, a policy $u\left(a_t|s_t\right)$ is a function that maps the state $s_t$ to the action $a_t$. The ultimate goal of an RL algorithm is to find out a certain policy to maximize the benefits. The illustration of an MDP is shown in Figure 6. Every time the agent takes an action $a_t$, the environment generates a reward $r_t$ based on the current state $s_t$, and then moves to the next state $s_{t+1}$.

For an MDP, the state-value function $V_{\pi }\left(s\right)$ and the action-value function $Q_{\pi }\left(s,a\right)$ are used to describe how acceptable the action taken in a certain state is. The state-value function $V_{\pi }\left(s\right)$ can be defined as:

$$V_{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[R_t|S_t=s\right]$$

(8)

which means the expected reward value by following the policy $\pi$ in condition $s$. $R_t$ is the estimated value of the cumulative future reward defined as:

$$R_t=r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\dots ={{\displaystyle \sum }}_{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

(9)

where $\gamma$ is a discount factor and $\gamma \in \left.\left[0,1\right.\right)$.

The action-value function $Q_{\pi }\left(s,a\right)$ can be defined as:

$$Q_{\pi }\left(s,a\right)={\mathbb{E}}_{\pi }\left[R_t|S_t=s,a_t=a\right]$$

(10)

which means the expected long-term benefits by taking the action $a_t$ in the state $s_t$.

The optimal policy ${\pi }^{*}$ refers to the policy that receives the highest cumulative reward. And the optimal state-value function and action-value function can be defined as:

$$V^{*}\left(s\right)=\underset{\pi }{\max }{V}_{\pi }\left(s\right)$$

(11)

$$Q^{*}\left(s,a\right)=\underset{\pi }{\max }{Q}_{\pi }\left(s,a\right)$$

(12)

To find out the optimal policy, RL algorithms can be divided into two categories: value-based approaches and policy-based approaches.

3.2.1. Value-Based Approaches

The deep Q-network (DQN) is a representative-value-based approach that marks the beginning of the flourishing of deep reinforcement learning. Developed by DeepMind, a DQN algorithm has outperformed human players in an Atari 2600 game, demonstrating the potential of deep learning in reinforcement learning [38]. In the DQN, a DNN is used to replace the tabular value in classic Q-learning. Moreover, many of the contributions of the DQN are groundbreaking. First, an experienced replay mechanism based on a replay buffer is developed to store transition tuples. Then, a target network is introduced in the process. The target network $Q\left(s,a|{\theta }^{\prime }\right)$ has the same structure with the online network $Q\left(s,a|\theta \right)$ but with different parameters. The parameters ${\theta }^{\prime }$ in the target network update slowly to overcome the overfitting problem. It is widely confirmed that learning with a target network performs better in many tasks [39]. Furthermore, an ε-greedy strategy is applied to encourage exploration to prevent the agent from acting within the explored space.

The Double DQN (DDQN) [40], dueling DQN [41], and Prioritized DQN [42] are improved approaches based on the DQN to solve the problem of action-value function estimation. In the DDQN, two neural networks are applied to select actions and calculate the target Q value, respectively, which helps to solve the problem of overestimation in the DQN, and improves the stability and performance of training, while, in the dueling DQN, the Q-value function is divided into two parts: the value function of the state and the advantage function of the action. The dueling DQN can learn the value information of states and the advantage information of actions, thus achieving a faster convergence. The prioritized DQN innovatively takes the temporal difference (TD) as an important consideration to ensure that important experiences can be replayed and learned first, which greatly improves the learning efficiency of the DQN.

3.2.2. Policy-Based Approaches

The value-based approaches output the value of all actions and select the action with the highest value, which takes the same action under the same circumstance. The policy-based approaches parameterize the strategy, take the weight parameter of the neural network as the parameter of the value function, and directly output the probability of various actions to be taken in the next step by analyzing the state. The actions are taken according to the probability. Each action has a corresponding probability to be selected.

Compared with value-based approaches, policy-based approaches are simpler to realize and have a better convergence. However, the size of the learning step for policy-based approaches is more difficult to determine. The trust region policy optimization (TRPO) algorithm introduces Kullback–Leibler divergence (KLD) to represent the gap between old and new strategies, and can finally solve the maximum step length in a confidence region that can make the strategy monotonically improve, reducing the fluctuation during training [43]. The proximal policy optimization (PPO) algorithm innovatively uses a new loss function, Clip, instead of KLD to solve the problems of the large computational load and complex implementation of TRPO [44]. The authors of [45] proposed a deterministic policy gradient (DPG) algorithm theoretically and provide the calculation formula and parameter-updating method. As suggested in this name, the deterministic method greatly improves the training speed compared with the stochastic strategy gradient method.

Actor–Critic is a policy-based approach that blends TD learning and a policy gradient. The actor refers to the policy function $\pi \left(a|s\right)$ and the critic refers to the value function $V_{\pi }\left(s\right)$. The deep deterministic policy gradient (DDPG) algorithm improves the DPG by using this Actor–Critic architecture to replace the linear function for the value function prediction through the neural network [46]. Since the action generated by the DPG in the same state is determined, the DDPG combined the advantages of the DPG and the DQN. Using the Actor–Critic Framework, the Actor to explore and the Critic to revise, it ensures the ability to explore the action space and improves the computational efficiency. Based on the Actor–Critic framework, various algorithms have been proposed, such as the Advantage Actor–Critic (A2C), Asynchronous Advantage Actor–Critic (A3C) [47], Soft Actor–Critic (SAC) [48], and twin delayed deep deterministic policy gradient (TD3) algorithm [49].

The RL-based approaches can be classified as shown in Figure 7.

4. DL and RL in Single VSG

The aim of the VSG is to mimic the inertia and damping characteristic of SGs. Apart from that, one of the advantages of the VSG is its flexibility. The parameters of SGs are fixed since the physical structure and state are fixed. But the parameters in the VSG are flexible towards change since the control algorithm is just a program running on a controller. To make better use of the flexibility of the VSG, various parameter adaptive control methods have been widely studied.

The mode-based method depends on the accurate system model to develop a certain adaptive law for tuning. In [50], an adaptive virtual inertia control method is proposed to improve the dynamic performance of the system. In [51], both virtual inertia and damping are taken into consideration. The effect of control parameters on the output power and frequency have been intuitively studied in [52] and a strategy with the features of a short response time and small overshoot is proposed.

With the rapid development of AI technology, its powerful ability to learn data relationships causes it to have a good application prospect in parameter adjustment.

DL algorithms are introduced to deal with this issue. In [53], taking the lifetime consumption into consideration, a double-ANN-based method is proposed to design the virtual inertia with stability and reliability. A deep convolutional neural network (DCNN) is designed in [54] to adjust the virtual inertia $J$ and damping $D$ online. The input data for the net are the frequency deviation and the rate of change of frequency. The convolution layer, pooling layer, and fully connected layer are involved in the network. The results show a better performance over the fixed-parameter method in active power and frequency response.

RL algorithms are also widely used in this field. Q-learning is first used in VSG parameter tuning in [55]. Even though the structure design is relatively simple (the virtual inertia $J$ is the only one-dimensional action output, and the reward function only considers the deviation of angular frequency), the method can still effectively improve the dynamic frequency and active power response of the VSG. In [56], a DQN algorithm is proposed, which has a continuous state set by replacing the Q-table in Q-learning with a neural network. The virtual inertia $J$ and damping $D$ are selected as actions with discrete actions spaces. The deviation of active power, reactive power, and frequency are all considered in the reward function. Reference [57] applies a decentralized DPG algorithm in order to obtain continuous control operations with continuous state observations. And the derivative of angular frequency, that is, the rate of change of frequency (RoCoF), is taken into consideration in order to obtain relatively slow and smooth electromechanical dynamics. However, when applied in a new system, the results of the proposed methods show no significant advantage over the droop controller. In [58], a physics-informed neural network combined with an SAC algorithm is proposed to adjust the virtual inertia $J$ and virtual damping $D$ of a VSG. To decrease the risk of grid instability, physical constraints are taken into consideration. The proposed method is tested in the IEEE-39 bus simulation system. Reference [59] uses an RL algorithm based on Actor–Critic to adjust the virtual inertia. In [60], virtual inertia is adjusted by a DDPG agent to achieve a better dynamic frequency response. A DDPG algorithm is proposed in [61] to adjust $J$ and $D$ adaptively. The author of [62] argues that the DDPG algorithm may suffer from overestimation bias and proposed a TD3 algorithm. Moreover, the settling time is involved in the reward function to ensure a fast response. Considering different reward functions and various working conditions, the results of the proposed TD3 method are fully compared with some other control strategies both in Simulink and a real-time digital simulator (RTDS). In [63], the VSG control is applied in a modular multilevel converter (MMC) structure and a TD3 algorithm is used to adjust the virtual inertia $J$, damping $D$, and the voltage reference $E$ according to the deviation of active power, reactive power, and frequency. The TD3 algorithm is also used to adjust $J$ and $D$ in [64], in which the authors make a comparative analysis between TD3 and other two RL algorithms, DDPG and SAC. Different methods are compared both in control performance and in training performance. In [65], an adaptive dynamic programming (ADP) algorithm is used to adjust the virtual inertia of a VSG and the angular deviation, frequency deviation, and RoCoF are taken into consideration. Compared with other research, Ref. [65] provides a bounded threshold of inertia to facilitate practical applications.

Besides virtual inertia and damping, some researchers use DL or RL to adjust other parameters in the VSG. In [66], a DDPG algorithm is used to adjust the active power reference value. Compared with the PI controller, the DDPG method achieves a better performance in the grid-connected mode and realizes the frequency restoration in the islanded mode. Reference [67] applies a safe RL method with which to adjust the active charging power reference of a battery energy storage system (BESS). The transient frequency regulation process is improved compared with the DDPG-based method and the SAC-based method. In [68], different grid conditions like an unbalanced grid, voltage drop, and a weak grid are all included. The ADP algorithm is used to realize the secondary control. The proposed method is tested both in simulation and in a lab. In [69], a model-dependent heuristic dynamic programing (MDHDP) algorithm is introduced. Receiving the power, frequency, and angle information of the system, the actor network outputs the peak value of the inverter’s voltage. And the simulation results show the effectiveness of the proposed method. The authors of [69] further carry out the examination on a test bed and the results are shown in [70].

Some research provides other solutions with which to improve the dynamic performance of the VSG. Instead of adjusting some core parameters in a reasonable range, another natural idea is to compensate the variables that the system needs to control. The traditional active power loop of the VSG is shown in Figure 8, while, in [37], a power compensation term is introduced to this loop, as shown in Figure 9. The action output of the DDPG agent is the power compensation to the input of the active power loop. With fixed parameters $J$ and $D$, the active power and frequency responses of the system are also improved. In [71], the author replaces the VSG controller with a DNN with five layers. The data generated by a traditional VSG control system under multiple working conditions are selected as the training data. The inputs of the network include the output voltages and currents of the inverters, loads, angular frequency, the required phase voltage, and frequency. The outputs are the duty ratio under the dq-frame and the angular frequency, which are sent to the PWM module after the Parker inverse transform calculation to control the inverters. In [72], the LSTM algorithm is applied to mimic the relationship between the inputs and outputs of a VSG control model when a phase-to-phase short circuit fault occurs. And the same idea is applied in [73] by combining the restricted Boltzmann machine (RBM) features and a DNN. With fewer inputs, the proposed controller can suppress the power oscillation and provide a stable frequency. The effectiveness is verified on an experimental platform in a laboratory.

The authors of [74] focus on the power-coupling problem of VSG. A new reactive-voltage loop is designed, which is symmetric to the active-frequency loop. By using an ADP algorithm, the parameters of the two loops above are adjusted. The constraints of PQ decoupling are also relaxed. The results performed on a test rig shows the effectiveness of the proposed method.

The references mentioned in this section and the methods used are shown in

Table 1. DL and RL in single VSG control.

References	Learning Algorithm		Objective			Method
	DL	RL	Power	Frequency	Voltage	Adjust Parameters	Replace VSG Module
[53]	√					√
[54]	√		√	√		√
[55]		√	√	√		√
[56]		√	√	√		√
[57]		√	√	√		√
[58]		√	√	√		√
[59]		√	√	√		√
[60]		√	√	√		√
[61]		√		√		√
[62]		√	√	√		√
[63]		√	√	√	√	√
[65]		√	√	√		√
[66]		√	√	√		√
[67]		√		√
[69]		√		√	√		√
[71]	√			√	√		√
[72]	√			√	√		√
[73]	√		√	√			√
[74]		√	√			√

.

5. DL and RL in Multi-VSG System

Power systems with regional energy internet or microgrids have the ability of local autonomy. The local power supply can be fully absorbed by local loads or ESSs and scheduled by a local governor, which requires that all parts of the microgrids work together. The co-ordinated control of a multi-VSG system consists of system transient stability prediction, power sharing, the suppression of current circulation, and power oscillation.

5.1. Power Sharing, Suppression of Current Circulation, and Power Oscillation

It can be drawn from the discussion above that the operation mechanism of the VSG is relatively more complex than other control methods. As a result, the dynamics of several paralleled VSGs can interact with each other. It is shown in [75,76] that the placement of virtual inertia and damping, “where” to place, rather than the total inertia and damping in a system, has a more significant influence on the efficiency and stability. Since then, more and more researchers have directed their attention towards the co-ordinated operation of a distributed system.

Some learning methods like particle swarm optimization (PSO) in [77,78] are introduced to attain optimal multi-VSG parameters on a detailed model. But these parameter optimization methods are offline and based on fixed working conditions. Some model-based methods in [79,80,81,82,83], realized by the dynamic model of the VSG, can attain optimal parameters and achieve a good performance. However, an accurate mathematical model of the multi-VSG system is hard to obtain and the working conditions change over time. Therefore, the model-free methods attract more attention in application.

It is analyzed in [84] that all VSGs output the same frequency when the virtual inertia, droop coefficient, and the equivalent disturbance are proportional. By adjusting the parameters of each VSG in real time, the oscillation can be suppressed. According to this theory, the author raises an SAC-based model-free method to modify the inertia and droop coefficient, which shows a good robustness under communication interruption and time delay.

In [85], an SAC algorithm is used to adjust the droop coefficients and the DC voltage reference values in a system with four AC/DC converters. Compared with the traditional fixed parameters control method, the SAC method performs better in frequency support when the transmission load is heavy.

In [86], to design a co-ordinated wide-area damping control method, two control schemes are proposed, one based on the DNN and the other based on the DDPG. The different two agents can both track the system dynamics and take actions to improve safety and reliability under various conditions. In [87], an ANN-based virtual inertia control system is introduced to support the frequency of a multi-area microgrid system. And the multi-agent RL method is used to find out the weights of each ANN. Compared with the PID method in a three-area interconnected system, the proposed method can reduce oscillations at high frequencies.

Reference [88] applies a TD3 algorithm to adjust the reactive power reference for each VSG to minimize the reactive-power-sharing error and ensure voltage stability. In [89], the multi-agent distributed frequency control strategy based on Q-learning is applied to realize frequency restoration and active power sharing among the four VSGs. The control actions are the active power compensation values for VSGs.

5.2. Transient Stability Prediction and Improvement

With the high penetration of power electronic devices, the nonlinear characteristics make it difficult to analyze the transient stability by traditional analysis methods. The development of AI technology brings solutions to this complicated process. Some ML methods like the Decision Tree (DT) and Support Vector Machine (SVM) are first raised in this field. In [90], phasor measurements and the DT are used to assess the dynamic security of a large-scale power system with a 2100-bus, 2600-line, and 240-generator model, which reaches an accuracy of 97%.

Afterwards, DL methods are introduced to this field. An online monitoring system (OMS) is proposed in [91] using a convolutional neural network (CNN). The OMS monitors the voltage profiles continuously, makes predictions on stability, and provides the most impacted generators. In [92], a deep feed-forward neural network (DFFNN) method is applied for transient stability prediction in a 3-VSG system. Compared with shallow ML methods, the DFFNN method realizes a more rapid prediction with 0.18 ms and a higher accuracy of 97.2% under complex working conditions. In [93], the author raises a message-passing graph neural network (MPGNN) to make it possible to use the same algorithm under different topologies of the system without training again, which improves the generalization ability compared with [92]. Besides accuracy, some of the research focuses on efficiency. Only using the virtual power angle data of each VSG, the long-short term memory (LSTM) neural network proposed in [94] takes 0.02 s to predict the virtual power angle of the next 0.25 s. To further reduce the amount of computation and the network size, a variable kernel–temporal convolutional network (VK-TCN) method is proposed in [95], which takes a minimum of 43 s to train. By analyzing the mechanism of voltage instability in the island microgrid, it narrows the range of eigenvalues and reduces trial and error in adjusting the hyperparameters of the network. Another control method to improve transient stability is called preventive control. According to [96], preventive control changes the operating point at a small cost by adjusting the power output of generators or the load configuration before a large disturbance to meet the requirement of transient stability. The solution to the preventive control problem is generally focused on the optimal power flow constrained by transient stability. In some works of research like [97,98], where DL methods are applied, the constraint conditions of converter control methods are not fully considered. In [99], the current limit value of each VSG is added as the input feature for a DFFNN. By minimizing the total cost of active output adjustment in system, the fast prediction of transient stability is achieved.

In [100], Q-learning is applied to adjust the virtual inertia of each VSG in a 4-VSG system. Two metrics are used to evaluate the performance, one of which is the critical clearing time (CCT) and the other is the percentage of conditions that becomes stable after the parameters are tuned. Compared with traditional methods, the Q-learning method performs better on both of the two metrics. In [101], the authors present an optimal preventive control (OPC) method by tuning the parameters of VSGs, and an RL method with the fault energy reward function is introduced to solve the OPC problem. The simulation results on various systems show that the proposed method can improve the CCT when a fault occurs. In [102], the learning vector quantization (LVQ) network is applied to distinguish the fault type, which can be performed within 5 ms, and adjust the parameters adaptively according to the type of the fault. The result is compared with some other AI algorithm to demonstrate the effectiveness.

5.3. Power Prediction and Scheduling

Power systems with distributed renewable energy provide power for local loads and achieve regional autonomy. With the volatility of renewable energy and the periodic characteristics of industrial and domestic electricity consumption, it is of practical significance that we arrange the power generation plan reasonably. Due to the different generation costs and dispatch costs of different power generation equipment, accurate prediction and reasonable scheduling can bring huge economic benefits to the power system.

In [103], all VSGs output the same frequency when the virtual inertia, the droop coefficient, and the equivalent disturbance are proportional. By adjusting the parameters of each VSG in real time, the oscillation can be suppressed. According to this theory, the author raises an SAC-based model-free method to modify the inertia and droop coefficient, which shows a good robustness under communication interruption and time delay.

Large disturbances and faults can lead to the loss of synchronization (LOS) for VSG control. The influence of virtual inertia varies under different fault conditions and needs different adjustments or modifications within the VSG control framework. Therefore, fault identification and diagnosis are essential to the transient stability of the power system.

The references mentioned in this section and the methods used are shown in

Table 2. DL and RL in multi-VSG system.

References	Learning Algorithm		Objective			Method
	DL	RL	Oscillation Suppression	Stability Prediction and Improvement	Power Scheduling	Adjust Parameters of each VSG	Predict with Real-Time Data
[84]		√	√			√
[85]		√		√		√
[86]	√		√			√
[86]		√	√			√
[87]		√	√			√
[88]		√	√			√
[89]		√	√			√
[90]	√			√			√
[91]	√			√			√
[92]	√			√			√
[93]	√			√			√
[94]	√			√			√
[95]	√			√			√
[99]	√			√			√
[100]		√		√		√
[101]		√		√		√
[103]		√			√	√
[102]	√			√		√

.

6. Discussion and Future Research Trends

Compared with the traditional centralized control strategy, the VSG can serve as a grid-forming control strategy. Compared with droop control, the VSG can provide inertia to the power system under conditions of the high penetration of DGs and power electronic devices. These advantages are destined to keep the VSG-based control strategy the optimal choice in energy internet. And researchers are finding out more solutions with which to overcome the disadvantages, such as difficulties in accurate modeling and parameter tuning, and complex operating conditions, with the help of DL and RL algorithms. It can be drawn from the research mentioned above that DL and RL methods show a powerful ability in improving the performance of the VSG, and, since the research on the implementation of DL and RL in the VSG is still in its infancy, there will be great potential and a lot of room for exploration in this field. Moreover, difficulties and limitations will also arise and need to be overcome. According to the reviewed literature, some discussions and future research trends are listed as follows:

• Flexibility. As mentioned at the beginning of this paper, flexibility is one of the important characteristics of modern power systems. The distributed power supply and loads should be able to be connected or cut off without affecting the conditions of other devices. And, with the further development of distributed power generation, new DGs should be able to connect the system freely, and the parameters of other VSGs should be adjusted automatically if possible, which ensures transient stability under system topology changes.
• Deployment in real devices or systems. With many AI-based control methods being proposed, most of them are verified only in a simulation platform or in lab conditions. Few of these methods are applied in real systems. This is due to the high reliability requirements of the power system and incomplete interpretability of DL and RL. However, the combination of an advanced control strategy, VSG, and advanced AI methods, DL and RL, should not remain only at the theoretical level.
• Efficiency and speed. Since the VSG control can be applied parallelly in a large system, in which the working conditions are complicated, a powerful DL network or RL agent may need plenty of data with which to train. According to the reviewed research, the training process often takes a relatively long time. A faster training process and a smaller training dataset will bring benefits to the practical deployment of the VSG.
• Generalization ability. Since all the DL and RL methods require training processes and all the working conditions of a huge system may not be easy to obtain, a network or agent may perform well on the training datasets but have bad behaviors when a new disturbance occurs. Moreover, topology changes may also lead to failures if operating in a new system with previous parameters. Therefore, the generalization ability of the VSG with DL and RL methods is under great demand. With the strong generalization ability of, the amount of computation and training will be greatly reduced when new systems are deployed.
• Voltage and reactive power control. As concluded in Section 3, most of the AI methods for VSG parameter tuning aim to have a better performance in active power and frequency response. The frequency in a system is a global quantity, so the output frequency of each VSG becomes consistent automatically. The active power output is controlled together with the frequency in the same loop, which makes it easier to control. However, due to the existence of line impedance, the deviation in voltage will cause a current circulation between VSGs with different voltage outputs. Since the voltage instability is more complex than the frequency, it is of great importance that we explore the potential of AI methods in voltage control.