Model-Free Resilient Grid-Forming and Grid-Following Inverter Control Against Cyberattacks Using Reinforcement Learning

Abstract

The U.S. movement toward clean energy generation has increased the number of installed inverter-based resources (IBR) in the grid, introducing new challenges in IBR control and cybersecurity. IBRs receive their set point through the communication link, which may expose them to cyber threats. Previous work has developed various techniques to detect and mitigate cyberattacks on IBRs, developing schemes for new inverters being installed in the grid. This work focuses on developing model-free control techniques for already installed IBR in the grid without the need to access IBR internal control parameters. The proposed method is tested for both the grid-forming and grid-following inverter control. Different detection and mitigation algorithms are used to enhance the accuracy of the proposed method. The proposed method is tested using the modified CIGRE 14-bus North American grid with seven IBRs in PSCAD/EMTDC. Finally, the performance of the detection algorithm is tested under grid normal transients, such as set point change, load change, and short-circuit fault, to make sure the proposed detection method does not provide false positives.

1. Introduction

The U.S. has the goal of clean energy generation by 2035, and the number of inverter-based resources (IBR), such as photovoltaic systems and wind turbines, is on the rise [1]. Furthermore, there is a global effort to move from fossil-fueled energy generation to 100% clean power grids [2]. The vast number of IBRs in the grid introduces new challenges to the control and cybersecurity of the power system [3]. IBRs have faster transient responses compared to synchronous machines, enhancing the importance of the inverter control algorithm in maintaining the stability of the grid under various transients. Furthermore, IBRs can be controlled either in a grid-forming or grid-following mode, where the IBR set points are adjusted using the communication link, which offer openings for cyberattacks [4,5]. Moreover, the advent of fast and reliable communication has facilitated the broad installation of these devices in the power system. They are primarily used for exchanging electrical data and commands to breakers and generation units [6]. Multiple cyberattacks have been reported previously on the power system. The Ukrainian power system was the target of a major cyberattack, causing a blackout that lasted for 6 hours, affecting 225,000 Ukrainians in 2015 [7]. Attackers targeted the SolarWinds’ Orion software through a supply chain attack, gaining unauthorized access to government and other systems in 2020 [8]. In 2023 alone, the U.S. Department of Energy (DOE) reported 120 disturbances to the power system, where 114 were due to vandalism and 6 were due to cyber events. The attack on March 2023 successfully interrupted the monitoring and control of a control center for roughly 30 min [9]. Standard protocols have known vulnerabilities, making equipment being used in the power system more susceptible to cyberattacks [10]. For instance, flaws in the transmission control protocol/internet protocol (TCP/IP) have been reported [8]. Using secure communication protocols increases the power system’s resilience against cyberattacks [11]. As a result, designing cyber-resilient IBR control techniques is necessary for preventing system outages due to cyberattacks.

Previous work has focused on designing an inverter control which either needs the microgrid model, can only be installed on the microgrid controller, or required an inverter control update [12,13,14,15]. The lack of a detailed grid model, often due to substantial changes in the grid, poses a significant challenge. Moreover, updating the control algorithms or parameters of the already installed IBR control algorithm is often not possible due to vendor policies. Model-free control approaches modifying the IBR set points without accessing or modifying the internal control parameter can be used to enhance the performance of installed IBRs [16,17,18]. However, the aforementioned model-free control approaches are still susceptible to cyberattacks, thus creating a research gap. A separate detection and mitigation layer can be added to develop a cyber-resilient model-free algorithm. Moreover, reinforcement learning (RL) is a popular choice for developing an optimal controller since it is model-free, can handle nonlinear behavior, and performs well for unknown dynamics and under highly uncertain conditions [19]. A model-free RL-based inverter control increases the grid’s resilience against cyberattacks.

Machine learning (ML) algorithms are widely used to detect different types of cyberattacks, such as false data injection (FDI) and denial of service attack (DoS). Moreover, ML algorithms can be used to mitigate the adverse effect of the attack after the attack has been detected [20,21,22,23]. Reference [5] detected anomalies on the real and reactive power set point of a grid-following inverter using an LSTM algorithm. Reference [24] proposes a machine learning-based mitigation method against DoS attacks in T-S fuzzy networked control systems. A semi-supervised learning approach is used in [25] to detect and mitigate FDI attacks that can pass conventional bad data detection algorithms utilizing autoencoders and generative adversarial networks. In [26], deep reinforcement learning (RL) is used to adjust compromised DERs, mitigating the voltage instability during a FDI attack.

This work uses a model-free approach, modifying the inverter set points after the attack has been detected to mitigate the adverse effects of the attack. This work’s proposed control algorithm has the following salient features:

• A detection algorithm for FDI attacks for both the grid-forming and grid-following inverter is developed.
• A mitigation algorithm is developed using reinforcement learning for FDI attacks.
• The detection algorithm correctly avoids interfering with the grid’s normal transients, such as load change, step change, and short circuit faults.

Section 2 discusses the inverter control, and Section 3 discusses the details of the proposed detection and mitigation method for both the grid-forming and grid-following inverter. Section 4 discusses the test system implementation, and Section 5 offers the performance evaluation. Section 6 discusses future works and Section 7 concludes the paper.

2. Inverter Control

An inverter can be controlled in an either grid-following or grid-forming mode [4].

2.1. Grid-Following Control

An inverter in the grid-following mode of operation receives real and reactive power set points and adjusts the inverter output voltage to keep the output’s real and reactive power as close to their set points as possible. Since the inverter output voltage is generated by switching and includes harmonics, the inverter is connected to the grid through an $RL$ filter [4]. The grid-following inverter uses a phase-locked loop (PLL) to estimate the voltage angle, which is later used in the grid-following control loop to convert voltage from $abc$-frame to $dq$-frame and vice versa. Moreover, $PI$ controllers are used to adjust the inverter output voltage. Figure 1a shows the control loop of a grid-following inverter, where the real power set point, $P_{\mathrm{set}}$, and the reactive power set point, $Q_{\mathrm{set}}$, are the inputs used to calculate the inverter current set point, $I_{\mathrm{set}}$. The inverter control uses electrical measurements such as the real power, P; reactive power, Q; inverter output current, I; terminal voltage, $V_s$; and inverter output voltage $V_t$, as shown in Figure 1a [4].

2.2. Model-Free RL-Based Grid-Following Control

The real and reactive power set point of an inverter in the grid-following mode of operation can be altered from its original value by launching cyberattacks, controller operator mistakes, or even communication noise, which may lead to voltage violation. Detection and mitigation techniques can be combined with the conventional grid-following inverter control [5,27]. This work proposes a cyber-resilient inverter control for grid-following inverters using a detection and mitigation algorithm based on an RL algorithm, as shown in Figure 1b. After the attack is detected, the RL algorithm modifies the real and reactive power set point by either increasing or decreasing it, ensuring the voltage is within the nominal range, from 0.96 pu to 1.04 pu, in this work.

2.3. Conventional Grid-Forming Control

An inverter in the grid-forming mode of operation helps maintain the grid voltage and frequency within the nominal range by adjusting the inverter output’s real and reactive power [4]. P − f and Q − V droop control algorithms can be used to maintain the grid voltage and frequency. Q − V droop modifies the inverter output reactive power based on the difference between the magnitude of the inverter terminal voltage and the set point voltage, as shown in Figure 2a. The P − f droop modifies the inverter output’s real power based on the frequency deviation, as shown in Figure 2a. Droop control outputs are the input of the voltage control loop, and the output of the voltage control loop is the input of the current control loop, as shown in Figure 2b. In a microgrid operating in the islanded mode, a specific number of inverters need to operate in the grid-forming mode to support the grid voltage and frequency.

2.4. Model-Free RL-Based Grid-Forming Control

The droop frequency, real power set point, and voltage are the inputs of an inverter in the grid-forming mode. The inverter voltage set point can be adjusted by the control center; however, the frequency is usually fixed and set to the grid frequency, 60 Hz. The voltage set point of an inverter in the grid-forming mode of operation can be altered from its original value by launching cyberattacks, controller operator mistakes, or even communication noise, which may lead to voltage violations. Detection and mitigation techniques can be combined with the grid-forming inverter control. This work proposes a cyber-resilient inverter control for grid-forming inverters using a detection and mitigation algorithm based on an RL algorithm as shown in Figure 2c. After the attack is detected, the RL algorithm modifies the voltage set point by either increasing or decreasing it, ensuring the voltage is within the nominal range, from 0.96 pu to 1.04 pu, in this work.

3. Detection and Mitigation Method

This section discusses the basics of RL, the basics of inverter control, the proposed inverter control, the detection method, and RL implementation.

3.1. Reinforcement Learning (RL) Basics

Reinforcement learning (RL), along with supervised and unsupervised learning, is one of the three categories that encompass machine learning algorithms. Unlike supervised and unsupervised learning, RL interacts with an environment, taking actions and evaluating their effect [28]. This is done by establishing a reward system that evaluates the taken action on the environment and connecting it to the state that the system is at when the action is taken. Over a period of trial-and-error, an RL system can analyze the state of the system and, using its developed dataset, predict the best action [28].

RL includes several methods that can be used to train the system. The Markov Decision Process (MDP) is one of the methods used for decision making in RL that uses the state–reward relationship to predict optimal actions at a given state [29]. Unlike Q-learning or other RL methods, an MDP is only concerned with the current state of the environment and does not consider any previous data in predicting the optimal action. The prediction of the action is independent of any previous states and is only concerned with the most up to date information.

Setting up a reinforcement learning system requires the establishment of the following information before further progress may ensue. RL requires an environment to interact with before any further progress can occur. An environment is the set of rules and boundaries that the system can interact with over the course of training and testing. This includes defining the actions that the system can take, the states that can be evaluated, as well as the reward system that is implemented [29]. The set of variables that the RL system intakes for prediction is defined as the state. A system may have a list of one or more variables that make up a state; however, this must be clearly defined in the environment. Actions are the changes that an agent may be allowed to perform within an environment [30]. In an RL environment, the number of actions may be large or small; however, a longer list of actions requires additional state-action pairs to be trained. After an action has been performed by an agent in the environment, a reward function is applied to develop a numerical value that represents the effectiveness of the previous action in performing the intended function. Typically, the reward function is structured so that the optimal reward is of the highest positive value; however, an agent may be customized to search for rewards that are mostly negative or closest to zero [29]. Once a training dataset has been developed, an agent can be used to analyze said dataset and select an action that best suits the intended output of the RL system.

In [31], a similar approach is introduced to establish a method for RL to be used for real-time strategy games. The method proposed uses a Semi-Markov Decision Process (SMDP) and an $ϵ$-greedy approach for the RL system. An $ϵ$-greedy selection method is used for training an RL system that has a large number of state–action pairs. $ϵ$-greedy is performed by taking a random action with probability $ϵ$ and the greedy action with probability $1-ϵ$ [31]. Training the system occurs by selecting an action via $ϵ$-greedy, performing the action, and evaluating the reward. The recorded information is used to adjust the option-value function used to select the optimal action. This work follows a similar approach to accomplish the designated task; however, $ϵ$-greedy is not used, since the number of state-action pairs is smaller compared to that of other RL algorithms.

This work details the use of RL and MDP to mitigate FDI attacks on a microgrid once an attack has been detected. A training method is developed to train the RL agent to choose the best action for the state that the system is in. Additionally, a reward function is implemented that aids in the agent’s search for the optimal action to take given the state of the system.

3.2. RL Implementation

This section discusses the implementation of RL for this work. In total, there are three RL models that are used in the construction of the system. For each GFL inverter, there is a RL model that makes predictions based on the changes in $P_{set}$ and another for changes in $Q_{set}$. For each GFM inverter, there is a RL model that makes predictions based on changes in $V_{ref}$. The RL models for $P_{set}$, $Q_{set}$, and $V_{ref}$ have states that correspond to the current voltage measured at the output of their respective inverters. The RL is designed as follows:

3.2.1. State s

s is defined as the discrete state of the system. The state for each RL implementation is the voltage which is read by the inverter at the given time step. It is used for calculating the reward as well as for tuning the agent to find the best action at a particular state. Error e is defined as the difference between s and the voltage of 1.0 pu.

3.2.2. Action a

a is the action that is chosen by the agent to be performed on the system. For this work, the action is the value $a\left(t\right)$ that changes the value of the given reference point. For $P_{ref}$, $Q_{ref}$, and $V_{ref}$, $a\left(t\right)$ has the following values: 0.01, −0.01, 0.001, −0.001, and 0. Testing indicated that $a\left(t\right)$ did not require a difference in its magnitude due to the observed effect that it has on the system with the time step used during simulation.

3.2.3. Reward Function R

R is the reward function that defines the reward that is applied to a state as a result of an action a. The purpose of the mitigation algorithm is to reduce the effects on voltage as a result of a FDI attack. The reward function is designed such that the mitigation system returns the voltage to a level within a specified range as quickly as possible. This reward function is designed with the following criteria in mind:

$$R\left(t\right)=\left\{\begin{array}{cc}-e-\lambda \Delta a,\hfill & \mathrm{if}\left|e\right|>0.04\hfill \\ 0,\hfill & \mathrm{if}\left|e\right|\le 0.04\mathrm{and}\Delta a=0\hfill \\ -10.\hfill & \mathrm{if}\left|e\right|\le 0.04\mathrm{and}\Delta a\ne 0\hfill \end{array}\right.$$

(1)

In (1), $\lambda$ is defined as the penalty factor applied to the value of $a\left(t\right)$ to optimize the agent for selecting an action that approaches the outer limit of the voltage [32].

3.2.4. Training

Given that the previous sections have established the criteria on which the system relies, the RL agent needs to be trained. To produce the intended results of the agent, a training algorithm must be developed for the agent to have enough information. This requires having a sufficient dataset that contains enough states with different actions for the agent to choose from. Moreover, the generated dataset needs to ensure the robustness of the RL algorithm against diverse attack scenarios. Collecting all possible attack scenarios is impractical since it leads to a very large dataset. Additionally, attackers can develop novel attacks that are not considered in the training dataset. One way to overcome the aforementioned challenges is to use a pseudo-random binary sequence (PRBS)-based technique. PRBS-based techniques are used in system identifications and for testing the controller under various transients [33]. Using a PRBS-based technique allows for the testing of the controller under various transients, mainly due to random sequence generation. This work uses PRBS-based techniques to generate the dataset needed to train the RL agent to ensure that the RL agent gives the intended results under various scenarios. Since there are three different agents (one each for $P_{set}$, $Q_{set}$, and $V_{ref}$), this requires three separate training algorithms to be developed.

Each dataset is trained over the course of 100 s, taking actions every 25 ms. Initially, the co-simulation is ran for 5 s for the microgrid to reach steady-state before values are taken. The agent is then be allowed to take a random action for 1.5 s at a given set point, with a reward being recorded along with the state and action that occurred. The system then returns to the starting set point for a period of 0.5 s. This occurs in intervals of 10 s before a new starting set point is chosen. This is done to capture the broadest number of states with enough actions taken to find the most suitable reward for a state. Training for $P_{set}$ has starting set points between 0 and 1 pu; for $Q_{set}$, it has starting set points between −1 and 1 pu; and for $V_{ref}$, the starting set points are between 0.85 and 1.15 pu. Figure 3 shows the plot for the collection of data for $P_{set}$ RL. Contrary to the other RL training methods, the value of $P_{set}$ can only have values between 0 and 1, which does not cover the entire range of values that the voltage may have. To remedy this, $Q_{set}$ is changed to increase the starting voltage for a third of the training length and to decrease for another third of the training length. This allows for further voltage ranges to be reached and actions to be taken. Figure 4 shows the plot for the collection of data for the $Q_{set}$ RL. Figure 5 shows the plot for the collection of data for the $V_{ref}$ RL. The training datasets for each of the RLs can now be used for the mitigation of simulated cyberattacks.

3.2.5. Agent

In the previous section, the training datasets were developed for each of the set points mitigated. The agent developed in this section uses the datasets developed to find the best action at the state the system is in at a given time step. The first step of the agent to predict the best action is to analyze the list of states that are available to choose from. The agent reads the current state of the system and then proceeds to find states inside the dataset that are within a range. For each of the agents, the range is set for states in the dataset that fall within 0.01 pu of the voltage read by the inverter. The agent then categorizes all of the states within the range with respect to the action that corresponds to them. Following the grouping of state–action relationships, the reward is averaged for each action that may be taken. This average reward for each action is known as the Q-value. The agent proceeds to select the action with a Q-value closest to a value of 0. The reward function is structured so that the optimal action to take is the one with the reward closest to 0, reducing the error. This process repeats during each time step for which the mitigation method is called to take action.

The value of 0.01 pu voltage for the search criteria is chosen to encompass enough data so that the Q-value is the most accurate. Cases where the search criteria are smaller may lead to taking the wrong action if one or more actions are not taken during training at that particular state range. In cases where the search criteria are too large, this could lead to taking actions when the voltage is inside the allowed range of tolerance or not taking an action when the voltage is outside the range of tolerance.

3.2.6. Libraries Used

This work utilizes Python 3.9.13 as the virtual environment where the system is designed. Additionally, this work utilizes three libraries that are fundamental to the development of the RL system. Numpy 1.26.3 is used for formatting arrays in the agent’s search algorithm. Additionally, mhi-cosim 1.1.3 is used for the interaction between PSCAD and the virtual environment. Finally, Pandas 2.2.0 is utilized for formatting the dataframes used for the collection and importation of data for the agent to search.

3.3. Detection and Mitigation Algorithm Implementation

The detection algorithm, presented in Algorithm 1, monitors the current and previous set points imposed on each inverter. In the event that the detection algorithm identifies that a FDI attack has occurred, the system calls for the mitigation method to take action. The detection algorithm identifies that at least 100 ms have lapsed since the last event. This is represented in the pseudocode by the change in the discrete time of the system and the updating_timer. Additionally, less than five actions must have been taken by the mitigation method before checking for the following conditions. If the detection algorithm identifies that a change in set point has occurred, then the system invokes a ready-flag on the system. When action = 4, then, change = 0, and therefore, no difference in set point occurs during the time step. In the event that the ready-flag has been invoked, and the predicted action = 4, then, the detection algorithm raises a detection-flag. This calls for the system to predict the action necessary to mitigate the system. This is represented in line 14 of the pseudocode. $V\left(t\right)$ is the discrete voltage state of the system. Otherwise, the mitigation algorithm does not take action and allows for the original adjustment of the set point to be made to the system. After the detection flag becomes 1, the RL action will be applied to the received set point and update the inverter set point to mitigate the adverse effect of the attack.

The size of the actions taken in each time step and the time-step values are tunable variables of the mitigation method. Varying the time step of the PSCAD co-simulation requires adjusting the reward function and action size to produce consistent results. The results indicate that the use of a smaller time step reduces the detection and mitigation times. However, less time for computation makes the real-time implementation of the algorithm unfeasible. A time step of 25 ms is used to perform all PSCAD simulations since the average mitigation algorithm takes 12.263 ms to find the best action. The size of the actions used is tuned to prevent the system from becoming unstable and unable to return to the original voltage. The value of $\lambda$ is chosen to create an optimal range between actions of different sizes. Larger distances of voltage from the tolerant range require a larger action. A value of 0.3 is selected for this work.

Algorithm 1 Detection algorithm pseudocode.

1: if $(\mathrm{time}-\mathrm{Activated}-\mathrm{time})>100\mathrm{ms}\mathrm{and}$ $\mathrm{Action}-\mathrm{counter}\le 5$ then 2:     if ${\mathrm{INV}}_{\mathrm{set}}-{\mathrm{INV}}_{\mathrm{set},\mathrm{prev}}\ne 0$ then 3:         $\mathrm{Ready}-\mathrm{flag}=0$ 4:     else 5:         $\mathrm{Ready}-\mathrm{flag}=1$ 6:         $\mathrm{Activated}-\mathrm{time}=0$ 7:     end if 8: end if 9: if $\mathrm{Ready}-\mathrm{flag}=1\mathrm{and}\mathrm{action}\ne 4$ then 10:   $\mathrm{Detection}-\mathrm{flag}=1$ 11:   $\mathrm{Action}-\mathrm{counter}=\mathrm{Action}-\mathrm{counter}+1$ 12: end if 13: if $\mathrm{Detection}-\mathrm{flag}=1$ then 14:     ${\mathrm{INV}}_{set,change}=\mathrm{find}\_\mathrm{action}(V(t))$ 15:     ${\mathrm{INV}}_{set}$ = ${\mathrm{INV}}_{set}$ + ${\mathrm{INV}}_{set,change}$ 16: end if

4. Implementation Test System

The detection and mitigation methods require a physical layer to interact with for both training and testing the system. The physical layer of this work is represented by PSCAD, since it is capable of modeling and simulating IBRs with high accuracy. Figure 6 shows the physical layer of the system, represented by a single-line diagram with seven IBRs. The base power of inverters 1 and 12 are 10 MVA, that of inverters 3, 4, 8, 10, and 14 is 5 MVA, and the base voltage of all inverters is 480 V. The primary and secondary voltages of the transformers connecting the IBRs to the grid are 480 V and 12.47 kV. Each IBR has an output voltage of 480 V and a DC bus voltage of 1.2 kV. The filter resistance and reactance of the IBRs is 2 mΩ and 30 uH. IBRs 1, 4, 12, and 14 are set to the grid-following mode, and IBRs 3, 8, and 10 are set to the grid-forming mode. Gains of the PI block for the grid-forming mode in P − f droop control are $k_P=01.5$ and $k_I=0.02$. In the Q − V droop control, the grid-forming gains of the PI block are $k_P=0.65$ and $k_I=0.015$. The gains of the PI block for the grid-following mode are $k_P$ = 0.5 and $k_I=0.05$.

5. Performance Evaluation

This section discusses the simulation results for FDI attacks on the P and Q channels of the GFL units and V channel of the GFM units and under normal grid transients.

5.1. FDI Attack Cases

5.1.1. Positive Ramp Attack of GFL Inverters

At $t=0.05$ s, a FDI cyberattack is launched by attackers that increases the real power set point of IBR 1 from 5 MW to 5.5 MW, IBR 4 from 3 MW to 3.3 MW, IBR 12 from 5 MW to 5.5 MW, and IBR 14 from 3.5 MW to 3.85 MW. The reactive power set point is increased for IBR 1 from 2.1 MVAR to 2.31 MVAR, IBR 4 from 1.26 MVAR to 1.39 MVAR, IBR 12 from 2.1 MVAR to 2.31 MVAR, and IBR 14 from 1.47 MVAR to 1.62 MVAR over the course of 200 ms. Figure 7a,g,m,s shows the detection signal with and without mitigation applied and without an attack. Figure 7b shows the real power set point of IBR 1, where its pre-attack value is 5 MW and increases to 5.5 MW after the attack is launched in an undetected attack case. Figure 7c shows the reactive power set point of IBR 1, where its pre-attack value is 2.1 MVAR and increases to 2.31 MVAR after the attack is launched in an undetected attack case. IBR 1 does not call for the mitigation to be applied since the attack does not increase the voltage of IBR 1 above 1.04 pu. Figure 7h shows the real power set point of IBR 4, where its pre-attack value is 3 MW and increases to 3.3 MW after the attack is launched in an undetected attack case. Figure 7i shows the reactive power set point of IBR 4, where its pre-attack value is 1.26 MVAR and increases to 1.39 MVAR after the attack is launched in an undetected attack case. IBR 4 does not call for the mitigation to be applied since the attack does not increase the voltage of IBR 4 above 1.04 pu. Figure 7n shows the real power set point of IBR 12, where its pre-attack value is 5 MW and increases to 5.5 MW after the attack is launched in an undetected attack case. Figure 7o shows the reactive power set point of IBR 12, where its pre-attack value is 2.1 MVAR and increases to 2.31 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 5.1 MW and the reactive power set point is adjusted to 1.81 MVAR. Figure 7r shows IBR 12 bus voltage, where its pre-attack value is 1.03 pu and increases to 1.07 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 12 bus voltage results in a voltage of 1.02 pu. Figure 7t shows the real power set point of IBR 14, where its pre-attack value is 3.5 MW and increases to 3.85 MW after the attack is launched in an undetected attack case. Figure 7u shows the reactive power set point of IBR 14, where its pre-attack value is 1.47 MVAR and increases to 1.62 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 3.71 MW and the reactive power set point is adjusted to 1.34 MVAR. Figure 7x shows the IBR 14 bus voltage, where its pre-attack value is 1.03 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 14 bus voltage results in a voltage of 1.03 pu. The detection method for IBR 12 takes 100 ms to recognize the attack and an additional 125 ms to mitigate the attack, The detection method for IBR 14 takes 100 ms to recognize the attack and an additional 100 ms to mitigate the attack.

5.1.2. Negative Bias Attack of GFL Inverters

At $t=0.05$ s, a FDI cyberattack is launched by attackers that decreases the real power set point of IBR 1 from 5 MW to 4.5 MW, IBR 4 from 3 MW to 2.7 MW, IBR 12 from 5 MW to 4.5 MW, and IBR 14 from 3.5 MW to 3.15 MW. The reactive power set point is decreased for IBR 1 from 1.5 MVAR to 1.35 MVAR, IBR 4 from 0.9 MVAR to 0.81 MVAR, IBR 12 from 1.5 MVAR to 1.35 MVAR, and IBR 14 from 1.05 MVAR to 0.95 MVAR. Figure 8a,g,m,s shows the detection signal with and without mitigation applied and without an attack. Figure 8b shows the real power set point of IBR 1, where its pre-attack value is 5 MW and decreases to 4.5 MW after the attack is launched in an undetected attack case. Figure 8c shows the reactive power set point of IBR 1, where its pre-attack value is 1.5 MVAR and decreases to 1.35 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 4.9 MW and the reactive power set point is adjusted to 1.63 MVAR. Figure 8r shows the IBR 1 bus voltage, where its pre-attack value is 0.96 pu and decreases to 0.95 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 1 bus voltage results in a voltage of 0.98 pu. Figure 8h shows the real power set point of IBR 4, where its pre-attack value is 3 MW and decreases to 2.7 MW after the attack is launched in an undetected attack case. Figure 8i shows the reactive power set point of IBR 4, where its pre-attack value is 0.9 MVAR and decreases to 0.81 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 2.94 MW and the reactive power set point is adjusted to 1.01 MVAR. Figure 8l shows the IBR 4 bus voltage, where its pre-attack value is 0.96 pu and decreases to 0.95 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 4 bus voltage results in a voltage of 0.98 pu. Figure 8n shows the real power set point of IBR 12, where its pre-attack value is 5 MW and decreases to 4.5 MW after the attack is launched in an undetected attack case. Figure 8o shows the reactive power set point of IBR 12, where its pre-attack value is 1.5 MVAR and decreases to 1.35 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 5 MW and the reactive power set point is adjusted to 1.85 MVAR. Figure 8r shows the IBR 12 bus voltage, where its pre-attack value is 0.97 pu and decreases to 0.94 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 12 bus voltage results in a voltage of 0.99 pu. Figure 8t shows the real power set point of IBR 14, where its pre-attack value is 3.5 MW and decreases to 3.15 MW after the attack is launched in an undetected attack case. Figure 8u shows the reactive power set point of IBR 14, where its pre-attack value is 1.05 MVAR and decreases to 0.95 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 3.29 MW and the reactive power set point is adjusted to 1.09 MVAR. Figure 8x shows the IBR 14 bus voltage, where its pre-attack value is 0.98 pu and decreases to 0.96 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 14 bus voltage results in a voltage of 1 pu. The detection method for IBR 1 takes 50 ms to recognize the attack and an additional 100 ms to mitigate the attack. The detection method for IBR 4 takes 50 ms to recognize the attack and an additional 100 ms to mitigate the attack. The detection method for IBR 12 takes 50 ms to recognize the attack and an additional 125 ms to mitigate the attack. The detection method for IBR 14 takes 75 ms to recognize the attack and an additional 50 ms to mitigate the attack.

5.1.3. Positive Ramp Attack on GFM Inverters

At $t=0.05$ s, a FDI cyberattack is launched by attackers that increases the voltage reference points of IBR 3 from 1 pu to 1.04 pu, IBR 8 from 1 pu to 1.04 pu, and IBR 10 from 1 pu to 1.04 pu over the course of 200 ms. Figure 9a,d,g shows the detection signal with and without mitigation applied and without an attack. Figure 9b shows the voltage reference point of IBR 3, where its pre-attack value is 1 pu and increases to 1.04 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1.01 pu. Figure 9c shows the IBR 3 bus voltage, where its pre-attack value is 1.01 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 3 bus voltage results in a voltage of 1.02 pu. Figure 9e shows the voltage reference point of IBR 8, where its pre-attack value is 1 pu and increases to 1.04 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1.01 pu. Figure 9f shows the IBR 8 bus voltage, where its pre-attack value is 1.01 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 8 bus voltage results in a voltage of 1.03 pu. Figure 9h shows the voltage reference point of IBR 10, where its pre-attack value is 1 pu and increases to 1.04 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1.02 pu. Figure 9i shows IBR 10 bus voltage, where its pre-attack value is 1.01 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 10 bus voltage results in a voltage of 1.02 pu. The detection method for IBR 3 takes 225 ms to recognize the attack and an additional 75 ms to mitigate the attack. The detection method for IBR 8 takes 225 ms to recognize the attack and an additional 75 ms to mitigate the attack. The detection method for IBR 10 takes 225 ms to recognize the attack and an additional 50 ms to mitigate the attack.

5.1.4. Negative Bias Attack on GFM Inverters

At $t=0.05$ s, a FDI cyberattack is launched by attackers that decreases the voltage reference points of IBR 3 from 1 pu to 0.96 pu, IBR 8 from 1 pu to 0.96 pu, and IBR 10 from 1 pu to 0.96 pu. Figure 10a,d,g shows the detection signal with and without mitigation applied and without an attack. Figure 10b shows the voltage reference point of IBR 3, where its pre-attack value is 1 pu and decreases to 0.96 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 0.991 pu. Figure 10c shows the IBR 3 bus voltage, where its pre-attack value is 0.98 pu and decreases to 0.94 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 3 bus voltage results in a voltage of 0.98 pu. Figure 10e shows the voltage reference point of IBR 8, where its pre-attack value is 1 pu and decreases to 0.96 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1 pu. Figure 10f shows the IBR 8 bus voltage, where its pre-attack value is 0.98 pu and decreases to 0.94 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 8 bus voltage results in a voltage of 0.98 pu. Figure 10h shows the voltage reference point of IBR 10, where its pre-attack value is 1 pu and decreases to 0.96 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1.01 pu. Figure 10i shows the IBR 10 bus voltage, where its pre-attack value is 0.98 pu and decreases to 0.94 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 10 bus voltage results in a voltage of 0.98 pu. The detection method for IBR 3 takes 225 ms to recognize the attack and an additional 100 ms to mitigate the attack. The detection method for IBR 8 takes 225 ms to recognize the attack and an additional 100 ms to mitigate the attack. The detection method for IBR 10 takes 225 ms to recognize the attack and an additional 125 ms to mitigate the attack.

5.1.5. Positive Ramp Attack on GFL Inverters and Positive Bias Attack on GFM Inverters

At $t=0.05$ s, a FDI cyberattack is launched by attackers that increases the real power set point of IBR 1 from 5 MW to 5.5 MW and IBR 14 from 3.5 MW to 3.85 MW. The reactive power set points are increased for IBR 1 from 2.1 MVAR to 2.31 MVAR and IBR 14 from 1.47 MVAR to 1.62 MVAR over the course of 200 ms. The voltage reference points are increased for IBR 3 from 1 pu to 1.05 pu and IBR 8 from 1 pu to 1.05 pu. Figure 11a,g and Figure 12a,d show the detection signal with and without mitigation applied and without an attack. Figure 11b shows the real power set point of IBR 1, where its pre-attack value is 5 MW and increases to 5.5 MW after the attack is launched in an undetected attack case. Figure 11c shows the reactive power set point of IBR 1, where its pre-attack value is 2.1 MVAR and increases to 2.31 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 5.2 MW and the reactive power set point is adjusted to 2.12 MVAR. Figure 11f shows the IBR 1 bus voltage, where its pre-attack value is 1.02 pu and increases to 1.06 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 1 bus voltage results in a voltage of 1.03 pu. Figure 11h shows the real power set point of IBR 14, where its pre-attack value is 3.5 MW and increases to 3.85 MW after the attack is launched in an undetected attack case. Figure 11i shows the reactive power set point of IBR 14, where its pre-attack value is 1.47 MVAR and increases to 1.62 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 3.15 MW and the reactive power set point is adjusted to 0.98 MVAR. Figure 11l shows the IBR 14 bus voltage, where its pre-attack value is 1.04 pu and increases to 1.07 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 14 bus voltage results in a voltage of 1.04 pu. Figure 12b shows the voltage reference point of IBR 3, where its pre-attack value is 1 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1.04 pu. Figure 12c shows the IBR 3 bus voltage, where its pre-attack value is 1.01 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 3 bus voltage results in a voltage of 1.02 pu. Figure 12e shows the voltage reference point of IBR 8, where its pre-attack value is 1 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 1.03 pu. Figure 12f shows the IBR 8 bus voltage, where its pre-attack value is 1.01 pu and increases to 1.05 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 8 bus voltage results in a voltage of 1.02 pu. The detection method for IBR 1 takes 100 ms to recognize the attack and an additional 75 ms to mitigate the attack. The detection method for IBR 14 takes 75 ms to recognize the attack and an additional 250 ms to mitigate the attack. The detection method for IBR 3 takes 125 ms to recognize the attack and an additional 25 ms to mitigate the attack. The detection method for IBR 8 takes 100 ms to recognize the attack and an additional 50 ms to mitigate the attack.

5.1.6. Negative Bias Attack on a GFL Inverter and Negative Ramp Attack on a GFM Inverter

At $t=0.05$ s, a FDI cyberattack is launched by attackers that decreases the real power set point of IBR 1 from 5 MW to 4.5 MW and IBR 12 from 5 MW to 4.5 MW. The reactive power set points are increased for IBR 1 from 1.5 MVAR to 1.35 MVAR and IBR 12 from 1.5 MVAR to 1.35 MVAR. The voltage reference points are decreased for IBR 3 from 1 pu to 0.95 pu and IBR 8 from 1 pu to 0.95 pu over the course of 200 ms. Figure 13a,g and Figure 14a,d show the detection signal with and without mitigation applied and without an attack. Figure 13b shows the real power set point of IBR 1, where its pre-attack value is 5 MW and decreases to 4.5 MW after the attack is launched in an undetected attack case. Figure 13c shows the reactive power set point of IBR 1, where its pre-attack value is 1.5 MVAR and decreases to 1.35 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 4.9 MW and the reactive power set point is adjusted to 1.65 MVAR. Figure 13f shows the IBR 1 bus voltage, where its pre-attack value is 0.97 pu and decreases to 0.92 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 1 bus voltage results in a voltage of 0.97 pu. Figure 13h shows the real power set point of IBR 12, where its pre-attack value is 5 MW and decreases to 4.5 MW after the attack is launched in an undetected attack case. Figure 13i shows the reactive power set point of IBR 12, where its pre-attack value is 1.5 MVAR and decreases to 1.35 MVAR after the attack is launched in an undetected attack case. With the mitigation method applied, the real power set point is adjusted to 5 MW and the reactive power set point is adjusted to 1.76 MVAR. Figure 13l shows the IBR 12 bus voltage, where its pre-attack value is 0.97 pu and decreases to 0.9 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 12 bus voltage results in a voltage of 0.98 pu. Figure 14b shows the voltage reference point of IBR 3, where its pre-attack value is 1 pu and decreases to 0.95 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 0.99 pu. Figure 14c shows the IBR 3 bus voltage, where its pre-attack value is 0.96 pu and decreases to 0.92 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the IBR 3 bus voltage results in a voltage of 0.97 pu. Figure 14e shows the voltage reference point of IBR 8, where its pre-attack value is 1 pu and decreases to 0.95 pu after the attack is launched in an undetected attack case. With the mitigation method applied, the voltage reference point is adjusted to 0.99 pu. Figure 14f shows the IBR 8 bus voltage, where its pre-attack value is 0.96 pu and decreases to 0.92 pu after the attack is launched in an undetected attack case.With the mitigation method applied, the IBR 8 bus voltage results in a voltage of 0.97 pu. The detection method for IBR 1 takes 50 ms to recognize the attack and an additional 100 ms to mitigate the attack. The detection method for IBR 12 takes 75 ms to recognize the attack and an additional 125 ms to mitigate the attack. The detection method for IBR 3 takes 50 ms to recognize the attack and an additional 100 ms to mitigate the attack. The detection method for IBR 8 takes 50 ms to recognize the attack and an additional 100 ms to mitigate the attack.

5.1.7. Overview of FDI Attack Cases

The results of the FDI attack cases are compiled into

Table 1. Results of all FDI attack cases.

Attack Case	Inverter	Detection Time (ms)	Mitigation Time (ms)	Mitigation Accuracy (%)
GFL positive ramp attack	IBR 12 IBR 14	100 100	125 100	99.029 100
GFL negative bias attack	IBR 1 IBR 4 IBR 12 IBR 14	50 50 50 75	100 100 125 50	97.917 97.917 97.938 97.959
GFM positive ramp attack	IBR 3 IBR 8 IBR 10	225 225 225	75 75 50	99.01 98.02 99.01
GFM negative bias attack	IBR 3 IBR 8 IBR 10	225 225 225	100 100 125	100 100 100
GFL positive ramp attack and GFM positive bias attack	IBR 1 IBR 3 IBR 8 IBR 14	100 125 100 75	75 25 50 250	99.02 99.01 99.01 100
GFL negative bias attack and GFM negative ramp attack	IBR 1 IBR 3 IBR 8 IBR 12	50 50 50 75	100 100 100 125	100 98.958 98.958 98.969

. The results show a low relative error with the application of the proposed mitigation method. The minimum detection and mitigation times are 50 ms, with two consecutive time steps. The maximum detection and mitigation times are 225 and 250 ms, respectively. The minimum mitigation accuracy in the study cases is 97.917%. In all cases, the proposed method successfully recovers the voltage to below 1.05 and above 0.95 pu.

5.2. Under Grid Transient

This section studies the performance of the proposed method under the grid transients.

5.2.1. Simultaneous Set Point and Load Changes

A load increase of 0.655 MW and 0.658 MVAR is attached to bus 1 at $t=0.05$ s. After 100 ms, the power set points of IBR 1, both real and reactive, are increased to manage the increase in load. Figure 15a shows the detection signal with and without the mitigation method. Figure 15b shows the decrease of bus 1 voltage to 0.96 pu after the load change and the increase to 1 pu following the IBR set point adjustments. Figure 15c,d show the power set points, real and reactive, with and without the mitigation method.

5.2.2. Three-Phase Short-Circuit Fault

A three-phase short circuit occurs on bus 1 at $t=0.05$ s. Figure 16a shows the detection signal with and without the mitigation method. Figure 16b shows the voltage of bus 1 with and without the mitigation method. The detection method ensures that the mitigation method does not take any actions during the fault.

6. Future Work

The proposed detection and mitigation method is tested using the modified CIGRE 14-bus North American grid with seven IBRs in PSCAD. Real-time simulation shows higher fidelity compared to software-based simulations and ensures the implementation of the proposed method in real time. Future work includes testing and validating the proposed method using a real-time testbed. However, simulating a grid with seven inverters requires a powerful real-time simulator with multiple cores, which may not always be available. Testing of the mitigation method found that it takes an average of 12.263 ms to find an action for $P_{set}$, $Q_{set}$, and $V_{ref}$. This was done using a PC with an i7 core CPU and 16 GB of RAM. Achieving 25 ms using an off-the-shelf microcontroller can be challenging due to computational complexity. The solution is to decrease the computational complexity of the RL model. Moreover, running the detection and mitigation algorithm for all inverters on one microcontroller can be computationally expensive. One solution to this problem is to use a separate microcontroller for each inverter. Using a larger grid also requires a more powerful real-time simulator capable of modeling a grid with more nodes and inverters.

7. Conclusions

IBRs receive their set point through the communication link, which may expose them to cyber threats. Previous work has developed various techniques to detect and mitigate cyberattacks on IBRs, developing scheme for new inverters being installed in the grid. This work focuses on developing model-free control techniques for already installed IBR in the grid without the need to access IBR internal control parameters. The proposed method is tested for both grid-forming and grid-following inverter control. Separate detection and mitigation algorithms are used to enhance the accuracy of the proposed method. The proposed method is tested using the modified CIGRE 14-bus North American grid with seven IBRs in PSCAD/EMTDC. Finally, the performance of the detection algorithm is tested under grid normal transients, such as set point change, load change, and short-circuit fault, to make sure the proposed detection method does not provide false positives. The proposed method works even when multiple inverters are under cyberattack.