Intelligent Integration of Renewable Energy Resources Review: Generation and Grid Level Opportunities and Challenges

Abstract

This paper reviews renewable energy integration with the electrical power grid through the use of advanced solutions at the device and system level, using smart operation with better utilisation of design margins and power flow optimisation with machine learning. This paper first highlights the significance of credible temperature measurements for devices with advanced power flow management, particularly the use of advanced fibre optic sensing technology. The potential to expand renewable energy generation capacity, particularly of existing wind farms, by exploiting thermal design margins is then explored. Dynamic and adaptive optimal power flow models are subsequently reviewed for optimisation of resource utilisation and minimisation of operational risks. This paper suggests that system-level automation of these processes could improve power capacity exploitation and network stability economically and environmentally. Further research is needed to achieve these goals.

1. Introduction

Considerable efforts are being made to de-carbonise electrical power networks, where renewable energy resources such as wind and solar present a viable alternative to carbon-based sources. The ongoing availability and security of global energy is one of the key blockages to future sustainability [1], and further research and investment are needed for effective large-scale adoption in the coming years. The continued growth of renewables capacity, led by wind and solar, complicates the power grid composition and, in particular, how it is operated to deliver energy reliably. Intelligent solutions are needed to ensure optimal exploitation and grid integration of renewables. This paper addresses two vital aspects of renewables integration by exploring possibilities for advanced solutions in this space from both the generating device and the power system operation perspectives. The first aspect examines how the capacity of existing wind turbine (WT) generators may be expanded at low cost through advanced control to exploit device design margins. The second aspect considers the application of machine learning methods to assist with the necessary power flow optimisation in a power network dominated by low-carbon renewables.

Enhanced utilisation of the existing wind turbine capacity essentially looks at a low-cost retrofittable extension of the wind generator’s nominal operational envelope. Such a solution could increase in-service capacity above the pre-installation design rating without replacing major system components. However, this requires the system components to be operated at a higher than nominal rating; hence, a better understanding of the in-service stresses is required. Improved sensing through advanced condition monitoring techniques for thermal feedback, integrated with the WT control for power generation management, is generally needed to facilitate such schemes. Thermal margins in WT electrical generators [2] and power electronic converters [3] can be sizeable, and their full exploitation could provide an increase in in-service system capacity.

The WT industry has reported limited exploration of technologies to increase the performance of in-service WT generators and hence annual energy production: up to a 5% increase is identified through the use of ‘over-rating control’, depending on the size and specifications of the upgraded WT system [4,5]. This was accomplished by inserting additional hardware and software upgrades, taking into account site conditions such as ambient temperature, wind speed, generator, and grid side voltages, and most importantly, the drive-train components’ current loading; these are seen as key factors in determining the WT operational envelope [4]. However, ref. [6] argues that the limiting factor is hotspot temperature rather than current, so credible real-time WT drive-train temperature measurements are necessary to extend the WT operation envelope. The existing WT drive-trains employ conventional temperature sensors such as thermocouples and resistance temperature detectors for this purpose [7]. Despite offering effective, low-cost sensing solutions, these conventional sensors have access limitations and are electrically conductive, which may cause safety issues [8]. Crucially, it is difficult to locate conventional sensors where the key device hotspots occur (e.g., generator winding coil centres, power electronic switch junctions).

Fibre optic Bragg grating (FBG) sensing technology has recently emerged as a viable alternative, offering the capability for in situ, in-service distributed hotspot measurement that simultaneously provides electrical isolation and is immune to electromagnetic interference [9]. Despite its wide commercial usage for WT blade strain monitoring [10], FBG sensing applications for WT drive-train temperature measurements have not yet received much attention among WT manufacturers. However, academic research works have shown the feasibility and robustness of FBG sensors for temperature measurements in various parts of electrical machines, such as end-windings, stator slot centres, and rotor surfaces, [9,11] but also in power electronic switches where direct thermal on-chip thermal sensing was shown to be possible [12,13,14]. These sensing applications allow for unparalleled awareness of the thermal conditions of key device locations and could be integrated into modern WT generators, which can directly be translated into a much-improved understanding of the in-service operating envelope.

The real-time integration of temperature sensors with an electrical machine and power converter controller would thus provide a way to extend the wind generator’s operating capacity in-service, past the conservative nominal values, in a controllable manner. Research works have demonstrated that bespoke electrical machine drives with closed-loop thermal feedback are integrated with the relevant field-oriented controllers for improved performance in automotive applications [15,16,17], but schemes of this type have not been widely researched in wind power generation. Similarly, the research on FBG sensing applications in electrical machines and drives has, to date, been largely devoted to understanding the sensing implementation without integrating these capable sensors with real-time control for improved performance management. This paper aims to review the available literature and build on this to explore a possible framework to implement FBG sensing and thermal management of a WT generator with overrating control, as well as the general requirements for its implementation.

The second aspect of this paper centres on the transformative impact that machine learning (ML) technologies have on optimal power flow (OPF) within modern power systems, which are integrating renewable energy sources at an unprecedented rate. As the energy landscape shifts towards renewables like wind and solar, the inherent variability and unpredictability of these sources pose significant challenges to traditional OPF models [18]. These models, originally designed for more stable and predictable energy sources, are not equipped to handle the dynamic fluctuations that renewable energies introduce. This situation necessitates a paradigm shift from static and deterministic OPF models to those that are dynamic and adaptive, capable of real-time analysis and response. ML offers an innovative solution, employing sophisticated algorithms to process continuous streams of data from grid sensors and smart meters. Thus, ML enables the real-time optimisation of power flows and predictive monitoring of the system’s operational health. This dynamic learning and adaptive response capability ensure that the grid can maintain stability and efficiency even under the fluctuating conditions that renewables introduce [19]. Moreover, the integration of ML into OPF can lead to more informed and proactive management strategies, enhancing the grid’s ability to cope with immediate and future challenges while optimising resource utilisation and minimising operational risks.

Expanding further, the incorporation of machine learning into OPF redefines the boundaries of grid management from a computational task to a strategic governance framework. With ML, the grid is not only a network of physical power flows but also a platform for intelligent decision-making, where data-driven insights lead to better control and optimisation decisions [20]. This advanced approach facilitates a transition from reactive to proactive grid management, where potential issues can be anticipated and mitigated before they escalate. Furthermore, the ability of ML to integrate with existing grid infrastructure introduces a layer of resilience and adaptability previously unattainable with conventional OPF methods [21]. This paper, therefore, also reviews the specific ML techniques that enhance OPF, such as deep learning and reinforcement learning, examining their roles in optimising grid operations against the backdrop of increasing renewable integration. This discussion complements this review by providing an outline of the necessary technological advancements and proposing changes in regulatory frameworks to effectively incorporate these intelligent systems into everyday grid operations. The underlying aim is to provide insights into a possible path forward for energy systems, emphasising the critical role of machine learning in ensuring that the grid not only survives but continues to improve its functionality in the face of evolving global energy demands and the push toward sustainability.

This paper is organised as follows. Section 2 starts with a brief background regarding the advanced monitoring and control for the over-rating operation of a WT. The rest of Section 2 is devoted to reviewing the relevant topics such as the WT thermal condition monitoring, electrical machine thermal design limitations and margins, thermal feedback integration electrical machine controller, and WT power curve upgrade. Section 3 reviews the application of machine learning methods to optimal power flow, discussing both deterministic and probabilistic OPF models, the integration of deep learning and reinforcement learning techniques, and the role of these technologies in enhancing real-time grid operation and management.

2. Advanced Monitoring and Control for Optimised Exploitation

This section presents a review of challenges and opportunities for improving the exploitation of WT generators through over-rating control, underpinned by advanced in situ thermal monitoring. The WT generator thermal monitoring is first reviewed, followed by an overview of the generator’s operating margins and limitations. The possible control architectures are then presented, and the general implementation requirements of the thermally controlled over-rating capability in variable-speed WT generators are explored.

Monitoring of thermal, mechanical, and electrical operating parameters in wind turbines (WTs) plays a vital role in managing their in-service utilisation. This is particularly relevant for the WT drivetrain and its generator and converter, which are the main electromechanical energy conversion components. In-service abnormalities can cause deviations from recognised parameter values for operation in the nominal range [22]. The ability to measure these key operating parameters of WT subassemblies whilst in-service is imperative for the recognition of abnormal operating states in time to establish mitigating actions.

Targeted observation of temperature rise in WT components has been used for fault detection [23]. The nominal current ratings of the WT generator and power converter are directly associated with permissible thermal levels in their windings and power electronic switch junctions, respectively. The accurate observation of worst-case, in-service temperature in these may permit over-rating power to be extracted by the WT in order to yield a desirable increase in energy recovery. The aim is to load the generator beyond the design temperature in nominal conditions whilst ensuring its integrity is not or is minimally compromised. For this to be achieved, in addition to improved monitoring, advanced control routines are needed that can react intelligently to improved sensing feedback and are able to deliver improved WT operational capability, while keeping its assets within safe integrity margins. Examples include allowing controlled overloads under cold ambient conditions or for short durations, with no or minimal risk of damage, or alternatively extending service life in faulty conditions through redistributing load to other WTs. The availability of such solutions would open attractive opportunities to develop more resilient WT systems needed to underpin our Net Zero transition.

2.1. Wind Turbine Thermal Condition Monitoring

WT thermal monitoring has long been used and remains standard in practical applications, with a range of thermal sensors fitted to WT drivetrains [24]. The sensor type, location, and its measurand fidelity and resolution can vary across different possible monitoring solutions, extending from, e.g., low-resolution measurements provided through WT supervisory control and data acquisition (SCADA) systems to higher resolution measurements from dedicated condition monitoring platforms [25]. This section provides an overview of the general thermal monitoring techniques and their use in WT drivetrains and identifies potential techniques for achieving improved sensing.

Existing regulations for WT system certification stipulate the minimal set of thermal and other sensing points for the entire WT structure and, in particular, its drivetrain [7]. Where thermal monitoring of the drivetrain and the generator is concerned, the use of conventional thermal sensing elements (e.g., thermocouple (TC) or resistance temperature detector (RTD)) is recommended. Sensors may be embedded in various locations of interest, such as the end winding, winding slot centre, and stator pack laminations [26]. A Siemens WT commercial condition monitoring system known as SIPLUS CM [27] utilises vibration signals as well as temperature signals measured from the WT drive train components, including the generator, through a SIMATIC S7 module supporting the use of various different TC and RTD sensors.

Conventional TC and RTD thermal sensors are electrically conductive and require wiring, so they cannot easily be placed in close contact with the active current-carrying copper conductor in an arbitrary position. Due to these sensors’ installation requirements and bulk, the locations where the hottest temperatures occur can be impractical or challenging to measure; hence, the hottest temperature measurement of the active copper conductors in a machine may be underestimated. One such scenario is illustrated in Figure 1, where for practical reasons, TC sensors were installed away from the slot centre where the hotspot temperature occurs [26].

The sensors and wiring can occupy a relatively large space, so they cannot provide sufficient temperature measurement points for detailed thermal mapping. The electrical conductivity of conventional sensor-based thermal sensing makes the monitoring system complicated and less reliable [28]. In addition, conventional sensors have low immunity to electromagnetic interference (EMI).

Due to the disadvantages of TC and RTD sensors for temperature monitoring, there is a continued interest in alternative temperature sensors, which could operate effectively in EMI-rich and electrically conductive environments. A fibre optic sensing technology known as fibre Bragg grating sensors (FBGs) has emerged that offers the desired features needed to provide improved in-service monitoring solutions for electrical machines. The FBGs can perform multi-physical sensing [11], possess a multiplexing capability, are electrically non-conductive, exhibit a high level of immunity to EMI, are suitable for use in harsh environments, and are of small size and thus suited to applications where weight and size are critical [29]. An additional beneficial feature of FBGs is their superior data transmission over a long distance without any data loss [30].

FBGs require a laser source and interrogation unit, which is expensive compared with TC and RTD technologies. However, FBGs are widely employed by WT manufacturers as strain sensors for WT blades and for structural health monitoring. The FBGs can be installed at multiple points on the turbine blades or tower, facilitating the detection of small cracks and abnormalities in rotating blades and their structure [10]. Examples of commercially installed FBGs in Portugal, the United Kingdom, and France for strain measurement operated at various sampling rates of 25 Hz, 100 Hz, and 2 kHz and were designed and utilised to detect cracks in the blades, unbalanced turbine towers or blades, and icing thickness [31].

Despite being commercially employed for strain measurements, FBGs have not yet received sufficient attention from WT manufacturers for drive train monitoring applications, particularly for generator and converter thermal condition monitoring. Recent research has demonstrated the feasibility of thermal sensing using FBGs embedded in various locations within electrical machines, such as stator end-windings [32], slot centres [8], rotors [33], and bearings [34], as well as power electronic switches [13]. In these studies, a single FBG or an array of FBGs was installed in the points of interest in the studied device geometry, and in-service tests were performed under different practical healthy and fault conditions, indicating reliable response and measurement of temperature.

There are specific requirements for the successful implementation of FBGs during installation and operation [35]. The FBG’s intrinsic cross-sensitivity to temperature and strain needs to be addressed through appropriate sensor packaging to allow exclusive sensing of thermal only or strain only [8]. FBGs are of small size and flexible and thus allow for in situ observation of localised excitation; however, the accurate determination of precise locations of highest excitation to sense can be a challenge in practical device geometries. Experimentally verified modelling [8] has therefore been employed to determine the optimal FBG sensor positions. The FBG sensor-to-measurand interface also requires careful consideration, where suitable packaging is often required to protect the sensor and ensure proper functionality during electrical machine operation [32]. While FBGs have shown reliable, in situ thermal and other monitoring, the interpretation of the diagnostic information contained in the high-fidelity thermal data requires further research [9], including for WT drivetrains.

2.2. Thermal Design Limits and Margins

Three factors, electromagnetic, mechanical, and thermal, limit the current or torque density in electrical machines [36]. The saturation level of the core magnetic materials selected in the design phase determines electromagnetic limitations. The maximum mechanical operating speed is constrained by the stiffness of the bearings and shaft. The thermal limit of electrical motors and generators is determined by the winding insulation temperature, which is one of the most vulnerable parts of the machine when subjected to thermal excitation caused by nominal or abnormal operating conditions. In permanent magnet machines, temperature dependency of the demagnetisation characteristic is also a constraint.

The National Electrical Manufacturers Association (NEMA) [2] classified insulation system classes by letters A, B, F, and H, specifying thermal ratings associated with each class. The ambient temperature of $40°\mathrm{C}$ has been established as a reference for all of the insulation classes, followed by the maximum temperature rise of each class. The combination of the ambient temperature and the temperature rise determines the maximum allowed operating temperature for a given insulation class. For example, for all induction machines rated above 1 kW, continuously operating at a service factor (SF) of 1 and 1.15, insulation class A has the lowest permitted temperature rise of 60 °C and 65 °C, respectively, while insulation class H has the highest temperature rise of 125 °C and 135 °C, as shown in

Table 1. Insulation class rating measured by resistance method at service factors 1 and 1.15 for all induction machines above 1 kW rating [2].

NEMA Insulation Class Rating Measured by Resistance Method	Temperature Rise in Degrees, °C Starting from the Ambient Temperature of 40 °C
Insulation class	SF 1	SF 1.15
A	60	65
B	80	85
F	105	110
H	125	135

. The Table 1 data are obtained by the average winding temperature measurement using the “resistance method” detailed by the IEEE Std 112 [37] since winding resistance is temperature-dependent. This method neglects winding hotspot temperature measurement. To overcome this issue, NEMA utilises slot-embedded TCs and RTDs temperature sensors to measure the winding hotspot temperature in the slots.

Table 2. Insulation class rating is measured by slot-embedded TC and RTD at service factors 1 and 1.15 for induction machines of over 1120 kW rating [2].

NEMA Insulation Class Rating Measured by Slot-Embedded Sensors	Temperature Rise in Degrees, °C Starting from the Ambient Temperature of 40 °C
Insulation class	SF 1	SF 1.15
A	65	75
B	85	95
F	110	120
H	135	145

shows the NEMA stipulated temperature rise of all insulation classes for induction machine ratings above 1120 kW at SF 1 and 1.15 (continuous operation) measured by the winding slot-embedded sensors. The limitations of slot-embedded TCs and RTDs have been detailed in the previous section: due to the practical challenges of measuring the point of highest temperature reliably with these sensors, a hot spot temperature allowance is often introduced to provide a thermal safety margin. An interesting in situ sensing alternative is presented by the FBG sensor, where sensors can be embedded in slot centre to facilitate credible measurement of the winding temperature hotspots without safety and size concerns [8].

The values in Table 1 and Table 2 give winding insulation temperature thresholds that are typically higher than the hotspot temperatures of in-service machines operating in their nominal rated conditions [38]. For instance, the winding hotspot temperatures of a commercial 0.55 kW induction motor and 5.5 kW permanent magnet motor, measured using FBGs in a healthy full-load continuous duty cycle (S1), were 96 °C at an ambient temperature of 23 °C [8] and 80 °C at an ambient temperature of 21 °C [32], respectively. The test motor insulations were class F with class B temperature rise, corresponding to a thermal rating of 155 °C, with an 85 °C rise, as specified by NEMA. For large machines, thermal sensing using FBGs for a 42 MW hydropower generator was reported in [39], where the recorded stator winding surface temperature was 95 °C during full-load operation conditions. Therefore, it is clear that, typically, there is a thermal design margin in practical applications. This margin offers insulation lifetime extension and further thermal safety [38], as the lifetime of winding insulation is inversely proportional to the winding operating temperature. For any 10 °C increase in winding temperature, the insulation lifetime is decreased by half [36]. Similarly, by lowering the winding operating temperature by 10 °C, the insulation lifetime is doubled. Steady-state operation within a lower temperature range can also increase thermal safety margins in overload conditions, with a variable-speed drive, and with time-varying duty cycles and transients [38]. However, the potential extra capacity that could be extracted through over-rating, by exploiting of the thermal design margins (i.e., by running windings hotter), can present attractive opportunities for increasing the output in some applications.

2.3. Integration of Closed-Loop Thermal Feedback with Electrical Machine Control

Despite the possible extra capacity contained in the thermal margins, only a limited number of researchers have explored the operation of electrical machines close to their thermal design limits. This would only be possible if the thermal state of the machine is reliably and accurately measured and integrated with real-time control, able to facilitate an optimal trade-off between more torque (or power) and higher temperature, which can potentially reduce insulation lifetime for a given operating scenario [40].

Closed-loop temperature feedback for active thermal management has been implemented on a switched reluctance motor [15], a permanent magnet motor [16], and an induction motor [17] for automotive applications to extract short bursts of higher manoeuvring torque. Refs. [15,16,17] employed model predictive control (MPC) in conjunction with simple and complex lumped parameter thermal networks for temperature estimation. Motor losses were first calculated as inputs to the thermal network, and then the temperatures were predicted, converted into a current limit, and fed back to a field-oriented torque controller, as illustrated in Figure 2. This mechanism enables a thermally controlled machine to limit the operating temperature to a desired reference point, which cannot be guaranteed in a controlled machine without thermal estimated temperature feedback either sensed or derived from a lumped parameter thermal network (LPTN) or an alternative estimator [16]. With this proposed active thermal control, if a measured temperature is lower than its set point, the machine can be pushed to allow a higher current and so torque, and if a machine’s temperature is close to/or exceeding the design limit, the controller acts to reduce the current or torque limit leading to temperature reduction. Further research explored an increase in performance of an emulated automotive drive using active thermal management integrating both the power electronic device and the motor winding temperatures in real-time with a field-oriented controller considering not only conventional voltage and current boundaries but also the thermal design limits [40]. The thermal monitoring in this work is, however, either estimated using a simple thermal model which could underestimate the temperature, or via a complex thermal model that could be subject to error and increases computational requirements. While limited, the existing research on active thermal control of electric motors and drives indicates a strong potential for intelligent and reasonably low-cost output capacity improvement. In addition, the existing work is largely based on utilising estimator type models for thermal monitoring; hence, improving the quality of real-time thermal measurement feedback would be of benefit to further improve the efficacy of schemes of this type in various applications.

2.4. Wind Turbine Overload Capability and Extracting More Energy

Improvements in the existing WT systems to capture more wind energy through over-rating have been investigated independently by WT manufacturers. The “Energy thrust” by Siemens Gamesa (Hamburg, Germany) [4] and “PowerPlus” by Vestas (Aarhus, Denmark) [5] both claim to enable an annual increased energy production (AEP) of up to 5%. Examples of the upgraded commercial turbine models are SWT 2.3, 3, and 3.6, manufactured by Siemens Gamesa, and V82-1.65MW, V90-1.8MW, and V100-1.8MW, manufactured by Vestas. Both turbine manufacturers have upgraded the entire power curve operating regions in this process: the maximum power point tracking (MPPT) region, the constant power region, and the cut-out wind speed extension. The original and the upgraded power curves reported for a typical WT by Vestas are displayed in Figure 3 [5,41]. In the MPPT region, the aerodynamics have been upgraded using vortex generators mounted on the turbine blades: this is claimed to enhance the process of lift generation on the blades and, therefore, provide a better aerodynamically performing wind turbine rotor, resulting in more power extraction from the wind stream. In the full load operating region (i.e., the constant power region), the original power curve has been uprated by making use of load margins performed by taking into account the site condition thresholds (ambient temperature, current ratings of WT system components, gusty wind level, and the magnitudes of both converter and grid side voltages), implemented through adjustment of control parameters. In the constant power region of operation, the WT operates at a new maximised capacity with no upgrade or replacement to the core components, such as the generator or power converter hardware. The turbine cut-out wind speed is also extended from 25 m/s to 30 m/s, contributing to the increase in turbine output power.

The manufacturers have indicated that the effective implementation of these techniques is highly reliant on more reliable sensing of multiple WT measurands. However, due to the lack of information available in the public domain, the details of the existing work on WT power curve upgrade through over-rating are not fully understood. The commercial work adjusts the current limits in key turbine power conversion components based on ambient temperature [41] with control of the WT operating point through a combination of reference torque and pitch control.

Since the fundamental physical constraint is temperature rather than current, thermal design limits and temperature measurements offer a better way to set the degree of WT over-rating. Enhanced closed-loop thermal feedback, similar to that discussed in the previous section for electric vehicles, could be applied to a WT [6,42] For increased power yield, a distributed FBG sensor network is proposed to monitor in situ thermal hotspots across the WT power electronic drive and the electrical generator to be integrated with a dedicated real-time controller (as illustrated in Figure 4). With such feedback, an appropriate control would be able to react to prevailing wind conditions and real-time grid demand to set the generator/drive operating point to achieve different goals, such as the following examples:

(a)

Operating close to or at the thermal design limit in conditions of high wind, so the WT energy yield is increased;

(b)

Temporarily exceeding the thermal design limit in a controlled fashion in scenarios requiring a sudden and large power injection into the grid, for grid frequency support, or to compensate for the failure of another WT.

The availability of such an active thermal sensing scheme would have the potential to provide more resilient WT drives capable of more intelligent usage of the existing hardware capacity.

2.5. Discussion and Summary

With the push towards clean energy, over-rating of existing renewable generation installations is attractive, particularly in wind power, where there are plentiful opportunities to uprate existing WTs to increase the available energy output. The key to this is ensuring improved, real-time monitoring of component temperatures with more intelligent power management. FBG temperature sensors have been shown to be effective in power conversion devices and generator systems. Moreover, FBG sensors have already been applied for structural monitoring in WTs, so some of the implementation infrastructure is available in the field. The integration of improved sensor feedback with enhanced control would allow the development of more resilient WT drives, able to utilise active thermal control for increased power output or grid support at a minimal cost where there is an already existing fibre optic interrogation infrastructure (such as that used for in situ blade strain monitoring). However, while the general cost of FBG sensing is continuously reducing and FBG sensors are now largely generally comparable in cost to alternative conventional sensing, the cost of interrogator systems needed to illuminate and operate the sensing fibres remains reasonably high. While this cost can be prohibitive for condition monitoring and sensing applications of FBG technology in low-value assets, for large high-value assets such as WT systems, it is comparable to alternative commercially available high-end condition monitoring solutions [9]. Furthermore, the operational advantages and possible ancillary service potential of WT systems retro-fitted with active thermal capability would have the potential to generate extra revenue from energy production and grid support that would, over time, offset the installation cost of in situ monitoring systems. Finally, the development of alternative low-cost solutions for reliable thermal feedback based on advanced in situ sensing-based validated thermal estimators would provide alternate low-cost methods for thermal monitoring but requires further research.

This would allow for both the improvement of legacy WT equipment that has been in field operation for an extended time and the enhancement of modern WT designs. There is already demonstrable industrial interest in the development and application of these techniques; however, much further work is needed to facilitate the over-rating functionality in the field on a large scale and ensure the methodology is transparent and applicable to more modern WT designs.

3. Optimal Power Flow with Machine Learning

An optimal power flow (OPF) was initially proposed in 1962 by Carpentier [43]. The OPF is a non-convex, non-linear, and large-scale optimisation problem. OPF problems have been solved by the grid operator by finding the most economical generation dispatch point to meet electric demand while satisfying all the equality and inequality constraints of the network [44]. In other words, OPF assists the grid operator in controlling the power flow within the power gird without violating grid constraints. Moreover, it gives the operator useful support in the planning and operation of the grid [45].

The OPF problems can be categorised into two groups. The first group is deterministic OPF (D-OPF), and the other group is probabilistic OPF (P-OPF) [46]. D-OPF has been widely used to solve optimal flow problems. This type of OPF does not consider stochastic features, which means explicit values of the electricity demand and sustainable generation are required to deal with this type of problem. A variety of methods have been developed to solve D-OPF, e.g., evolutionary algorithm [47] and swarm intelligence [48]. However, the nonlinearity characteristics of equality constraints in the power network introduced by loads or generators make the swarm intelligence approaches unsuitable for solving OPF problems effectively. In contrast, evolutionary algorithms can be highly effective in optimising P-OPF when the solution space is adequately small or a considerable amount of time is available for the optimisation process [49].

However, electrical power systems have now become highly stochastic and uncertain, especially when distributed generators (DGs) like wind turbines and solar photovoltaics are connected in the generation process. In fact, it is difficult to use the optimisation methods mentioned above in solving the OPF within a sufficient time, principally when the stochastic behaviour of the DGs and uncertainty of the demand are considered [50].

3.1. Machine Learning Methods for OPF

Recently, driven by the growing amount of data due to using extensively smart sensors and meters in energy production and consumption, data-driven approaches with machine learning (ML) have been developed to use these data to overcome the limitation of the aforementioned methods in solving the OPF problems. ML methods provide the system the capability to automatically learn from historical data and improve its abilities without requiring an entire system identification or prior information of the environment [51]. In other words, ML methods are an efficient tool to deal with the uncertainty of the power system by generating optimisation and control decisions in real time. Therefore, ML methods are very powerful for solving OPF in real time by taking into consideration the uncertainty and stochasticity of the power system variables. ML approaches are divided into many methods, two of which are considered the most promising approaches in solving OPF in real time, namely (i) deep learning (DL) and (ii) reinforcement learning (RL) [52].

DL is a part of machine learning. In DL, computers train the models to process and learn from raw data, and that is possible by using the deep neural networks (DNNs) model. The structure of DNNs is inspired by the human brain, which is made up of multiple layers. The first layer is the input layer, whereas the last layer is the output layer, and the layers in the middle are called hidden layers. These layers consist of many processors called neurons, which are connected to each other. The input layers receive raw data from an environment, e.g., the data from power grid components, which are sent to hidden neurons through connections. The hidden neurons become activated through weighted connections, and the results are produced from the output layer. This process is called a feed-forward neural network. If the results of DNN do not match the correct results, the backpropagation algorithm is used to update these weights optimally. The loss function is the difference between the true value and the predicted value that is obtained from DNN. The DNN uses the backpropagation algorithm, e.g., gradient descent, to reduce the difference between true and predicted values. A DL method is suitable for working in high-dimensional environments [53].

RL is also a subset of machine learning, concerned with how the agent takes a sequence of actions in a dynamic and uncertain environment in order to increase the cumulative reward. RL has a number of base elements, including agents, environments, states, actions, and rewards. An agent takes some actions in an environment to maximise the rewards. An action is the group of potential moves that the agent is able to make at each state. An environment is a place where the agent can take action. A state is a situation where the agent locates itself. RL can be formulated as a Markov decision process (MDP) that consists of state space, action space, reward function, transition probability function, and discount factor.

In data-driven RL OPF methods, the agent of RL shows great capabilities to make sequences of decisions in the absence of power grid information. Using reinforcement learning in a power grid decision-making has significant advantages. The agent seeks to make optimal actions for each state by interacting with grid components. RL agents do not require any initial knowledge to make these actions on the grid. Moreover, the RL agent can achieve many objectives through offline training and online implementation. Lastly, the RL is easier to apply in different scenarios in real-time OPF than traditional optimisation approaches. The reason is that a trained RL agent is able to calculate real-time optimisation problems in a grid within several milliseconds [54]. Consequently, the RL is a very efficient tool for solving real-time optimisation problems. However, RL does not work appropriately in continuous state space like OPF. Furthermore, it suffers from dealing with large-dimension data and faces various challenges related to transition function uncertainty and inefficient data usage.

To enhance ML performance, researchers have made efforts to fill the gap by combining RL with DL to create deep reinforcement learning (DRL). As mentioned above, RL has great capabilities to make sequences of decisions in an uncertain environment by learning the optimal action through interactions with a stochastic or deterministic environment. To increase the performance of RL in solving high-dimensional real-time problems, researchers have combined a deep neural network (DNN) with RL, where the DNN works as a function approximator.

3.2. OPF Based on Objective Functions

Each optimisation problem, like OPF, has a dedicated objective function, which needs to be optimised with respect to the target variables of the power system in the presence of constraints imposed on those variables. The aim of this part is to classify the OPF in terms of objective function. Different DRL approaches are applied to find the best OPF solution for the proposed objective function.

3.2.1. Operating Cost Minimisation

OPF supports the network operators, minimising operational costs. Since reducing the electricity cost is considered one of the main goals for the operator of the grid, it has been used widely as an objective function [55].

Due to the high-level penetration of distributed generators (e.g., solar PVs, wind turbines) in distributed networks, controlling these devices becomes very important to minimise the running cost. In ref. [18], a soft actor-critic is proposed for solving the optimal active power dispatch on the IEEE 118-bus. The Lagrange multiplier method is used to improve the performance of the soft actor-critic algorithm in a high-dimensional discrete action environment. The proposed algorithm is more effective in finding active power dispatch points than the proximal policy optimisation and double deep Q-network. To deal with a continuous action space, the authors in [56] introduce a Lagrangian-based DRL to solve the continuous real-time OPF. The objective of this work is to find the least generation dispatching cost while satisfying the security constraints. The critic networks are not used because they induce higher approximation errors. Instead, the deterministic gradient is approximated analytically. The proposed method reached the best solution compared to the supervised learning method. A twin-delayed deep deterministic policy gradient (TD3) algorithm is used in [19] to minimise the summation of production costs by determining the active power of the generators on the IEEE 118-bus system, where a Levenberg–Marquardt method is introduced to the TD3 to mitigate the risk of divergence solutions. The proposed method is able to find a better solution than the deep deterministic policy gradient (DDPG) that is used in [56].

Energy storage (ES), on the other hand, is used widely in electrical grids to store excess power from distributed generators and can be managed optimally to minimise operating costs. In [57], DDPG is proposed to control a battery with lookahead constraints in real time. A safety layer and two replay buffers are introduced to promote the RL agent’s action, where the goal of the agent is to increase revenue by operating the energy storage optimally. The proposed method can reach a cost that is close to the ideal cost while the computational time is reduced multiple times as compared with model predictive control (MPC). In [58], a DRL-based method is proposed to control energy storage and distributed generators in a microgrid to reduce the purchases of power from the main grid. The authors in [59] proposed a bottom-up energy internet architecture to model the integrated multi-microgrid to minimise the overall cost by the optimal control of the energy storage and distributed generators. The DRL method is utilised to manage the power sources in the bottom layer and dispatches the decision to the up layer, which is connected to the main grid. The simulation results show that the proposed method outperforms MPC in minimising the running cost.

Due to the increasing number of electric vehicles (EVs) that are able to work as a load or a power source, the authors in [60] developed a control strategy to minimise the power cost in a microgrid by considering stochasticity associated with electricity price and renewable resources. TD3 algorithm is utilised to control the distributed generators and electric vehicles, and simulation results show that the proposed control strategy outperforms the traditional particle swarm optimisation (PSO) method. To deal with the unknown transition probability of a distribution network equipped with large-scale electric vehicle charging and distributed generators, the nodal multi-target policy is proposed in [61] to schedule the optimal electric vehicle charging while a soft actor-critic algorithm is used to determine the target levels for the policy. The proposed approach achieves lower system costs than the proximal policy optimisation (PPO) method.

Flexible loads are considered one of the most efficient ways to minimise operating costs. To study the feasibility of using flexible loads, the authors in [62] proposed graph reinforcement learning to manage an electrical network that contains both energy storage and flexible loads. The proposed method is implemented based on a graph attention network to extract the topology structure information from the electrical grid and send this information to DDPG to find the optimal formulation in order to manage the controllable assets. The proposed method is carried out within an IEEE 123-bus system, and the simulation results show the ability of the method to find the optimal operational status compared to PSO. To exploit the interruptible loads at the demand side, the authors in [63] used the duelling deep Q network (DQN) algorithm to minimise the daily load cost.

When faults occur in the distributed networks, the grid operators often try to disconnect a number of buses to isolate the affected transmission lines, attempting to ensure the grid works continuously without considering the operational cost. The authors in [64] proposed a method to minimise the running cost even when the faults occur by optimal controlling the topology and distributed generators. Three-stage reinforcement learning is presented to manage an IEEE 33-bus system, and the simulation results show the capability of this approach to reduce operating costs even when one of the transmission lines is disconnected. In [65], a batch-constrained soft actor-critic algorithm is developed to minimise the operational cost by finding the optimal configuration for a power grid under unforeseen states. The test results show that the proposed method is better than DQN and SAC in terms of decreasing the system running costs.

Minimising power loss is deemed one of the techniques to reduce the overall operating cost by controlling the active and reactive power of the controllable component in the electric grid. In [55], the OPF is modelled as a stochastic nonlinear programming problem, and the proximal policy optimisation (PPO) is proposed to find the best solution for the optimisation problem by modifying the active and reactive power of the energy storage. The DRL-based approach reaches the least operational cost for IEEE 33-bus compared to stochastic programming. In [66], TD3 is presented to optimally control the community microgrid networks with integrated solar PVs, wind turbines, and energy storage. The DRL agent is able to manage the active and reactive power of the grid to minimise the total power loss. The related work for minimising the operating cost is summarised in

Table 3. OPF through minimising the operating cost.

Ref.	Optimisation Method	Objective Function	Application	Action	Power System Size
[18]	Lagrange multiplier SAC	Maximise operational rewards, minimise generation costs, maintain system constraints (power balance, voltage limits)	Optimal active power dispatch for DGs in systems with renewable energy	Discrete actions	IEEE 118-bus power system
[56]	Lagrangian-based DDPG	Minimise total generation cost	Optimising real-time power flow in systems with intermittent distributed renewable generators	Continuous action	IEEE 118-bus power system
[19]	TD3 and Levenberg–Marquardt	Minimise total generation cost	Real-time optimal power flow management, controlling DG outputs	Continuous action	IEEE 118-bus power system
[57]	DDPG with safety layer and dual replay buffer	Minimise expected time average cost over control horizon, accounting for monetary costs at PCC and dispatch plan deviation cost	Real-time control of ES in active distribution grids with distributed energy resources	Continuous action	34-bus Swiss grid
[58]	DQN	Minimise operational costs of microgrid	Management of conventional generators, DGs, ESs, and grid interactions	Discrete actions	Microgrid (size not specified)
[59]	A2C with Curriculum Learning	Minimise overall operational costs of microgrids, considering generation costs, storage management, and penalties for operational constraint violations.	Energy management of controllable DGs in a network of microgrids	Continuous action	Network of interconnected microgrids (size not specified)
[60]	TD3	Minimise operating costs, including generation, transaction, and EV charging costs, with incentives for renewable energy use	Intelligent energy management in a hybrid DG and EV system	Continuous action	Microgrid (size not specified)
[61]	SAC with nodal multi-target (NMT) approach	Minimise energy costs and penalties related to EV charging non-completion	EV charging scheduling in a power distribution network	Continuous action	IEEE 37-node test feeder with 2500 EV stations
[62]	Graph RL with Graph Attention Networks (GAT) and DDPG	Minimise costs, including network transactions, power losses, load control, and voltage deviations	Real-time optimal scheduling of FLs, DGs, ES systems, and SVCs in active distribution networks	Continuous action	Modified IEEE 33-bus system
[63]	DDQN	Maximise long-term profit by managing interruptible loads to reduce peak demand and operation costs while maintaining voltage limits	Demand response management of interruptible load (FL) in power distribution networks	Discrete actions	Enhanced IEEE 33-node test feeder system
[64]	Three-stage DDQN and DDPG	Minimise operational costs while ensuring network stability and reliability, reducing power losses, managing load demands, and optimising distributed energy resources (DERs)	Real-time operation of distribution networks, controlling DGs, load points, switches, and ESSs	Discrete and continuous actions	IEEE 33-bus
[65]	Batch-constrained soft actor-critic (BCSAC)	Minimise overall operational costs, including electricity consumption, line losses, and switching operations	Dynamic distribution network reconfiguration, controlling remotely operable switches	Discrete actions	119-bus distribution network
[20]	PPO	Minimise the cost of power loss across the distribution network with constraints related to renewable energy and storage device operations	Optimal power flow in networks with DGs and ESs	Continuous actions	Modified IEEE 33-bus network with added renewables and storage
[66]	TD3	Minimise total power loss within a community microgrid	Power flow optimisation in community microgrids, controlling DERs and ESS	Continuous actions	IEEE 14-bus test system
[67]	Improved DRL	Minimise operational costs in day-ahead dispatch, including costs from power purchases, network losses, and curtailment of renewable energy	Day-ahead optimal dispatch in active distribution networks, controlling DGs, ESS, and CBs	Discrete and continuous actions	IEEE 33-bus test system
[68]	Deep LSTM-based DQN	Minimise overall daily operational cost of the grid-tied microgrid, optimising power flow from BESS and managing grid interactions to reduce costs and maximise revenue	Economic energy dispatch of ES systems in a grid-tied microgrid	Discrete actions	Residential microgrid (specific size not provided
[69]	Multi-agent deep deterministic policy gradient (MADDPG)	Minimise total generation and interaction costs, balancing production costs and revenue/costs associated with power flow between microgrids and the main grid	Economic dispatch of ES and DG in active distribution networks with multiple microgrids	Continuous actions	Multiple microgrids, unspecified the size
[70]	Double DQN	Minimise reactive power-related losses and voltage deviations, expressed as a weighted sum of costs associated with line losses and voltage deviations	Reactive power optimisation in distribution networks, controlling reactive power compensators	Discrete actions	IEEE 37-bus test system
[71]	DQN	Minimise annual operational costs, including energy losses and the operation of dispersed generation units	Energy management in distribution networks with EVs and DGs	Discrete actions	57-bus IEEE grid

.

3.2.2. Voltage Deviation Minimisation

An increased number of distributed generators in an electrical grid may lead to a disturbance in the voltage of the grid. The high penetration of these resources could cause unforeseen fluctuations in the voltage profile due to their stochasticity nature [72]. Ineffective control of the grid voltage affects the power flow dispatch in the distribution networks; therefore, transmission line losses and electrical prices will eventually increase [73].

One of the techniques to improve the voltage quality is to control the distributed generators in an effective way. Optimal reactive power control of the distributed generators is used widely to decrease the fluctuation of the voltage. The authors in [74] used DDPG to control the reactive power of the PV inverters in a low-voltage network. Their simulation results show that the proposed method is able to keep the voltage fluctuation within the desired limits. The MADDPG algorithm and the attention model are used in [75] to enhance the voltage control strategy in the IEEE-123-bus system, where the results demonstrated that the proposed approach can achieve a better control performance as compared with a standard MADDPG algorithm. The authors in [76] proposed a two-stage control scheme to manage DG inverters in the IEEE 123-bus system. In the first stage, which is called an offline stage, a jointly adversarial soft actor-critic algorithm is used to make the inverter agents more robust to reach an optimal solution. Then, the SAC is used in the second stage (online stage) to control the inverters in real time. The proposed method outperforms the state-of-art DRL algorithm. Instead of using smart inverters to control the voltage profile, the authors in [77] proposed the PPO and imitation learning method to find the optimal set-points for 38 conventional generators in Illinois 200-bus systems to ensure the voltage within the acceptable range. The proposed method was able to solve the OPF problem much faster than the interior-point method.

Energy storage technologies have experienced huge development recently; as a result, they have become another feasible solution for reducing voltage fluctuation. Energy storage can play an important role in distribution networks to participate in minimising power fluctuations caused by distributed generators [78]. DQN is proposed in [79] to mitigate voltage fluctuations by controlling a single battery. The results showed that a battery is able to reduce voltage violation caused by the stochasticity of the distributed generators. Overvoltage issues are caused by high levels of penetration of distributed generators, and energy storages may not be sufficient to ingest extra power, especially during the light load intervals. Energy storage capacity problems are addressed in [21], where reinforcement learning is combined with MPC to prevent voltage violations under high generating conditions. Electric vehicles can be considered as mobile energy storage, which can play a significant role in supporting grid voltage. The average weighted deep double Q-network (DDQN) algorithm is introduced in [80] to work as a voltage controller for EVs. The proposed method outperformed DDQN and DQN in terms of keeping the voltage within safe limits. A multi-agent DQN approach is used in [81] to control EVs and ESs in a low-voltage grid. The distributed generators are integrated with energy storage units to mitigate the risk of voltage fluctuation. The authors of [82] used DDPG to find the optimal schedule of PV and energy storage inverters in an IEEE-34 bus system, and they achieved a better performance as compared to DQN in minimising voltage fluctuations.

Another approach to regulating the voltage of the grid is to use capacitor banks, which are essentially one type of reactive power compensation device. DQN algorithm is proposed to control two capacitor banks in a microgrid [83]. Capacitor banks are categorised as slow-timescale devices based on response speed. A two-timescale voltage management plan is, therefore, developed in [84] to minimise voltage deviations. DQN algorithm is utilised to optimise the setpoints of PV inverters on a fast timescale to reduce the instantaneous voltage violations. Capacitor banks can also be configured by the proposed algorithm to control long-term voltage deviations. The second type of reactive power compensation device is associated with the on-load tap changers, which can be used to regulate the voltage. The DDPG algorithm is used in [85] to learn an optimal setting of on-load tap changers in terms of mitigation of the voltage sags. A constrained soft actor-critic algorithm is presented in [86] to find an optimal configuration of on-load tap changers and capacitor banks. The simulation results show that the proposed algorithm achieves better performance than the state-of-the-art RL algorithms and the conventional optimisation-based algorithms. A static VAR compensator (SVC) is a compensation device used to provide fast-acting reactive power in distribution systems. The soft actor-critic algorithm is introduced in [87] to enhance the ability of the grid to accommodate the high fluctuation of the voltage caused by DGs. The proposed algorithm appears to be the best for controlling the reactive power of PV inverters and SVCs to mitigate the risk of voltage violations compared to the PSO algorithm. A multi-agent soft actor-critic algorithm is used to achieve decentralised control of SVCs and energy storage units for voltage regulation in the distribution system [88]. The sparse pseudo-Gaussian process is integrated with the proposed algorithm to learn the relationship between the power injections and voltage magnitude of each bus. The results show that the multi-agent soft actor-critic (MASAC) outperformed the single-agent SAC and the traditional optimisation-based algorithms.

Reconfiguration of the distribution network plays a significant role in increasing the voltage quality of the grid by finding the optimal configuration of switching devices over a particular time period. The DQN algorithm is used as a smart controller to manage the power flow by controlling grid switches to make the voltage fluctuation within acceptable limits [89]. To examine the ability of the network reconfiguration approach to reduce the voltage violation under different loading and generating conditions, the PPO algorithm is proposed in [90] to control nine switches (sectionalism and tie switches) in a microgrid. The experimental results show that the proposed algorithm produces an effective and much faster solution than DQN.

Load shedding is considered one of the most effective and economical approaches to protect the power system against voltage swings. The DDPG algorithm is combined with the convolutional neural networks to learn the optimal load-shedding configuration to maximise voltage stability [91]. The proposed method successfully increased the quality of the voltage by determining the location and amounts of load shedding in the New England 39-bus system. The MASAC approach is also presented in [92] for voltage regulation in a low-voltage network, where the MASAC algorithm uses a decentralised execution framework to control loads in commercial buildings for mitigating voltage swings. The experimental results demonstrate that MASAC outperformed the MADDPG algorithm. The related work for minimising the voltage deviation is summarised in

Table 4. OPF through minimising the voltage deviation.

Ref.	Optimisation Method	Objective Function	Application	Action	Power System Size
[74]	DDPG	Minimise voltage fluctuations	DGs	Continuous action	IEEE 21-bus
[75]	MADDPG	Minimise voltage fluctuations	DGs	Continuous action	123-bus systems
[76]	JASAC	Minimise voltage deviation and active power loss	DGs	Continuous action	IEEE 123-bus
[77]	PPO with IL	Minimise voltage deviation and generator costs	Conventional generators	Continuous action	IEEE 200-bus
[79]	DQN	Minimising the average voltage fluctuation and maximising the SoC of energy storage	ESs	Discrete action	IEEE 33-bus
[21]	RL with MPC	Minimise voltage deviation, active power loss, and overall cost	ESs	Continuous action	IEEE 33-bus
[80]	AWDDQN	Minimise voltage fluctuations	EVs and DGs	Discrete action	IEEE 123-bus system
[81]	MADQN	Increase the proportion of PV power generation used locally and minimise voltage fluctuations	EVs and ESs	Discrete action	Low voltage grid
[82]	DDPG	Voltage regulation and power loss minimisation	DGs and ESs	Continuous action	IEEE-34
[83]	DQN	Minimise voltage deviation	CBs	Discrete action	13-bus
[84]	DQN	Minimising the long and short-term average voltage deviation	DGs and CBs	Discrete action	The Southern California Edison 47-bus
[85]	DDPG	Minimising the voltage swell and power losses	TCs	Continuous action	IEEE 33-bus system
[86]	Constrained SAC	Minimising voltage deviation and generating costs	TCs and CBs	Discrete action	123-bus
[87]	SAC	Minimising voltage deviation and power loss and generating cost	DG inverters and CBs	Continuous action	33-bus
[88]	Multi-agent SAC and sparse pseudo-Gaussian process	Minimising the voltage deviation and PV curtailment	ESs, DGs, and SVCs	Continuous action	IEEE 123-bus
[93]	Multi-agent RL algorithms	Minimising voltage deviation and active power loss	Conventional generators and capacitor bank	Discrete action	IEEE 162-bus
[94]	DDPG	Minimising voltage deviation and active power loss	TCs	Continuous action	IEEE 123-bus
[95]	DQN	Minimising the voltage deviation	Conventional generators, CBs, and TCs	Discrete action	IEEE 14-bus
[96]	Multi-agent SAC	Minimising the voltage deviation	DGs, CBs, and TCs	Continuous action	IEEE 123-bus
[97]	DDPG and Monte Carlo	Minimising voltage deviation and power loss	EVs, CBs, and TCs	Continuous action	IEEE 123-bus
[89]	DQN	Minimising voltage deviation, power loss and switch action cost	Topology	Discrete action	Taiwan power company 84-bus
[90]	PPO	Minimising voltage deviation and power loss	Topology	Discrete action	IEEE 34-bus
[91]	DDPG with CNN	Minimising voltage deviation	Load shedding (FL)	Continuous action	New England 39-bus system
[92]	Multi-agent SAC	Minimising voltage deviation, energy cost and indoor thermal discomfort	Load shedding (FL)	Continuous action	Low-voltage network 6-bus
[98]	Convolutional LSTM with DQN	Minimising short-term voltage deviation	Load shedding (FL)	Discrete action	China Southern Power Grid 23-bus

.

3.2.3. Emission Cost Minimisation

Climate change and global warming are considered to be among the main challenges facing our world presently. Traditional generators and vehicles produce almost 60% of greenhouse gases [98]. The reason behind using fossil fuel-based generators is their low prices and reliability in contrast to DGs. Greenhouse gas emissions must be reduced to save our planet. Due to the increasing environmental awareness, the DGs have grown unprecedentedly in distribution systems; however, this growth creates significant problems for the grid. Power curtailment of DGs is necessary to minimise the voltage rise and congestion.

Real-time OPF can play an important role in minimising the curtailment of renewable energy and maximising the quality of the voltage and grid capacity. Management of the DG outputs is one approach to increase the penetration of renewable energy sources.

The authors in [99] proposed a duelling DQN algorithm to control DGs in an IEEE 14-bus system fed with 40% renewable energy sources. The experimental results showed that duelling DQN is able to increase the capacity of the grid to accommodate higher renewable energy rates and maintain the stability of the system. A two-timescale control framework is presented in [100] to manage a grid with high PV penetration (120% of the feeder capacity). In a slow timescale control, a model-based approach is used to organise the voltage. On the other hand, the DDPG algorithm is proposed to control the setpoints of PV inverters in a fast timescale. The results showed that the proposed framework achieved lower voltage deviations and PV curtailment in contrast to a traditional optimisation method based on Volt–VAR control. A soft actor-critic-based multi-agent DRL algorithm is proposed in [101] to control the active and reactive PV inverter for the Colorado U.S. grid with 80% penetration of renewable power. The proposed method succeeded in managing 77 PVs in the 759-bus system and minimised PV curtailment while keeping the grid voltage within acceptable limits as compared with the traditional Volt–VAR control method.

Energy storage units and EVs are effective methods to decrease the curtailment of DGs by storing excess power, especially when the demand is low. A microgrid with hydrogen storage units is introduced in [102], and the DDPG algorithm is used as a control agent to reduce the curtailment of PV generation. Simulation results showed that the DRL agent reduced the operation and emission cost by 5% when compared with the genetic algorithm. A vehicle-to-grid framework is developed in [103] to utilise EV features to support the grid. This framework helps the EVs to work in a cooperative way to achieve a number of goals, e.g., minimising the operational and emission costs. A hybrid multi-agent PPO algorithm is used to determine the routing and scheduling of the EVs inside the grid. Moreover, the parameter-sharing method is integrated with DRL to stabilise the training performance. The results showed that the proposed framework is able to reduce the travelling time of the EVs, as well as energy and emission costs.

The optimal configuration of the status of switches in a distribution network can play a vital role in raising the hosting capacity of transmission lines, and the curtailment of the excess renewable energy is, therefore, minimised. The DQN algorithm is used in [104] to find the best network configuration for a 16-bus distribution system. Simulation results showed that the proposed policy is able to minimise the operating cost and the curtailment power of DGs while the voltage profile is improved. Optimal management of reactive power devices can also be used to increase the hosting capacity of DGs. The multi-agent DRL algorithm is investigated in [105] to control the bus voltages by specifying the setpoint for the SVCs. The MADDPG agents succeeded in decreasing the system loss and improving the hosting capacity of the grid as compared with a conventional model-based method. The related work for minimising the emission cost is summarised in

Table 5. OPF through minimising the emission cost.

Ref.	Optimisation Method	Objective Function	Application	Action	Power System Size
[100]	Duelling DQN	Minimising operating costs and curtailment of RE	DGs	Discrete actions	IEEE 14-bus
[101]	DDPG and model-based approach	Minimising line losses, voltage deviations, and curtailment of RE	DGs	Continuous action	IEEE 34-bus
[102]	MASAC	Minimising voltage deviations and curtailment of RE	DGs	Continuous action	Colorado U.S. grid 759-bus
[103]	DDPG	Minimising operating and emission cost	DGs and ES	Continuous action	Microgrid
[104]	MAPPO with parameter-sharing	Minimising operating and emission cost	EVs	Continuous + discrete action	15-bus radial distribution
[105]	DQN	Minimising line losses, voltage deviations, and curtailment of RE	Reconfiguration (Topology)	Discrete action	16-bus
[106]	DQN with multi-objective bacterial foraging optimisation	Minimising PV power curtailment, power loss, and generation cost	Reconfiguration (Topology)	Discrete action	IEEE 118-bus
[107]	MADDPG	Minimising line losses, voltage deviations, and curtailment of RE	SVC	Continuous action	IEEE 300-bus and China 157-node

.

3.2.4. Increasing System Reliability

Solving real-time OPF problems is an effective method to increase the robustness and reliability of the power system to withstand any type of contingencies without violating system constraints. The authors in [108] utilised the PPO algorithm to find the optimal generator setpoints in a 200-bus system. One random transmission line outage is included to assess the performance of the proposed algorithm under a contingency state. The results demonstrated that the PPO algorithm is able to deal with topology changes and find near-optimal OPF solutions. Optimal management of PV inverters can also be used to increase the robustness of the system. The PPO algorithm is investigated in [109] to mitigate the voltage unbalance at the point of common coupling by controlling the Volt–VAR of the PV inverters.

Energy storage units and EVs can play essential roles in supporting system reliability. The DDPG algorithm is presented in [110] to reduce power fluctuations caused by large wind fluctuations. The proposed method is able to efficiently manage the energy storage units to minimise wind fluctuation as compared with the DQN algorithm. Energy storage units can be used as an efficient approach for peak load shifting. The DDPG algorithm is used in [111] to deal optimally with the uncertainty of load demand at peak time by controlling the storage units. Q-learning algorithm has been used in grid-to-vehicle and vehicle-to-grid services to increase the efficiency of the grid by minimising the peak load [112]. The authors in [113] used energy storage units controlled by the SAC algorithm to reduce the voltage violations in an unbalanced low-voltage grid.

Short-term voltage instability is a fast event that usually takes seconds, where fast actions are required to return the voltage to the normal range. Load shedding is one of the effective emergency methods to deal with voltage instability, especially short-term events. A parallel augment random search (PARS) algorithm is adopted in [114] to mitigate short-term voltage by shedding 20% of the total load. The DRL algorithm is integrated with LSTM to support the learning rate. The proposed algorithm outperformed the MPC approach in terms of computational efficiency and robustness in learning. The load-shedding method is often used in an emergency state. The DDPG algorithm is proposed in [115] to deal with line faults by using the load-shedding method, where the DRL approach is used to choose which bus participates in the shedding process and the amount of load shedding. The shedding must be less than 40% of the original load power. The results showed that the proposed approach is able to return the voltage of the bus to the normal value after the emergency event.

Topology reconfiguration is one of the best approaches for the grid operator to increase the stability of the network. Topology reconfiguration is considered the most economical solution for distribution violations when compared with other approaches like load shedding, peak shaving, and transmission line expansion. The actor-critic (A3C) algorithm is combined with domain knowledge of power system operators to prevent cascading line outages by using topology reconfiguration [116]. Due to the high generation of DGs in a distribution system, an online reconfiguration scheme is proposed in [117] to alleviate line congestion and voltage violations. The DQN algorithm is used as a controller to find optimal distribution topologies. The performance of the DRL algorithm outperformed the genetic algorithm and Brute-force Search. The related work for minimising the system instability is summarised in

Table 6. OPF through minimising the system instability.

Ref.	Optimisation Method	Objective Function	Application	Action	Power System Size
[108]	PPO	Maximise power system security	Conventional generators	Continuous action	200-bus
[109]	PPO	Minimise voltage unbalance at the PCC	DGs	Continuous action	IEEE 34 bus
[110]	DDPG	Minimising the power fluctuations and power cost	ESs	Continuous action	IEEE 14 bus
[111]	DDPG	Minimising the operating cost and net load fluctuations	ESs	Continuous action	11 bus
[118]	Safety-constrained SAC	Peak shaving and voltage regulation	DG inverters and ESs	Continuous action	IEEE 123-bus
[112]	Q-learning algorithm and enhanced Grasshopper optimisation	Peak shaving and minimising the power loss	DG and EVs	Discrete action	IEEE 33-bus
[113]	SAC	Minimising the total daily cost and voltage regulation in unbalanced grid	ESs	Continuous action	IEEE 34-bus
[114]	PARS with LSTM	Minimising the total load-shedding amount and the voltage violations	Load shedding (FL)	Continuous action	IEEE 300-bus
[115]	DDPG	Minimising voltage violations	Load shedding (FL)	Continuous action	IEEE 39-bus
[116]	A3C	Minimising the total line loading	Topology	Discrete action	IEEE 14-bus
[117]	DQN	Mitigating line congestion and voltage violations	Topology	Discrete action	IEEE 123-bus
[119]	Q-learning	Minimising the load shedding cost and frequency instability	DGs, ES, and demand response (FL)	Discrete action	IEEE 37-node MG
[120]	Clipped PPO	Maximising long-term voltage stability	ES and demand response (FL)	Continuous action	Nordic 32-bus
[121]	Q-learning with CNN	Maximising frequency stability after the fault	Load shedding (FL)	Discrete action	IEEE 39-bus

.

Abbreviations used in the tables are as follows: soft actor-critic: SAC; deep deterministic policy gradient: DDPG; twin delayed DDPG: TD3; duelling deep Q network: DDQN; proximal policy optimisation: PPO; multi-agent deep reinforcement learning: MADRL; distributed generators: DGs; energy storage: ES; electric vehicle: EV; flexible load: FL; static var compensation: SVC; microgrid: MG; jointly adversarial soft actor-critic algorithm: JASAC; imitation learning method: IL; weighted deep double Q-network: AWDDQN; capacitor bank: CB; tap-changer, TC.

3.3. Discussion and Summary

Figure 5 below reveals significant insights into the integration of DRL with OPF in terms of categories of objective function, application, optimisation method, and power system size. The use of objective functions emphasises a predominant focus on minimising costs (56%), followed by managing voltage fluctuations (26%) and emissions (11%), which underscore the economic, operational, and environmental imperatives in current research. Applications are largely dominated by DGs at 36%, highlighting a shift towards decentralised power generation models, while ES systems, EVs, and VRD (voltage regulation devices) at 20%, 5%, and 16%, respectively, appear as emerging roles in dynamic grid management. The utilisation of DQN and DDPG methods at 28% and 22% indicates a robust exploration of DRL techniques suitable for the complex and high-dimensional state spaces typical in power systems. This evolving integration from simple cost minimisation to complex objectives like volatility and emissions management reflects a maturing field where future work must address scalability and real-world applicability, ensuring advancements in ML to propel the transition toward more sustainable and resilient power systems while aligning with evolving regulatory frameworks to maximise benefits and mitigate associated risks.

Our reviews explore the significant role of ML in enhancing OPF for electrical networks that integrate renewable energy sources. OPF is crucial for determining the optimal operational points of distributed generators, with objectives such as minimising operational costs or reducing power loss. These operational points must adhere to stringent constraints, particularly maintaining network voltage within specified limits to ensure stability and efficiency. Exceeding these voltage thresholds renders the operational points unacceptable, highlighting the importance of precise control mechanisms. Through our review, we found that all studied approaches not only aim to optimise cost functions but also prioritise maintaining network voltage within acceptable limits. The integration of ML techniques has shown promising results in managing these complexities more effectively, providing real-time solutions that adapt to the variability inherent in renewable energy sources. Our reviews underscore the transformative potential of machine learning in making energy systems more efficient and sustainable.

The paper delves into the transformative role of ML in modernising OPF systems, highlighting a significant shift from traditional deterministic models to more dynamic, adaptive models equipped to handle the complexities introduced by the increasing use of renewable energy sources. This evolution fosters enhanced real-time decision-making and increases the resilience of power systems through the adoption of sophisticated ML techniques like DL and RL. These technologies not only adeptly manage the variability and unpredictability inherent in renewable sources such as wind and solar but also transform grid operations into intelligent, proactive management frameworks. This shift to a more anticipatory strategy improves the ability to forecast and react to changes in power flow, optimising both stability and efficiency. The integration of ML into OPF presents notable challenges, including high computational requirements and the critical dependence on the quality and availability of data. Furthermore, effective deployment of these advanced technologies requires supportive regulatory frameworks that facilitate innovation while ensuring alignment with broader objectives such as sustainability and public safety. As the paper indicates, embracing these challenges and opportunities is essential for developing power systems that are not only more efficient but also robust and adaptable to the evolving demands of energy management.

4. Conclusions

This paper reviews two vital aspects of renewable integration by exploring possibilities for advanced solutions from the generating device and power system operation perspectives. The review covers recent developments in thermal condition monitoring to examine how the capacity of existing renewable energy generators such as wind turbines can be expanded at low cost and in power flow optimisation with machine learning to examine how a low-carbon renewable dominated power network can be achieved. Utilisation of the thermal design margins for power equipment has the potential to expand renewable energy generation capacity. This is particularly true for wind power generation, where the uprating of existing wind turbines could increase the available renewable energy output. In situ fibre optic thermal sensing was identified as the technique that can facilitate the required thermal monitoring capability for WT generators; retrofitting fibre optic sensors to in-service machinery in situ may be challenging in practice, and the application of effective thermal estimators, where available, would also be of interest in these cases. Furthermore, the development of dynamic and adaptive optimal power flow models can lead to a more informed and proactive management strategy for the power grids while optimising resource utilisation and minimising operational risks. With machine learning, the power grid can redefine its management boundaries and realise a platform for intelligent decision-making. System-level automation of this process by combining thermal condition monitoring with optimal power flow is highly desirable, yet it remains a challenge for control and management of renewable energy integration into the grid. The advanced sensor/sensing systems and machine learning approaches reviewed in this paper hold the potential to provide a viable and efficient solution to improve power capacity exploitation and maintain network stability in an economical and environmentally affordable way. However, considerable further research is needed to achieve this goal.