Prospects and Challenges of the Machine Learning and Data-Driven Methods for the Predictive Analysis of Power Systems: A Review

Abstract

The use of machine learning and data-driven methods for predictive analysis of power systems offers the potential to accurately predict and manage the behavior of these systems by utilizing large volumes of data generated from various sources. These methods have gained significant attention in recent years due to their ability to handle large amounts of data and to make accurate predictions. The importance of these methods gained particular momentum with the recent transformation that the traditional power system underwent as they are morphing into the smart power grids of the future. The transition towards the smart grids that embed the high-renewables electricity systems is challenging, as the generation of electricity from renewable sources is intermittent and fluctuates with weather conditions. This transition is facilitated by the Internet of Energy (IoE) that refers to the integration of advanced digital technologies such as the Internet of Things (IoT), blockchain, and artificial intelligence (AI) into the electricity systems. It has been further enhanced by the digitalization caused by the COVID-19 pandemic that also affected the energy and power sector. Our review paper explores the prospects and challenges of using machine learning and data-driven methods in power systems and provides an overview of the ways in which the predictive analysis for constructing these systems can be applied in order to make them more efficient. The paper begins with the description of the power system and the role of the predictive analysis in power system operations. Next, the paper discusses the use of machine learning and data-driven methods for predictive analysis in power systems, including their benefits and limitations. In addition, the paper reviews the existing literature on this topic and highlights the various methods that have been used for predictive analysis of power systems. Furthermore, it identifies the challenges and opportunities associated with using these methods in power systems. The challenges of using these methods, such as data quality and availability, are also discussed. Finally, the review concludes with a discussion of recommendations for further research on the application of machine learning and data-driven methods for the predictive analysis in the future smart grid-driven power systems powered by the IoE.

1. Introduction

The increasing complexity of modern power systems, along with the growing penetration of renewable energy sources, has led to a significant increase in the use of machine learning and data-driven methods for predictive analysis of power systems in recent years. They offer new insights into system behavior, help to enhance forecasting accuracy, and provide real-time decision support, thereby increasing the efficiency and reliability of power systems [1,2].

One of the key reasons for the increasing use of machine learning and data-driven methods in predictive analysis of power systems is their ability to handle large volumes of data. With the proliferation of sensors and data acquisition devices in power systems, vast amounts of data are generated every second. Machine learning algorithms can process these data in real time, allowing for the identification of patterns and trends that would be impossible to detect using traditional methods. This can help utilities and system operators to optimize system performance, identify potential problems before they occur, and make data-driven decisions that improve the efficiency and reliability of power systems [3].

Another key advantage of machine learning and data-driven methods in predictive analysis of power systems is their ability to learn from experience. By analyzing historical data, machine learning algorithms can learn to recognize patterns and correlations and develop models that accurately predict future system behavior. This can help system operators to make informed decisions about system operation, maintenance, and planning, leading to better overall system performance and increased reliability [4].

Moreover, machine learning and data-driven methods offer a level of flexibility and adaptability that is unmatched by traditional methods. As power systems continue to evolve and become more complex, machine learning algorithms can be retrained to adapt to changing conditions, and new data sources can be easily integrated into existing models. This can help system operators to keep pace with changes in the system and respond quickly and effectively to new challenges [5].

Despite these benefits, there are also challenges associated with the use of machine learning and data-driven methods for predictive analysis of power systems. These include the need for large amounts of high-quality data, the difficulty of interpreting machine learning models, and the potential for model overfitting or bias. To address these challenges, researchers are exploring new techniques for data acquisition and processing, developing more interpretable machine learning models, and improving methods for model validation and testing.

1.1. Machine Learning and Data-Driven Methods in Power Systems

In general terms, power systems represent the complex networks that consist of generators, transformers, transmission lines, as well as distribution networks. These systems are designed to deliver electricity to consumers with high reliability and efficiency [6,7]. Predictive analysis plays a critical role in ensuring the reliable and efficient operation of power systems. It involves the use of historical data and statistical models to predict potential issues, such as equipment failures, and plan for optimal operation [8]. With the increasing demand for reliable, renewable and sustainable energy, the use of machine learning and data-driven methods for predictive analysis of power systems has become essential. Machine learning and data-driven methods have the potential to improve the accuracy and reliability of power system operations by utilizing large volumes of data generated from various sources [9,10].

Therefore, in the past few years, machine learning and data-driven methods have become increasingly popular for the predictive analysis in power systems due to their ability to handle large amounts of data and make accurate predictions. These methods involve the use of algorithms for identifying the patterns in data and making predictions based on those patterns [11,12]. The benefits of using these methods for predictive analysis in power systems include improved accuracy, reduced cost, and increased efficiency [13]. However, there are also some limitations to using them, such as the need for high-quality data and the risk of overfitting [14]. In general, the challenges to using machine learning and data-driven methods for predictive analysis in power systems are also quite significant. One of the main challenges is data quality and availability. Power system data can be complex, and there is often a lack of standardization in the data collected from different sources. This can make it challenging to train accurate machine learning models [15]. Another challenge is the need for domain expertise to ensure that the models are appropriately designed and validated [16]. Finally, there is a risk of overfitting when using machine learning and data-driven methods, which can result in models that are too complex and not generalizable to new data [17].

The use of machine learning and data-driven methods for the predictive analysis of power systems has been extensively studied in research literature. Various methods have been proposed for predictive analysis of power systems, including artificial neural networks (ANNs), support vector machines (SVMs), decision trees, and random forests. These methods have been applied to a range of power system applications, such as load forecasting, fault detection, or condition monitoring [18,19,20].

The problems and challenges include the need for high-quality and standardized data, the requirement for domain expertise to design and validate models, and the risk of overfitting. Power system data can be complex and lack standardization, making it difficult to train accurate machine learning models. Additionally, overfitting can result in models that are too complex and not generalizable to new data. Therefore, addressing these challenges is crucial to ensure the successful implementation of machine learning and data-driven methods in power systems.

1.2. Smart Grids

In general terms, smart grids are intelligent electric power systems that use advanced technologies such as sensors, smart meters, communication networks, and computers to optimize the energy use [21,22]. This type of grid allows for better control over the distribution of electricity, increasing the efficiency of the electrical system and providing more reliable service [23,24]. Smart grids can also increase national security by providing improved monitoring and protection from cyber-attacks [25]. The increased awareness of electricity use along with new technologies provides a great opportunity for cities on their recently increasingly popular transitions towards smart grids [26]. Smart grids enable more accurate control over power distribution networks by allowing customers to make better decisions about their electricity consumption. This leads to reduced energy costs and increased reliability in the electricity supply network [27]. In addition, smart grids are becoming a major part of the transition towards flexible and intelligent systems for electricity production, transmission, and distribution. Their deployment allows organizations to generate sustainable electricity more efficiently by using advanced communication technologies. This can help organizations better manage the electrical grid, reduce power outages, and respond quickly to changes in electric demand [28,29].

The smart grid architecture incorporates innovative technologies, such as advanced sensors and control systems, into the electricity network, enabling bidirectional communication and power flows between different components of the system. These technologies enable the integration of renewable energy resources, such as solar and wind power, into the grid. This integration results in an energy system that is more reliable, efficient, and sustainable than the conventional power systems [30,31].

All in all, the benefits of smart grid systems are vast and have been widely discussed in the literature. For example, smart grids can optimize energy consumption, reduce greenhouse gas emissions, increase the reliability of the energy system, and provide consumers with more control over their energy usage [32,33]. Furthermore, the integration of distributed energy resources, including renewable energy sources and electric vehicles, into the smart grid system enables the creation of a more resilient energy system that is less susceptible to blackouts and other disruptions [34].

Smart grids have the potential to revolutionize the way we generate, distribute, and consume electricity. However, the deployment of smart grids also brings challenges and obstacles such as the integration of various renewable energy sources and their intermittent nature, which requires a significant amount of forecasting and optimization to ensure reliable and efficient operation. Another issue is ensuring the cybersecurity of the grid, as the increasing use of digital communication and control systems can make the grid vulnerable to cybercriminals. Last but not least, the deployment of smart grids requires significant investments and regulatory changes.

1.3. Internet of Energy

In general terms, the Internet of Energy (IoE) is a network of intelligent technologies that together allow for more efficient control of the electrical system [35,36]. This includes the introduction of sensors to monitor and control energy distribution networks, as well as optimization load management and adaptive ways to distribute green energy from renewable energies [37]. All these elements work together in order to increase energy efficiency, perform energy optimization and control sensors in an effective way. The goal is to improve the electric power sector by providing an intelligent monitoring and management system that transmits electricity in a faster, smarter, and more adaptive way [38]. As a subset of the wider concept of the Internet of Things (IoT), IoE uses communication technologies, smart grids, and the integration of the renewable energy sources to enhance efficiency and reliability of electricity transmission and distribution networks [39]. In other words, it represents a technology-driven market which boosts businesses with applications that help manage electricity use, optimize energy production and consumption, increase safety, as well as to reduce costs. This is achieved by improving the stability of electric infrastructures as well as driving operational efficiency through the integration of renewable energy sources [40]. The IoE also helps to maximize energy potential while experiencing high demands from industries such as ICTs and national security. In addition, IoE helps businesses to boost their productivity by enabling them to purchase electricity on more cost-efficient terms and also helps to advance our green energy goals by providing safe access to renewable resources. This reduces the demand for fossil fuels, which leads to the decline in greenhouse gases emissions [41].

By using IoE technology, waste can be reduced, and efficiency can be increased in the existing energy infrastructure. This might result in significant cost savings for electricity consumers and help to preserve the power systems. The use of IoE could also include the utilization of renewable energy sources, the improvement of generation and transmission use, as well as the storage of transmission for later use [42]. Furthermore, IoE technology can also be used to allow energy producers and electricity infrastructures to generate electricity more efficiently, to reduce the costs of power transmission systems, and to better utilize the energy produced from farms and energy meters. Through the IoE technologies, many of these renewable sources such as wind farms and other distant power plants can draw their electricity from their own lines or generators, reducing the energy losses that are caused when large, centralized power plants are used [43]. Additionally, this infrastructure includes everything from the storage centers for generated electricity to the transmission lines that carry it to its destination.

The main challenges facing IOE include the integration of various energy sources and the management of their variability and unpredictability as well as the interoperability of various devices and systems in the IoE ecosystem, which requires standardization and harmonization. Additionally, the deployment of IoE raises privacy and security concerns, as the increasing use of data and communication networks can make the energy system vulnerable to cyber-attacks and data breaches.

1.4. Overview and Structure of the Review

Overall, the use of machine learning and data-driven methods for predictive analysis of power systems has gained significant attention in recent years due to their ability to handle large volumes of data, learn from experience, and provide flexibility and adaptability. While there are challenges associated with these methods, ongoing research is exploring new techniques for addressing these challenges, and their use is expected to continue to grow in the future, leading to increased efficiency, reliability, and sustainability of power systems.

This review paper explores the prospects and challenges of using machine learning and data-driven methods for the predictive analysis in both traditional and modern power systems using a comprehensive and structured overview of the research literature on the subject. The paper consists of seven sections that are arranged in a logical and systematic structure with the following outline: Section 2 focuses on the transition of the traditional power grids to the smart grids of the future. Section 3 provides an overview of the data-driven methods for the predictive analysis in power systems. Section 4 outlines how predictive analytics is used for making predictions about future events and conditions based on the past data retrieved from power systems. Section 5 discusses machine learning methods in power systems. Section 6 continues this discussion only focusing on using machine learning in smart grids. Finally, Section 7 concludes with overall conclusions, policy implications, major limitations, as well as future pathways for the research on this relevant and timely topic.

2. Traditional Power Grids and Smart Grids

The electric power grid constitutes a complex network that is responsible for delivering reliable and affordable electricity to homes, businesses, and industries. The conventional power grid has been in place for over a century, and it has been reliable for the most part [44,45]. However, with the advent of new technologies and the increasing demand for energy, the conventional power grid is facing numerous challenges, including inefficiencies, lack of flexibility, as well as aging infrastructure. The smart grid leverages digital technologies and two-way communication to optimize power delivery, improve efficiency, and support the integration of renewable energy sources [46,47].

The conventional power systems, illustrated in Figure 1, have historically relied on a small number of centralized and large power generation sources, typically utilizing hydropower or fossil fuel-based power generation systems. This power is transmitted over a large transmission network and ultimately delivered to consumers through a distribution system (Figure 1).

However, recently, the smart grids have gained increased attention of researchers as the power industry transitions towards a more advanced and efficient energy system. Compared to the conventional energy systems, smart grids are characterized by their active nature, which results from the two-way power and information flow enabled by the integration and contribution of every distributed and renewable energy resource. The conventional energy systems are passive due to the unidirectional power and communication flow [48,49]. Figure 2 below demonstrates the basic construction of the smart grid and highlights the distinctive traits of these advanced energy systems.

Smart grids are rapidly replacing older centralized electrical grids and are becoming increasingly prevalent as a part of the modern power systems. In addition, it becomes clear that the smart grid technologies allow for increased flexibility in transmission, improved power quality and an increased ability to integrate renewable energy sources into the electrical grid [50,51,52]. Smart grids can also improve distribution by allowing for distributed energy resources such as solar and wind to be integrated into the system, thereby increasing overall grid capacities [53]. With these advances in grid technologies, electricity generated from renewable sources can be more effectively utilized while older electricity grids with poor power quality can be replaced with the more efficient and flexible transmission networks [54]. This transition towards flexible, intelligent smart grids is revolutionizing the way electricity is generated and distributed, making it easier than ever before to incorporate renewable sources of energy into the existing electrical systems [55]. Additionally, smart grids provide a two-way exchange of information between the electricity supply and the consumers, allowing for better management of electricity consumption. This makes it easier to implement smart charging technology, which optimizes usage of renewable energy sources such as wind and solar while minimizing carbon emissions [56]. The implementation of electric vehicles is also facilitated by the presence of a smart grid, as they are capable of being charged wherever infrastructure is available with maximum efficiency. As these new solutions become more commonplace, it is becoming increasingly important to build efficient smart grids in order to meet our growing need for the renewable energy sources and reduce the dependence on the non-renewable ones [57,58].

Novel technologies such as the distributed energy resources (DERs) also provide many possibilities for smart grids that allow power generation at a local level instead of relying on large and centralized power plants [59,60]. This is an important change that can result in improved environmental performance and increased efficiency from renewable energy sources. In addition, these distributed systems also increase security by making it harder for malicious actors to disrupt the electrical grid or cause blackouts due to cyber threats or physical attacks on substations [61,62]. Sufficient smart grid systems are needed for ensuring reliable access to electricity for all consumers and businesses across the world. Smart grid technology is used to monitor the transmission and distribution of electrical power using advanced digital technology. This allows for two-way communication between utilities and consumers, providing them with insight into their energy use. Smart grids also allow for two-way flow of electricity by enabling electricity to be transmitted from both sources (utilities and consumers). This helps reduce the demand for utilities by allowing them to better manage their resources [63,64,65]. Additionally, smart grids can be equipped with various sensors that can detect any faults or problems with transmission lines or other elements of the grid infrastructure. This might help the utilities to quickly identify and resolve any outages or problems in a timely manner [66]. Modern power systems are transitioning towards flexible and intelligent smart grids to make energy distribution more efficient. The introduction of new technologies offers the utilities the ability to integrate renewable energy sources into their existing energy systems, increasing grid efficiency and allowing for variable renewable energy generation. Smart grids also allow for better control of power flows, enabling better management of intermittent renewable sources. In addition, communication technologies are used to enhance the efficiency of the grid itself by enabling faster response times and higher levels of accuracy in monitoring system performance [67,68].

The transition towards smarter grids offers the utilities, as well as other energy providers, new opportunities in managing their existing energy system more efficiently. Modern power systems are transitioning towards the intelligent, flexible smart grids to meet the increasing energy demands of the smart cities [69,70,71]. Smart electrical grids help utilities better manage energy generation and consumption, increasing energy efficiency and enabling the integration of renewable energies into electricity grids. Renewable solar, wind, hydroelectric and geothermal sources are becoming more cost effective for the electricity utilities to use for their power generation needs [72]. The development of those new technologies in the power sector has made it easier for electrical networks to be monitored remotely in real time, which results in being able to control resources more effectively. Moreover, the smart grid technology allows for greater optimization of renewable resources by utilizing advanced algorithms that can predict energy demand trends and enable more efficient utilization of the existing resources. This helps to reduce overall costs associated with electricity production while also helping to increase the reliability of electric supply networks [73]. With smarter grids, electricity can be generated from renewable sources at a faster rate than before. This increases both the reliability and sustainability of electric networks across different regions. By transitioning to more flexible, intelligent smart grids, energy communities can work together to ensure that reliable electrical infrastructures are in place [74]. Energy prosumers (those who not only consume the energy from the grid but also produce their own energy with renewable sources such as solar or wind power) would be able to connect directly with the electric network and use the energy they generate efficiently [75,76]. Smart grids also enable DERs such as storage batteries and renewable generation systems to be integrated into the existing electrical distribution system which allows for an increased level of flexibility when it comes to changing energy consumption patterns in response to demand [77]. Smart technologies are being implemented in modern power systems at a rapid pace, and include the advanced metering infrastructure (AMI), automated control systems, and communications platforms that allow for greater visibility of electricity usage patterns. All of these enable the utilities and other stakeholders within the energy system to better forecast needs and make necessary adjustments in order to ensure reliable delivery of electricity [78]. Modern power systems are transitioning towards flexible and intelligent smart grids, which can provide consumers with an unprecedented degree of control over their energy usage. Advanced sensors provide real-time information about energy production and consumption, allowing for smarter grid management. This makes it possible to automatically adjust electrical appliances in response to changes in the electricity network. Automation capabilities make it easier for utilities to respond quickly to large solar flares or other events that could affect the electric power system [79]. Smart meters provide the customers with access to more detailed information about their energy usage and help them to make informed decisions about how they use electricity [80].

3. Data-Driven Methods in the Predictive Analysis of Power Systems

In recent years, data-driven methods for predictive analysis of power systems have become increasingly important due to the vast amounts of data available for analysis [81,82]. Deep learning methods, computational intelligence, and machine learning are all being used to provide effective data analytics and developments in Big Data analytics are allowing for the increased accuracy in critical power generation information. Big Data sets also allow for deep learning algorithms and intelligent computing to be used to uncover critical information about a power system’s performance [83,84]. Data analytics is providing more accurate predictions on how the system would perform under certain conditions, allowing better decisions to be made by the analysts regarding energy generation and usage. By utilizing deep learning methods and Big Data analysis, it is possible to gain insights into the operation of a power system that would otherwise not be available when applying the traditional analytical methods [85]. With the help of machine learning algorithms and deep learning techniques, predictive analysis models can be built that are capable of predicting the performance and operation of power systems. With predictive analytics adopters, data mining is used to identify patterns in the data that can be used to anticipate potential issues or events [86]. Furthermore, reinforcement learning technologies can be used to automate system operations to optimize performance. The use of machine learning and deep learning capabilities allows for a greater level of accuracy when predicting how a power system would perform under certain and specific conditions. This enables predictive analytics adopters to make better decisions when dealing with the operational issues or potential risks within their systems [87]. Big Data also provides insight into large-scale trends that might affect the overall functioning of a power system, allowing experts to better understand how different elements interact with each other over time. By adopting the various machine learning techniques and deep learning methods for predictive analysis models, it is possible for organizations to gain insights into their power systems in order to improve their operational efficiency and reduce the risk associated with the unexpected changes or failures within their infrastructure [88]. Predictive analytics provides the ability to interpret data generated from power systems and forecast events, as well as to identify the important trends and patterns to generate accurate data predictions. With the help of sophisticated analytics techniques, such as statistical modeling and empirical methods, predictive power can be created by supplying parameters that are relevant to certain types of events or changes within a power system [89]. Big Data also plays a crucial role in providing the necessary information for machine learning algorithms to process and analyze in order to make more reliable predictions. Through careful application of predictive analysis models, organizations can gain valuable insights into their power systems by leveraging the immense potential offered by modern-day data analytics. Predictive analytics tools allow organizations to utilize current data and prior knowledge to gain useful information about the power system’s performance. This information can be used for various types of predictive analyses that are tailored to specific purposes, such as monitoring changes in customer demand or predicting failure rates of power grids [90]. Additionally, sophisticated predictive analytics tools can help organizations plan marketing campaigns at particular times when they are likely to yield the most positive results. In this way, data-driven methods for predictive analysis can provide invaluable insights into an organization’s power system and inform decisions related to its management. By leveraging the advantages offered by these powerful tools, organizations can gain relevant knowledge about their power systems that would enable them to make better use of their resources and optimize performance for a wide range of activities [91,92]. There are tools for organizations to collect their data from various sources and analyze them with the help of sophisticated algorithms. The results can be presented in the form of data visualization, graphs, tables, and other visuals which provide insights into the underlying trends in their power system. This enables companies to make informed decisions about how they can best manage their systems [93].

With Big Data systems in place, businesses are also able to store large amounts of data in a secure manner by using cloud-based storage services. Furthermore, by using a data warehouse, businesses can easily access and analyze huge amounts of information from multiple sources simultaneously. This allows them to leverage predictive analytics solutions to gain insights into the behavior of their customers, helping them create targeted analytics and provide enhanced customer experiences [94]. Additionally, with the help of intelligence tools, businesses can use smart data solutions to uncover hidden trends and discover new opportunities for growth [95]. Furthermore, cloud security and cloud technology are increasingly being used by financial service companies to secure their systems from external threats. Finally, systems research is also being used by business owners and organizations for developing more efficient tools for predictive analysis of power systems which can be deployed in the cloud environment. All in all, these tools allow businesses to access data quickly and securely while providing valuable insights into how power systems behave [96,97]. By leveraging these intelligent solutions, businesses can better understand their customers’ needs while gaining a competitive advantage over other organizations using the traditional methods of analysis.

In addition, the data-driven methods for predictive analysis of power systems offer an innovative approach to managing and monitoring the complexities of operations. Through the computational models, a modelled system can be used to simulate potential problems and to create more efficient power planning [98]. Approaches such as machine learning and artificial intelligence can be deployed for accurately predicting the future events from past data. Another method known as “system dynamics” is often used for analyzing the dynamics of complex systems by simulating battery conditions over time. This approach is useful for analyzing the prospects of using data-driven methods for predictive analysis of power systems [99]. The data partitioning methods used in this approach enable the assessment time to be effectively reduced, allowing for more accurate results. Using various data sets and quality assessment techniques, the method allows for partitioning of generation cathode materials into various compositions, as well as for prediction to be undertaken with improved performance over that of traditional approaches [100]. Thus, with system dynamics, it is possible to develop better models which can enable more accurate predictions of power system performance in less time than previously achievable. Furthermore, frequency prediction is an important area of predictive analysis, and prediction performance can be improved by using data partitioning methods to divide the training database into meaningful sets. Systematic analysis of the data sets can then be used to identify potential faults in the power system and accuracy might be improved using data-driven methods by introducing additional generators or changing the database configuration. Partitioning techniques are helpful for improving prediction accuracy as they allow a more accurate distinction between different types of power systems and, subsequently, a more efficient fault detection [101]. Consequently, it is possible to use data-driven methods for predictive analysis of power systems with greater accuracy than previously achievable.

4. Predictive Analysis and Historical Data in Power Systems

Predictive analytics is used to make predictions about future events and conditions based on past data from complex systems such as electricity power systems. Predictive analytics represents a branch of advanced analytics that uses data mining techniques, statistic modeling, and machine learning to look at current and historical data to detect trends and patterns [102,103].

Organizations use predictive analytics to sift through current and historical data to detect trends and forecast events and conditions that should occur at a specific time based on supplied parameters. Identified patterns can be used to forecast future events more accurately and reliably than previous tools [104]. The use of historical data in predictive analysis of the state of power systems is helping to facilitate data-driven approaches to system analysis and optimization. This includes significant advances in predictive analytics for adopters of this technology who can make more informed decisions about their energy choices, improve economic efficiency, as well as to reduce risk [105,106]. In addition to traditional measurements, predictive analytics can also leverage learning algorithms and machine learning methods to analyze the large amount of data that can be collected from the power networks. Data technology has enabled model-driven approaches for generation experimental planning and control in power systems as well as providing many new insights into fusion energy research [107]. The use of predictive analytics draws upon advanced analytics, intelligence data mining, and machine learning techniques to detect trends and patterns in current and historical data. This includes a variety of statistic modeling, data mining, machine learning, mathematical processes, and Big Data. Predictive analysis also uses modern methods of modeling statistical models for predicting future outcomes or validating the accuracy of assumptions [108]. With these techniques, it is possible to learn from numerous sources of past events to make informed decisions about the state of power systems in the present and future.

The process typically starts with building predictive models that use statistics to determine the likelihood of certain future outcomes. By using descriptive models and learning predictive modeling, it is possible to observe trends and discover patterns that are useful for predicting future outcomes. With this knowledge, it is then possible to build models that can determine the likelihood of these outcomes. Predictive analytics can be used to detect trends and make predictions based on current and historical data [109]. Descriptive statistical models can be used to generate data that would help in forecasting future events. By identifying patterns and structures in the existing data, underlying processes can be determined, and models can be used to make predictions. The application of predictive analytics is also beneficial as it uses supplied parameters to determine relationships between current and historical data which allows us to forecast future events based on the patterns identified in the existing data. Predictive models are used to make predictions about future events by basing predictions on observations of past behavior patterns [110].

All of this can help organizations to optimize the quality of their operations and services by predicting the likelihood of certain events or outcomes before they happen. Predictive analytics involves the use of the statistical methods, data mining techniques and artificial intelligence algorithms (such as neural networks and decision trees) in order to train algorithms that can generate such forecasts accurately. These tools are used in combination with historical data to create a more accurate prediction than what is achievable through human predictions alone. By using predictive models and incorporating historical data into predictive analysis, organizations are able to gain invaluable insights into the behavior patterns of their customers, machines, resources and systems in order to better plan for the future [111].

Predictive analytics can be used by businesses to gain a competitive advantage by better managing their operations and resources, enabling them to make more accurate forecasts. The embedded insights that can be gleaned from the predictive analytics provide a unique opportunity for companies to gain valuable insights into the customer base, product lines and market trends in order to provide better services for their customers. Predictive analytics can be also used to anticipate, forecast, and predict future customer behavior as well as the energy needs [112]. This kind of sophisticated predictive analytics tools can also be applied to maintenance predictive maintenance, which is the use of current data to anticipate equipment failures or forecast trends that may affect operations. By applying this approach, companies can identify and predict the impact of these events on their supply chain operations and equipment maintenance in order to reduce operating costs. Companies can also use such tools and models to accurately forecast material demand and failure forecasts that enable them to improve their operations [113,114]. Additionally, they can use historical data to make accurate predictions on market trends, predict risk, and provide early fraud detection using advanced data analytics. Then, organizations can collect more data on their employees, customers and products in order to make better predictions. With the right predictive analysis tools and models, companies are able to forecast the impact of their products, brands, employees, and customers. Using historical data in predictive analysis of the state of power systems allows companies to make impactful business decisions. This involves data mining, examining data, preparing data, and testing models to extract insights and identify trends that influence results [115]. Predictive analytics practice also consists of creating hypotheses and models and testing them against existing historical data before publishing them and using them to make decisions. This allows organizations to better understand and anticipate future customer actions or behaviors.

Using historical data such as previous financial statements or customers who bought a particular product, predictive analytics helps to correlate that data with predictions such as how many people clicked on a certain advertisement. By using control algorithms to predict lead likelihood, organizations can make decisions about issues such as project sales and craft a picture of what their future sales revenue might look like [116]. Companies can use predictive analytics techniques to boost customer experience, as well as improve their predictive technology. Predictive analytics can make truly valuable predictions by helping organizations plan businesses and build understanding. Network algorithms are used for studying the relationships between variables, allowing businesses to plan ahead and better anticipate future outcomes. With enough historical data, predictive analytics can help with business governance in terms of typical consumer behaviors [117].

There are also neural networks that are used with that data in the same way as human mind functions, mimicking the way the brain works to recognize patterns and create relationships. One can use neural networks for many applications such as social media, health care, and even business relationships. One of them is the predictive analysis of the state of power systems [118,119]. This requires predictive analytics, which in turn requires people to make judicious decisions. One can use the historical data from these systems to create data models and put predictions into operations strategy. This can help to improve shift and prolong asset life, thus improving performance and making the utilization of resources more efficient. Data centers can also benefit from this predictive analysis, where leadership can use the problematic data to recognize an increasingly vital need for improvement in their performance and therefore optimize the use of energy.

5. Machine Learning Methods in Power Systems

The application of machine learning in energy and power systems is becoming increasingly important. Machine learning (ML) represents a technology that enables machines to discover patterns in data without the need of explicit instructions. It found a wide use in the energy industry due to its ability to control and forecast energy demand [120,121]. Unsupervised machine learning helps data centers cluster large amounts of data for better visibility and applications, while supervised ML is used for more specific tasks such as forecasting or control applications. Reinforcement learning is also used in order to optimize and control the IoE in power grids [122,123].

In addition, there are artificial neural networks (ANNs) which represent the modern machine learning algorithms that use deep neural networks with massive datasets and high dimensional data to create much simpler models. The capability of machines to learn from raw, high-dimensional data allows them to outperform traditional approaches in many tasks. ANNs have been shown to increase the performance of network-wide energy management [124,125]. ML constitutes a key component to many of the learning projects, with supervised machine learning being one of the most popular. Supervised ML has different predictive capabilities, such as decision trees and support vector machines, allowing a data scientist to experiment with different data models and configurations. By using supervised ML models, it is possible to achieve better efficiency in demand forecasting. Moreover, these algorithms can improve the existing energy management systems by providing more accurate experimental results through an improved data model [126]. Ultimately, this leads to better efficiency in terms of demand forecasting and energy management systems.

Machine learning has the potential to revolutionize the power grid industry. It is being used in various applications such as supervised and unsupervised learning, wavelet transform, persistence forecasting, and more [127,128]. With the increasing complexity of energy fluctuations in power grids, machine learning techniques offer better performance than traditional methods. Quantile regression is one of the most popular machine learning models for load forecasting in power grids. It employs artificial neural networks in order to learn from data patterns and make predictions about future energy needs. The network includes layers of neurons that are connected to each other through weights which represent different features of energy demand or load on a grid system. By using deep learning algorithms, the network can be further trained to improve its accuracy when predicting energy needs or loads on a grid system [129,130].

Machine learning can be used to analyze a variety of data sources such as connected distribution energy, topology estimation, operational radial topology, and simulation module. Such data are then used to create a smart meter data set from which the system can identify patterns that lead to better load forecasting. This information is then used for outage management and load balancing [131]. The IoE provides an additional layer of intelligence by utilizing machine learning algorithms such as supervised learning, unsupervised learning, and reinforcement learning. Supervised learning allows for the classification of meter ping measurements, while unsupervised algorithms are used to cluster similar sections using power data sets [132]. Reinforcement learning allows for optimization in distribution systems by combining it with other optimization techniques such as heuristics based on historical load profiles or real-time control strategies such as peak shaving or demand response services [133]. These algorithms help improve accuracy when predicting loads on different parts of the grid system, allowing for more efficient operations and better utilization of resources within the power grid. Machine learning and IoE in power grid become the important research areas in the energy industry, as they enable resilient grid operations and help to anticipate future efficiency. Sensors feed data to intelligent surfaces which are then used to train deep neural networks that can generate optimal power injection [134]. Smart cities have already started utilizing this technology for the fuel economy, while research into more effective methods of data collection and analysis continues. Systems such as these rely on a powerful network of neural connections so that they can control the grid in real time with minimal errors. As machine learning algorithms become increasingly sophisticated, it is possible to develop advanced control systems that provide more efficient and cost-effective solutions for the power grid [135].

Several research papers have been published in recent years focusing on the potential of machine learning and IoE in power grid applications. These broad research areas include energy informatics, efficient cloud data analysis and management, intelligent surfaces, wireless networks, as well as artificial intelligence. Research interests are also focused on the use of machine learning algorithms for network management to improve system reliability [136,137,138]. Some researchers developed a few algorithms for intelligent network control systems as well as intelligent surfaces for data acquisition and communication with other systems within the network. This comprehensive approach promises to revolutionize how we manage our grids by providing an efficient way to collect, analyze, store and share information about energy usage across the grid [139,140]. Machine learning is an important part of this revolution, because it allows us to build systems that are able to learn from data and make decisions based on its observations. Unsupervised machine learning techniques can be used to detect patterns in large datasets that would otherwise be too complex for humans to discern. Training data can also be used with supervised machine learning algorithms such as music streaming or customer loyalty programs, allowing for a more accurate prediction of future user behavior [141]. Supervised learning algorithms require labelled training data to learn how to distinguish between different types of inputs [142]. For example, when training a supervised machine learning model on customer loyalty programs, the user has to provide labels for the various activities that customers engage in. The model then uses these labels to learn how to identify similar activities and predict future customer behavior. Unsupervised learning algorithms are also used in applications such as image recognition or natural language processing where there is no need for labelled training data [143].

Neural networks, adversarial networks, and generative adversarial networks are all popular machine learning systems used for recognizing patterns in spatial data. These systems learn from new data and can adapt quickly to any changes in future data sets. Geometrical locations or temporal relationships can be learned by these models and used to improve the accuracy of energy management decisions. Machine learning IoE in power grid has seen a tremendous increase in research and development due to its potential applications. It can revolutionize the way energy is managed and distributed [144]. Artificial intelligence (AI) is at the core of these technologies, raising many ethical questions. These systems are also being used to predict energy usage patterns and optimize distribution networks with the data collected from various sources such as buildings, cars, or power plants [145].

Nowadays, the rapidly changing energy sector is integrating renewable energy into the existing power systems, which requires continuous energy sector operations and irregular energy consumption patterns. Machine learning technology solutions are being used to maximize renewable energy resources, demand forecasting, and minimize carbon emissions. The application of Industry 4.0 digital technologies for power generation is also increasing as variable generation sources become more common [146]. This technology allows for the accurate prediction of generation net load and commodity pricing. Additionally, ML can be used to provide accurate forecasts in the long term by incorporating different energy sources that are available in the short term. By utilizing big data, energy suppliers can create a comprehensive model that can predict the needs of customers in order to provide reliable electrical infrastructures. This can help reduce the strain on overstressed centralized electrical grids and prevent outages [147]. Furthermore, machine learning applications in smart grids enable sustainable power production through distributed energy resources and renewable energy generation. This helps support power generation in Industry 4.0 developed countries that are increasingly focusing on following new environmental initiatives [148]. Smart meter implementation is also an effective way for energy suppliers to track and monitor the amount of electrical power being consumed, as well as allowing them to rate sources of energy consumption more accurately [149].

Therefore, machine learning applications have become increasingly important for providing reliable electrical infrastructures and supporting the implementation technology in order to focus on renewable sources of energy. The smart grid technology leverages machine learning algorithms to take available energy resources from utility sources, as well as other renewable sources, to restore the distribution system [150]. Machine learning algorithms are used to analyze data from the grid and identify patterns that can be used for forecasting, predicting demand, and for enhancing cybersecurity. Moreover, the smart grid relies on its ability to monitor and control the flow of electricity within a utility system more accurately than classic consumption models [151]. It is essential for generating electrical power from renewable energy sources, such as solar energy, wind energy and hydropower, which are difficult to utilize due to their intermittent production nature. Machine learning applications in the smart grid are designed to make the grid more efficient, reliable, and flexible by using transmission energy management systems and connected distribution energy resources. It supports deployed inverter-based DERs, estimated load demand from automated metering infrastructure (AMI), report estimation for distribution management systems, outage management system and distribution system [152]. In addition, meter ping measurements that compliments the estimation that is already the cornerstone of the power flow. Thence, the strength of the machine learning application in the smart grid consists of three aspects: energy sources, AMI to estimate the already present cornerstone of power flow, and integrated control systems. These three aspects enable forecasting renewable energy sources and demand side measurement accuracy with cybersecurity as its cornerstones [153]. Smart home demand forecasting is a challenging task because of the heterogeneity in energy consumption patterns and the presence of several variables that affect the consumption. To tackle this issue, researchers proposed models that exhibit different predictive capabilities. Experimental results showed that it can outperform other models by capturing the models’ general trends, as well as specific information from each of the forecasting electricity prices, residential demand response, and demand forecasting. The models can be used for residential customers to make decisions about purchasing energy at different times of the day depending on their needs [154].

Due to the energy IoE sensors and electric loads, the data-driven mode, including machine learning methods, has been widely used in energy research. In the past decades, significant advances have been made in the areas of power system analysis, disruption prediction, experimental planning, and demand forecasting. Moreover, ML methods have also been applied to renewable energy generation prediction, surrogate model generation, and response evaluation [155]. Nevertheless, there is still much to be achieved—significant efforts are being made to improve the adaptabilities of ML models to power systems and to develop new congestion control methods for transmission system operator congestion problems. The load side of the power grid transactions, power generation, and energy scheduling optimization are solved by the dynamic model. The price of the outputs of the model is based on the line congestion and optimal power flow problem solutions [156]. The development of ML technology in power systems can solve the problems in real time by using ML and neural networks, which will bring about a comprehensive change in power grid operations and energy systems in the future [157].

In addition, ML methods and tools include analyzing machine learning to improve energy efficiency, predicting energy consumption, and managing electric grid infrastructure. They provide many opportunities for using machine learning methods to improve the effectiveness of building energy efficiency systems, as well as for predicting future room occupancy and providing smart recommendations [158]. One can see the potential of using IoT deployment to increase the accuracy of predictions and provide more detailed insights into a system’s operations. In addition, the importance of adopting an IML (In-Memory Learning) method model for forecasting room occupancy, which can include further dimensions such as dispersed IoT systems and sensor data is also important within the context of the discussion above [159].

Some authors discuss the ways in which the ML can be used in highly sophisticated robots and intelligent controllers to create reliable, autonomous systems. They also go on to demonstrate that machine learning can be used to learn large amounts of data, including significant advances in fusion energy research [160,161]. This type of research requires intelligence for an efficient generation of experimental planning and models for accurate predictions. Machine learning is being utilized for several applications such as heating, ventilation, and air conditioning (HVAC) systems, energy management, building occupants, homes, and businesses. With the advent of IoE technologies, machine learning is being used to drive smart energy systems to increase energy efficiency and reduce energy consumption [162]. Smart city systems are incorporating smart grids, enabling sustainable energy, and enhancing governance. Machine learning is being used to detect power consumption attacks and improve data exchange between connected devices. AI chips are being used in learning sensor systems that are connected to an IoT network for collecting data from the environment for further analysis. This helps enhance the understanding of how energy is consumed and used in different areas of a city or region [163].

Furthermore, ML also assists with managing Big Data for better decision making, as well as creating predictive models that can be applied across various urban areas. It can help with optimizing resources, improving services and enhancing safety among citizens [164]. ML-driven smart electric power systems enable information technologies to be used for future electricity systems. These technologies are used for industrial demand side flexibility. The renewable energy sources affect pricing energy management as the new machine-learning technology examines the demand-side flexibility to profile economic advantages of a smart grid [165]. New opportunities, such as trading power system flexibility and examination of its potential benefits in the demand-side management also constitute an important aspect. The power system flexibility can be used to enhance the profile of the economic advantage with respect to pricing energy management [166]. There are various applications of these new technologies in the industrial robot manufacturing sector. This opens up a new trend in the application of programmable industrial robots and machine learning algorithms to improve IoE devices, which has been gaining more attention in recent years. Many researchers have been focusing on developing machine learning-driven smart electric power systems, which integrate renewable energy sources with smart grid solutions. Their research interests include machine learning algorithms and wireless sensor networks for cyber security applications in the industrial IoE domain [167,168].

Figure 3 below reports the dynamics of frequency of search requests (“interest over time”—the search interest relative to the highest point on the chart for the given region and time) of the keywords “machine learning” and “power systems” worldwide from 2013 to 2023. The figure was created based on the analysis obtained from the Google Trends toolkit (offered by Google search engine) which shows the change in search queries of the main relevant concepts in order to determine and compare the peak periods with the periods of the most significant changes in online searchers. It becomes apparent that while the term “power systems” had been a subject of higher search interest in 2013–2015, it was overpassed by the term “machine learning”. The latter search term has been yielding exponentially growing search interest from 2016 onwards (with a brief downfall during the COVID-19 pandemic in 2020–2021).

In addition, we conducted a bibliometric network analysis of the keywords “machine learning” and “power systems” in the research literature. Using the keywords “machine learning” and “power systems” in scientific papers, reports, proceedings, and book chapters indexed in the Web of Science Core Collection (WoS) database, a sample of 7457 papers was retrieved. The analysis of this sample was then conducted using the VOSViewer v. 1.6.15 software. The VOSViewer software is very popular nowadays for identifying the dominant trends in intersectoral research related to various topics, as well as for finding out which tools and instruments in various topics (such as the machine learning and smart power grids, in our case) are more relevant for academics and scholars [170].

Figure 4 below shows the results of the network map based on the text data. In total, seven main clusters were identified. The analysis of using key words and phrases in the publications retrieved from WoS revealed that terms connected with “machine learning” and “power systems” are most often related to the following: (1) energy management (Cluster 1); (2) artificial intelligence (Cluster 2); (3) anomaly detection (Cluster 3); (4) classifier (Cluster 4); (5) neural network (Cluster 5); (6) blockchain (Cluster 6); (7) automation (Cluster 7).

Figure 5 below shows the results of the network map based on the bibliometric data (e.g., keyword co-occurrences, citation, or bibliographic coupling). One can clearly see the variety of topics that are interconnected with the machine learning applications in smart grids (e.g., photovoltaic systems, anomaly detection, Internet of Energy, deep neural networks, or, for example, optimization).

6. Using Machine Learning in Smart Grids

As it has been already outlined above, one of the main advantages of using machine learning and data-driven methods for the predictive analysis of power systems is the ability to handle large volumes of data from various sources. These methods can effectively capture the complex and dynamic behavior of power systems which are difficult to model using traditional analytical methods. For instance, ANNs have been used for short-term load forecasting with high accuracy, while support vector machines (SVMs) have been applied to fault detection and diagnosis in power systems [171,172].

However, the use of machine learning and data-driven methods for the predictive analysis of power systems also poses a number of challenges. One of the main challenges is the availability and quality of data. Power system data are often incomplete, noisy, and non-stationary which can affect the accuracy and reliability of predictive models. In addition, the lack of interpretability of these models can make it difficult to understand the underlying mechanisms of power system behavior [173].

The use of machine learning and data-driven methods for predictive analysis in power systems is expected to grow in the future as more data becomes available and the algorithms become more sophisticated. However, there are also some challenges that need to be addressed to ensure the successful implementation of these methods. For example, there is a need for standardization in data collection and analysis to ensure the accuracy of the models. Additionally, there is a need for further research to develop more sophisticated algorithms that can handle the complexity of power system data and ensure reliable predictions [174,175]. Based on the detailed and exhaustive review of the research literature, we have identified four key areas of using machine learning in smart grids identified in the research literature;

Table 1. Similarities and differences in using machine learning in smart grids.

Key Area	Similarities	Differences
Forecasting in smart grids	Necessary for efficient grid operation and management	Two primary types of forecasting: electric load and price forecasting and renewable power generation prediction. Various methods and techniques are used for each type of forecasting, such as time series analysis, regression analysis, artificial neural networks, econometric models, and machine learning techniques.
Machine learning in fault and failure analysis	Machine learning algorithms can be used for prediction, classification, identification, and location of faults and failures.	Fault classification, identification, and location can be performed using data from various sensors with high accuracy.
Machine learning in demand-side management	DSM is an approach to energy management that involves changing the patterns of electricity consumption to align with the availability of energy supply.	Machine learning algorithms can be used to analyze historical data and identify patterns that can be used to predict future energy demands and prices.
Machine learning in energy trading	Machine learning algorithms can be used for price forecasting, energy trading strategy optimization, and energy market analysis.	Machine learning algorithms can be used to analyze data from various sources, such as energy consumption, weather, and social media, to predict energy prices and market trends.

Source: Own results

(that follows below) summarizes the main similarities and differences among those areas.

6.1. Forecasting in Smart Grids

Smart grid technology integrates advanced information and communication technologies to enhance the operation and management of the electrical power system. One of the essential components of the smart grid is the capability to forecast electricity load and renewable power generation accurately [176]. According to the researchers, forecasting in the smart grid features two primary types of forecasting, namely electric load and price forecasting as well as the renewable power generation prediction [177]. Many studies suggest that accurate forecasting is crucial in achieving efficient grid operation and management, as well as the different methods and techniques used for forecasting [178].

As outlined above in this review paper, the smart grid represents a modern electrical power system that integrates advanced information and communication technologies with the electrical power grid infrastructure. The smart grid provides advanced capabilities for efficient grid operation and management, such as real-time monitoring, control, and communication. Forecasting represents one of the critical components of this grid which plays a vital role in achieving efficient grid operation and management. Forecasting involves predicting future electric load and renewable power generation, which are necessary for the efficient scheduling of power generation, transmission, and distribution [179,180]. Electric load and price forecasting are the two primary types of forecasting used in the smart grid. Electric load forecasting involves predicting the future electric load of the power system, which is necessary for efficient grid operation and management. Accurate load forecasting helps utilities to plan and schedule power generation and transmission, which leads to efficient and cost-effective operation of the power system [181,182]. Various methods and techniques are used for electric load forecasting, such as time series analysis, regression analysis, and artificial neural networks. Price forecasting is another critical aspect of smart grid forecasting. Price forecasting involves predicting the future electricity prices in the power market, which are essential for efficient market operation and management [183,184]. Accurate price forecasting helps electricity market participants to make informed decisions regarding power generation, transmission, and consumption. Various methods and techniques are used for price forecasting, such as econometric models, time series analysis, and machine learning techniques [185].

Renewable power generation prediction is another critical aspect of smart grid forecasting. Renewable power generation, such as solar and wind power, is highly variable and uncertain, making it challenging to integrate into the power system. Researchers agree that timely and precise prediction of renewable power generation is quite essential for its efficient integration and management [186,187]. Sophisticated methods and techniques are used for renewable power generation prediction, such as the already mentioned statistical models, artificial neural networks, and hybrid models.

6.2. Machine Learning in Fault and Failure Analysis

The transformation of the conventional power grid to a smart grid has been ongoing for quite some time now, driven by the need for more efficient and reliable energy supply to consumers. Machine learning techniques have emerged as a useful tool in various applications of the smart grid, including prediction, classification, identification, and location of faults and failures [188]. All of that helped the applications of machine learning techniques in the transformation of the conventional power grid to a more efficient and reliable smart grid for ensuring the safe and reliable energy supply to the consumers [189].

The research literature shows that one of the essential applications of machine learning in the smart grid is prediction. This is because electric load and price forecasting are critical for efficient energy management in the smart grid. Machine learning algorithms can be trained to analyze historical data and identify patterns that can be used to predict future energy demands and prices [190]. These predictions can help utilities plan their energy generation and distribution strategies, reducing waste and increasing efficiency. Another important application of machine learning in the smart grid is classification. Fault classification is a crucial task in maintaining the reliability and safety of the power grid [191]. Machine learning algorithms can analyze data from various sensors and identify different types of faults and their severity. This information can be used to prioritize maintenance tasks and prevent potential failures. In addition, machine learning algorithms can be trained to classify different types of power quality issues, such as voltage sags, interruptions, and transients, allowing utilities to take corrective action to minimize their impact on consumers [192].

Many papers suggest that machine learning algorithms can also be used for identification purposes in the smart grid. For instance, identifying the type and location of a fault can be difficult in a large power grid. Machine learning algorithms can analyze data from various sensors and identify the type and location of a fault with high accuracy. This information can be used to prioritize maintenance tasks and reduce downtime, improving the reliability of the power grid [193]. Finally, machine learning algorithms can be used for location purposes in the smart grid. For instance, identifying the location of a fault is crucial in maintaining the reliability and safety of the power grid [194]. Machine learning algorithms can analyze data from various sensors and identify the location of a fault with high accuracy. This information can be used to direct maintenance crews to the location of the fault, reducing downtime and improving the reliability of the power grid [195].

6.3. Machine Learning in Demand-Side Management

The demand for electricity is increasing worldwide, and it is expected to continue growing as economies expand and populations increase. With the rise of renewable energy sources, it is important to ensure that the supply of electricity is balanced with demand [196]. Demand-side management (DSM) is an approach to energy management that involves changing the patterns of electricity consumption to align with the availability of energy supply. According to many researchers, DSM is becoming increasingly important for electric utilities to ensure reliable energy supply to their customers [197]. In recent years, machine learning techniques have been utilized in DSM planning, implementation, and monitoring. The role of machine learning in DSM planning, implementation, and monitoring of electric utilities specifically designed to influence customers’ electricity utilization is very important. DSM planning involves the identification of opportunities to manage the demand for electricity. Traditional DSM planning methods involve energy audits, customer surveys, and analysis of electricity consumption patterns [198]. However, these methods are time-consuming and may not provide accurate predictions. Machine learning techniques, on the other hand, can provide accurate and timely predictions of electricity demand. By analyzing historical data, machine learning algorithms can identify patterns in electricity consumption and predict future demand. These predictions can be used to plan DSM strategies, such as peak shaving and load shifting [199]. Peak shaving involves reducing electricity consumption during peak periods to reduce the demand for electricity. Load shifting represents the re-scheduling the electricity consumption towards the off-peak periods. Machine learning techniques can predict peak periods accurately, allowing for more effective peak shaving and load shifting strategies. For instance, the use of smart meters and machine learning algorithms can predict peak periods and send signals to appliances to reduce their electricity consumption during those periods [200,201].

DSM implementation involves the implementation of strategies to manage the demand for electricity. Machine learning techniques can help electric utilities to implement DSM strategies more effectively [202]. For example, machine learning algorithms can analyze data from smart meters and other sensors to identify appliances that consume the most electricity. This information can be used to target DSM strategies in a more efficient way. By targeting the most energy-consuming appliances, DSM strategies can be better implemented. In addition, machine learning can also be used to optimize DSM strategies [203]. For example, machine learning algorithms can analyze data on electricity consumption, weather patterns, and other factors to optimize peak shaving and load shifting strategies. By optimizing DSM strategies, electric utilities can reduce the demand for electricity during peak periods and improve the reliability of the energy supply [204]. Moreover, DSM monitoring involves the monitoring of the effectiveness of DSM strategies. Traditional monitoring methods involve manual data collection and analysis. However, machine learning techniques can provide more accurate and timely monitoring. Many researchers show that machine learning algorithms can analyze data from smart meters and other sensors to identify the effectiveness of DSM strategies. This information can be used to adjust DSM strategies and improve their effectiveness. Machine learning can also be used to identify anomalies in electricity consumption [205,206].

In addition, the research literature asserts that anomalies in electricity consumption can be an indication of faults or failures in the electricity supply system. By using machine learning algorithms to identify anomalies, electric utilities can detect faults and failures more quickly and reduce downtime. Machine learning algorithms can also be used to identify the location of faults and failures, allowing for more efficient repairs [207,208].

6.4. Machine Learning in Cybersecurity of Smart Grids

In the smart grid environment, the communication between various IoT devices and cloud computing resources is essential to ensure efficient and effective energy management. However, the rise of cyber-attacks targeting these devices has become a major concern [209,210]. Therefore, many researchers suggest that there is a need to deploy robust security mechanisms that can safeguard these devices against various security threats, including the stealthy attacks involving the sensors or smart meters within the smart systems; that might be important for the future research on the topic covered by this review [211,212]. Hence, the role of machine learning in cyberspace security to enable safe communication among IoT devices in the smart grid and cloud computing resources cannot be underestimated [213,214].

The relevant literature demonstrates that machine learning has shown promising results in various cybersecurity applications such as intrusion detection, malware analysis, and network security. It has the capability to learn from data and detect anomalous behavior in real time, which is crucial for identifying security threats [215,216,217]. One of the primary advantages of machine learning is its ability to learn from historical data, identify patterns, and predict future outcomes. Therefore, machine learning can be used to detect and prevent cyber-attacks in the smart grid environment [218,219].

Moreover, intrusion detection is one of the most critical aspects of cybersecurity. Traditional intrusion detection systems (IDS) are based on rule-based systems, which rely on predefined rules to identify attacks. However, these rule-based systems can be easily bypassed by attackers using sophisticated attack techniques [220]. Machine learning-based IDS, on the other hand, can detect new and unknown attacks by learning from historical data. Machine learning algorithms can detect anomalies in real time by identifying patterns in network traffic, user behavior, and system logs. Additionally, machine learning algorithms can adapt to new threats by continuously learning from new data [221].

Another crucial aspect with regard to the above discussion is the malware that represents a major security threat in the smart grid environment [222]. Malware can infect IoT devices and cloud computing resources and cause significant damage to the system. Therefore, it is essential to detect and remove malware from the system. Machine learning algorithms can be used to identify and classify malware based on its behavior. These algorithms can detect malware by analyzing its code and identifying patterns that indicate malicious behavior. Furthermore, machine learning algorithms can also predict the behavior of malware, which can be used to develop more effective countermeasures [223,224].

Finally, the relevant scientific literature demonstrates that network security constitutes another critical element of cybersecurity in the smart grid environment [225,226]. It demonstrates that machine learning can be used to detect network intrusions, identify vulnerable systems, and detect anomalies in network traffic. In addition, machine learning algorithms can analyze network traffic and identify patterns that indicate attacks. Moreover, machine learning can also be used to develop predictive models that can anticipate future network attacks and take preventive measures [227].

7. Conclusions and Recommendations

All in all, it appears that machine learning and data-driven methods have shown great promise for predictive analysis in modern day power systems that are starting to be dominated by the smart grids. With the penetration of the information and communication technologies (ICT) into all spheres of our daily lives, including the electricity power supply and energy grids, a plethora of data is being collected using smart meters and other devices. Then, it needs to be analyzed in real time to predict, optimize, and facilitate the two-way flow of energy between the prosumers, smart cities, smart industries, and generation of energy that are increasingly relying upon the renewable energy sources and solutions. These methods offer significant benefits, including improved accuracy, reduced cost, and increased efficiency. However, there are also some challenges to using these methods, such as the need for high-quality data and the risk of information flooding.

The use of machine learning and data-driven methods for predictive analysis of power systems has significant potential for improving the accuracy and reliability of power system operations. These methods offer the ability to handle large volumes of data generated from various sources and can effectively capture the complex and dynamic behavior of power systems. However, the availability and quality of data, as well as the lack of interpretability of these models, pose significant challenges. Future research should focus on addressing these challenges and developing more accurate and reliable predictive models for power systems.

Furthermore, it has been shown in our review that forecasting is a critical aspect of the smart grid. Accurate forecasting of electric load and price helps utilities and electricity market participants to make informed decisions regarding power generation, transmission, and consumption. Accurate prediction of renewable power generation is essential for efficient integration and management of renewable power generation. Various methods and techniques are used for forecasting, such as time series analysis, regression analysis, artificial neural networks, and machine learning techniques. The development and application of accurate forecasting techniques are necessary for achieving efficient and cost-effective operation of the smart grid. The transition from conventional power systems to smart grids represents a significant step forward in the evolution of the energy industry. Smart grid technologies enable the integration of distributed and renewable energy resources, resulting in a more reliable, efficient, and sustainable energy system. The smart grid architecture incorporates advanced sensors and control systems that facilitate bidirectional communication and power flow between different components of the system. Smart grid systems have numerous benefits that make them an attractive option for the future of energy.

In addition, it stems from our review that machine learning and data-driven methods appear to be crucial for the IoE. Today, IoE is becoming key in enhancing the distributed DERs that are represented by the small-scale renewable energy sources (e.g., rooftop photovoltaic panels or energy storage devices such as Tesla Powerwall). The IoE enables DERs to engage into the frequency regulation and voltage support. This can enhance the reliability and resilience of the electricity system, as well as reduce the dependence on centralized power generation in the traditional power systems, thus enabling the quicker transition to the smart grid of the future electricity markets. Future research should focus on addressing these challenges and developing more effective market design approaches for the IoE. Such applications as peer-to-peer (P2P) electricity generation and trading (some of them are already being used) might represent the future of the independent smart grids and the autonomic power systems, and therefore they need more attention from the researchers and academics.

It is quite clear that accurate forecasting is essential for efficient grid operation and management. Electric load and price forecasting help utilities to plan and schedule power generation and transmission efficiently, leading to cost-effective operation of the power system. Price forecasting helps electricity market participants to make informed decisions regarding power generation, transmission, and consumption. Renewable power generation prediction is essential for efficient integration and management of renewable power generation. Accurate prediction of renewable power generation helps to ensure reliable and cost-effective operation of the power system.

Furthermore, machine learning techniques have emerged as a useful tool in the transformation of the conventional power grid to a more efficient and reliable smart grid for ensuring the safe and reliable energy supply to the consumers. Machine learning algorithms can be used for various applications in the smart grid, including prediction, classification, identification, and location of faults and failures. These applications can improve the efficiency and reliability of the power grid, reducing waste and downtime, and ensuring safe and reliable energy supply to consumers. However, there are still several challenges associated with the use of machine learning techniques in the smart grid, including data quality, model interpretability, and security concerns. Further research is needed to address these challenges and develop more advanced machine learning techniques that can support the transformation of the power grid to a smarter and more reliable system.

Moreover, machine learning techniques are becoming increasingly important in DSM planning, implementation, and monitoring of electric utilities. By analyzing historical data, machine learning algorithms can provide accurate predictions of electricity demand and optimize DSM strategies. Machine learning algorithms can also be used to monitor the effectiveness of DSM strategies and detect faults and failures in the electricity supply system. As the demand for electricity continues to increase, the role of machine learning in DSM will become even more important in ensuring reliable and efficient energy supply to customers.

Finally, when it comes to the security of the smart grids of the future, it needs to be noted that due to the heavy usage of information and data (including the Big Data), the cybersecurity and data safety need to be ensured since they might constitute one of the biggest fears of the consumers and prosumers in accepting the novel technological solutions on the energy market. Our review has demonstrated that machine learning has the potential to revolutionize the cybersecurity landscape in the smart grid environment. It can enable safe communication among IoT devices and cloud computing resources protected against various security threats. Machine learning can detect and prevent cyber-attacks in real time, detect and remove malware, and identify vulnerable systems. Therefore, it is essential to integrate machine learning into the cybersecurity framework of the smart grid to ensure the safe and reliable operation of the system.

The limitations of the machine learning and data-driven methods in modern power systems also need to be mentioned here in order to provide a complete picture. One of the main limitations of machine learning and data-driven methods is the limited amount and quality of data available for analysis. In energy and power systems, the data available for analysis are often incomplete, noisy, and inconsistent. Additionally, the quality of the data can also affect the accuracy of the predictive models. Another limitation is the limited interpretability and transparency since machine learning and data-driven models are often considered black boxes, where it is difficult to understand how the model arrived at its predictions, which becomes problematic in energy and power systems, where the decisions made based on the predictions of the models can have significant consequences. Energy and power systems represent complex interconnections that involve a wide range of variables that interact with each other in complex ways. This complexity can be challenging for machine learning and data-driven methods which may struggle to handle complex relationships between variables. Yet another limitation is the difficulty in accounting for external factors that can affect energy and power systems (e.g., changes in government policies, weather conditions, or natural disasters (including the recent COVID-19 pandemic).

From our review and analysis of the available relevant literature sources, it becomes quite apparent that the prospects and challenges of the machine learning and data-driven methods for the predictive analysis of power systems in the 21st century can be very high and promising. Novel digital technologies are firmly grasping their place in the traditional power grids that are undergoing transformation on their way to becoming the modern smart grids. These technologies have gained even more attention and importance after the COVID-19 pandemic that brought about the “digital surge” in every sphere of economic and social life, including the energy sector. It is likely that in the near future, these current tendencies are going to further deepen and enhance themselves as the transformation of the energy sector is now ongoing with all its consequences and effects for the modern power systems.