A Comprehensive Review of Hybrid State Estimation in Power Systems: Challenges, Opportunities and Prospects

Abstract

Due to the increasing demand for electricity, competitive electricity markets, and economic concerns, power systems are operating near their stability margins. As a result, power systems become more vulnerable following disturbances, particularly from a dynamic point of view. To maintain the stability of power systems, operators need to continuously monitor and analyze the grid’s state. Since modern power systems are large-scale, non-linear, complex, and interconnected, it is quite challenging and computationally demanding to monitor, control, and analyze them in real time. State Estimation (SE) is one of the most effective tools available to assist operators in monitoring power systems. To enhance measurement redundancy in power systems, employing multiple measurement sources is essential for optimal monitoring. In this regard, this paper, following a brief explanation of the SE concept and its different categories, highlights the significance of Hybrid State Estimation (HSE) techniques, which combine the most used data resources in power systems, traditional Supervisory Control and Data Acquisition (SCADA) system measurements and Phasor Measurement Units (PMUs) measurements. Additionally, recommendations for future research are provided.

1. Introduction

Power systems are one of the most sophisticated and largest systems, with a significant number of substations that are interconnected through transformers and lines. Furthermore, different meters and protection devices are installed in power systems to protect and control them [1]. Hence, monitoring and protecting them from unexpected situations in real time are challenging issues for engineers. One of the most powerful tools that enable operators to monitor the system approximately in real-time is State Estimation (SE) [2]. The concept of power systems SE was introduced in the 1970s for the first time, and since then, numerous researchers have studied SE of power systems [3,4,5]. SE acts as a fundamental module of Energy Management Systems (EMSs) because EMS methods are vulnerable to bad or missing data, network topology, and any measurement or parameter errors; hence, they are not capable of using raw collected measurements [1,3,6]. The responsibility of SE is to process the raw data set, remove the errors, and find the optimum prediction of variables that contain the magnitude and phase angle of voltages for all buses. This generated information plays a vital role in modern EMSs, where a variety of applications depend on precise and updated snapshots of the operating condition of the system [7].

Although most SE methods are based on Static State Estimation (SSE) and assume that the system is operating in its steady-state mode, loads and generators are changing dynamically. Hence, current SSE methods employed in EMS cannot capture the system’s dynamics and need to be modified using new techniques like Dynamic State Estimation (DSE) [8]. In recent decades, the advent of Phasor Measurement Units (PMUs) and modern communication systems have enhanced the accuracy and redundancy of measurements, making implementing DSE possible [9,10].

By installing a PMU at a bus, operators have access to synchro phasor data of that bus, such as its voltage phasor and all its branch current phasors, in real time. Hence, to observe a power system dynamically, ideally, PMUs should be installed on all buses. However, today’s power systems are large and consist of a large number of buses, and PMU devices are costly. Therefore, in the present context or even in the near future, it is not practical to replace all traditional measurements such as Remote Terminal Units (RTUs) with PMUs [11]. Thus, in each system, in addition to traditional Supervisory Control and Data Acquisition (SCADA) system measurements, just a few PMU measurements are available [12]. Hence, to improve redundancy to monitor a system near real-time (dynamically) and in an optimum way, it is inevitable to employ multiple measurement sets [13]. In each power system, there are some measurement sets such as PMU, SCADA, intelligent electronic devices (IEDs), and merging units (MUs). However, the most widely used measurement sources are SCADA and PMU [14]. Hence, the focus of this research is on a kind of SE method called HSE, in which both SCADA and PMU measurements are utilized [15].

The contribution of this review study is summarized as follows: it provides a thorough review of existing SE methods in power systems, with an emphasis on HSE approaches that utilize both SCADA and PMU measurements. It discusses various algorithms and techniques that can be used for SE and covers the limitations, challenges, advantages, and disadvantages of each methodology. Then, the areas that require more investigation and improvement are presented.

2. State Estimation

The main goal of SE is to assign a value to unknown system state variables (voltage magnitude and angle of buses) based on available measurements, and it can be mathematically formulated in Equation (1) as follows [16].

z = h(x) + e

(1)

where h(x) is a non-linear function that indicates the relationship between the state variables and measurements, $z=[z_1,z_2,\dots ,{z_m]}^T$ is measurement vector and m is the number of measurements, $x=[x_1,x_2,\dots ,{x_n]}^T$ is a state vector where $x_i=[V_i,{\theta }_i]$, i = 1, 2, …, n, and n is the number of buses, e ~ N(0, R) is the measurement error vector. It is assumed that elements of this vector have random Gaussian distributions with zero mean and covariance matrix R as presented in Equation (2).

$$R=diag({{\sigma }_1}^2,{{\sigma }_2}^2,\dots ,{{\sigma }_m}^2)$$

(2)

where ${{\sigma }_k}^2$ is the variance of the k^th measurements, k = 1, 2, …, m [1].

Power systems comprise two interconnected parts, a physical part and a cyber part. Integrating these parts will result in a cyber-physical system [17] as shown in Figure 1.

Typically, a series of functions support the state estimator, as shown in Figure 1 [1,19]. The first block is to receive data from the meters of the system, which can be dynamic and alter with time or be static, like transmission line parameters.

Any information regarding topology changes and the estimated status is sent to an internal block called the topology processor. Based on the measurements and the network topology, the observability analyzer block is responsible for determining whether the redundancy of measurements is sufficient for full observability or not. Note that a bus that has a PMU installed on it or at one of its adjacent buses is referred to as a PMU observable bus. The measurement pre-filtering block is responsible for removing the measurements that are obviously incorrect. For example, any negative values for voltage magnitudes, values of power flow that exceed the bounds.

The next block is SE, in which, based on the results of previous blocks, the most accurate estimates are calculated. Since SE is vulnerable to bad data (BD), topology, and parameter errors, it needs an additional process to identify them. In this regard, the aim of the next block is to identify BD and errors based on the statistical characteristics of SE.

There has been an increase in research investigating different techniques for estimating the state of the power systems. Built on the research and studies conducted, the SE methods can be divided into two main categories depending on the number of measurement resources they employed: HSE and non-HSE methods. In HSE methods, multiple measurement sets are used to estimate the states of the power system, whereas in non-HSE approaches, only a single measurement source is employed.

Each of these methods is further divided into SSE and DSE approaches, which are, in turn, categorized into centralized and decentralized methods [8]. The proposed categorization of SE methods is shown in Figure 2.

2.1. Static State Estimation

Historically, SE methods deployed traditional measurements that were collected by the SCADA system. These measurements contain power injections of buses, voltage magnitudes, and line flows (active and reactive) with the rate of a sample every 0.5–2 s [20], which is inadequate for recording system dynamics [21]. These conventional methods are called SSE and employ Equation (1) to model the relationship between the state variables of the power system and their corresponding measurements at a time instant and then solve this model, typically using iterative methods.

Many researchers have investigated SSE, and various methods are proposed in the literature. To solve the SSE problem, some robust techniques such as Weighted Least Squares (WLS), Weighted Least Absolute Value (WLAV), and Schweppes–Huber-generalized M-estimator (SHGM) have been proposed. Comparing these methods shows that the WLS method is computationally lighter, has a better performance, and is more practical. Hence, in real power systems, control centers use WLS to estimate the states of the system statically [22,23].

In the following, the WLS formulations are presented. Using Taylor’s expansion and assuming that the system is operating at the point x₀, the measurement in Equation (1) is linearized, and it can be expressed as in Equation (3):

$$z=h\left(x_0\right)+\left(x-x_0\right)\left({\displaystyle \frac{\mathsf{\partial }h\left(x\right)}{\mathsf{\partial }x}}\right)+e\left(x\right)+h.o.t.$$

(3)

where h.o.t. is a short term for higher order terms.

After disregarding the h.o.t. in Equation (3), it can be expressed as Equation (4):

$$\Delta z=H\Delta x+e\left(x\right)$$

(4)

where H is defined as the Jacobian matrix and given by Equation (5):

$$H={\displaystyle \frac{\mathsf{\partial }h\left(x\right)}{\mathsf{\partial }x}}$$

(5)

Hence, the objective function of WLS can be presented by Equation (6) as:

$$\min f={\sum }_{n=1}^{m}W{[z-h\left(x\right)]}^2 or min{[z-h\left(x\right)]}^{T}W[z-h\left(x\right)]$$

(6)

where W = R⁻¹ is the matrix of measurement weight, R is the measurement covariance matrix, and G is the gain matrix as defined in Equation (7) as follows:

$$G=H^{T}WH$$

(7)

In full observability of the system, $G$ is positive definite and always non-singular.

Since the relationship between system states and the measurements is non-linear, iterative methods like the Newton–Raphson are applied to estimate the state of the system, as expressed in Equations (8) and (9) [7].

$$\Delta x=G^{-1}{H}^{T}W[z-h\left(x\right)]$$

(8)

$$x^{k+1}=x^k+\Delta x^{k+1}$$

(9)

where k represents the number of iterations.

However, the SSE method is not able to predict the state of the system across successive time intervals and needs to perform a new estimation for each time instant using a new set of measurements without any pre-existing knowledge about the previously estimated states [24]. Performing the entire procedure for each time instant is computationally intensive and thus time-consuming. These are the main disadvantages of SSE [25,26].

Furthermore, the main assumption in SSE is that the system is working in its quasi-steady state mode and changing very slowly. However, in a power system, loads and generators are continuously changing, which means that the state of the system is not steady [8,27]. Furthermore, widespread integration of distributed energy resources (DERs) into power networks on the generator side, complex loads, and advancements in demand response devices on the demand side, including Internet of Things (IoT) technology, make the situation even worse and increase systems’ uncertainties. As a result, the assumption of working in a quasi-steady state becomes controversial, and SSE techniques are unable to adequately represent these dynamics in a real-world system [8]. Hence, it is imperative to improve SSE techniques currently employed in practice in EMS [1,3,4,5] by utilizing new monitoring methods like DSE.

2.2. Dynamic State Estimation

The advent of smart meters like PMUs leads to a revolution in SE. PMUs can measure voltage and current phasors, which, in comparison with SCADA measurements, are synchronized, more accurate, and have a higher resolution (sampling rate). Their high sampling rate (60–120 samples per second), enables operators to capture state variables in real-time and track the system dynamically [28]. Researchers have employed PMU measurements as a powerful tool to address the drawbacks of SSE and introduced DSE, which is also known as forecasted-aided state estimation (FASE).

A mathematical model of DSE is presented in Equation (10) as follows [29]:

$$x_{k+1}=F_k{x}_k+g_k+q_k$$

(10)

where x, g, q are state, input and the process noise vector, respectively, F is the matrix of state transition and k indicates the time instant.

The elements F and g can be calculated as presented in Equations (11) and (12).

$$F_k={\alpha }_k\left(1+{\beta }_k\right)I$$

(11)

$$g_k=\left(1+{\beta }_k\right)\left(1-{\alpha }_k\right)x_k^{-}-{\beta }_k{a}_{k-1}+\left(1-{\beta }_k\right)b_{k-1}$$

(12)

where ${\alpha }_k$ and ${\beta }_k$ are two scalers in the interval (0, 1), I is the identity matrix, and its size is the same as the state vector size. $x_k^{-}$ is estimated states at time instant k. a and b are two parameter vectors, which are expressed at time k in Equations (13) and (14).

$$a_k={\alpha }_k{x}_k+(1-{\alpha }_k)x_k^{-}$$

(13)

$$b_k={\beta }_k\left(a_k-a_{k-1}\right)+(1-{\beta }_k)b_{k-1}$$

(14)

DSE can predict the state of system in time instant k and k + 1 using two sets of data, measurements and predicted data, from the former time instant [30]. It can predict the state of the system one time instant ahead, which gives more time for operators to analyze the system in emergency situations. Similarly, it helps to identify anomalies such as topology errors and sudden changes. These properties make it an important tool for today’s modern EMS [31].

In DSE methods, under the assumption of full observability of the system by PMUs, either electrotechnical parameters, such as rotor angle and generator speed, are estimated [30,32,33,34,35,36,37,38,39,40,41], or both dynamic and static states are estimated in sequence [42,43]. To solve the DSE problem, two types of methods are applied: model-based [30,32,33,34,35,38,39,41,42,43] and data-driven approaches [36,37,40]. In the model-based approaches, at first, with the help of DSEs, the system is mathematically simulated, and then by applying recursive filters like Kalman filters (KF) [44], least squares [45], particle filters [46], etc., the state of variables is estimated. For instance, an Extended Kalman Filter (EKF)-based approach is proposed in [47] to solve the reliability problem of the medium voltage DC integrated power system (MIPS) under the circumstances of sudden pulse load changes. However, the EKF methodology is not reliable under the strong nonlinearity of the power system. To address this issue, authors in [48] developed an Unscented Kalman Filter (UKF)-based approach. However, due to the observability problem, this method is unable to estimate a large number of parameters. The most widely used method in systems with large state vectors is the Cubature Kalman Filter (CKF), as it is both accurate and stable in such cases [49]. However, CKF-based methods are unable to handle non-Gaussian and outliers. A DSE methodology based on the CKF and the L_p norm estimator is proposed in [50], which is able to handle the effect of non-Gaussian noise (see also Section 3 for further discussion on KF performance). Under some assumptions, these methods have acceptable operations and results. First, it is assumed that the system operation and observation noise have zero means, and their covariance matrices are known at all instants. The second assumption is that they are following Gaussian distribution, and the last one is knowing the model of the system accurately [8]. Without either of these assumptions, the state estimator may lead to inaccurate estimation or even not converge [8,42]. In data-driven methods, the SE problem is addressed using Artificial Intelligence (AI), e.g., Neural Networks (NN). These trainable methods can describe the non-linear performance of the system by learning from the training dataset without considering the mathematical model of the system. Much research has investigated these model-free methods, and the results have demonstrated that they are flexible, offer superior performance, and have faster computational times compared to model-based approaches. However, to have acceptable performance, the training dataset must involve all aspects of transient and steady states of the system under investigation. This dependency is the main drawback of data-driven methods and may result in erroneous results in some cases.

To monitor the entire system and implement the DSE method, two different techniques, namely centralized and decentralized DSE, are employed. These techniques are described in the following subsections.

2.2.1. Centralized DSE

Several centralized SE techniques are available in the literature that provide accurate results regardless of the size or topology of the system [51,52,53,54,55,56].

The basic concept underlying this technique is that the measurements from all meters (e.g., RTUs) are sent to a center to be processed; hence, the redundancy of measurements is high, and they are robust to security and data quality issues [57]. However, in centralized methods, a large amount of data from the entire system needs to be analyzed, which results in a considerable computational burden, especially in the case of large-scale power systems. Furthermore, to transfer this huge amount of data, they require a communication system with minimum latency, which is not available in some power systems [58].

Centralized DSE methods are based on two main assumptions: first, the system is fully observable by PMUs, and second, Kron reduction is applicable to simplify the system to the terminals of its generators. In these methods, it is imperative to have real-time PMU measurements as well as accurate knowledge about the parameters of all system components. However, it is not possible to extract a dynamic model that exactly displays the whole system, and every model has some inaccuracies, which results in inaccurate predictions [8]. The following

Table 1. Centralized DSE challenges.

Category	Challenges	References
Centralized DSE	- It is prone to communication problems and single points of failure. - It might be difficult to handle massive amounts of data from sensors and measurement devices. - It is vulnerable to cyber attacks.	[1,59,60,61,62]

represents the summary of centralized DSE challenges.

2.2.2. Decentralized DSE

To address the challenges of centralized techniques, researchers have proposed decentralized SE as an alternative solution [63].

In decentralized DSE, it has been assumed that the system is divided into several sub-areas based on different characteristics such as geographic location, communication resources, or operational resemblance properties [57]. In each sub-area, decentralized DSE can be executed by employing measurements of that area, and the states are estimated in all areas simultaneously. In decentralized DSE, it is imperative that areas have the minimum data exchange rate with each other at their borders, i.e., the solution of decentralized and centralized SE will be the same. Accordingly, one way to classify decentralized DSE methods is based on their overlapping degrees, as described below [7].

• None-overlapping areas.

In this case, none of the buses or branches are in common, and areas are just connected to each other by tie-lines, as shown in Figure 3.

2.

Border-bus overlapping areas.

In this scenario, adjacent areas do not have any tie-lines linking them, and areas overlap across a single layer of border buses, as represented in Figure 4. The prior case can be used to artificially generate this scenario by constructing a virtual border bus at each tie-line mid-point and then extending each area up to the virtual bus (see Figure 5).

3.

Tie-line overlapping areas.

In this situation, as shown in Figure 6, tie-lines as well as border buses are shared.

4.

Deep or extended overlapping areas.

Where adjacent zones share multiple levels of border buses (see Figure 7).

Generally, the approach that is being used will determine how much data must be shared. In this regard, a comprehensive review of decentralized approaches considering some indices, such as the rate of convergency and data exchange, has been completed in [64], which strongly demonstrates that methods are different based on the amount of exchanged data.

Decentralized DSE techniques can also be categorized based on whether areas are fully distributed or under the supervision of a central controller. Hence, these methods can be divided into two groups: hierarchical decentralized DSE techniques that have a central controller [65,66,67] or are fully distributed so that adjacent areas may exchange data [64,68,69], as shown in Figure 8. Both of these techniques lead to acceptable results, for example, the network is divided into areas by choosing buses randomly, and a fully distributed SE approach is proposed in [70] based on the network gossiping technique. Nevertheless, comparing the results with the centralized DSE indicates a noticeable error. In [71], the power system is partitioned into sub-areas based on the number of State Load Dispatch Centers (SLDCs) and their boundaries. In this decentralized approach, the estimated states of all sub-areas are sent to the central coordinator, who works as an angle referencing. The proposed method in [72] is a hierarchical decentralized DSE in which sensitivity functions of all estimators are shared instead of the estimated states or boundaries information.

It is obvious that executing the dynamic model of a sub-area is faster than modeling the entire system; furthermore, any inaccuracy in other parts of the system does not affect the dynamic model of the understudied area. Hence, to implement DSE for bulk power systems, it is more reasonable to apply decentralized DSE. However, the low redundancy of local measurements is one of the disadvantages of decentralized methods, which makes it difficult to deal with the measurements’ quality and security problems [8]. The challenges of implementing decentralized DSE for a power system are shown in

Table 2. Decentralized DSE challenges.

Category	Challenges	References
Decentralized DSE	- To have reliable estimation, it is important to employ an accurate and fast SE algorithm. - It is imperative to control the amount of data exchange between sub-areas to computational burdens and communication resources.	[60,64,69,73,74]

.

2.3. Hybrid State Estimation

Due to the high price of PMUs, it is not economically practical to install a PMU on each bus of today’s large-scale power systems, and just a few numbers of PMUs are available.

Nevertheless, different sampling rates of these sets are a significant challenge of hybrid methods [75]. Because of the higher sampling rate of PMUs, between each two successive SCADA measurements, for example, time instances of t and t + 1, there are some PMU measurements, as is obvious in Figure 9. Although the system is observable at specific time instances t and t + 1, due to the limited number of PMUs in the system, there may be periods between t and t + 1 when the system is not fully observable.

HSE methods can be categorized into two groups: static HSE (SHSE) and dynamic HSE (DHSE) [71,72]. SHSE methods assume that the system state does not change significantly over the period of analysis, making them suitable for steady-state scenarios. In contrast, DHSE methods account for changes in the system state over time, making them ideal for systems experiencing fluctuations or transient conditions. The details of both dynamic and static HSE methods are described in the following subsections.

2.3.1. Dynamic Hybrid State Estimation

In the literature, numerous DHSEs have been proposed, which generally are composed of three categories: sequential measurement-state fusion (two-stage SE), parallel state fusion, and direct measurement fusion (one-stage SE) [14]. The details are described in the following subsections.

• Sequential measurement-state fusion.

These HSE methods, also known as two-stage SE, involve two stages where a traditional state estimator in one stage is combined with a linear one in sequence. As shown in Figure 10, there are two different estimator orders to process measurements. In the first sequence of processing, in the first stage, SCADA measurements are processed, then the estimated states are transformed into rectangular coordinates to be compatible with the linear estimator, and the second stage is to process traditional SE output as well as PMU measurements using a linear estimator [7,13,21,76,77,78,79,80,81,82,83,84]. In [16,85], the process is carried out in a reversed sequence, where using a linear estimator, PMU measurements are handled in the first stage, and then the results are combined with the SCADA measurements to obtain the final results by employing a non-linear estimator. In these references, to solve the unobservability problem in the first stage, only state variables in PMU buses or their adjacent buses are estimated.

For instance, to estimate the state variables of the power system under PMU phase errors, authors in [86] proposed a distributed two-stage HSE method. In this approach, the power system is divided into sub-areas, each with its own estimator, while the current SCADA system remains intact. In [87], a physics-embedded data-driven SE is proposed to estimate the states of the power system in real time. In this method, the purpose of the physics part is to generate node features by employing both SCADA and PMU measurements, while the data-driven part is responsible for estimating the state variables of the system by utilizing the multi-head graph attention (MGAT) algorithm. In [85], a linear estimator is employed in the first stage, while in the second stage, an iterative estimator is utilized to estimate the states of the system based on both conventional measurements and pseudo-measurements derived from the first stage. In [88], the procedure of SE and the data collection is carried out in parallel to accelerate the estimation process of large-scale power systems based on EKF. A two-stage HSE is proposed in [78], where in the presence of both SCADA and PMU measurements, a non-linear SE method is applied, and when just PMU measurements are available, a linear SE method is used.

Utilizing the existing traditional SE methods is one clear benefit of two-stage approaches. This implies that they should be less complicated and costly compared to the estimators needed for one-stage HSE methods since they apply LSE for the post-processing stage. Additionally, it is not imperative to have full observability of the network from PMU devices. Conversely, the major drawback of the two-stage method is that the system has to be completely observable from SCADA measurements (RTUs). Each power system has enough RTUs available, making it completely observable with a considerable level of redundancy. Nevertheless, due to RTU aging or failure over time, the system could not be fully observable. In such a scenario, PMUs are unable to compensate for the lack of RTUs since the traditional estimator in the first stage is incompatible with PMU measurements and cannot utilize them directly. Consequently, the best solution under this circumstance is to substitute obsolete RTUs with new ones to protect the HSE operation.

2.

Parallel state fusion.

In parallel fusion methods, two estimators are working in parallel simultaneously to independently evaluate PMU and SCADA measurements, and incorporating their results leads to the final SE [89,90,91,92,93], as shown in Figure 11. For example, in [91] a parallel SE algorithm is proposed, taking into account the statistics of unknown measurement noise. In this method, PMU measurements are buffered to address the issue of varied sampling rates.

Since the number of PMUs in a power system is limited, the linear estimator may encounter unobservability problems, which is the main drawback of parallel methods. Some techniques, such as utilizing data obtained from SCADA estimator in previous time instances [89] or using pseudo-measurements [90], are proposed to solve this problem. However, SCADA measurements are not accurate, which potentially results in less accurate SE, similarly, pseudo-measurements may not always be guaranteed, and inaccurate pseudo-data can lead to erroneous SE. In [94], a parallel method based on the KF is proposed, which employs multi-rate data to predict the state of the power system. However, this standard KF is accurate only in linear systems, making it unsuitable for use in real power systems. To address this issue, the authors in [95] employed the UKF, which is capable of accounting for the non-linear behavior of the power system. The main assumption of this method is that loads are constant impedance. However, this load modeling is unrealistic, and in real power systems, loads are rarely constant impedance.

3.

Direct measurement fusion.

These HSE methods are also known as integrated or one-stage SE since they have only one stage where both SCADA and PMU measurements are incorporated with each other and make one dataset [51,96], as shown in Figure 12.

The primary benefit of one-stage SE over two previous methods is that it requires less execution time due to its single processing stage and the contribution of both PMU and SCADA measurements to keep the system fully observable. Consequently, despite two-stage methods, it will not encounter any unobservability issues in the long term. Nevertheless, it is not possible to use previous estimators for one stage SE, and new SE methods need to be created. These developed estimators will be more complicated and may be more costly than other hybrid methods. Much research has been conducted to develop direct measurement fusion methods and integrate PMU and SCADA measurements, which can be categorized into model-based and data-based approaches.

In the model-based methods, models of the system are employed to determine how future states are related to past states [97,98,99,100,101,102,103]. For instance, authors in [104,105] utilized the model of the system, and voltage phasor measurements were incorporated into the SE for the first time. The basic idea of these methods is to set the Jacobian matrix elements that are related to the voltage phasor data to 1. Conversely, PMUs measure both voltage and current phasors; hence, they employed the minimum data to build the Jacobian matrix. To solve this problem, current and voltage phasors are both incorporated into the traditional state estimator and developed the Jacobian matrix in [28,106]. To tackle the numerical issue, especially undefined elements in the Jacobian matrix during flat start initialization, phasor measurements transformed from polar to rectangular coordinates. However, this transformation magnifies PMU measurement errors, which leads to erroneous SE. To address this problem and minimize any errors in transformation, the authors proposed applying rectangular coordinates for the problematic iteration [51] or only for the initial iteration [107], where polar coordinates are used for the other iterations. The drawback of these techniques is that they were designed for SCADA sampling rates and were not appropriate for dynamic systems.

An integrated HSE method, which is based on a static estimator, is introduced in [98], where the states of unobservable buses are calculated using PMU measurements and an interpolation matrix, which is formed from system admittances. This interpolation matrix is updated by assigning differential weights to each measurement whenever the SCADA measurement set comes in, or significant changes occur in phasor measurements. Nevertheless, these weights are determined by how far each measurement is from the fault, which is unreasonable, particularly when multiple faults occur. Hence, the authors in [100] introduced an adaptive weight function to adjust each measurement weight dynamically. However, the main assumption in constructing the interpolation matrix is that loads are static while most loads represent dynamic behavior in power systems. To compensate for the low sampling rate of SCADA measurements, distributed Compressive Sensing (CS) is used in [101] to take advantage of PMU measurements’ spatial–temporal correlation. By employing distributed CS, SCADA measurements are rebuilt in time instances when they are unavailable. To maintain an observable power system at the interval between two successive SCADA measurements, authors in [102] allocate pseudo-measurements optimally within these gaps. Then, by employing a very short-term load forecasting technique based on KF, pseudo-measurements are anticipated, which are used as inputs of WLS along with PMU measurements to estimate the states of the system.

In order to estimate the status of variables in distribution systems, a mixed measurement set made up of micro PMU and RTU measurements is employed in [103]. The suggested estimator is based on the CKF, and this article proposed the Optimally Weighted Average (OWA) method to interpolate asynchronous data to solve the asynchronous measurement problem. In the work documented in [108], the authors employed the EKF algorithm and introduced a new integrated SE method. Although KF is a widely used tool in solving SE problems, it encounters some weaknesses. The standard KF is accurate only in linear systems and cannot be used directly in non-linear systems such as power systems. Hence, non-linear modifications of KF must be employed in non-linear systems. In most of these modified KFs, the Taylor series is executed to create a linearized system from a non-linear one in a limited area around the operating point. Consequently, the filter’s effectiveness is limited to a small region surrounding the operating point, and its accuracy declines when dealing with a broad operating range, resulting in significant errors in outputs [109].

Since these methods are based on the systems model, any inaccuracy or unavailability in the physical model can affect the estimation results. Hence, data-based methods are introduced to extract historical measurements more precisely to make an accurate forecasting [110,111,112]. Despite model-based approaches that require system information, data-based methods rely on data to predict hidden (unknown) states of the system. For example, using both PMU and SCADA measurements, a Linear Regression (LR) model is trained in [113] to predict the hidden states. Although LR models are straightforward and understandable, LR-based methods have limited capacity. To address this problem and expand the capacity, improvements can be made by either (1) enhancing the non-linearity of the single model to have a more accurate estimation or (2) by combining different models. For instance, authors in [114] integrated several different LRs and proposed a Bagged Averaging of Multiple Linear Regression model. Nevertheless, it is challenging for these methods to fully capture the complicated temporal and spatial correlations of data found in SCADA and PMU measurements.

To predict hidden states in the presence of complicated spatial–temporal correlations effectively, deep learning techniques are proposed [115,116,117]. These techniques enable learning complicated data patterns by artificial neural networks and capture spatial and/or temporal features through feature learning. For example, authors in [115] captured temporal correlation of data by applying Recurrent Neural networks (RNNs), which are able to learn data patterns by creating a memory and predicting each state by considering its past inputs and outputs and in [116], Convolutional Neural Networks (CNNs) employed to obtain a spatial correlation of data using sliding windows. However, these methods require comparative big datasets with a variety of information to be able to predict the state of the system accurately. To address this issue, authors in [117] proposed a new Deep Neural Network (DNN), which is combined with an expectation maximization method. This heterogeneous data deep expectation maximization method (Hd-Deep-EM) can also consider temporal–spatial correlations between SCADA and PMU measurements. This iterative algorithm retrains the network to solve the problem of a limited number of SCADA measurements for the training step and handles the need for large datasets for the training step. To evaluate the method, it is tested on 200- and 500-bus systems and the results are compared with a feed-forward DNN. The results represent that in comparison with the feed-forward DNN, the proposed Hd-Deep-EM algorithm has an average testing MSE reduction of 0.01 and 0.02 for 200 and 500 bus systems, respectively, and captures the spatial–temporal correlation more effectively.

2.3.2. Static Hybrid State Estimation

Most of the hybrid methods focus on static estimation of variables, in which the fact of the high sampling rate of PMUs is neglected, and the state of variables is estimated based on the update rate of SCADA measurements [16,118,119]. Similar to the DHSE methods, SHSE approaches can be divided into three categories: two-stage SE, parallel state fusion, and one-stage SE.

In two-stage static estimation methods, similar to dynamic methodologies, two estimators are working in a cascade architecture [120,121], as shown in Figure 10. For instance, in [15] a two-stage HSE is introduced, in which a traditional SE based on SCADA measurements is executed in the first stage, and then in the second stage, an LSE method based on both PMU measurements and the outputs of the first stage is applied. In [122], a 2 × 2 block version of fast Givens rotations is introduced to handle the assigned weights for the measurements. In this approach, traditional and linear estimators are working in sequence.

Like the previous category, there are two estimators in parallel SHSE approaches that work in parallel, as illustrated in Figure 11. For example, authors in [123] proposed an improved parallel SHSE methodology based on Hachtel’s augmented matrix and reduced the requirement of parallel methods for pseudo-measurements.

As shown in Figure 12, in one-stage static estimation methods, both SCADA and PMU measurements are processed by one estimator [106,124,125]. For instance, the authors in [126] proposed a one-stage SE method where both SCADA and PMU measurements are directly employed in the SE process directly without any transformation. Since data transformation can increase errors, the proposed algorithm prevents the propagation of uncertainties. The work documented in [127] employed a linear WLS framework to estimate the state variables of the power system. Since the proposed methodology is not iterative, traditional BD detection approaches can be employed as post-processing approaches. Authors in [128], employed SCADA measurements and a limited number of PMU measurements for the purpose of SE. In this method, instead of the Jacobian matrix, current phasors are updated in each iteration based on the previously estimated states.

However, as previously stated, since static estimators are not capable of capturing the dynamic changes in the system, they are not suitable for the analysis of the power system dynamically [129]. Therefore, DHSE, which is able to track the dynamic behavior of the power system, has received a lot of attention [130,131].

3. Conclusions and Future Research Prospects

In this paper, a comprehensive review of power systems SE that covers different SE problems, such as SSE, DSE, and HSE, is presented, and current approaches are properly classified and examined based on the main challenges resolved. Some other concepts, such as centralized DSE and decentralized DSE, are also investigated in detail. Despite various research work in this field, there are still significant gaps that need further investigation, which are suggested as follows.

Although data-driven SE approaches have achieved acceptable results, their predictions may be physically impossible or inconsistent due to extrapolation or biases in observations. In other words, any biases, gaps, or errors in the training data, or any extrapolation outside the scope of observed data, can result in impractical outputs. To address this and design an accurate SE method that can estimate the state variables of the power system as close as possible to their true values, it is imperative to consider the topology of the system and integrate underlying physical rules into data-driven models. By incorporating physical rules, such as power flow equations, states will be estimated based not only on data but also on the physical aspects of the system, leading to reduced errors and improved reliability of the estimated states.

In power systems, analog-to-digital converters (ADCs) are used to transform continuous measurement signals measured by PMUs into digital formats that can be examined and processed for the purpose of SE. However, the stability of the reference voltage, which ADCs rely on, has a significant effect on their performance. If the reference voltage is not stable, the quality of the ADCs’ output can degrade, leading to inaccurate or noisy data. Therefore, considering this factor in SE models can enhance the accuracy of the estimation results. Consequently, to improve the stability and accuracy of SE, it is essential to integrate the performance and stability of ADCs, specifically, the characteristics of their reference voltage, into the SE process.

As previously mentioned, due to the high cost of PMUs, only a limited number are available in each power system. This shortage is even more critical in distribution systems, which have a lower priority for monitoring and control compared to transmission systems. Hence, to increase redundancy, it is imperative to integrate other data sources, such as smart inverter technology, which is increasingly used in renewable energy resources. For instance, rooftop solar systems can report data every five minutes, including power and AC voltage. Furthermore, employing new methods to generate pseudo-measurements can enhance SE accuracy. This data can be obtained from advanced metering infrastructure (AMI), consumer billing data, or calculated based on daily load profiles.