Leveraging Behavioral Correlation in Distribution System State Estimation for the Recognition of Critical System States

Abstract

State estimation for distribution systems faces the challenge of dealing with limited real-time measurements and historical data. This work describes a Bayesian state estimation approach tailored for practical implementation in different data availability scenarios, especially when both real-time and historical data are scarce. The approach leverages statistical correlations of the state variables from a twofold origin: (1) from the physical coupling through the grid and (2) from similar behavioral patterns of customers. We show how these correlations can be parameterized, especially when no historical time series data are available, and that accounting for these correlations yields substantial accuracy gains for state estimation and for the recognition of critical system states, i.e., states with voltage or current limit violations. In a case study, the approach is tested in a realistic European-type, medium-voltage grid. The method accurately recognizes critical system states with an aggregated true positive rate of 98%. Compared to widely used approaches that do not consider these correlations, the number of undetected true critical cases can be reduced by a factor of up to 9. Particularly in the case where no historical smart meter time series data is available, the recognition accuracy of critical system states is nearly as high as with full smart meter coverage.

1. Introduction

The accelerated addition of distributed energy resources (DERs) such as electric vehicle charging infrastructure, heat pumps, and photovoltaic (PV) installations drives distribution systems to their limits [1]. With a high share of DERs, these systems are expected to frequently be in critical states where branch capacity is limited or voltage limits are exceeded. An important aspect of operating these systems is state estimation, as it provides knowledge of the grid state and enables the operator to make informed decisions about how to control the system and identify faults and critical system states.

Unlike the transmission level of a power system, the primary distribution (at medium voltage (MV)) and, more so, the secondary distribution have a low coverage with real-time measurements [2,3,4]. This poses a challenge for state estimation, which relies on real-time measurements to process them into accurate estimates of the grid state. An approach to meet this challenge is to include additional data sources that, while not available in real-time, provide more accurate characterizations of past and expected system states. These data sources might include, e.g.,:

• Time-resolved Smart-Meter (SM) measurements from customers in the low voltage (LV) grids [5];
• Time-resolved power recordings (RLM), which are mandatory for large customers with annual energy demands larger than 100 MWh (typically directly connected to MV nodes) [6];
• Annual energy demand data available through the billing system from analog or digital meters from customers without time-resolved measurements;
• Standard load profiles (SLP), representing the average behavior of a consumer group [7];
• Data from exogenous sources like solar irradiance or wind velocity measurements to estimate generation from photovoltaic (PV) or wind turbines.

An approach particularly suited to process data from different sources and at such different levels of detail is Bayesian state estimation [8,9,10,11,12,13,14,15]. Before processing real-time measurements, Bayesian state estimation models any prior knowledge of the system state using a prior probability distribution. The above data sources are then used to accurately parameterize this distribution, characterizing the expected values of the state variables, such as voltages and currents, and their covariance. As real-time measurements are obtained, the prior distribution is updated using the Bayes rule to incorporate the new information.

In applying Bayesian probability theory, it is important to note that state variables are not statistically independent. If, for instance, the voltage magnitude at one node increases, the voltage at the neighboring nodes also increases. These correlations are valuable for state estimation because they allow one to use data from one node to infer information about other nodes, particularly unmeasured ones. Two origins of these correlations can be distinguished:

• Physical correlations are caused by the grid’s physical coupling, for example, that a high voltage at one grid node is transmitted through a line to another node.
• Behavioral correlations or load correlations are caused by similar (correlated) behavior of grid customers independent from the grid, i.e., even for electrically unconnected customers: As an example, if it is a warm day, electricity consumption for air conditioning will be higher than the historical average for all households at the same time, causing the voltage to drop in different grid segments, which might not even be physically connected.

Properly accounting for these correlations in the context of state estimation can drastically improve the accuracy of the results, as will be shown in this paper. In the Bayesian framework, behavioral correlations are encoded in a prior probability distribution for the customers’ loads—the load probability distribution (LPD), specifically in the covariance matrix of the LPD. Physical correlations are explicitly modeled using a power system model to obtain the prior probability distribution for the state variables [8].

Distribution system state estimation (DSSE) methods that are based on the Weighted-Least-Square (WLS) approach [16], which is the state-of-the-art method for higher voltage levels, compensate for the lack of real-time data by processing the above-mentioned data sources about expected states into so-called pseudo-measurements [17,18]. These pseudo-measurements are then treated in the same way as the real-time measurements, but by giving them lower weights, one ensures that they have a smaller influence on the estimation result. WLS approaches assume these pseudo-measurements to be uncorrelated, like the real-time measurements. As a consequence, they do not take into account behavioral correlations as described above. Papers that adjust the WLS algorithm to consider load correlations [19] show that the consideration improves the state estimation results. A complementary set of works [20,21,22,23] has developed optimal planning and control approaches for energy systems in the presence of uncertainty.

In [8], the basic principle for a Bayesian Linear DSSE is demonstrated. The parameters for the LPD are taken from historical SM time series, and loads are assumed to be uncorrelated. There is also a proposal for three-phase Bayesian DSSE [9], in which the authors use a machine learning model to get forecast data for the background LPD distribution, and the authors in [11] used probabilistic graph models for state estimation. They do not use behavioral correlation information. The simplification of zero correlations in [8] is addressed in [12], where the authors use correlation information between loads calculated from historical data assuming full SM coverage. In [13], the correlation coefficients for active power are sampled randomly, while a full correlation between active and reactive power is assumed. As in [12], the authors assume full meter coverage. Another approach combines deep learning with the Bayes rule using grids with full SM coverage to learn the LPD [14]. In [15], the authors explicitly investigated correlation coefficients between different types of loads for a historical data set. Their focus is on the impact of the correlation coefficients on the standard deviations of state variables. A more recently proposed Bayesian approach [10] combines varying input sources with different time resolutions and considers load correlations. In particular, it considers real-time measurements, smart meters, and historical load data, which are at least 30 min time-resolved.

All the above approaches that consider LPD correlations have in common that they assume complete availability of historical time-resolved power measurements (e.g., Smart Meter data or recorded power measurements) across the entire grid to parameterize correlation coefficients.

However, only a few European countries or US states have 100%-SM coverage [24,25,26]. While with the Third Energy Package (European Union, 2009), European member states are required to implement SMs in the future, currently, a mixed situation is found. Sweden, Norway, Spain, and Italy have a high SM coverage of over 97%. Conversely, Poland, Hungary, and Slovakia have a coverage smaller than 10% [24,25]. Similarly, in the US, the SM coverage for Utah and New Mexico is below 20% while California, Nevada, Georgia, and Maine achieve high coverage rates over 80% [26].

In summary, for both US- and European grids, only a few real-time measurements are available, and only a few countries have 100% SM coverage. In most countries, there is a substantial share of customers for which only annual energy consumption data are available. While this is sufficient to characterize expectation values for load profiles, data are lacking to parameterize the correlation between load profiles accurately. This paper shows that accounting for behavioral correlations in the load profiles improves the state estimation accuracy (confirming findings for WLS-based approaches [19] also for Bayesian state estimation) and how to relax the assumption of complete SM coverage. A method is developed to synthesize LPDs from mixed sources of input data while adequately accounting for correlations between different loads, regardless of the available type of load measurements. This paper focuses on MV grids, where most nodes represent the aggregated load of multiple customers. The topology and line parameters of the grid are assumed to be known.

The main methodological contributions of this paper are as follows:

• We highlight the role of behavioral correlations and provide a deeper understanding of their effects on state estimation accuracy (Section 2);
• We develop a flexible approach to derive background probability information for parts of the network with different levels of smart meter coverage (Section 3.2);
• We present a method to estimate covariance data for the derived background probability information and validate this approach by a statistical analysis of load correlations between LV grids (Section 3.3 and Section 3.4);
• We demonstrate the accuracy improvement in state estimation and detecting critical system states that can be achieved by accounting for load correlation in Bayesian state estimation in general and by using the approach presented in this article in particular.

Our contributions extend the well-established Bayesian state estimation framework by a method to parameterize load probability distributions in different, practically relevant levels of data availability ranging from complete time-resolved measurement sets to yearly power consumption data only, rendering unnecessary the constraint of having a full set of time-resolved power measurements for all customers. The application of the presented method is demonstrated in a case study using the example of Bayesian DSSE, but the synthesis of accurate load profile correlations can also be applied to WLS-based methods. The application of the presented method is demonstrated in a case study using the example of Bayesian DSSE, but the synthesis of accurate load profile correlations can also be applied to WLS-based methods.

In Section 2, the modeling of the grid and system states is outlined, followed by a brief introduction of both WLS and Bayes approaches, focusing on how these approaches treat load correlations. Subsequently, the principle of the Bayesian linear state estimator is outlined. Section 3 starts with the proposal of the Correlation-Aware LPD Synthesis Method and details its integration into a state estimation framework. In Section 4, the accuracy gains of considering behavioral correlations for recognizing critical system states are demonstrated for a 107 bus, 20 kV MV grid under different measurement instrumentation scenarios.

2. State Estimation

2.1. Modeling Power Grids and System States

The distribution grid is modeled with $N+1$ nodes, N non-slack buses marked with index “$\mathrm{ns}$”, and one slack bus with index “0”, and B branches, including lines and transformers. The vector of system state variables is denoted by x. Throughout the remainder of the paper, matrices and vectors are printed in bold to distinguish them from scalar values. The complex node voltage $\mathit{V}\in {\mathbb{C}}^{N+1}$ are used as state variables. It is distinguished between true and estimated state variables, $x_{\mathrm{true}}$ and $\widehat{x}$, respectively. Complex-valued variables can alternatively be represented in cartesian form (real and imaginary part of the variable: $x_{\mathrm{re}}$, $x_{\mathrm{im}}$) or in polar form (magnitude and phase: $x_{\mathrm{mag}}$, $x_{\mathrm{ang}}$). The asterisk ${}^{*}$ is used to denote complex conjugation. The transpose operation of matrix $\mathit{A}$ is denoted by ${\mathit{A}}^{\mathrm{T}}$ and the Hermitian conjugation by ${\mathit{A}}^{\mathrm{H}}={\mathit{A}}^{\mathrm{T}*}$.

2.2. Bayesian and WLS Approaches

State estimation generally describes an approach to processing raw measurement data and topology information to an estimate of the current state of a power system. The process can consist of a number of steps linked with each other, including grid topology processing, observability analysis, a state estimation algorithm, and bad data detection [27]. In this paper, we focus on the role of (load) correlations when using non-real-time data for state estimation in settings with low availability of real-time measurements. While all of the above steps can play a role in addressing this challenge, Bayesian and WLS-based state estimation approaches deal with the issue of non-real-time data in a fundamentally different way, which has implications for how load correlations can be handled. To highlight these differences and set the framework for the load probability distribution synthesis method presented in Section 3, we briefly re-visit the basic algorithm approaches of Bayesian state estimation below and put it into the context with WLS-based approaches. For this, the following part focuses on the state estimation algorithm, assuming the grid topology and measurement data are already processed and available (Large measurement errors are not considered in this paper. Hence, no bad data are included in the state estimation method).

For most state estimation tasks, firstly, a measurement model is assumed mapping the system state. The conventional measurement model is given by [3]

$$\begin{array}{c}\hfill \mathit{z}=\mathit{h}\left(\mathit{x}\right)+\mathit{e}\end{array}$$

(1)

where $\mathit{z}$ is an M-dimensional measurement vector, $\mathit{h}$ is the measurement function, and $\mathit{e}$ is the measurement error. The measurement errors are assumed to be independent and normally distributed with mean vector ${\mathit{\mu }}_{\mathrm{e}}=0$ and standard deviation vector ${\mathit{\sigma }}_{\mathrm{e}}$.

In a Bayesian framework, according to Bayes theorem, the conditional probability of x given evidence z is

$$\begin{array}{cc}\hfill p\left(\mathit{x}\mid \mathit{z}\right)& =\frac{p\left(\mathit{z}\mid \mathit{x}\right)p\left(\mathit{x}\right)}{p\left(\mathit{z}\right)}.\hfill \end{array}$$

(2)

The left side of the equation ($p\left(\mathit{x}\mid \mathit{z}\right)$) is called posterior probability. $p(\mathit{z}\mid \mathit{x})$ is the likelihood probability for $\mathit{z}$ given $\mathit{x}$, $p\left(\mathit{x}\right)$ is the prior probability, and $p\left(\mathit{z}\right)$ is the measurement probability [28]. The Bayes theorem can be stated more explicitly using probability density functions for states and measurements. Let ${\Phi }_a$ denote the probability density of the (multivariate) Gaussian distribution of a D-dimensional random variable $\mathit{a}$ with expectation value ${\mathit{\mu }}_a$ and covariance matrix ${\mathit{\Sigma }}_a$:

$$\begin{array}{c}\hfill {\Phi }_a=\mathcal{N}\left(\mathit{a}\mid {\mathit{\mu }}_a,{\mathit{\Sigma }}_a\right)=\frac{1}{{\left(2\pi \right)}^{\frac{D}{2}}{\left|{\mathit{\Sigma }}_a\right|}^{\frac{1}{2}}}exp\left\{-\frac{1}{2}{\left(\mathit{a}-{\mathit{\mu }}_a\right)}^{\mathrm{T}}{\mathit{\Sigma }}_a^{-1}\left(\mathit{a}-{\mathit{\mu }}_a\right)\right\}\end{array}$$

(3)

We assume that both the prior distribution of voltage phasors ${\Phi }_{V,\mathrm{pr}}$ (which represents our belief about the system state based on information available prior to receiving measurements, see Section 2.3 for a more detailed explanation) and the likelihood distribution ${\Phi }_{z\mid x}$ (which represents the distribution of measurement errors) are normally distributed as

$${\Phi }_{V,\mathrm{pr}}=\mathcal{N}\left(\mathit{V}\mid {\mathit{\mu }}_{V,\mathrm{pr}},{\mathit{\Sigma }}_{V,\mathrm{pr}}\right)\mathrm{and}{\Phi }_{z\mid x}=\mathcal{N}\left(\mathit{z}\mid \mathit{h}\left(\mathit{x}\right),{\mathit{\Sigma }}_{z\mid x}\right),$$

(4)

where ${\mathit{\mu }}_{V,\mathrm{pr}}$ and ${\mathit{\Sigma }}_{V,\mathrm{pr}}$ are the expectation value and the covariance matrix of the voltage prior distribution. The covariance matrix of the likelihood distribution ${\mathit{\Sigma }}_{z\mid x}$ is built from the standard deviations of the measurement devices ${\mathit{\sigma }}_{\mathrm{z}}$: ${\mathit{\Sigma }}_{z\mid x}=diag{\left(\frac{1}{{\sigma }_z}\right)}^2$. Then (according to Equation (2)) the posterior probability distribution of voltage phasors ${\Phi }_{V,\mathrm{po}}$ is proportional to their product

$$\begin{array}{c}\hfill {\Phi }_{V,\mathrm{po}}\propto {\Phi }_{z\mid x}\times {\Phi }_{V,\mathrm{pr}}.\end{array}$$

(5)

A derivation of the explicit form of the posterior, Equation (5), is given in Appendix A, together with the relevant sources for the derivation.

In Bayesian state estimation, one typically computes a Maximum-A-Posterior estimate for $\mathit{x}$, i.e., a value $\mathit{x}$ that maximizes the posterior density function [10,11]. This is equivalent to finding $\mathit{x}$ that minimizes the negative sum of the exponent of the likelihood density function ${(\mathit{z}-\mathit{h}\left(\mathit{x}\right))}^{\mathrm{T}}{\mathit{\Sigma }}_{z\mid x}^{-1}(\mathit{z}-\mathit{h}\left(\mathit{x}\right))$ and the exponent of the prior density function ${(\mathit{x}-{\mathit{\mu }}_{V,\mathrm{pr}})}^{\mathrm{T}}{\mathit{\Sigma }}_{V,\mathrm{pr}}^{-1}(\mathit{x}-{\mathit{\mu }}_{V,\mathrm{pr}})$:

$$\begin{array}{cc}\hfill {\widehat{\mathit{x}}}_{\mathrm{MAP}}& =\underset{x}{arg\; min}\left[{(\mathit{z}-\mathit{h}\left(\mathit{x}\right))}^{\mathrm{T}}{\mathit{\Sigma }}_{z\mid x}^{-1}(\mathit{z}-\mathit{h}\left(\mathit{x}\right))+{(\mathit{x}-{\mathit{\mu }}_{V,\mathrm{pr}})}^{\mathrm{T}}{\mathit{\Sigma }}_{V,\mathrm{pr}}^{-1}(\mathit{x}-{\mathit{\mu }}_{V,\mathrm{pr}})\right]\hfill \end{array}$$

(6)

Note that the Maximum-A-Posterior estimate for Gaussian distributions is equal to the expectation of the posterior distribution ${\mathit{\mu }}_{x,\mathrm{po}}$ and is hence equal to the Minimum Mean Square Error (MMSE) estimate defined by

$$\begin{array}{c}\hfill {\widehat{\mathit{x}}}_{\mathrm{MMSE}}=\mathbb{E}[\mathit{x}\mid \mathit{z}]={\mathit{\mu }}_{x_{\mathrm{po}}}={\widehat{\mathit{x}}}_{\mathrm{MAP}}.\end{array}$$

(7)

In contrast, state estimation approaches based on the Weighted-Least-Squares approach minimize the squared sum of residuals ($\mathit{r}=\mathit{z}-h\left(\mathit{x}\right)$) weighted by the standard deviation of measurement errors [3,17,29]. It is assumed that all measurement errors are uncorrelated, and hence the covariance matrix is, in fact, a diagonal matrix, i.e., ${\mathit{W}}^{-1}=diag{\left(\frac{1}{{\sigma }_e}\right)}^2$:

$$\begin{array}{cc}\hfill {\widehat{\mathit{x}}}_{\mathrm{WLS}}& =\underset{x}{arg\; min}\left[{(\mathit{z}-\mathit{h}\left(\mathit{x}\right))}^{\mathrm{T}}{\mathit{W}}^{-1}(\mathit{z}-\mathit{h}\left(\mathit{x}\right))\right]\hfill \end{array}$$

(8)

Viewed in this way, ${\widehat{\mathit{x}}}_{\mathrm{WLS}}$ is actually a special case of the Maximum-Likelihood estimator (${\widehat{\mathit{x}}}_{\mathrm{ML}}$), which maximizes the conditional probability for $\mathit{x}$ given $\mathit{z}$ (Note that the Maximum-Likelihood estimator can in turn be seen as a special case of the Maximum-A-Posterior estimator where the prior distribution is assumed to be uniform and thus $p\left(\mathit{x}\right)$ is a constant for all $\mathit{x}$):

$$\begin{array}{cc}\hfill {\widehat{\mathit{x}}}_{\mathrm{ML}}& =\underset{x}{arg\; max}p\left(\mathit{z}\mid \mathit{x}\right)=\underset{x}{arg\; min}\left[{(\mathit{z}-h\left(\mathit{x}\right))}^{\mathrm{T}}{\mathit{\Sigma }}_{z\mid x}^{-1}(\mathit{z}-h\left(\mathit{x}\right))\right]\hfill \end{array}$$

(9)

As can be seen, the Weighted-Least-Squares estimate is actually a Maximum-Likelihood estimator for normally distributed measurement errors under the additional assumption that ${\mathit{\Sigma }}_{z\mid x}$ is diagonal (and hence all measurement errors are independent).

Bayesian and WLS DSSE differ in how they approach a low measurement coverage, i.e., unavailable real-time data, and how they use additional offline data. In Bayesian DSSE, data from additional offline sources were used to parameterize the prior state distribution, in particular, accounting for load and physical correlations. In WLS DSSE, in contrast, additional data sources are used to parameterize pseudo-measurements, which (like all measurements) are treated as uncorrelated by default. Load correlations can, however, be included in the WLS approach by relaxing the assumption that measurement errors are uncorrelated (and hence, the weighting matrix is diagonal), which leads to the full Maximum-Likelihood estimator as described above.

Finally, note that even most Bayesian DSSE approaches only consider the (deterministic) Maximum-A-Posterior estimator (${\widehat{x}}_{\mathrm{MAP}}$ as the result of the state estimation process. As demonstrated in [30], however, there is additional information contained in the full posterior distribution (e.g., its variance) that can be leveraged for use cases such as the identification of critical system states. An example of such a use case is also given in the case study below (Section 4).

The following section examines how load correlations impact the prior distribution in Bayesian Linear State Estimation.

2.3. Correlation in Bayesian Linear State Estimation

Multiple papers have shown how the posterior distribution can be analytically calculated in the context of state estimation [8,12,30]. The state estimation workflow performed in this paper is given below. For the initial setup, the prior distribution is computed via the following steps:

• Determine the (multivariate) background load probability distribution ${\Phi }_S$;
• Re-write the background load probability distribution ${\Phi }_S=\mathcal{N}\left({\mathit{\mu }}_S,{\mathit{\Sigma }}_S\right)$ in complex form of a normal distribution ${\Phi }_S^{\mathrm{C}}=\mathcal{N}\left({\mathit{\mu }}_S,{\Gamma }_S,{\mathit{C}}_S\right)$ to incorporate the load correlations (For the detailed steps to calculate ${\Gamma }_S$ and ${\mathit{C}}_S$ from ${\mathit{\Sigma }}_S$, see Equations (A1) and (A2));
• Calculate the prior voltage distribution ${\Phi }_{V,\mathrm{pr}}^C$ using linearized power flow calculation (see, Equations (A5)–(A8));
• Re-transform the complex form of ${\Phi }_{V,\mathrm{pr}}^{\mathrm{C}}$ to the real component form of a normal distribution ${\Phi }_{V,\mathrm{pr}}$ (see Equation (A9)).

After the initial set-up, the following step is performed whenever a new estimate of the system state is required (e.g., whenever real-time measurements are available):

5

Combine the real-time measurements and the prior distribution to derive the posterior voltage distribution ${\Phi }_{V,\mathrm{po}}$ (For the explicit calculation of expectation value and covariance matrix of ${\Phi }_{V,\mathrm{po}}$, see Equations (A12) and (A13));

The workflow starts in step 1 with determining a background distribution for loads ${\Phi }_S$, which is the LPD for the complex power $\mathit{S}\in {\mathbb{C}}^N$ of load and generation units at the non-slack nodes. ${\mathit{\mu }}_S$ and ${\mathit{\Sigma }}_S$ are the expectation value and covariance matrix of the background distribution. When using the multivariate normal distribution, the real and imaginary parts of the complex-valued vectors and matrices are stacked as shown in Equation (10).

$$\begin{array}{c}\hfill {\Phi }_S=\mathcal{N}\left(\mathit{S}\mid {\mathit{\mu }}_S,{\mathit{\Sigma }}_S\right)=\mathcal{N}\left(\left(\begin{array}{c}{\mathit{S}}_{\mathrm{re}}\\ {\mathit{S}}_{\mathrm{im}}\end{array}\right)\mid \left(\begin{array}{c}{\mathit{\mu }}_{S,\mathrm{re}}\\ {\mathit{\mu }}_{S,\mathrm{im}}\end{array}\right),\left(\begin{array}{cc}{\mathit{\Sigma }}_{S,\mathrm{rr}}& {\mathit{\Sigma }}_{S,\mathrm{ri}}\\ {\mathit{\Sigma }}_{S,\mathrm{ir}}& {\mathit{\Sigma }}_{S,\mathrm{ii}}\end{array}\right)\right)\end{array}$$

(10)

Here, the covariance matrices ${\mathit{\Sigma }}_S\in {\mathbb{R}}^{2N\times 2N}$ is composed of blocks for the real and imaginary parts, which are denoted by a combination of indices r and i. The elements ${\mathit{\Sigma }}_{S,nm}$ for $n,m\in 2N$ of the covariance can be written more explicitly:

• The diagonal elements ($n=m$) represent the marginalized variances ${\mathit{\Sigma }}_{S,nn}={\mathit{\sigma }}_{S,n}^2$, which are equal the squared standard deviations ${\mathit{\sigma }}_{S,n}$.
• The non-diagonal elements ($n\ne m$) can be determined from load correlations $r_{S,nm}$ and standard deviations, where the relation is given by equation for Pearson correlation coefficient: ${\mathit{\Sigma }}_{S,nm}=r_{S,nm}·\sqrt{{\mathit{\Sigma }}_{S,nn}}·\sqrt{{\mathit{\Sigma }}_{S,mm}}$.

As mentioned in the introduction, these covariance matrices represent correlations between active and reactive power at an individual load and correlations between different loads due to similarities in consumer behavior (e.g., household loads) or caused by weather patterns (e.g., for PV and Wind power generation). The voltage prior distribution is obtained from the LPD by a linearized power flow transformation (see step 3 above and Equations (A5)–(A8)) and includes the correlation between the voltages at different nodes in the non-diagonal covariance matrix elements. The node voltage correlation results from the physical correlation (resulting from the coupling through the grid) and the behavioral correlations from the background distribution.

The following section focuses on estimating the LPD for incorporating load correlations, which is the input for state estimation approaches.

3. Correlation-Aware Load Probability Distribution Synthesis

In this section, the aim is to estimate the LPD parameters $\left({\mathit{\mu }}_{\mathrm{S}},{\mathit{\Sigma }}_{\mathrm{S}}\right)$ by combining non-real-time inputs with varying levels of detail. The input data

• Contains recorded power measurements at MV nodes or 15 min time-resolved SM [5] and RLM [6] measurements as well as annual energy demand values and SLP classes of LV nodes;
• Can be extended by further exogenous measurements like solar irradiation measurements for power estimation of PV modules or wind velocity measurements for estimated power of wind turbines.

Further external inputs are required: the measurement type and the index of the MV node, the assignment of connected LV nodes to the corresponding MV node given by grid topology, and the smart meter coverage of the underlying LV grid.

The correlation-aware LPD synthesis method consists of the following steps:

• Classifying every MV node into a Measurement Instrumentation Scenario (MIS), the scenarios are explained in detail in Section 3.1;
• Determination of load time series (apparent power) for every MV node according to its assigned MIS as described in Section 3.2;
• Determination of LPD parameters ${\mathit{\mu }}_S$ and ${\mathit{\Sigma }}_S$, which is explained in Section 3.3.

3.1. Measurement Instrumentation Scenario

A classification for MV nodes into different Measurement Instrument Scenarios (MIS) is proposed in

Table 1. Classification of MV nodes into different measurement instrumentation scenarios (MIS).

MIS	Description
MV${}_{\mathrm{meas}}$	Measurement at MV node
SM${}_{100\%}$	$100\%$ SM coverage in underlying LV grid
SM${}_{\ge {\alpha }_{\mathrm{thres}}}$	SM coverage $\ge {\alpha }_{\mathrm{thres}}$ in LV grid
SM${}_{<{\alpha }_{\mathrm{thres}}}$	SM coverage $<{\alpha }_{\mathrm{thres}}$ in LV grid

.

The first scenario (MV${}_{\mathrm{meas}}$) covers all MV nodes with recorded real-time measurements or with RLMs. Here, the meter devices are directly installed into the MV nodes. This usually only applies to the primary substation, central MV nodes, or MV nodes with directly connected large customers. Most of the MV nodes are not equipped with a metering device. However, historical data from the SMs of the underlying LV grids can be used as an information source: If all customers in an LV grid are equipped with SMs, their summed-up demand would be approximately the demand at the connected MV node (neglecting the line and transformer losses).

Three scenarios are defined for the underlying LV grids, corresponding to full, substantial, or low SM coverage. Here, a low SM coverage means that the set of customers with SMs cannot be assumed to be statistically representative of the behavior of all consumers. The scenario where the LV grid is fully covered with SMs is denoted by SM${}_{100\%}$. To differentiate between LV grids with substantial and low SM coverage, a threshold fraction ${\alpha }_{\mathrm{thres}}$ is defined (e.g., ${\alpha }_{\mathrm{thres}}=60\%$). Using this threshold fraction, the measurement instrumentation scenario SM${}_{\ge {\alpha }_{\mathrm{thres}}}$ represents grids where SM measurements are available for at least ${\alpha }_{\mathrm{thres}}$ of all customers. Analogously, the scenario SM${}_{<{\alpha }_{\mathrm{thres}}}$ represents grids where SM measurements are available for less than ${\alpha }_{\mathrm{thres}}$ of all customers. Figure 1 shows an exemplary MV node for each proposed MIS. Green points mark the placement of SMs.

To estimate the power consumption of non-metered grid users, system operators often conventionally use SLP [7]. In the proposed method, it is also possible to take SLPs for the LV grid profiles $S_{{\mathrm{LV}}_{\mathrm{comp}}}$ of MV nodes assigned to SM${}_{<{\alpha }_{\mathrm{thres}}}$. However, using SLPs does not correctly reproduce correlations between groups of customers. Simply calculating the sample correlations between two MV nodes using the same SLP time series results in an unrealistically high correlation value of $100\%$. One way to avoid such unrealistic correlation values is to consider correlations between comparable LV grids. Hence, exemplary synthetic power time series are used. A more detailed description is given in

Table 2. The determination of time series values $S_{\mathrm{MV},n,t}$ for MV nodes according to MIS classification is given. In the case of SM${}_{<{\alpha }_{\mathrm{thres}}}$ a set of K exemplary, synthetic time series is created to achieve correlation-aware LPD parameters.

MIS	Determination of Time Series for MV Nodes
MV${}_{\mathrm{meas}}$	Recorded MV node measurements	$S_{\mathrm{MV},n,t}=S_{{\mathrm{MV}}_{\mathrm{meas}},n,t}$
SM100%	Sum up SM data from the underlying LV grid	$S_{\mathrm{MV},n,t}\approx S_{{\sum }_{\mathrm{SM},n,t}}={\displaystyle \sum _{l=1}^{N_{\mathrm{SM},n}}}{S}_{\mathrm{SM},n,l,t}$
SM${}_{\ge {\alpha }_{\mathrm{thres}}}$	Sum up data from SM nodes & scale up to meet total annual energy consumption	$S_{\mathrm{MV},n,t}\approx S_{{\sum }_{\mathrm{SM},n,t}}·f_{\mathrm{scale},n}$ with $f_{\mathrm{scale},n}=\frac{E_{\sum \mathrm{LV},n}}{E_{{\sum }_{\mathrm{SM}},n}}$
SM${}_{<{\alpha }_{\mathrm{thres}}}$	Calculate K exemplary, synthetic time series from sum of SM-data for SM nodes & sum of comparable profiles for non-SM nodes	$S_{{\mathrm{MV}}_{\mathrm{synth}},n,t,k}=S_{{\sum }_{\mathrm{SM},n,t}}+S_{{\sum }_{\mathrm{LV}}\mathrm{comp},n,t,k}$ with $k\in \left\{1,...,K\right\}$

.

3.2. Determination of Time Series Values

The MV node time series are denoted by $S_{\mathrm{MV},n,t}$ at every MV node $n\in \{1,..,N\}$ for every time step $t\in \{1,...,T\}$. In Table 2, the calculation steps for the determination of time series values are shown for each MIS.

For MV nodes in scenario MV${}_{\mathrm{meas}}$, the recorded measurements $S_{{\mathrm{MV}}_{\mathrm{meas}},n,t}$ at the MV nodes can directly be used as time series input. The other scenarios use the power time series from the SMs of the underlying LV grid. The SM power time series for LV node $l\in \{1,...,N_{\mathrm{SM},n}\}$ connected to MV node n are denoted by $S_{\mathrm{SM},n,l,t}$. $N_{\mathrm{SM},n}$ is the number of the LV nodes equipped with smart meters of a corresponding MV node n. For the scenario with the assumption of $100\%$ coverage of SM in LV grids, the active and reactive power of the SM are summed up for the corresponding MV node n and result in a time series $S_{{\sum }_{\mathrm{SM},n,t}}$ with a quarter-hourly resolution. This approximation neglects distribution losses. These typically amount to no more than $5\%$ for active power and $10\%$ for reactive power, which justifies this simplification.

For LV grids with larger SM coverage than the set threshold ${\alpha }_{\mathrm{thres}}$, it is assumed that the summed-up time series $S_{{\sum }_{\mathrm{SM},n,t}}$ of the SM measured grid users is representative for the behavior of this LV grid. To achieve correct energy demand values at the MV node, the aggregated SM time series are multiplied with a scaling factor $f_{\mathrm{scale},n}$ compensating for the missing power contributions from unmeasured customers. This factor is calculated as the ratio of the summed-up annual energy demand of all LV nodes of this grid $E_{\sum \mathrm{LV},n}$ and the summed-up energy demand recorded by SM $E_{\sum \mathrm{SM},n}$.

In the last scenario SM${}_{<{\alpha }_{\mathrm{thres}}}$, the SM information cannot be assumed to be representative of the non-SM nodes due to the small sample size. Therefore, a set of K exemplary, synthetic time series $S_{{\mathrm{MV}}_{\mathrm{synth}},n,t,k}$ for $k\in \left\{1,...,K\right\}$ is created. These exemplary time series will be used to estimate the parameters of an LPD representing this MV node (see Section 3.3). The number of power profiles K should be high enough to achieve sufficient sampling. A residential profile from comparable grids is randomly assigned for every LV node which is not an SM node. The profiles are scaled to the LV nodes’ annual consumption. For each k, the synthesized time series $S_{{\mathrm{MV}}_{\mathrm{synth}},n,t,k}$ is now defined as the sum of the aggregated SM profiles $S_{{\sum }_{\mathrm{SM},n,t}}$ for the SM nodes and of the aggregated comparable profiles for the non-SM nodes $S_{{\sum }_{\mathrm{LV}}\mathrm{comp},n,t,k}$. In the extreme special case where no smart meter measurements are available at all (i.e., $N_{\mathrm{SM},n}=0$), each synthesized time series $S_{{\mathrm{MV}}_{\mathrm{synth}},n,t,k}$ consists only of aggregated, synthesized time series and $S_{{\sum }_{\mathrm{SM},n,t}}=0$. This case will be evaluated in detail in our case study in Section 4.

3.3. Determination of LPD Parameters

The mean ${\mathit{\mu }}_{S,n}$ is calculated from the time series values as the sample expectation value ${\mathit{\mu }}_{S,n,\mathrm{samp}}$. This is equivalent to the annual energy consumption divided by the number of time steps T ($\frac{1}{T}·E_{\sum \mathrm{LV},n}$). As the synthetic time series are scaled to the annual consumption, the mean is always the same for every exemplary K.

$$\begin{array}{c}\hfill {\mathit{\mu }}_{S,n,\mathrm{samp}}=\frac{1}{T}\sum _{t=1}^T{S}_{\mathrm{MV},n,t}\end{array}$$

(11)

The elements ${\mathit{\Sigma }}_{S,nm}$ for the LPD covariance are calculated according to

Table 3. Determination ${\mathit{\Sigma }}_S$ element for $(n,m)$-pairs and included MIS.

MIS of n and m	Calculation of Elements for ${\mathit{\Sigma }}_{\mathit{S}}$
$n,m\notin$ SM${}_{<{\alpha }_{\mathrm{thres}}}$	${\mathit{\Sigma }}_{S,nm,\mathrm{samp}}=\frac{1}{T-1}{\sum }_{t=1}^T\left(S_{\mathrm{MV},n,t}-{\mathit{\mu }}_{S,n,\mathrm{samp}}\right)\left(S_{\mathrm{MV},m,t}-{\mathit{\mu }}_{S,m,\mathrm{samp}}\right)$
$n\in$ SM${}_{<{\alpha }_{\mathrm{thres}}}$ or $m\in$ SM${}_{<{\alpha }_{\mathrm{thres}}}$	${\mathit{\Sigma }}_{S,nm,\mathrm{comp}}=\frac{1}{K}{\displaystyle \sum _{k=1}^K}\frac{1}{T-1}{\displaystyle \sum _{t=1}^T}\left(S_{n,t}-{\mathit{\mu }}_{S,n,\mathrm{samp}}\right)\left(S_{m,t}-{\mathit{\mu }}_{S,m,\mathrm{samp}}\right)$
	$S_{n,t}=\left\{\begin{array}{cc}{S}_{{\mathrm{MV}}_{\mathrm{synth}}n,k,t}\hfill & ,\mathrm{if}n\in {\mathrm{SM}}_{<{\alpha }_{\mathrm{thres}}}\hfill \\ S_{\mathrm{MV},n,t}\hfill & ,\mathrm{else}\hfill \end{array}\right.$

: For $n,m$-pairs, where neither n or m are assigned to scenario SM${}_{<{\alpha }_{\mathrm{thres}}}$, the covariance matrix elements are calculated as the sample covariance ${\mathit{\Sigma }}_{S,nm,\mathrm{samp}}$. If at least one of the nodes is classified as SM${}_{<{\alpha }_{\mathrm{thres}}}$ (i.e., no sufficient measurement coverage), then the sample covariance is calculated for every k and then averaged over all k resulting in ${\mathit{\Sigma }}_{S,nm,\mathrm{comp}}$. To calculate the sample covariance for $k\in 1...K$, the k-th synthesized time series $S_{{\mathrm{MV}}_{\mathrm{synth}}n,k,t}$ is used for every $n\in$ SM${}_{<{\alpha }_{\mathrm{thres}}}$ and the measured time series$S_{\mathrm{MV},n,t}$ for every $n\notin$ SM${}_{<{\alpha }_{\mathrm{thres}}}$. To validate this approach, a statistical analysis of an exemplary set of residential active power profiles was performed in the following Section 3.4.

The steps of the Correlation-Aware LPD Synthesis Method, described in the sections above, are summarized in Figure 2.

3.4. Correlation Analysis between LV Grids

For the statistical analysis of an exemplary set of residential active power profiles, the profiles are exemplary taken from OpenEI for Washington state. OpenEI provides a “publicly available dataset of calibrated and validated 15-min resolution load profiles for all major residential and commercial building types and end uses, across all climate regions in the United States” [31]. This database has previously been used by other authors for state estimation methods [12]. Overall, 4947 individual residential profiles were chosen. The correlation coefficient between the individual power profiles is, on average, $11\%$.

For the analysis of correlations between sub-grids with a varying number d of households, for each pair of sub-grid sizes ($d_1$ and $d_2$), one thousand samples of $d_1$ and $d_2$ are randomly chosen. The required power profiles are taken from over 4900 Washington state profiles. They are summed up to obtain MV node time series ($S_{\mathrm{MV},n,t}$). Based on these profiles, the correlation coefficients are calculated. As a result, 1000 correlation coefficients are obtained for each pair $d_1$ and $d_2$ of sub-grid sizes. The resulting mean and standard deviations of the correlation coefficients are plotted as heat maps in Figure 3. Already, for subgrids consisting of only 20 customers each, the variance in correlation values drops to ∼4%. This indicates that the correlation of two similarly sized subgrids taken from a comparable region provides a reliable estimate for the correlation coefficient of two individual subgrids. Furthermore, if the size of the subgrids increases, so does the accuracy of the estimate: For a sufficiently large number of households in the subgrids (>80), the mean of correlation coefficients ${\mathit{\mu }}_{\mathrm{r}}$ is an accurate estimation of the correlation as the standard deviation for correlation coefficients ${\mathit{\sigma }}_{\mathrm{r}}$ is lower than $1\%$.

4. Case Study—Correlation–Aware State Estimation with Synthesized Load Profile Distributions

This section evaluates the accuracy of the proposed state estimation approach based on correlation–aware load probability distributions from two different perspectives:

• Recognition of critical system states.
• Accuracy of estimated prior voltage distribution.

The first perspective sheds light on the practical benefits expected from the approach when applied to a typical use case. The second perspective helps to give a more fundamental understanding of where improvements in accuracy are coming from and what the specific differences are compared to conventional approaches. At first, the performance metrics used in this evaluation are defined in Section 4.1. The simulation environment is presented in Section 4.2, and the different test cases with respect to SM coverage and load correlation assumption are given in Section 4.3. Finally, the recognition of critical system states is evaluated in Section 4.4, and the accuracy of the estimated voltage prior distribution is assessed in Section 4.5.

4.1. Performance Metrics

To evaluate the recognition of critical system states, a critical system state must be defined first. A critical system state is present if a voltage band ($V_{\mathrm{up}}$, $V_{\mathrm{low}}$) or thermal current limit ($I_{\mathrm{th}}$) is violated for any node or branch element. In this case study, voltage limits are set to ±6% of the nominal voltage, which are limits commonly used by distribution system operators (see [30]). The thermal current limits are taken from the chosen test power system (defined below). From the state estimation results (step 5 of state estimation workflow in Section 2.3), the probability of critical systems states $p_{\mathrm{css}}$ can be calculated (see Equation (A14)): It is the probability that the system state described by the posterior distribution ${\Phi }_{V,\mathrm{po}}$ violates operational limits (e.g., voltage band or thermal current limits). Note that the voltage posterior distribution ${\Phi }_{V,\mathrm{po}}$ can also be used to derive a posterior distribution of branch currents ${\Phi }_{I_{\mathrm{po}}}$ via the branch admittance matrix $\mathit{Y}$, since the branch currents $\mathit{I}=\mathit{Y}\mathit{V}$.

Finally, a probability threshold ($p_{\mathrm{thres}}$) must be defined at which the state is classified as critical. A system state is classified as critical if any value in the range of $\mathit{\mu }\pm {\mathit{\sigma }}_{\mathrm{po}}$ is critical. The range $\mathit{\mu }\pm {\mathit{\sigma }}_{\mathrm{po}}$ contains $68.3\%$ of all estimates from the posterior distribution. Hence, $15.85\%$ of the distribution remain on each side (=$(100\%-68.3\%)/2$). Therefore, the probability threshold $p_{\mathrm{thres}}$ is set to ≈15.9%. If the calculated probability of critical system states $p_{\mathrm{css}}$ exceeds this threshold, the value is inside the range $\mathit{\mu }\pm {\mathit{\sigma }}_{\mathrm{po}}$ and therefore assumed to be critical. The key performance indicator (KPI), used to measure how well critical system states are recognized, is the aggregated true positive rate ${\mathit{tpr}}_{lim}^{\widehat{x}}$ (see [30]). An estimate is considered a true-positive if the estimate ${\widehat{x}}_{ϵ,t}$ for a grid element $ϵ\in \{1,...,E\}$ (node or branch) for time step $t\in \{1,...,T\}$ correctly recognizes a critical system state corresponding to the given limit ($lim$). In this case, ${\mathit{tp}}_{ϵ,t,lim}^{\widehat{x}}=1$ (otherwise, ${\mathit{tp}}_{ϵ,t,lim}^{\widehat{x}}=0$). The aggregated true positive rate, ${\mathit{tpr}}_{lim}^{\widehat{x}}$, is now given by

$$\begin{array}{cc}\hfill {\mathit{tpr}}_{lim}^{\widehat{x}}& =\frac{1}{{\mathit{pos}}_{\mathrm{true}}}\sum _{t=1}^T\sum _{ϵ=1}^E{\mathit{tp}}_{ϵ,t,lim}^{\widehat{x}}\hfill \end{array}$$

(12)

where ${\mathit{pos}}_{\mathrm{true}}$ is the number of all limit violations (over all time steps and grid elements) in the true system state time series. In some cases, one is primarily interested not in the fraction of critical cases correctly recognized but in the fraction of critical cases incorrectly missed. To measure this, the false negative rate $\mathit{fnr}=1-\mathit{tpr}$ is used.

The second KPI evaluates how accurately the prior state distribution can be estimated. The estimated and true prior distributions (available in the simulation) are compared in terms of the true and estimated expectation value and covariance matrix components. The normalized Root-Mean-Square-Error $NRMSE$ is used as an error metric. It is defined by the $RMSE$ between true and estimated multi-component quantities, $y_{\mathrm{true},n}$ and $y_{\mathrm{calc},n}$, normalized to the mean of the true values:

$$\begin{array}{cc}\hfill NRMSE& =\frac{RMSE}{\frac{1}{N}{\displaystyle \sum _{n=1}^N}{y}_{\mathrm{true},n}}·100\%\mathrm{with}RMSE=\sqrt{\sum _{n=1}^N{(y_{\mathrm{true},n}-y_{\mathrm{calc},n})}^2}.\hfill \end{array}$$

(13)

We applied this error metric to the expectation value and covariance matrix components. The resulting KPIs are a measure of how close the estimated and the true prior distribution are.

4.2. Simulation Environment

The present approach was tested in a simulation study to evaluate the effect of correlations in general and the performance of the synthesized correlation-aware load probability distributions. A 20 kV MV grid with 107 buses from [32] (1-MV-comm–0-sw) was taken as a test system. The model is used in the pandapower format [33] and contains consumption and generation profiles for one year with a 15-min resolution for every MV node. The grid topology is shown in Figure 4. The renewable generation plants directly connected to MV nodes include PV, wind, biomass, and hydropower units. In summary, 10 large RES units and 19 large commercial customers are connected at the MV level. At 79 MV nodes, LV grids are located. The LV grids have varying numbers of customers, 13 to 118.

In the original dataset from Simbench, load and generation profiles are assigned to nodes in the test model from a relatively small pool of profiles, resulting in the same profile being assigned to multiple nodes. While this is accurate enough for most grid analyses, it leads to unrealistically high correlations in the behavior of different loads. Since these correlations play an important role in evaluating the performance of the proposed approach, the test system must realistically reproduce the correlations between power profiles of different MV nodes. We, therefore, retain only the static specification of the grid topology and the load types from the simbench data set but apply a bottom-up approach to synthesize the load profiles for them. Since the load profiles from Washington state (as described in Section 3.4) will be used as comparable profiles to estimate LPDs for subgrids with low smart meter coverage, we draw household load profiles for the test system from an independent pool of 7000 OpenEI load profiles for a different state (New York state) [31]. This process avoids our comparable load profiles being unrealistically similar to the true load profiles in the test data. For MV nodes with underlying LV grids dominated by household profiles, every household is assigned a new, unique residential profile from that pool. The commercial profiles from simbench are replaced by OpenEI profiles of the same type. The PV and wind profiles are taken from simbench. For the reactive power profiles, the power angle $\phi$ between active ${\mathit{S}}_{\mathrm{re}}$ and reactive power ${\mathit{S}}_{\mathrm{im}}$ is assumed to be constant.

To simulate the true system state (ground truth), a power flow calculation is executed for every time step using the load profiles described above. Simulated measurements are then obtained from true system states without adding synthetic measurement errors. For the measurement scenario, all MV nodes with large commercial consumers and RES are assumed to be equipped with RLMs. Furthermore, five voltage PMU measurements (assumed accuracy: ${\mathit{\sigma }}_{{\mathrm{V}}_{\mathrm{mag}}}:0.2\%,{\mathit{\sigma }}_{{\mathrm{V}}_{\mathrm{ang}}}:0.{11}^{\circ }$) are placed at buses 2, 5, 23, 77, and 87 (see Figure 4). For the household loads, two different test cases are investigated, corresponding to smart meter coverage of 0 and $100\%$.

In the terminology of Section 3, this results in the following MIS for the different MV nodes:

• The 5 MV nodes with real-time measurements and 21 MV nodes with RLMs (large commercial consumers and RES) are assigned to MV${}_{\mathrm{meas}}$.
• The 79 MV nodes with underlying LV grids are assigned to SM${}_{100\%}$/SM${}_{\ge {\alpha }_{\mathrm{thres}}}$/SM${}_{<{\alpha }_{\mathrm{thres}}}$. The SM coverage is either 0 or $100\%$, depending on the test case.

On this test dataset, the performance of the approach is evaluated in the context of a typical use case: the recognition of critical system states. To this end, the approach is integrated into a complete Bayesian state estimation workflow, described in detail in Section 2.3. The workflow uses a single LPD for all state estimation tasks. Note that the accuracy of Bayesian state estimation can further be improved if a more accurate prior voltage distribution is available for a specific state estimation task. One example would be distinguishing between weekdays and weekends or different seasons, using a separate LPD (and hence prior voltage distribution) for each setting.

4.3. Test Cases

For the evaluation, four test cases are defined:

Test case 1: 

The SM coverage of all underlying LV grids is assumed to be $100\%$ (SM${}_{100\%}$), i.e., 15 min “time-resolved” measured power profiles at every load are available for the estimation. The sample LPD covariance ${\mathit{\Sigma }}_{S,nm,\mathrm{samp}}$ is calculated according to our approach as described in Section 3.3. Hence, it considers the behavioral correlations (correlation aware, denoted in results tables by $\mathrm{cor}$).

Test case 2: 

The SM coverage is also assumed to be $100\%$, but the load correlations are not considered (correlation-unaware, denoted in results tables by ${\mathrm{cor}}_0$). This results in a diagonal covariance matrix of the LPD.

Test case 3: 

The SM coverage of $0\%$ (SM${}_{0\%}$) is assumed for all LV grids, i.e., only the annual energy demand is available for the estimation approach. The LPD is calculated according to our approach as described in Section 3.3, using a value of $K=1000$ and drawing comparable profiles for non-SM nodes (in this case, all nodes in the LV grid) from the pool of 4947 OpenEI power profiles described in Section 3.4. Hence, it considers the load correlations (correlation aware, denoted in results tables by $\mathrm{cor}$).

Test case 4: 

The same SM coverage as for the third test (SM${}_{0\%}$) is assumed, but the correlation is neglected (correlation-unaware, denoted in results tables by ${\mathrm{cor}}_0$).

4.4. Recognition of Critical System States

As mentioned above, the evaluation starts with a test to recognize critical system states. For every time step $t\in \{1,...,T\}$, the Bayesian linear state estimation is conducted as described in Section 4.2 based on LPDs calculated according to the approach described in Section 3. From the resulting posterior distribution, the probability of critical system states $p_{\mathrm{css}}$ is calculated as described in Section 4.1. To quantify the accuracy, the true positive rate $\mathit{tpr}$ (see Section 4.1) was used (The aggregated true-negative rate, which gives the percentage of correctly recognized non-critical system states, is for all eight cases larger than $99.6\%$). The results are shown in

Table 4. Aggregated true-positive rates in % for recognition of critical system states (lower voltage limit violation or thermal current limit violation) by Bayesian DSSE for test cases with different smart meter coverage (SM${}_{100\%}$ and SM${}_{0\%}$), as well as with ($\mathrm{cor}$) and without (${\mathrm{cor}}_0$) considering load correlation.

MIS		SM${}_{100\%}$		SM${}_{0\%}$
Approach		$\mathrm{cor}$	${\mathrm{cor}}_0$	$\mathrm{cor}$	${\mathrm{cor}}_0$
${\mathit{tpr}}_{V_{\mathrm{low}}}^{\widehat{x}}$		$98.0$	$82.1$	$98.0$	$80.7$
${\mathit{tpr}}_{I_{\mathrm{th}}}^{\widehat{x}}$		$97.6$	$76.7$	$95.2$	$72.8$

below.

First, the results of the SM${}_{100\%}$ scenario are compared. In this case, taking into account behavioral correlations for calculating the probability of critical system states improves ${\mathit{tpr}}_{lim}^{\widehat{x}}$ by $15.9$ percentage points for voltage limit violations and by $20.9$ percentage points for thermal current limit violations. The picture is similar for the SM${}_{0\%}$ scenario: Here, ${\mathit{tpr}}_{lim}^{\widehat{x}}$ improves by $17.3$ percentage points for voltage limit violations and by $22.4$ percentage points for thermal current limit violations.

The benefit of considering behavioral correlations becomes even more striking when viewed from the perspective of critical system states that are missed (i.e., not correctly recognized) by each approach. This can be captured, e.g., looking at the false negative rate $\mathit{fnr}$: In the scenario SM${}_{0\%}$ where no smart meter measurements are available, the conventional approach ${\mathrm{cor}}_0$ misses $19.3\%$ of true critical system states caused by voltage limit violations. Using the approach $\mathrm{cor}$ based on the correlation-aware estimation of background load probability distributions in this article, that fraction drops to $2\%$, a reduction by a factor of $9.7$. Analogously, for thermal current limit violations, a reduction of the $\mathit{fnr}$ by a factor of $5.7$ is observed.

These findings confirm the importance of considering load correlations in DSSE algorithms, as found in the context of WLS state estimation by [19]. It also demonstrates the relevance of the Correlation-Aware LPD Synthesis method presented in this article: The method enables correlation-aware state estimation even for nodes with no smart meter coverage. The first and third test cases (SM${}_{100\%}$ vs. SM${}_{0\%}$, both with correlation awareness) result in similar true positive rates. Even without smart meter coverage, the accuracy is much better than what can be achieved without considering load correlations, even if background distributions are based on full historical smart meter data.

It is important to note that, as described in Section 3.2, the proposed approach only uses comparable load profiles for the estimation of correlations in the case SM${}_{0\%}$. These give us an approximate indication of the true behavioral correlation between loads in the system (in fact, the estimated correlation values differ from the empirical correlation in our ground truth by $12\%$ on average, with a maximum value of $39\%$). However, as the results above show, this approximation is good enough to substantially improve the accuracy of the resulting state estimation and, consequently, the recognition of critical system states.

In summary, it could be confirmed that considering load correlations allows for substantially more accurate recognition of critical system states. Furthermore, using the approach described in this paper, this improvement can be leveraged even in cases where no smart meter measurements are available that would allow an empirical estimation of load correlations. Instead, insights from a set of comparable (but different) load profiles from an entirely different geographical region can be used to achieve accuracy comparable to the case where full smart meter coverage is available.

4.5. Accuracy of Estimated Prior Voltage Distribution

To get a more detailed view of the improved state estimation accuracy behind the results in the previous section, we now look closer at the prior voltage distribution used in the Bayesian state estimation approach.

To achieve accurate state estimation results (and hence a reliable detection of critical system states), the approximation of ${\Phi }_{V,\mathrm{pr}}$ should represent the true empirical voltage distribution ${\Phi }_{V,\mathrm{pr},\mathrm{true}}$ (i.e., the distribution of voltage values in the simulated ground truth system state, resulting from a power flow calculation for every time step) as accurately as possible. To illustrate this fact, we have run the state estimation based on true, empirical voltage prior distribution ${\Phi }_{V,\mathrm{pr},\mathrm{true}}$, which results in recognition accuracies for critical system states of >99% (${\mathit{tpr}}_{V_{\mathrm{low}}}^{\widehat{x}}$: $99.9\%$, ${\mathit{tpr}}_{I_{\mathrm{th}}}^{\widehat{x}}$: $99.0\%$).

A closer look at the Bayesian state estimation algorithm shows that changes in the LPD covariance matrix ${\mathit{\Sigma }}_S$ only affect the covariance matrix ${\mathit{\Sigma }}_{V,\mathrm{pr}}$ of the prior voltage distribution (the expectation value ${\mathit{\mu }}_{V,\mathrm{pr}}$ is not affected). Hence, to assess the impact of changes in LPD covariances on the estimated prior voltage distribution, the calculated prior voltage covariance matrices are compared to the true empirical covariance matrix ${\mathit{\Sigma }}_{V,\mathrm{pr},\mathrm{true}}$ obtained from the simulated ground truth system state. Specifically, the following two characteristics of the prior voltage distribution are evaluated:

• Standard deviation ${\mathit{\sigma }}_{V,\mathrm{pr}}$ (overall variation of voltage values for each node);
• Correlation coefficient $r_{V,\mathrm{pr}}$ (correlation between voltage values at different nodes).

On a qualitative level, the standard deviation calculated without considering load correlations always substantially underestimates the true standard deviation, thus assuming a much lower variation of voltage values in the prior distribution (see Figure 5). On the other hand, if load correlations are considered, the resulting standard deviation is very close to the true empirical values (there are small deviations with a maximum of ±2.7% relative to the mean).

Table 5. $NRMSE$ for ${\mathit{\sigma }}_{V,\mathrm{pr}}$ and $r_{V,\mathrm{pr}}$ for calculated and true voltage prior covariance depending on MIS and load correlation $r_S$ assumption in %.

	${\mathit{\sigma }}_{\mathit{V},\mathit{pr}}$				${\mathit{r}}_{\mathit{V},\mathit{pr}}$
MIS	SM${}_{\mathit{100}\%}$		SM${}_{\mathit{0}\%}$		SM${}_{\mathit{100}\%}$		SM${}_{\mathit{0}\%}$
Correlation	$\mathrm{cor}$	${\mathrm{cor}}_0$	$\mathrm{cor}$	${\mathrm{cor}}_0$	$\mathrm{cor}$	${\mathrm{cor}}_0$	$\mathrm{cor}$	${\mathrm{cor}}_0$
$V_{\mathrm{pr},\mathrm{mag}}$	$15.5$	$71.9$	$27.6$	$72.3$	$2.1$	$40.9$	$4.8$	$41.3$
$V_{\mathrm{pr},\mathrm{ang}}$	$1.5$	$44.7$	$11.6$	$44.8$	$1.6$	$36.4$	$3.8$	$36.6$

summarizes the results separately for ${\mathit{\sigma }}_{{\mathrm{V}}_{\mathrm{pr}}}$ and $r_{{\mathrm{V}}_{\mathrm{pr}}}$. It shows the $NRMSE$ (see Equation (13)) for each of the two characteristics of the prior voltage distribution and for each of the four test cases from Section 4.3, separately for magnitude and angle components of V.

We first focus on the $NRMSE$ for the standard deviation of the prior voltage distribution. If load correlations are not taken into account (${\mathrm{cor}}_0$ columns), relatively high $NRMSE$ values (∼72% for voltage magnitude and ∼45% for voltage angle) are observed, confirming the qualitative observation from Figure 5. This is true independently of the level of smart meter coverage. Accounting for load correlations, on the other hand, reduces that error dramatically in the case of full smart meter coverage. Even in the case where no smart meter measurements are available, using the approach presented in this paper, error levels can be reduced to ∼$1/3$ (for $V_{\mathrm{pr},\mathrm{mag}}$, $27.6\%$ vs. $72.3\%$) and ∼$1/4$ (for $V_{\mathrm{pr},\mathrm{ang}}$, $11.6\%$ vs. $44.8\%$) compared to the baseline ${\mathrm{cor}}_0$.

Regarding the $NRMSE$ for the correlation coefficients of the prior distribution, an even more dramatic improvement can be observed. While the $NRMSE$ is somewhat lower if load correlations are not taken into account (∼41% for voltage magnitude and ∼36% for voltage angle), the increase in accuracy when considering load correlations is larger than before: In the case where no smart meter measurements are available, the approach reduces error levels to ∼$1/9$ (for both $V_{\mathrm{pr},\mathrm{mag}}$ and $V_{\mathrm{pr},\mathrm{ang}}$) compared to the baseline.

The comparison of true and calculated ${\mathit{\sigma }}_{V,\mathrm{pr}}$ and $r_{V,\mathrm{pr}}$ shows that the consideration of behavioral correlations in LPD for Bayesian DSSE strongly improves the accuracy of the estimated prior distribution ${\Phi }_{V,\mathrm{pr}}$. This translates into a more accurate posterior estimate of ${\Phi }_{V,\mathrm{po}}$, which can be observed in our assessment of recognition of critical system states above (Section 4.4).

Naturally, the achieved accuracy w.r.t the prior voltage distribution cannot be quite as high, when no smart meter measurements are available as in the case with full smart meter coverage. However, the results of this paper show that the approach presented in this article can improve the accuracy of the prior voltage distribution impressively, even without any smart meter measurements (see Section 4.5). In particular, the level of accuracy is sufficient for practical use cases, e.g., enabling a reliable recognition of critical system states (see Section 4.4).

5. Conclusions

This paper highlights the importance of accurately modeling load correlations in Bayesian state estimation. A method is proposed to calculate load probability distributions and accurate load correlations for medium voltage grids in different measurement instrumentation scenarios, which was previously only possible if a complete set of time-resolved power measurements was available at all underlying grid nodes (e.g., from smart meters). The resulting load probability distributions can be used as background distributions for Bayesian state estimation, but also to enhance WLS approaches that consider correlations between pseudo-measurements.

The main inferences of this paper are:

• Using the correlation-aware synthesis module for estimating the background distribution for Bayesian state estimation substantially improves the accuracy of state estimation results;
• Using the approach presented in this article, up to $98\%$ of critical system states are correctly identified, reducing the fraction of missed critical system states by a factor of up to nine compared to conventional approaches that do not take behavioral correlations in background load probability distributions into account;
• The approach is especially beneficial for cases with no or very limited data availability, where a factor improvement of nine can be achieved.
• The results emphasize the importance of accurately modeling the statistical properties of customer behavior in distribution system state estimation by achieving up to 22% percentage points higher detection rates.

For future research, the correlation analysis should be evaluated for different types of low-voltage grids. The distinction between rural, semi-urban, and urban grids could open a broader application scope for the proposed method. For future decarbonization scenarios, further research would be needed to determine load correlations between low-voltage grids with high photovoltaic, electric vehicles, and heat pump penetrations. Furthermore, validation in experiments with actual field data is required to confirm the practical applicability.