Probabilistic Short-Term Load Forecasting Incorporating Behind-the-Meter (BTM) Photovoltaic (PV) Generation and Battery Energy Storage Systems (BESSs)

Abstract

Increased behind-the-meter (BTM) solar generation causes additional errors in short-term load forecasting. To ensure power grid reliability, it is necessary to consider the influence of the behind-the-meter distributed resources. This study proposes a method to estimate the size of behind-the-meter assets by region to enhance load forecasting accuracy. This paper proposes a semi-supervised approach to BTM capacity estimation, including PV and battery energy storage systems (BESSs), to improve net load forecast using a probabilistic approach. A co-optimization is proposed to simultaneously optimize the hidden BTM capacity estimation and the expected improvement to the net load forecast. Finally, this paper presents a net load forecasting method that incorporates the results of BTM capacity estimation. To describe the efficiency of the proposed method, a study was conducted using actual utility data. The numerical results show that the proposed method improves the load forecasting accuracy by revealing the gross load pattern and reducing the influence of the BTM patterns.

1. Introduction

Renewable generation has rapidly increased based on governmental support to reduce carbon emissions and improve energy sustainability. Many countries have provided financial incentives and deregulated the installation of renewable energy systems to achieve penetration goals. The incentive schemes were designed to favor small-scale distributed energy resources (DERs), especially solar plus batteries, and most are developed as small-scale photovoltaic (PV) power systems that are more profitable than other generation types. Most small-scale PV companies have signed power purchase agreements (PPAs) with utilities to avoid financial risk and there are also privately owned small PVs such as rooftop solar systems. Unlike utility-scale PV systems, facility deregulation allows small-scale PV systems to use inexpensive metering systems that cannot monitor power in real-time or distinguish between solar and battery output.

Short-term load forecasting is essential for efficient power grid operations [1]. Electric demand forecasting was carried out in consideration of weather conditions, time variables, and holiday information [2]. In recent years, however, it has been more difficult to accurately forecast the net load because the capacity of BTM DERs has increased while it cannot be properly monitored [3]. In the past few years, many studies have been presented to address the impact of BTM on net load. There are mainly four contents of related work: (i) BTM estimation using alternative verification methods without true data points, (ii) BTM estimation with verification using true data points, (iii) various BTM estimation algorithms to disaggregate the BTM pattern from net load, and (iv) load forecasting methods to address the underlying uncertainties and BTM patterns.

2. Literature Review

BTM capacity estimation methods pose inherent challenges in deploying ground-truth data points. From a practical perspective, there are limited methods to verify the accuracy of the estimated BTM capacities. Thus, prior research has validated BTM capacity estimation results, excluding true data points. Wang et al. represented all BTM PV outputs by adopting virtual equivalent PV, was derived from weather and location data [4]. Shaker applied multiple BTM penetration scenarios to demonstrate the overall estimation accuracy [5]. the BTM capacity in low-voltage networks was analyzed using the weather condition changes and net load difference. Sun et al. focused on extreme weather condition and compared them to close days when electric demand patterns were less affected [6]. System operators need to continuously track the quantity of behind-the-meter DERs, which is a prerequisite for improving the transparency of the grid and reducing the uncertainties in net load patterns. Pennsylvania-New Jersey-Maryland Interconnection (PJM), which is a system operator in the United States, reconstitutes the net load using BTM capacity/pattern estimation and improves the results of net load forecasting integrated with PV forecast [2,3]. Load forecasting accuracy improvement was generally used in previous studies to validate the BTM capacity estimation results and the California Independent System Operator (CAISO) used the p-value and Pearson correlation coefficient to measure the load forecast improvement [7].

However, some research was able to validate its efficiency using actual data. Li et al. presented a two-stage decoupled estimation for disaggregate net load into load and BTM PV. The results were validated through actual utility data [8]. Shaffery et al. proposed a framework based on a Bayesian process to decompose the net load into the gross load and BTM PV patterns [9]. Representative solar sites were selected from the available historical data and used to estimate the entire BTM PV pattern [6,10,11,12]. Several previous studies have considered BTM resource behavior from a bottom-up approach using low-voltage network data. The BTM BESS was estimated by matching the assumed charging/discharging schedule patterns using California power system data [13].

BTM capacity estimation are derived from the various methodology. Kabir et al. presented an unsupervised framework used to disaggregate the residential load patterns and applied a mixed hidden Markov model (MHMM) to model the human behavior [14]. Li et al. proposed two-stage approach to disaggregate the effect of BTM PV from residential load pattern and adopted support vector machine (SVR) and an ensemble technique to address the analysis for 300 customers [15]. To consider regional differences, Mason et al. used a deep neural network to determine the representative BTM PV characteristics, such as tilt and azimuth [16]. Pan et al. showed the approach to disaggregate the impact of BTM PV through variability analysis [17]. Bu demonstrated a data-driven approach on netload disaggregation using game theory [18]. Peng et al. presented the method to determine the investment configuration including BTM PV plus BESS operations [19].

Recently, artificial neural networks have been widely used in short-term load prediction studies and show advanced results when combined with quantitative regression [20,21]. The net load forecasting accuracy was improved through a probabilistic approach that consider the uncertainties of input variables [22]. Saeedi et al. proposed an adaptive method to consider changes on load trend, human behavior, and the impact of additional BTM PV [23]. PJM and CAISO also used neural network framework to improve net load forecasting accuracy considering BTM patterns [3,7]. Kwon et al. showed that an online learning method improves the accuracy by quickly adapting underlying changes on net load [24]. In addition, autoencoder and recurrent neural network models are also used to forecast PV patterns using weather conditions, capacity data, and locational data [25,26].

From the previous studies, as shown in

Table 1. Summary of previous studies for BTM estimation and net load forecasting.

Category	Result Validation	Forecasting Target	Characteristics	Techniques and Publications
(i) BTM estimation using alternative methods	Virtual Scenarios	Net load	Regional BTM PVs were modeled by adopting equivalent PV models.	MIC, GBM, QRNN [4]
	Actual limited onsite PVs data	BTM PV	Used randomly chosen scenarios to overcome limited PV data.	Fuzzy [5]
		Net load	BTM patterns were used for post-correction of forecasted net load.	ANN [27]
	Load forecast improvement	Net load	BTM PV estimation results validated by alternative method	GBM [6], ANN [2,3,7]
(ii) BTM validation with true data points	Actual utility data	Residential load	BTM PV and BESS patterns were estimated simultaneously to improve housing load prediction.	Heuristic [13], Sensitivity Model [17], DNN [9]
		BTM PV	BTM PV were estimated without considering net load forecasting accuracy or BTM BESS.	PCA [10], Fuzzy [11], HMM [12]
		CBL	BTM PV estimation was used for improving the demand response analysis	SVR [15], K-means [8]
(iii) Various BTM Estimation methods	Actual utility data	BTM PV	BTM patterns were disaggregated from net load using the probabilistic modeling method.	MHMM [14], Random Forest, DNN, etc [23].
		BTM PV	Test dataset was generated using the simulator.	DNN [16]
		Net load and BTM PV	A data-driven approach was used to estimate the behavior of BTM PVs.	Game Theory [18]
	Economic evaluation	-	Estimating the configuration of BTM system using economic evaluation	Mixed-integer optimization [28], ROI, IRR [19]
(iv) Load forecasting methods	Load forecast improvement	Net load	Various machine learning techniques were used for the net load forecasting	QRNN [20,21,22], LSTM [24]
	Actual PV data	PV	Using weather conditions and locational data to forecast the PV outputs	Auto Encoder, LSTM [25], RNN [26]

, it can be observed that the accuracy of net load forecasting is improved through the process of disaggregating the net load into gross load and BTM patterns. It is obvious that BTM capacity estimation accuracy and net load forecasting accuracy are complementary to each other. Furthermore, the net load forecasting accuracy was improved by considering hidden BTM patterns [27]. This study also aimed to improve short-term load forecasting by considering the presence of behind-the-meter PVs and BESS. This study proposes a method for considering the uncertainties inherent in the BTM capacity estimation results for short-term load forecasting. Even incompletely, the provided true values had a positive effect on verifying the research results. From a practical perspective, this study referred to statistical data points from authorities, which are not precise but are sufficiently referenced. Front-the-meter (FTM) patterns by region were applied to address regional PV pattern differences. Referring to previous studies on regional differences in BTM PV patterns, this study also aimed to estimate and disaggregate the BTM capacity into eight major regions incorporating regional weather differences. The impact of BTM BESS is estimated based on metering system configuration and operation scheduling considering incentive policy structure.

The remainder of this paper is organized as follows: Section 3 presents the problem description. Section 4 formulates the proposed approach to estimate the BTM resources and applies the probabilistic load forecast method using previous results. The numerical studies and results are described in Section 5. Section 6 concludes the paper.

3. Problem Description

This section introduces the approach for formulating the behind-the-meter estimation problem and the proposed method for integrated short-term load forecasting. In general, load forecasting is a practical/alternative approach to modeling human behavior regarding electricity consumption; for this reason, weather information is commonly used for load forecasting to consider heating/cooling device loads. Based on this approach, traditional load forecasting methods can be formulated as in Equation (1):

$$Y_t=f\left(X,Y_{t-1}\right)+e_1$$

(1)

$$Y_t=f\left(X,Y_{t-1}\right)+g\left(X,Y_{t-1}\right)+e_2$$

(2)

where the error term $e_1$ can be minimized if the load forecasting model $f$ performs well; it can be defined as white noise.

However, model $f$ is fitted to follow the major human behavior and without additional information, it is difficult to find another behavior $g$ presented in Equation (2). This study proposes a hierarchical structure to clarify the relationship between power resources, as shown in Figure 1. From a bottom-up perspective, the pattern of each power resource group can be explained simply [5,10]. Conversely, the net load and BTM pattern can be expressed by Equations (3) and (4)., respectively. Furthermore, $e_1$ can be minimized by extracting the BTM pattern from $e_1$:

$$L_t=y_t+b_t+e_3$$

(3)

$$b_t=B_{pv,t}+B_{in,t}+B_{out,t}$$

(4)

where $y_t$ and $L_t$ are the net and gross loads, respectively. $b_t$ is an integrated BTM pattern, including solar and battery charging/discharging.

In previous studies, BTM patterns were derived from solar radiation and locational information using physical PV modeling [4,13]. However, the variation in installation status is quite different in the real world. The integrated outputs of PVs are highly correlated with solar radiation but also distributed owing to other weather parameters, as shown in Figure 2a. To consider regional weather differences and the physical properties of PVs, it is reasonable to incorporate front-the-meter PV data. This study assumed that the equivalent properties of the BTM are the same as those of the FTM for the same period.

In addition, Figure 2b shows that the slope between power generation and solar radiation, or power generation efficiency, decreased. This can be explained through the operation of behind-the-meter batteries. Behind-the-meter batteries are encouraged to charge during the day and discharge at night to receive the maximum renewable incentives in South Korea. It can also be observed in the upper tail of Figure 2b. BTM batteries are charged and discharged according to the scheduling of the operator, regardless of external environmental changes. To estimate the output of BTM BESS, this study assumed that all BTM BESS are operated to maximize the incentive revenue. In addition, this study assumed that the BTM capacity does not decrease because PVs and batteries last for at least 15 years.

Figure 3 shows the overall process of the proposed approach, which is combined with three components: probabilistic net load forecasting, BTM capacity estimation, and beam search. For probabilistic net load forecasting, this study applied a weak forecasting model to generate the forecasting error, which is considered a system-biased error, such as BTM solar pattern. The proposed BTM capacity estimation extracts the BTM pattern from the forecasting error using the regulated gradient descent. For the beam search stage, the final BTM estimation results are derived by selecting an advantageous option among the various probabilistic cases.

Finally, the gross load was reconstituted using the estimation results and short-term load forecasting method is proposed that combines the forecasted gross load and the forecasted BTM pattern into net load.

4. Proposed Load Forecasting Method

The proposed behind-the-meter estimation method has four components, as shown in Figure 3. Each component is based on a data-driven approach. The detailed process details and data descriptions are provided in this section.

4.1. Preprocess to Decompose PV and BESS

The FTM output data require preprocessing to decompose the solar and BESS outputs because the solar plus BESS resources were measured in an integrated form. Some solar plants can distinguish between solar power and BESS output by extracting data from inverters whereas some simply install more metering systems. For economic reasons, non-mandatory facilities have been minimized; hence, most solar power plants only use one meter, as shown in Figure 4.

Owing to the phase difference, the integrated output is slightly less than the numerical sum of the solar and battery AC powers but can be neglected in the utility-scale problem. The BTM BESS charging/discharging schedule can be estimated by considering the incentive policy of the utility. BTM BESSs are motivated to be charged between 11 a.m. and 4 p.m. and discharged after 7 p.m. until 9 a.m. the following day. Figure 5 shows the daily operation of a PV plus BESS plant.

The BESS facility capacities, such as the inverter output and battery size, can be reasonably estimated through empirical facts. Many studies have proposed optimization investment methods to determine the capacities of inverters and batteries for solar plus battery systems [28]. However, the BTM pattern is combined with only PV and PV plus BESS plants. Moreover, the bottom-up method of estimating individual BESS operations in various places risks leading to the spread of errors [11].

For this reason, this study applied the simplified ratio in

Table 2. Average Ratio of PV Plus BESS Size on Front-the-Meter Data.

Category	PV Capacity (MW)	Inverter Capacity (MW)	Battery Capacity (MWh)
Only PV	1.0	-	-
PV Plus BESS	1.0	0.8	3.2

and estimated the size of the BTM inverter and battery.

Table 2 can be justified because, for operational safety, the battery management system (BMS) approves charging when the solar output is larger than a certain threshold lb; usually lb is set to 10% of the solar capacity, indicated by the green dotted line in Figure 5, so the inverter capacity is less than PV. According to official statistics, the expected PV generation time is 3.7 h. Considering only the amount of power generated during the incentive period, this is approximately 320% of the PV capacity.

As shown in Figure 6, it is possible to approximate the size of the combined PV and battery based on the estimated inverter size, as presented in Table 2.

This study focused on BTM patterns after sunset, i.e., after 8 p.m., and is considered the maximum inverter output. To approach conservatively, the inverter capacity was derived on average, excluding the maximal and minimal 5 days from the monthly output at 8 p.m.; this method is proposed herein and is referred to as Mid (20/30). Individual differences by site were neglected, and the average BESS round-trip efficiency $\eta$ was considered as 95%. The BTM inverter capacity was estimated as follows:

$$\begin{array}{cc}{C}_{inverter}=Mid\left(\raisebox{1ex}{$20$}\!\left/ \!\raisebox{-1ex}{$30$}\right.\right)\left(P_{d,20H}\right)\hfill & \hfill \forall d in a month\end{array}$$

(5)

The BTM battery capacity was estimated as follows:

$$C_{battery,d}=\frac{3.2}{\eta }\times C_{inverter}$$

(6)

4.2. Regulated Probabilistic Net Load Forecast

The behind-the-meter patterns are hidden in the load forecast error patterns, but they cannot be simply identified owing to other factors, such as weather conditions, holidays, and long-term load growth. To minimize the impact of external factors, the load forecast error was derived using weekdays and reconstituted gross loads. Each month, the forecasting models were fitted using data from the preceding 12 months to consider the monthly changes on load pattern [29]. In general, short-term load forecasting models can be formulated as in Equation (7). The models use recent loads, observed weather, weather forecasts, and calendar features:

$${\widehat{y}}_t=model\left(y_{t-w:t-1}, o_{t-w:t-1},f_{t-w:t+h},c_{t-w:t+h}\right)$$

(7)

where ${\widehat{y}}_t$ is the forecasted net load at time $t$. $o_{t-w:t-1}$ and $f_{t:t+h}$ are observed weather in the past window w and weather forecasts in the forecasting horizon h, respectively. $c_{t-w:t+h}$ denotes calendar features, such as year, month, and day of the week, which are known for all times.

However, in this study, the model was designed as an ex-post analysis, and there is a limit to the input data to avoid effects other than the BTM effect. Observed weather is used to avoid weather forecast errors, and previous loads are not used as input data to avoid pre-reflected BTM effects and autoregressive properties. The proposed regulated load forecasting model for detecting the BTM pattern is given in Equation (8):

$${\widehat{y}}_{t-w:t-1}=model\left(o_{t-w:t-1}, c_{t-w:t-1}\right)$$

(8)

The net load forecasting errors, which are used for BTM capacity estimation, are derived from the quantile forecasting method. Quantile regression is widely used to measure forecast points in consideration of underlying uncertainties and to estimate the reliability of forecasted results [20,22]. This study used quantile forecasting to generate quantile forecast values that provide a probabilistic range for each forecasted value.

This study adopted a quantile forecasting method to address the reliability of the BTM capacity estimation. The pinball loss is used as a loss function, as described in Equation (9):

$$Pinball\left(y_t,\tau \right)=\left\{\begin{array}{cc}\left(y_t-z\right)\tau \hfill & \hfill if y\ge z\\ \left(z-y_t\right)\left(1-\tau \right)\hfill & \hfill if z>y\end{array}\right.$$

(9)

where $\tau$ is generally a quantile value that lies between 0.1 and 0.9. $y_t$ is the target value and $z$ is the forecasted value at quantile $\tau$. Pinball loss is the sum of losses in all quantiles.

Applying pinball loss can treat weak net load forecasting models as probabilistic detectors to reveal hidden patterns, regardless of specific forecasting date issues. The quantile forecasting results are shown in Figure 7. It can be observed that the range of forecasted results varies over time. The gap is narrow at night, when net loads are less volatile, and widens during dawn or daytime.

4.3. BTM Estimation Using Probabilistic Forecasting Error

This study proposes a method for extracting reasonable BTM capacities from probabilistic net load errors. Using this method, the BTM solar, inverter, and battery sizes can be estimated. Previous studies have proposed BTM pattern estimation using insolation data and an equivalent solar panel model by region [4,13]. However, it should assume an ideal environment in which all solar panels have the same characteristics, such as generation efficiency, location, installed angle, and shading. This study used front-the-meter data, including solar and battery, that could represent average physical specifications by diversity. From a practical perspective, FTM data enables realistic analysis because it reflects the smoothing effect that must be considered when analyzing aggregated BTM resources.

The proposed BTM estimation method comprises three components, as shown in Figure 8. BTM PV estimation, BTM BESS estimation, and net load accuracy gain calculation. These processes are combined as an optimization problem to maximize the expected net load forecasting accuracy improvement using the gradient descent method.

Figure 9 depicts the estimation method used to find additional BTM PV capacity. As introduced in Section 4.1, the FTM data are decomposed into PV and BESS patterns. Regional FTM PV patterns were used to estimate the BTM patterns.

In Figure 9, P is a constant matrix, which is a given FTM pattern. W is a learnable weight matrix, which indicates additional BTM PV capacities by region. The ReLU function was applied based on the assumption that the BTM capacity did not decrease. C is a constant matrix, which is a BTM capacity estimation result from the former iteration and its details are formulated as in Equation (10). The weight value of W is initially randomly selected from the standard distribution. Weight values are updated through optimization iterations and fit to maximize accuracy gain. Matrix C is updated after iteration completes as in Equation (11).

From the proposed hierarchical relationship:

$$\begin{array}{cc}{B}_{PV,t}\hfill & =PP{A}_t+SEL{F}_t\hfill \\ \hfill & =\left(ReLU\left(W\right)\oplus C_t\right)·\max \left(B_{PV,t-1}\right)\hfill \end{array}$$

(10)

$$C_t=BT{M}_{t-1}/\max \left(B_{PV,t-1}\right)$$

(11)

where $PP{A}_t$ is a matrix of known BTM solar capacities at time $t$, which are given data from the utility. $SEL{F}_t$ is a matrix of self-use BTM capacities and can be derived using the estimated BTM results and PPA data. Scaling is necessary to improve the convergence of the optimization process.

The BTM BESS capacity was estimated using the B matrix in Figure 9. B is a learnable weight matrix that represents regional inverter capacity, at the same time regional battery capacity can be estimated using Equation (6). The BTM BESS operation was modeled based on the greedy algorithm using previous size estimation. The daily BTM BESS operation can be classified by three time periods. The BTM BESS operation was generated as follows:

For t in 11~16 h:

$$\begin{gathered} B_{in,t}=\sqrt{\eta }·\max (0,\min (C_{battery}-{\displaystyle \sum }_{t=0}^{t-1}{B}_{in,t},B_{PV,t}-lb·C_{PV}, C_{inverter})) \\ {B}_{out,t}=0 \end{gathered}$$

(12)

For t in 18~24 h:

$$\begin{gathered} B_{out,t}=\sqrt{\eta }·\max (0, \min (C_{battery}-{\displaystyle \sum }_{t=0}^t{B}_{out,t} , C_{inverter}-B_{PV,t})) \\ {B}_{in,t}=0 \end{gathered}$$

(13)

otherwise:

$$B_{in,t}=0 and B_{out,t}=0$$

(14)

where E is the energy charged in a day. $B_{in,t}$ and $B_{out,t}$ are the charging and discharging powers, respectively. $\eta$ is the BESS round-trip efficiency. The BESS operating lower bound lb is set to 10%.

After estimating the BTM pattern, depicted as Step I in Figure 8, estimating the expected gain improves the net load forecasting accuracy. The expected accuracy gain is the convolution area between the net load error and estimated BTM pattern, as shown in Figure 10. In this step, the proposed accuracy gain is calculated for all quantiles and all hours in the target period. The proposed accuracy gain can be formulated as follows:

$$G\left(t\right)={\displaystyle \sum }_{t,\tau }g\left(r_{t,\tau },b_t\right)$$

(15)

$$g\left(r_{t,\tau },b_t\right)=\left\{\begin{array}{cc}{b}_t\hfill & \hfill \mathrm{if} \left|r_{t,\tau }\right|\ge \left|b_t\right| \mathit{and} r_{t,\tau }{b}_t\ge 0\\ -b_t\hfill & \hfill if r_{t,\tau }{b}_t\le 0\\ 2r_{t,\tau }-b_t\hfill & \hfill \mathit{otherwise}\end{array}\right.$$

(16)

where $g$ is the proposed accuracy gain function. $r_{t,\tau }$ is an error by quantile and time t. $b_t$ is a BTM pattern at time t.

The gain matrix consists of the accuracy gains per quantile. The quantile results can indicate the confidence level of the weak net load forecasting. BTM capacity is estimated conservatively because the sum of the gain matrix increases with narrower confidence level. This design also improves the convergence of the optimization process.

By maximizing the proposed gain, the proposed weights, such as W and B, are fitted to minimize the net load forecasting errors. This study used the gradient descent method to optimize this problem and provided sufficient iterations for convergence. Through this process, W and B are determined by quantiles. The proposed method can be applied to any length; however, in this study, it was applied monthly.

4.4. Navigation Method Using Beam Search

The beam search method is an expanded form of the greedy search algorithm. By tracking and updating the top-K paths in sequential programming, the beam search method can provide a more effective path than the greedy algorithm. Recently, beam search has been widely used in neural machine translation.

The previous results are only optimal BTM capacity estimates for the target period; thus, the best results are not guaranteed when used for the next estimation. In this study, a beam search algorithm was used to choose more profitable results for overall periods. The proposed navigation method is illustrated in Figure 4. The proposed method keeps the top-K paths and drops the rest. After estimating the next period using the remaining K paths, the method keeps the next top-K paths based on the connected gain and moves on. Finally, this method determines the result path with the highest gain among the remaining top-K paths.

4.5. BTM Pattern Forecasting

In the prediction stage, a model for forecasting the BTM pattern is required. Because the operation of the BTM BESS is determined by the BTM PV pattern, the BTM PV pattern is forecasted first. Then, the pattern of the BTM BESS can be derived using Equations (12)–(14). and the forecasted BTM solar pattern. The proposed BTM PV forecasting model applies a neural network and uses the weather features [25,30]. The details are formulated in Equation (17):

$$B_{pv,t:t+h}=BT{M}_{PV,t}·Model\left(w_{t:t+h}\right)$$

(17)

where $B_{pv,t:t+h}$ is the target BTM solar pattern. The model output was scaled up using the estimated BTM solar capacity. In addition, $w_{t:t+h}$ means a set of observed weather features including solar irradiation, temperature, rainfall, and cloud amount. $f_{t:t+h}$ is for weather forecasts and replace $w_{t:t+h}$ during the training. In this study, the solar forecasting models were fitted for each region and, on the 1st of every month, they were trained for the last 12 months.

4.6. Net Load Forecasting Incorporating with BTM Forecast

From the BTM capacity estimation results, it is possible to restore the gross load from the actual net load and BTM estimation results. The BTM capacity was estimated using a probabilistic method but derived deterministically. Therefore, the gross load may be restored through a linear combination of the net load and BTM pattern. However, simply combining the two forecasting results may increase the error of the individual forecasting models. In addition, it can be estimated that solar power generation will affect the power usage behavior of users who have self-used solar systems. Therefore, the net load forecasting model uses gross load $L$ and BTM pattern $b$ as input data and can be formulated by modifying Equation (7) as follows:

$${\widehat{y}}_t=model\left(L_{t-w:t-1}, b_{t-w:t+h}, o_{t-w:t-1},f_{t-w:t+h},c_{t-w:t+h}\right)$$

(18)

Unlike the previous weak net load forecasting model, this model has an input/output structure using autocorrelation and was implemented by applying a deep neural network technique.

5. Results

In this section, a short-term load forecasting simulation is provided to verify the efficacy of the proposed method. The data set used in this study consisted of observed weather and forecast data, calendar data, and power market data, as detailed in

Table 3. Classification of Distribution Energy Resources Based on Contract Type and Visibility.

Category	Contract Type	Meter Resolution	Capacity Monitoring
Front-the-Meter	Wholesale Market 1	Hourly/Regional	Daily
Behind-the-Meter	PPA	Monthly/Regional	Irregularly
	Private	Monthly/Regional	No

¹ PPA: Power Purchase Agreement.

and

Table 4. Data Resources for BTM Capacity Estimation and Day-Ahead Load Forecasts.

Name	Period	Feature Resolution	Provided Features
Load	2013.01–2020.07	Hourly	Observed Net Load (MW)
Demand Response Outputs	2015.01–2020.07	Hourly	Estimated Demand Response (MW)
FTM Outputs	2015.01–2020.07	Hourly	Integrated solar and BESS outputs by region
FTM Solar Capacity	2015.01–2020.07	Daily
PPA Solar Capacity	2015.01–2020.07	Irregularly
Observed Weather	2013.01–2020.07	Hourly	Temperature, Humidity, Cloud Amount, etc. by region
Forecasted Weather	2013.01–2020.07	3 Hourly

. The net load forecasting model performance is evaluated using the mean absolute percentage error (MAPE) and the standard deviation of MAPE. MAPE is defined as follows:

$$MAPE\left(\%\right)=100\times \frac{1}{n}{\displaystyle \sum }_{t=1}^n\frac{\left|{\widehat{y}}_t-y_t\right|}{y_t}$$

(19)

5.1. Data Description and Applying Demand Response

Historical net load and front-the-meter solar power data were used for the BTM capacity estimation. To increase the contribution of this study, the dataset used actual data from South Korea, as provided by official organizations. Solar data, including capacity, address, and hourly output, are managed by the Korea Power Exchange (KPX) and Korea Electric Power Corporation (KEPCO), according to the contract type. KPX is an isolated system operator and KEPCO is a monopoly utility in South Korea. The details are listed in Table 3.

Hourly load data were provided by KPX and weather data were collected by API service from the Korea Meteorological Administration (KMA). Weather forecast data were also prepared for a realistic numerical simulation. Because of the divided management described in Table 3, the FTM and BTM solar data followed different standards; however, the same regional division was applied to both. The details are listed in Table 4.

This study aims to uniformly model human behavior by using the gross load in a load forecasting model. Demand-side flexible resources such as time-of-use (ToU) and real-time pricing (RTP) can change the shape of the system load, but can be considered part of natural human behavior. On the other hand, direct load control (DLC), which is one of the incentive-based demand-side programs, is operated by the system operator. The participation amount of DLC, which is demand response output in Table 4, is estimated by the system operator. The effect of DLC can be incorporated by adjusting the observed net load. Therefore, the net load used in this study was derived by adding the estimated demand response output to the observed net load.

This simulation is performed in the following steps, as introduced in Section 4. Based on the FTM data, the FTM pattern was decomposed into PV and BESS pattern. As shown in Figure 11, the PPA solar capacity data provided by KEPCO show regional differences, which can be observed to worsen.

5.2. BTM Capacity Estimation

In this subsection, the generated net load forecast error and FTM data between January 2015 and August 2020 were used. To generate the net load forecasting error, a weak forecasting model M1(weak) was used, which comprises four fully connected layers with hidden size = 32. The quantile of the pinball loss was applied with nine values, from 0.1–0.9. Techniques for improving the performance of neural network, such as batch normalization, and hyperparameter optimization, were excluded to show the generality of this study. The Adam optimizer was used for training, with β1 = 0.9, β2 = 0.98, and learning rate = 0.06. The maximum epoch was set to 200 and early stopping was not applied. The BTM capacity was estimated monthly, and its suitability was quantified using the proposed gain. A beam search was performed using k = 3.

The optimal monthly proposed accuracy gain derived from the results of the beam search is shown in Figure 12. In some cases, it can be observed that the proposed gain decreased, and it is interpreted that reflecting the BTM local optimal result in that month does not help to improve overall load forecasting errors. In Figure 13, it can be seen that the distribution of prediction errors was narrowed, and in particular, the errors of positive numbers decreased.

In Figure 13, it can be seen that the error distribution is reduced after considering BTM. The normality of the error distribution can be measured using the Shapiro-Wilk test. According to the Shapiro-Wilk test, if the distribution follows a normal distribution with the null hypothesis, the test statistic decreases, and the p-value is less than the criterion 0.05.

Table 5. Shapiro-Wilk Test results on net load forecasting errors after BTM capacity estimation from May to July 2020.

	Case without BTM		Case Considering BTM
Time Period	Test Statistics	p-Value	Test Statistics	p-Value
7:00–10:00	0.9696	0.0010	0.9767	0.0062
10:00–13:00	0.9977	0.9970	0.9934	0.6418
13:00–16:00	0.9898	0.2725	0.9696	0.0010
16:00–19:00	0.9624	0.0002	0.9615	0.0001
Average	0.9799	0.3174	0.9753	0.1623

shows that the p-values of 13–16H are changed to satisfy normality after considering BTM. Furthermore, average test statistics and p-value are also reduced after considering BTM, so it can be seen that the proposed method is effective in extracting the BTM pattern from the forecasting error. The regional BTM solar capacities were determined from the above results and are described in Figure 14. Comparing Regions 4 and 7, a noticeable result can be observed. Region 7 is the region with the largest PPA penetration rate but the BTM estimate is lower than that of Region 4.

To demonstrate the contributions of this study to the net load forecasting method, the BTM utilization method is applied to each model in five cases: without BTM, estimate BTM pattern using physical PV model as introduced in [4], estimate BTM using a fixed ratio as is currently used by KPX, adding BTM after forecast, and the proposed method, which uses BTM as the input variable. Two additional neural network models were simulated to confirm that the proposed method is generally effective: M2 (linear) comprises four fully connected layers. M3 (LSTM) comprises two long short-term memory layers as the input layer and one fully connected layer as the output layer. Each case was simulated using M1 (weak), M2 (linear), and M3 (LSTM). The test results are presented in

Table 6. Comparison of Day-Ahead Load Forecast Error for Cases by Applying BTM Methods based on Three Different Scenarios.

Scenarios	Cases	M1 (Weak)		M2 (Linear)		M3 (LSTM)
		MAPE (%)	StDev	MAPE (%)	StDev	MAPE (%)	StDev
Scenario A (Base)	Case I	3.78	3.55	3.31	3.22	2.98	1.92
	Case II	3.61	3.37	2.95	2.68	2.86	2.54
	Case III	3.55	3.30	3.04	2.97	2.56	2.47
	Case IV	3.56	3.34	2.96	2.70	2.43	2.25
	Case V (Proposed)	-	-	2.85	2.61	2.35	2.10
Scenario B (Without PPA)	Case I	3.78	3.55	3.31	3.22	2.98	1.92
	Case II	3.69	3.46	3.27	3.10	2.90	2.61
	Case III	-	-	-	-	-	-
	Case IV	3.60	3.48	3.15	2.65	2.85	2.44
	Case V (Proposed)	-	-	3.13	2.67	2.84	2.44
Scenario C (Without ReLU)	Case I	3.78	3.55	3.31	3.22	2.98	1.92
	Case II	3.73	3.46	3.05	2.94	2.94	2.63
	Case III	3.55	3.30	3.04	2.97	2.56	2.47
	Case IV	3.62	3.49	2.92	2.83	2.79	2.50
	Case V(Proposed)	-	-	2.88	2.80	2.58	2.26

.

Five case studies were applied to each model to compare how BTM patterns were incorporated into net load forecasting. In Case I, the model did not use BTM as an input variable or post-correction. In Case II, the BTM pattern was predicted using the estimated BTM capacity and the physical PV model, which is introduced in [4]. The forecasted net load result is obtained through the post-correction of subtracting the BTM pattern from the model output. Case III shows the result of existing method currently used by KPX. In Case III, the hidden BTM PV capacity was estimated by 50% of PPA capacity, and the impact of BTM BESS was ignored. Case IV is similar to Case II, but the proposed BTM pattern prediction model was used instead of the physical PV model. For Case V, the model used the proposed BTM pattern as an input variable and forecasted the net load directly.

In Scenario A, which is base scenario, case studies were applied using PPA data and each BTM PV estimation results were corrected to be equal to or larger than PPA. In addition, under the proposed assumption, BTM capacities were estimated that the BTM capacities are not decreased and thus the ReLU function was used for the proposed process. However, Scenario B assumed that PPA data was not provided, and Scenario C does not used ReLU assuming that BTM capacity could be decreased. Comparing scenarios A and B, it can be observed that using PPA data enables effective BTM estimation, improving the accuracy of day-ahead load forecast. Through comparison between scenarios A and C, it can be shown that proposed assumptions are needed to improve the day-ahead net load forecasting accuracy.

In all scenarios, it can be observed that each model performed best when the BTM pattern was used as the input variable. Adding the BTM pattern also improves the forecasting performance; thus, this result indicates that the proposed approach is reasonable, and it is possible to increase the model accuracy by applying BTM patterns. The results of Case II show that the physical PV model can improve the MAPE of M1, M2 and M3. However, it can be seen from the results that the proposed method is more effective and that the FTM pattern data makes this difference possible. Case III showed the best result in Scenario C, but in Scenario A to which all assumptions were applied, the proposed method was 0.21%p more accurate, showing a distinct difference. Furthermore, Case III has a disadvantage in that it cannot be applied to Scenario B because it depends on PPA data. Through comparison between Case IV and V, the accuracy was more improved when BTM pattern is used as an input variable. From the results in Table 6, it can be concluded that the proposed short-term load forecasting method is improved by applying the BTM patterns.

6. Conclusions

This paper proposes a short-term load forecasting method that incorporates the estimation of behind-the-meter resources, including solar plants, batteries and demand resources. A preprocessing method to reflect the demand response and preprocessing method to decompose front-the-meter data into solar and battery patterns has been proposed.

To reveal the BTM patterns hidden in the net load, a regulated probabilistic net load forecasting model was presented using pinball loss. This paper also proposes a BTM battery size estimation method based on an incentive policy and a data-driven approach. The proposed accuracy gain quantified the effect of the BTM pattern on improving the load forecasting accuracy, allowing the BTM capacity to be estimated through optimization. Each optimization result was connected using a beam-search algorithm to increase the overall efficiency.

The simulation shows that the proposed method is applicable for estimating BTM resources and improves short-term net load model accuracy. The test results are presented using actual utility data in South Korea from January 2015 to August 2020. The regional BTM capacity estimation results obtained by the proposed method had the effect of reducing the uncertainty of the historical net load pattern. As a result of the Shapiro test analysis, the normality of the net load forecast error between 13:00 and 16:00 was significantly improved, reducing the p-value from 0.2725 to 0.0010, and the p-value between 7:00 and 19:00 from 0.3174 to 0.1623. It can be seen from the test results that the proposed BTM capacity estimation effectively detected the BTM pattern hidden in the net load. Using BTM capacity estimation, it can be observed through both the existing and proposed methods that net load forecast accuracy is improved. Comparing the existing methods with the proposed method in various scenarios, the best result of the existing methods was 2.56%, while the proposed method showed clearly improved to 2.35%. Furthermore, the BTM pattern is more suitable when used as an input variable than when used for post-correction. The proposed method was used based on the degree of improvement in net load forecasting accuracy by referring to existing studies to validate the BTM estimation results without actual data points for the BTM capacity. However, the proposed method is different from other studies in that it complementarily improves net load prediction accuracy while estimating the BTM PV and BESSs and corrects the estimation process using auxiliary data. In addition, the results are improved compared to the existing BTM estimation methods.

This study has a limitation that the scheduling of BTM BESS is fitted for the dominant objective function. Considering the various purposes of BTM BESS operation, future research is needed to directly estimate the BTM capacity. The future study will help to increase the visibility of the system considering distributed energy resource operation and improve the load forecasting performance.