Enhancing Probabilistic Solar PV Forecasting: Integrating the NB-DST Method with Deterministic Models

Abstract

Accurate quantification of uncertainty in solar photovoltaic (PV) generation forecasts is imperative for the efficient and reliable operation of the power grid. In this paper, a data-driven non-parametric probabilistic method based on the Naïve Bayes (NB) classification algorithm and Dempster–Shafer theory (DST) of evidence is proposed for day-ahead probabilistic PV power forecasting. This NB-DST method extends traditional deterministic solar PV forecasting methods by quantifying the uncertainty of their forecasts by estimating the cumulative distribution functions (CDFs) of their forecast errors and forecast variables. The statistical performance of this method is compared with the analog ensemble method and the persistence ensemble method under three different weather conditions using real-world data. The study results reveal that the proposed NB-DST method coupled with an artificial neural network model outperforms the other methods in that its estimated CDFs have lower spread, higher reliability, and sharper probabilistic forecasts with better accuracy.

1. Introduction

Solar photovoltaic (PV) generation has been penetrating the power grid at an accelerating speed over the past decade as the world moves toward a more sustainable energy grid. Integrating a large number of solar PV power plants into the existing conventional grid presents significant challenges because solar PV generation is not dispatchable and is often treated as a form of negative load in power grid operations. To efficiently maintain the balance between generation and load in the power grid, both load and solar PV generation need to be forecasted. These forecasts enable the proactive scheduling of dispatchable generation resources, allowing for the management of power ramping and the timely allocation of generation reserves to address power uncertainty [1,2]. Moreover, solar PV power forecasting plays a significant role in predicting energy market imbalances and optimizing bid scheduling [3].

To support the bidding strategy and operation of a day-ahead electric power market, many data-driven forecasting methods have been developed to forecast solar PV generation. Typically, these day-ahead forecasting methods have forecast horizons between 24 and 48 h, with a lead time of 24 h. In other words, the first instance of the forecast is 24 h ahead in the future relative to the current time [4]. Because of the large lead time, all forecasting methods inherently exhibit some levels of forecast errors that cannot be overlooked. Based on how they handle the uncertainty associated with the forecast errors, the forecasting methods can be broadly categorized into deterministic and probabilistic approaches. While deterministic methods provide a single-value forecast without quantifying forecast uncertainty, probabilistic methods quantify forecast uncertainty by estimating statistical properties of forecast variables, such as probability distributions and prediction intervals (PIs) [5].

Many deterministic methods have been developed to accurately forecast solar PV generation, gaining widespread acceptance in real-world applications [6,7,8,9,10]. Well-recognized forecasting methods include artificial neural networks (ANN) [11], recurrent neural networks (RNN) [12], long-short term memory (LSTM) [13], convolutional neural network (CNN) [14], and support vector regression (SVR) [15,16]. Deterministic forecasting methods excel at estimating the expected solar PV generation and are valuable for managing power ramping. Yet, they cannot be directly used to address the uncertainty associated with solar PV generation.

Many probabilistic methods have been developed to quantify the uncertainty inherent in solar PV forecasts. For instance, the persistence ensemble (PerEn) method assumes that the forecasted solar PV generation follows a Gaussian distribution [17] and subsequently estimates its mean and variance. Meanwhile, the ensemble learning method (ELM) [18] integrates various deterministic forecasting methods, including k-nearest neighbors (kNN), decision trees, gradient boosting, random forest, lasso, and ridge regression, to estimate forecast distributions. For deep learning, an improved deep ensemble method, CNN-BiLSTM, is proposed in [19] for probabilistic wind speed forecasting, where the outputs of the final CNN-BiLSTM layer are fitted to a Gaussian distribution to estimate the mean and variance of the wind speed. Quantile regression (QR), on the other hand, estimates the conditional quantiles of solar irradiance based on numerical weather prediction (NWP) data [20]. In addition, an analog ensemble (AnEn) method is developed by [21] and compared to QR and PerEn methods. The AnEn method requires a frozen meteorological model to identify forecast points in the training dataset most similar to the current forecast. Furthermore, Doubleday et al. propose combining several deterministic NWP model outputs using the Bayesian model averaging (BMA) technique to estimate forecast distributions [22]. In [23], Gaussian process regression (GPR) with specific kernel functions is applied to obtain prediction intervals of solar power forecast. This method tries to quantify the forecast uncertainty by fitting the forecast errors to a standard normal distribution. A sparse GPR method is proposed in [24] for probabilistic wind-gust forecasting by combining NWP data and on-site measurements. A variant of this method has shown promise for solar power forecasting of geographically sparse distributed solar plants [25]. The uncertainty quantified by probabilistic forecasting methods serves as a valuable guide for operators in booking appropriate generation reserves.

However, these probabilistic forecasting methods have certain limitations and challenges. For example, the AnEn, ELM, CNN-BiLSTM, and BMA rely on the assumption that a predefined family of probabilistic distributions can adequately characterize their forecasts. The GPR-based approaches, however, non-parametrically forecast the solar power generation but fit the forecast error to a standard normal distribution to quantify the uncertainty. As a result, discrepancies between actual and assumed distributions may lead to significant errors. In addition, both the ELM and BMA methods require forecast outputs from several NWP models, which is time-consuming and may introduce additional uncertainty due to variation across models. As such, these methods have not gained widespread adoption in practical applications.

To overcome the above-mentioned limitations inherent in existing probabilistic forecasting methods and leverage the widespread acceptance of deterministic forecasting methods, this paper proposes a data-driven non-parametric statistical method for day-ahead probabilistic forecasting of PV power [26]. First, the method estimates the probability of the forecast error falling into a certain interval using the Naïve Bayes (NB) classification technique. Then, the estimated evidence is combined using a suitable statistical inference technique. Bayesian inference is a well-known method for combining independent evidence and has been utilized for many probabilistic forecasting applications, such as wind-gust speed forecasting [27]. However, it requires prior probabilities. To address this limitation, a frequentist approach known as the Dempster–Shafer theory (DST) is utilized in this paper to combine probabilities from various independent sources, specifically from NB’s outputs. Hence, the proposed method is named the NB-DST method. As a non-parametric approach, the proposed NB-DST method provides the flexibility needed for real-world applications. Distinguishing itself from other probabilistic methods such as BMA and ELM, the proposed NB-DST method only requires a single deterministic model output. A similar method is utilized for probabilistic wind-power forecasting, which uses sparse Bayesian classification networks that require proper initialization of hyperparameters and a large training dataset [28]. In comparison, the NB classifiers used in this paper are easier to train with a small training dataset and outperform other methods like logistic regression [29].

The proposed NB-DST method is a novel approach in the landscape of probabilistic solar PV power forecasting. Unlike traditional methods that rely on predefined probabilistic distributions or extensive model ensembles, the NB-DST method uniquely combines NB classification with DST to offer a non-parametric scalable solution for quantifying forecast uncertainty. This method not only addresses the limitations of existing probabilistic approaches by eliminating the need for prior probability distributions but also enhances adaptability to real-world forecasting scenarios. The purpose of this research is to demonstrate the efficacy and practicality of the NB-DST method in improving the accuracy and reliability of day-ahead solar power forecasts, thereby supporting more informed decision making in grid management and energy market operations.

The structure of the rest of this paper is as follows: Section 2 outlines the proposed NB-DST method. The performance evaluation metrics are introduced in Section 3. In Section 4, the performance of the proposed method is compared with other methods utilizing real-world data from a rooftop solar PV plant. Finally, conclusions are drawn and future work is discussed in Section 5.

2. NB-DST Method for Day-Ahead Solar PV Forecasting

This section introduces the proposed NB-DST method to estimate the cumulative distribution function (CDF) of day-ahead solar PV forecasts. The application of the proposed NB-DST method consists of the following four steps: (1) collect historical solar PV generation and related weather data; (2) apply a deterministic forecasting method such as SVR and ANN to forecast solar PV generation and determine the forecast errors; (3) utilize the NB method to determine the probability of the forecast errors falling into probability intervals; and (4) employ the DST method to determine the CDF of forecast errors. The procedure of applying the NB-DST method to quantify the day-ahead solar PV forecast errors is detailed as follows.

2.1. Data Collection

To perform day-ahead solar PV forecasting, day-ahead weather forecast, and solar power generation data are collected. For a particular geographic location, historical weather forecast data (${\mathit{w}}_h^d$) and historical solar power generation data ($s_h^d$) for the targeted hour (h) and day (d) are collected. After identifying and removing bad data [30], the dataset is divided into three subsets (see Figure 1): (1) training dataset for building deterministic forecasting models (about 75%), i.e., $T{r}_1=\left\{{\mathit{w}}_h^d, s_h^d | d\in D_1\right\}$, (2) training dataset for building the NB models (about 15%), i.e., $T{r}_2=\left\{{\mathit{w}}_h^d, s_h^d | d\in D_2\right\}$, and (3) testing dataset for evaluating the performance of the NB-DST method (about 10%), i.e., $TS=\left\{{\mathit{w}}_h^d, s_h^d | d\in D_3\right\}$.

2.2. Deterministic Forecasting Methods and Forecast Errors

Using the training dataset Tr₁ as the input, deterministic forecasting methods aim to build models for forecasting solar PV generation 24 h in advance: ${\widehat{s}}_h^d=f_h^{wc}\left({\mathit{w}}_h^d, s_h^{d-1}\right)$. Here, the output of the deterministic forecasting methods is a forecasting model, denoted as $f_h^{wc}(·)$, which may include models used by SVR, ANN, and persistence methods. To catch hourly variations, separate models are trained for each hour ($h$) of the day, using historical data for the corresponding hour.

Furthermore, three distinct forecasting models are trained to accommodate different weather conditions ($wc$): clear, overcast, and partially cloudy. These trained forecasting models generate hourly day-ahead deterministic forecasts (${\widehat{s}}_h^d$) using dataset Tr₂. Then, forecast errors (${\epsilon }_h$) can be calculated using (1) as the difference between the observations ($s_h^d$) and forecasts.

$${\epsilon }_h^d=s_h^d-{\widehat{s}}_h^d$$

(1)

To apply the NB method, ${\epsilon }_h^d$ is cast into equally divided intervals. First, the lower and upper limits of historical forecast errors are determined as ${\xi }_h^{min}=\underset{d\in D_2}{\min }{\epsilon }_h^d$ and ${\xi }_h^{max}=\underset{d\in D_2}{\max }{\epsilon }_h^d$, respectively. Then, the historical error range $\left[{\xi }_h^{min},{\xi }_h^{max}\right]$ is divided into $L$ intervals of equal width. These intervals are defined as $I_h^l=interval\left(Lo{w}_h^l, U{p}_h^l\right)$, where $Lo{w}_h^l={\xi }_h^{min}+\left(l-1\right){\Delta }_h$ and $U{p}_h^l={\xi }_h^{min}+l{\Delta }_h$ for l = 1, 2, …, L and ${\Delta }_h=\frac{{\xi }_h^{max}-{\xi }_h^{min}}{L}$.

Finally, an NB binary classifier model is trained for each error interval at hour $h$. This approach is chosen because the results of each NB model will serve as evidence for the DST. Training a single multi-class NB classifier would result in only one mass function, where estimated probabilities for each class sum to ‘1’ and individual class probabilities do not form meaningful evidence for the DST. To train the NB classifier for hour h, the day-ahead weather forecast data (${\mathit{w}}_h^d$) and solar power forecast data (${\widehat{s}}_h^d)$ are used as predictors (${\mathit{x}}_h^d={\left[{\mathit{w}}_h^d, {\widehat{s}}_h^d\right]}^T$). The target is represented as a binary class label, $\left\{0, 1\right\}$, where ‘0’ represents the event that the forecast error falls outside the corresponding interval and ‘1’ represents the event that the forecast error falls inside the corresponding interval. Mathematically, the labeling can be written as follows:

$$y_h^l=𝟙\left({\epsilon }_h^d\in I_h^l\right)=\{\begin{array}{c}1 if {\epsilon }_h^d\in I_h^l\\ 0 otherwise\end{array}$$

(2)

where $y_h^l$ is the class label of a training sample of the $l^{th}$ NB classifier of hour $h$ and $𝟙(·)$ is an indicator function.

NB classifiers are trained using dataset Tr₂ and validated through the k-fold cross-validation technique [31]. For the $l^{th}$ interval at hour $h$, the NB classifier provides two probabilities: one for the forecast falling into the interval ($p_h^l$) and another for it falling out of the interval ($1-p_h^l$), where $l\in \left\{1, 2, \dots , L\right\}$. This procedure is illustrated in Figure 2.

2.3. Probability Estimation with the NB Method

The NB method, a straightforward probabilistic classification algorithm based on Bayes’ theorem of probabilistic inference [32], serves as a crucial tool in this study. Bayes’ theorem connects the conditional and marginal probabilities of two random variables and is often used to calculate posterior probabilities based on given data. In this study, the NB method is used to estimate $P>({\epsilon }_h^d\in I_h^l | {\mathit{x}}_h^d)$, i.e., the probability of forecast errors ${\epsilon }_h^d$ falling into interval $I_h^l$ given the predictor feature vector ${\mathit{x}}_h^d$.

To be concise, the following study will be focused on interval l at hour h of day d, so that the superscripts l and h can be dropped. Assume that the training data $\left\{{\mathit{x}}^{\mathit{d}},y^d\right\}$ are given for $d=1, 2, \dots , N_2$. Here, $N_2$ is the total number of days in Tr₂. Symbol ${\mathit{x}}^{\mathit{d}}={\left[x_1^d, x_2^d,\cdots ,x_v^d,\cdots , x_V^d\right]}^T$ is a $V$-dimensional feature vector, which consists of day-ahead weather forecast ${\mathit{w}}^d$ and solar power forecast ${\widehat{s}}^d$. Symbol $y^d$ is the corresponding class label defined in (2). An NB classifier computes the probabilities that an unclassified sample belongs to certain class label(s) conditioned on the feature values using Bayes’ theorem, as depicted in (3).

$$P\left(Y=C|\mathit{X}=\mathit{x}\right)=\frac{f_{X|Y=C}\left(\mathit{x}\right)·P\left(Y=C\right)}{f_{X|Y=0}\left(\mathit{x}\right)·P\left(Y=0\right)+f_{X|Y=1}\left(\mathit{x}\right)·P\left(Y=1\right)}$$

(3)

Here, $C ϵ \left\{0,1\right\}$ is the class label, $f_{X|Y=C}\left(x\right)$ is the probability density function (PDF) of feature vector $\mathit{x}$ given that $Y$ attains class label $C$, and $P\left(Y=C\right)$ is the prior probability of class $C$. It is assumed that the prior probability follows a Bernoulli distribution.

The NB classifier assumes conditional independence of the features, meaning that the value of a particular feature is considered independent of other features for a given class, as expressed in (4).

$$f_{X|Y=C}\left(\mathit{x}\right)={\displaystyle \prod }_{v=1}^V{f}_{X_v|Y=C}\left(x_v\right)$$

(4)

Although this assumption may not always hold in some practical applications, NB classifiers, when coupled with kernel density estimation (KDE), can still achieve accurate estimations in such cases [32]. KDE is a non-parametric estimation technique for estimating PDF, which can be described by (5).

$${\widehat{f}}_{\mathit{X}}\left(\mathit{x}\right)=\frac{1}{N_2\lambda }{\displaystyle \sum }_{d=1}^{N_2}K\left(\frac{\mathit{x}-{\mathit{x}}^d}{\lambda }\right)$$

(5)

Here, ${\widehat{f}}_{\mathit{X}}\left(\mathit{x}\right)$ represents the KDE of the PDF of a random variable $\mathit{X}$ given $N_2$ realizations of $x$, $\left[x_1, x_2, \dots , x_{N_2}\right]~f_X\left(x\right)$. The smoothing parameter $\lambda$ is also known as bandwidth. Additionally, $K(·)$ denotes the kernel function. A commonly used choice for this function is the Gaussian kernel, which is defined in (6).

$$K\left(\frac{\mathit{x}-{\mathit{x}}^d}{\lambda }\right)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{\frac{\mathit{x}-{\mathit{x}}^d}{\lambda }}^2}$$

(6)

The KDE of the conditional PDF of the feature vector $\mathit{X}$ given Y = C can be obtained through (7). Here, $N_C$ is the number of training samples belonging to class $C$ and ${\lambda }_v^C$ is the bandwidth for the $v^{th}$ feature for class $C$ and selected to optimize the Gaussian kernel in (6).

$${\widehat{f}}_{X|Y=C}\left(\mathit{x}\right)={\displaystyle \prod }_{v=1}^V\left\{\frac{1}{N_C{\lambda }_v^C}{\displaystyle \sum }_{n_C=1}^{N_C}K\left(\frac{x_v^{n_c}-x_v^{n_c}}{{\lambda }_v^C}\right)\right\}$$

(7)

An NB classifier for a specific error interval is trained by simultaneously simulating (3) and (7) for each training sample of the corresponding interval of a particular hour in dataset Tr₂. The output probabilities of an NB classifier for the $l^{th}$ interval of hour $h$ can be written as (8) and (9).

$$p_h^l=P\left(Y=1|\mathit{X}=\mathit{x}\right)$$

(8)

$$1-p_h^l=P\left(Y=0|\mathit{X}=\mathit{x}\right)$$

(9)

2.4. Review of the DST

The DST is a mathematical framework employed to combine evidence from multiple independent sources in the presence of uncertainty [26,33]. The DST operates by collecting information from individual sources represented by a mass function, also known as (a.k.a.) the basic probability assignment (BPA). The BPAs are then merged using Dempster’s rule of combination to determine the degrees of belief in a set of events of interest.

When formulating a problem for DST application, a set of all possible solutions, a.k.a. the frame of discernment, is considered. It is assumed that the events of interest are subsets of this frame. Let $\mathsf{\Omega }$ represent frame of discernment and $E$ represent an event of interest, where $E\subseteq \mathsf{\Omega }$. Then, $\overline{E}\cong \mathsf{\Omega }-E$ represents the events in $\mathsf{\Omega }$ that are distinct from event $E$. The BPA for $\mathsf{\Omega }$ is a function $m: 2^{\mathsf{\Omega }}\to \left[0,1\right]$ that has the following two properties:

(1)

$m\left(\varphi \right)=0$, where $\varphi$ denotes the empty set

(2)

${{\displaystyle \sum }}_{Eϵ2^{\mathsf{\Omega }}}m\left(E\right)=1$ where $m\left(E\right)$ is the degree of belief assigned to event $E$.

Let $m_1, m_2,\dots , m_z$ represent the BPAs for independent evidence $F_1, F_2, \dots , F_z$ in support of event $E$ from $\mathsf{\Omega }$. Dempster’s rule of combination can be exploited to construct a new mass function for combined evidence in support of $E$ using (10).

$$m\left(E\right)=\left(m_1\oplus m_2\oplus \dots \oplus m_z\right)\left(E\right)=\frac{1}{U}{\displaystyle \sum }_{F_1{\displaystyle \cap }{F}_2{\displaystyle \cap } \dots {\displaystyle \cap }{F}_z=E}{m}_1\left(F_1\right)·m_2\left(F_2\right)· \cdots ·m_z\left(F_z\right)$$

(10)

Here, $\oplus$ denotes the operator for Dempster’s rule of combination, which performs orthogonal sum of mass functions supporting event $E$. And $U$ is a normalization constant representing the degree of conflict between individual BPAs, as described by (11).

$$U={\displaystyle \sum }_{F_1{\displaystyle \cap }{F}_2{\displaystyle \cap } \dots {\displaystyle \cap }{F}_z\ne \varphi }{m}_1\left(F_1\right)·m_2\left(F_2\right)· \cdots ·m_z\left(F_z\right)$$

(11)

2.5. Incorporating Evidence from NB Classifiers Using the DST

The NB classifiers provide individual probabilities indicating the likelihood of forecast errors falling within a specific interval. To accurately determine the final probability of an error falling into a specified interval, DST is used to combine the evidence that the error falls into that interval with the evidence that error does not fall into any other intervals. Because there are $L$ possible intervals for the error at hour $h$, there are $L$ possible events, which are denoted as $E_1^h, E_2^h, \dots ,E_l^h, \dots , E_L^h$. Here, $E_l^h$ represents the event that the error falls into interval $I_h^l$, where $\left\{l\in {\mathbb{Z}}^{+}:1\le l\le L\right\}$. The frame of discernment at hour $h$, denoted by ${\mathsf{\Omega }}_h$, can be written as (12).

$${\mathsf{\Omega }}_h=E_1^h{\displaystyle \cup }{E}_2^h{\displaystyle \cup }\dots {\displaystyle \cup }{E}_l^h{\displaystyle \cup }\dots {\displaystyle \cup } E_L^h$$

(12)

The mass functions of these individual events, also known as BPAs, are obtained from the NB classifiers. The mass function $m_l\left(E_l^h\right)$ of the event $E_l^h$ is defined in (13)–(14) for all values of $l$. Here, $\overline{E_l^h}$ denotes the event that the error does not fall into the $l^{th}$ interval.

$$m_l\left(E_l^h\right)=p_h^l$$

(13)

$$m_l\left(\overline{E_l^h}\right)=1-p_h^l$$

(14)

Because solar generation is inherently non-negative and cannot exceed the generation capacity of the solar power plant, the constraint described by (15) is imposed on solar power generation. Here, $s_{cap}$ represents the maximum generation capacity of the solar power plant.

$$0\le s_h\le s_{cap}$$

(15)

The range of the forecast error is then established using (16) and (17). The bounds on the index of error interval are set through (18)–(19). Here, $\lceil ·\rceil$ represents the ceiling operation and $l_{lower}$ and $l_{upper}$ are the indices of the lowermost and uppermost error intervals such that $1\le l_{lower}<\dots <l_{upper}\le L$.

$$0\le {\widehat{s}}_h+{\epsilon }_h\le s_{cap}$$

(16)

$$-{\widehat{s}}_h\le {\epsilon }_h\le s_{cap}-{\widehat{s}}_h$$

(17)

$$l_{lower}=\lceil \frac{-{\widehat{s}}_h-{\xi }_h^{min}}{{\Delta }_h}\rceil$$

(18)

$$l_{upper}=L-\lceil \frac{{\xi }_h^{max}-\left(s_{cap}-{\widehat{s}}_h\right)}{{\Delta }_h}\rceil$$

(19)

The frame of discernment is updated according to the lowermost and uppermost error intervals, as shown in (20).

$${\mathsf{\Omega }}_h=E_{l_{lower}}^h{\displaystyle \cup }\dots {\displaystyle \cup } E_{l_{upper}}^h$$

(20)

Here, instead of combining the evidence from the whole set of error intervals, the frame of discernment is shrunk to a subset of $M$ intervals, where $M=l_{upper}-l_{lower}+1$ and $M\le L$. Next, integrate these pieces of evidence using (10) and (11) to determine the probabilities of the forecast errors that fall into each of these $M$ intervals, which constitute the mass function $m\left(E_i^h\right) \forall i:l_{lower}\le i\le l_{upper}$, as illustrated in (21).

$$m\left(E_i^h\right)=\left(m_{l_{lower}}\oplus m_{l_{lower}+1}\oplus \dots \oplus m_{l_{upper}}\right)\left(E_i^h\right)$$

(21)

The CDF of the forecast error is derived by accumulating the probabilities. The set of error values $\left\{{\epsilon }_h^1, {\epsilon }_h^2, \dots , {\epsilon }_h^k, \dots , {\epsilon }_h^M\right\}$ and their corresponding probabilities $\{{\widehat{p}}_h^1, {\widehat{p}}_h^2, \dots , {\widehat{p}}_h^k, \dots ,{\widehat{p}}_h^M\}$ for the CDF is calculated using the interval bounds and the width, as outlined in (22)–(25).

$${\epsilon }_h^1={\xi }_h^{min}+\left(l_{lower}-1\right) {\Delta }_h$$

(22)

$${\epsilon }_h^k={\epsilon }_h^1+\left(k-1\right){\Delta }_h\mathrm{for}k=2, 3, \cdots , M$$

(23)

$${\widehat{p}}_h^1=m\left(E_{l_{lower}}^h\right)$$

(24)

$${\widehat{p}}_h^k={\widehat{p}}_h^{k-1}+m\left(E_{l_{lower}+k-1}^h\right)\mathrm{for}k=2, 3, \cdots , M$$

(25)

Finally, the CDF of the solar generation at hour $h$ is calculated by adding the day-ahead deterministic forecast to the error values for hour $h$, as shown in (26). Here, ${\widehat{S}}_h^k$ represents the NB-DST forecast whose CDF value is ${\widehat{p}}_h^k$.

$${\widehat{S}}_h^k={\epsilon }_h^k+{\widehat{s}}_h\mathrm{for}1\le k\le M$$

(26)

The framework of the NB-DST method is summarized in Figure 3. In this study, the NB-DST method is coupled with three deterministic forecasting methods: SVR, ANN, and QR (median only), resulting in SVR-NB-DST, QR-NB-DST, and ANN-NB-DST models, respectively. To mitigate the risk of overfitting the NB-DST models to the training data, a separate dataset, TS, is reserved for generating performance metrics to evaluate the models’ effectiveness.

3. Performance Metrics for Evaluating Probabilistic Forecasts

Assessing probabilistic forecasts is more challenging than deterministic ones because probabilistic forecasts have various statistical properties. As can be found in the literature, many state-of-the-art probabilistic forecasting methods are only assessed using certain sets of evaluation metrics. But the evaluation process needs to balance several critical aspects, such as forecast accuracy, reliability, sharpness, and uncertainty. This paper aims at addressing this challenge by evaluating the probabilistic forecasting methods using a handful of metrics that assess distinct aspects of the forecasts, providing the forecasters with a framework for critical evaluation. In this section, these evaluation metrics are defined to evaluate the performance of the probabilistic forecasts.

3.1. Continuous Rank Probability Score

The continuous rank probability score (CRPS) is an evaluation metric widely used in various forecasting domains, such as weather, solar and wind power, and load forecasting [5,34,35]. It effectively measures forecast accuracy, reliability, and sharpness. CRPS is defined in (27).

$$\mathrm{CRPS}=\frac{1}{N}{\displaystyle \sum }_{k=1}^N{{\displaystyle \int }}_{-\infty }^{+\infty }{\left[{\widehat{p}}_{fcst}^k\left(x\right)-o_{s_h}^k\left(x\right)\right]}^{2}dx$$

(27)

Here, ${\widehat{p}}_{fcst}^k\left(x\right)$ represents the CDF of the forecast at instant $k$. Symbol $o_{s_h}^k\left(x\right)$ denotes the CDF of the observation at instance $k$, which is defined as $o_{s_h}^k\left(x\right)=𝟙\left(x\ge s_h\right)$. Here, $𝟙(·)$ is an indicator function and $s_h$ denotes the observed value of solar generation. $N$ stands for the total number of forecast instances in dataset TS.

A lower CRPS value is desirable as it indicates that the estimated CDF has a lower spread around the observation. It is evident from (27) that the CRPS penalizes any discrepancies between the forecasted CDF and the actual observations. Moreover, for a deterministic forecast, the CRPS transforms into the mean absolute error (MAE). This property makes the CRPS a preferred metric for comparing probabilistic and deterministic forecasts. CRPS can be further decomposed into three components as outlined in (28) [36]. The methods for calculating CRPS, along with its components—REL, UNC, and RES—are detailed in [36] and used in this paper.

$$\mathrm{CRPS}=\mathrm{Reliability}\left(\mathrm{REL}\right)+\mathrm{Uncertainty}\left(\mathrm{UNC}\right)-\mathrm{Resolution}\left(\mathrm{RES}\right)$$

(28)

3.2. Brier Score

The Brier score (BS) is a scoring function widely used to measure the accuracy of probabilistic forecasts [37]. The BS for a probabilistic forecast at hour $h$ is defined in (29).

$${\mathrm{BS}}_h=\frac{1}{M}{\displaystyle \sum }_{j=1}^M{\left[{\widehat{p}}_h^j-y_h^j\right]}^2$$

(29)

Here, ${\widehat{p}}_h^j$ represents the forecasted probability of event $j$ happening at hour h. Term $y_h^j$ is defined as $y_h^j=𝟙\left(s_h\ge {\widehat{S}}_h^j\right)$. $M$ denotes the total number of events considered in the probability forecast at hour $h$. To compute the BS for an entire day’s forecast, one can average $B{S}_h$ across all hours, from $h=1 to N_h$, where $N_h$ represents the total number of hours of the day.

3.3. Prediction Interval Coverage Probability

After the CDF of the forecast is obtained from a probabilistic forecasting method, a prediction interval (PI) of the forecast can be calculated. For a confidence level of $\left(1-\alpha \right)\times 100\%$, the PI is determined using (30). Here, ${\widehat{S}}_{h,\tau }$ represents the ${\tau }^{th}$ quantile of the forecasted CDF, meaning $P(S_h\le {\widehat{S}}_{h,\tau })=\tau$.

$${\widehat{PI}}_{h,\left(1-\alpha \right)\times 100\%}=Interval\left({\widehat{S}}_{h,\tau =\frac{\alpha }{2}},{\widehat{S}}_{h,\tau =1-\frac{\alpha }{2}}\right)$$

(30)

The prediction interval coverage probability (PICP) is a measure of the reliability of a probabilistic forecasting method. PICP can be calculated from the $\left(1-\alpha \right)\times 100\%$ PI as outlined in (31) [38].

$$\mathrm{PICP}\left(\alpha \right)=\frac{1}{N_h}{\displaystyle \sum }_{h=1}^{N_h}𝟙\left({\widehat{s}}_h\in {\widehat{PI}}_{h,\left(1-\alpha \right)\times 100\%}\right)$$

(31)

Here, ${\widehat{s}}_h$ denotes the forecast at hour $h$, $N_h$ denotes the total number of forecasting hours in a day, and $𝟙(·)$ is an indicator function. A forecasting method is considered more reliable if its $\mathrm{PICP}\left(\alpha \right)$ is closer to the nominal coverage for the confidence intervals of the same size.

3.4. Prediction Interval Normalized Average Width

The prediction interval normalized average width (PINAW) is a metric for measuring the sharpness of a probabilistic forecast method [38]. Mathematically, PINAW is defined in (32), where $s_h$ denotes the observed solar generation at hour $h$ on the forecast day. In this formula, the numerator computes the average width of the $\left(1-\alpha \right)\times 100\%$ PI of the probabilistic forecast, while the denominator computes the average hourly solar power generation. A lower PINAW value is desirable, as it indicates a shaper forecast.

$$\mathrm{PINAW}\left(\alpha \right)=\frac{\frac{1}{N_h}{{\displaystyle \sum }}_{h=1}^{N_h}\left({\widehat{S}}_{h,\tau =1-\frac{\alpha }{2}}-{\widehat{S}}_{h,\tau =\frac{\alpha }{2}}\right)}{\frac{1}{N_h}{{\displaystyle \sum }}_{h=1}^{N_h}{s}_h}$$

(32)

3.5. Coverage Width Criterion

The coverage width criterion (CWC) is defined in (33) [39] and can effectively address the limitations of both the PICP and PINAW. While PICP values closer to the nominal coverage indicate a more reliable forecast, they do not assess the sharpness of the forecast. Conversely, a lower PINAW signifies a sharper forecast but does not measure its reliability. The CWC integrates both PICP and PINAW, providing a comprehensive metric to evaluate the reliability and sharpness of a forecast simultaneously.

$$\mathrm{CWC}=\{\begin{array}{cc}\beta ·\mathrm{PINAW}& for \mathrm{PICP}\ge \mathrm{PINC}\\ \left(\kappa +\beta \cdot \mathrm{PINAW}\right)\cdot \left(1+exp\left(-\eta \cdot \frac{\mathrm{PICP}-\mathrm{PINC}}{100}\right)\right)& for \mathrm{PICP}<\mathrm{PINC}\end{array}$$

(33)

Here, $\mathrm{PINC}$ represents the prediction interval nominal coverage. Symbols $\mathsf{\kappa },\mathsf{\beta },$ and $\mathsf{\eta }$ represent hyperparameters, which are chosen as suggested in [39]. A lower CWC value is desirable, as it indicates that a probabilistic forecasting method is more reliable and sharper.

4. Case Study

In this section, day-ahead probabilistic forecasting of solar PV generation is performed using the proposed NB-DST method. In addition to the SVR-NB-DST, QR-NB-DST, and ANN-NB-DST methods, the study also implements the AnEn and PerEn methods [21] as benchmarks. Although there are many probabilistic methods in the literature, they have not been widely adopted in practice. For example, the PerEn method is still preferred for benchmarking within both industry settings and research environments, such as those conducted by the US Department of Energy [40]. The effectiveness of these methods is evaluated and compared using the performance metrics outlined in Section 3 using real-world data.

4.1. Data Collection and Selection

The solar generation data used in this study are collected from a 450-kW solar PV plant located on the rooftop of the Target^® store in Vestal, NY, USA (latitude 42°05′37.0″ N, longitude 76°00′06.0″ W). This plant was established in 2016 with support from the New York State Energy Research and Development Authority (NYSERDA). Hourly solar generation data of this plant between 2016 and 2022 are downloaded from NYSERDA’s website [41].

Additionally, this study incorporates various weather variables that significantly influence solar irradiance [42]. Historical weather data for the study site from 2016 to 2021, along with historical forecast data for 2022, are collected on an hourly basis from the website of Visual Crossing weather data service [43]. The weather variables available in this dataset are listed in

Table 1. Weather variables for the study site are available from the Visual Crossing weather data service (2016–2022).

Weather Variable	Description	Unit
Temperature	Actual temperature	°C
Dew point	Dew point	°C
Wind chill	Wind chill factor	°C
Heat index	Heat index	°C
Feels like	Combination of temperature, wind chill, and heat index	°C
Humidity	Relative humidity	%
Precipitation	Amount of liquid equivalent precipitation	mm
Snow depth	Average amount of snow on the ground	cm
Wind speed	Wind speed at 10 m above ground	kmph
Wind direction	Direction of wind in reference to the north direction	degrees
Pressure	Sea level pressure	mbars
Cloud cover	Amount of sky covered with cloud	%
Visibility	Distance visible in daylight	km
Conditions	Weather condition reported by the weather station	none

. To select the predictor variables for the forecasting algorithms, the correlation between solar generation and each numerical weather variable is measured using Pearson’s correlation coefficient (PCC) [44]. After pre-processing and removing outliers from the data, the PCC for each numerical weather predictor variable with solar generation data is calculated. The results are presented in Figure 4. It is observed that temperature and visibility have a strong positive correlation with solar power generation, while relative humidity and cloud cover have a strong negative correlation. The correlations of other predictor variables are weak, as shown in Figure 4, and thus, these variables are excluded from the model. Solar generation data from the previous day’s corresponding hours are also included in the model to account for the auto-correlation of the solar power generation data.

The datasets used in this study are divided as follows: the data from 2016 to 2019 are designated as the Tr₁ dataset, while data recorded from 2020 to 2021 are used as the Tr₂ dataset. The data recorded in 2022 are used as the TS dataset. Because weather conditions are complex and have varying impacts on solar PV generation, separate forecasting models are built and evaluated under different cloud patterns. Accordingly, the datasets are further divided into three categories on a daily basis according to the weather conditions using the approach described in [45]: “clear”, “overcast”, and “partially cloudy”. The daily solar generation profile on a clear day typically exhibits a smooth bell curve, with the power generation peaking near the generation capacity during noon. On overcast days, the total daily solar generation is significantly lower. On partially cloudy days, the hourly solar generation shows considerable fluctuation, characterized by multiple peaks and troughs throughout the day.

Subsequently, the forecasting models are first trained using Tr₁ and Tr₂ datasets for each hour of the day under these distinct weather conditions. Then, the forecasting models are evaluated using the performance metrics computed through the TS dataset. Note that out of 365 days in the TS dataset, there are 124 days classified as “clear”, 129 days classified as “overcast”, and 112 days classified as “partially cloudy”. The effectiveness of the forecasting models is evaluated using a $5$-fold cross-validation approach.

4.2. Performance Evaluation under “Clear” Weather Conditions

This subsection focuses on assessing the probabilistic forecasting methods during “clear” weather conditions. The performance metrics for the 124 “clear” days in the TS dataset are calculated for all the methods and summarized in

Table 2. Performance metrics of all the forecasting methods evaluated over the 124 “Clear” days.

Method	CRPS (%)	CRPS Decomposition			BS (%)	PICP (%)	PINAW	CWC
		REL	RES	UNC
PerEn	11.14	1.20	48.38	58.32	6.87	91.62	19.87	119.32
AnEn	15.38	1.51	44.45	58.32	7.44	84.71	14.67	88.12
QR-NB-DST	7.86	1.58	52.04	58.32	5.73	87.18	9.33	56.08
SVR-NB-DST	7.97	1.41	51.76	58.32	5.41	86.35	9.81	58.96
ANN-NB-DST	6.28	1.38	53.42	58.32	4.62	88.42	7.25	43.60

. It can be observed from Table 2 that the three NB-DST methods outperform the AnEn and PerEn methods in terms of the CRPS. Also, the BS values for the NB-DST methods are considerably lower than those for the other methods, which indicates NB-DST’s higher accuracy. These three NB-DST methods show satisfactory reliability and coverage with resolution metrics notably surpassing those of the AnEn and PerEn methods. Also, a slightly lower value of the PINAW for the ANN-NB-DST gives it an advantage over the other two NB-DST methods. While the PICP values for the three NB-DST methods are comparable to those for the AnEn and PerEn methods, the NB-DST methods, particularly ANN-NB-DST, exhibit superior overall performance as indicated by their lower CWC values.

To further illustrate the performance, Figure 5 shows the estimated 95% PIs from all the forecasting methods for a particular “clear” day (19 May 2022). The 95% PIs are obtained using (22) by setting α = 0.05. It can be observed that the 95% PIs of the three NB-DST methods are narrower than those of the two benchmark methods. Figure 6a shows the CDF estimate for PV power at 10:00 AM on this “clear” day (19 May 2022). It can be observed that the CDFs estimated using the ANN-NB-DST and the QR-NB-DST methods exhibit the lowest deviation from the actual observation CDF. Although the CDF estimated by the SVR-NB-DST method shows a deviation slightly higher than other NB-DST methods, it still performs commendably when compared to the benchmark methods.

4.3. Performance Evaluation under “Overcast” Weather Conditions

In this subsection, the performance metrics are calculated over the 129 “overcast” days in the TS dataset for all the probabilistic forecasting methods and are summarized in

Table 3. Performance metrics of all the forecasting methods evaluated over the 129 “Overcast” days.

Method	CRPS (%)	CRPS Decomposition			BS (%)	PICP (%)	PINAW	CWC
		REL	RES	UNC
PerEn	5.89	0.51	7.33	12.71	4.73	91.68	1316.20	7897.3
AnEn	6.21	0.54	7.04	12.71	5.61	89.54	1353.90	8123.5
QR-NB-DST	5.74	0.48	7.45	12.71	3.68	86.61	632.80	3796.9
SVR-NB-DST	6.78	0.39	6.32	12.71	3.87	85.47	670.94	4025.7
ANN-NB-DST	5.37	0.41	7.75	12.71	3.13	88.25	581.42	3488.6

. Among these methods, the ANN-NB-DST stands out by achieving the lowest CRPS. It also exhibits superior reliability and resolution, as indicated by its REL and RES values, compared to the other methods. Notably, the resolution of all the NB-DST methods is superior to the two benchmark methods as their PINAW values are much lower. While the PICP values for the NB-DST methods are further from the ideal 95% nominal coverage than the benchmark methods, their lower CWC values reflect better overall performance considering both forecast reliability and sharpness. Additionally, the lower BS values of the NB-DST methods suggest that the NB-DST methods are more accurate than the other methods. Among the three NB-DST methods, the ANN-NB-DST method exhibits the most impressive performance.

To further illustrate the performance, Figure 7 shows the 95% PIs obtained from the forecasting methods on a particular “overcast” day (28 December 2022). It can be observed that the 95% PIs for the AnEn and PerEn methods are notably broader than those for the NB-DST methods. Additionally, the SVR-NB-DST method meets the targeted coverage rate of 95% PIs, whereas the coverage rates of the ANN-NB-DST and QR-NB-DST methods are lower than the targeted 95%. This variation highlights the challenges of forecasting under overcast conditions. The estimated CDFs of the hourly solar generation estimated by the forecasting methods at 10:00 AM on this “overcast” day (28 December 2022) are illustrated in Figure 6b. It can be observed from Figure 6b that the CDFs estimated by the NB-DST methods align more closely to the actual observation CDF than those estimated by the two benchmark methods.

4.4. Performance Evaluation under “Partially Cloudy” Weather Conditions

In this subsection, the performance metrics are calculated over the 112 “partially cloudy” days in the TS dataset for all the probabilistic forecasting methods and are summarized in

Table 4. Performance metrics of all the forecasting methods were evaluated over the 112 “partially cloudy” days.

Method	CRPS (%)	CRPS Decomposition			BS (%)	PICP (%)	PINAW	CWC
		REL	RES	UNC
PerEn	9.30	0.55	33.93	42.68	5.24	90.86	27.67	166.12
AnEn	9.98	0.64	33.34	42.68	6.77	86.33	28.60	171.70
QR-NB-DST	8.15	0.65	35.18	42.68	5.72	88.28	16.92	101.62
SVR-NB-DST	8.55	0.78	34.91	42.68	5.28	84.91	17.37	104.32
ANN-NB-DST	6.21	0.49	36.96	42.68	3.94	87.89	13.24	79.54

. It can be observed that the ANN-NB-DST method achieves the lowest CRPS value, which indicates the best overall performance. Additionally, the reliability and resolution of the ANN-NB-DST method are also superior to those of other NB-DST and benchmark methods, as indicated by its lower value of REL and higher value of RES. Moreover, the lower PINAW values of the NB-DST methods suggest that they have better resolution than the AnEn and the PerEn methods. While the PICP values for the NB-DST methods are not as good as those for the benchmark methods, the CWC values, which consider both PICP and PINAW, suggest a better overall performance for the NB-DST methods. Notably, the ANN-NB-DST method stands out with the lowest BS, indicating the highest accuracy among the five methods.

To further illustrate the performance, Figure 8 shows the 95% PIs obtained from all the forecasting methods on a particular “partially cloudy” day (9 July 2022). It can be observed that the 95% PIs obtained from the benchmark methods are wider than those from the NB-DST methods. Also, in Figure 8, some measured points fall outside the 95% PIs obtained from the CDFs estimated by the NB-DST methods. In contrast, such occurrences are not observed with the AnEn and PerEn methods. The observation highlights a strategic trade-off made by the NB-DST methods, prioritizing improved resolution over coverage probability. Figure 6c shows the CDFs estimated by all the methods at 10:00 AM on this “partially cloudy” day (9 July 2022). The CDFs estimated by the NB-DST methods show smaller deviations from the actual observed solar power CDF than those estimated by the AnEn and PerEn methods.

4.5. Overall Comparative Analysis

The results from Table 2, Table 3 and Table 4 are synthesized to evaluate the overall performance of the proposed method over the 1-year testing dataset, TS. The findings are presented in

Table 5. Performance metrics of all the forecasting methods evaluated over the 1-year testing dataset.

Method	CRPS (%)	CRPS Decomposition			BS (%)	PICP (%)	PINAW	CWC
		REL	RES	UNC
PerEn	8.72	0.76	29.44	37.40	5.61	91.41	480.42	2882.61
AnEn	10.48	0.90	27.82	37.40	6.59	86.91	492.26	2953.67
QR-NB-DST	7.20	0.91	31.11	37.40	5.00	87.32	232.01	1392.15
SVR-NB-DST	7.73	0.86	30.53	37.40	4.83	85.60	245.79	1474.82
ANN-NB-DST	5.94	0.76	32.23	37.40	3.88	88.20	212.01	1272.18

. Notably, the NB-DST methods outperform the benchmark methods on nearly all performance metrics except for the PICP. This exception is understandable because the PI estimates of the PerEn and AnEn methods are considerably wider than those of the NB-DST methods. Even with the exception, the significantly better PINAW and CWC values suggest the overall superiority of the NB-DST methods. According to Table 5, the ANN-NB-DST emerges as the best-performing method among the NB-DST variants.

Figure 9 further illustrates the percentage improvements in CRPS and CWC of the ANN-NB-DST method over the PerEn method across all weather conditions. As explained before, CRPS measures the overall deviation of the estimated CDF from the actual measurements, indicating the forecast accuracy, while CWC measures the reliability and sharpness of the forecasted CDF. Notably, the ANN-NB-DST method achieves over 33% improvement in CRPS under clear and partially cloudy weather conditions. This result indicates that the CDF estimated by the ANN-NB-DST method deviates less from the actual solar power measurements than the PerEn method. Also, the improvement in CRPS is less pronounced (<10%) under overcast weather conditions, reflecting the challenges in predictive accuracy under such weather. Despite this, results from Figure 7 and Table 3 suggest the overall superiority of the ANN-NB-DST method in accuracy and sharpness with a slight reduction in reliability. Moreover, the CWC improvement remains consistent (more than 52%) across all the weather conditions, indicating the higher reliability and better sharpness of the ANN-NB-DST method than the benchmark method.

5. Conclusions and Future Work

This paper proposes the NB-DST method, a novel approach for quantifying forecast errors of a deterministic forecasting method by estimating the CDF using a non-parametric technique. The NB-DST method has been integrated with deterministic forecasting methods such as SVR, ANN, and QR to generate probabilistic forecasts under different weather conditions. Comparisons with established benchmark methods (AnEn and PerEn) using real-world data reveal that the NB-DST methods consistently outperform the benchmarks in terms of lower CRPS, BS, and CWC values. Among the tested methods, the ANN-NB-DST method emerges as the top performer, consistently producing reliable and high-resolution CDF estimates across various weather conditions, establishing it as the superior option for probabilistic photovoltaic (PV) forecasting among the methods evaluated.

Future studies will focus on exploring a wider range of deterministic forecasting methods in conjunction with the NB-DST method to enhance forecast robustness and consistency. Additionally, this study considered only a limited number of meteorological variables due to data availability limitations. Thus, an important direction for future investigation is to incorporate a more comprehensive set of meteorological variables into the forecasting models. This expansion will enable a deeper exploration of their impact on forecasting accuracy and provide more comprehensive insights into the dynamics of probabilistic PV forecasting.