A Resilient Cyber-Physical Demand Forecasting System for Critical Infrastructures against Stealthy False Data Injection Attacks

Abstract

The safe and efficient function of critical national infrastructure (CNI) relies on the accurate demand forecast. Cyber-physical system-based demand forecasting systems (CDFS), typically found in CNI (such as energy, water, and transport), are highly vulnerable to being compromised under false data injection attacks (FDIAs). The problem is that the majority of existing CDFS employ anomaly-based intrusion detection systems (AIDS) to combat FDIAs. Since the distribution of demand time series keeps changing naturally with time, AIDS treat a major change in the distribution as an attack, but this approach is not effective against colluding FDIAs. To overcome this problem, we propose a novel resilient CDFS called PRDFS (Proposed Resilient Demand Forecasting System). The primary novelty of PRDFS is that it uses signature-based intrusion detection systems (SIDS) that automatically generate attack signatures through the game-theoretic approach for the early detection of malicious nodes. We simulate the performance of PRDFS under colluding FDIA on High Performance Computing (HPC). The simulation results show that the demand forecasting resilience of PRDFS never goes below 80%, regardless of the percentage of malicious nodes. In contrast, the resilience of the benchmark system decreases sharply from about 99% to less than 30%, over the simulation period as the percentage of malicious nodes increases.

1. Introduction

Our critical national infrastructure (CNI) systems like those driving power generation and distribution, water treatment, electricity production, and other platforms are being increasingly digitized and controlled by the Internet of Things (IoT) [1]. The task of forecasting demand for CNI is fundamental and crucially important to run CNI smoothly and make sound operational decisions [2]. Demand specifies the number of goods and services that will be used or consumed in an economy at a given time. Cyber physical system (CPS)-based demand forecasting systems (CDFS) are ubiquitous in CNI [2] where sensor nodes are deployed in the environment to monitor the current demand for the services of CNI (e.g., traffic and electricity consumption) [3,4,5,6,7]. The sensors transmit their measurements at a regular interval to a base station through one or more intermediate nodes depending on the distance between the sensor and the base station. In the base station, the measurements of all sensors are aggregated using aggregate operators like average and sum. The forecasting model stored in the base station uses the values of the aggregate demand time series to develop an estimate of expected demand during a specified future period. The demand forecast is transmitted to the application layer to adjust the supply of the services of CNI such as electricity, water, gas, transportation, etc., to the forecasted demand. Inaccurate demand forecasts can create an imbalance of demand and supply which may have a devastating impact in terms of lives lost, infrastructure damage, and economic disruption [8,9,10]. The accuracy of demand forecasting depends on the reliability of the inputs to the forecasting model. In CPS, sensor nodes are deployed in remote areas without any physical protection [11]. Furthermore, sensors are resource-constrained tiny nodes, due to which the implementation of traditional cryptographic techniques is not possible [11]. Hence, sensor nodes are highly vulnerable to attacks, and attackers can easily tamper with the sensor node and compromise it [6,12]. Once a sensor node is compromised, attackers can launch any form of attack they desire [13]. A false data injection attack (FDIA) is the worst possible attack that could be made on a forecasting system. In FDIAs, attackers modify the measurements of the compromised sensors to change the demand forecasts to their desired level [14,15]. To resolve this issue, it is necessary to improve the resilience of CDFS to stealthy attacks. Resilience is twofold. Firstly, the CDFS must be robust against attacks (that is, be able to either prevent the attacks or detect them at a very early stage), and secondly, it should bounce back quickly to its baseline before the attack [4].

Recently, several resilient CDFS have been proposed that consider any drastic change in demand distribution as an attack [9,16,17,18]. The problem with this approach is that the joint probability distribution of demand time series keeps changing naturally with time [2]. The approach often produces an extremely high false positive rate [9]. In addition, the attackers have figured out how to avoid detection. They launch collusion attacks in which the attacker takes control of multiple sensor nodes and alters the readings of those sensors slightly, such that the aggregate value of all sensors’ readings is skewed towards the attacker’s desired direction [19]. This aggregate measurement estimated at the time step $\mathit{t}$ is incorporated into the training set. The demand forecasting models are adapted to recent history to make out-of-sample forecasts so that the machine learning forecasting system dynamically adapts to data distributions that evolve over time. Since the forecasting model is evolving toward false data, over time, the normal sensors are marked as compromised. They have an increasingly reduced influence on the model’s evolution. This way, through careful selection of reported values, the attackers manage to shift the accepted value far away from the actual value. To make the attack even stealthier, at each time point, attackers randomly select a subset of sensors from the set of compromised sensors, and only the selected compromised nodes inject false data. In contrast, the other compromised sensors provide unadjusted, untampered measurements [20,21]. To protect the CDFS from stealthy attacks, we propose a novel CDFS called PRDFS (Proposed Resilient Demand Forecasting System). The key advantage of the PRDFS over the existing CDFS is its signature-based intrusion detection system (SIDS) trained on autonomously generated attack signatures.

The rest of this paper is organised as follows: Section 2 provides definitions of several important concepts used within this paper, an overview of challenges in demand forecasting, and how the existing resilient CDFS deal with these challenges. Section 2 also contains an overview of existing defence schemes against stealthy FDIA. Section 3 presents our proposed CDRFS (PRDFS). Section 4 explains the research methodology, including a description of the benchmark CDFS, adversary model, performance criteria, and evaluation methodologies for resilient CDFS. Section 5 presents the results, wherein we discuss the advantages and disadvantages of PRDFS and the benchmark CDFS in detail. Section 6 concludes this work.

2. Related Work

Resilient demand forecasting is a relatively new area of research [9,16,17,18]. Most of the existing resilient CDFS consist of two subsystems—the demand forecasting system (DFS) and the intrusion detection system (IDS). The first subsystem consists of normal time series forecasting algorithms that predict future demand for a particular product/service. Kalman Filtering models and Long Short-Term Memory (LSTM) networks (a special variant of recurrent neural networks) are by far the most popular choices for forecasting demand among the existing resilient CDFS [9,17,22,23]. The Kalman Filter is a model-based linear estimator unsuitable for complex demand signals with unknown characteristics.

The purpose of the second subsystem (IDS) of the resilient CDFS is to mitigate the adverse effects of attacks on demand forecasting. We organise this section as follows. Important concepts are defined in Section 2.1. The major challenges in demand forecasting and how the existing resilient CDFS address these challenges are discussed in Section 2.2. An overview of existing attack detection and mitigation algorithms used in IDS is discussed in Section 2.3.

2.1. Definitions

This section aims to provide an operational definition of “an attack on CDFS” and define three core concepts (trust, trustworthiness, and reputation) that are frequently used in this paper.

An attack on a system is defined as “a threat action that undesirably alters system operation by adversely modifying system functions or data” [24]. For this study, an attack on CDFS is defined as systematically deviating the point forecasts and interval forecasts for k-step ahead aggregate demands from the actual values to a certain direction (upwards or downwards) over time by adversely modifying sensor data. The success of the attack is measured on the following two indicators: (1) the higher the deviations of the single-point demand forecast from the actual demand for a given period, the more successful the attack, and (2) the ultimate success of the attack is achieved when the actual demand falls outside of the forecasted bounds.

Although the terms ‘trust’, ‘trustworthiness’, and ‘reputation’ are widely used in the literature, there remains debate about the definitions of all three [11,13,22,23,25,26,27,28,29]. Different studies have different definitions of these terms. In this study, we adopt the following definitions for trust, trustworthiness, and reputation. We use ‘trust’ and ‘trustworthiness’ interchangeably. If a node needs to evaluate the trust score of another node, the evaluating node is called the subject node, and the evaluated node is called the object node [20]. ‘Trust’ or ‘trustworthiness’ of a node is the subjective probability assigned to an object node by a subject node at time step $\mathit{t}$ based on various factors (e.g., the object node’s long-term reputation, the similarity of its measurements with the measurements generated by other nodes received at the same time step etc.) that the measurement received from the object node is accurate [27,30]. The trustworthiness of data provenance is the probability assigned by a subject node to the provenance that the measurement received from the data provenance is accurate. Data provenance refers to the metadata that captures the complete history of a piece of data from origin to destination. The data provenance of measurement consists of two basic components [31]. The first component is the route of the measurement. A route refers to the path a measurement travels on a network. The route includes every node that handles the measurement between its source and destination. The second component of the data provenance is the list of actions performed by participating nodes. The trustworthiness of a measurement value is the probability that the measurement is accurate [11]. The reputation of an object node is the overall probability estimated from the trust scores assigned to the object node by all subject nodes over time [26,30].

2.2. Challenges in Demand Forecasting

Demand forecasting is a mature area of research [32]. Nonetheless, for accurate predictions of demand, many challenges remain to be addressed [32,33,34]. One of the major challenges is the curse of dimensionality. In demand forecasting, the predictor variables are lagged variables. The lagged variables are highly correlated. Consequently, even a moderate number of predictor variables may trigger the curse of dimensionality, and demand forecasting models often suffer from overfitting. Most of the existing CDFS do not take special measures to protect themselves against the curse [9,17]. They train a single network (e.g., LSTM) with a set of lagged aggregate demand time series values to forecast the out-of-sample data.

Another major challenge in demand forecasting is the presence of non-stationarity in demand time series [32,33,34]. In a non-stationary time series, the frequency content changes in time; all frequencies do not always exist. There are currently three main approaches for dealing with time-varying frequency components [35]. The first approach is to apply the differencing operator on-demand time series and remove these patterns from the time series. However, the main difficulty of the differencing approach is that it requires exact knowledge about the locations of all frequency components in the time series, which is by no means easy to capture.

The second approach to deal with the time-varying frequency components is to enter a large number of non-seasonal lagged variables in a neural network (e.g., LSTM) during the training so that it automatically selects an optimal number of seasonal and non-seasonal lagged variables and models time-varying frequency components. Most of the existing resilient CDFS adopt this mechanism [9,17]. This approach leads to a large number of lagged input variables in the model which might exacerbate the curse of dimensionality.

The third approach to deal with the time-varying frequency components is to use a hybrid of wavelet transform and neural networks [35,36]. This approach has been drawing increasingly more attention in recent years. This approach uses wavelet techniques to split the original time series into multiple component time series. A neural network (e.g., LSTM) is trained with the non-seasonal lagged variables generated from the component time series to forecast the future values of the original demand curve. Currently, more comparative studies are required to confirm the superiority of one approach over the other.

Perhaps the biggest challenge in developing a demand forecasting model is that the older values of the demand time series are less useful than the newer ones as training data and become obsolete quickly, since the distribution of demand time series keeps changing with time [30]. Several studies suggest ensuring that the forecasting model adjusts rapidly to the changing distribution. Based on these studies, the higher penalty should be assigned to in-sample forecast errors on more recent demand observations during the training phase. However, this approach poses many practical problems. The key problem is how to weigh the values of demand time series appropriately in the presence of time-varying frequency components. An inadequate weighting scheme may cause more harm than good, as it may destroy the distribution of the original time series. Hence, the common practice is to keep adapting the forecasting model to the new observations as the sensors gather them without applying an arbitrary weighting scheme [9].

2.3. Overview of Existing Defense Strategies against Stealthy False Data Injection Attacks (FDIAs)

Existing defense systems to deal with stealthy false data injection attacks (FDIAs) can be grouped into two broad categories: anomaly-based intrusion detection systems (AIDS) and provenance-based trust management systems (TMS) [9,13,22,31,37,38].

The goal of AIDS is to detect malicious nodes and exclude their readings from the aggregation of sensor measurements while estimating the most trustworthy aggregate measurement at a given time step. When AIDS is used to deal with FDIA, the simple average is usually used to aggregate the measurements of the normal sensors. Existing AIDS can be broadly split into two categories: confidence interval-based AIDS [9,22,37] and clustering-based AIDS [38]. Both types of AIDS are based on the hard clustering concept that classifies a sensor node as either malicious (1) or normal (0). Confidence interval-based AIDS divides the search space into only two clusters (one for each class: malicious and normal). On the contrary, clustering-based AIDS allow multiple clusters for each class; this type of AIDS performs better on search space with local patterns.

AIDS uses unsupervised hard clustering algorithms to perform clustering of the search space to classify the sensor nodes as malicious and normal. Different search spaces have been explored for detecting stealthy malicious nodes, including measurement space, forecasting residual space, Spatio-temporal correlation space, the skewness space of residuals in the forecasting demand model, and reputation space [9,22,37,38]. The major limitation of AIDS is that they use a set of arbitrary rules to cluster the search space and to label the clusters as malicious and normal. Past studies suggest that AIDS suffer from low malicious node detection accuracy and a high false positive rate (FPR) [9].

The goal of TMS, in contrast to AIDS, is not to classify the sensor nodes as malicious and normal [13,31]. Instead, they carry out unsupervised fuzzy clustering in the sensor measurement space to assign a reputation score to each sensor to control its influence on the sensor measurement aggregation process. During the aggregation process, TMS assigns a trustworthiness score to each measurement at time step $\mathit{t}$ based on the data similarity and provenance similarity among measurements, and the trustworthiness of its data provenance, which is usually set to the reputation score of the least reputed node in the provenance. TMS then determines the most trustworthy measurement at time step $\mathit{t}$ using the weighted average method. Most of the existing TMS use linear models to update the reputation scores of nodes based on the trustworthiness scores of their measurements received at time step $\mathit{t}$. However, the linear models may not be able to accurately estimate the reputation scores of nodes in the face of stealthy FDIA.

3. Proposed Resilient Demand Forecasting Systems

We name our proposed resilient demand forecasting system PRDFS (Proposed Resilient Demand Forecasting System). PRDFS consists of five modules: (1) Trust Management System (TMS), (2) Demand Forecasting System (DFS), (3) Data Generator (DG), (4) Ethical Hacking System (EHS), and (5) Signature-based Intrusion Detection System (SIDS). Codes and pseudocodes of PRDFS are provided in Supplementary Materials. Figure 1 shows the architecture of PRDFS. The modules are linked to each other via the input and output they exchange. Figure 1 depicts the exchanges of inputs and outputs among the modules. The internal workings of the modules are shown in the flowcharts of Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6.

Table 1. The Definitions of Terms used in Section 3.

Term	Definition
$N$	Total number of DPVs
$t$	Current time step
$DP{V}_i$	Data provenance of the ith measurement
$DP{V}_j$	Data provenance of the jth measurement
$nc$	The total number of nodes common to $DP{V}_i$ and $DP{V}_j$
$n_1$	The lenght of the longer DPV between $DP{V}_i$ and $DP{V}_j$
$\overrightarrow{m{v}_i\left(t\right)}$	Time series of the ith DPV’s measurements containing both past and current measurements
$\overrightarrow{m{v}_i\left(t-1\right)}$	Time series of the ith DPV’s past measurements
$m{v}_i\left(t\right)$	The received measurement of the ith DPV at time step t
$m{v}_j\left(t\right)$	The received measurement of the jth DPV at time step t
$D{S}_{i,j}\left(t\right)$	The data similarity between $m{v}_i\left(t\right)$ and $m{v}_j\left(t\right)$ estimated using MAAPE
$P{S}_{i,j}\left(t\right)$	Provenance similarity between the DPVs of $m{v}_j\left(t\right)$ and $m{v}_i\left(t\right)$$: P{S}_{i,j}\left(t\right)=nc/n_1$
$\widehat{m{v}_i\left(t\right)}$	One step ahead forecast of the conditional mean of $m{v}_i\left(t\right)$
${\epsilon }_i\left(t\right)$	${\epsilon }_i\left(t\right)=m{v}_i\left(t\right)-\widehat{m{v}_i\left(t\right)}$
$\overrightarrow{{\epsilon }_i\left(t\right)}$	The residual series for the one step ahead forecasts of ith DPVs measurements containing both the past and current residuals
$\overrightarrow{{\epsilon }_i\left(t-1\right)}$	The residual series for the one step ahead forecasts of ith DPVs measurements containing the past residuals
${\sigma }_i^2\left(t\right)$	The observed conditional variance of $m{v}_i\left(t\right)$$:{\sigma }_i^2\left(t\right)={\epsilon }_i^2\left(t\right)$
$\widehat{{\sigma }_i^2\left(t\right)}$	One step ahead forecast of ${\sigma }_i^2\left(t\right)$
$ENe{t}_i^M$	The ensemble of CNNs forecasting $\widehat{m{v}_i\left(t\right)}$
$ENe{t}_i^V$	The ensemble of CNNs forecasting $\widehat{{\sigma }_i^2\left(t\right)}$ based on $\overrightarrow{{\epsilon }_i^2\left(t-1\right)}$
${ϵ}_i\left(t\right)$	${ϵ}_i\left(t\right)={\sigma }_i^2\left(t\right)-\widehat{{\sigma }_i^2\left(t\right)}$
$f\left(\widehat{m{v}_i\left(t\right)},\sqrt{\widehat{{\sigma }_i^2\left(t\right)}}\right)$	A gaussian distribution with mean $\widehat{m{v}_i\left(t\right)}$ and standard deviation $\sqrt{\widehat{{\sigma }_i^2\left(t\right)}}$
$SE$	A matrix containing the residual series for the one step ahead forecasts of all DPVs measurements
$SK$	A matrix containing the time series of skewness of the series in $SE$
$\overrightarrow{MV\left(t\right)}$	The measurements of all DPVs at $t: \overrightarrow{MV}\left(t\right)=\left[m{v}_1\left(t\right),\dots ,m{v}_N\left(t\right)\right]$
$D\left(t\right)$	The most trustworthy aggregate demand measurement at time step $t$ based on $\overrightarrow{MV\left(t\right)}$
$\overrightarrow{D\left(t\right)}$	The time series of aggregate demand
$\overrightarrow{tr{s}_i\left(t\right)}$	Time series of the assessed trust scores of the ith DPV including its assessed trust score at time step t
$\overrightarrow{tr{s}_i\left(t-1\right)}$	Time series of the assessed trust scores of the ith DPV, not including its assessed trust score at time step t
$tr{s}_i\left(t\right)$	The assessed trust score of the ith DPV at time step t
$\widehat{tr{s}_i\left(t\right)}$	The forecasted trust score of the ith sensor at time step t
$m_i\left(t\right)$	The forecasted conditional mean of $tr{s}_i\left(t\right)$ based on $\overrightarrow{tr{s}_i\left(t-1\right)}$
${\delta }_i^2\left(t\right)$	The forecasted conditional variance of $m_i\left(t\right)$
$GRN{N}_i^M$	The ith member of GRNN of the ensemble of GRNNs forecasting $m_i\left(t\right)$
$GRN{N}_i^V$	The ith member of GRNN of the ensemble of GRNNs forecasting ${\delta }_i^2\left(t\right)$
$TS$	$TS=\left[\overrightarrow{tr{s}_1\left(t\right)},\dots ,\overrightarrow{tr{s}_N\left(t\right)}\right]$
$D\left(t\right)$	The most trustworthy aggregate demand measurement at time step t
$\overrightarrow{D\left(t\right)}$	The time series of aggregate demand (based on $\overrightarrow{MV\left(t\right)}$) including $D\left(t\right)$
$\overline{D\left(t+k\right)}$	The k-step-ahead forecasted conditional mean aggregate demand based on $\overrightarrow{D\left(t\right)}$
$V\left(t+k\right)$	The k-step-ahead forecasted value of the conditional variance of $\overline{D\left(t+k\right)}$
$L\left(t+k\right)$	$L\left(t+k\right)=\overline{D\left(t+k\right)}-3\sqrt{V\left(t+k\right)}$
$U\left(t+k\right)$	$U\left(t+k\right)=\overline{D\left(t+k\right)}+3\sqrt{V\left(t+k\right)}$
$Ensembl{e}^{mean}$	The ensemble of CNNs forecasting $\overline{D\left(t+k\right)}$
$Ensembl{e}^{var}$	The ensemble of CNNs forecasting $V\left(t+k\right)$
$gm{v}_i\left(t\right)$	Measurement generated for the ith DPV at time step t by DG
$\overrightarrow{GMV\left(t\right)}$	$\overrightarrow{GMV\left(t\right)}$ contains the measurements generated by DG for all sensors at t: $\overrightarrow{GMV\left(t\right)}=\left[gm{v}_1\left(t\right),\dots ,gm{v}_N\left(t\right)\right]$
$\overline{D_g\left(t\right)}$	The most trustworthy aggregate demand measurement at time step t based on $\overrightarrow{GMV\left(t\right)}$
$\overrightarrow{D_g\left(t\right)}$	The time series of aggregate demand (based on $\overrightarrow{GMV\left(t\right)}$) including $\overline{D_g\left(t\right)}$
$\widehat{D_g\left(t+k\right)}$	The k-step-ahead forecasted conditional mean aggregate demand based on $\overrightarrow{D_g\left(t\right)}$
$\overrightarrow{po{s}_q\left(t,I\right)}$	The position of the qth particle at time step t and iteration I.
$\overline{D_q\left(t\right)}$	The most trustworthy aggregate demand measurement at time step t based on $\overrightarrow{po{s}_q\left(t,I\right)}$
$\overrightarrow{D_q\left(t\right)}$	The time series of aggregate demand (based on $\overrightarrow{po{s}_q\left(t,I\right)}$) including $\overline{D_q\left(t\right)}$
$\widehat{D_q\left(t+k\right)}$	The k-step-ahead forecasted conditional mean aggregate demand based on
$\overrightarrow{Ve{l}_q\left(t,I\right)}$	$\overrightarrow{Ve{l}_q\left(t,I\right)}$ is the velocity of the qth particle.
$fitness(\overrightarrow{po{s}_q\left(t, I\right)}$	The fitness score of $\overrightarrow{po{s}_q\left(t, I\right)}$
$\overrightarrow{pbes{t}_q}$	The personal best position of the qth particle
$fitness\left(\overrightarrow{pbes{t}_q}\right)$	The fitness score of $\overrightarrow{pbes{t}_q}$
$\overrightarrow{gbest}$	The global best position of all the particles
$fitness\left(\overrightarrow{gbest}\right)$	The fitness score of $\overrightarrow{gbest}$
$ST$	$ST$ is a matrix containing the time series of the states (malicious and normal) of each sensor at previous time steps
$\overrightarrow{S{T}_q\left(t,I\right)}$	A vector containing the state of each DPV in the qth particle at time step t and iteration I
$\overrightarrow{ST\left(t=1, I=1\right)}$	A vector containing the state of each DPV at the beginning of a game
$\overrightarrow{ST}$	A vector containing the state of each DPV in $\overrightarrow{gbest}$

shows the definitions of the terms used in this section.

The first layer of PRDFS is only for graphical representation (Figure 1). Each node of this layer represents a data provenance (DPV). In our simulation study, for simplicity purposes, each DPV contains only one sensor node. The DPVs transmit streams of measurements of demand $\left(\overrightarrow{\mathit{M}\mathit{V}\left(\mathit{t}\right)}\right)$ to the following two units: TMS and DG.

TMS: The flowchart for TMS is presented in Figure 2. TMS produces the following three outputs, which the other modules use as inputs.

Output 1: TMS derives the aggregate demand time series $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ of individual DPVs by averaging the measurements. DFS uses $\overrightarrow{\mathit{D}\left(\mathit{t}\right)}$ as inputs. If all DPVs were equally trustworthy, a simple average would be appropriate to aggregate the measurements of all DPVs. However, since the measurements of all DPVs are not always equally trustworthy due to adversarial attacks and mechanical issues, a weighted average is more appropriate than a simple average for the derivation of aggregate demand $\left(\mathit{D}\left(\mathit{t}\right)\right)$. TMS first determines the trustworthiness weights of the measurements based on the data similarity, provenance similarity, past measurements, and the predicted trustworthiness of the corresponding DPVs at a given time step using the ensembles of Convolutional Neural Networks (CNNs) [39] and Generalized Regression Neural Networks (GRNNs) [40], and then performs the weighted average. TMS sends the aggregate demand time series $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ to DFS.

Output 2: At the end of each time step, TMS assesses the trust scores $\left[\mathit{t}\mathit{r}{\mathit{s}}_{\mathbf{1}}\left(\mathit{t}\right)\mathbf{,}\mathbf{\dots }\mathbf{,}\mathit{t}\mathit{r}{\mathit{s}}_{\mathit{N}}\left(\mathit{t}\right)\right]$ for all DPVs based on the distances between their measurements and the most trustworthy aggregated measurement $\left(\mathit{D}\left(\mathit{t}\right)\right)$ at that time step. TMS updates $\mathit{T}\mathit{S}\mathbf{=}\left[\overrightarrow{\mathit{t}\mathit{r}{\mathit{s}}_{\mathbf{1}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)}\mathbf{,}\mathbf{\dots }\mathbf{,}\overrightarrow{\mathit{t}\mathit{r}{\mathit{s}}_{\mathit{N}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)}\right]$ by adding $\left[\mathit{t}\mathit{r}{\mathit{s}}_{\mathbf{1}}\left(\mathit{t}\right)\mathbf{,}\mathbf{\dots }\mathbf{,}\mathit{t}{r}{\mathit{s}}_{\mathit{N}}\left(\mathit{t}\right)\right]$ to $\mathit{T}\mathit{S}$. The updated $\mathit{T}\mathit{S}$ is $\mathit{T}\mathit{S}\mathbf{=}\left[\overrightarrow{\mathit{t}\mathit{r}{\mathit{s}}_{\mathbf{1}}\left(\mathit{t}\right)}\mathbf{,}\mathbf{\dots }\mathbf{,}\overrightarrow{\mathit{t}\mathit{r}{\mathit{s}}_{\mathit{N}}\left(\mathit{t}\right)}\right]$.

Output 3: At each time step, TMS produces the residuals $\left[{\mathbf{\epsilon }}_{\mathbf{1}}\left(\mathit{t}\right)\mathbf{,}\mathbf{\dots }\mathbf{,}{\mathbf{\epsilon }}_{\mathit{N}}\left(\mathit{t}\right)\right]$ of one-step ahead forecasts of the measurements of all DPVs at that time step. TMS then updates $\mathit{S}\mathit{E}\mathbf{=}\left[\overrightarrow{{\mathbf{\epsilon }}_{\mathbf{1}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)}\mathbf{,}\mathbf{\dots }\mathbf{,}\overrightarrow{{\mathbf{\epsilon }}_{\mathit{N}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)}\right]$ to $\mathit{S}\mathit{E}\mathbf{=}\left[\overrightarrow{{\mathbf{\epsilon }}_{\mathbf{1}}\left(\mathit{t}\right)}\mathbf{,}\mathbf{\dots }\mathbf{,}\overrightarrow{{\mathbf{\epsilon }}_{\mathit{N}}\left(\mathit{t}\right)}\right]$ by adding $\left[{\mathbf{\epsilon }}_{\mathbf{1}}\left(\mathit{t}\right)\mathbf{,}\mathbf{\dots }\mathbf{,}{\mathbf{\epsilon }}_{\mathit{N}}\left(\mathit{t}\right)\right]$ to $\mathit{S}\mathit{E}$.

DFS: The flowchart for DFS is provided in Figure 3. DFS performs k-step ahead out-of-sample forecasting of demand based on $\overrightarrow{\mathit{D}\left(\mathit{t}\right)}$ derived by TMS. It uses wavelet decomposition [41] to split $\overrightarrow{\mathit{D}\left(\mathit{t}\right)}$ into component series of various frequencies. It then uses a pair of hierarchical ensembles of CNNs to estimate the distribution of demand forecasts based on the values of the component time series at non-seasonal lags. The member CNNs are trained sequentially such that the first member CNN is trained on the component time series of $\overrightarrow{\mathit{D}\left(\mathit{t}\right)}$ and each of the other member CNNs is trained on the component time series of the residuals of the member CNN trained just before it. DFS keeps increasing the number of member CNNs in the ensemble until the residual series of the last member CNN becomes white noise. We call this ensemble of CNNs ‘$\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$’. $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$ forecasts the conditional mean demand $\left(\overline{\mathit{D}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\right)$. DFS also trains another ensemble of CNNs called ‘$\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{v}\mathit{a}\mathit{r}}$’ in the same way it trained $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$. The only difference is that the first member CNN of $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{v}\mathit{a}\mathit{r}}$ is trained on the component time series of the squared residuals of $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{m}\mathit{e}\mathit{a}{n}}$ and each of the remaining member CNNs is trained on the component time series derived from the time series of the absolute values of residuals obtained from the above member CNN. $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{v}\mathit{a}\mathit{r}}$ forecasts the conditional variance $\left(\mathit{V}\left(\mathit{t}\mathbf{+}\mathit{k}\right)\right)$ of $\left(\overline{\mathit{D}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\right)$. During forecasting, each ensemble member in $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$ and $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{v}\mathit{a}\mathit{r}}$ forecasts values separately. The overall forecast $\left(\overline{\mathit{D}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\right)$ of $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}\mathit{b}\mathit{l}{\mathit{e}}^{\mathit{m}\mathit{e}\mathit{a}\mathit{n}}$ is the summation of the forecasts of its all members. The overall forecast $\left(\mathit{V}\left(\mathit{t}\mathbf{+}\mathit{k}\right)\right)$ of $\mathit{E}\mathit{n}\mathit{s}\mathit{e}\mathit{m}{b}\mathit{l}{\mathit{e}}^{\mathit{v}\mathit{a}\mathit{r}}$ is the summation of the absolute values of the forecasts of all its member CNNs. The point forecast (i.e., the most expected demand) of the DFS module at $\mathit{t}\mathbf{+}\mathit{k}$ is $\overline{\mathit{D}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}$. The interval forecast of DFS at $\mathit{t}\mathbf{+}\mathit{k}$:

The lower bound of the forecast distribution:

$$\mathit{L}\left(\mathit{t}\mathbf{+}\mathit{k}\right)\mathbf{=}\overline{\mathit{D}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\mathbf{-}\mathbf{3}\sqrt{\mathit{V}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\mathbf{ }$$

(1)

The upper bound of the forecast distribution:

$$\mathit{U}\left(\mathit{t}\mathbf{+}\mathit{k}\right)\mathbf{=}\overline{\mathit{D}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\mathbf{+}\mathbf{3}\sqrt{\mathit{V}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\mathbf{ }$$

(2)

DG: The flowchart for DG is provided in Figure 4. The DG learns the joint probability distribution of the individual time series of each DPV and generates an infinite stream of $\left(\overrightarrow{\mathit{G}\mathit{M}\mathit{V}\left(\mathit{t}\right)}\right)$ for each DPV. DG sends $\overrightarrow{\mathit{G}\mathit{M}\mathit{V}\left(\mathit{t}\right)}$ to EHS to facilitate the generation of attack signatures.

EHS: The flowchart for EHS is given in Figure 5. The task of EHS is to generate all possible attack signatures for the training of SIDS. To do so, the EHS uses the Particle Swarm Optimization (PSO) algorithm [42] with a game theoretic approach [43]. It plays repeated games with the other modules of PRDFS. In these games, EHS plays as an attacker while TMS and SIDS are teamed up to play as a defender of DFS. The attacker and defender play under the following rules: (1) in each game, the attacker has a fixed number of malicious nodes at the beginning. (2) If the defender detects a malicious node, it becomes a normal node. From the next step onwards, the attacker cannot modify its measurements. (3) An attacker cannot compromise a new or recovered node in the middle of the game. (4) EHS works with complete knowledge of the PRDFS. (5) The attacker randomly chooses one of the following two objectives to pursue in a game: (i) shift the demand trend upward, and (ii) shift the demand trend downward.

To generate optimal attack signatures, the PSO (employed by EHS) follows the following steps.

Step 1. Initialize the positions and velocities of ten particles. The position $\left(\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\mathbf{=}\left[\mathit{g}\mathit{m}{\mathit{v}}_{\mathit{1}}\left(\mathit{t}\right)\mathbf{\dots }\mathbf{,}\mathit{g}\mathit{m}{\mathit{v}}_{\mathit{N}}\left(\mathit{t}\right)\right]\right)$ of each particle $\left(\mathit{q}\right)$ at time step $\mathit{t}$ and iteration $\mathit{I}$ is a vector containing the measurements of all DPVs at time step $\mathit{t}$, wherein the measurements of normal DPVs are the same as the generated measurements by the DG and the measurements of the malicious DPVs are randomly initialized. The velocity $\left(\overrightarrow{\mathit{V}\mathit{e}{\mathit{l}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\mathbf{=}\left[\mathit{V}\mathit{e}{\mathit{l}}_{\mathit{1}}\left(\mathit{t}\right)\mathbf{,}\mathbf{\dots }\mathbf{,}\mathit{V}\mathit{e}{\mathit{l}}_{\mathit{N}}\left(\mathit{t}\right)\right]\right)$ of each particle $\left(\mathit{q}\right)$ at time step $\mathit{t}$ and iteration $\mathit{I}$ is a vector that has the same length as the position vector. Each element of the velocity vector contains the direction and speed at which the measurement of the corresponding DPV is updated from the one iteration to the next during the PSO search. In each particle, the velocities of normal DPVs are set to zero and regarded as constant. In addition, each particle has a state vector $\left(\overrightarrow{\mathit{S}{\mathit{T}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right)$ containing the state of each DPV. Initially, the state vectors of all particles are the same.

Step 2. The fitness of each particle’s position $\left(\mathit{f}\mathit{i}\mathit{t}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\left(\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right)\right)$ is determined using the following method:

• Present $\overrightarrow{\mathit{G}\mathit{M}\mathit{V}\left(\mathit{t}\right)}$ to TMS. TMS estimates the most trustworthy aggregated measurement of demand $\overline{{\mathit{D}}_{\mathit{g}}\left(\mathit{t}\right)}$ and sends it to DFS to forecast the k-step ahead demand $\widehat{{\mathit{D}}_{\mathit{g}}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}$.
• Present $\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}$ to TMS. TMS estimates the most trustworthy aggregated measurement of demand $\overline{{\mathit{D}}_{\mathit{q}}\left(\mathit{t}\right)}$ and sends it to DFS to forecast the k-step ahead demand $\widehat{{\mathit{D}}_{\mathit{q}}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}$. TMS also updates the time series of assessed trust scores $\left(\mathit{t}\mathit{S}\right)$ of DPVs by adding the assessed trust scores of DPVs (found in $\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}$) at time step $\mathit{t}$ to $\mathit{t}\mathit{S}$, as well as the time series of the residuals of one-step ahead forecasts of each DPV’s measurements $\left(\mathit{S}\mathit{E}\right)$ by adding the residuals of forecasting the measurements of DPVs at time step $\mathit{t}$ to $\mathit{S}\mathit{E}$. SIDS uses the updated $\mathit{t}\mathit{S}$ and $\mathit{S}\mathit{E}$ to predict the current states (malicious and normal) of DPVs.
• If SIDS detects any malicious node correctly, set $\mathit{f}\mathit{i}\mathit{t}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\left(\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right)$ to zero and change the state(s) of the detected malicious node(s) from malicious to normal in $\left(\overrightarrow{\mathit{S}{\mathit{T}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right)$. Else if $\widehat{{\mathit{D}}_{\mathit{q}}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\mathbf{<}\widehat{{\mathit{D}}_{\mathit{g}}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}$ when the objective is shifting the demand upward or $\widehat{{\mathit{D}}_{\mathit{q}}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}\mathbf{>}\widehat{{\mathit{D}}_{\mathit{g}}\left(\mathit{t}\mathbf{+}\mathit{k}\right)}$ when the objective is shifting the demand downward, set $\mathit{f}\mathit{i}\mathit{t}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\left(\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right)$ to zero. Else estimate the fitness using the following formula:

$$\mathit{f}\mathit{i}\mathit{t}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\left(\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right)\mathbf{=}\mathbf{100}\mathbf{\times }{\mathbf{tan}}^{\mathbf{-}\mathbf{1}}\left(\left|\frac{\widehat{{\mathit{D}}_{\mathit{g}}\left(\mathit{t}\right)}\mathbf{-}\widehat{{\mathit{D}}_{\mathit{q}}\left(\mathit{t}\right)}}{\widehat{{\mathit{D}}_{\mathit{g}}\left(\mathit{t}\right)}}\right|\right)\mathbf{ }\mathbf{ }$$

(3)

Step 3. Find the personal best position of each particle at the current Ith iteration from the positions it has visited before and set the personal best position $\left(\overrightarrow{\mathit{p}\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{q}}}\right)$ of the qth particle to it. If it is the first iteration, set the $\overrightarrow{\mathit{p}\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{q}}}$ to its current position. We denote the fitness of $\overrightarrow{\mathit{p}\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{q}}}$ by $\left(\mathit{f}\mathit{i}\mathit{t}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\left(\overrightarrow{\mathit{p}\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{q}}}\right)\right)$. Find the global best position of all particles $\left(\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}\right)$ from the positions they have visited before. We denote the fitness of $\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}$ by $\left(\mathit{f}\mathit{i}\mathit{t}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\left(\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}\right)\right)$.

Step 4. Update the velocity of each malicious DPV in qth particle using the following equation:

$$\overrightarrow{\mathit{V}\mathit{e}{\mathit{l}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\mathbf{+}\mathbf{1}\right)}\mathbf{=}\overrightarrow{\mathit{V}\mathit{e}{\mathit{l}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\mathbf{+}{\mathit{C}}_{\mathbf{1}}\mathbf{\times }\mathit{r}\mathit{a}\mathit{n}\mathit{d}\left(\mathbf{1}\mathbf{,}\mathbf{1}\right)\mathbf{\times }\left|\overrightarrow{\mathit{p}\mathit{b}\mathit{e}\mathit{s}{\mathit{t}}_{\mathit{q}}}\mathbf{-}\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right|\mathbf{+}{\mathit{C}}_{\mathbf{2}}\mathbf{\times }\mathit{r}\mathit{a}\mathit{n}\mathit{d}\left(\mathbf{1}\mathbf{,}\mathbf{1}\right)\mathbf{\times }\left|\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}\mathbf{-}\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\right|\mathbf{ }\mathbf{ }\mathbf{ }$$

(4)

where $\mathit{r}\mathit{a}\mathit{n}\mathit{d}\left(\mathbf{1}\mathbf{,}\mathbf{1}\right)$ is a random number between (0,1). ${\mathit{C}}_{\mathbf{1}}$ and ${\mathit{C}}_{\mathbf{2}}$ are learning factors. We set the learning factor to 2 $\left({\mathit{C}}_{\mathbf{1}}\mathbf{=}{\mathit{C}}_{\mathbf{2}}\mathbf{=}\mathbf{2}\right)$.

Step 5. Update each particle’s position using the following equation:

$$\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\mathbf{+}\mathbf{1}\right)}\mathbf{=}\overrightarrow{\mathit{p}\mathit{o}{\mathit{s}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\right)}\mathbf{+}\overrightarrow{\mathit{V}\mathit{e}{\mathit{l}}_{\mathit{q}}\left(\mathit{t}\mathbf{,}\mathit{I}\mathbf{+}\mathbf{1}\right)}\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(5)

Repeat steps 2–5 until the stopping criterion is satisfied.

Step 6. $\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}$ contains the optimal measurements for the malicious sensors. $\left[\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}\mathbf{,}\overrightarrow{\mathit{S}\mathit{T}}\right]$ is an optimal attack signature, wherein $\overrightarrow{\mathit{S}\mathit{T}}$ is a vector containing the state of each DPV in $\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}$.

EHS generates attack signatures through repeated games and keeps updating the attack signature database by adding new signatures.

SIDS: The flowchart for SIDS is provided in Figure 6. The task of SIDS is to classify DPVs into one of the two groups: malicious (1) and normal (0). It uses an ensemble of GRNNs to perform the classification task. Each member GRNN is trained to classify one DPV as either normal or malicious. Target outputs vary between member GRNNs. During training, each GRNN’s target outputs are the states (malicious state or normal state) of the corresponding DPV at a given time step. However, the input vectors of all member GRNNs are the same. During training, the raw input is $\overrightarrow{\mathit{g}\mathit{b}\mathit{e}\mathit{s}\mathit{t}}$. During testing, the raw input is the vector of measurements transmitted by the DPVs. SIDS sends the raw input to TMS. TMS updates $\mathit{T}\mathit{S}$ and $\mathit{S}\mathit{E}$ accordingly. SIDS estimates the time series of the skewness coefficients of residuals at each time step by moving a sliding window over the residual series in $\mathit{S}\mathit{E}$. This yields the matrix $\mathit{S}\mathit{K}$ containing the skewness series of each DPV. The input vector contains 100 consecutive most recent lagged variables from $\mathit{T}\mathit{S}$ and $\mathit{S}\mathit{K}$.

Figure 6. Flowchart for SIDS.

Figure 6. Flowchart for SIDS.

It is worth noting that when malicious DPVs are detected, they get converted to normal DPVs at the very time step, so that from the next time on, they provide the actual measurements. However, there are no ways to recover the actual measurements of the recovered DPVs at time step t when they are first detected as having compromised. Hence, SIDS cannot improve the performance of estimating the most trustworthy aggregate demand $\left(\mathit{D}\left(\mathit{t}\right)\right)$ at time step t. By detecting and recovering the malicious DPVs at time step t, SIDS ensures the better performance of TMS in estimating the most trustworthy aggregate demand at the later time steps. In PRDFS, TMS does not exclude the measurements obtained from the malicious DPVs while estimating $\mathit{D}\left(\mathit{t}\right)$. This is because there is a chance of the detected malicious DPVs being in benign modes at time step t, that means they have reported the actual measurements. Even if the detected malicious DPVs are in malicious modes at time step t, the distance between the malicious measurements may not be far away from the actual measurements if the attack is at the initial stage. Hence, to avoid losing information from malicious DPVs by discarding their readings, TMS adjusts the weights of the measurements to control their influence on the estimation of $\mathit{D}\left(\mathit{t}\right)$. Thus, by providing a trustworthy time series of historical aggregate demands $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$, TMS ensures the performance of DFS in forecasting the k-step ahead values of $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ is least affected by the attack.

4. Research Methodology

We compare PRDFS with a benchmark resilient demand forecasting system. No existing resilient CDFS have all the features that have been demonstrated to be effective for resilient demand forecasting systems. Hence, we implement a benchmark resilient CDFS by combining the promising solutions proposed in previous studies that complement each other and strengthen the performance of the resilient demand forecasting system. We call this benchmark algorithm BRDFS. The remaining section is organized as follows: the evaluation methodology for PRDFS and BRDFS is discussed in Section 4.1. The performance metrics against which the performance of PRDFS and BRDFS is measured are discussed in Section 4.2. A brief description of BRDFS is presented in Section 4.3. PRDFS and BRDFS are evaluated on synthetic data in the presence of on-off colluding FDIA. The details of the adversary model that generated stealthy FDIA attacks are presented in Section 4.4. The synthetic data generation process is described in Section 4.5.

4.1. Evaluation Methodology for PRDFS and BRDFS

To compare PRDFS and BRDFS, we simulate two CPS. For simplicity purposes, each CPS consists of two layers of nodes. The first layer contains 14 sensors that monitor demand continuously. The second layer contains a base station. All sensor nodes are directly connected to the base station. They transmit their measurements to the base station at regular intervals. PRDFS resides in the base station of one CPS, and BRDFS resides in the base station of the other CPS. PRDFS and BRDFS are implemented in MATLAB and their performance is tested on High Performance Computing (HPC). PRDFS and BRDFS both play multiple games with the adversary model under different percentages of malicious nodes. The important rules of the games are stated below: (1) at the beginning of each game, the attacker compromises a fixed number of malicious nodes. (2) If the defender detects a malicious node, it becomes a normal node. From the next step onwards, the attacker cannot modify its measurements. (3) An attacker cannot compromise a new or recovered sensor node. (4) The duration of each game is 1000-time steps. (5) The adversary model has partial knowledge. At the start of the game, the adversary model gets the most recent 500 measurements of all malicious sensors, whereas PRDF and BRDFS get the most recent 1000 measurements of all sensors. (6) At each time step, the data generator (described in Section 4.5) generates the measurements of sensors at that time step. The adversary model modifies the measurements of malicious sensors. All measurements are then transmitted to the base station. PRDFS and BRDFS estimate the most trustworthy measurement based on the noisy sensor measurements, append the most trustworthy measurement to the aggregate demand time series, retrain the forecasting models on the updated aggregate demand time series, and forecast one-step ahead of demand using the updated forecasting model, and classify each sensor node into one of the two types: malicious and normal. The adversary model also adapts itself based on the new measurements of malicious sensors.

4.2. Performance Measures

PRDFS and BRDFS are compared in terms of two measures. The first performance measure is the accuracy of detecting malicious nodes. The second performance measure is the resilience of the demand forecasting system in the face of ongoing attacks. These measures are described in detail in Section 4.2.1 and Section 4.2.2.

4.2.1. Accuracy in Detecting Malicious Nodes

PRDFS and BRDFS are compared with respect to their ability to correctly classify sensors as malicious (i.e., involved in stealthy FDIA attacks) or normal (i.e., not involved in attacks). When the classification is compared with the true status, there are four possible outcomes: true positive (TP), true negative (TN), false positive (FP), and false negative (FN) (details are available in

Table 2. Performance Metric.

		Predicted Label
		Positive (Malicious)	Negative (Normal)
Actual Status	Positive (Malicious)	True Positive (TP)	False Negative (FN)
	Negative (Normal)	False Positive (FP)	True Negative (TN)

).

We use accuracy rate and false positive rate (FPR) as the performance criteria metric based on the above performance metric shown in Table 2. The accuracy rate and FPR are measured using the following formulae.

$$\mathit{A}\mathit{c}\mathit{c}\mathit{u}\mathit{r}\mathit{a}\mathit{c}\mathit{y}\mathbf{ }\mathit{r}\mathit{a}\mathit{t}\mathit{e}\mathbf{=}\frac{\mathit{T}\mathit{P}\mathbf{+}\mathit{T}\mathit{N}}{\mathit{T}\mathit{P}\mathbf{+}\mathit{T}\mathit{N}\mathbf{+}\mathit{F}\mathit{P}\mathbf{+}\mathit{F}\mathit{N}}\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(6)

$$\mathit{F}\mathit{P}\mathit{R}\mathbf{=}\frac{\mathit{F}\mathit{P}}{\mathit{F}\mathit{P}\mathbf{+}\mathit{t}\mathit{N}}\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(7)

4.2.2. The Resilience of the Demand Forecasting System in the Face of Ongoing Attack

The resilience of PRDFS and BRDFS are measured using the following metric for resilience [44]:

$$\mathit{R}\mathbf{=}{\mathit{P}}_{\mathit{n}}\mathbf{/}\mathit{P}\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(8)

where ${\mathit{P}}_{\mathit{n}}$ and $\mathit{P}$ present the out-of-sample one step ahead demand forecasting accuracy with and without attacks, respectively. The predictive accuracy is measured in both point estimates and confidence intervals. In the single point estimate, the predicted output is the most expected value of demand. We use the mean arctangent absolute percentage error (MAAPE) to measure the accuracy of single point forecasts [45]. In the MAAPE approach, the accuracy is defined as follows:

$$\mathit{A}\mathit{c}\mathit{c}\mathit{u}\mathit{r}\mathit{a}\mathit{c}\mathit{y}\mathbf{=}\mathbf{100}\mathbf{-}\mathit{M}\mathit{A}\mathit{A}\mathit{P}\mathit{E}\mathbf{\times }\mathbf{100}\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(9)

$$\mathit{M}\mathit{A}\mathit{A}\mathit{P}\mathit{E}\mathbf{=}\frac{\mathbf{1}}{\mathit{N}}{\displaystyle \mathbf{\sum }}_{\mathit{t}\mathbf{=}\mathbf{1}}^{\mathit{N}}\left(\mathit{A}\mathit{A}\mathit{P}{\mathit{E}}_{\mathit{t}}\right)\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathit{f}\mathit{o}\mathit{r}\mathbf{ }\mathit{t}\mathbf{=}\mathbf{1}\mathbf{,}\mathbf{\dots }\mathbf{,}\mathit{N}$$

(10)

$$\mathit{A}\mathit{A}\mathit{P}{\mathit{E}}_{\mathit{t}}\mathbf{=}{\mathbf{tan}}^{\mathbf{-}\mathbf{1}}\left(\left|\frac{{\mathit{A}}_{\mathit{t}}\mathbf{-}{\mathit{F}}_{\mathit{t}}}{{\mathit{A}}_{\mathit{t}}}\right|\right)\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(11)

where ${\mathit{A}}_{\mathit{t}}$ and ${\mathit{F}}_{\mathit{t}}$ denote the actual and forecasted values at data point $\mathit{t}$, respectively. $\mathit{N}$ is the number of data points.

In the interval estimate, the predicted outputs are the upper and lower bounds of forecasts. In this approach, if the actual demand lies between the forecasted upper and lower bounds, the forecast is correct. Interval estimates are made for the 99.7% confidence level. The 99.7% limits lie three standard deviations below and three above the mean.

4.3. Description of the Benchmark Algorithm (BRDFS)

BRDFS consists of three modules: a demand forecasting system (DFS), a trust management system (TMS), and a malicious node detection system. The details of these systems are given below.

4.3.1. Demand Forecasting System (DFS) of BRDFS

Most of the existing resilient demand forecasting systems use a single Long-Term Short-Term Memory (LSTM) for forecasting the point estimate of the demand [9,17]. For comparison purposes, the DFS of BRDFS consists of two LSTMs for forecasting the interval estimate of the demand. LSTM1 is trained to forecast the conditional mean of demand and LSTM2 is trained to forecast the conditional variance of demand. LSTM1 is trained on the original aggregate demand time series and LSTM2 is trained on the squared residuals of LSTM1. Most of the existing resilient demand forecasting systems do not perform differencing or wavelet decomposition on the demand time series to deal with non-stationarity. Hence, BRDFS does not apply these measures as well.

4.3.2. Trust Management System (TMS) of BRDFS

BRDFS uses a popular TMS proposed in [31] to estimate the most trustworthy measurement at time step $\mathit{t}$ based on the measurements from all available sensors. This TMS is implemented as follows. Normalize all measurements at time step $\mathit{t}$ using the min-max normalization. Find the Euclidean distance between each pair of measurements. Estimate the similarity $\left(\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\right)$ between the measurements $\left(\mathit{m}{\mathit{v}}_{\mathit{i}}\left(\mathit{t}\right)\mathbf{ }\mathit{a}\mathit{n}\mathit{d}\mathbf{ }\mathit{m}{\mathit{v}}_{\mathit{j}}\left(\mathit{t}\right)\right)$ of sensors $\mathit{i}$ and $\mathit{j}$ using the following equation:

$$\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{=}\mathbf{1}\mathbf{-}\mathit{E}\mathit{u}\mathit{c}\mathit{l}\mathit{i}\mathit{d}\mathit{e}\mathit{a}\mathit{n}\mathbf{ }\mathit{d}\mathit{i}\mathit{s}\mathit{t}\mathit{a}\mathit{n}\mathit{c}\mathit{e}\mathbf{ }\mathit{b}\mathit{e}\mathit{t}\mathit{w}\mathit{e}\mathit{e}\mathit{n}\mathbf{ }\mathit{m}{\mathit{v}}_{\mathit{i}}\left(\mathit{t}\right)\mathbf{ }\mathit{a}\mathit{n}\mathit{d}\mathbf{ }\mathit{m}{\mathit{v}}_{\mathit{j}}\left(\mathit{t}\right)\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(12)

Estimate the provenance similarity $\left(\mathit{P}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\right)$ between $\mathit{m}{\mathit{v}}_{\mathit{i}}\left(\mathit{t}\right)$ and $\mathit{m}{\mathit{v}}_{\mathit{j}}\left(\mathit{t}\right)$ which is the ratio of the total number of common nodes in both provenances to the total number of nodes in the provenance with the longer length. Adjust $\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}$ to $\mathit{P}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}$ using the following pseudocode:

$$\mathit{i}\mathit{f}\mathbf{ }\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{\ge }\mathbf{0.6}\mathbf{ }\mathbf{then}\mathbf{ }\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{=}\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{+}\left(\mathbf{1}\mathbf{-}\mathit{P}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\right)\mathbf{ }$$

$$\mathit{e}\mathit{l}\mathit{s}\mathit{e}\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{=}\mathit{D}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{-}\mathit{P}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}\mathbf{ }\mathbf{ }\mathit{e}\mathit{n}\mathit{d}\mathbf{ }$$

Since in our simulation, we assume that all sensor nodes are directly linked to the based station, $\mathit{P}{\mathit{S}}_{\mathit{i}\mathbf{,}\mathit{j}}$ is always zero. Find the average similarity ${\overline{\mathit{D}\mathit{S}}}_{\mathit{j}}$ of each measurement $\left(\mathit{m}{\mathit{v}}_{\mathit{j}}\left(\mathit{t}\right)\right)$ to all other measurements.

$${\overline{\mathit{D}\mathit{S}}}_{\mathit{j}}\mathbf{=}\frac{{{\displaystyle \mathbf{\sum }}}_{\mathit{i}\mathbf{=}\mathbf{1}}^{\mathit{n}\mathbf{-}\mathbf{1}}{\widehat{\mathit{D}\mathit{S}}}_{\mathit{i}\mathbf{,}\mathit{j}}}{\mathit{n}\mathbf{-}\mathbf{1}}\mathbf{ }\mathbf{ }\mathbf{ }$$

(13)

where $\mathit{n}$ represents the total number of measurements.

Find the measurement $\mathit{m}{\mathit{v}}_{\mathit{j}}\left(\mathit{t}\right)$ with the highest ${\overline{\mathit{D}\mathit{S}}}_{\mathit{j}}$ as the mean represents the total number of measurements $\left(\mathit{M}\right)$ measurement (i.e., the most trustworthy measurement). Estimate the standard deviation $\left(\mathit{\sigma }\right)$ of $\mathit{M}$. Generate a Gaussian function $\mathit{f}\left(\mathit{M}\mathbf{,}\mathit{\sigma }\right)$ with $\mathit{M}$ and $\mathit{\sigma }$. Assign an intermediate trust score ${\widehat{\mathit{s}}}_{\mathit{d}}\left(\mathit{t}\right)$ to each measurement at time step $\mathit{t}$ using $\mathit{f}\left(\mathit{M}\mathbf{,}\mathit{\sigma }\right)$. Estimate the final score $\left({\overline{\mathit{s}}}_{\mathit{d}}\left(\mathit{t}\right)\right)$ of each measurement using the following linear model.

$${\overline{\mathit{s}}}_{\mathit{d}}\left(\mathit{t}\right)\mathbf{=}{\mathit{c}}_{\mathit{d}}{\overline{\mathit{s}}}_{\mathit{n}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)\mathbf{+}\left(\mathbf{1}\mathbf{-}{\mathit{c}}_{\mathit{d}}\right){\widehat{\mathit{s}}}_{\mathit{d}}\left(\mathit{t}\right)\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }\mathbf{ }$$

(14)

where ${\overline{\mathit{s}}}_{\mathit{n}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)$ is current reputation score of the sensor node made the measurement. ${\mathit{c}}_{\mathit{d}}$ is a constant to be specified by the user. We set ${\mathit{c}}_{\mathit{d}}$ to 0.5.

Set ${\overline{\mathit{s}}}_{\mathit{d}}\left(\mathit{t}\right)$ as the intermediate reputation score $\left({\widehat{\mathit{s}}}_{\mathit{n}}\left(\mathit{t}\right)\right)$ of the corresponding sensor. Estimate the final reputation score $\left({\overline{\mathit{S}}}_{\mathit{n}}\left(\mathit{t}\right)\right)$ of the sensor node at the end of time step $\mathit{t}$ using the following equation and we set the value of ${\mathit{c}}_{\mathit{d}}$ to 0.5.

$${\overline{\mathit{s}}}_{\mathit{n}}\left(\mathit{t}\right)\mathbf{=}{\mathit{c}}_{\mathit{d}}{\overline{\mathit{s}}}_{\mathit{n}}\left(\mathit{t}\mathbf{-}\mathbf{1}\right)\mathbf{+}\left(\mathbf{1}\mathbf{-}{\mathit{c}}_{\mathit{d}}\right){\widehat{\mathit{s}}}_{\mathit{n}}\left(\mathit{t}\right)\mathbf{ }\mathbf{ }\mathbf{ }$$

(15)

4.3.3. Malicious Node Detection System of BRDFS

BRDFS uses a skewness detector proposed in [37] and a correlation detector CAD proposed in [38] to detect the compromised nodes. If a node is found compromised by the skewness detector and/or CAD, the node is considered as compromised. The details of the above anomaly detectors are given below.

Skewness Detector

This skewness detector is proposed in [37]. The idea is, if a sensor is involved in a collusion attack, the distribution of forecasting residuals of the measurements from the sensor (that usually tend to be normal) should be skewed to the left or right.

The training phase of the skewness detector includes the following steps. Step 1: Fit a forecasting model to the time series of the sensor’s readings. Step 2: Find the residuals of the forecasting model. Step 3: Estimate the rolling skewness coefficients of the residual series using the rolling window of $\mathit{N}$ time steps. We set $\mathit{N}$ to 30. This gives a series of skewness coefficients. Step 4: Estimate the unconditional mean $\left(\mathit{m}\right)$ and standard deviation $\left(\mathit{s}\right)$ of the skewness series. Then compute the acceptable range $\left(\mathbf{-}\mathit{\delta }\mathbf{=}\mathit{m}\mathbf{-}\mathbf{3}\mathit{s}\mathbf{,}\mathbf{ }\mathit{a}\mathit{n}\mathit{d}\mathbf{+}\mathit{\delta }\mathbf{=}\mathit{m}\mathbf{+}\mathbf{3}\mathit{s}\right)$ for the skewness of the forecasting error distribution.

During the stealthy attack distribution, the following steps are followed. Step 1: Take the new sequence $\left({\mathit{r}}_{\mathbf{0}}\right)$ of forecasting residuals over the last $\mathit{N}$ time steps. Step 2: Generate a sequence $\left({\mathit{r}}_{\mathit{r}\mathit{a}\mathit{n}\mathit{d}}\right)$ of normal random number with $\mathit{\mu }$ and ${\mathit{\sigma }}^{\mathbf{2}}$ estimated in the training phase. The length of ${\mathit{r}}_{\mathit{r}\mathit{a}\mathit{n}\mathit{d}}$ is $\mathbf{0.05}\mathit{N}$. Step 3: Replace a small proportion of entries in ${\mathit{r}}_{\mathbf{0}}$ with ${\mathit{r}}_{\mathit{r}\mathit{a}\mathit{n}\mathit{d}}$ and generate a new sequence ${\mathit{r}}_{\mathit{t}\mathit{e}\mathit{s}\mathit{t}}$ for testing. Step 4: Compute the skewness coefficient $\left(\mathit{S}\mathit{C}\right)$ of ${\mathit{r}}_{\mathit{t}\mathit{e}\mathit{s}\mathit{t}}$. If $\mathit{S}\mathit{C}$ falls outside the acceptable range $\left(\mathbf{-}\mathit{\delta }\mathbf{,}\mathbf{+}\mathit{\delta }\right)$, raise the alarm that the sensor is participating in the collusion attack.

Correlation Detector ‘CAD’

The correlation detector, called CAD, is proposed in [38] to detect compromised nodes. CAD includes two sub-schemes: temporal collusion detection scheme and spatial collusion detection scheme. If both sub-schemes find a node anomalous, the node is regarded as compromised.

Temporal collusion detection sub-scheme performs the following steps to detect the anomalous nodes. Step 1: Estimate the correlation between each node’s measurements at time step $\mathit{t}$ and at time step $\mathit{t}\mathbf{+}\mathbf{1}$. Step 2: Estimate a predefined confidence interval for temporal correlations based on the bootstrap distribution. If at a given time window, the temporal correlation falls outside the confidence interval, the node is considered a malicious node.

The spatial collusion detection sub-scheme performs the following steps. Step 1: Compute the Pearson’s correlation coefficient between the measurements of each pair of sensors over time window $\mathit{t}$ (a time window consists of multiple time steps). This gives a correlation matrix which is called the correlation map. Each vector of the matrix contains the pairwise correlation coefficients between the measurements of one particular sensor $\left(\mathit{i}\right)$ and that of every other sensor over time window $\mathit{t}$—this vector is the location of sensor $\mathit{i}$ in the location map at time window $\mathit{t}$. Step 2: Estimate the global centroid which is the vector in the matrix that has the smallest distance from all vectors in the matrix. Step 3: Perform $\mathit{k}$-means clustering to partition the vectors into $\mathit{k}$ clusters. Step 4: If a vector’s distance from the global centroid is greater than 0.5, and if the vector’s distance from the centroid of its closest cluster is greater than 0.3, the sensor represented by the vector is classified as compromised.

4.4. Adversary Model

Let there be $\mathit{N}$ total number of sensors in a complete sensor network. The attacker can only compromise $\mathit{k}$ out of $\mathit{N}$ sensors. Step 1, randomly choose $\mathit{k}$ sensors to be compromised: generate a random number for each sensor. Sort the sensors based on the corresponding random numbers in ascending order. Select the first $\mathit{k}$ sensors for compromising. Step 2, record each compromised sensor’s untampered readings over the next 500-time steps. Build and train a pair of convolutional neural networks (CNN) for each compromised sensor. The first CNN predicts the conditional mean of the individual time series at time step $\mathit{t}$—the inputs of this network are the past values of the time series, and the desired outputs are one-step-ahead values of the time series. The second network predicts the conditional variance of the conditional mean at time step $\mathit{t}$—the inputs of this network are the past squared residuals of the first CNN, and the desired outputs are one-step-ahead squared residuals from the first CNN. Step 3, to estimate the modified readings for the compromised sensors, construct a 95% confidence interval for the reading of each compromised sensor $\left(\mathit{l}\right)$ at time step $\mathit{t}$ based on the corresponding conditional mean $\left({\widehat{\mathit{m}}}_{\mathit{l}}\left(\mathit{t}\right)\right)$ and conditional variance $\left({\widehat{\mathit{v}}}_{\mathit{l}}^{\mathbf{2}}\left(\mathit{t}\right)\right)$ forecast. Step 4, Generate $\mathit{k}$ random numbers in the range $\left[\mathbf{0}\mathbf{,}\mathbf{1}\right]$, one for each compromised sensor. If the random number is less than 0.5, the corresponding compromised sensor $\left(\mathit{l}\right)$ uses the following formula to calculate its tampered measurement $\left(\mathit{m}{\mathit{v}}_{\mathit{l}}\right)$ at time step $\mathit{t}$:

$$\mathit{m}{\mathit{v}}_{\mathit{l}}\mathbf{=}{\widehat{\mathit{m}}}_{\mathit{l}}\left(\mathit{t}\right)\mathbf{+}{\widehat{\mathit{\vartheta }}}_{\mathit{l}}\left(\mathit{t}\right)\mathbf{\times }\mathit{r}\mathbf{ }\mathbf{ }\mathbf{ }$$

(16)

where $\mathit{r}$ is the normal random number with mean 0 and standard deviation 1.

Otherwise, it reports the upper or lower bound of the 99.7% confidence interval as the reading of the compromised sensor at time step $\mathit{t}$ to shift the trend upward or downwards, respectively. Step 5, if the defender fails to identify the $\mathit{l}$th compromised sensor at time step $\mathit{t}$, retrain the corresponding CNN pair with the reported measurement at time step $\mathit{t}$. Step 6, repeat steps 3–5 to estimate the readings for the compromised sensors at each time step.

4.5. Data Generator

We build a data generator that produces data for an infinite time, so that we can evaluate the performance of the proposed forecasting system. In non-stationary time series, various local frequencies exist at different resolution levels. We use wavelets to generate synthetic non-stationary time series. The mother wavelet and the resolution levels $\left(\mathit{n}\right)$ along with the values of approximation coefficients at the highest level and detail coefficients at each level must be specified to generate a time series. We complete the following steps to generate a time series for each sensor. Step 1: Choose a wavelet and a set of scales (or level of resolution) (say 1 to $\mathit{n}$) where the most valuable frequencies will be assigned. We use Daubechies mother wavelet of order 2 $\left(\mathit{d}\mathit{b}\mathbf{2}\right)$. The number of resolution levels $\left(\mathit{n}\right)$ is chosen to be three. Step 2: We define a separate sinusoidal function for the generation of the approximation coefficients at level 3 and the detail coefficients at each resolution level. The higher the resolution level, the smaller the number of coefficients, due to subsampling. To generate a signal of length 10,000 with ‘db2’ as mother wavelet and three levels of resolution, we need to specify 1252 level-3 approximation coefficients, 1252 level-3 detail coefficients, 2502 level-2 detail coefficients, and 5001 level-1 detail coefficients.

$${\mathit{w}}_{\mathit{j}\mathbf{,}\mathit{t}}\mathbf{=}\mathit{A}\mathbf{ }\mathit{sin}\left(\mathit{B}\left(\mathit{t}\mathbf{+}\mathit{C}\right)\right)\mathbf{+}\mathit{D}\mathbf{+}\mathit{\sigma }\mathit{r}\mathbf{ }\mathbf{ }\mathbf{ }$$

(17)

where ${\mathit{w}}_{\mathit{j}\mathbf{,}\mathit{t}}$ is the (approximation or detail) coefficient at scale $\mathit{j}$ and time point $\mathit{t}$. $\mathit{A}\mathbf{=}\mathit{a}\mathit{m}\mathit{p}\mathit{l}\mathit{i}\mathit{t}\mathit{u}\mathit{d}\mathit{e}$; $\mathit{C}\mathbf{=}\mathit{p}\mathit{h}\mathit{a}\mathit{s}\mathit{e}\mathbf{ }\mathit{s}\mathit{h}\mathit{i}\mathit{f}\mathit{t}$; $\mathit{D}\mathbf{=}\mathit{v}\mathit{e}\mathit{r}\mathit{t}\mathit{i}\mathit{c}\mathit{a}\mathit{l}\mathbf{ }\mathit{s}\mathit{h}\mathit{i}\mathit{f}\mathit{t}$; $\mathit{B}\mathbf{=}\mathbf{2}\mathit{\pi }\mathit{f}$; $\mathit{t}\mathbf{=}\mathit{t}\mathit{i}\mathit{m}\mathit{e}\mathbf{ }\mathit{s}\mathit{t}\mathit{e}\mathit{p}$; $\mathit{\sigma }\mathbf{=}\mathit{s}\mathit{t}\mathit{a}\mathit{n}\mathit{d}\mathit{a}\mathit{r}\mathit{d}\mathbf{ }\mathit{d}\mathit{e}\mathit{v}\mathit{i}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}$; $\mathit{r}\mathbf{=}\mathit{G}\mathit{a}\mathit{u}\mathit{s}\mathit{s}\mathit{i}\mathit{a}\mathit{n}\mathbf{ }\mathit{n}\mathit{o}\mathit{i}\mathit{s}\mathit{e}$. The values of function parameters $\left(\mathit{A}\mathbf{,}\mathit{f}\mathbf{,}\mathit{C}\mathbf{,}\mathit{D}\mathbf{,}\mathit{\sigma }\right)$ are arbitrarily assigned.

Step 3: Reconstruct the signal based on the selected mother wavelet and resolution level using the MATLAB function ‘waverec’. We perform augmented Dickey-Fuller tests on the time series and its first and second order derivatives to check whether the time series is stationary or not. The test confirms that the time series and its derivatives are non-stationary. Step 4: At each time step, estimate the mean $\left(\mathit{\mu }\right)$ of all measurements of demand obtained from 14 sensors to get the aggregate demand time series.

5. Results and Discussion

The performances of PRDFS and BRDFS were comparatively tested on 11 games with an adversary model. Each game was conducted with different percentages of malicious nodes. The duration of each game was 1000-time steps. The simulation results provide interesting insight into the comparative efficacy of PRDFS and BRDFS.

5.1. The Comparative Performance of PRDFS and BRDFS in Terms of Demand Forecasting Resilience to Stealth Attacks

Major Findings

• PRDFS exhibits an almost perfect mean resilience value of $\mathbf{~}\mathbf{0.99}$ over the time frame of the game (i.e., 1000-time steps) regardless of the percentage of the nodes compromised by the adversary (Figure 7). This is mainly because it detects and recovers all compromised nodes at the very early stages of games (within first 30-time steps) (Figure 10).

Figure 7. A comparison of resilience between PRDFS and BRDFS under different percentages of malicious nodes. Figure 7 shows the mean resilience of PRDFS and BRDFS in demand forecast accuracy under different percentages of malicious nodes. The X-axis represents the percentages of malicious nodes, and the Y-axis represents the mean resilience. The mean resilience of a system under a given percentage of malicious nodes is the ratio of mean (point or interval) demand forecasting accuracy under attacks with the given percentage of malicious nodes to the mean (point or interval) demand forecasting accuracy under no attack. The red and blue lines plot the mean resilience of PRDFS in respect to point demand forecast accuracy and interval demand forecast accuracy, respectively, over 1000-time steps under different percentages of malicious nodes. The pink and green lines plot the mean resilience of PRDFS in terms of point demand forecast accuracy and interval demand forecast accuracy, respectively, over the first 40-time steps under different percentages of malicious nodes. The orange and black line plots the mean resilience of BRDFS in terms of point demand forecast accuracy and interval demand forecast accuracy, respectively, over 1000-time steps under different percentages of malicious nodes. The figure shows that PRDFS demonstrates near perfect mean resilience over 1000-time steps in both point and interval forecasting accuracy for all percentages of malicious nodes. However, the mean resilience of PRDFS over the first 40-time steps shows the signs of weakening with the increase in malicious nodes. In BRDFS, the mean resilience over 1000-time steps falls sharply as the percentage of malicious nodes increases. PRDFS is observed to have higher resilience in the interval demand forecasting accuracy than in the point forecasting accuracy. The opposite trend is seen in BRDFS.

Figure 7. A comparison of resilience between PRDFS and BRDFS under different percentages of malicious nodes. Figure 7 shows the mean resilience of PRDFS and BRDFS in demand forecast accuracy under different percentages of malicious nodes. The X-axis represents the percentages of malicious nodes, and the Y-axis represents the mean resilience. The mean resilience of a system under a given percentage of malicious nodes is the ratio of mean (point or interval) demand forecasting accuracy under attacks with the given percentage of malicious nodes to the mean (point or interval) demand forecasting accuracy under no attack. The red and blue lines plot the mean resilience of PRDFS in respect to point demand forecast accuracy and interval demand forecast accuracy, respectively, over 1000-time steps under different percentages of malicious nodes. The pink and green lines plot the mean resilience of PRDFS in terms of point demand forecast accuracy and interval demand forecast accuracy, respectively, over the first 40-time steps under different percentages of malicious nodes. The orange and black line plots the mean resilience of BRDFS in terms of point demand forecast accuracy and interval demand forecast accuracy, respectively, over 1000-time steps under different percentages of malicious nodes. The figure shows that PRDFS demonstrates near perfect mean resilience over 1000-time steps in both point and interval forecasting accuracy for all percentages of malicious nodes. However, the mean resilience of PRDFS over the first 40-time steps shows the signs of weakening with the increase in malicious nodes. In BRDFS, the mean resilience over 1000-time steps falls sharply as the percentage of malicious nodes increases. PRDFS is observed to have higher resilience in the interval demand forecasting accuracy than in the point forecasting accuracy. The opposite trend is seen in BRDFS.

• During the first 40-time steps of games while the attack is still occurring or just ended, the mean demand forecasting resilience of PRDFS slightly decreases but it never goes below 0.8 (its resilience ranges from 0.82 to 1), regardless of the percentage of the malicious nodes (Figure 7). In contrast, the mean demand forecasting resilience sharply decreases in BRDFS over the simulation period (i.e., 1000 time-steps) as the percentage of malicious nodes increases (Figure 7).
• In PRDFS, the interval demand forecast accuracy demonstrates greater resilience than the point demand forecast accuracy, while the opposite trend is observed in BRDFS (Figure 7). In BRDFS, the interval demand forecast accuracy has weaker resilience compared to the point demand forecast accuracy and the trend becomes more pronounced as the percentage of malicious nodes increases. This is mainly because BRDFS fails to detect most malicious nodes (Figure 11). All colluding malicious nodes report similar measurements and thereby artificially reduce the uncertainty around the mean estimate, which results in low interval demand forecast accuracy and yields a very high FPR in detecting malicious nodes.

We attribute the high resilience of PRDFS in the face of an attack to the combined effect of the strong performances from its modules: SIDS, TMS, and DFS. Section 5.2, Section 5.3 and 5.4 discuss the comparative performances of the individual subsystems.

5.2. The Comparative Performance of PRDFS and BRDFS in Detecting Malicious Nodes

5.2.1. Major Findings:

• PRDFS maintains a high malicious node detection accuracy of around 95% (ranging from 97% to 89%) while still maintaining a low FPR, ranging from 3–9%, with the increasing percentage of malicious nodes (Figure 8 and Figure 9). In contrast, the accuracy of BRDFS sharply falls with the rising FPR as the percentage of malicious nodes increases (Figure 8 and Figure 9).

Figure 8. A comparison of accuracy between PRDFS and BRDFS for classifying normal and malicious nodes. Figure 8 depicts the node classification accuracy of PDRFS and BRDFS under the varying percentage of malicious sensor nodes. The X-axis represents the percentages of sensor nodes that are malicious in the simulation trial. The Y-axis represents the mean accuracy of classifying the sensor nodes into one of the two classes (malicious and normal) over the 1000-time steps. The classification is done at each time step. The blue line shows the classification accuracy of PRDFS under different percentages of malicious nodes, whereas the orange line shows the mean classification accuracy of BRDFS under different percentages of malicious nodes. PRDFS achieves high malicious node detection accuracy. The accuracy of PRDFS is almost always over 90%. The orange line shows a dramatic fall in the accuracy of BRDFS as the number of malicious nodes increases. Its accuracy varies from over 90% to 0%.

Figure 8. A comparison of accuracy between PRDFS and BRDFS for classifying normal and malicious nodes. Figure 8 depicts the node classification accuracy of PDRFS and BRDFS under the varying percentage of malicious sensor nodes. The X-axis represents the percentages of sensor nodes that are malicious in the simulation trial. The Y-axis represents the mean accuracy of classifying the sensor nodes into one of the two classes (malicious and normal) over the 1000-time steps. The classification is done at each time step. The blue line shows the classification accuracy of PRDFS under different percentages of malicious nodes, whereas the orange line shows the mean classification accuracy of BRDFS under different percentages of malicious nodes. PRDFS achieves high malicious node detection accuracy. The accuracy of PRDFS is almost always over 90%. The orange line shows a dramatic fall in the accuracy of BRDFS as the number of malicious nodes increases. Its accuracy varies from over 90% to 0%.

Figure 9. A comparison of False Positive Rates (FPR) for PRDFS and BRDFS for detecting malicious nodes. Figure 9 shows the false positive rates (FPR) for PRDFS and BRDFS when detecting malicious nodes. The X-axis represents the percentages of sensor nodes that were malicious in the simulation trial. The Y-axis represents the false positive rates (FPR). The blue line shows the FPR of PRDFS under different percentages of malicious nodes whereas the orange line shows the FPR of BRDFS under different percentages of malicious nodes. The FPR in PRDFS never exceeds 9%. The FPR of BRDFS rises sharply with the percentage of malicious nodes. The FPR of BRDFS varies from less than 10% to around 75%.

Figure 9. A comparison of False Positive Rates (FPR) for PRDFS and BRDFS for detecting malicious nodes. Figure 9 shows the false positive rates (FPR) for PRDFS and BRDFS when detecting malicious nodes. The X-axis represents the percentages of sensor nodes that were malicious in the simulation trial. The Y-axis represents the false positive rates (FPR). The blue line shows the FPR of PRDFS under different percentages of malicious nodes whereas the orange line shows the FPR of BRDFS under different percentages of malicious nodes. The FPR in PRDFS never exceeds 9%. The FPR of BRDFS rises sharply with the percentage of malicious nodes. The FPR of BRDFS varies from less than 10% to around 75%.

• PRDFS successfully detects all malicious nodes within the first 30-time steps regardless of the percentage of the malicious nodes, whereas BRDFS fails to detect any malicious node after 15-time steps (Figure 10). It appears that BRDFS has a shorter time window for detecting malicious nodes than PRDFS.
• BRDFS manages to correctly detect malicious nodes when the percentage of malicious nodes is low (Figure 10 and Figure 11). BRDFS fails to detect any malicious node when the percentage of malicious nodes is 30% or over (Figure 11).

Figure 10. The distribution of detected malicious nodes by PRDFS and BRDFS over time. (a) PRDFS and (b) BRDFS showing the distributions of the number of malicious nodes detected by PRDFS and BRDFS, respectively, in contrast to total number of malicious nodes in the system in different time windows in all games. The length of each time window is equal to five-time steps. The X-axis represents the time window, and the Y-axis represents the number of malicious nodes. In one game, the total number of nodes is 14. Ten games out of all 11 games contain malicious nodes. The total number of nodes in ten games together is 140, among which 77 nodes are malicious. Nodes are compromised at the beginning of the game. If a malicious node is detected, it is converted to benign nodes. If an algorithm detects a higher number of malicious nodes early in the games, there will be a lower total number of malicious nodes later. We add up malicious nodes detected in all games at each time window to get the total number of malicious nodes at the given time window. The orange part of each stacked bar represents the total number of malicious nodes undetected at the beginning of each time window, whereas the blue part represents the total number of malicious nodes detected in that time window. PRDFS detects the largest number of malicious nodes at the fourth time window. BRDFS appears to have a shorter time window for detecting malicious nodes than PRDFS. BRDFS have not detected any malicious nodes after the third time window.

Figure 10. The distribution of detected malicious nodes by PRDFS and BRDFS over time. (a) PRDFS and (b) BRDFS showing the distributions of the number of malicious nodes detected by PRDFS and BRDFS, respectively, in contrast to total number of malicious nodes in the system in different time windows in all games. The length of each time window is equal to five-time steps. The X-axis represents the time window, and the Y-axis represents the number of malicious nodes. In one game, the total number of nodes is 14. Ten games out of all 11 games contain malicious nodes. The total number of nodes in ten games together is 140, among which 77 nodes are malicious. Nodes are compromised at the beginning of the game. If a malicious node is detected, it is converted to benign nodes. If an algorithm detects a higher number of malicious nodes early in the games, there will be a lower total number of malicious nodes later. We add up malicious nodes detected in all games at each time window to get the total number of malicious nodes at the given time window. The orange part of each stacked bar represents the total number of malicious nodes undetected at the beginning of each time window, whereas the blue part represents the total number of malicious nodes detected in that time window. PRDFS detects the largest number of malicious nodes at the fourth time window. BRDFS appears to have a shorter time window for detecting malicious nodes than PRDFS. BRDFS have not detected any malicious nodes after the third time window.

Figure 11. Number of malicious nodes detected vs. number of malicious nodes undetected by PRDFS and BRDFS. (a) PRDFS, and (b) BRDFS. Each bar in the subplots ‘(a) PRDFS’ and ‘(b) BRDFS’ shows the number of malicious nodes detected and the number of malicious nodes undetected by PRDFS and BRDFS, respectively, under a given percentage of malicious nodes. In both subplots, the X-axis represents the percentage of nodes compromised, whereas the Y-axis represents the number of nodes. The blue part of the bar indicates the number of detected malicious nodes. The orange part of the bar indicates the number of undetected malicious nodes. In subplot ‘(a)PRDF’, there is no orange part in any bar. This is because PRDFS has detected all the malicious nodes under all percentages of malicious nodes. In subplot ‘(b)BRDFS’, on the contrary, all bars are completely orange, except for the first two bars. A completely orange bar indicates that no malicious node is detected by BRDFS under the given percentage of malicious nodes. BRDFS identifies all malicious nodes when the percentage of malicious nodes is 10%. It can detect most of the malicious nodes under attacks with 20% malicious nodes.

Figure 11. Number of malicious nodes detected vs. number of malicious nodes undetected by PRDFS and BRDFS. (a) PRDFS, and (b) BRDFS. Each bar in the subplots ‘(a) PRDFS’ and ‘(b) BRDFS’ shows the number of malicious nodes detected and the number of malicious nodes undetected by PRDFS and BRDFS, respectively, under a given percentage of malicious nodes. In both subplots, the X-axis represents the percentage of nodes compromised, whereas the Y-axis represents the number of nodes. The blue part of the bar indicates the number of detected malicious nodes. The orange part of the bar indicates the number of undetected malicious nodes. In subplot ‘(a)PRDF’, there is no orange part in any bar. This is because PRDFS has detected all the malicious nodes under all percentages of malicious nodes. In subplot ‘(b)BRDFS’, on the contrary, all bars are completely orange, except for the first two bars. A completely orange bar indicates that no malicious node is detected by BRDFS under the given percentage of malicious nodes. BRDFS identifies all malicious nodes when the percentage of malicious nodes is 10%. It can detect most of the malicious nodes under attacks with 20% malicious nodes.

5.2.2. Further Observations

We attribute the above observations to the following characteristics of PRDFS and BRDFS:

• PRDFS uses a supervised intrusion detection system (SIDS) for detecting malicious nodes.

  ⮚

  SIDS learns from the input and output data sets that are contamination-free since they are generated through the game theoretic approach. It employs a supervised learning algorithm (GRNN) to distinguish between malicious and normal nodes.

  ⮚

  It can detect malicious nodes even when all nodes are malicious because it compares the present behaviors of the nodes in respect to dynamic spatio-temporal patterns of trustworthiness and the skewness of residual distribution with previous correct behaviors.

• BRDFS uses an unsupervised anomaly-based intrusion detection system (AIDS) for detecting malicious nodes.

⮚

AIDS learn from unlabeled real data that are contaminated with many undetected malicious nodes. Hence, its learning is not very effective.

⮚

AIDS learn under the assumption that statistically rare or atypical behaviors are abnormal and are possible signs of maliciousness, whereas typical or common behaviors are normal. Hence, AIDS can only detect malicious nodes when the percentage of malicious nodes is low, and the malicious nodes are operating for a short period of time where it is not possible to inject a significant number of malicious instances in the data set. However, stealthy malicious nodes are highly difficult to detect at very early stage.

⮚

The skewness detector and the temporal collusion detection scheme of CAD uses confidence interval-based anomaly detection. If a node’s behavioral traits fall outside the confidence interval, the node is detected as malicious. However, such a strategy is ineffective against collusion attacks. In collusion attacks, the adversary takes control of multiple sensors and changes the readings of these sensors to the desired upward or downward direction in such a way that the malicious nodes always lie within the confidence intervals of correlation coefficients and skewness coefficients. One of the major challenges in time series forecasting is that the distribution of the time series changes over time. Hence, the forecasting model needs to be adapted to the most recent data—the adversaries take advantage of this situation. Returning the upper or lower extreme of the acceptable range as the readings of the compromised sensors, they can gradually move the forecasting model in the wrong direction without getting caught. Since the forecasting model is evolving towards the false data, over time, the non-compromised sensors are marked as compromised one after another and will lose all influence over the model’s evolution, thereby accelerating the compromise rate of the entire forecasting system.

⮚

The spatial collusion detector of the CAD in BRDFS uses unsupervised clustering to separate malicious nodes. These clustering algorithms are based on arbitrary rules. Thus, it is natural that its accuracy is not as good as PRDFS.

5.3. The Comparative Performance of PRDFS and BRDFS in Estimating the Most Trustworthy Measurement of the Current Aggregate Demand $\left(\overrightarrow{D\left(t\right)}\right)$

5.3.1. Major Findings

• The demand forecast resilience is directly correlated to the accuracy in estimating current aggregate demand $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ that acts as the input signal while forecasting the demand time series (Figure 7 and Figure 12).
• Since PRDFS detects and recovers all compromised nodes at the very early stages of games (within first 30-time steps) (Figure 10), the mean accuracy of estimating $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ in PRDFS is almost constant to around 98% over the time frame of the game (i.e., 1000-time steps) regardless of the percentage of the nodes compromised by the adversary (Figure 12).
• During the first 40-time steps of games, while the attack is still occurring or just ended, the mean $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ estimation accuracy in PRDFS decreases from around 98% to 83% as the percentage of malicious nodes increases. In contrast, the mean $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ estimation accuracy in BRDFS falls sharply over the simulation period (i.e., 1000 time-steps) as the percentage of malicious nodes increases.

Figure 12. A comparison of accuracy between PRDFS and BRDFS for estimating current aggregate demand. Figure 12 depicts the prediction accuracy of current aggregate demands in PDRFS and BRDFS under the varying percentage of malicious sensor nodes. The accuracy is estimated based on the difference between the most trustworthy measurement of the current aggregate demand predicted by the TMS and the actual current aggregate demand (which is the simple average of the actual current measurements of the sensors). The X-axis represents the percentages of sensor nodes that were malicious in the simulation trial. The Y-axis represents the accuracy. The accuracy is estimated based on MAAPE. The green and blue lines plot the mean accuracy of PRDFS in respect to the mean current aggregate demand over 1000-time steps and over the first 40-time steps, respectively, under different percentages of malicious nodes. The red line plots the mean accuracy of BRDFS in terms of the mean current aggregate demand over 1000-time steps under different percentages of malicious nodes. The green line is almost flat to X-axis, indicating that the mean accuracy of mean current aggregate demand over 1000-time steps predicted by (the TMS of) PRDFS is not sensitive to the percentage of malicious nodes in the system. The blue line drops gradually as the percentage of malicious nodes increases, indicating that the mean accuracy of mean current aggregate demand over the first 40-time steps predicted by PRDFS is mildly sensitive to the percentage of malicious nodes in the system. The red line falls at an accelerated rate indicating that the mean accuracy of mean current aggregate demand over 1000-time steps predicted by BRDFS is highly sensitive to the percentage of malicious nodes in the system.

Figure 12. A comparison of accuracy between PRDFS and BRDFS for estimating current aggregate demand. Figure 12 depicts the prediction accuracy of current aggregate demands in PDRFS and BRDFS under the varying percentage of malicious sensor nodes. The accuracy is estimated based on the difference between the most trustworthy measurement of the current aggregate demand predicted by the TMS and the actual current aggregate demand (which is the simple average of the actual current measurements of the sensors). The X-axis represents the percentages of sensor nodes that were malicious in the simulation trial. The Y-axis represents the accuracy. The accuracy is estimated based on MAAPE. The green and blue lines plot the mean accuracy of PRDFS in respect to the mean current aggregate demand over 1000-time steps and over the first 40-time steps, respectively, under different percentages of malicious nodes. The red line plots the mean accuracy of BRDFS in terms of the mean current aggregate demand over 1000-time steps under different percentages of malicious nodes. The green line is almost flat to X-axis, indicating that the mean accuracy of mean current aggregate demand over 1000-time steps predicted by (the TMS of) PRDFS is not sensitive to the percentage of malicious nodes in the system. The blue line drops gradually as the percentage of malicious nodes increases, indicating that the mean accuracy of mean current aggregate demand over the first 40-time steps predicted by PRDFS is mildly sensitive to the percentage of malicious nodes in the system. The red line falls at an accelerated rate indicating that the mean accuracy of mean current aggregate demand over 1000-time steps predicted by BRDFS is highly sensitive to the percentage of malicious nodes in the system.

5.3.2. Further Observations

The following strategy differences between the TMS of PRDFS and that of BRDFS may explain their striking differences in $\left(\overrightarrow{\mathit{D}\left(\mathit{t}\right)}\right)$ estimation accuracy in the face of an attack:

• The TMS of PRDFS considers the following three factors in determining the trustworthiness of a sensor node at a time distant $\mathit{t}$: (1) the trustworthiness of the node, (2) the similarity of the sensor’s reading with the other sensors’ readings at the same time step, and (3) the similarity of the sensor’s measurement at time step $\mathit{t}$ with its past measurements, but in contrast, the TMS of BRDFS uses only the first two factors.
• The TMS of BRDFS updates the reputation score of a node at the end of a time step $\mathit{t}$ after determining the most trustworthy measurement at that time step. In the presence of on-off attacks, the normal behavior of a node at the last few time steps does not guarantee normal behavior from the node at the next time step. For this reason, the TMS of PRDFS predicts the trust score of a node at the beginning of a time step $\mathit{t}$ (before determining the most trustworthy measurement at that time step) using GRNN, which is a powerful pattern recognition algorithm capable of learning local patterns.
• In BRDFS, the reputation score of a node at time step $\mathit{t}$ is a linear function of its long-term reputation and its intermediate reputation score computed at time step $\mathit{t}$. In contrast, the TMS of PRDFS uses a powerful fuzzy clustering-based non-linear algorithm (GRNN) to predict the trust score of a node at time step $\mathit{t}$ based on its past trustworthiness scores.
• The TMS of BRDFS sets the reputation score of a node at time step $\mathit{t}$ to the estimated expected (i.e., mean) reputation score of the node at time step $\mathit{t}$. In stealthy on-off attacks, where at each time step the malicious nodes randomly choose whether to act maliciously or not, the variance of their reputation is an important indicator of their conditions. Hence, the point estimate of the expected reputation score is not very reliable. To overcome this problem, the TMS of PRDFS sets the trust score of a node to the lower bound of the predicted distribution of the plausible trust scores of the node at time step $\mathit{t}$

5.4. The Comparative Performance of PRDFS and BRDFS in Terms of Demand Forecasting when No Attack Is Underway

5.4.1. Major Findings

• PRDFS outperforms BRDFS in terms of demand forecasting accuracy by 15% or more (Figure 13).

Figure 13. Comparison of demand forecasting performance between PRDFS and BRDFS when no attack is present. Figure 13 shows the demand forecasting accuracy of PRDFS and BRDFS when no attack takes place. The Y-axis of the figure represents the accuracy. The first two bars from the left are for PRDFS and the second two bars are for BRDFS. Blue bars represent the point forecasting accuracy, whereas the orange bars represent the interval forecasting accuracy. The figure shows that PRDFS outperforms BRDFS in both point and interval forecasting accuracy.

Figure 13. Comparison of demand forecasting performance between PRDFS and BRDFS when no attack is present. Figure 13 shows the demand forecasting accuracy of PRDFS and BRDFS when no attack takes place. The Y-axis of the figure represents the accuracy. The first two bars from the left are for PRDFS and the second two bars are for BRDFS. Blue bars represent the point forecasting accuracy, whereas the orange bars represent the interval forecasting accuracy. The figure shows that PRDFS outperforms BRDFS in both point and interval forecasting accuracy.

5.4.2. Possible Explanations

We attribute the finding to the robustness of PRDFS against non-stationarity and high dimensionality. To deal with the non-stationarity in the demand time series, the DFS of PRDFS applies wavelet multiresolution analysis on the raw demand time series and decomposes the original input time series into multiple component time series with a simpler structure. Each component time series contains the information content of the original demand time series at a particular level or scale. Our study suggests that the wavelet analysis eliminates the requirement of selecting optimal seasonal lagged predictor variables (i.e., non-consecutive lagged variables) from the component time series for the demand forecasting model, which are highly difficult to identify. In the process, it also reduces the dimensionality of the feature space.

• The curse of dimensionality hits particularly hard on time series forecasting models of high or even moderate dimensional input data, since they use lagged variables as predictor variables and the lagged variables are highly correlated with each other. Hence, to reduce the dimensionality further, the DFS of PRDFS uses a hierarchical ensemble model where each member CNN is trained with only a small number of non-seasonal lagged variables. Ensemble members are trained sequentially. The first ensemble member is trained on the original demand time series. Each of the remaining ensemble members is trained on the residual series of the previous trained ensemble member. Additionally, the DFS of PRDFS uses CNNs as ensemble members. Our study suggests that CNN is more robust to the high dimensionality compared to other existing learning algorithms.
• The DFS of BRDFS does not have any such effective strategy to deal with non-stationarity and high dimensionality. It just uses a single LSTM regression network to forecast the demand.

6. Conclusions

In this paper, we propose a novel resilient cyber-physical demand forecasting system (CDFS) called PRDFS that is resilient to false data injection attacks (FDIAs) since it can distinguish a natural change in the dynamics of demand over time, and a change due to the adversary’s attack. To achieve this capability, it uses a signature-based intrusion detection system (SIDS) of malicious nodes and normal nodes. SIDS is trained on labeled training data set using a supervised learning algorithm (GRNN). To automatically generate the training dataset for SIDS with all possible attack paths (defined in the spaces of assessed node trustworthiness and the skewness of residuals of forecasting models for individual sensor time series), an ethical hacking system (EHS) is run. EHS plays repeated attacker-defender games with the defense system (SIDS and TMS) of PRDFS. The training data of SIDS are continuously automatically generated by these games. This is the main novelty of PRDFS.

PRDFS uses a trust management system (TMS) to determine the most trustworthy measurement at time step $\mathit{t}$ from noisy sensor measurements for the aggregate demand time series. It considers the following information while determining the most trustworthy measurement at time step $\mathit{t}$: spatial data similarity, temporal data similarity, and the minimum expected trustworthiness of the corresponding sensor node at time step $\mathit{t}$. TMS employs GRNNs to predict the minimum expected trust score of each sensor based on the past trustworthiness scores of its measurements.

The demand forecasting system (DFS) of PRDFS uses the lagged values of aggregate demand time series to make out-of-sample forecasts. Our study suggests that among all existing approaches for dealing with the non-stationarity problem, the wavelet multiresolution decomposition approach is the most effective. Our DFS decomposes the time series into multiple component time series using wavelet transform. To avoid the curse of dimensionality, the proposed DFS uses a hierarchical ensemble of convolutional neural networks (CNNs) where each ensemble member is trained with a relatively small number of non-seasonal lag values of the component time series. Each ensemble member is trained sequentially. The first ensemble member is trained on the component time series derived from the original demand time series. Each of the remaining members is trained on the component time series derived from the residual series of the previous ensemble member. The number of ensemble members depends on how many members are required to make the final residual series of the ensemble a white noise series. This is another novelty of the proposed PRDFS.

For comparison purposes, we implement a hybrid resilient demand forecasting system that incorporates the best modules of existing demand forecasting systems. We call this resilient demand forecasting system BRDFS. We compare the performance of PRDFS and BRDFS against stealthy FDIAs. The data streams of all sensors are generated synthetically. Both were tested in 11 simulation setups with different percentages of malicious nodes. The performance of our algorithm is highly encouraging. It detects all malicious nodes within the first 30-time steps and it has low false positive rates (FPRs), never exceeding 9% regardless of the percentage of malicious nodes. Its demand forecasting accuracy never drops below 80% in the face of attacks.