Hidden Markov Mixture of Gaussian Process Functional Regression: Utilizing Multi-Scale Structure for Time Series Forecasting

Abstract

The mixture of Gaussian process functional regressions (GPFRs) assumes that there is a batch of time series or sample curves that are generated by independent random processes with different temporal structures. However, in real situations, these structures are actually transferred in a random manner from a long time scale. Therefore, the assumption of independent curves is not true in practice. In order to get rid of this limitation, we propose the hidden-Markov-based GPFR mixture model (HM-GPFR) by describing these curves with both fine- and coarse-level temporal structures. Specifically, the temporal structure is described by the Gaussian process model at the fine level and the hidden Markov process at the coarse level. The whole model can be regarded as a random process with state switching dynamics. To further enhance the robustness of the model, we also give a priori parameters to the model and develop a Bayesian-hidden-Markov-based GPFR mixture model (BHM-GPFR). The experimental results demonstrated that the proposed methods have both high prediction accuracy and good interpretability.

1. Introduction

The time series considered in this paper has a multi-scale structure: the coarse level and the fine level. We have observations $({\mathit{y}}_1,\dots ,{\mathit{y}}_T)$, where each ${\mathit{y}}_t=(y_{t,1},\dots ,y_{t,L})$ itself is a time series of length L. The whole time series is arranged as

$$y_{1,1},y_{1,2},\dots ,y_{1,L},y_{2,1},y_{2,2},\dots ,y_{2,L},\dots ,y_{T,1},y_{T,2},\dots ,y_{T,L}.$$

(1)

The subscripts ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$ are called coarse-level indices, while the subscripts ${\left\{y_{t,i}\right\}}_{i=1}^L$ are called fine-level indices. Throughout this paper, we took the electricity load dataset as a concrete example, as illustrated in Figure 1. The electricity load dataset consists of $T=365$ consecutive daily records, and in each day, there are $L=96$ samples recorded every quarter-hour. In this example, the coarse-level indices denote “day”, while the fine-level indices correspond to the time resolution of 15 min. The aim was to forecast both short-term and long-term electricity loads based on historical records. There may be partial observations $y_{T+1,1},\dots ,y_{T+1,M}$ with $M<L$, so the entire observed time series has the form:

$$y_{1,1},y_{1,2},\dots ,y_{1,L},y_{2,1},y_{2,2},\dots ,y_{2,L},\dots ,y_{T,1},y_{T,2},\dots ,y_{T,L},y_{T+1,1},\dots ,y_{T+1,M}.$$

(2)

The task is to predict future response $y_{t_{*},i_{*}}$, where $t_{*}\ge T+1,1\le i_{*}\le L$ are positive integers.

The coarse level and fine level provide different structural information about the data generation process. In the coarse level, each ${\mathit{y}}_t$ can be regarded as a time series, and there is a certain cluster structure [1,2] underlying these time series ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$: we can divide ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$ into groups such that the time series within each group share a similar evolving trend. Back to the electricity load dataset, such groups correspond to different electricity consumption patterns. We use $z_t$ to denote the cluster label of ${\mathit{y}}_t$. In the fine level, observations ${\left\{y_{t,i}\right\}}_{i=1}^L$ can be regarded as a realization of a stochastic process, and the properties of the stochastic process are determined by the cluster label $z_t$.

The mixture of Gaussian processes functional regression (mix-GPFR) model [1,3] is powerful for analyzing functional data or batch data, and it is applicable to the multi-scale time series forecasting task. Mix-GPFR assumes there are K Gaussian process functional regression (GPFR) [4] components, and associated with each ${\mathit{y}}_t$, there is a latent variable $z_t$ indicating which ${\mathit{y}}_t$ is generated by which GPFR component. Since GPFR is good at capturing temporal dependency, this model successfully utilizes the structure information in the fine level. However, the temporal information in the coarse level is totally ignored since mix-GPFR assumes ${\left\{z_t\right\}}_{t=1}^T$ are i.i.d.

In this work, we propose to model the temporal dependency in the coarse level by the hidden Markov model, which characterizes the switching dynamics of $z_1,\dots ,z_T$ by the transition probability matrix. We refer to the proposed model as HM-GPFR. Mix-GPFR is able to effectively predict $y_{T+1,M+1},\dots ,y_{T+1,L}$ when $M>0$. To predict the responses $y_{T+1,i_{*}}$, we must determine the cluster label $z_{T+1}$ based on observations $y_{T+1,1},\dots ,y_{T+1,M}$; otherwise, we do not know which ${\mathit{y}}_{T+1}$ is governed by which evolving pattern. If there is no observation at day $T+1$ (i.e., $M=0$), then mix-GPFR fails to identify the stochastic process that generates ${\mathit{y}}_{T+1}$. For the same reason, mix-GPFR is not suitable for long-term forecasting ($t_{*}>T+1$). On the other hand, HM-GPFR is able to infer $z_{t_{*}}$ for any $t_{*}$ based on the transition probabilities of the hidden Markov model even for $M=0$. Therefore, HM-GPFR makes use of coarse-level temporal information and solves the cold start problem in mix-GPFR. Besides, when a new day’s records ${\mathit{y}}_{T+1}$ have been fully observed, one needs to re-train a mix-GPFR model to utilize ${\mathit{y}}_{T+1}$, while HM-GPFR can adjust the parameters incrementally without retraining the model.

2. Related Works

The Gaussian process [5] is a powerful non-parametric Bayesian model. In [6,7,8], The GP was applied for time series forecasting. Shi et al.proposed the GPFR model to process batch data [4]. To effectively model multi-modal data, the mixture structure was further introduced to GPFR, and the mix-GPFR model was proposed [1,3]. In [2,9,10], GP-related methods for electricity load prediction were evaluated thoroughly. However, in these works, daily records were treated as i.i.d. samples, and the temporal information in the coarse level was ignored.

Multi-scale time series were proposed in [11,12], and further developments in this direction have been achieved in recent years. The time series considered in this work is different from the multi-scale time series since, at the coarse level, there is no aggregated observation from the samples at the fine level. In this paper, we mainly emphasize the multi-scale structure of the time series.

3. Preliminaries

3.1. Hidden Markov Model

For a sequence of observations ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$, the hidden Markov model (HMM) [13,14] assumes there is a hidden state variable $z_t$ associated with ${\mathit{y}}_t$. The sequence of hidden states ${\left\{z_t\right\}}_{t=1}^T$ forms a homogeneous Markov process. Usually, ${\left\{z_t\right\}}_{t=1}^T$ are categorical variables taking values in $\{1,\dots ,K\}$, and the transition dynamics is governed by $\mathbb{P}(z_t=l|z_{t-1}=k)=P_{kl}$. There are K groups of parameters ${\left\{{\mathit{\theta }}_k\right\}}_{k=1}^K$, and $z_t=k$ indicates that the observation ${\mathit{y}}_t$ is generated by $\mathbb{P}(\mathit{y};{\mathit{\theta }}_k)$. The goal of learning is to identify the parameters and infer the posterior distribution of hidden states ${\left\{z_t\right\}}_{t=1}^T$. Usually, the Baum–Welch algorithm [15,16] is utilized to learn the HMM, which can be regarded as a specifically designed EM algorithm based on the forward–backward algorithm. Once the model has been trained, we are able to simulate the future behavior of the system.

3.2. Gaussian Process Functional Regressions

The Gaussian process is a stochastic process for which any finite-dimensional distribution of samples is a multivariate Gaussian distribution. The property of a Gaussian process is determined by the mean function and the covariance function. We write the mean function as $\mu (·)$ and the covariance function as $c(·,·)$. Suppose that we have a dataset $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^L$. The relationship between the input and output is connected by a function $\mathcal{Y}$, i.e., $\mathcal{Y}\left(x_i\right)=y_i$. Let $\mathit{x}={[x_1,x_2,\dots ,x_L]}^{\mathrm{T}},\mathit{y}={[y_1,y_2,\dots ,y_L]}^{\mathrm{T}}$, then we assume $\mathit{y}|\mathit{x}\sim \mathcal{N}(\mathit{\mu },\mathbf{C})$, where $\mathit{\mu }={[\mu \left(x_1\right),\mu \left(x_2\right),\dots ,\mu \left(x_L\right)]}^{\mathrm{T}}$ and ${\mathbf{C}}_{ij}=c(x_i,x_j)$. In machine learning, the mean function and the covariance function are usually parameterized. Here, we used the squared exponential covariance function [2,3,5] $c(x_i,x_j;\mathit{\theta })={\theta }_1^{2}exp\left(-{\theta }_2^2\frac{{(x_i-x_j)}^2}{2}\right)+{\theta }_3^2{\delta }_{ij}$, where ${\delta }_{ij}$ is the Kronecker delta function and $\mathit{\theta }=[{\theta }_1,{\theta }_2,{\theta }_3]$. The mean function is modeled as a linear combination of B-spline basis functions [3,4]. Suppose that we have D B-spline basis functions ${\left\{{\varphi }_d\left(x\right)\right\}}_{d=1}^D$. Let $\mu \left(x\right)={\sum }_{d=1}^D{b}_d{\varphi }_d\left(x\right)$ and $\mathsf{\Phi }$ be an $L\times D$ matrix with ${\mathsf{\Phi }}_{id}={\varphi }_d\left(x_i\right),\mathit{b}={[b_1,b_2,\dots ,b_D]}^{\mathrm{T}}$, then $\mathit{y}|\mathit{x}\sim \mathcal{N}(\mathsf{\Phi }\mathit{b},\mathbf{C})$. From the function perspective, this model can be denoted as $\mathcal{Y}\left(x\right)\sim \mathcal{GPFR}(x;\mathit{b},\mathit{\theta })$.

We can use the Gaussian process to model the multi-scale time series considered in this paper, and the key point is transform the multi-scale time series into a batch dataset. For each coarse-level index t, we can construct a dataset ${\mathcal{D}}_t={\left\{(x_{t,i},y_{t,i})\right\}}_{i=1}^L$, where $x_{t,i}$ is the sampling time of the i-th sample in time series ${\mathit{y}}_t$. Let ${\mathcal{Y}}_t$ be the function underlying dataset $\mathcal{D}$, i.e., ${\mathcal{Y}}_t\left(x_{t,i}\right)=y_{t,i}$, then these ${\left\{{\mathcal{D}}_t\right\}}_{t=1}^T$ can be regarded as independent realizations of a GPFR, which assumes ${\mathcal{Y}}_t\left(x\right)\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathcal{GPFR}(x;\mathit{b},\mathit{\theta })$. Without loss of generality, we may assume $x_{t,i}=i$, and thus, ${\mathsf{\Phi }}_{id}={\varphi }_d\left(i\right),{\mathbf{C}}_{ij}=c(i,j;\mathit{\theta })$ do not depend on the coarse-level index t. Therefore, it is equivalent to assume ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$ are independently and identically distributed as $\mathcal{N}(\mathsf{\Phi }\mathit{b},\mathbf{C})$. To learn the parameters $\mathit{b}$ and $\mathit{\theta }$, we applied the Type-II maximum likelihood estimation technique [3,5].

As for prediction, given a new record ${\left\{(x_{t_{*},i},y_{t_{*},i})\right\}}_{i=1}^M$ and that we want to predict the corresponding output $y_{t_{*},i_{*}}$ at $x_{t_{*},i_{*}}$, where $M<i_{*}\le L$, from the definition of the Gaussian process, we immediately know that $y_{t_{*},i_{*}}$ also obeys a Gaussian distribution [5]. Let

$$\begin{array}{c}{\mathit{x}}_{*}={[x_{t_{*},1},\dots ,x_{t_{*},M}]}^{\mathrm{T}},{\mathit{y}}_{*}={[y_{t_{*},1},\dots ,y_{t_{*},M}]}^{\mathrm{T}},\end{array}$$

(3)

$$\begin{array}{c}{\mathit{\mu }}_{*}={[\mu \left(x_{t_{*},1}\right),\dots ,\mu \left(x_{t_{*},M}\right)]}^{\mathrm{T}},{\left[{\mathbf{C}}_{*}\right]}_{ij}=c(x_{t_{*},i},x_{t_{*},j}),\end{array}$$

(4)

then the mean of $y_{t_{*},i_{*}}$ is $\mu \left(x_{t_{*},i_{*}}\right)+\mathbf{c}(x_{t_{*},i_{*}},{\mathit{x}}_{*}){\mathbf{C}}_{*}^{-1}({\mathit{y}}_{*}-{\mathit{\mu }}_{*})$ and the variance of $y_{t_{*},i_{*}}$ is $c(x_{t_{*},i_{*}},x_{t_{*},i_{*}})-\mathbf{c}(x_{t_{*},i_{*}},{\mathit{x}}_{*}){\mathbf{C}}_{*}^{-1}\mathbf{c}({\mathit{x}}_{*},x_{t_{*},i_{*}})$. Note that, if $M=0$, the prediction is simply given by $\mathcal{N}(\mu \left(x_{t_{*},i_{*}}\right),c(x_{t_{*},i_{*}},x_{t_{*},i_{*}}))$, which equals the prior distribution of $y_{t_{*},i_{*}}$ and fails to utilize the temporal dependency with recent observations. In the electricity load prediction example, this means we can only effectively predict a new day’s electricity loads when we already have the first few observations of this day. In practice, however, it is very common to predict a new day’s electricity loads from scratch.

3.3. The Mixture of Gaussian Process Functional Regressions

GPFR implicitly assumes that all ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$ are generated by the same stochastic process, which is not the case in practice. In real applications, they may be generated from different signal sources; thus, a single GPFR is not flexible enough to model all the time series, especially when there are a variety of evolving trends. Take the electricity load dataset for example: the records corresponding to winter and summer are very likely to have significantly different trends and shapes. To solve this problem, Shi et al. [1] suggested introducing the mixture structure to GPFR and proposed the mixture of Gaussian process functional regressions (mix-GPFR). In mix-GPFR, there are K GPFR components with different parameters ${\{{\mathit{b}}_k,{\mathit{\theta }}_k\}}_{k=1}^K$, and the mixing proportion of the k-th GPFR component is ${\pi }_k$. Intuitively, there are K different signal sources or evolving patterns in mix-GPFR to describe temporal data with different temporal properties. For each ${\mathit{y}}_t$, there is an associated latent indicator variable $z_t\in \{1,2,\dots ,K\}$, and $z_t=k$ indicates that ${\mathit{y}}_t$ is generated by the k-th GPFR component. The generation process of mix-GPFR is

$$\begin{array}{c}{z}_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Categorical}({\pi }_1,{\pi }_2,\dots ,{\pi }_K),\\ {\mathcal{Y}}_t\left(x\right)|z_t=k\sim \mathcal{GPFR}(x;{\mathit{b}}_k,{\mathit{\theta }}_k).\end{array}$$

(5)

Let ${\mathbf{C}}_k\in L\times L$ be the covariance matrix calculated by ${\mathit{\theta }}_k$, i.e., ${\left[{\mathbf{C}}_k\right]}_{ij}=c(i,j;{\mathit{\theta }}_k)$, then the above equation is equivalent to ${\mathit{y}}_t\sim \mathcal{N}(\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k)$.

Due to the existence of latent variables, the parameter learning of mix-GPFR involves the EM algorithm [1,17]. As for prediction, K GPFR components of mix-GPFR first make predictions individually, then we weight these predictions based on the posterior probability $\mathbb{P}(z_{t_{*}}=k|{\mathit{y}}_{t_{*}};{\mathit{b}}_k,{\mathit{\theta }}_k)$. Note that if $M=0$, then $\mathbb{P}(z_{t_{*}}=k|{\mathit{y}}_{t_{*}};{\mathit{b}}_k,{\mathit{\theta }}_k)={\pi }_k$, which equals the mixing proportions and also fails to utilize recent observations. Therefore, mix-GPFR also suffers from the cold start problem.

4. Hidden-Markov-Based Gaussian Process Functional Regression Mixture Model

4.1. Model Specification

Similar to mix-GPFR, the hidden-Markov-based Gaussian process functional regression mixture model also assumes the time series is generated by K signal sources. The key difference is that the signal source may switch between consecutive observations in the time resolution of the coarse level. The temporal structure in the coarse level is characterized by the transition dynamics of ${\left\{z_t\right\}}_{t=1}^T$, and the temporal dependency in the fine level is captured by Gaussian processes. Precisely,

$$\begin{array}{c}{z}_1\sim \mathrm{Categorical}({\pi }_1,{\pi }_2,\dots ,{\pi }_K),\\ \mathbb{P}(z_t=l|z_{t-1}=k)=P_{kl},t=2,3,\dots ,T\\ {\mathcal{Y}}_t\left(x\right)|z_t=k\sim \mathcal{GPFR}(x;{\mathit{b}}_k,{\mathit{\theta }}_k),t=1,2,\dots ,T.\end{array}$$

(6)

Here, $\mathit{\pi }=[{\pi }_1,{\pi }_2,\dots ,{\pi }_K]$ is the initial state distribution, and $\mathbf{P}={\left[P_{kl}\right]}_{K\times K}$ is the transition probability matrix. We refer to this model as HM-GPFR, and the probabilistic graphical model is shown in Figure 2. In GPFR and mix-GPFR, the observations ${\left\{{\mathit{y}}_t\right\}}_{t=1}^T$ are modeled as independent and exchangeable realizations of stochastic processes; thus, the temporal structure in the coarse level is destroyed. However, in HM-GPFR, consecutive ${\mathit{y}}_{t-1},{\mathit{y}}_t$ are connected by the transition dynamics of their corresponding latent variables $z_{t-1},z_t$, which is more suitable for time series data. For example, if today’s electricity loads are very high, then it is unlikely that tomorrow’s electricity loads are extremely low.

4.2. Learning Algorithm

Due to the existence of latent variables ${\left\{z_t\right\}}_{t=1}^T$, we applied the EM algorithm to learn the HM-GPFR model. We write $\mathcal{T}={\left\{{\mathcal{D}}_t\right\}}_{t=1}^T$ to denote observations, $\mathsf{\Theta }={\left\{P_{kl}\right\}}_{k,l=1}^K\cup {\{{\pi }_k,{\mathit{b}}_k,{\mathit{\theta }}_k\}}_{k=1}^K$ to denote all parameters, and $\mathsf{\Omega }={\left\{z_t\right\}}_{t=1}^T$ to denote all latent variables. First, the complete data log-likelihood is

$$\begin{array}{cc}\hfill \mathcal{L}(\mathsf{\Theta };\mathcal{T},\mathsf{\Omega })=& \sum _{k=1}^K\mathbb{I}(z_1=k)log{\pi }_k+\sum _{t=1}^{T-1}\sum _{k=1}^K\sum _{l=1}^K\mathbb{I}(z_{t+1}=l,z_t=k)log{P}_{kl}\hfill \\ \hfill & +\sum _{t=1}^T\sum _{k=1}^K\mathbb{I}(z_t=k)log\mathbb{P}({\mathit{y}}_t;{\mathit{b}}_k,{\mathit{\theta }}_k).\hfill \end{array}$$

(7)

In the E-step of the EM algorithm, we need to calculate the expectation of Equation (7) with respect to the posterior distribution of latent variables $\mathsf{\Omega }$ to obtain the Q-function. However, it is not necessary to explicitly calculate $\mathbb{P}\left(\mathsf{\Omega }\right|\mathcal{T};\mathsf{\Theta })$, which is a categorical distribution with $K^N$ possible values, and it suffices to obtain $\mathbb{P}(z_{t+1}=l,z_t=k|\mathcal{T};\mathsf{\Theta })$ and $\mathbb{P}(z_t=k|\mathcal{T};\mathsf{\Theta })$. We first introduce some notations as follows:

$$\begin{array}{c}{\alpha }_t\left(k\right)=\mathbb{P}({\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_t,z_t=k;\mathsf{\Theta }),\\ {\beta }_t\left(k\right)=\mathbb{P}({\mathit{y}}_{t+1},{\mathit{y}}_{t+2},\dots ,{\mathit{y}}_T|z_t=k;\mathsf{\Theta }),\\ {\gamma }_t\left(k\right)=\mathbb{P}(z_t=k|\mathcal{T};\mathsf{\Theta }),\\ {\xi }_t(k,l)=\mathbb{P}(z_t=k,z_{t+1}=l|\mathcal{T};\mathsf{\Theta }).\end{array}$$

(8)

The key point is to calculate ${\gamma }_t\left(k\right)$ and ${\xi }_t(k,l)$. Note that

$$\begin{array}{cc}\hfill {\gamma }_t\left(k\right)& =\mathbb{P}(z_t=k|\mathcal{T};\mathsf{\Theta })\propto \mathbb{P}(z_t=k,\mathcal{T};\mathsf{\Theta })={\alpha }_t\left(k\right){\beta }_t\left(k\right),\hfill \\ \hfill {\xi }_t(k,l)& ={\alpha }_t\left(k\right)P_{kl}\mathcal{N}({\mathit{y}}_{t+1};\mathsf{\Phi }{\mathit{b}}_l,{\mathbf{C}}_l){\beta }_{t+1}\left(l\right).\hfill \end{array}$$

(9)

Therefore, the problem boils down to calculating ${\alpha }_t\left(k\right)$ and ${\beta }_t\left(k\right)$. We can derive them recursively based on the forward–backward algorithm [16,18]. According to the definition of ${\alpha }_t\left(k\right)$, we have

$${\alpha }_1\left(k\right)={\pi }_k\mathcal{N}({\mathit{y}}_1;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k),{\alpha }_t\left(k\right)=\left(\sum _{l=1}^K{\alpha }_{t-1}\left(l\right)P_{lk}\right)\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k).$$

(10)

Similarly, according to the definition of ${\beta }_t\left(k\right)$, we have

$${\beta }_T\left(k\right)=1,{\beta }_t\left(k\right)=\sum _{l=1}^K{P}_{kl}\mathcal{N}({\mathit{y}}_{t+1};\mathsf{\Phi }{\mathit{b}}_l,{\mathbf{C}}_l){\beta }_{t+1}\left(l\right).$$

(11)

To summary, in the E-step, we first use Equations (10) and (11) to calculate ${\alpha }_t\left(k\right),{\beta }_t\left(k\right)$ recursively based on the current parameters, then calculate ${\gamma }_t\left(k\right),{\xi }_t(k,l)$ according to Equation (9). The Q-function is given by

$$\mathcal{Q}\left(\mathsf{\Theta }\right)=\sum _{k=1}^K{\gamma }_1\left(k\right)log{\pi }_k+\sum _{t=1}^{T-1}\sum _{k=1}^K\sum _{l=1}^K{\xi }_t(k,l)log{P}_{kl}+\sum _{t=1}^T\sum _{k=1}^K{\gamma }_t\left(k\right)log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k).$$

(12)

In the M-step, we need to maximize $\mathcal{Q}$ with respect to the parameters. The parameters ${\left\{{\pi }_k\right\}}_{k=1}^K$ and ${\left\{P_{kl}\right\}}_{k,l=1}^K$ can be optimized in closed-form:

$${\pi }_k=\frac{{\gamma }_1\left(k\right)}{{\sum }_{l=1}^K{\gamma }_1\left(l\right)},P_{kl}=\frac{{\sum }_{t=1}^{T-1}{\xi }_t(k,l)}{{\sum }_{t=1}^{T-1}{\sum }_{m=1}^K{\xi }_t(k,m)}.$$

(13)

The parameters ${\{{\mathit{b}}_k,{\mathit{\theta }}_k\}}_{k=1}^K$ cannot be solved in closed-form, and we applied the gradient ascent algorithm to optimize $\mathcal{Q}\left(\mathsf{\Theta }\right)$ with gradients

$$\begin{array}{c}\frac{\partial \mathcal{Q}\left(\mathsf{\Theta }\right)}{\partial {\mathit{\theta }}_k}=\frac{1}{2}{\displaystyle \sum _{t=1}^T}{\gamma }_t\left(k\right)\mathrm{tr}\left(\left({\mathbf{C}}_k^{-1}({\mathit{y}}_t-\mathsf{\Phi }{\mathit{b}}_k){({\mathit{y}}_t-\mathsf{\Phi }{\mathit{b}}_k)}^{\mathrm{T}}{\mathbf{C}}_k^{-1}-{\mathbf{C}}_k^{-1}\right)\frac{\partial {\mathbf{C}}_k}{\partial {\mathit{\theta }}_k}\right),\\ \frac{\partial \mathcal{Q}\left(\mathsf{\Theta }\right)}{\partial {\mathit{b}}_k}={\displaystyle \sum _{t=1}^T}{\gamma }_t\left(k\right){\mathsf{\Phi }}^{\mathrm{T}}{\mathbf{C}}_k^{-1}({\mathit{y}}_t-\mathsf{\Phi }{\mathit{b}}_k).\end{array}$$

(14)

The complete algorithm is summarized in Algorithm 1. When the partial observations $y_{T+1,1},\dots ,y_{T+1,M}$ become complete as we collect more data, we can adjust the parameters incrementally without retraining the model. This is achieved by continuing EM iterations with current parameters until the iteration converges again.

4.3. Prediction Strategy

After the parameters have been learned, we assign the latent variable ${\widehat{z}}_t=$${argmax}_{k=1,\dots ,K}{\gamma }_t\left(k\right)$ and regard ${\left\{{\widehat{z}}_t\right\}}_{t=1}^T$ as deterministic. For the prediction, we considered two cases: $t_{*}=T+1$ and $t_{*}>T+1$. When $t_{*}=T+1$, the latent variable $z_{T+1}$ is determined by both the conditional transition probability $z_{T+1}|{\widehat{z}}_T$ and partial observations ${\mathit{y}}_{T+1}$. More precisely, suppose ${\widehat{z}}_T=l$, then

$${\omega }_k=\mathbb{P}(z_{T+1}=k|\mathcal{T},{\mathit{y}}_{T+1},{\widehat{z}}_T=l;\widehat{\mathsf{\Theta }})\propto {\widehat{P}}_{lk}\mathcal{N}({\mathit{y}}_{T+1};\mathsf{\Phi }[1:M,:]{\widehat{\mathit{b}}}_k,\mathbf{C}[1:M,1:M]),$$

(15)

where the square brackets denote slicing operation. If $M=0$, then ${\omega }_k={\widehat{P}}_{lk}$ is determined by the last hidden state and transition dynamics, which is more accurate than mix-GPFR. Suppose the prediction of the k-th component is $y_{*}^{\left(k\right)}$, then the final prediction is given by ${\sum }_{k=1}^K{\omega }_k{y}_{*}^{\left(k\right)}$.

We next considered the case $t_{*}>T+1$, the main difference being the posterior distribution of $z_{t_{*}}$. In this case, we need to use the transition probability matrix recursively. First, we calculated the distribution of $z_{T+1}$ according to Equation (15). Then, by the Markov property, we know

$${\omega }_k=\mathbb{P}(z_{t_{*}}=k|\mathcal{T},{\mathit{y}}_{T+1},z_T=l;\widehat{\mathsf{\Theta }})\propto \sum _{m=1}^K\mathbb{P}(z_{T+1}=m|\mathcal{T},{\mathit{y}}_{T+1},{\widehat{z}}_T=l;\mathsf{\Theta }){\left[{\widehat{\mathbf{P}}}^{t_{*}-T-1}\right]}_{mk}.$$

(16)

The final prediction is also given by ${\sum }_{k=1}^K{\omega }_k{y}_{*}^{\left(k\right)}={\sum }_{k=1}^K{\omega }_k\mathsf{\Phi }[i_{*},:]{\mathit{b}}_k$.

Algorithm 1: The EM algorithm for learning HM-GPFR.

5. Bayesian-Hidden-Markov-Based Gaussian Process Functional Regression Mixture Model

5.1. Model Specification

One drawback of HM-GPFR is that there are too many parameters, and thus, it has the risk of overfitting. In this section, we further developed a fully Bayesian treatment of HM-GPFR. We placed a Gaussian prior $\mathcal{N}({\mathbf{m}}_b,{\mathsf{\Sigma }}_b)$ on the coefficients of B-spline functions ${\left\{{\mathit{b}}_k\right\}}_{k=1}^K$. For the transition probabilities, let ${\mathit{p}}_k={[P_{k1},P_{k2},\dots ,P_{kK}]}^{\mathrm{T}}$ be the probabilities from state k to other states, then we assumed ${\mathit{p}}_k$ obeys a Dirichlet prior $\mathrm{Dir}(a_0,\dots ,a_0)$. The generation process of Bayesian HM-GPFR is

$$\begin{array}{c}{\mathit{b}}_k\sim \mathcal{N}({\mathbf{m}}_b,{\mathsf{\Sigma }}_b),k=1,2,\dots ,K\\ {\mathit{p}}_k\sim \mathrm{Dir}(a_0,\dots ,a_0),k=1,2,\dots ,K\\ z_1\sim \mathrm{Categorical}({\pi }_1,{\pi }_2,\dots ,{\pi }_K),\\ \mathbb{P}(z_t=l|z_{t-1}=k)=P_{kl},t=2,3,\dots ,T\\ {\mathcal{Y}}_t\left(x\right)|z_t=k\sim \mathcal{GPFR}(x;{\mathit{b}}_k,{\mathit{\theta }}_k),t=1,2,\dots ,T.\end{array}$$

(17)

The probabilistic graphical model of Bayesian HM-GPFR is shown in Figure 2.

5.2. Learning Algorithm

We still used the EM algorithm to learn the parameters of the BHM-GPFR model. However, this case is more complicated since there are more latent variables. The complete data log-likelihood is

$$\begin{array}{cc}\hfill \mathcal{L}(\mathsf{\Theta };\mathcal{T},\mathsf{\Omega })=& \sum _{k=1}^{K}log\mathcal{N}({\mathit{b}}_k;{\mathbf{m}}_b,{\mathsf{\Sigma }}_b)+\sum _{k=1}^K\sum _{l=1}^K(a_0-1)log{P}_{kl}\hfill \\ \hfill & +\sum _{k=1}^K\mathbb{I}(z_1=k)log{\mathit{\pi }}_k+\sum _{t=1}^{T-1}\sum _{k=1}^K\sum _{l=1}^K\mathbb{I}(z_{t+1}=l,z_t=k)log{P}_{kl}\hfill \\ \hfill & +\sum _{t=1}^T\sum _{k=1}^K\mathbb{I}(z_t=k)log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k).\hfill \end{array}$$

(18)

Compared with Equation (7), the first two terms are extra due to the prior distributions. In the E-step of the EM algorithm, we need to take the expectation of Equation (18) with respect to the posterior distribution of the latent variables. However, the posterior of $\mathsf{\Omega }$ is intractable since ${\left\{{\mathit{b}}_k\right\}}_{k=1}^K$, ${\left\{{\mathit{p}}_k\right\}}_{k=1}^K$, and ${\left\{z_t\right\}}_{t=1}^T$ are correlated. We used the variational inference method and tried to find an optimal approximation of $\mathbb{P}\left(\mathsf{\Omega }\right|\mathcal{T};\mathsf{\Theta })$ with a simple form. We adopted the mean-field family approximation, which factorizes the joint distribution of $\mathsf{\Omega }$ to a product of several independent distributions:

$$\mathbb{Q}\left(\mathsf{\Omega }\right)=\prod _{k=1}^K\mathbb{Q}\left({\mathit{b}}_k\right)\prod _{k=1}^K\mathbb{Q}\left({\mathit{p}}_k\right)\mathbb{Q}\left(\mathit{z}\right).$$

(19)

Similar to the HM-GPFR case, $\mathbb{Q}\left(\mathit{z}\right)$ is a categorical distribution with $K^T$ possible values, but we do not need to calculate $\mathbb{Q}\left(\mathit{z}\right)$ explicitly and only need to calculate ${\gamma }_t\left(k\right)=\mathbb{Q}(z_t=k)$ and ${\xi }_t(k,l)=\mathbb{Q}(z_{t+1}=l,z_t=k)$. According to the variational inference theory, we iterated $\mathbb{Q}\left({\mathit{b}}_k\right)$, $\mathbb{Q}\left({\mathit{p}}_k\right)$, and $\mathbb{Q}\left(\mathit{z}\right)$ alternately until convergence.

For $\mathbb{Q}\left({\mathit{b}}_k\right)$,

$$\begin{array}{cc}\hfill \mathbb{Q}\left({\mathit{b}}_k\right)\propto & exp{\mathbb{E}}_{{\prod }_{k=1}^K\mathbb{Q}\left({\mathit{p}}_k\right)\mathbb{Q}\left(\mathit{z}\right)}\left[\mathcal{L}(\mathsf{\Theta };\mathcal{T},\mathsf{\Omega })\right]\hfill \\ \hfill =& exp{\mathbb{E}}_{\mathbb{Q}\left(\mathit{z}\right)}\left[log\mathcal{N}({\mathit{b}}_k;{\mathbf{m}}_b,{\mathsf{\Sigma }}_b)+\sum _{t=1}^T\mathbb{I}(z_t=k)log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k)\right]\hfill \\ \hfill \propto & exp\left(-\frac{1}{2}log\left|{\mathsf{\Sigma }}_b\right|-\frac{1}{2}{({\mathit{b}}_k-{\mathbf{m}}_b)}^{\mathrm{T}}{\mathsf{\Sigma }}_b^{-1}({\mathit{b}}_k-{\mathbf{m}}_b)\right.\hfill \\ \hfill & +\left.\sum _{t=1}^T{\gamma }_t\left(k\right)\left(-\frac{1}{2}log\left|{\mathbf{C}}_k\right|-\frac{1}{2}{({\mathit{y}}_t-\mathsf{\Phi }{\mathit{b}}_k)}^{\mathrm{T}}{\mathbf{C}}_k^{-1}({\mathit{y}}_t-\mathsf{\Phi }{\mathit{b}}_k)\right)\right)\hfill \end{array}$$

(20)

By completing the square, we obtained the approximate posterior of ${\mathit{b}}_k$ as $\mathcal{N}({\mathbf{m}}_k,{\mathsf{\Sigma }}_k)$ with

$${\mathsf{\Sigma }}_k={\left({\mathsf{\Sigma }}_b+\sum _{t=1}^T{\gamma }_t\left(k\right){\mathsf{\Phi }}^{\mathrm{T}}{\mathbf{C}}_k^{-1}\mathsf{\Phi }\right)}^{-1},{\mathbf{m}}_k={\mathsf{\Sigma }}_k\left({\mathsf{\Sigma }}_b^{-1}{\mathbf{m}}_b+\sum _{t=1}^T{\gamma }_t\left(k\right){\mathsf{\Phi }}^{\mathrm{T}}{\mathbf{C}}_k^{-1}{\mathit{y}}_t\right).$$

(21)

For $\mathbb{Q}\left({\mathit{p}}_k\right)$,

$$\begin{array}{cc}\hfill \mathbb{Q}\left({\mathit{p}}_k\right)& \propto exp{\mathbb{E}}_{{\prod }_{k=1}^K\mathbb{Q}\left({\mathit{b}}_k\right)\mathbb{Q}\left(\mathit{z}\right)}\left[\mathcal{L}(\mathsf{\Theta };\mathcal{T},\mathsf{\Omega })\right]\hfill \\ \hfill & =exp{\mathbb{E}}_{\mathbb{Q}\left(\mathit{z}\right)}\left[\sum _{l=1}^K(a_0-1)log{P}_{kl}+\sum _{t=1}^{T-1}\sum _{l=1}^K\mathbb{I}(z_{t+1}=l,z_t=k)log{P}_{kl}\right]\hfill \\ \hfill & =exp\left(\sum _{l=1}^K(a_0-1)log{P}_{kl}+\sum _{t=1}^{T-1}\sum _{l=1}^K{\xi }_t(k,l)log{P}_{kl}\right)\hfill \\ \hfill & =\prod _{l=1}^K{P}_{kl}^{a_0+{\sum }_{t=1}^{T-1}{\xi }_t(k,l)-1}\hfill \end{array}$$

(22)

Therefore, the approximate posterior of ${\mathit{p}}_k$ is $\mathrm{Dir}(a_{k1},\dots ,a_{kK})$ with $a_{kl}=a_0+{\sum }_{t=1}^{T-1}{\xi }_t(k,l)$.

For $\mathbb{Q}\left(\mathit{z}\right)$,

$$\begin{array}{cc}\hfill \mathbb{Q}\left(\mathit{z}\right)\propto & exp{\mathbb{E}}_{{\prod }_{k=1}^K\mathbb{Q}\left({\mathit{b}}_k\right){\prod }_{k=1}^K\mathbb{Q}\left({\mathit{p}}_k\right)}\left[\mathcal{L}(\mathsf{\Theta };\mathcal{T},\mathsf{\Omega })\right]\hfill \\ \hfill =& exp{\mathbb{E}}_{{\prod }_{k=1}^K\mathbb{Q}\left({\mathit{b}}_k\right){\prod }_{k=1}^K\mathbb{Q}\left({\mathit{p}}_k\right)}\left[\sum _{k=1}^K\mathbb{I}(z_1=k)log{\mathit{\pi }}_k+\sum _{t=1}^{T-1}\sum _{k=1}^K\sum _{l=1}^K\mathbb{I}(z_{t+1}=l,z_t=k)log{P}_{kl}\right.\hfill \\ \hfill & +\left.\sum _{t=1}^T\sum _{k=1}^K\mathbb{I}(z_t=k)log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k)\right]\hfill \\ \hfill =& exp\left(\sum _{k=1}^K\mathbb{I}(z_1=k)log{\mathit{\pi }}_k+\sum _{t=1}^{T-1}\sum _{k=1}^K\sum _{l=1}^K\mathbb{I}(z_{t+1}=l,z_t=k){\mathbb{E}}_{\mathbb{Q}\left({\mathit{p}}_k\right)}[log{P}_{kl}]\right.\hfill \\ \hfill & +\left.\sum _{t=1}^T\sum _{k=1}^K\mathbb{I}(z_t=k){\mathbb{E}}_{\mathbb{Q}\left({\mathit{b}}_k\right)}[log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k)]\right).\hfill \end{array}$$

(23)

Note that this equation has exactly the same form as Equation (7); thus, we can use the forward–backward algorithm to obtain ${\gamma }_t\left(k\right)$ and ${\xi }_t(k,l)$. To see this, let

$$\begin{array}{c}{\tilde{P}}_{kl}=exp{\mathbb{E}}_{\mathbb{Q}\left({\mathbf{p}}_k\right)}[log{P}_{kl}]=exp\left(\psi \left(a_{kl}\right)-\psi \left(\sum _{l=1}^K{a}_{kl}\right)\right),\\ \tilde{\mathbb{P}}({\mathit{y}}_t;{\mathbf{m}}_k,{\mathsf{\Sigma }}_k,{\mathit{\theta }}_k)=exp{\mathbb{E}}_{\mathbb{Q}\left({\mathit{b}}_k\right)}[log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k)]=\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathbf{m}}_k,{\mathbf{C}}_k)exp\left(-\frac{1}{2}\mathrm{tr}\left({\mathsf{\Sigma }}_k\mathsf{\Phi }{\mathbf{C}}_k^{-1}{\mathsf{\Phi }}^{\mathrm{T}}\right)\right),\end{array}$$

(24)

then Equation (23) can be rewritten as

$$\begin{array}{cc}\hfill log\mathbb{Q}\left(\mathit{z}\right)=& \sum _{k=1}^K\mathbb{I}(z_1=k)log{\mathit{\pi }}_k+\sum _{t=1}^{T-1}\sum _{k=1}^K\sum _{l=1}^K\mathbb{I}(z_{t+1}=l,z_t=k)log{\tilde{P}}_{kl}\hfill \\ \hfill & +\sum _{t=1}^T\sum _{k=1}^K\mathbb{I}(z_t=k)log\tilde{\mathbb{P}}({\mathit{y}}_t;{\mathbf{m}}_k,{\mathsf{\Sigma }}_k,{\mathit{\theta }}_k).\hfill \end{array}$$

(25)

To obtain ${\gamma }_t\left(k\right)$ and ${\xi }_t(k,l)$, we ran the Baum–Welch algorithm with sufficient statistics ${\pi }_k,{\tilde{P}}_{kl},\tilde{\mathbb{P}}({\mathit{y}}_t;{\mathbf{m}}_k,{\mathsf{\Sigma }}_k,\mathit{\theta })$.

Taking expectation of Equation (18) with respect to the approximate posterior $\mathbb{Q}\left(\mathsf{\Omega }\right)$, the Q function is

$$\begin{array}{cc}\hfill \mathcal{Q}\left(\mathsf{\Theta }\right)=& \sum _{k=1}^K{\mathbb{E}}_{\mathbb{Q}\left({\mathit{b}}_k\right)}[log\mathcal{N}({\mathit{b}}_k;{\mathbf{m}}_b,{\mathsf{\Sigma }}_b)]+\sum _{k=1}^K{\gamma }_1\left(k\right)log{\mathit{\pi }}_k+\sum _{t=1}^T\sum _{k=1}^K{\gamma }_t\left(k\right){\mathbb{E}}_{\mathbb{Q}\left({\mathit{b}}_k\right)}[log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathit{b}}_k,{\mathbf{C}}_k)]\hfill \\ \hfill =& \sum _{k=1}^K(log\mathcal{N}({\mathbf{m}}_k;{\mathbf{m}}_b,{\mathsf{\Sigma }}_b)-\frac{1}{2}\mathrm{tr}\left({\mathsf{\Sigma }}_k{\mathsf{\Sigma }}_b^{-1}\right))+\sum _{k=1}^K{\gamma }_1\left(k\right)log{\mathit{\pi }}_k\hfill \\ \hfill & +\sum _{t=1}^T\sum _{k=1}^K{\gamma }_t\left(k\right)(log\mathcal{N}({\mathit{y}}_t;\mathsf{\Phi }{\mathbf{m}}_k,{\mathbf{C}}_k)-\frac{1}{2}\mathrm{tr}\left({\mathsf{\Sigma }}_k\mathsf{\Phi }{\mathbf{C}}_i^{-1}{\mathsf{\Phi }}^{\mathrm{T}}\right))].\hfill \end{array}$$

(26)

Maximizing $\mathcal{Q}\left(\mathsf{\Theta }\right)$ with respect to ${\pi }_k$, ${\mathbf{m}}_b$, and ${\mathsf{\Sigma }}_b$, we obtain

$${\pi }_k=\frac{{\gamma }_1\left(k\right)}{{\sum }_{l=1}^K{\gamma }_1\left(l\right)},{\mathsf{\Sigma }}_b=\frac{1}{K}\sum _{k=1}^K\left({\mathsf{\Sigma }}_k+({\mathbf{m}}_k-{\mathbf{m}}_b){({\mathbf{m}}_k-{\mathbf{m}}_b)}^{\mathrm{T}}\right),{\mathbf{m}}_b=\frac{1}{K}\sum _{k=1}^K{\mathbf{m}}_k.$$

(27)

The parameters ${\left\{{\mathit{\theta }}_k\right\}}_{k=1}^K$ cannot be solved in closed-form, and we applied the gradient ascent algorithm to optimize $\mathcal{Q}\left(\mathsf{\Theta }\right)$. The gradient of $\mathcal{Q}\left(\mathsf{\Theta }\right)$ with respect to ${\mathit{\theta }}_k$ is

$$\frac{\partial \mathcal{Q}\left(\mathsf{\Theta }\right)}{\partial {\mathit{\theta }}_k}=\sum _{t=1}^T\frac{1}{2}{\gamma }_t\left(k\right)\mathrm{tr}\left({\mathbf{C}}_k^{-1}{\mathit{S}}_{t,k}{\mathbf{C}}_k^{-1}\frac{\partial {\mathbf{C}}_k}{\partial {\mathit{\theta }}_k}\right),{\mathit{S}}_{t,k}=({\mathit{y}}_t-\mathsf{\Phi }{\mathbf{m}}_k){({\mathit{y}}_t-\mathsf{\Phi }{\mathbf{m}}_k)}^{\mathrm{T}}+{\mathsf{\Phi }}^{\mathrm{T}}{\mathsf{\Sigma }}_k\mathsf{\Phi }-{\mathbf{C}}_k.$$

(28)

The complete algorithm is summarized in Algorithm 2.

We point out that the EM algorithm is not guaranteed to converge to local minima under mild conditions; thus, Algorithms 1 and 2 are aimed at practical applications rather than theoretical analysis. Besides, applying the forward–backward learning algorithm to the Bayesian HMM with unnormalized parameters ${\tilde{P}}_{kl},\tilde{\mathbb{P}}({\mathit{y}}_t;{\mathbf{m}}_k,{\mathsf{\Sigma }}_k,\mathit{\theta })$ was justified in [19].

5.3. Prediction Strategy

After learning, we set the latent variables to their maximum a posteriori (MAP) estimates $\widehat{\mathsf{\Omega }}$. Specifically, ${\widehat{\mathit{b}}}_k={\mathbf{m}}_k,{\widehat{P}}_{kl}=\frac{a_{kl}}{{\sum }_{m=1}^K{a}_{km}},{\widehat{z}}_t={argmax}_{k=1,2,\dots ,K}{\gamma }_t\left(k\right)$. The rest of the prediction was the same as HM-GPFR.

Algorithm 2: The variational EM algorithm for learning BHM-GPFR.

6. Experimental Results

6.1. Experiment Settings

In this section, we used the electricity load dataset issued by the State Grid of China for a city in northwest China. The dataset records electricity loads every 15 min; thus, there are 96 records per day. Using the electricity load records of 2010 for training, we predicted the subsequent S-step electricity loads in a time series prediction fashion, where $S=1,2,3,4,5,10,20,30,50,80,100,200,500,1000$. This setting allowed both short-term and long-term predictions to be evaluated. For a more comprehensive and accurate assessment of the performance, we rolled the time series by 100 rounds. Based on the electricity loads of 2010, the r-th round also puts the first $(r-1)$ records of 2011 into the training set. In each round, we predicted the subsequent S-step electricity loads. In the r-th round, suppose the ground-truths are $y_1,y_2,\dots ,y_S$ and the predictions are ${\widehat{y}}_1,\widehat{y_2},\dots ,{\widehat{y}}_S$; we used the mean absolute percentage errors (MAPEs) to evaluate the prediction results. Specifically, ${\mathrm{MAPE}}_r=\frac{1}{S}{\sum }_{s=1}^S\frac{|y_s-{\widehat{y}}_s|}{|y_s|}$. For the overall evaluation, we report the average of 100 MAPEs to obtain $\mathrm{MAPE}=\frac{1}{100}{\sum }_{r=1}^{100}{\mathrm{MAPE}}_r$. Since the algorithms are influenced by randomness, we repeated the algorithms for 10 runs and report the average results.

We implemented HM-GPFR and BHM-GPFR in MATLAB and compared them with other time series forecasting methods. Specifically:

• Classical time series forecasting methods: auto-regressive (AR), moving average (MA), auto-regressive moving average (ARMA), auto-regressive integrated moving average (ARIMA), seasonal auto-regressive moving average (SARMA).
• Machine learning methods: long short-term memory (LSTM), feedforward neural network (FNN), support vector regression (SVR), enhanced Gaussian process mixture model (EGPM).
• GPFR-related methods: the mixture of Gaussian process functional regressions (mix-GPFR), the mixture of Gaussian processes with nonparametric mean functions (mix-GPNM), Dirichlet-process-based mixture of Gaussian process functional regressions (DPM-GPFR).

For AR, MA, ARMA, ARIMA, and SARMA, we set the model order L in $\{4,8,12\}$. For SARMA, the seasonal length was set to be 96 since there are 96 records per day, which implicitly assumes that the overall time series exhibits periodicity in days. LSTM, NN, SVR, and EGPM transform the time series prediction problem into a regression problem, i.e., they use the latest L observations to predict the output at the next point and then use the regression method to train and predict. In the experiment, we set L in $\{4,12,24,48\}$. The neural network in the FNN has two hidden layers with 10 and 5 neurons, respectively. The kernel function in SVR is the Gaussian kernel, whose scale parameters were adaptively selected by cross-validation. The number of components for EGPM was set in $\{3,5,10\}$. In addition, we used the recursive method [20] for multi-step prediction. For the comparison algorithms implemented using the MATLAB toolbox (including AR, MA, ARMA, ARIMA, ARIMA, SARMA, LSTM, FNN, and SVR), the settings of the hyper-parameters were adaptively tuned by the toolboxes. For other comparison algorithms (EGPM, mix-GPFR, mix-GPNM), the parameters were set according to the original papers. For mix-GPFR, mix-GPNM, and DPM-GPFR, we first converted the time series data into curve datasets and then used these methods to make the predictions. The number of components K in mix-GPFR and mix-GPNM was set to 5, and the number of B-spline basis functions D in mix-GPFR and DPM-GPFR was set to 30. The setting of K and D will be further discussed in Section 6.6.

6.2. Performance Evaluation and Model Explanation

The prediction results of various methods on the electricity load dataset are shown in

Table 1. MAPE of various methods on the electricity loads dataset under different step lengths and parameter settings.

Method	Parameter	Step Length S
		1	2	3	4	5	10	20	30	50	80	100	200	300	500	1000
AR	$L=4$	$1.02\%$	$1.36\%$	$1.75\%$	$2.13\%$	$2.53\%$	$4.37\%$	$6.95\%$	$8.88\%$	$11.79\%$	$13.92\%$	$15.0\%$	$18.06\%$	$17.82\%$	$16.94\%$	$16.46\%$
	$L=8$	$1.01\%$	$1.36\%$	$1.75\%$	$2.14\%$	$2.55\%$	$4.47\%$	$7.0\%$	$8.7\%$	$11.28\%$	$13.36\%$	$14.5\%$	$17.8\%$	$17.64\%$	$16.83\%$	$16.4\%$
	$L=12$	$1.01\%$	$1.35\%$	$1.74\%$	$2.13\%$	$2.54\%$	$4.46\%$	$6.96\%$	$8.63\%$	$11.17\%$	$13.23\%$	$14.38\%$	$17.74\%$	$17.6\%$	$16.81\%$	$16.39\%$
MA	$L=4$	$3.39\%$	$5.74\%$	$8.03\%$	$9.94\%$	$11.35\%$	$14.08\%$	$15.13\%$	$15.2\%$	$15.66\%$	$16.39\%$	$16.97\%$	$19.05\%$	$18.48\%$	$17.33\%$	$16.65\%$
	$L=8$	$2.23\%$	$3.5\%$	$4.74\%$	$5.81\%$	$6.83\%$	$11.11\%$	$13.65\%$	$14.21\%$	$15.07\%$	$16.01\%$	$16.67\%$	$18.9\%$	$18.38\%$	$17.27\%$	$16.62\%$
	$L=12$	$1.83\%$	$2.76\%$	$3.66\%$	$4.46\%$	$5.21\%$	$8.61\%$	$12.32\%$	$13.33\%$	$14.54\%$	$15.68\%$	$16.41\%$	$18.77\%$	$18.29\%$	$17.22\%$	$16.59\%$
ARMA	$L=4$	$1.01\%$	$1.34\%$	$1.73\%$	$2.12\%$	$2.52\%$	$4.42\%$	$6.93\%$	$8.6\%$	$11.22\%$	$13.18\%$	$14.31\%$	$17.64\%$	$17.54\%$	$16.77\%$	$16.38\%$
	$L=8$	$1.01\%$	$1.34\%$	$1.72\%$	$2.09\%$	$2.48\%$	$4.34\%$	$6.87\%$	$8.52\%$	$11.12\%$	$13.05\%$	$14.13\%$	$17.5\%$	$17.44\%$	$16.71\%$	$16.34\%$
	$L=12$	$1.02\%$	$1.36\%$	$1.76\%$	$2.14\%$	$2.55\%$	$4.39\%$	$6.8\%$	$8.4\%$	$10.99\%$	$12.89\%$	$13.93\%$	$17.31\%$	$17.3\%$	$16.62\%$	$16.3\%$
ARIMA	$L=4$	$0.98\%$	$1.34\%$	$1.74\%$	$2.14\%$	$2.57\%$	$4.58\%$	$7.27\%$	$9.08\%$	$11.99\%$	$14.43\%$	$15.34\%$	$18.65\%$	$18.67\%$	$17.95\%$	$17.65\%$
	$L=8$	$1.01\%$	$1.36\%$	$1.75\%$	$2.14\%$	$2.57\%$	$4.56\%$	$7.24\%$	$9.2\%$	$12.48\%$	$14.53\%$	$15.09\%$	$18.67\%$	$18.79\%$	$18.2\%$	$18.37\%$
	$L=12$	$1.01\%$	$1.4\%$	$1.82\%$	$2.24\%$	$2.68\%$	$4.93\%$	$8.64\%$	$11.92\%$	$18.41\%$	$22.65\%$	$21.83\%$	$24.05\%$	$24.24\%$	$24.2\%$	$29.52\%$
SARMA	$L=4$	$0.83\%$	$1.08\%$	$1.33\%$	$1.55\%$	$1.76\%$	$2.66\%$	$4.06\%$	$5.15\%$	$6.38\%$	$7.57\%$	$8.67\%$	$10.69\%$	$9.96\%$	$7.62\%$	$7.62\%$
	$L=8$	$0.83\%$	$1.08\%$	$1.32\%$	$1.55\%$	$1.76\%$	$2.67\%$	$4.04\%$	$5.12\%$	$6.35\%$	$7.54\%$	$8.64\%$	$10.67\%$	$9.93\%$	$7.58\%$	$7.58\%$
	$L=12$	$0.82\%$	$1.07\%$	$1.3\%$	$1.52\%$	$1.72\%$	$2.62\%$	$4.06\%$	$5.16\%$	$6.31\%$	$7.17\%$	$8.11\%$	$10.55\%$	$10.09\%$	$7.86\%$	$7.83\%$
LSTM	$L=4$	$12.89\%$	$12.9\%$	$12.91\%$	$12.97\%$	$13.04\%$	$13.55\%$	$14.56\%$	$15.16\%$	$16.24\%$	$16.99\%$	$17.25\%$	$19.48\%$	$19.01\%$	$17.88\%$	$17.28\%$
	$L=12$	$12.39\%$	$12.32\%$	$12.32\%$	$12.35\%$	$12.39\%$	$12.78\%$	$13.9\%$	$14.83\%$	$16.38\%$	$17.38\%$	$17.42\%$	$19.77\%$	$19.39\%$	$18.27\%$	$17.73\%$
	$L=24$	$11.48\%$	$11.43\%$	$11.43\%$	$11.46\%$	$11.5\%$	$11.81\%$	$12.69\%$	$13.49\%$	$14.73\%$	$15.72\%$	$16.28\%$	$18.97\%$	$18.8\%$	$17.83\%$	$17.44\%$
	$L=48$	$10.1\%$	$10.11\%$	$10.11\%$	$10.15\%$	$10.2\%$	$10.49\%$	$11.22\%$	$11.96\%$	$12.94\%$	$13.09\%$	$13.56\%$	$16.53\%$	$17.52\%$	$17.98\%$	$18.57\%$
FNN	$L=4$	$0.96\%$	$1.29\%$	$1.64\%$	$1.94\%$	$2.27\%$	$3.99\%$	$6.21\%$	$8.13\%$	$11.56\%$	$14.4\%$	$15.49\%$	$18.71\%$	$18.71\%$	$17.87\%$	$17.61\%$
	$L=12$	$0.85\%$	$1.1\%$	$1.37\%$	$1.62\%$	$1.88\%$	$3.13\%$	$5.38\%$	$7.25\%$	$9.94\%$	$13.24\%$	$14.87\%$	$20.44\%$	$20.72\%$	$19.91\%$	$19.81\%$
	$L=24$	$0.85\%$	$1.07\%$	$1.27\%$	$1.43\%$	$1.6\%$	$2.39\%$	$3.94\%$	$5.38\%$	$7.43\%$	$10.1\%$	$11.54\%$	$14.57\%$	$15.27\%$	$15.62\%$	$17.87\%$
	$L=48$	$0.85\%$	$1.0\%$	$1.15\%$	$1.28\%$	$1.39\%$	$1.99\%$	$3.21\%$	$4.12\%$	$5.49\%$	$7.62\%$	$8.93\%$	$10.26\%$	$9.42\%$	$7.72\%$	$8.34\%$
SVR	$L=4$	$0.98\%$	$1.33\%$	$1.71\%$	$2.05\%$	$2.43\%$	$4.16\%$	$5.85\%$	$7.81\%$	$10.82\%$	$14.1\%$	$15.07\%$	$18.94\%$	$19.84\%$	$19.41\%$	$19.5\%$
	$L=12$	$1.05\%$	$1.33\%$	$1.62\%$	$1.91\%$	$2.17\%$	$3.6\%$	$6.59\%$	$9.09\%$	$13.5\%$	$17.89\%$	$19.2\%$	$24.47\%$	$27.77\%$	$28.52\%$	$29.88\%$
	$L=24$	$1.06\%$	$1.29\%$	$1.5\%$	$1.68\%$	$1.85\%$	$2.73\%$	$4.82\%$	$6.85\%$	$9.56\%$	$12.47\%$	$13.85\%$	$17.34\%$	$17.83\%$	$17.56\%$	$18.33\%$
	$L=48$	$1.25\%$	$1.46\%$	$1.64\%$	$1.8\%$	$1.95\%$	$2.66\%$	$4.1\%$	$5.27\%$	$7.9\%$	$11.33\%$	$13.05\%$	$12.39\%$	$9.87\%$	$8.45\%$	$8.07\%$
EGPM	$L=4,K=3$	$0.97\%$	$1.29\%$	$1.65\%$	$1.98\%$	$2.33\%$	$4.05\%$	$6.42\%$	$7.54\%$	$10.22\%$	$13.49\%$	$15.06\%$	$18.17\%$	$17.97\%$	$17.15\%$	$16.81\%$
	$L=4,K=5$	$0.97\%$	$1.28\%$	$1.64\%$	$1.97\%$	$2.32\%$	$4.03\%$	$6.38\%$	$7.53\%$	$10.18\%$	$13.45\%$	$15.04\%$	$18.18\%$	$17.97\%$	$17.16\%$	$16.83\%$
	$L=4,K=10$	$0.97\%$	$1.29\%$	$1.65\%$	$1.98\%$	$2.33\%$	$4.04\%$	$6.42\%$	$7.57\%$	$10.23\%$	$13.5\%$	$15.07\%$	$18.19\%$	$17.98\%$	$17.16\%$	$16.82\%$
	$L=12,K=3$	$0.93\%$	$1.19\%$	$1.49\%$	$1.77\%$	$2.08\%$	$3.65\%$	$5.92\%$	$8.16\%$	$11.44\%$	$14.12\%$	$15.35\%$	$18.99\%$	$19.32\%$	$18.78\%$	$18.44\%$
	$L=12,K=5$	$0.92\%$	$1.18\%$	$1.47\%$	$1.76\%$	$2.06\%$	$3.63\%$	$5.89\%$	$8.17\%$	$11.45\%$	$14.13\%$	$15.35\%$	$19.02\%$	$19.32\%$	$18.79\%$	$18.47\%$
	$L=12,K=10$	$0.95\%$	$1.21\%$	$1.51\%$	$1.79\%$	$2.1\%$	$3.67\%$	$5.92\%$	$8.13\%$	$11.4\%$	$14.15\%$	$15.39\%$	$19.05\%$	$19.32\%$	$18.77\%$	$18.42\%$
	$L=24,K=3$	$0.94\%$	$1.19\%$	$1.41\%$	$1.6\%$	$1.81\%$	$2.95\%$	$5.09\%$	$7.09\%$	$9.79\%$	$13.34\%$	$15.26\%$	$19.03\%$	$19.66\%$	$20.47\%$	$21.91\%$
	$L=24,K=5$	$0.97\%$	$1.22\%$	$1.43\%$	$1.62\%$	$1.83\%$	$2.97\%$	$5.01\%$	$6.96\%$	$9.51\%$	$12.62\%$	$14.39\%$	$17.57\%$	$17.85\%$	$18.66\%$	$20.4\%$
	$L=24,K=10$	$0.95\%$	$1.2\%$	$1.42\%$	$1.62\%$	$1.82\%$	$2.91\%$	$4.89\%$	$6.84\%$	$9.19\%$	$12.32\%$	$14.18\%$	$17.67\%$	$18.28\%$	$19.14\%$	$20.78\%$
	$L=48,K=3$	$1.02\%$	$1.29\%$	$1.52\%$	$1.74\%$	$1.93\%$	$2.83\%$	$5.42\%$	$7.03\%$	$9.0\%$	$11.88\%$	$13.93\%$	$23.48\%$	$32.01\%$	$38.54\%$	$45.35\%$
	$L=48,K=5$	$1.02\%$	$1.28\%$	$1.52\%$	$1.74\%$	$1.93\%$	$2.92\%$	$5.68\%$	$7.44\%$	$9.46\%$	$12.35\%$	$14.42\%$	$24.12\%$	$32.28\%$	$38.92\%$	$45.36\%$
	$L=48,K=10$	$1.02\%$	$1.29\%$	$1.53\%$	$1.76\%$	$1.95\%$	$2.94\%$	$5.7\%$	$7.43\%$	$9.38\%$	$12.29\%$	$14.38\%$	$23.61\%$	$32.3\%$	$38.86\%$	$45.69\%$
Mix-GPFR	$P=30,K=5$	$0.82\%$	$0.97\%$	$1.12\%$	$1.25\%$	$1.39\%$	$2.09\%$	$3.37\%$	$4.24\%$	$5.75\%$	$7.94\%$	$9.19\%$	$10.67\%$	$9.65\%$	$7.19\%$	$7.24\%$
Mix-GPNM	$K=5$	$0.78\%$	$0.94\%$	$1.11\%$	$1.26\%$	$1.4\%$	$2.16\%$	$3.47\%$	$4.34\%$	$5.85\%$	$8.02\%$	$9.27\%$	$10.71\%$	$9.67\%$	$7.2\%$	$7.25\%$
DPM-GPFR	$P=30$	$0.83\%$	$0.91\%$	$0.97\%$	$1.03\%$	$1.09\%$	$1.4\%$	$2.09\%$	$2.61\%$	$3.38\%$	$4.14\%$	$4.8\%$	$10.15\%$	$12.35\%$	$12.26\%$	$12.81\%$
HM-GPFR	$P=30,K=5$	$0.93\%$	$1.12\%$	$1.3\%$	$1.48\%$	$1.66\%$	$2.51\%$	$4.07\%$	$5.18\%$	$6.79\%$	$8.8\%$	$9.83\%$	$10.76\%$	$9.49\%$	$6.82\%$	$6.77\%$
BHM-GPFR	$P=30,K=5$	$0.77\%$	$0.92\%$	$1.07\%$	$1.18\%$	$1.3\%$	$1.89\%$	$2.88\%$	$3.59\%$	$4.89\%$	$6.88\%$	$8.04\%$	$9.85\%$	$9.21\%$	$6.94\%$	$7.15\%$

. From the table, we can see that the prediction accuracy of the classical time series forecast methods decreased significantly as we increased the prediction step. Among them, SARMA outperformed AR, MA, ARMA, and ARIMA, because SARMA takes the periodicity of data into consideration and can fit the data more effectively. The results of machine learning methods LSTM, NN, SVR, and EGPM also had similar phenomena, that is, when S was small, the prediction accuracy was high, and when S was large, the prediction accuracy was low. This observation indicates that these methods are not suitable for long-term prediction. In addition, machine learning methods are also sensitive to the settings of the parameters. For example, the results of FNN and SVR were better when $L=4$, which was close to SARMA, while the long-term prediction accuracy of EGPM decreased significantly when L was relatively large. It is challenging to appropriately set the hyper-parameters in practice. When making a long-term prediction, classical time series prediction methods and machine learning methods need to recursively predict the subsequent values based on the estimated values, which will cause the accumulation and amplification of errors. On the other hand, GPFR-related methods first make predictions according to the mean function, then finely correct these predictions based on observed data. The mean function part can better describe the evolution law of the data, which enables the use of historical information and structural information in the data more effectively. Mix-GPFR, mix-GPNM, and DPM-GPFR obtained similar results in long-term prediction compared with SARMA and could even achieve the best results in short-term prediction. This observation demonstrates the effectiveness of GPFR-related methods. However, these methods cannot deal with long-term prediction tasks well due to the “cold start” problem. Overall, the performances of the proposed HM-GPFR and BHM-GPFR were more comprehensive. For medium-term and short-term prediction, the results of HM-GPFR and BHM-GPFR were slightly worse than those of mix-GPFR, mix-GPNM, and DPM-GPFR, but they still enjoyed significant advantages compared with the other comparison methods. In terms of long-term forecasting, HM-GPFR and BHM-GPFR outperformed mix-GPFR, mix-GPNM, and DPM-GPFR, which shows that considering the multi-scale temporal structure between daily electricity load time series can effectively improve the accuracy of long-term forecasting. In addition, BHM-GPFR was generally better than HM-GPFR, which shows that giving prior distributions to the parameters and learning in a fully Bayesian way can further increase the robustness of the model and improve the prediction accuracy.

HM-GPFR and BHM-GPFR have strong interpretability. Specifically, the estimated values of the hidden variables obtained after training ${\left\{{\widehat{z}}_i\right\}}_{i=1}^n$ divided the daily electricity load records into K categories according to the evolution law. Each evolution pattern can be represented by the mean function of the GPFR component, and these evolution patterns transfer to each other with certain probabilities. The transfer law is characterized by the transfer probability matrix in the model. In Figure 3, we visualize the evolution patterns and transfer laws learned by HM-GPFR and BHM-GPFR. We call the evolution law corresponding to the mean function represented by the orange curve (at the top of the figure) Mode 1, and we call the five evolution modes as Mode 1 to Mode 5, respectively, in clockwise order. Combined with the practical application background, some meaningful laws can be found according to the results of the learned models. Examples are as follows:

• The electricity load of Mode 1 was the lowest. Besides, Mode 1 was relatively stable: when the system was in this evolution pattern, then it would stay in this state in the next step with a probability of about $0.5$. In the case of state transition, the probability of transferring to the mode with the second-lowest load (Mode 2 in Figure 3a and Mode 3 in Figure 3b) was high, while the probability of transferring to the mode with the highest load (Mode 5 in Figure 3a and Mode 2 and Mode 5 in Figure 3b) was relatively low;
• The evolution laws of Mode 2 and Mode 5 in Figure 3b are very similar, but the probabilities of transferring to other modes are different. From the perspective of electricity load alone, both of them can be regarded as the mode with the highest load. When the system was in the mode with the highest load (Mode 5 in Figure 3a and Mode 2 and Mode 5 in Figure 3b), the probability of remaining in this state in the next step was the same as that of transferring to the mode with the lowest (Mode 1);
• When the system was in the mode with the second-highest load (Mode 3 in Figure 3a and Mode 4 in Figure 3b), the probability of remaining in this state in the next step was low, while the probabilities of transferring to the modes with the lowest load and the highest load were high.

These laws are helpful for us to understand the algorithm, have a certain guiding significance for production practice, and can also be further analyzed in combination with expert knowledge.

The case of $S=1$ in Table 1 is the most common in practical applications, that is a one-step-ahead rolling forecast. As discussed in sec:HMMGPFR, when making a rolling prediction, HM-GPFR and BHM-GPFR can dynamically adjust the model incrementally after collecting new data without retraining the model. The results of the one-step-ahead rolling prediction of HM-GPFR and BHM-GPFR on the electricity load dataset are shown in Figure 4. It can be seen that the predicted values of HM-GPFR and BHM-GPFR were very close to the ground-truths, indicating that they are effective for rolling prediction. In the figure, the color of each point is the weighted average of the colors corresponding to each mode in Figure 3 according to the weight ${\omega }_K$. Note that there are color changes in some electricity load curves in Figure 4a,b. Taking the time series in Figure 4a in the range of about 1100–1200 as an example, when there are few observation data on that day, HM-GPFR believes that the electricity load evolution pattern of that day is more likely to belong to Mode 3. With the gradual increase of observation data, the model tends to think that the electricity load evolution pattern of that day belongs to Mode 5 and then tends to Mode 3 again. This shows that HM-GPFR and BHM-GPFR can adjust the value of $z_{i_{*}}$ in a timely manner according to the latest information during the rolling prediction.

6.3. Clustering Structure

The estimated values of latent variable ${\widehat{z}}_i$ also indicate the evolution mode corresponding to the data of the i-th day. Figure 5 visualizes some training data with different colors indicating different evolution modes, so we can intuitively see the multi-scale structure in the electricity load time series. According to the learned transition probability, we can obtain the stationary distribution of the Markov chain $(z_1,z_2,\dots ,z_N)$, which is ${[0.4825,0.2026,0.0513,0.1124,0.1513]}^{\mathrm{T}}$ in HM-GPFR and ${[0.4501,0.0427,0.2992,0.1381,0.0700]}^{\mathrm{T}}$ in BHM-GPFR. The proportion of each mode in Figure 5 is roughly consistent with the stationary distribution.

6.4. Ablation Study

In this section, we mainly compared HM-GPFR, BHM-GPFR with mix-GPFR, mix-GPNM, and DPM-GPFR to explore the impact of introducing coarse-grained temporal structure on the prediction performance. The MAPEs reported in

Table 2. MAPE of GP-related methods under the cold start setting ($r=1$).

Method	Step Length S
	1	2	3	4	5	10	20	30	50	80	100	200	300	500	1000
Mix-GPFR	$6.72\%$	$6.76\%$	$6.98\%$	$7.02\%$	$7.21\%$	$7.12\%$	$7.18\%$	$7.07\%$	$8.98\%$	$8.69\%$	$8.49\%$	$11.66\%$	$10.85\%$	$7.93\%$	$7.4\%$
Mix-GPNM	$6.72\%$	$6.76\%$	$6.98\%$	$7.03\%$	$7.2\%$	$7.11\%$	$7.18\%$	$7.07\%$	$8.98\%$	$8.69\%$	$8.48\%$	$11.66\%$	$10.85\%$	$7.93\%$	$7.4\%$
DPM-GPFR	$11.79\%$	$11.81\%$	$12.05\%$	$12.14\%$	$12.35\%$	$12.41\%$	$12.66\%$	$12.53\%$	$13.75\%$	$12.97\%$	$12.68\%$	$15.35\%$	$11.93\%$	$8.4\%$	$6.41\%$
HM-GPFR	$6.47\%$	$6.43\%$	$6.61\%$	$6.62\%$	$6.78\%$	$6.6\%$	$6.71\%$	$6.58\%$	$8.41\%$	$8.14\%$	$7.92\%$	$11.52\%$	$10.48\%$	$7.44\%$	$6.77\%$
BHM-GPFR	$4.58\%$	$4.61\%$	$4.84\%$	$4.88\%$	$5.07\%$	$4.98\%$	$5.15\%$	$5.32\%$	$7.3\%$	$7.04\%$	$6.76\%$	$10.67\%$	$10.23\%$	$7.58\%$	$7.24\%$

are averaged with respect to $r=1,\dots ,100$, while in this section, we paid special attention to the case of $r=1$. In this case, the observed data are the electricity load records in 2010, and there are no partial observations on January 1, 2011 (i.e., $M=0$ in Equation (2)). Therefore, mix-GPFR, mix-GPNM, and DPM-GPFR will encounter the cold start problem. Table 2 reports the MAPE of these methods at different prediction steps when $r=1$. It can be seen from the table that the prediction accuracy of HM-GPFR and BHM-GPFR was higher than that of mix-GPFR, mix-GPNM, and DPM-GPFR at almost every step, which shows that coarse-grained temporal information is helpful to improve the prediction performance, and the use of the Markov chain to model the transfer law of electricity load evolution patterns can make effective use of coarse-grained temporal information.

Figure 6 further shows the results of the multi-step prediction of these methods on the electricity load dataset. Here is also the case of “cold start” ($r=1$), and we predicted the electricity loads in the next 10 days (960 time points in total). It can be seen from the figure that these methods can effectively utilize the periodic structure in the time series; the prediction results showed periodicity, but the prediction results of HM-GPFR and BHM-GPFR were slightly different from the other methods. Due to the problem of the “cold start”, the predictions of mix-GPFR, mix-GPNM, and DPM-GPFR for each day were the same, i.e., ${\widehat{\mathit{y}}}_{N+1}={\widehat{\mathit{y}}}_{N+2}=\dots ={\widehat{\mathit{y}}}_{N+10}$, while HM-GPFR and BHM-GPFR use coarse-grained temporal information when making predictions and then adjust the predicted values of each day. Based on the predicted values of the other methods, it can be seen from the figure that the predicted values of HM-GPFR and BHM-GPFR on the first day were higher, and with the increase in step size, the predicted values will tend to the weighted average value of the mean function of each GPFR component.

6.5. Multi-Step Prediction under Cold Start Setting

In order to more clearly see the role of the Markov chain structure of hidden variables in the cold start setting, in Figure 7 and Figure 8, we show the predicted values of HM-GPFR and BHM-GFPR for electricity load in the next five days ${\widehat{\mathit{y}}}_{N+1},\dots ,{\widehat{\mathit{y}}}_{N+5}$ and the distributions of the latent variables $z_{N+1},\dots ,z_{N+5}$ conditioned on ${\widehat{z}}_N=k$. It can be seen from the figure that HM-GPFR and BHM-GPFR have different predictions for each day’s electricity load, which will be adjusted according to the transition probability of the evolution law. For example, in Figure 7, when ${\widehat{z}}_N=1$, the power load on that day is low, and the predicted value of HM-GPFR on the $(N+1)$-th day is also low. When $hat{z}_N=5$, the electricity load on that day is higher, and the predicted value of HM-GPFR on the $(N+1)$-th day is also higher. Figure 8 has a similar phenomenon. In addition, it can be seen that, with the increase of $i_{*}$, $\mathbb{P}\left(z_{i_{*}}\right)$ quickly converged to the stable distribution of the Markov chain, and the predicted value ${\widehat{\mathit{y}}}_{i_{*}}$ also tended to be the weighted average of the mean function in each GPFR component. In conclusion, these phenomena demonstrated that HM-GPFR and BHM-GPFR can effectively use the coarse-grained temporal structure to adjust the prediction of each day.

6.6. Sensitivity of Hyper-Parameters

There are two main hyper-parameters in HM-GPFR and BHM-GPFR: the number of B-spline basis functions D and the number of GPFR components K. Here, we mainly focused on the selection of K. We varied K in $\{3,4,5,6,7,8,9,10,15,20,30,50\}$, trained HM-GPFR and BHM-GPFR, respectively, and report the results in

Table 3. Sensitivity of HM-GPFR and BHM-GPFR with respect to the number of components K.

Method	K	Step Length S
		1	2	3	4	5	10	20	30	50	80	100	200	300	500	1000
HM-GPFR	3	$0.84\%$	$1.01\%$	$1.19\%$	$1.35\%$	$1.52\%$	$2.35\%$	$3.85\%$	$4.82\%$	$6.36\%$	$8.48\%$	$9.63\%$	$10.71\%$	$9.43\%$	$6.78\%$	$6.75\%$
	4	$0.9\%$	$1.09\%$	$1.27\%$	$1.44\%$	$1.61\%$	$2.46\%$	$4.02\%$	$5.09\%$	$6.6\%$	$8.57\%$	$9.68\%$	$10.71\%$	$9.43\%$	$6.78\%$	$6.75\%$
	5	$0.93\%$	$1.12\%$	$1.3\%$	$1.48\%$	$1.66\%$	$2.51\%$	$4.07\%$	$5.18\%$	$6.79\%$	$8.8\%$	$9.83\%$	$10.76\%$	$9.49\%$	$6.82\%$	$6.77\%$
	6	$1.09\%$	$1.32\%$	$1.57\%$	$1.81\%$	$2.04\%$	$3.13\%$	$4.77\%$	$5.83\%$	$7.3\%$	$9.04\%$	$10.03\%$	$10.9\%$	$9.59\%$	$6.88\%$	$6.8\%$
	7	$1.06\%$	$1.27\%$	$1.49\%$	$1.7\%$	$1.91\%$	$2.87\%$	$4.5\%$	$5.66\%$	$7.28\%$	$9.17\%$	$10.13\%$	$10.88\%$	$9.58\%$	$6.87\%$	$6.79\%$
	8	$0.97\%$	$1.16\%$	$1.36\%$	$1.54\%$	$1.73\%$	$2.57\%$	$4.14\%$	$5.32\%$	$6.99\%$	$8.95\%$	$9.92\%$	$10.81\%$	$9.57\%$	$6.88\%$	$6.79\%$
	9	$1.1\%$	$1.33\%$	$1.56\%$	$1.79\%$	$2.01\%$	$3.06\%$	$4.77\%$	$5.93\%$	$7.49\%$	$9.3\%$	$10.29\%$	$10.99\%$	$9.6\%$	$6.88\%$	$6.8\%$
	10	$1.18\%$	$1.41\%$	$1.65\%$	$1.88\%$	$2.11\%$	$3.22\%$	$4.97\%$	$6.05\%$	$7.53\%$	$9.39\%$	$10.45\%$	$11.17\%$	$9.71\%$	$6.95\%$	$6.83\%$
	15	$1.25\%$	$1.48\%$	$1.72\%$	$1.94\%$	$2.17\%$	$3.29\%$	$5.03\%$	$6.12\%$	$7.63\%$	$9.42\%$	$10.5\%$	$11.22\%$	$9.71\%$	$6.94\%$	$6.82\%$
	20	$1.31\%$	$1.54\%$	$1.77\%$	$2.0\%$	$2.22\%$	$3.33\%$	$5.07\%$	$6.14\%$	$7.65\%$	$9.57\%$	$10.78\%$	$11.52\%$	$9.85\%$	$7.01\%$	$6.86\%$
	30	$1.37\%$	$1.62\%$	$1.87\%$	$2.12\%$	$2.38\%$	$3.62\%$	$5.45\%$	$6.5\%$	$7.98\%$	$9.76\%$	$10.92\%$	$11.57\%$	$9.86\%$	$7.01\%$	$6.86\%$
	50	$1.47\%$	$1.72\%$	$1.99\%$	$2.25\%$	$2.5\%$	$3.7\%$	$5.5\%$	$6.62\%$	$8.29\%$	$10.35\%$	$11.66\%$	$12.01\%$	$10.06\%$	$7.12\%$	$6.91\%$
BHM-GPFR	3	$0.85\%$	$1.02\%$	$1.19\%$	$1.36\%$	$1.52\%$	$2.35\%$	$3.87\%$	$4.85\%$	$6.37\%$	$8.46\%$	$9.6\%$	$10.7\%$	$9.47\%$	$6.84\%$	$6.82\%$
	4	$0.78\%$	$0.93\%$	$1.07\%$	$1.18\%$	$1.29\%$	$1.86\%$	$2.82\%$	$3.49\%$	$4.8\%$	$6.96\%$	$8.23\%$	$9.91\%$	$9.04\%$	$6.68\%$	$6.85\%$
	5	$0.77\%$	$0.92\%$	$1.07\%$	$1.18\%$	$1.3\%$	$1.89\%$	$2.88\%$	$3.59\%$	$4.89\%$	$6.88\%$	$8.04\%$	$9.85\%$	$9.21\%$	$6.94\%$	$7.15\%$
	6	$0.8\%$	$0.96\%$	$1.1\%$	$1.23\%$	$1.36\%$	$2.02\%$	$3.17\%$	$3.97\%$	$5.32\%$	$7.22\%$	$8.32\%$	$9.91\%$	$9.3\%$	$7.01\%$	$7.18\%$
	7	$0.79\%$	$0.95\%$	$1.1\%$	$1.22\%$	$1.33\%$	$1.94\%$	$3.01\%$	$3.79\%$	$5.12\%$	$6.89\%$	$7.99\%$	$9.76\%$	$9.34\%$	$7.18\%$	$7.39\%$
	8	$0.78\%$	$0.94\%$	$1.08\%$	$1.19\%$	$1.31\%$	$1.89\%$	$2.94\%$	$3.71\%$	$5.03\%$	$6.74\%$	$7.79\%$	$9.7\%$	$9.49\%$	$7.46\%$	$7.7\%$
	9	$0.78\%$	$0.93\%$	$1.07\%$	$1.18\%$	$1.29\%$	$1.86\%$	$2.86\%$	$3.61\%$	$4.92\%$	$6.69\%$	$7.77\%$	$9.73\%$	$9.52\%$	$7.53\%$	$7.8\%$
	10	$0.82\%$	$0.98\%$	$1.13\%$	$1.26\%$	$1.4\%$	$2.11\%$	$3.29\%$	$4.09\%$	$5.37\%$	$7.01\%$	$8.04\%$	$9.94\%$	$9.8\%$	$7.86\%$	$8.12\%$
	15	$0.79\%$	$0.94\%$	$1.07\%$	$1.18\%$	$1.29\%$	$1.84\%$	$2.86\%$	$3.64\%$	$4.95\%$	$6.66\%$	$7.7\%$	$9.89\%$	$9.96\%$	$8.25\%$	$8.6\%$
	20	$0.79\%$	$0.94\%$	$1.07\%$	$1.17\%$	$1.28\%$	$1.83\%$	$2.83\%$	$3.6\%$	$4.88\%$	$6.51\%$	$7.5\%$	$9.95\%$	$10.32\%$	$8.88\%$	$9.31\%$
	30	$0.8\%$	$0.95\%$	$1.07\%$	$1.18\%$	$1.29\%$	$1.83\%$	$2.82\%$	$3.58\%$	$4.86\%$	$6.52\%$	$7.53\%$	$10.04\%$	$10.46\%$	$9.07\%$	$9.52\%$
	50	$0.83\%$	$0.98\%$	$1.11\%$	$1.22\%$	$1.33\%$	$1.88\%$	$2.9\%$	$3.68\%$	$4.96\%$	$6.5\%$	$7.46\%$	$10.14\%$	$10.71\%$	$9.45\%$	$9.9\%$

. For HM-GPFR, its prediction performance tended to deteriorate with the increase of K. In short-term prediction, the MAPE increased significantly, while the MAPE changed less in long-term prediction. With the increase of K, the number of parameters in the model also increased, and the model tended to suffer from overfitting. For BHM-GPFR, with the increase of K, its long-term prediction performance decreased significantly, while the medium-term and short-term prediction results did not change much. This showed that BHM-GPFR can prevent overfitting to a certain extent after introducing the prior distributions to the parameters. In addition, we also note that, when $K\le 10$, the difference between the results corresponding to different K was not significant, which is a more reasonable choice. From the perspective of applications, we set $K=5$ in the experiment, which can take both the expression ability and interpretability of the model into consideration.

7. Conclusions and Discussions

In this paper, we proposed the concept of multi-scale time series. Multi-scale time series have two granularity temporal structures. We established the HM-GPFR model for multi-scale time series forecasting and designed an effective learning algorithm. In addition, we also gave a priori parameters to the model and obtained a more robust BHM-GPFR model. Compared with conventional GPFR-related methods (mix-GPFR, mix-GPNM, DPM-GPFR), the proposed method can effectively use the temporal information of both the fine level and coarse level, alleviates the “cold start” problem, and has good performance in short-term prediction and long-term prediction. HM-GPFR and BHM-GPFR not only achieved high prediction accuracy; they also had good interpretability. Combined with the actual problem background and domain knowledge, we can explain the state transition law learned by the model.

In practice, the number of hidden states K in HM-GPFR/BHM-GPFR can be set by expert knowledge. However, how to set K in a data-driven way is an interesting direction, and this is usually referred to as the model selection problem. The model selection problem is both important and challenging. One can run the algorithms with different K and use certain criteria (such as AIC, BIC) to choose the best one; however, this procedure is time-consuming, and the obtained result is generally unstable. For mix-GPFR, the Dirichlet process is utilized to tackle the model selection problem [21]. We suggest hierarchical-Dirichlet-process-based hidden Markov models [22,23,24] as a promising method for the model selection of HM-GPFR/BHM-GPFR. It is also promising to reduce the computational cost by introducing inducing points [25,26,27] to our proposed models, but how to balance the trade-off between performance and computational cost generally depends on the particular application scenario.