Multiple Time Series Forecasting Using Quasi-Randomized Functional Link Neural Networks

Abstract

We are interested in obtaining forecasts for multiple time series, by taking into account the potential nonlinear relationships between their observations. For this purpose, we use a specific type of regression model on an augmented dataset of lagged time series. Our model is inspired by dynamic regression models (Pankratz 2012), with the response variable’s lags included as predictors, and is known as Random Vector Functional Link (RVFL) neural networks. The RVFL neural networks have been successfully applied in the past, to solving regression and classification problems. The novelty of our approach is to apply an RVFL model to multivariate time series, under two separate regularization constraints on the regression parameters.

1. Introduction

In this paper, we are interested in obtaining forecasts for multiple time series, by taking into account the potential nonlinear relationships between their observations. This type of problem has been tackled recently by Exterkate et al. (2016), who applied kernel regularized least squares to a set of macroeconomic time series. The Long Short-Term Memory neural networks (LSTM) architectures (introduced by Hochreiter and Schmidhuber (1997)) are another family of models, which are currently widely used for this purpose. As a basis for our model, we will use (quasi-)randomized neural networks known as Random Vector Functional Link neural networks (RVFL networks hereafter).

The forecasting method described in this paper, provides useful inputs for Insurance Quantitative Risk Management models; the interested reader can refer to Bonnin et al. (2015) for an example.

To the best of our knowledge, randomized neural networks were introduced by Schmidt et al. (1992), and the RVFL networks were introduced by Pao et al. (1994). An early approach for multivariate time series forecasting using neural networks is described in Chakraborty et al. (1992). They applied a back propagation algorithm from Rumelhart et al. (1988) to trivariate time series, and found that the combined training of the series gave better forecasts than a separate training of each individual series. The novelty of the approach described in this paper is to derive an RVFL model for multiple time series, under two separate regularization constraints on the parameters, as it will be described in detail in Section 2.3.

RVFL networks are multilayer feedforward neural networks, in which there is a direct link between the predictors and the output variable, aiming at capturing the linear relationships. In addition to the direct link, there are new features: the hidden nodes (the dataset is augmented), that help to capture the nonlinear relationships between the time series. These new features are obtained by random simulation over a given interval. More details on the direct link and the hidden nodes will be provided in the next section.

The RVFL networks have been successfully applied to solving different types of classification and regression problems, see for example Dehuri and Cho (2010). More specifically, they have been applied to univariate time series forecasting by Ren et al. (2016). A comprehensive survey can be found in Zhang and Suganthan (2016), where a large number of model specifications are tested on classification problems, including changing the range of hidden layer’s randomization.

Here, we will use RVFL networks with one hidden layer. Furthermore, instead of relying on fully randomized nodes, we will use sequences of deterministic quasi-random numbers. Indeed, with fully randomized nodes, the model fits obtained are dependent on the choice of a simulation seed. Typically, a different fitting solution would be obtained for each seed.

In our various numerical examples from Section 3, we will apply the RVFL networks to forecasting trivariate time series, notably (but not only) in a Dynamic Nelson Siegel (DNS) framework (see Nelson and Siegel 1987; Diebold and Li 2006). We will obtain point forecasts and predictive distributions for the series, and see that in this RVFL framework, one (or more) variable(s) can be stressed, and influence the others. More precisely, about this last point, it means that it is possible, as in dynamic regression models (Pankratz 2012) to assign a specific future value to one regressor, and obtain forecasts of the remaining variables. Another advantage of the model described here is its ability to integrate multiple other exogenous variables, without overfitting in-sample data.

2. Description of the Model

The general procedure for obtaining the model’s optimal parameters and predictions is summarized in Figure 1.

This procedure is described in detail in the next sections, especially in Section 2.3 and Section 2.4.

2.1. On a Single Layer RVFL Network

We rely on single layer feed forward neural networks (SLFN). Considering that an output variable $y\in {\mathbb{R}}^n$ is to be explained by a set of observed predictors $Z^{(j)}\in {\mathbb{R}}^n$, $j\in \left\{1,\dots ,p\right\}$, the RVFL networks we will use to explain y can be described for $i\in \left\{1,\dots ,n\right\}$ as:

$$y_i={\beta }_0+\sum _{j=1}^p{\beta }_j{Z}_i^{(j)}+\sum _{l=1}^L{\gamma }_{l}g\left(\sum _{j=1}^p{W}^{(j,l)}{Z}_i^{(j)}\right)+{ϵ}_i$$

g is called activation function, L is the number of nodes in the hidden layer, $W^{(j,l)}$ are elements of the hidden layer, and the parameters ${\beta }_j$ and ${\gamma }_l$ are to be learned from the observed data $Z^{(j)},j\in \left\{1,\dots ,p\right\}$. The ${ϵ}_i$’s are the residual differences between the output variable values and the RVFL model.

This type of model can be seen as one explaining $y_i$, by finding a compromise between linear and potentially non-linear effects of the original predictors $Z^{(j)}$ and transformed predictors

$$\mathsf{\Phi }{(\mathbf{Z})}^{(l)}=g\left(\sum _{j=1}^p{W}^{(j,l)}{Z}_i^{(j)}\right)$$

$\left\{1,\dots ,L\right\}$ on the response. Common choices for function g in neural networks regression are the sigmoïd activation function

$$g:x\mapsto \frac{1}{1+e^{-x}}$$

the hyperbolic tangent function,

$$g:x\mapsto tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$$

or the Rectified Linear Units, known as ReLU

$$g:x\mapsto max(x,0)$$

The main differences between the RVFL framework and a classical SLFN framework are:

• The inclusion of a linear dependence between the output variable and the predictors: the direct link, ${\beta }_0+{\sum }_{j=1}^p{\beta }_j{Z}_i^{(j)}$.
• The elements $W^{(j,l)}$ of the hidden layer are typically not trained, but randomly and uniformly chosen on a given interval. Different ranges for these elements of the hidden layer are tested in (Zhang and Suganthan 2016).

Solving for the optimal parameters ${\beta }_j$’s and ${\gamma }_l$’s can be done by applying directy a least squares regression of y on the set of observed and transformed predictors. But since these input predictors are likely to be highly correlated—especially in our setting, with time series data—we do not search each of these parameters on the entire line, but in restricted regions where we have:

$$\sum _{j=1}^p{\beta }_j^2\le u$$

and

$$\sum _{l=1}^L{\gamma }_l^2\le v$$

for $u,v\in {\mathbb{R}}^{\ast }$. That is, applying some kind of Tikhonov regularization or ridge regression model (Hoerl and Kennard 1970) of y on the set of observed and transformed predictors. Having two constraints instead of one allows for more flexibility in the covariance structure between the predictors and the output, with ${\beta }_j$’s and ${\gamma }_l$’s moving in separate balls. For these constraints to be applicable, the input variables will need to be standardized, so as to be expressed on the same scales, and the response variable will be centered.

Imposing these restrictions to the model’s parameters increases their interpretability—by reducing their variance—at the expense of a slight increase in in-sample bias. It also prevents the model from overfitting the data as in ridge regression (Hoerl and Kennard 1970). One of the advantages of RVFL networks is that they are relatively fast to train, due to the availability of closed-form formulas for the model’s parameters, as it will be presented in the next section.

On the other hand, RVFL networks incorporate some randomness in the hidden layer, which makes each model relatively dependent on the choice of a simulation seed. Each seed would indeed produce a different set of parameters ${\beta }_j$’s and ${\gamma }_l$’s for the model. For that reason, we will also use sequences of deterministic quasi-random numbers in the hidden layer. The elements $W^{(j,l)}$ of the hidden layer are taken from a quasi-random (deterministic) sobol sequence on $[0,1]$, which is shifted in such a way that they belong to $[-1,1]$.

Sobol sequences are part of quasi-random numbers, which are also called low discrepancy sequences. As described intuitively in Boyle and Tan (1997), the discrepancy of a sequence of N points in a subcube $V\in {[0,1)}^d$ is defined as:

$$su{p}_{V\in {[0,1)}^d}|\frac{numberofpointsinV}{N}-v(V)|$$

where $v(V)$ is the volume of V. It describes how well the points are dispersed within V. The idea is to have points which are more or less equidispersed in V. Joe and Kuo (2008) describe the generation of the $i^{th}$ term, $j^{th}$ component ($x_{i,j}$) of a Sobol sequence. The generation starts with obtaining the binary representation of i. That is, obtaining i as:

$$i=\sum _k{i}_k{2}^k={(\dots i_3{i}_2{i}_1)}_2$$

For example, $5=1\times 2^2+0\times 2^1+1\times 2^0$ can be expressed as ${(101)}_2$ in binary representation. Then, by using the sequence of bits describing i in base 2, we can obtain $x_{i,j}$ as:

$$x_{i,j}=i_1{v}_{1,j}\oplus i_2{v}_{2,j}\oplus \dots$$

(1)

where ⊕ is a bitwise exclusive-or operation, and the numbers $v_{i,j}$ are called the direction numbers, defined for $k\ge 1$ as:

$$v_{k,j}=\frac{m_{k,j}}{2^k}=\frac{2a_{1,j}{m}_{k-1,j}\oplus 2^2{a}_{2,j}{m}_{k-2,j}\oplus \dots \oplus 2^{s_j-1}{a}_{s_j-1,j}{m}_{k-s_j+1,j}\oplus 2^{s_j}{m}_{k-s_j,j}\oplus m_{k-s_j,j}}{2^k}$$

(2)

A few details on Equation (2):

• The bitwise exclusive-or operation ⊕ applied to two integers p and $q\in \left\{0,1\right\}$ returns 1 if and only if one of the two (but not both) inputs is equal to 1. Otherwise, $p\oplus q$ is equal to 0.
• The second term of the equation relies on primitive polynomials of degree $s_j$, with coefficients $a_{i,j}$ taken in $\left\{0,1\right\}$:

  $$x^{s_j}+a_{1,j}{x}^{s_j-1}+a_{2,j}{x}^{s_j-2}+\dots +a_{s_j-1,j}x+1$$

  (3)
• The terms $m_{k,j}$ are obtained recursively, with the initial values $m_{1,j},m_{2,j},\dots ,m_{k-s_j,j}$ chosen freely, under the condition that $m_{k,j},1\le k\le s_j$ is odd and less than $2^k$.

A more complete treatment of low discrepancy sequences can be found in Niederreiter (1992). In addition, an example with $s_j=3$, $a_{1,j}=0$, $a_{2,j}=1$ is given in Joe and Kuo (2008).

2.2. Applying RVFL Networks to Multivariate Time Series Forecasting

We consider $p\in {\mathbb{N}}^{\ast }$ time series ${(X_t^{(j)})}_{t\ge 0},j=1,\dots ,p$, observed at $n\in {\mathbb{N}}^{\ast }$ discrete dates. We are interested in obtaining simultaneous forecasts of the p time series at time $n+h$, $h\in {\mathbb{N}}^{\ast }$, by allowing each of the p variables to be influenced by the others (in the spirit of VAR models, see Lütkepohl 2005).

For this purpose, we use $k<n$ lags of each of the observed p time series. The output variables to be explained are:

$$Y^{(j)}={\left(X_n^{(j)},\dots ,X_{k+1}^{(j)}\right)}^T$$

(4)

for $j\in \left\{1,\dots ,p\right\}$. Where $X_n^{(j)}$ is the most contemporaneous observed value of the j-th time series, and $X_{k+1}^{(j)}$ was observed k dates earlier in time for ${(X_t^{(j)})}_{t\ge 0}$. These output variables are stored in a matrix:

$$\mathbf{Y}\in {\mathbb{R}}^{(n-k)\times p}$$

and the predictors are stored in a matrix:

$$\mathbf{X}\in {\mathbb{R}}^{(n-k)\times (k\times p)}$$

where $\mathbf{X}$ consists in p blocks of k lags, for each one of the observed p time series. For example, the $j_0^{th}$ block of $\mathbf{X}$, for $j_0\in \left\{1,\dots ,p\right\}$ contains in columns:

$${\left(X_{n-i}^{(j_0)},\dots ,X_{k+1-i}^{(j_0)}\right)}^T$$

(5)

with $i\in \left\{1,\dots ,k\right\}$. It is also possible to add other regressors, such as dummy variables, indicators of special events, but for clarity, we consider only the inclusion of lags.

As described in the previous section, an additional layer of transformed predictors is added to $\mathbf{X}$, in order to capture the potentially non-linear interactions between the predictors and the output variable. Adding the transformed predictors to the original ones leads to a new matrix of predictors with dimensions $(n-k)\times (k\times p+L)$, where L is the number of nodes in the hidden layer. We are then looking for simultaneous predictions

$${\widehat{X}}_{n+h|n,\dots ,1}^{(j)}=:{\widehat{X}}_{n+h}^{(j)}$$

for $h\in {\mathbb{N}}^{\ast }$, and $j\in \left\{1,\dots ,p\right\}$. This is a multi-task learning problem (see Caruana 1998), in which the output variables will all share the same set of predictors.

For example, we have $p=2$ time series $(X_{t_1}^{(1)},\dots ,X_{t_5}^{(1)})$ and $(X_{t_1}^{(2)},\dots ,X_{t_5}^{(2)})$ observed at $n=5$ dates $t_1<\dots <t_5$, with $k=2$ lags, and $L=3$ nodes in the hidden layer. In this case, the response variables are stored in:

$$\mathbf{Y}=\left(\begin{array}{cc}{X}_{t_5}^{(1)}& X_{t_5}^{(2)}\\ X_{t_4}^{(1)}& X_{t_4}^{(2)}\\ X_{t_3}^{(1)}& X_{t_3}^{(2)}\end{array}\right)$$

The predictors are stored in:

$$\mathbf{X}=\left(\begin{array}{cccc}{X}_{t_4}^{(1)}& X_{t_3}^{(1)}& X_{t_4}^{(2)}& X_{t_3}^{(2)}\\ X_{t_3}^{(1)}& X_{t_2}^{(1)}& X_{t_3}^{(2)}& X_{t_2}^{(2)}\\ X_{t_2}^{(1)}& X_{t_1}^{(1)}& X_{t_2}^{(2)}& X_{t_1}^{(2)}\end{array}\right)$$

And the coefficients in the hidden layer are:

$$\mathbf{W}=\left(\begin{array}{ccc}{W}^{(1,1)}& W^{(1,2)}& W^{(1,3)}\\ W^{(2,1)}& W^{(2,2)}& W^{(2,3)}\\ W^{(3,1)}& W^{(3,2)}& W^{(3,3)}\\ W^{(4,1)}& W^{(4,2)}& W^{(4,3)}\end{array}\right)$$

2.3. Solving for $\widehat{\beta }$’s and $\widehat{\gamma }$’s

We let y be the $j_0^{th}$ column (out of p) of the response matrix $\mathbf{Y}$, and $\mathsf{\Phi }(\mathbf{X})$ be the matrix of transformed predictors obtained from $\mathbf{X}$ by the hidden layer described at the beginning of Section 2.1. We also denote the set of regression parameters associated with this $j_0^{th}$ time series, as:

$${\beta }_m^{(j_0)}=:{\beta }_m$$

and

$${\gamma }_l^{(j_0)}=:{\gamma }_l$$

for $m\in \left\{1,\dots ,k\right\}$; $l\in \left\{1,\dots ,L\right\}$. Solving for the regression parameters for the $j_0^{th}$ time series, under the constraints

$$\sum _{m=1}^{k\times p}{\beta }_m^2\le u$$

and

$$\sum _{l=1}^L{\gamma }_l^2\le v$$

for $u,v\in {\mathbb{R}}^{\ast }$, leads to minimizing a penalized residual sum of squares. Hence, for vectors $\beta \in {\mathbb{R}}^{(k\times p)}$ and $\gamma \in {\mathbb{R}}^L$ containing the regression parameters, we obtain the Lagrangian:

$$\mathcal{L}(\mathbf{X};\beta ,\gamma )={\left(y-\mathbf{X}\beta -\mathsf{\Phi }(\mathbf{X})\gamma \right)}^T\left(y-\mathbf{X}\beta -\mathsf{\Phi }(\mathbf{X})\gamma \right)+{\lambda }_1{\beta }^T\beta +{\lambda }_2{\gamma }^T\gamma$$

where ${\lambda }_1$ and ${\lambda }_2$ are Lagrange multipliers. Taking the first derivatives of $\mathcal{L}$ relative to $\beta$ and $\gamma$ leads to:

$$\begin{array}{ccc}\hfill \frac{\partial \mathcal{L}(\mathbf{X};\beta ,\gamma )}{\partial \beta }& =& -y^T\mathbf{X}-{\mathbf{X}}^{T}y+2({\mathbf{X}}^T\mathbf{X})\beta +{\mathbf{X}}^T\mathsf{\Phi }(\mathbf{X})\gamma +{\left(\mathsf{\Phi }(\mathbf{X})\gamma \right)}^T\mathbf{X}+2{\lambda }_1\beta \hfill \\ & =& 2({\mathbf{X}}^T\mathbf{X}+{\lambda }_1{I}_{k\times p})\beta -y^T\mathbf{X}-{\mathbf{X}}^{T}y+{\mathbf{X}}^T\mathsf{\Phi }(\mathbf{X})\gamma +{\left(\mathsf{\Phi }(\mathbf{X})\gamma \right)}^T\mathbf{X}\hfill \\ & =& 2({\mathbf{X}}^T\mathbf{X}+{\lambda }_1{I}_{k\times p})\beta -2{\mathbf{X}}^{T}y+2{\mathbf{X}}^T\mathsf{\Phi }(\mathbf{X})\gamma \hfill \end{array}$$

where $I_{k\times p}$ is the identity matrix with dimensions $(k\times p)\times (k\times p)$ and equivalently

$$\begin{array}{ccc}\hfill \frac{\partial \mathcal{L}(\mathbf{X};\beta ,\gamma )}{\partial \gamma }& =& 2(\mathsf{\Phi }{(\mathbf{X})}^T\mathsf{\Phi }(\mathbf{X})+{\lambda }_2{I}_L)\gamma -2\mathsf{\Phi }{(\mathbf{X})}^{T}y+2\mathsf{\Phi }{(\mathbf{X})}^T\mathbf{X}\beta \hfill \end{array}$$

where $I_L$ is the identity matrix with dimensions $L\times L$. And setting these first derivatives to 0 leads to:

$$\left\{\begin{array}{c}({\mathbf{X}}^T\mathbf{X}+{\lambda }_1{I}_{k\times p})\beta +{\mathbf{X}}^T\mathsf{\Phi }(\mathbf{X})\gamma ={\mathbf{X}}^{T}y\hfill \\ (\mathsf{\Phi }{(\mathbf{X})}^T\mathsf{\Phi }(\mathbf{X})+{\lambda }_2{I}_L)\gamma +\mathsf{\Phi }{(\mathbf{X})}^T\mathbf{X}\beta =\mathsf{\Phi }{(\mathbf{X})}^{T}y\hfill \end{array}\right.$$

That is:

$$\left(\begin{array}{cc}{\mathbf{X}}^T\mathbf{X}+{\lambda }_1{I}_{k\times p}\hfill & {\mathbf{X}}^T\mathsf{\Phi }(\mathbf{X})\hfill \\ \mathsf{\Phi }{(\mathbf{X})}^T\mathbf{X}\hfill & \mathsf{\Phi }{(\mathbf{X})}^T\mathsf{\Phi }(\mathbf{X})+{\lambda }_2{I}_L\hfill \end{array}\right)\left(\begin{array}{c}\beta \\ \gamma \end{array}\right)=\left(\begin{array}{c}{\mathbf{X}}^{T}y\\ \mathsf{\Phi }{(\mathbf{X})}^{T}y\end{array}\right)$$

Now, if we denote:

$$A=\left(\begin{array}{cc}{\mathbf{X}}^T\mathbf{X}+{\lambda }_1{I}_{k\times p}\hfill & {\mathbf{X}}^T\mathsf{\Phi }(\mathbf{X})\hfill \\ \mathsf{\Phi }{(\mathbf{X})}^T\mathbf{X}\hfill & \mathsf{\Phi }{(\mathbf{X})}^T\mathsf{\Phi }(\mathbf{X})+{\lambda }_2{I}_L\hfill \end{array}\right)=:\left(\begin{array}{cc}B& C^T\\ C& D\end{array}\right)$$

and $S=D-C{B}^{+}{C}^T$. Then, using the algorithm described in Cormen (2009) for blockwise matrix inversion, we obtain:

$$A^{+}=\left(\begin{array}{cc}{B}^{+}+B^{+}{C}^T{S}^{+}C{B}^{+}& -B^{+}{C}^T{S}^{+}\\ -S^{+}C{B}^{+}& S^{+}\end{array}\right)=:\left(\begin{array}{cc}{A}_1^{+}& A_2^{+}\\ A_3^{+}& A_4^{+}\end{array}\right)$$

where $S^{+}$ and $B^{+}$ are the Moore-Penrose pseudo-inverse (Penrose 1955) of matrixes S and B. Hence for each column y of $\mathbf{Y}$, we have the solutions:

$$\left(\begin{array}{c}\widehat{\beta }\\ \widehat{\gamma }\end{array}\right)=\left(\begin{array}{cc}{A}_1^{+}& A_2^{+}\\ A_3^{+}& A_4^{+}\end{array}\right)\left(\begin{array}{c}{\mathbf{X}}^{T}y\\ \mathsf{\Phi }{(\mathbf{X})}^{T}y\end{array}\right)$$

And the whole set of parameters, for all the p observed time series is given by:

$$\left(\begin{array}{c}\widehat{\underline{\beta }}\\ \widehat{\underline{\gamma }}\end{array}\right):=\left(\begin{array}{cc}{A}_1^{+}& A_2^{+}\\ A_3^{+}& A_4^{+}\end{array}\right)\left(\begin{array}{c}{\mathbf{X}}^T\mathbf{Y}\\ \mathsf{\Phi }{(\mathbf{X})}^T\mathbf{Y}\end{array}\right)$$

The objective function to be minimized (the least squares) is convex, and so is the set of feasible solutions. The solutions $\widehat{\underline{\beta }}$ and $\widehat{\underline{\gamma }}$ found here are hence global minima.

2.4. h-Steps Ahead Forecasts and Use of Dynamic Regression

Having obtained the optimal set of parameters $\widehat{\underline{\beta }}$ and $\widehat{\underline{\gamma }}$ as described in the previous section, a new set of predictors is constructed by using the former output variables contained in response matrix $\mathbf{Y}$’s columns. The first k elements of each one of the p columns of $\mathbf{Y}$, which are the most contemporaneous values of the p series, constitute the new predictors.

Hence, if we denote the new predictors as:

$$\begin{array}{ccc}\hfill {\mathbf{X}}_n^{\ast }& =& \left(X_n^{(1)},\dots ,X_{n-k+1}^{(1)},\dots ,\dots ,X_n^{(p)},\dots ,X_{n-k+1}^{(p)}\right)\hfill \end{array}$$

(6)

The 1-step ahead forecasts are obtained as:

$$\left({\widehat{X}}_{n+1}^{(1)},\dots ,{\widehat{X}}_{n+1}^{(p)}\right)=\left({\mathbf{X}}_n^{\ast }\mathsf{\Phi }({\mathbf{X}}_n^{\ast })\right)\left(\begin{array}{c}\widehat{\underline{\beta }}\\ \widehat{\underline{\gamma }}\end{array}\right)$$

The h-step ahead forecasts are obtained in a similar fashion; with the new forecasts $\left({\widehat{X}}_{n+1}^{(1)},\dots ,{\widehat{X}}_{n+1}^{(p)}\right)$ being added to the set of most contemporaneous values of the p series, and used as part of the new predictors.

In order to obtain confidence intervals around the point forecasts, we fit an ARIMA model to the in-sample residuals ${ϵ}_i$ of each one of the p time series, as in dynamic regression models (see Pankratz (2012)). An illustration can be found in the next section. Other models for the autocorrelated residuals could be envisaged, though.

3. Numerical Examples

3.1. A Dynamic Nelson-Siegel Example

The following examples are not exhaustive benchmarks, but aim at illustrating the forecasting capabilities of the model described in this paper1. All the results on RVFL use the ReLU activation function. We use calibrated discount rates data from Deutsche Bundesbank website (http://www.bundesbank.de/Navigation/EN/Statistics/Time_series_databases/time_series_databases.html), observed on a monthly basis, from the beginning of 2002 to the end 2015. There are 167 curves, observed at 50 maturities in the dataset. We obtain curves’ forecasts in a Dynamic Nelson Siegel (Nelson and Siegel 1987) framework (DNS), in the spirit of Diebold and Li (2006) and other similar models2.

In Figure 2, we present the data that we use, and

Table 1. Summary of observed discount rates from Deutsche Bundesbank website, from 2002 to the end 2015.

Maturity	Min	1st Qrt	Median	3rd Qrt	Max
1	−0.116	0.858	2.045	3.072	5.356
5	0.170	1.327	2.863	3.807	5.146
15	0.711	2.616	3.954	4.702	5.758
30	0.805	2.594	3.962	4.814	5.784
50	0.749	2.647	3.630	4.590	5.467

contains a summary of these data; the minimum, maximum, median, first and third quartiles of the discount rates observed at given maturities. There are alternate cycles of increases and decreases of the discount rates, with a generally decreasing trend. Some of the discount rates, at the most recent dates, and lower maturities, are negative.

In the DNS framework, the spot interest rates observed at time t, for time to maturity $\tau$ are modeled as:

$$R_t(\tau )={\alpha }_{1,t}+{\alpha }_{2,t}\left(\frac{1-e^{-\tau /\lambda }}{e^{-\tau /\lambda }}\right)+{\alpha }_{3,t}\left(\frac{1-e^{-\tau /\lambda }}{e^{-\tau /\lambda }}-e^{-\tau /\lambda }\right)$$

(7)

The factor loadings 1, $\left(\frac{1-e^{-T/\lambda }}{e^{-T/\lambda }}\right)$ and $\left(\frac{1-e^{-T/\lambda }}{e^{-T/\lambda }}-e^{-T/\lambda }\right)$ are used to represent the level, slope, and curvature of the Yield Curve. We obtain estimations of ${\alpha }_{i,t},i=1,\dots ,3$ for each cross-section of yields by fixing $\lambda$, and doing a least squares regression on the factor loadings. The three time series ${\alpha }_{i,t},i=1,\dots ,3$ associated to the loadings for each cross-section of yields, are those that we wish to forecast simultaneously, by using an RVFL model.

This type of model (DNS) cannot be used for no-arbitrage pricing as is, but it could be useful for example, for stressing the yield curve factors under the historical probability. It can however be made arbitrage-free, if necessary (see Diebold and Rudebusch 2013). We will benchmark the RVFL model applied to the three time series ${\alpha }_{i,t},i=1,\dots ,3$, against ARIMA (AutoRegressive Integrated Moving Average) and VAR (Vector AutoRegressive) models. Diebold and Li (2006) applied an autoregressive AR(1) model separately to each one of the parameters, ${\alpha }_{i,t},i=1,\dots ,3$.

We will apply to these parameters’ series: an ARIMA model (Hyndman and Khandakar 2008), and a Vector Autoregressive model (VAR, see Pfaff et al. 2008; Lütkepohl 2005); with the parameter $\lambda$ of the DNS factor loadings, used as an hyperparameter for the time series cross-validation. In the RVFL and the VAR model, the number of lags is also used as an hyperparameter for the cross-validation. For the RVFL, the most recent values of ${\alpha }_{i,t},i=1,\dots ,3$ are stored in matrix $\mathbf{Y}$, as described in Section 2.3, ordered by date of arrival, whereas matrix $\mathbf{X}$ contains the lags of the three series.

A rolling forecasting methodology (see Bergmeir et al. 2015) is implemented in order to obtain these benchmarks. A fixed 12 months-length window is established for training the model, and the following 12 months are used for testing; the origin of the training set is then advanced of 1 month, and the training/testing procedure is repeated. The measure of forecasting performance is the Root Mean Squared Error ($RMSE$). Figure 3 presents boxplots for the distribution of out-of-sample errors obtained in the cross-validation procedure, and Figure 4 presents the 12 months-ahead out-of-sample errors over time. ARIMA (separate (Hyndman and Khandakar 2008) ARIMA models applied to each series ${\alpha }_{i,t},i=1,\dots ,3$) gives good results, as already suggested by Diebold and Li (2006). They are nearly comparable to results from RVFL, but a bit more volatile, with an outlier point observed on the $log(RMSE)$ box plot.

The unrestricted VAR model results include more volatility than the two other methods on this specific example, especially in the period of financial crisis between 2007 to 2009, as seen on Figure 4.

Table 2. Comparison of 12 months ahead out-of-sample $RMSE$ (log(Root Mean Squared Error)), for the ARIMA(AutoRegressive Integrated Moving Average), RVFL, and VAR (Vector AutoRegressive).

Method	Min.	1st Qrt	Median	Mean	3rd Qrt	Max.
RVFL	0.1487	0.3092	0.4491	0.5041	0.6414	1.1535
ARIMA	0.2089	0.3470	0.5187	0.6358	0.7516	5.3798
VAR	0.1402	0.3493	0.5549	1.9522	0.8619	122.2214

is to be read in conjuction with the $log(RMSE)$ box plot presented in Figure 3. It summarises the results obtained by the different methods on the out-of-sample $RMSE$.

Table 3. 95% confidence interval around the difference of out-of-sample $RMSE$.

Method	Lower Bound	Upper Bound	Mean
RVFL-ARIMA	−0.2116	−0.0518	−0.1317
RVFL-VAR	−3.1888	0.2927	−1.4480
ARIMA-VAR	−2.9937	0.3610	−1.3163

contains $95\%$ confidence intervals around the mean of the differences between the three methods.

Another advantage of RVFL over ARIMA or AR(1) in this context is that it would be possible to add other variables to the RVFL regression, such as inflation, or dummy variables for external events, and combine their effects. It is also possible to stress one variable, and see the effects on the other variables, as presented in the Appendix A.1: the parameter ${\alpha }_{1,t}$ is increased (from $0.75$ to $1.25$) and decreased (from $0.75$ to $0.25$), and the other parameters ${\alpha }_{2,t}$ and ${\alpha }_{3,t}$ forecasts move slightly, consecutively to these stresses. The corresponding median forecast curves for these stresses, and some additional ones, are presented in Figure 5.

3.2. Forecasting 1 Year, 10 Years and 20 Years Spot Rates

For this second example, we forecast the 1-year, 10-year and 20-year spot rates time series from the previous dataset, on a 12-month horizon. As described in the previous section, we use a rolling forecasting methodology, with a training window of 12 months’ length.

Figure 6 presents the three time series of data, and a summary of the data can be found in

Table 4. Summary of the data for 1-year, 10-years and 20-years spot rates time series (in %).

	Min.	1st Qrt	Median	Mean	3rd Qrt	Max.
1-year rate	−0.116	0.858	2.045	2.062	3.072	5.356
10-years rate	0.560	2.221	3.581	3.322	4.354	5.570
20-years rate	0.790	2.685	4.050	3.782	4.830	5.850

and

Table 5. Summary of data for 1-year, 10-year and 20-year spot rates time series.

Correlations	1-Year Rate	10-Years Rate	20-Years Rate
1-year rate	1.0000	0.8729	0.8118
10-years rate	0.8729	1.0000	0.9900
20-years rate	0.8118	0.9900	1.0000

.

The three time series globally exhibit a decreasing trend, and are highly positively correlated. The spot rates for short-term maturities can also be negative, as it has been observed recently in 2016. The spreads between the spot rates time series are extremely narrow during the 2007–2009 crisis. The tables below (

Table 6. Comparison of 12 months ahead out-of-sample $RMSE$, for the RVFL, and VAR.

Method	Min.	1st Qrt	Median	Mean	3rd Qrt	Max.
RVFL	0.1675	0.2906	0.4704	0.5452	0.6469	1.8410
VAR	0.1382	0.4025	0.6469	1.0310	1.0750	13.020

and

Table 7. 95% confidence interval around the difference of out-of-sample $RMSE$.

Method	Lower Bound	Upper Bound	Mean
RVFL-VAR	−0.2622	−0.7087	−0.4854

) contain the results of a comparison between the RVFL model and an unrestricted VAR model (with one lag, best parameter found) on the forecasting problem. The best RVFL model, with the lowest out-of-sample $RMSE$, uses one lag, four hidden nodes, and ${\lambda }_1=5.80$, ${\lambda }_2=19.66$.

3.3. Forecasting on a Longer Horizon, with a Longer Training Window

In this third example, as in Section 3.1, we apply the DNS framework to the forecasting of spot rates. However, in this example we have a longer training set (36 months), and a longer horizon for the test set (36 months as well). We use interest rate swaps data from the Federal Reserve Bank of St Louis website3, observed on a monthly basis from July 2000 to September 2016, with maturities equal to $1,2,3,4,5,7,10,30$, and a tenor equal to three months.

In Figure 7, we present three of the eight time series of swap rates, observed for time to maturities equal to $3,10$ and 30. The swap rates for different maturities generally exhibit a decreasing trend, and are nearly equal to 0 by the end of 2016, for the shortest maturities.

We also observe that the spreads between swap rates with different maturities start to narrow in 2006 until the end of 2007, and the swap rates for short term maturities are relatively high during the same period. This is the period corresponding to the Liquidity and Credit Crunch 2007–2008.

Table 8. Descriptive statistics of St Louis Federal Reserve data for 1-year, 10-years and 30-years swap rates (in %).

Maturity	Min.	1st Qrt	Median	Mean	3rd Qrt	Max.
1	0.260	0.500	1.340	2.108	3.360	7.050
10	1.390	2.610	4.190	3.881	5.020	7.240
30	1.750	3.270	4.650	4.404	5.375	7.200

presents the descriptive statistics for these three time series.

All the swap rates (for all the maturities available) were then transformed into zero rates, by using a single curve calibration methodology (that is, ignoring the counterparty credit risk) with linear interpolation between the swaps’ maturities. Then, the Nelson & Siegel model was used for fitting and forecasting the curves in a DNS framework, with both auto.arima and the RVFL model presented in this paper, applied to the three factors. In the fashion of Section 3.1. However, now we obtain 36-month’s ahead forecasts, from a rolling training windows with a fixed 36 month’s length. The average out-of-sample $RMSE$ are then calculated for each method.

The best hyperparameters—associated with the lowest out-of-sample average $RMSE$—for each model are obtained through a search on a grid of values. We have:

• DNS with ARIMA (auto.arima): $\lambda =1.4271$ (Nelson Siegel parameter).
• DNS with RVFL: number of lags for each series: 1, activation function: ReLU, number of nodes in the hidden layer: 45, ${\lambda }_1=4.6416$, ${\lambda }_2=774.2637$ (RVFL parameters) and $\lambda =24$ (Nelson Siegel parameter).

With these parameters, the results detailed in

Table 9. Descriptive statistics for out-of-sample $RMSE$, with rolling training window = 36 months, and testing window = 36 months.

Method	Min.	1st Qrt	Median	Mean	3rd Qrt	Max.	Std. Dev
ARIMA	0.0036	0.0070	0.0104	0.0149	0.0161	0.2150	0.0213
RVFL	0.0032	0.0078	0.0115	0.0120	0.0148	0.0256	0.0055

are obtained, for the out-of-sample $RMSE$. A $95\%$ confidence interval around the difference of out-of-sample $RMSE$ between ARIMA (applied to each one of the three factors) and RVFL is presented in

Table 10. 95% confidence interval around the difference of out-of-sample $RMSE$.

Method	Lower Bound	Upper Bound	Mean
RVFL-ARIMA	−0.0064	0.0007	−0.0028

.

Figure 8 presents the evolution of the out-of-sample $log(RMSE)$ over the training/testing windows. The grey rectangle indicating the Liquidity and Credit crunch is larger here, because in this example, a training set starting in 2004 has its test set starting 36 months later, in 2007. Again, we observe that the results from the RVFL model exhibit a low out-of-sample error, along with a low volatility.

Figure 9, presents the convergence of the out-of-sample $log(RMSE)$ for the DNS + RVFL model from this section, as a function of $log({\lambda }_1)$ and $log({\lambda }_2)$. ${\lambda }_1$ and ${\lambda }_2$ both range from ${10}^{-2}$ to ${10}^4$, with 10 equally-spaced points each (hence, a grid of 100 points $\left(log({\lambda }_1),log({\lambda }_2),log(RMSE)\right)$.

The number of nodes in the hidden layer is equal to 45, and the value of $\lambda$, parameter from the Nelson and Siegel (1987) model presented in Section 3.1, is fixed and equal to 24. The one hundred points $\left(log({\lambda }_1),log({\lambda }_2),log(RMSE)\right)$ that we use for Figure 9 can be found in Appendix A.3.

There is a rectangular region at the top, in the middle of the figure, where the $log(RMSE)$ is the lowest. In this region, the lowest value of the out-of-sample $log(RMSE)$ is observed for ${\lambda }_1=4.6416$ and ${\lambda }_2=464.1589$ and the out-of-sample $RMSE$ is equal to $0.01206$ (75th point in Appendix A.3).

4. Conclusions

We present a model which could be used for multiple time series forecasting, based on a single layer quasi-randomized neural network. In this model, the lags of the different time series are used as in a dynamic regression model, and include the response variable lags. An additional layer of variables is added to the regression, whose nodes are not trained but obtained from a low discrepancy sequence. It is possible to add new variables to the regression, as indicators of special events, or to stress one variable, and observe the implied effect on the others’ forecast. The model is tested on raw historical spot rates, and in a Dynamic Nelson Siegel framework. It produces robust forecast results when compared to other usual (unpenalized) models in the same framework.