Machine Learning for Multiple Yield Curve Markets: Fast Calibration in the Gaussian Affine Framework

Abstract

Calibration is a highly challenging task, in particular in multiple yield curve markets. This paper is a first attempt to study the chances and challenges of the application of machine learning techniques for this. We employ Gaussian process regression, a machine learning methodology having many similarities with extended Kálmán filtering, which has been applied many times to interest rate markets and term structure models. We find very good results for the single-curve markets and many challenges for the multi-curve markets in a Vasiček framework. The Gaussian process regression is implemented with the Adam optimizer and the non-linear conjugate gradient method, where the latter performs best. We also point towards future research.

1. Introduction

It is the aim of this paper to apply machine learning techniques to the calibration of bond prices in multi-curve markets in order to predict the term structure of basic instruments. The challenges are two-fold: On the one side, interest rate markets are characterized by having not only single instruments, such as stocks, but also full term structures, i.e., the curve of yields for different investment periods. On the other side, in multi-curve markets, not only one term structure is present but multiple yield curves for different lengths of the future investment period are given in the market and have to be calibrated. This is a very challenging task (see Eberlein et al. (2019) for an example using Lévy processes).

The co-existence of different yield curves associated to different tenors is a phenomenon in interest rate markets which originates with the 2007–2009 financial crisis. In this time, spreads between different yield curves reached their peak beyond 200 basis points. Since then, the spreads have remained on a non-negligible level. The most important curves to be considered in the current economic environment are the overnight indexed swap (OIS) rates and the interbank offered rates (abbreviated as Ibor, such as Libor rates from the London interbank market) of various tenors. In the European market, these are, respectively, the Eonia-based OIS rates and the Euribor rates. The literature on multiple yield curves is manifold and we refer to the works of Grbac and Runggaldier (2015) and Henrard (2014) for an overview. The general theory and affine models have been developed and applied, among others, in Grbac et al. (2016, 2019); Fontana et al. (2020); Grbac et al. (2015); Mercurio (2010).

The recent developments have seen many machine learning techniques, in particular deep learning became very popular. While deep learning typically needs big data, here we are more confronted with small data together with a high-dimensional prediction problem, since a full curve (the term structure), and in the multi-curve market, even multiple curves have to be calibrated and predicted. To be able to deal with this efficiently, one would like to incorporate information from the past, and a Bayesian approach seems best suited to this. We choose Gaussian process regression (GPR) as our machine learning approach which ensures fast calibration (see De Spiegeleer et al. (2018)). This is a non-parametric Bayesian approach to regression and is able to capture non-linear relationships between variables. The task of learning in Gaussian processes simplifies to determining suitable properties for the covariance and mean function, which determine the calibration of our model. We place ourselves in the context of the Vasiček model, which is a famous affine model (see the works of Filipović (2009) and Keller-Ressel et al. (2018) for a guide to the literature and details.)

It is generally accepted that machine learning performs better with higher-dimensional data. In a calibration framework such as the one we are studying here with GPR, it turns out that this seems not to be the case. While we find very good results for the single-curve markets, we find many challenges for the multi-curve markets in a Vasiček framework. A possible reason for this is that, in the small data setting considered here, the GPR is not able to exploit sparseness in the data.

Related Literature

Calibration of log-bond prices in the simple Vasiček model framework via Gaussian processes for machine learning has already been applied in Beleza Sousa et al. (2012) for a single maturity and in Beleza Sousa et al. (2014) for several maturities. They both rely on the theory of Gaussian processes presented in Rasmussen and Williams (2006). While in Section 3 we extend Beleza Sousa et al. (2012) by presenting an additional optimization method, Beleza Sousa et al. (2014) and Section 4 constitute a different access to the calibration of interest rate markets with several maturities. While Beleza Sousa et al. (2014) calibrated solely zero-coupon log-bond prices, from which one is not able to construct a post-crisis multi-curve interest rate market, we calibrate zero-coupon log-bond prices and log-$\delta$-bond prices on top in order to encompass forward rate agreements and to be conform with the multi-curve framework (cf. Fontana et al. (2020) for the notion of $\delta$-bonds). The modeling of log-$\delta$-bond prices allows building forward-rate-agreements (FRAs) so that we have the basic building blocks for multi-curve interest rate markets.

2. Gaussian Process Regression

Following Rasmussen and Williams (2006, chps. 2 and 5), we provide a brief introduction to Gaussian process regression (GPR). For a moment, consider a regression problem with additive Gaussian noise: we observe $y_1,\dots ,y_n$ together with covariates $x_1,\dots ,x_n$ and assume that

$$y_i=f(x_i,\theta )+{ϵ}_i,i=1,\dots ,n,$$

where $f(·,\theta )$ is a regression function depending on the unknown parameter $\theta$ and $ϵ$ is a d-dimensional noise vector which we assume to consist of i.i.d. mean-zero Gaussian errors and standard deviation $\widehat{\sigma }$.

In the Bayesian approach, we are not left without any knowledge on $\theta$ but may start from a prior distribution; sometimes this distribution can be deducted from previous experience in similar experiments, while otherwise one chooses an uninformative prior. Assuming continuity of the prior, we denote the prior density of $\theta$ by $p(\theta )$. Inference is now performed by computing the a posteriori distribution of $\theta$ conditional on the observation $(x,y)$. This can be achieved by Bayes’ rule, i.e.,

$$p(\theta |x,y)=\frac{p(\theta ,y|x)}{p(y|x)}=\frac{p(y|x,\theta )p(\theta )}{p(y|x)},$$

where we assume only that the distribution of $\theta$ does not depend on x. Similarly, we can compute $p(y|x)$ from $p(y|x,\theta )$ by integrating with respect to $p(\theta )$.

For a moment, we drop the dependence on x in the notation. Assuming only that the observation $y\sim \mathcal{N}({\mu }_y,{\Sigma }_{yy})$ is normally distributed, we are already able to state the marginal likelihood $p(y)$: it is given up to a normalizing constant c by

$$\begin{array}{c}\hfill logp(y)=c-\frac{1}{2}{(y-{\mu }_y)}^{\top }{\Sigma }_{yy}^{-1}(y-{\mu }_y).\end{array}$$

(1)

If we assume moreover that $\xi =f(\theta )$ is jointly normally distributed with y, we arrive at the multivariate Gaussian case. Hence, the conditional distribution $p(\xi |x,y)$ is again Gaussian and can be computed explicitly. Starting from ${(\xi ,y)}^{\top }\sim \mathcal{N}(\mu ,\Sigma )$, where we split $\mu ={({\mu }_{\xi },{\mu }_y)}^{\top }$ and

$$\Sigma =\left(\begin{array}{cc}{\Sigma }_{\xi \xi }& {\Sigma }_{\xi y}\\ {\Sigma }_{y\xi }& {\Sigma }_{yy}\end{array}\right),$$

we can compute $p(\xi |y)$ through some straightforward calculations. First, observe that $y\sim \mathcal{N}({\mu }_y,{\Sigma }_{\xi \xi }+{\widehat{\sigma }}^2{I}_n)$. We obtain ${\Sigma }_{yy}={\Sigma }_{\xi \xi }+{\widehat{\sigma }}^2{I}_n$. Hence, the marginal likelihood is given by

$$\begin{array}{c}\hfill logp(y)=\tilde{c}-\frac{1}{2}{(y-{\mu }_y)}^{\top }{\left({\Sigma }_{\xi \xi }+{\widehat{\sigma }}^2{I}_n\right)}^{-1}(y-{\mu }_y).\end{array}$$

(2)

Second, we compute

$$\begin{array}{c}\hfill \xi |y\sim \mathcal{N}\left({\mu }_{\xi }+{\Sigma }_{\xi y}{\Sigma }_{yy}^{-1}(y-{\mu }_y),{\Sigma }_{\xi \xi }-{\Sigma }_{\xi y}{\Sigma }_{yy}^{-1}{\Sigma }_{y\xi }\right).\end{array}$$

(3)

This formula is the basis for the calibration in the Vasiček model, as we show now.

3. The Single-Curve Vasiček Interest Rate Model

As a first step, we calibrate the single-curve Vasiček interest rate model following Beleza Sousa et al. (2012). To begin with, we want to mention the important difference in mathematical finance between the objective measure $\mathbb{P}$ and the risk-neutral measure $\mathbb{Q}$. The statistical propagation of all stochastic processes takes place under $\mathbb{P}$. Arbitrage-free pricing means computing prices for options; of course, the prices depend on the driving stochastic factors. The fundamental theorem of asset pricing now yields that arbitrage-free prices of traded assets can be computed by taking expectations under a risk-neutral measure $\mathbb{Q}$ of the discounted pay-offs. For a calibration, the risk-neutral measure has to be fitted to observed option prices, which is our main target.

In this sense, we consider zero-coupon bond prices under the risk-neutral measure $\mathbb{Q}$. The Vasiček model is a single-factor model driven by the short rate $r={(r_t)}_{t\ge 0}$ which is given by the solution of the stochastic differential equation

$$d{r}_t=\kappa (\theta -r_t)dt+\sigma d{W}_t,t\ge 0,$$

(4)

with initial value $r_0$. Here, W is a $\mathbb{Q}$-Brownian motion and $\kappa ,\theta ,\sigma$ are positive constants. For a positive $\kappa$, the process r converges to the long-term mean $\theta$.

The price process of the zero-coupon bond price with maturity T is denoted by ${(P(t,T))}_{0\le t\le T}$ with $P(T,T)=1$. The Vasiček model is an affine bond price model, which implies that bond prices take an exponential affine form, i.e.,

$$P(t,T)=e^{-A(T-t)-B(T-t)r_t}\mathrm{for}t\le T.$$

(5)

Here, the functions $(A(·),B(·)):[0,T]\to \mathbb{R}\times \mathbb{R}$ are given by

$$\begin{array}{cc}\hfill A(T-t)& =\frac{\theta }{\kappa }\left(e^{-\kappa (T-t)}+\kappa (T-t)-1\right)\hfill \\ \hfill & +\frac{{\sigma }^2}{4{\kappa }^3}\left(e^{-2\kappa (T-t)}-4e^{-\kappa (T-t)}-2\kappa (T-t)+3\right),\hfill \\ \hfill B(T-t)& =\frac{1}{\kappa }(1-e^{-\kappa (T-t)}).\hfill \end{array}$$

(6)

We recall that the solution of the stochastic differential Equation (4) is given by (cf. Brigo and Mercurio (2001, chp. 4.2)),

$$\begin{array}{c}\hfill r_t=r_0{e}^{-\kappa t}+\theta (1-e^{-\kappa t})+\sigma e^{-\kappa t}{\int }_0^t{e}^{\kappa u}d{W}_u.\end{array}$$

Together with Equation (5), this implies that the zero coupon bond prices are log-normally distributed. To apply the Gaussian process regression, we consider in the sequel log-bond prices for $t\le T$,

$$y(t,T)=logP(t,T)=-A(T-t)-B(T-t)r_t.$$

The corresponding mean function is given by

$$\begin{array}{cc}\hfill \mu (t,T)& :={\mathbb{E}}_{\mathbb{Q}}\left[logP(t,T)\right]=-A(T-t)-B(T-t)\left(r_0{e}^{-\kappa t}+\theta (1-e^{-\kappa t})\right),\hfill \end{array}$$

(7)

and the covariance function is given by

$$\begin{array}{cc}\hfill c(s,t,T)& :={\mathbb{E}}_{\mathbb{Q}}\left[\left(logP(s,T)-{\mathbb{E}}_{\mathbb{Q}}[logP(s,T)]\right)·\left(logP(t,T)-{\mathbb{E}}_{\mathbb{Q}}[logP(t,T)]\right)\right]\hfill \\ \hfill & =B(T-s)B(T-t)\left({\mathbb{E}}_{\mathbb{Q}}\left[r_s{r}_t\right]-{\mathbb{E}}_{\mathbb{Q}}\left[r_s\right]{\mathbb{E}}_{\mathbb{Q}}\left[r_t\right]\right)\hfill \\ \hfill & =B(T-s)B(T-t)\frac{{\sigma }^2}{2\kappa }{e}^{-\kappa (s+t)}(e^{2\kappa (s\wedge t)}-1).\hfill \end{array}$$

(8)

An observation now consists of a vector $y={(y(t_1,T),\dots ,y(t_n,T))}^{\top }+ϵ$ of log-bond prices with additional Gaussian i.i.d. noise with variance ${\widehat{\sigma }}^2$. Consequently, $y\sim \mathcal{N}({\mu }_y,{\Sigma }_{yy})$ with

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mu }_y& ={(\mu (t_1,T),\dots ,\mu (t_n,T))}^{\top }\hfill \\ \hfill {({\Sigma }_{yy})}_{i,j}& =c(t_i,t_j,T)+{\widehat{\sigma }}^2{\mathbb{1}}_{\{i=j\}}.\hfill \end{array}\end{array}$$

(9)

We define the vector of hyper-parameters as $\mathsf{\Theta }={(r_0,\kappa ,\theta ,\sigma )}^{\top }$ and aim at finding the most reasonable values of hyper-parameters, minimizing the negative log marginal likelihood defined in Equation (1) with the underlying mean covariance function from Equation (9).

Remark 1.

[Extending the observation] Note that, in the set-up considered here, the driving single factor process r is considered not observable at $t_0,t_1,\dots ,t_n$; compare Equation (9) which depends on $r_0$ only (and on Θ of course). Such an approach may be appropriate for short time intervals. The propagation of $r_0$ to the future times $t_1,\dots ,t_n$ is solely based on the risk-neutral evolution under the risk-neutral measure $\mathbb{Q}$ via μ and c.

If, on the contrary, updating of r should be incorporated, the situation gets more complicated since the evolution of r (taking place under the statistical measure $\mathbb{P}$) needs to be taken into account. One can pragmatically proceed as follows: assume that at each time point r is observable. We neglect the information contained in this observation about the hyper-parameters Θ. Then, the formulas need to be modified only slightly by using conditional expressions. For example, $\mu (t,T)$ conditional on $r_s$ for $s\le t$ equals

$${\mathbb{E}}_{\mathbb{Q}}[logP(t,T)|r_s]=-A(T-t)-B(T-t)(r_s{e}^{-\kappa (t-s)}+\theta (1-e^{-\kappa (t-s)})),$$

and, if $t=s$, this simplifies even further.

In a similar spirit, one may extend the observation by adding bond prices with different maturities or, additionally, derivatives. A classical calibration example would be the fitting of the hyper-parameters to the observation of bond prices and derivatives, both with different maturities, at a fixed time $t_0$. The Gaussian process regression hence also provides a suitable approach for this task. We refer to De Spiegeleer et al. (2018) for an extensive calibration example in this regard.

We refer to Rasmussen and Williams (2006) for other approaches to model selection such as cross validation or alignment. For our application purposes, maximizing the log-marginal likelihood is a good choice since we already have information about the choice of covariance structure, and it only remains to optimize the hyper-parameters (cf. Fischer et al. (2016)).

3.1. Prediction with Gaussian Processes Regression

The prediction of new log-bond prices given some training data is of interest in the context of the risk management of portfolios and options on zero-coupon bonds and other interest derivatives. The application to pricing can be done along the lines of Dümbgen and Rogers (2014), which we do not deepen here. Once having found the calibrated parameters ${\mathsf{\Theta }}^{\ast }$, we can apply those calibrated parameters to make predictions for new input data.

For the dynamic prediction of log-bond prices, we seek a function that is able to predict new output values given some new input values, initially based on a set of training data. For this purpose, we specify in a first step the prior distribution, expressing our prior beliefs over the function. In Figure 1, we plot for the sake of illustration one simulated log-bond price and an arbitrary chosen prior, expressing our beliefs. This could correspond to the log-prices of a one-year maturity bond, plotted over 252 days. The blue line constitutes the prior mean and the shaded area nearly twice the prior standard deviation corresponding to the 95% confidence interval (CI). Without observing any training data, we are able to neither narrow the CI of our prior nor enhance our prediction.

The next step is to incorporate a set of training data, to enhance our prediction. The Gaussian process regression has a quite fascinating property in this regard: consider an observation at the time point t, $t\in \{1,\dots ,251\}$. The prediction for $y(t)$ is of course $y(t)$ itself—and hence perfect. The prediction of $y(s)$, be it for $s<t$ (which is often called smoothing) or the prediction for $s>t$ (called filtering) is normally distributed, but, according to Equation (3), has a smaller variance compared to our initial prediction without the observation of $y(t)$. Increasing the number of observations to, e.g., $y(t_1),\dots ,y(t_n)$ improves the prediction dramatically. We illustrate this property in Figure 2. Depending on the location of s relative to $t_1,\dots ,t_n$, the prediction can become very exact (bottom right-for $s\in [0,130]$) and may still have a significant error (e.g., for s around 180).

We now formalize the question precisely: given some training data $y={(y(t_1,T),\dots ,y(t_n,T))}^{\top }$ we aim at predicting the vector $\tilde{y}={(y(s_1,T),\dots ,y(s_m,T))}^{\top }$.

The joint distribution of y and $\tilde{y}$ can directly be obtained from the mean and covariance functions in Equations (7) and (8): since they are jointly normally distributed, we obtain that

$$\begin{array}{c}\hfill \left(\begin{array}{c}y\hfill \\ \tilde{y}\hfill \end{array}\right)\sim \mathcal{N}\left(\left(\begin{array}{c}{\mu }_y\hfill \\ {\mu }_{\tilde{y}}\hfill \end{array}\right),\left(\begin{array}{cc}{\Sigma }_{yy}\hfill & {\Sigma }_{y\tilde{y}}\hfill \\ {\Sigma }_{\tilde{y}y}\hfill & {\Sigma }_{\tilde{y}\tilde{y}}\hfill \end{array}\right)\right).\end{array}$$

Recall that ${\mu }_y$ and ${\Sigma }_{yy}$ are already specified in Equation (9). Analogously, we obtain that

$$\begin{array}{cc}\hfill {({\mu }_{\tilde{y}})}_i& =\mu (s_i,T)\hfill \\ \hfill {({\Sigma }_{y\tilde{y}})}_{i,j}& =c(t_i,s_j,T)\hfill \\ \hfill {({\Sigma }_{\tilde{y}y})}_{i,j}& =c(s_i,t_j,T)\hfill \\ \hfill {({\Sigma }_{\tilde{y}\tilde{y}})}_{i,j}& =c(s_i,s_j,T).\hfill \end{array}$$

The posterior distribution is obtained by using Equation (3): conditional on the training data y, we obtain that

$$\tilde{y}|y\sim \mathcal{N}\left(\tilde{\mu },\tilde{\Sigma }\right),$$

(10)

where

$$\tilde{\mu }={\mu }_{\tilde{y}}+{\Sigma }_{\tilde{y}y}{\left({\Sigma }_{yy}\right)}^{-1}(y-{\mu }_y)$$

and

$$\tilde{\Sigma }={\Sigma }_{\tilde{y}\tilde{y}}-{\Sigma }_{\tilde{y}y}{\left({\Sigma }_{yy}\right)}^{-1}{\Sigma }_{y\tilde{y}}.$$

Given some training data, Equation (10) now allows calculating the posterior distribution, i.e., the conditional distribution of the predictions $\tilde{y}$ given the observation y, and to make predictions, derive confidence intervals, etc.

The posterior distributions, i.e., the prior conditioned on some training data of 1, 3, 100, and 102 observations, respectively, is shown in Figure 2. The red dotted line represents a simulated trajectory of the log-bond price, while the red dots constitute the observed training data. The blue line is the posterior mean, which we take as our prediction, and the shaded areas denote the 95% confidence interval (CI). We observe that the more training data we have, the more accurate our prediction gets. While in Figure 2a the CI is quite large, two additional training data narrow the CI significantly down (see Figure 2b). Of course, the observation times $t_i$ play an important role in this: if we had observations $t_i$ close to $t_{i-1}$, the additional gain in knowledge would most likely be small, while, in Figure 2b, the $(t_i)$ are nicely spread. Consequently, if one is able to choose the $(t_i)$, one can apply results from optimal experimental design to achieve a highly efficient reduction in the prediction variance.

In most practical cases, the $(t_i)$ cannot be chosen freely, and just arrive sequentially, as we illustrate in Figure 2c,d: here, we aim at predicting log-bond prices of a one-year maturity bond which we have observed daily until day $t_n=125$. Figure 2c describes the situation in which we do not have any information about future prices of the one-year maturity bond, i.e., the second half of the one year. Figure 2d depicts the situation in which we assume to know two log-bond prices or option strikes on zero-coupon bonds in the future, which enhances the prediction and narrows the CI down.

3.2. Performance Measures

To monitor the performance of the calibration or the prediction, one typically splits the observed dataset into two disjoint sets. The first set is called training set and is used to fit the model. The second is the validation set and used as a proxy for the generalization error. Given the number of observed data points and number of parameters to optimize for our simulation task, we recommend splitting the training and validation test set in a $70\%/30\%$ ratio. The quality of the predictions can be assessed with the standardized mean squared error (SMSE) loss and the mean standardized log loss (MSLL).

The SMSE considers the average of the squared residual between the mean prediction and the target of the test set and then standardizes it by the variance of the targets of the test cases.

$$\begin{array}{cc}\hfill \mathrm{SMSE}& =\frac{1}{{\tilde{\sigma }}^2}\frac{1}{m}\sum _{i=1}^m{({\tilde{y}}_i-{\tilde{\mu }}_i)}^2,\hfill \end{array}$$

where ${\tilde{\sigma }}^2$ denotes the variance of the targets of the test cases. The SMSE is a simple approach and the reader should bear in mind that this assessment of the quality does not incorporate information about the predictive variance. However, as it represents a standard measure of the quality of an estimator, it is useful to consider it for the sake of comparability with other literature.

The MSLL as defined in Rasmussen and Williams (2006) is obtained by averaging the negative log probability of the target under the model over the test set and then standardizing it by subtracting the loss that would be obtained under the trivial model with mean and variance of the training data as in Equation (1),

$$\begin{array}{cc}\hfill & \mathrm{MSLL}=-\frac{1}{m}\sum _{i=1}^{m}logp({\tilde{y}}_i|y)+logp(y).\hfill \end{array}$$

(11)

The MSLL will be around zero for simple methods and negative for better methods. Note that in Equation (11) we omit in the notation the dependence on the parameter set $\mathsf{\Theta }$, since now the parameter set is fixed. Moreover, we notice that the MSLL incorporates the predictive mean and the predictive variances unlike the SMSE.

In the following section, we simulate log-bond prices in a Vasiček single-curve model and calibrate the underlying parameters of the model. This is performed for one maturity. Since we deal with simulated prices, we consider the noise-free setting and set ${\widehat{\sigma }}^2=0$ in Equation (2).

3.3. Calibration Results for the Vasiček Single-Curve Model

We generate 1000 samples of log-bond price time series, each sample consisting of 250 consecutive prices (approximately one year of trading time) via Equations (4) and (5). As parameter set, we fix

$$r_0=0.5,\kappa =2,\theta =0.1,\sigma =0.2.$$

For each of the 1000 samples, we seek the optimal choice of hyper-parameters $\mathsf{\Theta }=\{r_0,\kappa ,\theta ,\sigma \}$ based on the simulated trajectory of 250 log-bond prices. For this purpose, we minimize the negative log marginal likelihood in Equation (1) with the underlying mean and covariance functions in Equations (7) and (8), respectively, by means of two optimization methods: the non-linear conjugate gradient (CG) algorithm and the adaptive moment estimation (Adam) optimization algorithm. The CG algorithm uses a non-linear conjugate gradient by Polak and Ribiere (1969). For details on the CG algorithm, we refer to Nocedal and Wright (2006). The Adam optimization algorithm is based on adaptive estimates of lower-order moments and is an extension of the stochastic gradient descent. For details of the Adam optimization algorithm, we refer to Kingma and Ba (2014).

After a random initialization of the parameters to optimize, the CG optimization and the Adam optimization were performed using the Python library SciPy and TensorFlow, respectively. Parallelization of the 1000 independent runs was achieved with the Python library multiprocessing. The outcomes can be summarized as follows.

• The results of the calibration via the conjugate gradient optimization algorithm are very satisfying. In Figure 3, the learned parameters $r_0$, $\kappa$, $\theta$, and $\sigma$ of 1000 simulated log-bond prices are plotted in 50-bin histograms. The red dashed lines in each sub-plot indicate the true model parameters.
  The mean and standard deviation of the learned parameters are summarized in

  Table 1. Single curve calibration results with mean and standard deviations (StDev) of the learned parameters (params.) of 1000 simulated log-bond prices as well as the true Vasiček parameters.

  	Params.	${\mathit{r}}_0$		$\mathit{\kappa }$		$\mathit{\theta }$		$\mathit{\sigma }$
  Optimizer		Mean	StDev	Mean	StDev	Mean	StDev	Mean	StDev
  CG		0.496	0.135	2.081	0.474	0.104	0.106	0.202	0.020
  Adam		0.510	0.144	2.339	0.403	0.121	0.093	0.213	0.018
  True value		0.5		2		0.1		0.2

  . We observe that the mean of the learned parameters reaches the true parameters very closely while the standard deviation is reasonable.
• The results of the calibration via the Adam algorithm are satisfying, even though we note a shift of the mean reversion parameter $\kappa$ and the volatility parameter $\sigma$. The learned parameters $r_0$, $\kappa$, $\theta$, and $\sigma$ of 1000 simulated log-bond prices are plotted in 50-bin histograms in Figure 4 and summarized with their mean and standard deviation in Table 1. The Adam algorithm slowly reduces the learning rate over time to speed up the learning algorithm. Nonetheless, one needs to specify a suitable learning rate. If the learning rate is small, training is more reliable, but the optimization time to find a minimum can increase rapidly. If the learning rate is too big, the optimizer can overshoot a minimum. We tried different learning rates of 0.0001, 0.001, 0.01, 0.05, and 0.1. Finally, we decided in favor of a learning rate of 0.05 and performed the training over 700 epochs to achieve a suitable trade-off between accuracy and computation time. We expect the results to improve slightly with more epochs, at the expense of a longer training time.

We conclude that, in the single-curve Vasiček specification, both optimization algorithms provide reliable results. The optimization by means of the CG algorithm outperforms the Adam algorithm (see Table 1). However, there is hope that the results obtained by the Adam algorithm can be improved by increasing the number of training epochs or by choosing a smaller learning rate in addition to more training epochs.

4. Multi-Curve Vasiček Interest Rate Model

The financial crisis of 2007–2009 has triggered many changes in financial markets. In the post-crisis interest rate markets, multiple yield curves are standard: due to credit and liquidity issues, we obtain for each tenure a different curve (see, for example, Grbac and Runggaldier (2015) and the rich literature referenced therein).

The detailed mechanism in the multi-curve markets is quite involved, and we refer Fontana et al. (2020) for a precise description. Intuitively, traded instruments are forward-rate agreements which exchange a fixed premium against a floating rate over a future time interval $[T,T+\delta ]$. Most notably, in addition to the parameter maturity, a second parameter appears: the tenor $\delta$. While before the crisis, the curves were independent of $\delta$, after the crisis the tenure can no longer be neglected. The authors of Fontana et al. (2020) showed that forward-rate agreements can be decomposed into $\delta$-bonds, which we denote by $P(t,T,\delta )$. If $\delta =0$ we write $P(t,T)=P(t,T,0)$, which is the single-curve case considered previously.

In the multi-curve Vasiček interest rate model, we consider two maturity time points $T\le T^{\prime }$, hence $\delta =T^{\prime }-T$. We calibrate zero-coupon bond prices $P(t,T,0)$ and tenor-$\delta$-bond prices $P(t,T,\delta )$ under the risk-neutral measure $\mathbb{Q}$ at time t for the maturity T.

The calibration in the multi-curve framework corresponds to the situation in which we are given several datasets of different bond prices, where all of them are sharing the same hyper-parameters. We call this situation multi-task learning and reveal the different meaning in contrast to (Rasmussen and Williams 2006, sct. 5.4.3) where the corresponding log-marginal likelihoods of the individual problems are summed up and the result is then optimized with respect to the hyper-parameters. As our multi-curve example exhibits correlation, the latter approach cannot be applied.

We utilize a two-dimensional driving factor process $r={(r^1,r^2)}^{\top }$, generalizing Equation (4). Modeling r as a two-dimensional Ornstein–Uhlenbeck process can be done as follows: let r be the unique solution of the SDE

$$d{r}_t^i={\kappa }_i({\theta }_i-r_t^i)dt+{\sigma }_{i}d{W}_t^i,t\ge 0,i=1,2,$$

(12)

where $W^1,W^2$ are two standard $\mathbb{Q}$-Brownian motions with correlation $\rho$. The zero-coupon bond prices only depend on $r^1$ and we utilize the single-curve affine framework:

$$P(t,T)=e^{-A(T-t)-B(T-t)r_t^1}\mathrm{for}t\le T.$$

(13)

The functions $(A(·),B(·)):[0,T]\to \mathbb{R}\times \mathbb{R}$ satisfy again the Riccati equations. They are obtained from Equation (5) by replacing $\kappa$ by ${\kappa }_1$, etc. As in Equations (7) and (8), we obtain that zero-coupon log-bond prices $y(t,T,0)=logP(t,T,0)$ are normal with mean function

$$\begin{array}{cc}\hfill \mu (t,T,0)& :={\mathbb{E}}_{\mathbb{Q}}\left[logP(t,T,0)\right]\hfill \\ \hfill & =-A(T-t)-B(T-t)\left(r_0^1{e}^{-{\kappa }_{1}t}+{\theta }_1(1-e^{-{\kappa }_{1}t})\right),t\le T\hfill \end{array}$$

(14)

For two given time points $t_i,t_j\le T$, the covariance function is

$$\begin{array}{cc}\hfill c(t_i,t_j,T,0)& :=B(T-t_i)B(T-t_j)\frac{{\sigma }_1^2}{2{\kappa }_1}{e}^{-{\kappa }_1(t_i+t_j)}\left(e^{2{\kappa }_1(t_i\wedge t_j)}-1\right).\hfill \end{array}$$

(15)

The next step is to develop the prices for the tenor-$\delta$ bonds. We assume that, while for the tenor 0 the interest rate is $r^1$, the interest rate for tenor $\delta$ is $r^1+r^2$. This implies that $P(t,T,\delta )={\mathbb{E}}_{\mathbb{Q}}\left[e^{-{\int }_t^T(r_s^1-r_s^2)ds}|{\mathcal{F}}_t\right].$ Since r is an affine process, this expectation can be computed explicitly. Using the affine machinery, we obtain that

$$\begin{array}{cc}\hfill P(t,T,\delta )& =exp\left(\mathsf{\Phi }(T-t)+\mathsf{\Psi }{(T-t)}^{\top }{r}_t\right),\mathrm{for}t\le T\hfill \end{array}$$

(16)

where the function $(\mathsf{\Phi }(·),\mathsf{\Psi }(·)):[0,T]\to \mathbb{R}\times {\mathbb{R}}^2$ satisfies the Riccati equations. This implies that

$$\begin{array}{cc}\hfill {\mathsf{\Psi }}_1(T-t)& =-B(T-t)=-\frac{1}{{\kappa }_1}\left(1-e^{-{\kappa }_1(T-t)}\right),\hfill \\ \hfill {\mathsf{\Psi }}_2(T-t)& =\frac{1}{{\kappa }_2}\left(1-e^{-{\kappa }_2(T-t)}\right)\mathrm{and}\hfill \\ \hfill \mathsf{\Phi }(T-t)& =-({\theta }_1-{\theta }_2)(T-t)-\frac{{\theta }_1}{{\kappa }_1}\left(e^{-{\kappa }_1(T-t)}-1\right)+\frac{{\theta }_2}{{\kappa }_2}\left(e^{-{\kappa }_2(T-t)}-1\right)\hfill \\ \hfill & +\frac{{\sigma }_1^2}{2{\kappa }_1^2}\left(T-t+\frac{2}{{\kappa }_1}{e}^{-{\kappa }_1(T-t)}-\frac{1}{2{\kappa }_1}{e}^{-2{\kappa }_1(T-t)}-\frac{3}{2{\kappa }_1}\right)\hfill \\ \hfill & +\frac{{\sigma }_2^2}{2{\kappa }_2^2}\left(T-t+\frac{2}{{\kappa }_2}{e}^{-{\kappa }_2(T-t)}-\frac{1}{2{\kappa }_2}{e}^{-2{\kappa }_2(T-t)}-\frac{3}{2{\kappa }_2}\right)\hfill \\ \hfill & -\frac{\rho {\sigma }_1{\sigma }_2}{{\kappa }_1{\kappa }_2}(T-t+\frac{1}{{\kappa }_1}(e^{-{\kappa }_1(T-t)}-1)+\frac{1}{{\kappa }_2}(e^{-{\kappa }_2(T-t)}-1)\hfill \\ \hfill & -\frac{1}{{\kappa }_1+{\kappa }_2}\left(e^{-({\kappa }_1+{\kappa }_2)(T-t)}-1\right)).\hfill \end{array}$$

(17)

Proposition 2.

Under the above assumptions, $logP(·,T,\delta )$, $0\le t\le T$ is a Gaussian process with mean function

$$\begin{array}{cc}\hfill \mu (t,T,\delta )& =\mathsf{\Phi }(T-t)+{\mathsf{\Psi }}_1(T-t)(r_0^1{e}^{-{\kappa }_{1}t}+{\theta }_1(1-e^{-{\kappa }_{1}t}))\hfill \\ \hfill & +{\mathsf{\Psi }}_2(T-t)(r_0^2{e}^{-{\kappa }_{2}t}+{\theta }_2(1-e^{-{\kappa }_{2}t})),t\le T,\hfill \end{array}$$

and with covariance function

$$\begin{array}{cc}\hfill c(s,t,T,\delta )& ={\mathsf{\Psi }}_1(T-s){\mathsf{\Psi }}_1(T-t)\frac{{\sigma }_1^2}{2{\kappa }_1}{e}^{-{\kappa }_1(s+t)}(e^{2{\kappa }_1(s\wedge t)}-1)\hfill \\ \hfill & +{\mathsf{\Psi }}_1(T-s){\mathsf{\Psi }}_2(T-t)\frac{\rho {\sigma }_1{\sigma }_2}{{\kappa }_1+{\kappa }_2}{e}^{-({\kappa }_{1}s+{\kappa }_{2}t)}(e^{({\kappa }_1+{\kappa }_2)(s\wedge t)}-1)\hfill \\ \hfill & +{\mathsf{\Psi }}_1(T-t){\mathsf{\Psi }}_2(T-s)\frac{\rho {\sigma }_1{\sigma }_2}{{\kappa }_1+{\kappa }_2}{e}^{-({\kappa }_{1}t+{\kappa }_{2}s)}(e^{({\kappa }_1+{\kappa }_2)(t\wedge s)}-1)\hfill \\ \hfill & +{\mathsf{\Psi }}_2(T-s){\mathsf{\Psi }}_2(T-t)\frac{{\sigma }_2^2}{2{\kappa }_2}{e}^{-{\kappa }_2(s+t)}(e^{2{\kappa }_2(s\wedge t)}-1),0\le s,t\le T.\hfill \end{array}$$

Note that $\mu$ and c in the above proposition do not depend on $\delta$. We rather use $\delta$ as index to distinguish mean and covariance functions for the zero-coupon bonds and the $\delta$-tenor bonds.

The proof is relegated to Appendix A. The next step is to phrase observation and prediction in the Gaussian setting explicitly. We denote $y={(y^0,y^{\delta })}^{\top }$ where $y^0=(y(t_1,T,0),\dots ,y(t_n,T,0))$ and $y^{\delta }=(y(t_1,T,\delta ),\dots ,y(t_n,T,\delta ))$—of course, at each time point $t_i$ we observe two bond prices now: $P(t_i,T,0)$ and $P(t_i,T,\delta )$. The vector y is normally distributed, $y\sim \mathcal{N}({\mu }_y,{\Sigma }_y)$ with

$$\begin{array}{c}\hfill {\mu }_y:=\left(\begin{array}{c}\mu (·,T,0)\hfill \\ \mu (·,T,\delta )\hfill \end{array}\right),{\Sigma }_y:=\left(\begin{array}{cc}{\Sigma }_y^{00}\hfill & {\Sigma }_y^{0\delta }\hfill \\ {\Sigma }_y^{\delta 0}\hfill & {\Sigma }_y^{\delta \delta }\hfill \end{array}\right).\end{array}$$

(18)

We calculate these parameters explicitly and, analogously to the single-curve set-up, the calibration methodology with Gaussian process regression follows.

To begin with, note that ${\Sigma }_y^{00}$ coincides with ${\Sigma }_{yy}$ from Equation (9), when we replace the parameters $r_0,\kappa ,\theta ,\sigma$ by $r_0^1,{\kappa }^1,{\theta }^1,{\sigma }^1$, respectively. The next step for computing the covariance matrix is to compute

$$\begin{array}{cc}\hfill {({\Sigma }_y^{0\delta })}_{i,j}& :={\mathbb{E}}_{\mathbb{Q}}\left[\left(logP(t_i,T,0)-{\mathbb{E}}_{\mathbb{Q}}\left[logP(t_i,T,0)\right]\right)·\left(logP(t_j,T,\delta )-{\mathbb{E}}_{\mathbb{Q}}\left[logP(t_j,T,\delta )\right]\right)\right]\hfill \\ \hfill & ={\mathsf{\Psi }}_1(T-t_i){\mathsf{\Psi }}_1(T-t_j)\frac{{\sigma }_1^2}{2{\kappa }_1}{e}^{-{\kappa }_1(t_i+t_j)}(e^{2{\kappa }_1(t_i\wedge t_j)}-1)\hfill \\ \hfill & +{\mathsf{\Psi }}_1(T-t_i){\mathsf{\Psi }}_2(T-t_j)\frac{\rho {\sigma }_1{\sigma }_2}{{\kappa }_1+{\kappa }_2}{e}^{-({\kappa }_1{t}_i+{\kappa }_2{t}_j)}(e^{({\kappa }_1+{\kappa }_2)(t_i\wedge t_j)}-1),\hfill \end{array}$$

and, analogously,

$$\begin{array}{cc}\hfill {({\Sigma }_y^{\delta 0})}_{i,j}& :={\mathbb{E}}_{\mathbb{Q}}\left[\left(logP(t_i,T,\delta )-{\mathbb{E}}_{\mathbb{Q}}\left[logP(t_i,T,\delta )\right]\right)·\left(logP(t_j,T,0)-{\mathbb{E}}_{\mathbb{Q}}\left[logP(t_j,T,0)\right]\right)\right]\hfill \\ \hfill & ={\mathsf{\Psi }}_1(T-t_i){\mathsf{\Psi }}_1(T-t_j)\frac{{\sigma }_1^2}{2{\kappa }_1}{e}^{-{\kappa }_1(t_i+t_j)}(e^{2{\kappa }_1(t_i\wedge t_j)}-1)\hfill \\ \hfill & +{\mathsf{\Psi }}_1(T-t_j){\mathsf{\Psi }}_2(T-t_i)\frac{\rho {\sigma }_1{\sigma }_2}{{\kappa }_1+{\kappa }_2}{e}^{-({\kappa }_1{t}_j+{\kappa }_2{t}_i)}(e^{({\kappa }_1+{\kappa }_2)(t_i\wedge t_j)}-1);\hfill \end{array}$$

in a similar way to ${\Sigma }_y^{00}$, we obtain ${\Sigma }_y^{\delta \delta }$.

In the multi-curve calibration, we aim at minimizing the log marginal likelihood in Equation (1) with the corresponding mean function and covariance matrix in Equation (18).

Calibration Results

For the calibration in the multi-curve setting, we have to consider log-bond prices and the logarithm of tenor-$\delta$ bond prices. To be specific, we generate 1000 sequences of log-bond prices and 1000 sequences of log tenor-$\delta$ bond prices with maturity 1 by means of Equations (13), (16), and (17). We choose as parameters

$$\begin{array}{cc}\hfill r_0^1& =0.5,{\kappa }_1=2,{\theta }_1=0.1,{\sigma }_1=0.2\hfill \\ \hfill r_0^2& =0.7,{\kappa }_2=0.5,{\theta }_2=0.03,{\sigma }_2=0.8.\hfill \end{array}$$

For each sequence, we generate 125 training data points of log-bond prices and 125 training data points of log tenor-$\delta$ bond prices. This corresponds to a computational effort similar to the single-curve specifications, since the underlying covariance matrix will be $(250\times 250)$-dimensional. Based on the simulated prices. we aim at finding the optimal choice of hyper-parameters

$$\mathsf{\Theta }=\{r_0^1,{\kappa }_1,{\theta }_1,{\sigma }_1,r_0^2,{\kappa }_2,{\theta }_2,{\sigma }_2\}.$$

For this purpose, we apply the CG optimization algorithm and the Adam optimization algorithm. After a random initialization of the parameters to optimize, we perform the calibration procedure using SciPy and TensorFlow for the CG and Adam optimizer, respectively. Parallelization of the 1000 independent runs is achieved with the library multiprocessing. The outcomes are as follows.

• Considering the results of the calibration by means of the CG algorithm, we note several facts. In this regard, the mean and the standard deviation of the calibrated parameters can be found in

  Table 2. Calibration results with mean and standard deviations (StDev) of the learned parameters (params.) of 1000 simulated log-bond prices and log tenor-$\delta$ bond prices, as well as the true Vasiček parameters. We note that the CG optimizer yields for every parameter a higher standard deviation than the Adam algorithm. This can in particular be observed for the parameter ${\theta }_2$.

  	Params.	${\mathit{r}}_0^1$		${\mathit{\kappa }}_1$		${\mathit{\theta }}_1$		${\mathit{\sigma }}_1$
  Optimizer		Mean	StDev	Mean	StDev	Mean	StDev	Mean	StDev
  CG		0.477	0.119	1.994	0.781	0.101	0.121	0.150	0.033
  Adam		0.487	0.053	2.694	0.284	0.157	0.029	0.170	0.013
  True value		0.5		2		0.1		0.2
  	Params.	${\mathit{r}}_0^2$		${\mathit{\kappa }}_2$		${\mathit{\theta }}_2$		${\mathit{\sigma }}_2$
  Optimizer		Mean	StDev	Mean	StDev	Mean	StDev	Mean	StDev
  CG		0.678	0.601	0.529	0.545	0.385	1.732	0.602	0.113
  Adam		0.477	0.139	0.813	0.277	0.876	0.254	0.640	0.055
  True value		0.7		0.5		0.03		0.8

  . Figure 5 shows the learned parameters of the processes $r^1$ and $r^2$ in 50-bin histograms for 1000 simulations. The red dashed line in each of the sub-plots indicates the true model parameter value. First, except for the long term mean, the volatility for $r^2$ is higher than the one for $r^1$. This implies more difficulties in the estimation of the parameters for $r^2$, which is clearly visible in the results. For $r^1$, we are able to estimate the parameters well (in the mean), with the most difficulty in the estimation of ${\kappa }^1$ which shows a high standard deviation. For estimating the parameters of $r^2$, we face more difficulties, as expected. The standard deviation of ${\theta }_2$ is very high—it is known from filtering theory and statistics that the mean is difficult to estimate, which is reflected here. Similarly, it seems difficult to estimate the speed of reversion parameter ${\kappa }_2$ and we observe a peak around 0.02 in ${\kappa }_2$. This might be due to a local minimum, where the optimizer gets stuck.
• For the calibration results by means of the Adam algorithm, we note the following. The learned parameters are illustrated in a 50-bin histogram (see Figure 6), and mean and standard deviation of each parameter are stated in Table 2. After trying several learning rates of 0.0001, 0.001, 0.01, 0.05, and 0.1, we decided in favor of the learning rate 0.05 and chose 750 training epochs. While most mean values of the learned parameters are not as close to the true values as the learned parameters from the CG algorithm, we notice that the standard deviation of the learned parameters is smaller compared to the standard deviation of the learned parameters from the CG. Especially, comparing the values of ${\theta }_2$ in Figure 5 and Figure 6, we observe the different range of calibrated values.

5. Conclusions

Based on the simulation results, we can state that the calibration in the multi-curve framework constitutes a more challenging task compared to the calibration in the single-curve framework, since we need to find eight parameters instead of four parameters. In particular, to keep the computational complexity similar to the single-curve framework, we chose 125 training data time points, which results in a covariance matrix of the same dimension as in the single-curve setting. We are confident that doubling the training input data points would improve the results at the expense of computation time.

It would also be interesting to analyze how other estimators perform in comparison to the shown results, for example classical maximum-likelihood estimators. Since it is already known that it is difficult to estimate the variance with ML-techniques, it could also be very interesting to mix classical approaches with ML approaches, which we leave for future work.

A first step to extend the presented approach could consist in studying further optimization techniques such as the adaptive gradient algorithm (AdaGrad) or its extension Adadelta, the root mean square propagation (RMSProp), and the Nesterov accelerated gradient (NAG); one could further investigate the prediction of log-bond prices with the learned parameters in order to obtain decision-making support for the purchase of options on zero-coupon bonds and use the underlying strike prices as worst-case scenarios. A next step could be the development of further short rate models such as the Cox–Ingersoll–Ross, the Hull–White extended Vasiček, or the Cox–Ingersoll–Ross framework. An interesting application is the calibration of interest rate models including jumps. Beyond that, the study of the calibration of interest rate markets by means of Bayesian neural networks seems very promising and remains to be addressed in future work.

The Gaussian process regression approach naturally comes with a posteriori distribution which contains much more information compared to the simple prediction (which contains only the mean). It seems to be highly interesting to utilize this for assessing the model risk of the calibration and compare it to the non-linear approaches recently developed in Fadina et al. (2019) and in Hölzermann (2020).

Summarizing, the calibration of multiple yield curves is a difficult task and we hope to stimulate future research with this initial study showing promising results on the one side and many future challenges on the other side.