Multifidelity Model Calibration in Structural Dynamics Using Stochastic Variational Inference on Manifolds

Abstract

Bayesian techniques for engineering problems, which rely on Gaussian process (GP) regression, are known for their ability to quantify epistemic and aleatory uncertainties and for being data efficient. The mathematical elegance of applying these methods usually comes at a high computational cost when compared to deterministic and empirical Bayesian methods. Furthermore, using these methods becomes practically infeasible in scenarios characterized by a large number of inputs and thousands of training data. The focus of this work is on enhancing Gaussian process based metamodeling and model calibration tasks, when the size of the training datasets is significantly large. To achieve this goal, we employ a stochastic variational inference algorithm that enables rapid statistical learning of the calibration parameters and hyperparameter tuning, while retaining the rigor of Bayesian inference. The numerical performance of the algorithm is demonstrated on multiple metamodeling and model calibration problems with thousands of training data.

1. Introduction

Modern engineering tasks are often characterized by the need to perform large scale expensive laboratory experiments or amortize hours of computation, performing simulations that are based on sophisticated mathematical formulations. While these “high-fidelity” sources of information provide detailed insight into the complex physical process, one usually faces a heavy computational runtime or a massive financial investment. In addition to this, obtaining data by running experiments or simulations needs more advanced insight, that might not always be extricated from the data by applying state-of-the-art methods used to build data-driven metamodels [1]. Finally, with the advent of Industry 4.0 [2], developing digital twins, which are commonly probabilistic surrogate models representing the underlying physical process, is becoming a routine practice across the industry. In a realistic scenario, the paucity of data and noise in the recorded measurements are challenges that also need to be taken into account.

Surrogate modeling methods that have shown promise in dealing with problems of the aforementioned kind, typically include Gaussian process (GP) regression [3,4,5], probabilistic deep neural networks [6,7,8] or polynomial chaos expansions [9,10,11]. The application of these methods has been extended to problems from different domains, such as manufacturing [12,13], flow through porous media [10,14], and combustion mechanics [15]. Classic formulations of these methods provide a meaningful representation of a model from uncertainty and noise, and they demonstrate strong predictive performance on unseen data. However, these approaches are susceptible to challenges such as limited training data, multiple sources of information that model the same process, and the lack of identifiability of model parameters [16]. By referring to the model at-hand that meets the accuracy required by the current application as a “high-fidelity” model, one standard approach in the literature is to employ similar “low-fidelity” models whose characteristic is that they provide a lower accuracy on the model output, though they are cheaper to evaluate. A detailed review on multifidelity approaches, that is, approaches that employ more than one models to approximate the same physical process with different levels of accuracy, can be found in [17]. In brief, the most common ways in which the low accuracy–low computational cost is achieved are by simplified physics models (coarse-grid PDE solver) [18], reduced-order models [19], data-fit interpolation models [20,21] and machine learning and mathematical surrogates [22,23].

In this work, our focus is on applying GP regression to problems that have thousands of data [24]. Secondly, we focus on the use of GP regression in both the single-fidelity (where only a high-fidelity model is considered) and the multifidelity (both a low- and a high-fidelity model are employed) modeling scenarios. In the second scenario, we focus on the case where data from two sources of varying fidelity are available, and the task involves calibrating the so-called tuners of the low-fidelity source. In all these tasks, we resort to a fully Bayesian formulation of the GP regression, differentiating ourselves from the works of [25,26,27], the details of which are discussed in Ghosh et al. [28]. This is a critical aspect of this work, as retaining a fully Bayesian treatment for the metamodeling and model calibration tasks with GPs is a major challenge from a computational and numerical perspective. In some of the authors’ previous work (see Pandita et al. [29]), it was demonstrated how savings in computational time could be achieved using adaptive sequential Monte Carlo methods fused with a fully Bayesian treatment, applied to tasks of the above kind. However, the utilization of hundreds of computational processing units or cores is not always practically possible, necessitating the need for alternative approaches. Other adaptive algorithms that accelerate Markov chain Monte Carlo methods for Bayesian inference [30,31,32] and optimal-transport-based approaches that circumvent the need for MCMC methods [33] have shown promise in recent years.

Most of the above-mentioned works rely on computational power and heavy use of large-scale computing in order to overcome the challenges of training the models. Our main contribution in this work is to achieve computational efficiency by leveraging a variational formulation of Bayesian inference, commonly known as black-box variational inference (BBVI) [34], and by improving the performance of the optimization scheme involved using efficient subsampling, rather than resorting to online access to exorbitant computational resources.

Variational methods [35,36] to Bayesian inference have shown promise in various tasks that resort to a Bayesian formalism in order to train surrogate models [37,38], calibrate physical models [39], and more recently across a swathe of deep learning tasks [40,41,42]. The key ingredient in variational inference (VI) that enables efficient posterior density exploration conditioned on large quantities of data is to perform the required likelihood function evaluations using random batch-sampling. Introducing this additional level of stochasticity in the algorithm, resulting in what is known as stochastic variational inference (SVI) [43], allows for fast likelihood evaluations during the optimization procedure and scales the algorithm, while a full exploration of the available training dataset is still guaranteed. SVI has been previously successfully applied for training deep GP models [44] and sparse GPs in big data scenarios [45]. In this work, we apply SVI to train hybrid Gaussian process models that make use of training data stemming from multiple levels of fidelity, while at the same time they can incorporate calibration parameters. Specifically, we adopt the well-known Kennedy–O’Hagan formulation [46] that relies on an autoregressive GP scheme, and we develop a training algorithm that scales BBVI for big data problems using batch-sampling. We identify the optimal Gaussian approximations to the true posterior densities of the model’s hyperparameters by solving the variational problem with respect to full covariance matrices, thus capturing all correlations between the parameters. To achieve this, we make use of a manifold gradient ascent algorithm that performs the optimization directly on the manifold of symmetric positive semidefinite matrices, as opposed to solving complex constrained optimization problems.

The outline of the paper is as follows: We present the mathematical details of the autoregressive multifidelity calibration model in Section 2. In Section 3 and Section 4, we expand on the details of the black-box variational inference and its use in scaling up for big data problems, and we introduce the manifold gradient ascent optimization scheme, to be used for carrying out the optimization task. To illustrate the direct applicability of the proposed approach on calibrating models using data from sources of varying fidelity, we use a set of synthetic functions in Section 5.1. We demonstrate the impact of the extended variational formulation on a benchmark machine learning dataset with thousands of training data, in Section 5.2. In Section 5.3, we highlight the impact of the proposed formulation on a challenging multifidelity problem, in the high-sample regime with over ten thousand training data, where the parameters of interest include the uncertain tuners of the low-fidelity simulation model. We summarize our conclusions and directions for future work in Section 6.

2. Multifidelity Gaussian Process Modeling and Calibration

2.1. Autoregressive Gaussian Processes

We consider the Kennedy and O’Hagan’s formulation [47], where two simulators are available, namely $y_h\left(\mathbf{x}\right),y_l(\mathbf{x},\theta )$, where $y_h$ represents some high-fidelity computer code and $y_l(\mathbf{x},\mathit{\theta })$ represents a low-fidelity simulation code. The design variable $\mathbf{x}$ is assumed to take values within a space of feasible designs $\mathcal{X}\subset {\mathbb{R}}^D$, while $\mathit{\theta }$ is a set of calibration parameters that characterize the low-fidelity simulator.

The relationship between the two codes is assumed to be

$$y_h(\mathbf{x},\theta )=\rho y_l(\mathbf{x},\theta )+\delta \left(\mathbf{x}\right)+ϵ\left(\mathbf{x}\right),$$

(1)

where $\delta \left(\mathbf{x}\right)$ is a discrepancy term that is statistically independent of $y_l(\mathbf{x},\theta )$ and $ϵ\left(\mathbf{x}\right)$ accounts for measurement noise and is independent of both $y_l(\mathbf{x},\theta )$ and $\delta \left(\mathbf{x}\right)$. The coefficient $\rho$ satisfies

$$\rho =\frac{\mathrm{cov}[y_h(\mathbf{x},\theta ),y_l(\mathbf{x},\theta )]}{\mathrm{var}\left[y_l(\mathbf{x},\theta )\right]}$$

(2)

and therefore accounts for the correlation between the models. Although in general, $\rho$ can be considered a function of $\mathbf{x}$ [48,49], we assume for simplicity that it is constant throughout this work. Further, we take $y_l(\mathbf{x},\theta )$, $\delta \left(\mathbf{x}\right)$ to be Gaussian processes with zero mean and variances ${\sigma }_l^2{r}_l(\mathbf{x},{\mathbf{x}}^{\prime })$ and ${\sigma }_{\delta }^2{r}_{\delta }(\mathbf{x},{\mathbf{x}}^{\prime })$, respectively, where $r_l$ and $r_{\delta }$ are correlation kernels, here to be taken as squared exponential functions

$$r_t(\mathbf{x},{\mathbf{x}}^{\prime })=exp\left[-\sum _{i=1}^D\frac{{(x_i-x_i^{\prime })}^2}{{\ell }_{i,t}^2}\right],t=l,\delta ,$$

(3)

with ${\ell }_{i,t}$ being the correlation length or length scale along dimension i, for the two kernels ($t=l,\delta$).

The framework defined above may suffer from issues that pertain to recovering the correct solutions for the parameters being calibrated, also known as identifiability issues. These drawbacks are known in the literature and have been discussed in various works [50,51,52]. In this work, we limit our focus on improving the computational efficiency in a fully Bayesian formulation, while acknowledging this characteristic of the multifidelity framework.

2.2. Posterior Distribution

Assume a set of observations are available, namely, ${\mathcal{D}}_l={\{{\mathbf{x}}_i,{\theta }_i,y_i\}}_{i=1}^{N_l}$ and ${\mathcal{D}}_h={\{{\mathbf{x}}_i,y_i\}}_{i=1}^{N_h}$ are the input to output sets of points corresponding to the low- and high-fidelity simulators, respectively. Conditioning the distribution of $y_h({\mathbf{x}}^{*},\theta )$ evaluated at some test point ${\mathbf{x}}^{*}$ on the available data $\mathcal{D}:={\mathcal{D}}_l\cup {\mathcal{D}}_h$ and taking into account the prior choices and the independence between $y_l(·)$ and $\delta (·)$, we can write the posterior density as a Gaussian process with mean and variance given by [46]

$${\mu }_{y_h}({\mathbf{x}}^{*},\theta )=t_h({\mathbf{x}}^{*},\theta )V_h^{-1}\mathbf{y}$$

(4)

and

$${\sigma }_{y_h}^2({\mathbf{x}}^{*},\theta )={\sigma }_h^2\left({\mathbf{x}}^{*}\right)-t_h({\mathbf{x}}^{*},\theta )V_h^{-1}{t}_h({\mathbf{x}}^{*},\theta ).$$

(5)

In the above expressions, we use $\mathbf{y}={({\mathbf{y}}_l^T,{\mathbf{y}}_h^T)}^T$,

$$\begin{array}{c}\hfill V_h\left(\theta \right)=\left[\begin{array}{cc}{V}^{(l,l)}& V^{(l,h)}\left(\theta \right)\\ V^{(h,l)}\left(\theta \right)& V^{(h,h)}\left(\theta \right)\end{array}\right]\end{array}$$

(6)

where the diagonal block matrices are given by

$$\begin{array}{c}\hfill \begin{array}{ccc}{V}^{(l,l)}\hfill & =& {\sigma }_l^2\left(R_l\left({\mathcal{D}}_l\right)+{\sigma }_{{ϵ}_l}^{2}I\right),\hfill \\ V^{(h,h)}\left(\theta \right)\hfill & =& {\sigma }_{\delta }^2\left(R_{\delta }\left({\mathcal{D}}_h\right)+{\sigma }_{{ϵ}_h}^{2}I\right)+{\sigma }_l^2{\rho }^2\left(R_l\left({\mathcal{D}}_h\left(\mathit{\theta }\right)\right)+{\sigma }_{{ϵ}_l}^{2}I\right),\hfill \end{array}\end{array}$$

(7)

and $R_t\left({\mathcal{D}}_t\right)$ is the correlation matrix with entries $r_t(\mathbf{x},{\mathbf{x}}^{\prime })$ for $\mathbf{x}$, ${\mathbf{x}}^{\prime }\in {\mathcal{D}}_t$, $t=l,\delta$. In the above, ${\mathcal{D}}_h\left(\theta \right):={\left\{({\mathbf{x}}_i,\theta )\right\}}_{i=1}^{N_h}$ for ${\mathbf{x}}_i\in {\mathcal{D}}_h$. The off-diagonal blocks are written as

$$V^{(l,h)}\left(\theta \right)=\rho V^{(l,l)}({\mathcal{D}}_l,{\mathcal{D}}_h\left(\theta \right)).$$

(8)

At last, we define the vector

$$\begin{array}{c}\hfill t_h({\mathbf{x}}^{*},\theta )=\left(\begin{array}{c}\rho {\sigma }_l^2{R}_l(({\mathbf{x}}^{*},\theta ),{\mathcal{D}}_l)\\ {\rho }^2{\sigma }_l^2{R}_l(({\mathbf{x}}^{*},\theta ),{\mathcal{D}}_l)+{\sigma }_{\delta }^2{R}_{\delta }({\mathbf{x}}^{*},{\mathcal{D}}_h)\end{array}\right).\end{array}$$

(9)

3. Variational Inference

Throughout this section we present the main ingredients of the variational inference framework for the purpose of training Gaussian process models by means of exploring a Bayesian posterior density. The target distribution in our case is the posterior distribution of the Gaussian process hyperparameters $\mathit{\omega }$, defined as the set of length scales ${\ell }_{i,t}$, $t=l,h$ along each dimension of $\mathcal{X}$, the variance parameters ${\sigma }_l^2$, ${\sigma }_h^2$, ${\sigma }_{{ϵ}_t}^2$, $t,h$, and the calibration parameters $\mathit{\theta }$. This posterior density is conditioned on the training data $\mathcal{D}$, which in general consist of the high- and low-fidelity input and output observations. From Bayes’ rule

$$p\left(\mathit{\omega }\right|\mathcal{D})=\frac{p\left(\mathcal{D}\right|\mathit{\omega }\left)p\right(\mathit{\omega })}{p\left(\mathcal{D}\right)}$$

(10)

the posterior density is known as a function of the likelihood term and the prior density, up to a proportionality constant. Variational inference [53,54] bypasses the challenge of sampling from the posterior, by approximating it by an element $q\left(\mathit{\omega }\right)$ chosen from a parametric family of distributions $\mathcal{Q}=\left\{q\right(\mathit{\omega }\left|\lambda \right):\lambda \in \Lambda \}$, where $\Lambda$ is some set that determines the parameterization of the densities in $\mathcal{Q}$. The criterion for choosing the optimal density from the family is minimizing the Kullback–Leibler (KL) divergence between the candidate and the target densities. We define the KL divergence between the candidate and target densities as follows:

$$\mathrm{KL}\left[q\left(\mathit{\omega }\right|\lambda )\right|\left|p\left(\mathit{\omega }\right)\right]=\int q\left(\mathit{\omega }\right|\lambda )log\left(\frac{q\left(\mathit{\omega }\right|\lambda )}{p\left(\mathit{\omega }\right|\mathcal{D})}\right)d\mathit{\omega }.$$

(11)

Several techniques for solving the optimization problem exist in the literature [35] such as mean-field VI [55] or nonparametric VI [39], and they are typically tailored to problem-specific choices of prior densities, approximating family of distributions, and the inference problem under investigation.

One common characteristic of the approaches mentioned above is that they all transform the problem of minimizing the KL divergence to an equivalent maximization problem by substituting (10) into (11) to obtain

$$logp\left(\mathcal{D}\right)=\mathrm{KL}\left[q\right(\mathit{\omega }\left|\lambda \right)\left|\right|p\left(\mathit{\omega }\right)]+\mathcal{F}[q],$$

(12)

where

$$\mathcal{F}\left[q\right]=\mathcal{H}\left[q\right]+\int q\left(\mathit{\omega }\right|\lambda )log\left(p(\mathcal{D},\mathit{\omega })\right)d\mathit{\omega }$$

(13)

and $\mathcal{H}\left[q\right]$ is the entropy of $q\left(\mathit{\omega }\right|\lambda )$. Since the left-hand side of (12) is constant, we can conclude that the variational solution can be obtained by maximizing $\mathcal{F}\left[q\right]$, which is referred to as the evidence lower bound (ELBO).

Black-Box Variational Inference

One of the most popular choices for optimizing (13) is to directly employ a stochastic gradient descent or ascent algorithm, after observing that the objective function can be written as an expectation

$$\mathcal{F}\left[q\right]={\mathbf{E}}_q[logp(\mathcal{D},\mathit{\theta })-logq\left(\mathit{\omega }\right|\lambda )],$$

(14)

where the expectation is taken with respect to $q\left(\mathit{\omega }\right|\lambda )$. The gradient of this expression with respect to the parameters $\lambda$ that we seek to optimize is

$${\nabla }_{\lambda }\mathcal{F}\left[q\right]={\mathbf{E}}_q\left[{\nabla }_{\lambda }logq\left(\mathit{\omega }\right|\lambda )\left(logp(\mathcal{D},\mathit{\omega })-logq\left(\mathit{\omega }\right|\lambda )\right)\right],$$

(15)

where the gradient ${\nabla }_{\lambda }logq\left(\mathit{\omega }\right|\lambda )$ is known as the score function for any probability density q and the joint density can be expanded using Bayes’ rule to $p(\mathcal{D},\mathit{\omega })=p\left(\mathcal{D}\right|\mathit{\omega }\left)p\right(\mathit{\omega })$. A Monte Carlo estimator of (15) can be written as

$$\widehat{{\nabla }_{\lambda }\mathcal{F}}\left[q\right]=\frac{1}{N}\sum _{i=1}^N{\nabla }_{\lambda }logq\left({\mathit{\omega }}^i\right|\lambda )\left(logp(\mathcal{D},{\mathit{\omega }}^i)-logq\left({\mathit{\omega }}^i\right|\lambda )\right),$$

(16)

where ${\mathit{\omega }}^i\sim q\left(\mathit{\omega }\right|\lambda )$. Note that in the above expression, the gradient appears only on the score function, and can, in general, be computed analytically for certain families of distributions. On the contrary, the log-joint term $logp(\mathcal{D},\mathit{\omega })$ which depends on the Bayesian model under investigation, needs not be differentiated. The gradient expression does not make any further assumptions and applies generically on every Bayesian inference problem, justifying the term coined to this approach as black-box variational inference [34].

To further scale the algorithm, we perform the log-joint function evaluations $p(\mathcal{D},{\mathit{\omega }}^i)=p\left(\mathcal{D}\right|{\mathit{\omega }}^i)p\left({\mathit{\omega }}^i\right)$ using batch sampling throughout the available dataset $\mathcal{D}$, where each time, a random subset of the dataset is used to form the likelihood term. To put things in a realistic multifidelity context, it is highly unlikely that a big data problem will consist of a large number of high-fidelity observations. Therefore, in this work, we consider the following scenario where the number of training data points in ${\mathcal{D}}_l$ is significantly larger that the number of high-fidelity observations ${\mathcal{D}}_h$, that is $|{\mathcal{D}}_l|\gg |{\mathcal{D}}_h|$, thus, the batch sampling approach is applied only on ${\mathcal{D}}_l$. At every evaluation of Equation (17), let ${\mathcal{D}}_l^i$ be a random subset of ${\mathcal{D}}_l$ and ${\mathcal{D}}^i={\mathcal{D}}_l^i\cup {\mathcal{D}}_h$, then Equation (17) is rewritten as follows:

$$\widehat{{\nabla }_{\lambda }\mathcal{F}}\left[q\right]=\frac{1}{N}\sum _{i=1}^N{\nabla }_{\lambda }logq\left({\mathit{\omega }}^i\right|\lambda )\left(logp({\mathcal{D}}^i,{\mathit{\omega }}^i)-logq\left({\mathit{\omega }}^i\right|\lambda )\right),$$

(17)

where ${\mathcal{D}}_l$ is subsampled N times, that is, the number of Monte Carlo samples used to estimate $\widehat{{\nabla }_{\lambda }\mathcal{F}}\left[q\right]$. This scaling approaching was previously introduced in the literature as stochastic variational inference (SVI) [43].

4. Stochastic Optimization

Manifold Gradient Ascent

For the case where the approximating family of distributions $\mathcal{Q}$ consists of multivariate Gaussian densities, that is, $\mathcal{Q}:=\left\{q\right(\mathit{\omega }\left|\lambda \right):=\mathcal{N}\left(\mathit{\omega }\right|\mu ,\Sigma \left)\right\}$, a suitable optimization scheme needs to be employed over the parameters $\lambda =(\mu ,\Sigma )$ such that the symmetric positive semidefiniteness property of the covariance matrix is not violated. Here, we employ a stochastic optimization scheme that is tailored particularly to our problem. The scheme applies a momentum algorithm for updating $\mu$ while performing the $\Sigma$ update using a manifold gradient ascent step. For such a case, we make use of the natural gradient [56] as it is known to be invariant under parameterization [57].

The natural gradient on Riemannian manifolds is defined as

$${\nabla }_{\lambda }^{nat}\mathcal{F}\left[q\right]=I_F^{-1}{\nabla }_{\lambda }\mathcal{F}\left[q\right]$$

(18)

where ${\nabla }_{\lambda }\mathcal{F}\left[q\right]$ is the regular gradient and $I_F$ is the Fisher information for density q that is defined as

$$I_F\left(\lambda \right)={\mathbf{E}}_q\left[{\nabla }_{\lambda }logq\left(\mathit{\omega }\right|\lambda ){\left({\nabla }_{\lambda }logq\left(\mathit{\omega }\right|\lambda )\right)}^T\right].$$

(19)

In the Gaussian distribution case, the Fisher information matrix becomes

$$\begin{array}{c}\hfill I_F(\mu ,\Sigma )=\left(\begin{array}{cc}{\Sigma }^{-1}& 0\\ 0& I_F(\Sigma )\end{array}\right),\end{array}$$

(20)

where the elements of $I_F(\Sigma )$ are ${\left(I_F(\Sigma )\right)}_{{\sigma }_{ij},{\sigma }_{kl}}=\frac{1}{2}\mathrm{tr}\left({\Sigma }^{-1}\frac{\partial \Sigma }{\partial {\sigma }_{ij}}{\Sigma }^{-1}\frac{\partial \Sigma }{\partial {\sigma }_{kl}}\right)$, and the inverse simplifies to

$$\begin{array}{c}\hfill I_F{\left(\lambda \right)}^{-1}\approx \left(\begin{array}{cc}\Sigma & 0\\ 0& \Sigma \otimes \Sigma \end{array}\right),\end{array}$$

(21)

where “⊗” is the Kronecker product. Finally, the natural gradient of $\mathcal{F}\left[q\right]$ can be written as

$$\begin{array}{c}\hfill \begin{array}{ccc}{\nabla }_{\mu }^{nat}\mathcal{F}\left[q\right]& =& \Sigma {\nabla }_{\mu }\mathcal{F}\left[q\right]\hfill \\ {\nabla }_{\Sigma }^{nat}\mathcal{F}\left[q\right]& =& \Sigma {\nabla }_{\Sigma }\mathcal{F}\left[q\right]\Sigma \hfill \end{array}.\end{array}$$

(22)

In our stochastic gradient ascent scheme, the parameters $\mu$ are updated using a momentum algorithm with updating step

$${\mu }_{t+1}={\mu }_t+\gamma m_{{\mu }_t}$$

(23)

where the momentum term $m_{{\mu }_t}$ is given by

$$\begin{array}{c}\hfill m_{{\mu }_{t+1}}=\upsilon m_{{\mu }_t}+(1-\upsilon ){\nabla }_{\mu }^{nat}\mathcal{F}\left[q\right].\end{array}$$

(24)

For the update on $\Sigma$, it is necessary to map the point on the tangent space, indicated by the steepest ascent direction, back to the manifold. For that, we use a retraction mapping that approximates the exponential map of the manifold of symmetric positive semidefinite matrices [58].

In our case, we use

$$R_{\Sigma }\left(\xi \right)=\Sigma +\xi +\frac{1}{2}\xi {\Sigma }^{-1}\xi .$$

(25)

Further, for the momentum update on the manifolds, we apply a vector transport that further projects the translated points back to the tangent space, as was first done in [59]. For our purposes, we apply the following mapping:

$${\Gamma }_{{\Sigma }_1\to {\Sigma }_2}\left(\xi \right)=U\xi U^T,U={\left({\Sigma }_2{\Sigma }_1^{-1}\right)}^{1/2}.$$

(26)

Finally, our computational algorithm is summarized in Algorithm 1.

Algorithm 1: Manifold gradient ascent

5. Numerical Examples

We studied the performance of the proposed algorithm on three problems. One metamodeling problem and two multifidelity model calibration problems are used in the sections that follow.

5.1. Academic Example

We first considered the following mathematical functions

$$\begin{array}{c}\hfill \begin{array}{ccc}{f}_1(\mathbf{x},\mathit{\theta })& =& {\theta }_1(8{\mathbf{w}}^T\mathbf{x}-2)sin(5{\mathbf{w}}^T\mathbf{x}-4)+{\theta }_2(2{\mathbf{w}}^T\mathbf{x}+\frac{1}{2})\hfill \\ f_2(\mathbf{x},\mathit{\theta })& =& f_1(\mathbf{x},\mathit{\theta })+30{\left({\mathbf{w}}^T\mathbf{x}\right)}^2,\hfill \end{array}\end{array}$$

(27)

with the coupling indicating that $f_1(\mathbf{x},\mathit{\theta })$ can be considered to be a low-fidelity simulator and $f_2(\mathbf{x},\mathit{\theta })$ the high-fidelity function. We took $\mathbf{x}\in {\mathbb{R}}^{10}$ and the vector $\mathbf{w}$ was considered a set of known parameters projecting the 10-dimensional vector $\mathbf{x}$ to $\mathbb{R}$. For this example, we took

$$\begin{array}{c}\hfill \mathbf{w}=\left[\begin{array}{c}\hfill 0.14042\\ \hfill -0.35474\\ \hfill 0.42674\\ \hfill -0.09312\\ \hfill -0.21463\\ \hfill 0.26425\\ \hfill 0.25603\\ \hfill -0.18959\\ \hfill 0.00467\\ \hfill -0.66800\end{array}\right].\end{array}$$

(28)

A set of ${10}^4$ training points was generated from the low-fidelity function, that is, ${\mathcal{D}}_l={\{{\mathbf{x}}_i,{\mathit{\theta }}_i,y_i\}}_{i=1}^{{10}^4}$ while ${\mathcal{D}}_h={\{{\mathbf{x}}_i,y_i\}}_{i=1}^{200}$ consisted of 200 points simulated from $f_2$, where the calibration parameters were fixed to $\mathit{\theta }=(3/2,30)$. All inputs were generated using a uniform Latin hypercube sampling on ${[-2,2]}^{10}$, while the ${\theta }_i$’s were sampled uniformly within $[0.5,2.5]\times [20,40]$. The data are shown in Figure 1.

To test for the robustness of the approach, we first performed the ELBO optimization corresponding to training an autoregressive GP model on the available training data using a varying number of Monte Carlo samples used to evaluate the ELBO gradient estimate (17), namely $N=10,50,$ and 100. We ran $5\times {10}^3$ iterations of algorithm 1 using an initial learning rate ${\gamma }_0=0.0001$, momentum weight parameters $\upsilon =0.6$, and a random batch size equal to 50 data points (0.5% of the full dataset) to enable the SVI feature. As expected, the runtimes scaled linearly from 13 min for $N=10$ to 61 min for $N=50$, and 125 min for $N=100$. Improving the quality of the MC estimate of the $\mathcal{F}\left[q\right]$ function was expected to improve optimization performance. This is highlighted in

Table 1. Academic example: final ELBO values after 5000 iterations and the computational runtimes for each training case ($N=10,50$, and 100).

N	Final $\mathcal{F}\left[\mathit{q}\right]$	Mean Predictive Std	Runtime
10	228.77	34.71	13’
50	265.64	27.82	61’
100	439.75	19.28	125’

which shows the final ELBO values $\mathcal{F}\left[q\right]$ achieved after the optimization procedure was over and the mean predictive standard deviation (std) over all test points. In addition, Figure 2 shows the convergence of the objective function. It can be observed that increasing N resulted in more robust convergence and attaining higher values closer to the true optimum that subsequently resulted in better posterior estimates. The decrease in mean predictive std indicated that the predictive accuracy was improved as well and the confidence increased.

Next, Figure 3 shows the comparison between the observations and the trained model predictions along with a 45-degree line plot for the case where the number of MC samples was as low as 10. As can be seen, the red “•” marks that correspond to the discrepancy-adjusted prediction match exactly the observations and the variance remains low. The blue “×” marks corresponding to the inferred low-fidelity simulator $\eta (\mathbf{x},\mathit{\theta })$ fall below the line, which agrees with the observed trends of the true functions as seen in Figure 1. Specifically, the low-fidelity function appears to be the closest possible to the high-fidelity one on design points $\mathbf{x}$ corresponding to values of ${\mathbf{w}}^T\mathbf{x}$ near the origin, which is when we should expect the discrepancy term points to be the closest to the 45-degree line. When $\eta (\mathbf{x},\mathit{\theta })$ reaches very low or very high values (near $-200$ or 200, respectively), the discrepancy is the largest, and indeed the points are far from the 45-degree line.

Figure 4 shows the prediction versus observations plots for 500 test data points along with a 45-degree line plots again for the three cases where the number of MC samples was 10, 50, and 100, respectively. At last, Figure 5 shows the posterior densities of the two calibration parameters $\mathit{\theta }=({\theta }_1,{\theta }_2)$ obtained using the VI framework. We observe a clear improvement in the accuracy of the ${\theta }_1$ estimate as the number of Monte Carlo samples increase from 10. To ensure numerical stability in our implementation, the Gaussian approximation was applied on the $log\mathit{\theta }$ and the resulting density plots were based on a kernel density estimation using $5\times {10}^3$ samples from the optimal log-normal approximation that was obtained using the VI approach.

5.2. Chicago Crimes Statistics Dataset

In this section, we demonstrate the applicability of the proposed approach on a metamodeling task. The dataset used for this problem was one of the three datasets under the Query Analytics Workloads Dataset section, hosted by the University of California Irvine open-source machine learning data repository (https://archive.ics.uci.edu/ml/datasets/Query+Analytics+Workloads+Dataset accessed on 31 May 2022). This dataset was used in other recent works [60,61] in order to benchmark the performance of the proposed novel machine learning algorithms and has been derived from synthetic query analytics workloads from (https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-Present/ijzp-q8t2 accessed on 31 May 2022). The quantity of interest being modeled was the number of crimes reported or simply the count of crimes in a particular region, in the city of Chicago [60]. The variables used to define the region included the x and y coordinates of the center of the region and the radius of the region. Thus, the problem had three inputs and one output. The dataset had ten thousand pairs of inputs and outputs. We leveraged nine thousand points for training the fully Bayesian metamodel and left out one thousand points as test data in order to evaluate the predictive performance of the trained model and we ran $2\times {10}^4$ iterations of our optimization scheme.

Two clear observations from Figure 6 are: (a) the predictive performance visibly improves as the batch size of subsampled training data increases from across the three subfigures (here we used batch sizes equal to 45, 67, and 90 samples that corresponded to using 0.5%, 0.75%, and 1% of the available dataset), and (b) the predictive epistemic uncertainty of the trained model also decreases, indicating a higher confidence in the model.

Table 2. Chicago crimes dataset: root-mean-squared error (RMSE) values and mean standard deviations for the resulting model trained using different batch sizes.

Batch Size	RMSE Value	Mean Predictive Std
45	6.445	24.626
67	6.315	15.065
90	5.254	7.889

shows the improvement of the predictive accuracy of the model as the batch size used for training is increased, as that is illustrated through the root-mean-squared error (RMSE) and the mean std values.

In addition to these, Figure 7 shows the increase in runtime of the algorithm as the batch size of subsampled training data increases. For reference, we also present the runtimes of the sparse GP implementations presented in [4] using the GPy package [62] for the same number of iterations and batch size and a latent variable with 80 data points. As can be seen, our approach reduced the runtime significantly for very small batch sizes while the performance of the two algorithms was about the same when batch size became 90.

5.3. Torsional Vibration Problem

We considered the torsional vibration problem on the system depicted in Figure 8 consisting of three shafts and two discs of varying geometric characteristics and elasticity properties. Our goal was to built a Gaussian process metamodel on the quantity of interest that expressed the lowest natural frequency, given as

$$Y=\sqrt{\frac{-b-\sqrt{b^2-4ac}}{2}}/2\pi ,$$

(29)

where $a=1$,

$$\begin{array}{cc}b=-\left(\frac{K_1+K_2}{J_1}+\frac{K_2+K_3}{J_2}\right),& c=\frac{K_1{K}_2+K_2{K}_3+K_1{K}_3}{J_1{J}_2}.\end{array}$$

(30)

The torsional stiffness values were given by

$$K_i={\theta }_1\frac{\pi G_i{d}_i}{32L_i},i=1,2,3$$

(31)

and the polar moments of inertia were given by

$$J_j={\theta }_2{M}_j{\left(\frac{D_j}{2}\right)}^2,i=1,2$$

(32)

with $M_j=\frac{{\rho }_j}{g}\pi t_j\frac{D_j}{4}$, $j=1,2$. We considered a high-fidelity simulator where $Y_{hf}$ was evaluated using ${\theta }_1=\pi /32$, ${\theta }_2=\frac{1}{2}$ and shaft diameters $d_1=2$, $d_2=1.825$, and $d_3=2.25$ in expressions (31) and (32), while data from a low-fidelity $Y_{lf}$ were also used, where ${\theta }_1$ and ${\theta }_2$ were considered unknown parameters to be inferred and all diameters were taken equal $d_1=d_2=d_3=2$. All 12 remaining geometric and elasticity properties of the system were assumed to be design parameters and are described in

Table 3. Torsional vibration problem: description of the 12-dimensional input parameters and their values ranges. Length, diameters and thicknesses are given in inches, moduli of rigidity are in lb/sq inch, and weight densities are expressed in lb/cubic inch.

Part	Parameter	Value Range
Shaft 1	Length $L_1$	[9, 11]
	Modulus of rigidity $G_1$	[1053, 1287] $\times {10}^5$
Shaft 2	Length $L_2$	[10.8, 13.2]
	Modulus of rigidity $G_2$	[558, 682] $\times {10}^4$
Shaft 3	Length $L_3$	[7.2, 8.8]
	Modulus of rigidity $G_3$	[351, 429] $\times {10}^4$
Disk 1	Diameter $D_1$	[10.8, 13.2]
	Thickness $t_1$	[2.7, 3.3]
	Weight density ${\rho }_1$	[0.252, 0.308]
Disk 2	Diameter $D_2$	[12.6, 15.4]
	Thickness $t_2$	[3.6, 4.4]
	Weight density ${\rho }_2$	[0.09, 0.11]

.

We considered again an experimental scenario in the big data regime, where ${10}^4$ simulation data points were generated from $J_1$ and a much smaller number of high-fidelity observations were available from $J_2$. We tested the robustness of the approach by varying the number of high-fidelity observations from only 50 points up to 250 and we compared the runtimes. Due to the increasing number of data points used to optimize the ELBO, it became necessary to adjust the maximum number of iterations for which the optimization algorithm ran, and therefore, the resulting runtime was affected. For the first three cases, we performed 1000 iterations; for the case $N_{hf}=200$, we performed 1500 iterations; and for the remaining case ($N_{hf}=250$), 2000 iterations were found to be necessary. Figure 9 shows the convergence of the ELBO function along with the root-mean-squared error (RMSE) values obtained for each trained model, based on 100 test data points. As expected, the RMSE goes down with an increasing number of high-fidelity data as shown in Figure 9 bottom.

The posterior results for the calibrated parameters along with the runtime for each case are shown in

Table 4. Torsional vibration problem: posterior statistics for the calibration parameters $({\theta }_1,{\theta }_2)$ and the computational runtimes for each training case.

${\mathit{N}}_{\mathit{hf}}$	${\mathit{\theta }}_{\mathbf{1}}$ (Mean, Std)	${\mathit{\theta }}_{\mathbf{2}}$ (Mean, Std)	Runtime
50	(0.092, 0.01)	(0.484, 0.042)	9.7’
100	(0.091, 0.01)	(0.487, 0.111)	12.9’
150	(0.132, 0.03)	(0.682, 0.179)	14.1’
200	(0.145, 0.27)	(0.554, 1.614)	44.6’
250	(0.088, 0.0008)	(0.450, 0.0007)	54.4’

. As can be seen, the true values (0.98 and 0.5) fall within the reported mean values of $\mathit{\theta }$ ± 2 standard deviations for all cases. At last, the comparison of the model prediction versus observation, along with the 45-degree line plots, is provided for the worse and best cases ($N_{hf}=50,250$) in Figure 10.

6. Conclusions

We enhanced and extended the state-of-the-art stochastic variational Bayesian formulation for tasks that use GPs for multifidelity metamodeling and model calibration tasks, in order to treat problems with tens of thousands of training data and model calibration problems with more than ten inputs. The proposed mathematical formulation extended two classic approaches, the so-called black-box VI and stochastic VI, while utilizing a manifold gradient ascent scheme to accomplish the task of inferring the GP hyperparameters as well as the calibration parameters. The major impact of our work was being able to perform a fully Bayesian uncertainty quantification while training and calibrating models using multifidelity GPs, albeit with large datasets and a moderately large number of inputs. Numerical results on two challenging engineering problems visibly demonstrated a scale up of classical Bayesian GPs for multifidelity modeling to calibrate untuned computer simulators, by enabling savings in computation. This speed-up is critical for engineering applications, especially in the industry, where repeated model calibration tasks are a common occurrence and can lead to accumulated savings using the proposed approach.

This work showed promise for accelerating the training procedure in Gaussian process based metamodels without relying on enormous computational power. The key characteristic in our approach was the batch-sampling step used in the stochastic variational inference framework, which allowed the fast computation of the likelihood term and accelerated the optimization task. One key challenge in our approach is that fine tuning of the optimization is required in order to ensure a sufficiently large updating step in the optimization scheme, while at the same time avoiding overshooting. Fine tuning the algorithm heavily depends on the size of the batch samples being used, which is also relative to the original data size that is available. Extremely small batch samples can result in very inaccurate likelihood evaluations and eventually miss the optimum. Another important aspect mentioned above is the number of Monte Carlo samples used for approximating the ELBO function. Very small number of samples can lead to inaccurate estimates with large variance that fail to converge, while on the other hand, a high number of samples makes the algorithm computationally expensive and fails to achieve the desired speed up. Typically, big data problems in Bayesian inference exhibit a well-defined posterior, therefore optimizing the ELBO should always be a feasible task given that some fine tuning has been performed. A limitation of the approach would the case where a big part of the data set is corrupted or contains high noise, in which case, the exploration of the posterior via VI might become challenging due to the complex nature of the true posterior. In such cases, more complex variational approximations need to be considered which could, however, make the algorithm less computationally efficient.

Other general challenges, not associated specifically with our approach, are problems of extremely high input and output dimensions as well as highly nonsmooth response functions. In such cases, a further development of our framework might be necessary such that it aligns with similar approaches in the literature, for instance, enabling covariance matrix sparsity, employing nonsmooth correlation kernels and last, but not least, leveraging parallel computing.

Directions for future work include scaling up the proposed approach to problems with higher input dimensionality, i.e., hundreds of inputs and with more than one sources of information with lower-fidelity and large training data. Additionally, the proposed approach needs more work in order to be applied to problems where the different sources do not share the same inputs.