Bayesian Inference and Deep Learning for Inverse Problems ^†

Abstract

Inverse problems arise anywhere we have an indirect measurement. In general, they are ill-posed to obtain satisfactory solutions, which needs prior knowledge. Classically, different regularization methods and Bayesian inference-based methods have been proposed. As these methods need a great number of forward and backward computations, they become costly in computation, particularly when the forward or generative models are complex, and the evaluation of the likelihood becomes very costly. Using deep neural network surrogate models and approximate computation can become very helpful. However, in accounting for the uncertainties, we need first to understand Bayesian deep learning, and then we can see how we can use it for inverse problems. In this work, we focus on NN, DL, and, more specifically, the Bayesian DL particularly adapted for inverse problems. We first give details of Bayesian DL approximate computations with exponential families; then, we see how we can use them for inverse problems. We consider two cases: First, we consider the case where the forward operator is known and used as a physics constraint, and the second examines more general data-driven DL methods.

1. Introduction

Inverse problems arise almost everywhere in science and engineering. In fact, anytime and in any application when we have indirect measurements related to what we really want to measure through some mathematical relation, this is called the forward model. Then, we have to infer the desired unknown from the observed data using this forward model or a surrogate one. In general, many inverse problems are ill-posed, and many methods for finding well-posed solutions for them are mainly based either on regularization theory or Bayesian inference. We mention those in particular, which are based on the optimization of a criterion with two parts: a data model output matching criterion (likelihood part in the Bayesian) and a regularization term (prior model in the Bayesian). Different criteria for these two terms and a great number of standard and advanced optimization algorithms have been proposed and used with great success. When these two terms are distances, they can have a Bayesian maximum a posteriori (MAP) interpretation, where these two terms correspond, respectively, to the likelihood and prior probability models. The Bayesian approach gives more flexibility in choosing these terms via the likelihood and the prior probability distributions. This flexibility goes much further with the hierarchical models and appropriate hidden variables [1]. Also, the possibility of estimating the hyperparameters gives much more flexibility for semisupervised methods.

However, full Bayesian computations can become very heavy computationally. In particular, this occurs when the forward model is complex, and the evaluation of the likelihood has a high computational cost [2]. In those cases, using surrogate simpler models can become very helpful to reduce the computational costs, but then we have to account for the uncertainty quantification (UQ) of the obtained results [3]. Neural networks (NNs), with their diversity such as convolutional neural networks (CNNs), deep learning (DL), etc., have become tools for fast and low computational surrogate forward models.

Over the last decades, machine learning (ML) methods and algorithms have gained great success in many tasks, such as classification, clustering, segmentation, object detection, and many other areas. There are many different structures of neural networks (NNs), such as feed-forward, Convolutional NNs (CNNs), Deep NNs, etc. [4]. Using these methods directly for inverse problems, as intermediate preprocessing or as tools for performing fast approximate computation in different steps of regularization or Bayesian inference, has also been successful, but not as much as could be possible. Recently, physics-informed neural networks have gained great success in many inverse problems, thereby proposing interaction between the Bayesian formulation of forward models and optimization algorithms and ML-specific algorithms for intermediate hidden variables. These methods have become very helpful to obtain approximate practical solutions to inverse problems in real-world applications [5,6,7,8,9,10,11].

In this paper, first, in Section 2, some mathematical notations for dealing with NNs are given. In Section 3, a detailed presentation of the Bayesian inference and the approximate computation needed for BDL are given. Then, in Section 4, we consider a focus on the NN and DL methods for inverse problems. First, we present the same cases where we know the forward and its adjoint model. Then, we consider the case where we may not have this knowledge and want to propose directly data-driven DL methods [12,13].

2. Neural Networks, Deep Learning, and Bayesian DL

The main objective of the NN for a regression problem can be described as follows:

$${\mathbf{x}}_i\to \boxed{\mathrm{NN}:f\left({\mathbf{x}}_i\right)}\to y_i.$$

The objective is to infer the function $f:{\mathbb{R}}^M\to \mathbb{R}$ from the observations $\mathbf{y}={(y_1,\cdots ,y_N)}^{\mathrm{T}}$ at locations given by $\mathbf{x}=({\mathbf{x}}_1,\cdots ,{\mathbf{x}}_N)$. The usual NN learning approach is to define a parametric family of functions ${\varphi }_{\mathit{\theta }}:{\mathbb{R}}^M\times \mathit{\theta }\to \mathbb{R}$ that is flexible enough so that $\exists {\mathit{\theta }}^{★}$ such that $\varphi (·)\approx {\varphi }_{{\mathit{\theta }}^{★}}(·)$:

$$\{{\mathbf{x}}_i,y_i\}\to \boxed{\mathrm{NN}\mathrm{Learning}:y_i=\varphi \left({\mathbf{x}}_i\right)\approx {\varphi }_{{\mathit{\theta }}^{★}}\left({\mathbf{x}}_i\right)}\to {\mathit{\theta }}^{★}.$$

Deep learning focuses on learning the optimal parameters ${\mathit{\theta }}^{★}$, which can then be used for predicting the output ${\widehat{y}}_j$ for any input ${\mathbf{x}}_j$:

$${\mathbf{x}}_j\to \boxed{\mathrm{NN}:{\varphi }_{{\mathit{\theta }}^{★}}\left({\mathbf{x}}_j\right)}\to {\widehat{y}}_j.$$

In this approach, there is no uncertainty quantification.

The Bayesian deep learning approach can be summarized as follows:

$$\{{\mathbf{x}}_i,y_i\}\to \boxed{p\left(\mathit{\theta }\right|\mathcal{D})=\frac{p\left(\mathcal{D}\right|\mathit{\theta }\left)p\right(\mathit{\theta })}{p\left(\mathcal{D}\right)}}\to p\left(\mathit{\theta }\right|\mathcal{D}),$$

$${\mathbf{x}}_j\to \boxed{{\displaystyle p\left(y_j\right|{\mathbf{x}}_j,\mathcal{D})=\int p\left(y_j\right|{\mathbf{x}}_j,\mathit{\theta },\mathcal{D})p\left(\mathit{\theta }\right|\mathcal{D})\mathrm{d}\mathit{\theta }}}\to {\widehat{y}}_j.$$

As we can see, uncertainties are accounted for in both steps of the parameter estimation and prediction. However, as we will see, the computational costs are important. We need to find solutions to perform fast computation.

3. Bayesian Inference and Approximate Computation

In a general Bayesian framework for NNs and DL, the objective is to infer the parameters $\mathit{\theta }$ from the data, $\mathcal{D}:\{{\mathbf{x}}_i,y_i\}$, using the Bayes rule:

$$p\left(\mathit{\theta }\right|\mathcal{D})=\frac{p\left(\mathcal{D}\right|\mathit{\theta }\left)p\right(\mathit{\theta })}{p\left(\mathcal{D}\right)},$$

(1)

where $p\left(\mathit{\theta }\right)$ is the prior, $\mathit{\ell }\left(\mathcal{D}\right|\mathit{\theta })=p(\mathcal{D}\left|\mathit{\theta }\right)$ is the likelihood, $p\left(\mathit{\theta }\right|\mathcal{D})$ is the posterior, and ${\displaystyle p(\mathcal{D})=\int p(\mathcal{D}|\mathit{\theta })p(\mathit{\theta })\mathrm{d}\mathit{\theta }}$ is called the evidence. We can also write $p\left(\mathit{\theta }\right|\mathcal{D})\propto \mathit{\ell }(\mathcal{D}\left|\mathit{\theta }\right)p\left(\mathit{\theta }\right)$, where the classical maximum likelihood estimation (MLE) is defined as $\widehat{\mathit{\theta }}=\arg {max}_{\mathit{\theta }}\left\{\mathit{\ell }\left(\mathcal{D}\right|\mathit{\theta })\right\}$.

A particular point of the posterior is of high interest, because we may be interested in maximum a posterior (MAP) estimation: $\widehat{\mathit{\theta }}=\arg {max}_{\mathit{\theta }}\left\{\mathit{\ell }\left(\mathcal{D}\right|\mathit{\theta }\left)p\right(\mathit{\theta })\right\}$. We may also be interested in mean squared error (MSE) estimation, which is shown that it corresponds to ${\displaystyle \widehat{\mathit{\theta }}={\mathrm{E}}_{p\left(\mathit{\theta }\right|\mathcal{D})}\left\{\mathit{\theta }\right\}=\int \mathit{\theta }p\left(\mathit{\theta }\right|\mathcal{D})\mathrm{d}\mathit{\theta }}$.

The exact expression of the posterior and the computations of these integrals for great dimensional problems may be very computationally costly. For this reason, we need to perform approximate computation. In the following subsections, we review a few solutions.

3.1. Laplace Approximation

Rewriting the general Bayes rules slightly differently gives us the following:

$$p\left(\mathit{\theta }\right|\mathcal{D})=\frac{1}{p\left(\mathcal{D}\right)}p\left(\mathcal{D}\right|\mathit{\theta }\left)p\right(\mathit{\theta })=\frac{1}{Z}exp\left[\mathcal{L}\left(\mathit{\theta }\right|\mathcal{D})\right],$$

(2)

the Laplace approximations use a second-order expansion of $\mathcal{L}\left(\mathit{\theta }\right|\mathcal{D})=lnp(\mathcal{D}\left|\mathit{\theta }\right)+lnp\left(\mathit{\theta }\right)$ around ${\widehat{\mathit{\theta }}}_{MAP}$ to construct a Gaussian approximation of $p\left(\mathit{\theta }\right|\mathcal{D})$:

$$\mathcal{L}\left(\mathit{\theta }\right)\approx \mathcal{L}\left({\widehat{\mathit{\theta }}}_{MAP}\right)+\frac{1}{2}{(\mathit{\theta }-{\widehat{\mathit{\theta }}}_{MAP})}^{\prime }\left({\nabla }_{\theta }^2{\mathcal{L}\left(\mathit{\theta }\right)|}_{{\widehat{\mathit{\theta }}}_{MAP}}\right)(\mathit{\theta }-{\widehat{\mathit{\theta }}}_{MAP}),$$

(3)

where the first-order term vanishes at the ${\widehat{\mathit{\theta }}}_{MAP}$. This is equivalent to calculating the following approximation:

$$\boxed{\mathrm{Approximate}p\left(\mathit{\theta }\right|\mathcal{D})\mathrm{by}q\left(\mathit{\theta }\right)=\mathcal{N}\left(\mathit{\theta }|{\widehat{\mathit{\theta }}}_{MAP},\mathit{\Sigma }=\left[{\nabla }_{\theta }^2\mathcal{L}\left(\mathit{\theta }\right){|}_{{\widehat{\mathit{\theta }}}_{MAP}}\right]\right).}$$

With this approximation, the evidence $p\left(\mathcal{D}\right)$ is approximated by the following:

$$Z=p\left(\mathcal{D}\right)={\left(2\pi \right)}^{d/2}{\left|\mathit{\Sigma }\right|}^{1/2}exp\left[-\mathcal{L}\left({\widehat{\mathit{\theta }}}_{MAP}\right)\right].$$

(4)

For great dimensional problems such as BDL, the full computation of $\mathit{\Sigma }$ is very costly. We have still to do more approximations. The following are a few solutions for scalable approximations for BDL:

• Work with the subnetwork or last layer (transfer learning);
• Perform covariance matrix decomposition (low rank, Kronecker-factored approximate curvature (KFAC), Diag);
• Conduct the computation during the hyperparameter tuning using crossvalidation;
• Use approximate predictive computation.

3.2. Approximate Computation: Variational Inference

The main idea then is to perform approximate Bayesian computation (ABC) by approximating the posterior $p\left(\mathit{\theta }\right|\mathcal{D})$ using a simpler expression $q\left(\mathit{\theta }\right)$. When approximation is performed by minimizing

$$\mathrm{KL}\left[q\left(\mathit{\theta }\right):p\left(\mathit{\theta }\right|\mathcal{D})\right]=\int q\left(\mathit{\theta }\right)\log \frac{q\left(\mathit{\theta }\right)}{p\left(\mathit{\theta }\right|\mathcal{D})}\mathrm{d}\mathit{\theta },$$

(5)

the method is called the variational Bayesian approximation (VBA). When $q\left(\mathit{\theta }\right)$ is chosen to be separable in some components of the parameters $q\left(\mathit{\theta }\right)={\prod }_j{q}_j\left({\theta }_j\right)$, the approximation is called the mean field VBA (MFVBA).

Let us come back to the general VBA, ${\displaystyle \mathrm{KL}\left[q:p\right]=\int q\log \frac{q}{p}={\mathrm{E}}_q\left\{\log q\right\}-{\mathrm{E}}_q\left\{\log p\right\}}$, and note using $L={\mathrm{E}}_q\left\{\log p\right\}$ the expected likelihood and using $S=-{\mathrm{E}}_q\left\{\log q\right\}$ the entropy of q. Then, we have the following:

$$q*=arg\underset{q}{min}\left\{\mathrm{KL}\left[q:p\right]\right\}=arg\underset{q}{max}\left\{E=S+L\right\}.$$

(6)

E is also called the evidence lower bound (ELBO):

$$ELBO=-{\mathrm{E}}_q\left\{\log q\right\}+{\mathrm{E}}_q\left\{\log p\right\}.$$

(7)

At this point, it is important to note one main property of the VBA: When $p\left(\mathit{\theta }\right|\mathcal{D})$, the posterior probability law p and the approximate probability law q are in the exponential family; then, ${\mathrm{E}}_p\left\{\mathit{\theta }\right\}={\mathrm{E}}_q\left\{\mathit{\theta }\right\}$.

3.3. VBA and Natural Exponential Family

If q is chosen to be in a natural exponential family, $q\left(\mathit{\theta }\right|\mathit{\eta })=exp\left[{\mathit{\eta }}^{\prime }\mathit{\theta }-A\left(\mathit{\eta }\right)\right]$, then it is entirely characterized by its mean $\mathbf{m}={\mathrm{E}}_q\left\{\mathit{\theta }\right\}$, and if q is conjugate to p, then $q^{*}\left(\mathit{\theta }\right|\mathit{\eta })=exp\left[{{\mathit{\eta }}^{*}}^{\prime }\mathit{\theta }-A\left({\mathit{\eta }}^{*}\right)\right]$, which is entirely characterized by its mean ${\mathbf{m}}^{*}={\mathrm{E}}_{q^{*}}\left\{\mathit{\theta }\right\}$. We can then define the objective E as a function of $\mathbf{m}$, and the first order condition of the optimality is $\frac{\partial E}{\partial \mathbf{m}}{|}_{\mathbf{m}=\mathbf{m}\ast }=0$. From this property, we can obtain a fixed-point algorithm to compute ${\mathbf{m}}^{*}={\mathrm{E}}_{q^{*}}\left\{\mathit{\theta }\right\}$:

$$\frac{\partial E}{\partial \mathbf{m}}{|}_{\mathbf{m}=\mathbf{m}\ast }=0\to \frac{\partial E}{\partial \mathbf{m}}{|}_{\mathbf{m}=\mathbf{m}\ast }+\mathbf{m}=\mathbf{m}\to \mathbf{M}\left(\mathbf{m}\right)=\mathbf{m}.$$

Iterating on this fixed-point algorithm gives us the following:

$$\mathbf{M}\left(\mathbf{m}\right)={\mathbf{m}}^{(k-1)}\mathrm{with}\mathbf{M}\left(\mathbf{m}\right):=\frac{\partial E}{\partial \mathbf{m}}+\mathbf{m}$$

(8)

which converges to ${\mathbf{m}}^{*}={\mathrm{E}}_{q^{*}}\left\{{\mathit{\theta }}^{*}\right\}$ and is also ${\mathbf{m}}^{*}={\mathrm{E}}_q\left\{{\mathit{\theta }}^{*}\right\}$. This algorithm can be summarized as follows:

• Having chosen the prior and likelihood, find the expression of $p\left(\mathit{\theta }\right|\mathcal{D})\propto p(\mathcal{D}\left|\mathit{\theta }\right)p\left(\mathit{\theta }\right)$;
• Choose a family q and find the expressions of $L={\mathrm{E}}_q\left\{lnp\right\}$ and $S={\mathrm{E}}_q\left\{lnq\right\}$, which thus yield $E=L+S$ as a function of $\mathbf{m}={\mathrm{E}}_q\left\{\mathit{\theta }\right\}$;
• Find the expression of the vector operator $\mathbf{M}={\nabla }_{\mathbf{m}}E+\mathbf{m}$ and update it $\mathbf{M}\left(\mathbf{m}\right)={\mathbf{m}}^{(k-1)}$ until convergence, which results in ${\mathbf{m}}^{*}={\mathrm{E}}_{q^{*}}\left\{{\mathit{\theta }}^{*}\right\}={\mathrm{E}}_{p^{*}}\left\{{\mathit{\theta }}^{*}\right\}$.

At this point, it is important to note that, in this approach, even if the mean is well approximated, the variances or the covariance are underestimated. Some authors who are interested in this approach have proposed solutions to better estimate the covariance. See [14] for one of the solutions.

4. Deep Learning and Bayesian DL

As introduced before, in classical DL, the training and prediction steps can be summarized as follows:

$$\{{\mathbf{x}}_i,y_i\}\to \boxed{\mathrm{NN}\mathrm{Learning}:{\varphi }_{{\mathit{\theta }}^{★}}(·)}\to {\mathit{\theta }}^{★}{\mathbf{x}}_j\to \boxed{\mathrm{NN}:{\varphi }_{{\mathit{\theta }}^{★}}\left({\mathbf{x}}_j\right)}\to {\widehat{y}}_j$$

In this approach, there is no uncertainty quantification. The Bayesian deep learning can be summarized as follows:

$$\{{\mathbf{x}}_i,y_i\}\to \boxed{p\left(\mathit{\theta }\right|\mathcal{D})=\frac{p\left(\mathcal{D}\right|\mathit{\theta }\left)p\right(\mathit{\theta })}{p\left(\mathcal{D}\right)}}\to p\left(\mathit{\theta }\right|\mathcal{D})\mathrm{or}q\left(\mathit{\theta }\right|\mathcal{D}),$$

$${\mathbf{x}}_j\to \boxed{{\displaystyle p\left(y_j\right|{\mathbf{x}}_j,\mathcal{D})=\int p\left(y_j\right|{\mathbf{x}}_j,\mathit{\theta },\mathcal{D})p\left(\mathit{\theta }\right|\mathcal{D})\mathrm{d}\mathit{\theta }}}\to p\left(y_i\right|{\mathbf{x}}_j,\mathcal{D})\mathrm{or}q\left(y_i\right|{\mathbf{x}}_j,\mathcal{D}).$$

As we can see, uncertainties are accounted for in both steps of the parameter estimation and prediction. However, computational costs are important. We need to find solutions to perform fast computation. As mentioned before, the different possible tools are Laplace and Gaussian approximation, variational inference, and more controlled approximation to design new deep learning algorithms, which can scale up for practical situations.

Exponential Family Approximation

Let us consider the case of general exponential families $q\left(\mathit{\theta }\right|\mathit{\lambda })=exp\left[{\mathit{\lambda }}^T\mathbf{t}\left(\mathit{\theta }\right)-F\left(\mathit{\lambda }\right)\right]$, where $\mathit{\theta }$ represents the original parameters, $\mathit{\lambda }$ represents the natural parameters, $\mathbf{t}\left(\mathit{\theta }\right)$ represents sufficient statistics, $F\left(\mathit{\lambda }\right)$ represents the log partition function, and we define the expectations parameters as $\mathit{\mu }:={\mathrm{E}}_q\left\{\mathbf{t}\left(\mathit{\theta }\right)\right\}$. Let us also define the dual function $F^{*}\left(\mathit{\eta }\right)$ and the dual parameters $\mathit{\eta }$ via the Legendre transform:

$$G\left(\mathit{\eta }\right)=F^{*}\left(\mathit{\eta }\right)=\underset{\mathit{\lambda }}{sup}\{<\mathit{\lambda },\mathit{\eta }>-F\left(\mathit{\lambda }\right)\}.$$

Then, we can show the triangular relation between $\mathit{\theta }\in \mathrm{\Theta }$, $\mathit{\lambda }\in \mathrm{\Lambda }$, and $\mathit{\mu }\in \mathcal{M}$ shown in Figure 1.

With these notations, applying the VBA rule ${min}_{q\in \mathcal{Q}}{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\}-H\left(q\right)$ results in the following updating rule for the natural parameters:

$$\mathit{\lambda }\leftarrow \mathit{\lambda }-\rho {\nabla }_{\mathit{\mu }}\left(\mathcal{L}\left(\mathit{\theta }\right)-H\left(q\right)\right).$$

(9)

For example, by considering the Gaussian case

$$q\left(\mathit{\theta }\right)=\mathcal{N}\left(\mathit{\theta }\right|\mathbf{m},{\mathbf{S}}^{-1})\propto exp\left[-\frac{1}{2}{(\mathit{\theta }-\mathbf{m})}^T\mathbf{S}(\mathit{\theta }-\mathbf{m})\right]\propto exp\left[{\left(\mathbf{S}\mathbf{m}\right)}^T\mathit{\theta }+\mathrm{Tr}(-\frac{\mathbf{S}}{2}\mathit{\theta }{\mathit{\theta }}^T)\right].$$

(10)

we can identify the natural parameters as $\mathit{\lambda }=[\mathbf{S}\mathbf{m},-\mathbf{S}/2]\to \mathit{\mu }:=\left[{\mathrm{E}}_q\right\{\mathit{\theta }\},{\mathrm{E}}_q\{\mathit{\theta }{\mathit{\theta }}^T\left\}\right]$, and we easily obtain the following algorithm:

$$\left\{\begin{array}{c}\mathbf{S}\mathbf{m}\leftarrow (1-\rho )\mathbf{S}\mathbf{m}-\rho {\nabla }_{{\mathrm{E}}_q\left\{\mathit{\theta }\right\}}{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\},\hfill \\ \mathbf{S}\leftarrow (1-\rho )\mathbf{S}-\frac{\rho }{2}{\nabla }_{{\mathrm{E}}_q\left\{\mathit{\theta }{\mathit{\theta }}^T\right\}}{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\}.\hfill \end{array}\right.$$

(11)

where

$$\left\{\begin{array}{c}{\nabla }_{{\mathrm{E}}_q\left\{\mathit{\theta }\right\}}{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\}={\mathrm{E}}_q\left\{{\nabla }_{\mathit{\theta }}\mathcal{L}\left(\mathit{\theta }\right)\right\}-2{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\},\hfill \\ {\nabla }_{{\mathrm{E}}_q\left\{\mathit{\theta }{\mathit{\theta }}^T\right\}}{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\}={\mathrm{E}}_q\left\{\mathbf{H}\left(\mathit{\theta }\right)\right\},\hfill \end{array}\right.$$

which results, explicitly, in

$$\left\{\begin{array}{c}\mathbf{m}\leftarrow \mathbf{m}-\rho {\mathbf{S}}^{-1}{\nabla }_{\mathbf{m}}\mathcal{L}\left(\mathbf{m}\right),\hfill \\ \mathbf{S}\leftarrow (1-\rho )\mathbf{S}+\rho {\mathbf{H}}_{\mathbf{m}}.\hfill \end{array}\right.$$

(12)

For a linear model $\mathbf{y}=\mathbf{X}\mathit{\theta }$ and Gaussian priors, we have

$$\mathcal{L}\left(\mathit{\theta }\right)={(\mathbf{y}-\mathbf{X}\mathit{\theta })}^T(\mathbf{y}-\mathbf{X}\mathit{\theta })+\gamma {\mathit{\theta }}^T\mathit{\theta }=-2{\mathit{\theta }}^T\left({\mathbf{X}}^T\mathbf{y}\right)+\mathrm{Tr}\left[\mathit{\theta }{\mathit{\theta }}^T({\mathbf{X}}^T\mathbf{X}+\gamma \mathbf{I})\right],$$

(13)

and ${\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\}={\mathit{\lambda }}^T\mathit{\mu },{\nabla }_{\mathit{\mu }}{\mathrm{E}}_q\left\{\mathcal{L}\left(\mathit{\theta }\right)\right\}=\mathit{\lambda }$.

It is interesting to note that many classical algorithms for updating the parameters, such as forward–backward, sparse variational inference, and variational message passing, become special cases. A main remark here is that this linear generating function case will link us to the linear inverse problems $\mathbf{g}=\mathbf{H}\mathbf{f}+\mathit{ϵ}$ if we replace $\mathbf{y}$ with $\mathbf{g}$, $\mathbf{X}$ with $\mathbf{H}$, and $\mathit{\theta }$ with $\mathbf{f}$.

5. NN, DL, and Bayesian Inference for Linear Inverse Problems

To show the possibilities of the interaction between inverse problems methods, neural networks and deep learning, the best way is to give a few examples.

5.1. First Example: A Known Linear Forward Model

The first and easiest example is the case of linear inverse problems $\mathbf{g}=\mathbf{H}\mathbf{f}+\mathit{ϵ}$, where we know the forward model $\mathbf{H}$ and we assume the Gaussian likelihood $p\left(\mathbf{g}\right|\mathbf{f})=\mathcal{N}\left(\mathbf{g}\right|\mathbf{H}\mathbf{f},{\sigma }_{ϵ}^2\mathbf{I})$ and the Gaussian prior $p\left(\mathbf{f}\right)=\mathcal{N}(\mathbf{f}|0,{\sigma }_f^2\mathbf{I})$ are the easiest cases to consider, where we know that the posterior is Gaussian $p\left(\mathbf{f}\right|\mathbf{g})=\mathcal{N}\left(\mathbf{f}\right|\widehat{\mathbf{f}},\widehat{\mathit{\Sigma }})$ with

$$\widehat{\mathbf{f}}={({\mathbf{H}}^t\mathbf{H}+\lambda \mathbf{I})}^{-1}{\mathbf{H}}^t\mathbf{g}=\mathbf{A}\mathbf{g}=\mathbf{B}{\mathbf{H}}^t\mathbf{g}\mathrm{or}\mathrm{still}\widehat{\mathbf{f}}={\mathbf{H}}^t{(\frac{1}{\lambda }\mathbf{H}{\mathbf{H}}^t+\mathbf{I})}^{-1}\mathbf{g}={\mathbf{H}}^t\mathbf{C}\mathbf{g},$$

where $\lambda ={\sigma }_{ϵ}^2/{\sigma }_f^2$, $\mathbf{A}={({\mathbf{H}}^t\mathbf{H}+\lambda \mathbf{I})}^{-1}{\mathbf{H}}^t$, $\mathbf{B}={({\mathbf{H}}^t\mathbf{H}+\lambda \mathbf{I})}^{-1}$, $\mathbf{C}={(\frac{1}{\lambda }\mathbf{H}{\mathbf{H}}^t+\mathbf{I})}^{-1}$, and $\widehat{\mathit{\Sigma }}={({\mathbf{H}}^t\mathbf{H}+\lambda \mathbf{I})}^{-1}$.

These relations can be presented schematically as

$$\mathbf{g}\to \boxed{\mathbf{A}}\to \widehat{\mathbf{f}},\mathbf{g}\to \boxed{{\mathbf{H}}^t}\to \boxed{\mathbf{B}}\to \widehat{\mathbf{f}},\mathbf{g}\to \boxed{\mathbf{C}}\to \boxed{{\mathbf{H}}^t}\to \widehat{\mathbf{f}}.$$

We can then consider replacing $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ with appropriate deep neural networks and apply all the previous BDL methods to them. As we can see, these relations directly induce a linear feed-forward NN structure. In particular, if $\mathbf{H}$ represents a convolution operator, then ${\mathbf{H}}^t$, ${\mathbf{H}}^t\mathbf{H}$, and $\mathbf{H}{\mathbf{H}}^t$ are too, as well as the operators $\mathbf{B}$ and $\mathbf{C}$. Thus, the whole inversion can be modeled using a CNN [15,16].

For the case of computed tomography (CT), the first operation is equivalent to an analytic inversion, the second corresponds to back projection that is first followed by 2D filtering in the image domain, and the third corresponds to the famous filtered back projection (FBP), which is implemented on classical CT scans. These cases are illustrated in Figure 2.

5.2. Second Example: A Deep Learning Equivalence of Iterative Gradient-Based Algorithms

One of the classical iterative methods in linear inverse problem algorithms is based on the gradient descent method to optimize $J\left(\mathbf{f}\right)={\parallel \mathbf{g}-\mathbf{H}\mathbf{f}\parallel }^2$:

$${\mathbf{f}}^{(k+1)}={\mathbf{f}}^{\left(k\right)}+\alpha {\mathbf{H}}^t(\mathbf{g}-\mathbf{H}{\mathbf{f}}^{\left(k\right)})=\alpha {\mathbf{H}}^t\mathbf{g}+(\mathbf{I}-\alpha {\mathbf{H}}^t\mathbf{H}){\mathbf{f}}^{\left(k\right)},$$

(14)

where the solution to the problem is obtained recursively. Everybody knows that when the forward model operator $\mathbf{H}$ is singular or ill-conditioned, this iterative algorithm starts by converging, but it may diverge easily. One of the experimental methods to obtain an acceptable approximate solution is just to stop the iterations after K iterations. This idea can be translated to a deep learning NN by using K layers. Each layer represents one iteration of the algorithm. See Figure 3.

This DL structure can easily be extended to a regularized criterion.

$J\left(\mathbf{f}\right)=\frac{1}{2}{\parallel \mathbf{g}-\mathbf{H}\mathbf{f}\parallel }^2+\lambda {\parallel \mathbf{D}\mathbf{f}\parallel }^2$, which can also be interpreted as the MAP or posterior mean solution with a Gaussian likelihood and prior. In this case, we have the following:

$${\mathbf{f}}^{(k+1)}={\mathbf{f}}^{\left(k\right)}+\alpha [{\mathbf{H}}^t(\mathbf{g}-\mathbf{H}{\mathbf{f}}^{\left(k\right)})-\lambda {\mathbf{D}}^t\mathbf{D}]=\alpha {\mathbf{H}}^t\mathbf{g}+(\mathbf{I}-\alpha {\mathbf{H}}^t\mathbf{H}-\alpha \lambda {\mathbf{D}}^t\mathbf{D}){\mathbf{f}}^{\left(k\right)}.$$

(15)

We just need to replace $(\mathbf{I}-\alpha {\mathbf{H}}^t\mathbf{H})$ with $(\mathbf{I}-\alpha {\mathbf{H}}^t\mathbf{H}-\alpha \lambda {\mathbf{D}}^t\mathbf{D})$.

This structure can also be extended to all the sparsity-enforcing regularization terms such as ${\mathit{\ell }}_1$ and the total variation (TV) using appropriate algorithms such as the ISTA (iterative soft threshold algorithm) or its fast version FISTA by replacing the update expression and by adding an NL operation, much like the ordinary NNs. A simple example is given in the following subsection.

5.3. Third Example: ${\mathit{\ell }}_1$ Regularization and NN

The case of the MAP solution with a Gaussian likelihood and double exponential prior becomes equivalent to the ${\mathit{\ell }}_1$ regularization criterion:

$$J\left(\mathbf{f}\right)={\parallel \mathbf{g}-\mathbf{H}\mathbf{f}\parallel }_2^2+\lambda {\parallel \mathbf{f}\parallel }_1,$$

(16)

where the solution can be obtained with an iterative optimization algorithm, such as ISTA:

$${\mathbf{f}}^{(k+1)}=Pro{x}_{{\mathit{\ell }}_1}\left({\mathbf{f}}^{\left(k\right)},\lambda \right)\stackrel{▵}{=}{\mathcal{S}}_{\lambda \alpha }\left(\alpha {\mathbf{H}}^t\mathbf{g}+(\mathbf{I}-\alpha {\mathbf{H}}^t\mathbf{H}){\mathbf{f}}^{\left(k\right)}\right),$$

(17)

where ${\mathcal{S}}_{\theta }$ is a soft threshold operator, and $\alpha \le \left|eig\right({\mathbf{H}}^t\mathbf{H}\left)\right|$ is the Lipschitz constant of the normal operator. When $\mathbf{H}$ is a convolution operator, then

• $(\mathbf{I}-\alpha {\mathbf{H}}^t\mathbf{H}){\mathbf{f}}^{\left(k\right)}$ can also be approximated by a convolution and thus considered as a filtering operator;
• $\frac{1}{\alpha }{\mathbf{H}}^t\mathbf{g}$ can be considered as a bias term and is also a convolution operator;
• ${\mathcal{S}}_{\theta =\lambda \alpha }$ is a nonlinear pointwise operator. In particular, when $\mathbf{f}$ is a positive quantity, this soft threshold operator can be compared to the ReLU activation function of the NN. See Figure 4.

Using the iterative gradient-based algorithm with a fixed number of iterations for computing a GI or a regularized one, as explained in the previous section, can be used to propose a DL structure with K layers, with K being the number of iterations before stopping. Figure 5 shows this structure for a quadratic regularization, which results in a linear NN, and Figure 6 shows the case of ${\mathit{\ell }}_1$ regularization.

In all these examples, we could directly obtain the structure of the NN from the forward model and known parameters. However, in these approaches, there are some difficulties that consist of the determination of the structure of the NN. For example, in the first example, obtaining the structure of $\mathbf{B}$ depends on the regularization parameter $\lambda$. The same difficulty arises in determining the shape and the threshold level of the threshold bloc of the network in the second example. The same need for the regularization parameter, as well as many other hyperparameters, makes it necessary to create the NN structure and weights. In practice, we can decide, for example, on the number and structure of a DL network, but as their corresponding weights depend on many unknown or difficult-to-fix parameters, ML may become of help. In the following, we first consider the training part of a general ML method. Then, we will see how to include the physics-based knowledge of the forward model in the structure of learning [4,10,17,18].

5.4. Decomposition of the NN Structure to Fixed and Trainable Parts

The first easiest and most understandable method consists of decomposing the structure of the network $\mathbf{W}$ in two parts: a fixed part and a learnable part. As the simplest example, we can consider the case of analytical expression of the quadratic regularization $\widehat{\mathbf{f}}={(\mathbf{H}{\mathbf{H}}^t+\lambda \mathbf{D}{\mathbf{D}}^t)}^{-1}{\mathbf{H}}^t\mathbf{g}=\mathbf{B}{\mathbf{H}}^t\mathbf{g}$, which suggests having a two-layer network with a fixed part structure ${\mathbf{H}}^t$ and a trainable one $\mathbf{B}={(\mathbf{H}{\mathbf{H}}^t+\lambda \mathbf{D}{\mathbf{D}}^t)}^{-1}$. See Figure 6.

It is interesting to note that in X-ray-computed tomography (CT), the forward operator $\mathbf{H}$ is called projection, the adjoint operator ${\mathbf{H}}^t$ is called back projection (BP), and the $\mathbf{B}$ operator is assimilated to a 2D filtering (convolution).

6. DL Structure and Deterministic or Bayesian Computation

To be able to look at the DNN and analyze it either in a deterministic or Bayesian manner, let us come back to the general notations and consider the following NN with input $\mathbf{x}$, output $\mathbf{y}$, and intermediate hidden variables ${\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_L$:

$$\mathbf{x}\to \boxed{\mathrm{first}\mathrm{layer}}\to {\mathbf{z}}_1\to \boxed{\mathrm{second}\mathrm{layer}}\to {\mathbf{z}}_2\cdots \to \boxed{\mathrm{last}\mathrm{layer}}\to \mathbf{y}.$$

In the deterministic case, each layer is defined by its parameters $(W_l,b_l)$:

$$\mathbf{x}\to \boxed{f_{W_0,b_0}\left(\mathbf{x}\right)}\to {\mathbf{z}}_1\to \boxed{f_{W_1,b_1}\left({\mathbf{z}}_1\right)}\to {\mathbf{z}}_2\cdots \to \boxed{f_{W_L,b_L}\left({\mathbf{z}}_{L-1}\right)}\to \mathbf{y},$$

and we can write:

$$\mathbf{y}=f_{\mathit{\theta }}\left(\mathbf{x}\right)=(f_{(W_0,b_0)}\odot f_{(W_1,b_1)}\odot \cdots f_{(W_L,b_L)})\left(\mathbf{x}\right)$$

with

$$\mathit{\theta }=((W_0,b_0),(W_1,b_1),\cdots ,(W_L,b_L))$$

and

$$\mathbf{y}=f_L(W_L{\mathbf{z}}_{L-1}+b_L),{\mathbf{z}}_l=f_l(W_l{\mathbf{z}}_{l-1}+b_l),l=L,\cdots ,1,{\mathbf{z}}_0=\mathbf{x}.$$

(18)

6.1. Deterministic DL Computation

In general, during the training steps, the parameters $\mathit{\theta }$ are estimated via

$${\mathit{\theta }}^{*}=\arg \underset{\mathit{\theta }}{min}\left\{\sum _i\mathit{\ell }(y_i,f_{\mathit{\theta }}\left({\mathbf{x}}_i\right))+\sum _k{\lambda }_k{\varphi }_k\left(W_k,b_k\right)\right\}.$$

(19)

The main point here is to choose how to choose ${\lambda }_k$ and $\varphi (.)$, as well as which optimization algorithm to choose for better convergence.

When parameters are obtained (the model is trained), we can use it easily via

$${\mathbf{z}}_0=\mathbf{x},{\mathbf{z}}_l=f_l(W_l{\mathbf{z}}_{l-1}+b_l),\mathbf{y}=f_L(W_L{\mathbf{z}}_{L-1}+b_L).$$

(20)

6.2. Bayesian Deep Neural Network

In Bayesian DL, the question of choosing ${\lambda }_k$ and $\varphi (.)$ in the previous case becomes the choice of the prior, $p\left(\mathit{\theta }\right)$, which can also be assumed to be a priori separable in the components of $\mathit{\theta }$ or not. Then, we have to choose the expression of the likelihood (in general Gaussian) and find the expression of the posterior $p\left(\mathit{\theta }\right|\mathcal{D})$. As explained extensively before, directly using this posterior is almost impossible. Hopefully, we have a great number of approximate computation methods, such as MCMC sampling, slice sampling, nested sampling, data augmentation, and variational inference, which can still be used in practical situations. However, the training step in the Bayesian DL still stays very costly, particularly if we want to quantify the uncertainties.

Prediction Step

In the prediction step, we again have to consider choosing a probability law $p\left(\mathbf{x}\right)$ for the class of the possible inputs and for all the outputs ${\mathbf{z}}_l$ that are conditional to their inputs ${\mathbf{z}}_{l-1}$. Then, we can consider, for example, the Gibbs sampling scheme. A comparison between deterministic and Bayesian DL is shown here:

$$\begin{array}{ccc}\mathrm{Deterministic}:\hfill & & \mathrm{Bayesian}:\hfill \\ \left\{\begin{array}{c}{\mathbf{z}}_0=\mathbf{x},\hfill \\ {\mathbf{z}}_l=f_l(W_l{\mathbf{z}}_{l-1}+b_l),l=1,\cdots ,L,\hfill \\ \mathbf{y}=f_L(W_L{\mathbf{z}}_{L-1}+b_L).\hfill \end{array}\right.\hfill & \to & \left\{\begin{array}{c}{\mathbf{z}}_0\sim p\left(\mathbf{x}\right),\hfill \\ {\mathbf{z}}_l\sim p\left({\mathbf{z}}_l\right|{\mathbf{z}}_{l-1}),l=1,\cdots ,L,\hfill \\ \mathbf{y}\sim p\left(\mathbf{y}\right|{\mathbf{z}}_{L-1}).\hfill \end{array}\right.\hfill \end{array}$$

If we consider Gaussian laws for the input and all the conditional variables, then we can write the following:

$${\mathbf{z}}_0\sim N\left({\mathbf{z}}_0\right|\mathbf{x},{\tau }_0\mathbf{I}),{\mathbf{z}}_l\sim N\left({\mathbf{z}}_l\right|f_l\left({\mathbf{z}}_{l-1}\right),{\tau }_l\mathbf{I}),l=1,\cdots ,L,\mathbf{y}\sim p\left(\mathbf{y}\right|{\mathbf{z}}_{L-1}).$$

(21)

Here too, the main difficulty occurs when there are nonlinear activation functions, particularly in the last layer, where the Gaussian approximation may not be more valid.

7. Application: Infrared Imaging

Infrared (IR) imaging is used to diagnose and to survey the temperature field distribution of sensitive objects in many industrial applications. These images are, in general, low resolution and very noisy. The real values of the temperature also depend on many other parameters, such as emissivity, attenuation, and diffusion, due to the distance of the camera to the sources. To be really useful in practice, we need to reduce the noise, calibrate and increase the resolution, segment for detection of the hot area, and finally survey the temperature values of the different areas during the time to be able to conduct preventive diagnosis and possible maintenance.

Reducing the noise can be accomplished by filtering using the Fourier transform, wavelet transform, or other sparse representations of images. To increase the resolution, we may use deconvolution methods if we can obtain the point spread function (PSF) of the camera, or if not by using blind deconvolution techniques. The segmentation and detection of the hot area and the temperature value estimation at each area are also very important steps in real applications. Any of these steps can be performed separately, but trying to propose global processing using DL or BDL is necessary for real applications. As any of these steps are in fact different inverse problems, and it is difficult to fix the parameters in each step in a robust way, we propose a global process using the BDL. See the global scheme in Figure 7.

In the first step, as the final objective is to segment the image to obtain different levels of temperature (for example, four levels: background, normal, high, and very high), we propose to design an NN that obtains as input a low resolution and noisy image and outputs a segmented image with those four levels and, at the same time, a good estimate of the temperature at each segment. See Figure 8.

To train this NN, we can generate different known shaped images to consider as the ground truth and simulate the blurring effects of temperature diffusions via the convolution of different appropriate PSFs. We can also add some noise to generate realistic images. We can also use black body thermal sources and acquire different images at different conditions. All these images can be used for the training of the network. See an example of the obtained result in Figure 9.