An Overview of Neural Network Methods for Predicting Uncertainty in Atmospheric Remote Sensing

Abstract

In this paper, we present neural network methods for predicting uncertainty in atmospheric remote sensing. These include methods for solving the direct and the inverse problem in a Bayesian framework. In the first case, a method based on a neural network for simulating the radiative transfer model and a Bayesian approach for solving the inverse problem is proposed. In the second case, (i) a neural network, in which the output is the convolution of the output for a noise-free input with the input noise distribution; and (ii) a Bayesian deep learning framework that predicts input aleatoric and model uncertainties, are designed. In addition, a neural network that uses assumed density filtering and interval arithmetic to compute uncertainty is employed for testing purposes. The accuracy and the precision of the methods are analyzed by considering the retrieval of cloud parameters from radiances measured by the Earth Polychromatic Imaging Camera (EPIC) onboard the Deep Space Climate Observatory (DSCOVR).

1. Introduction

In atmospheric remote sensing, the retrieval of atmospheric parameters is an inverse problem that is usually ill posed. Due to its ill posedness, measurement errors can lead to large errors in the retrieved quantities. It is therefore desirable to characterize the retrieved value by an estimate of uncertainty describing a range of values that probably produce a measurement [1].

The retrieval algorithms are mostly based on deterministic or stochastic optimization methods. From the first category, the method of Tikhonov regularization and iterative regularization methods deserve to be mentioned, while from the second category, the Bayesian methods are the most representative. The Bayesian framework [2,3] provides an efficient way to deal with the ill-posedness of the inverse problem and its uncertainties. In this case, the solution of the inverse problem is given by the a posteriori distribution (the conditional distribution of the retrieval quantity given the measurement) that accounts for all assumed retrieval uncertainties. Under the assumptions that (i) the a priori knowledge and the measurement uncertainty are both Gaussian distributions, and (ii) the forward model is moderately nonlinear, the a posteriori distribution is approximately Gaussian with a covariance matrix that can be analytically calculated. However, even neglecting the validity of these assumptions, the method is not efficient for the operational processing of large data volumes. The reason is that the computations of the forward model and its Jacobian are expensive computational processes.

Compared to Bayesian methods, neural networks are powerful tools for the design of efficient retrieval algorithms. Their capability to approximate any continuous function on a compact set to an arbitrary accuracy makes them well suited to approximate the input–output function represented by a radiative transfer model. An additional feature is that the derivatives of a neural network model with respect to its inputs can be analytically computed. Thus, a neural network algorithm can produce, in addition to the approximation of the radiative quantities of interest, also an estimation of their derivatives with respect to the model inputs. While the training of a neural network may require a significant amount of time, a trained neural network may deliver accurate predictions of the forward model and its Jacobian, typically in a fraction of a millisecond.

Neural networks were initially developed for the emulation of radiative transfer models [4,5,6,7,8] and subsequently for atmospheric remote sensing. The latter include:

• Neural networks that are used to emulate the forward model and are applied in conjunction with a Bayesian approach to solve the inverse problem [9,10,11,12];
• Neural networks that are developed to directly learn the retrieval mappings from data [13,14,15,16,17,18,19,20]; and
• Neural networks that are designed to directly invert the atmospheric parameters of interest which are then used as initial values in an optimization algorithm, e.g., the method of Tikhonov regularization [21,22].

In the first case, the total uncertainty sums up the contributions of the uncertainties in the data and in the neural network model, whereby the model uncertainty is computed from the statistics of the errors over the data set. However, the retrieval algorithm is still based on the assumption that the forward model is nearly linear, which is generally not true. In the second case, the probabilistic character of the inverse problem is neglected and uncertainty estimates are provided as mean errors computed over the data set; this is a disadvantage compared to Bayesian methods. Remarkable exceptions to the approaches listed above are the works of Aires [23,24], in which a Bayesian framework in conjunction with the Laplace approximation was used to model the retrieval errors (estimated from the error covariance matrix observed on the data set), and of Pfreundschuh et al. [25], in which a quantile regression neural network was designed. Unfortunately, the Laplace approximation is only suitable for small training sets and simple networks, while quantile regression is suitable for the retrieval of scalar quantities (it is unclear whether a reasonable approximation of the quantile contours of the joint a posteriori distribution can be obtained).

This paper is devoted to an overview of neural networks methods for predicting uncertainty in atmospheric remote sensing. In addition to the method based on a neural network for simulating the radiative transfer model and a Bayesian approach for solving the inverse problem (Case 1 above), methods relying on Bayesian networks are described. These methods, in which the network activations and weights can be modeled by parametric probability distributions, are standard tools for uncertainty predictions in deep neural networks, and can be applied to nonlinear retrieval problems, and to the best knowledge of the authors, have not yet been used in atmospheric remote sensing. The paper, which is merely of pedagogical nature, is organized as follows. In Section 1, we present the theoretical background of this study. This is then used in Section 2 to develop several neural network methods and apply them to a specific cloud parameters retrieval problem. Section 3 contains a few concluding remarks.

2. Theoretical Background

We consider a generic model $\mathbf{y}=\mathbf{F}(\mathbf{x}),$ where $\mathbf{x}\in {\mathbb{R}}^{N_{\mathrm{x}}}$ is the input vector, $\mathbf{F}$ is some deterministic function and $\mathbf{y}\in {\mathbb{R}}^{N_{\mathrm{y}}}$ is the output vector. For an atmosphere characterized by a set of state parameters, the signal measured by an instrument at different wavelengths can be computed by a radiative transfer model $\mathbf{R}$. Specifically, we will refer to the measurement signals as data, and split the vector of state parameters in (i) the vector of atmospheric parameters that are intended to be retrieved, and (ii) the vector of atmospheric parameters that are known to have some accuracy, but are not included in the retrieval (forward model parameters). In this study, we will use neural networks to model the radiative transfer function $\mathbf{R}$, as well as its inverse ${\mathbf{R}}^{-1}$. In this regard and in order to simplify the notation, we will consider two cases. In the first case, referred to as the direct problem, the input $\mathbf{x}$ is the set of atmospheric parameters, the output $\mathbf{y}$ is the set of data and the forward model $\mathbf{F}$ coincides with the radiative transfer model $\mathbf{R}$, while in the second case, referred to as the inverse problem, the situation is reversed: the input $\mathbf{x}$ includes the sets of data and forward model parameters, while the output $\mathbf{y}$ includes the set of atmospheric parameters to be retrieved (in the absence of forward model parameters, the forward model $\mathbf{F}$ reproduces the inverse of the radiative transfer model ${\mathbf{R}}^{-1}$).

In machine learning, the task is to approximate $\mathbf{F}(\mathbf{x})$ by a neural network model $\mathbf{f}(\mathbf{x},\mathit{\omega })$ characterized by a set of parameters $\mathit{\omega }$ [26,27]. For doing this, we consider a set of inputs $\mathbf{X}={\left\{{\mathbf{x}}^{(n)}\right\}}_{n=1}^N$ and a corresponding set of outputs $\mathbf{Y}={\left\{{\mathbf{y}}^{(n)}\right\}}_{n=1}^N$, given by ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})$, where N is the number of samples. In a regression problem, $\mathcal{D}={\left\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$ forms a data set—or more precisely, a training set—from which the neural network model $\mathbf{f}(\mathbf{x},\mathit{\omega })$ can be inferred. Traditional neural networks are comprised of units or nodes arranged in an input layer, an output layer, and a number of hidden layers situated between the input and output layers. Let $L+1$ be the number of layers, and $N_l$ be the number of units in layer l, where $l=0,\dots ,L$. The input layer corresponds to $l=0$, the output layer to $l=L$, so that $N_{\mathrm{x}}=N_0$ and $N_{\mathrm{y}}=N_L$. In feed-forward networks, the signals $y_{i,l-1}$ from units $i=1,\dots ,N_{l-1}$ in layer $l-1$ are multiplied by a set of weights $w_{ji,l}$, $j=1,\dots ,N_l$, $i=1,\dots ,N_{l-1}$, and then summed and combined with a bias $b_{j,l}$, $j=1,\dots ,N_l$. This calculation forms the pre-activation signal $u_{j,l}={\sum }_{i=1}^{N_{l-1}}{w}_{ji,l}{y}_{i,l-1}+b_{j,l}$ which is transformed by the layer activation function $g_l$ to form the activation signal $y_{j,l}$ of unit $j=1,\dots ,N_l$ in layer l. Defining the matrix of weights ${\mathsf{W}}_l\in {\mathbb{R}}^{N_l\times N_{l-1}}$ and the vector of biases ${\mathbf{b}}_l\in {\mathbb{R}}^{N_l}$ by ${\left[{\mathsf{W}}_l\right]}_{ji}=w_{ji,l}$ and ${\left[{\mathbf{b}}_l\right]}_j=b_{j,l}$, respectively, and letting $\mathit{\omega }={\{{\mathsf{W}}_l,{\mathbf{b}}_l\}}_{l=1}^L$ be the set of network parameters, the feed-forward operations can be written in matrix form as

$$\begin{array}{cc}\hfill {\mathbf{y}}_0& =\mathbf{x},\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{u}}_l& ={\mathsf{W}}_l{\mathbf{y}}_{l-1}+{\mathbf{b}}_l,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\mathbf{y}}_l& =g_l({\mathbf{u}}_l),l=1,\dots ,L,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \mathbf{f}(\mathbf{x},\mathit{\omega })& ={\mathbf{y}}_L,\hfill \end{array}$$

(4)

where ${\left[{\mathbf{y}}_l\right]}_i=y_{i,l}$ and ${\left[{\mathbf{u}}_l\right]}_j=u_{j,l}$. Deep learning is the process of regressing the network parameters $\mathit{\omega }$ on the data $\mathcal{D}$. The standard procedure is to compute a point estimate $\widehat{\mathit{\omega }}$ as the minimizer of some loss function by using the back-propagation algorithm [28]. In a stochastic framework, the loss function is usually defined as the log likelihood of the data set, with an eventual regularization term to penalize the network parameters. From a statistical point of view, this procedure is equivalent to a maximum a posteriori (MAP) estimation when regularization is used, and maximum likelihood estimation (MLE) when this is not the case.

In this section, we review the basic theory which serves as a basis for the development of different neural network architectures. In particular, we describe (i) the methodology for computing point estimates; (ii) the different types of uncertainty; and (iii) Bayesian networks.

2.1. Point Estimates

A data model with output noise is given by

$$\begin{array}{cc}\hfill \mathbf{y}& =\mathbf{f}(\mathbf{x},\mathit{\omega })+{\delta }_{\mathbf{y}},\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\delta }_{\mathbf{y}}& \sim \mathcal{N}(\mathbf{0},{\mathsf{C}}_{\mathbf{y}}^{\delta }),\hfill \end{array}$$

(6)

where here and in the following, the notation $\mathcal{N}(\mathbf{x};\overline{\mathbf{x}},{\mathsf{C}}_{\mathbf{x}})$, or more simply and when no confusion arises, $\mathcal{N}(\overline{\mathbf{x}},{\mathsf{C}}_{\mathbf{x}})$ stands for a Gaussian distribution with the mean $\overline{\mathbf{x}}$ and covariance matrix ${\mathsf{C}}_{\mathbf{x}}$. When the true input $\mathbf{x}$ is hidden (so that it cannot be observed) but samples from a random vector $\mathbf{z}=\mathbf{x}+{\delta }_{\mathbf{x}}$ with ${\delta }_{\mathbf{x}}\sim \mathcal{N}(\mathbf{0},{\mathsf{C}}_{\mathbf{x}}^{\delta })$, i.e., $p(\mathbf{z}|\mathbf{x})=\mathcal{N}(\mathbf{z};\mathbf{x},{\mathsf{C}}_{\mathbf{x}}^{\delta })$ are available, the pertinent model is the data model with input and output noise, that is:

$$\begin{array}{cc}\hfill \mathbf{y}& =\mathbf{f}(\mathbf{z},\mathit{\omega })+{\Delta }_{\mathbf{y}},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\Delta }_{\mathbf{y}}& \sim \mathcal{N}(\mathbf{0},{\mathcal{C}}_{\mathrm{y}}^{\delta }).\hfill \end{array}$$

(8)

The error ${\Delta }_{\mathbf{y}}$ sums up the contributions of the output error and of the input error that propagates through the network in the output space. Specifically, when the noise process in the input space is small and the linearization:

$$\begin{array}{cc}\hfill \mathbf{f}(\mathbf{x},\mathit{\omega })& =\mathbf{f}(\mathbf{z},\mathit{\omega })+{\mathsf{K}}_{\mathbf{x}}(\mathbf{z},\mathit{\omega })(\mathbf{x}-\mathbf{z}),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill {\mathsf{K}}_{\mathbf{x}}(\mathbf{z},\mathit{\omega })& =\frac{\partial \mathbf{f}}{\partial \mathbf{x}}(\mathbf{z},\mathit{\omega }),\hfill \end{array}$$

(10)

is assumed, we find (cf. Equations (5), (7), and (9)) ${\Delta }_{\mathbf{y}}={\delta }_{\mathbf{y}}+{\mathsf{K}}_{\mathbf{x}}(\mathbf{z},\mathit{\omega })(\mathbf{x}-\mathbf{z})$, and further:

$${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })={\mathsf{C}}_{\mathbf{y}}^{\delta }+{\mathsf{K}}_{\mathbf{x}}(\mathbf{z},\mathit{\omega }){\mathsf{C}}_{\mathbf{x}}^{\delta }{\mathsf{K}}_{\mathbf{x}}^T(\mathbf{z},\mathit{\omega }).$$

(11)

To design a neural network, we consider a data set $\mathcal{D}$ associated to each data model, meaning that:

• An exact data set $\mathcal{D}={\left\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$, where ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})$;
• A data set with output noise $\mathcal{D}={\left\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$, where ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})+{\delta }_{\mathbf{y}}$; and
• A data set with input and output noise $\mathcal{D}={\left\{({\mathbf{z}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$, where ${\mathbf{z}}^{(n)}={\mathbf{x}}^{(n)}+{\delta }_{\mathbf{x}}$ and ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})+{\delta }_{\mathbf{y}}$.

In a stochastic framework, a neural network can be regarded as a probabilistic model $p(\mathbf{y}|\mathbf{z},\mathit{\omega })$; given an observable input $\mathbf{z}$ and a set of parameters $\mathit{\omega }$, a neural network assigns a probability to each possible output $\mathbf{y}$. In view of Equations (7) and (8), the a priori confidence in the predictive power of the model is given by

$$p(\mathbf{y}|\mathbf{z},\mathit{\omega })=\mathcal{N}(\mathbf{y};\mathbf{f}(\mathbf{z},\mathit{\omega }),{\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })).$$

(12)

The process of learning from the data $\mathcal{D}$ can be described by the posterior $p(\mathit{\omega }|\mathcal{D})=p(\mathit{\omega }|\mathbf{Z},\mathbf{Y})$, which represents the Bayes plausibility for the parameters $\mathit{\omega }$ given the data $\mathcal{D}$. This can be estimated by using the Bayesian rule:

$$\begin{array}{cc}\hfill p(\mathit{\omega }|\mathcal{D})& =\frac{p(\mathcal{D}|\mathit{\omega })p(\mathit{\omega })}{p(\mathcal{D})}\propto p(\mathcal{D}|\mathit{\omega })p(\mathit{\omega })\propto exp[-E(\mathit{\omega })],\hfill \end{array}$$

(13)

where $p(\mathcal{D}|\mathit{\omega })$ is the likelihood or the probability of the data, $p(\mathit{\omega })$ is the prior over the network parameters, $p(\mathcal{D})=\int p(\mathcal{D}|\mathit{\omega })p(\mathit{\omega })\mathrm{d}\mathit{\omega }$ the evidence, and:

$$E(\mathit{\omega })=E_{\mathrm{D}}(\mathit{\omega })+E_{\mathrm{R}}(\mathit{\omega })$$

(14)

is the loss function. The first term $E_{\mathrm{D}}(\mathit{\omega })$ in the expression of the loss function $E(\mathit{\omega })$ is the contribution from the likelihood $p(\mathcal{D}|\mathit{\omega })$, written as the product (cf. Equation (12)):

$$\begin{array}{cc}\hfill p(\mathcal{D}|\mathit{\omega })& =p(\mathbf{Y}|\mathbf{Z},\mathit{\omega })=\prod _{n=1}^{N}p({\mathbf{y}}^{(n)}|{\mathbf{z}}^{(n)},\mathit{\omega })\propto exp[-E_{\mathrm{D}}(\mathit{\omega })],\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill E_{\mathrm{D}}(\mathit{\omega })& =\frac{1}{2}\sum _{n=1}^N{[{\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{z}}^{(n)},\mathit{\omega })]}^T{\left[{\mathcal{C}}_{\mathbf{y}}^{\delta }({\mathbf{z}}^{(n)},\mathit{\omega })\right]}^{-1}[{\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{z}}^{(n)},\mathit{\omega })],\hfill \end{array}$$

(16)

while the second term $E_{\mathrm{R}}(\mathit{\omega })$ is the contribution from the prior $p(\mathit{\omega })$, chosen for example, as the Gaussian distribution:

$$\begin{array}{cc}\hfill p(\mathit{\omega })& =\mathcal{N}(\mathit{\omega };\mathbf{0},{\mathsf{C}}_{\mathit{\omega }})\propto exp[-E_{\mathrm{R}}(\mathit{\omega })],\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill E_{\mathrm{R}}(\mathit{\omega })& =\frac{1}{2}{\mathit{\omega }}^T{\mathsf{C}}_{\mathit{\omega }}^{-1}\mathit{\omega }.\hfill \end{array}$$

(18)

In this regard, point estimates with regularization are computed by maximizing the posterior $p(\mathit{\omega }|\mathcal{D})$:

$$\begin{array}{cc}\hfill \widehat{\mathit{\omega }}={\mathit{\omega }}_{\mathrm{MAP}}& =arg\underset{\mathit{\omega }}{max}logp(\mathit{\omega }|\mathcal{D})=arg\underset{\mathit{\omega }}{min}E(\mathit{\omega }),\hfill \end{array}$$

(19)

while point estimates without regularization are computed by maximizing the likelihood $p(\mathcal{D}|\mathit{\omega })$:

$$\begin{array}{cc}\hfill \widehat{\mathit{\omega }}={\mathit{\omega }}_{\mathrm{MLE}}& =arg\underset{\mathit{\omega }}{max}logp(\mathcal{D}|\mathit{\omega })=arg\underset{\mathit{\omega }}{min}{E}_{\mathrm{D}}(\mathit{\omega }).\hfill \end{array}$$

(20)

Some comments are in order.

• For a data model with output noise, we have $\mathbf{z}=\mathbf{x},$ ${\mathcal{C}}_{\mathrm{y}}^{\delta }={\mathsf{C}}_{\mathbf{y}}^{\delta }$, and $\mathcal{D}={\left\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$ with ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})+{\delta }_{\mathbf{y}}$. Moreover, for the covariance matrix ${\mathsf{C}}_{\mathbf{y}}^{\delta }={\sigma }_{\mathbf{y}}^2\mathsf{I}$, where $\mathsf{I}$ is the identity matrix, we find:

  $$E_{\mathrm{D}}(\mathit{\omega })=\sum _{n=1}^N\frac{1}{2{\sigma }_{\mathbf{y}}^2}||{\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{z}}^{(n)},\mathit{\omega }){||}^2,$$

  (21)

  or more precisely:

  $$E_{\mathrm{D}}(\mathit{\omega })=\sum _{n=1}^N\left[\frac{1}{2{\sigma }_{\mathbf{y}}^2}||{\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{z}}^{(n)},\mathit{\omega }){||}^2+\frac{N_{\mathrm{y}}}{2}log{\sigma }_{\mathbf{y}}^2\right],$$

  (22)
  Assuming ${\mathsf{C}}_{\mathit{\omega }}={\sigma }_{\mathit{\omega }}^2\mathsf{I}$ and using Equation (21), we infer that the point estimate $\widehat{\mathit{\omega }}={\mathit{\omega }}_{\mathrm{MAP}}$ is the minimizer of the Tikhonov function:

  $$\begin{array}{cc}\hfill E(\mathit{\omega })& =\frac{1}{2}\sum _{n=1}^N||{\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{x}}^{(n)},\mathit{\omega }){||}^2+{\alpha ||\mathit{\omega }||}^2,\hfill \end{array}$$

  (23)

  where $\alpha ={\sigma }_{\mathbf{y}}^2/(2{\sigma }_{\mathit{\omega }}^2)$ is the regularization parameter.
• A model with exact data can be handled by considering the data model with output noise and by making ${\sigma }_{\mathbf{y}}^2\approx 0$ in the representation of the data error covariance matrix ${\mathsf{C}}_{\mathbf{y}}^{\delta }={\sigma }_{\mathbf{y}}^2\mathsf{I}$. For ${\sigma }_{\mathbf{y}}^2\approx 0$, the relation ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})+{\delta }_{\mathbf{y}}$ yields ${\mathbf{y}}^{(n)}\approx \mathbf{F}({\mathbf{x}}^{(n)})$, while Equation (23) and the relation $\alpha ={\sigma }_{\mathbf{y}}^2/(2{\sigma }_{\mathit{\omega }}^2)\approx 0$ gives:

  $$\begin{array}{cc}\hfill \widehat{\mathit{\omega }}& \approx {\mathit{\omega }}_{\mathrm{MLE}}=arg\underset{\mathit{\omega }}{min}E(\mathit{\omega }),\hfill \\ \hfill E(\mathit{\omega })& =\frac{1}{2}\sum _{n=1}^N||{\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{x}}^{(n)},\mathit{\omega }){]||}^2.\hfill \end{array}$$
  Thus, when learning a neural network with the exact data, the maximum likelihood estimate minimizes the sum of square errors. Note that in this case, regularization is not absolutely required, because the output data are exact.
• For a data model with input and output noise, the computation of the estimate $\widehat{\mathit{\omega }}$ is not a trivial task, because the covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega }),$ which enters in the expression of $E_{\mathrm{D}}(\mathit{\omega })$, depends on $\mathit{\omega }$. Moreover, in Equation (11), ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })$ corresponds to a linearization of the neural network function under the assumption that the noise process in the input space is small. This problem can be solved by implicitly learning the covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })$ from the loss function [29]. Specifically, we assume that ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })$ is a diagonal matrix with entries ${\sigma }_j^2(\mathbf{z},\mathit{\omega })$, that is:

  $${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })=\mathrm{diag}{\left[{\sigma }_j^2(\mathbf{z},\mathit{\omega })\right]}_{j=1}^{N_{\mathrm{y}}},$$

  (24)

  implying:

  $$logp(\mathbf{y}|\mathbf{z},\mathit{\omega })\propto -\sum _{j=1}^{N_{\mathrm{y}}}\left\{\frac{1}{2{\sigma }_j^2(\mathbf{z},\mathit{\omega })}{[y_j-{\mu }_j(\mathbf{z},\mathit{\omega })]}^2+\frac{1}{2}log{\sigma }_j^2(\mathbf{z},\mathit{\omega })\right\}.$$

  (25)
  Here, we identified $\mathit{\mu }\equiv \mathbf{f}$, and set ${\mu }_j={\left[\mathit{\mu }\right]}_j$ and $y_j={\left[\mathbf{y}\right]}_j$. To learn the variances ${\sigma }_j^2(\mathbf{z},\mathit{\omega })$ from the loss function, we use a single network with input $\mathbf{z}$, and output $[{\mu }_j(\mathbf{z},\mathit{\omega }),{\sigma }_j^2(\mathbf{z},\mathit{\omega })]\in {\mathbb{R}}^{2N_{\mathrm{y}}}$; thus, in the output layer, $N_{\mathrm{y}}$ units are used to predict ${\mu }_j$ and $N_{\mathrm{y}}$ units to predict ${\sigma }_j^2$. In practice, to increase numerical stability, we train the network to predict the log variance ${\rho }_j=log{\sigma }_j^2$, in which case, the likelihood loss function is:

  $$\begin{array}{cc}\hfill E_{\mathrm{D}}(\mathit{\omega })& =\sum _{n=1}^N{E}_{\mathrm{D}}^{(n)}(\mathit{\omega }),\hfill \end{array}$$

  (26)

  with:

  $$\begin{array}{cc}\hfill E_{\mathrm{D}}^{(n)}(\mathit{\omega })& =\frac{1}{2}\sum _{j=1}^{N_{\mathrm{y}}}\{exp[-{\rho }_j({\mathbf{z}}^{(n)},\mathit{\omega })]{[y_j^{(n)}-{\mu }_j({\mathbf{z}}^{(n)},\mathit{\omega })]}^2+{\rho }_j({\mathbf{z}}^{(n)},\mathit{\omega })\}.\hfill \end{array}$$

  (27)
  It should be pointed out that from Equation (25) that it is apparent that the model is stopped from predicting high uncertainty through the $log{\sigma }_j^2$ term, but also low uncertainty for points with high residual error, as low ${\sigma }_j^2$ will increase the contribution of the residual. On the other hand, it should also be noted that a basic assumption of the model is that the covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })$ is diagonal, which unfortunately, is contradicted by Equation (11).
• In order to generate a data set with input and output noise, that is, $\mathcal{D}={\left\{({\mathbf{z}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$, where ${\mathbf{z}}^{(n)}={\mathbf{x}}^{(n)}+{\delta }_{\mathbf{x}}$, ${\delta }_{\mathbf{x}}\sim \mathcal{N}(\mathbf{0},{\mathsf{C}}_{\mathbf{x}}^{\delta })$, and ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})+{\delta }_{\mathbf{y}}$, we used the jitter approach. According to this approach, at each forward pass through the network, a new random noise ${\delta }_{\mathbf{x}}$ is added to the original input vector ${\mathbf{x}}^{(n)}$. In other words, each time a training sample is passed through the network, random noise is added to the input variables, making them different every time it is passed through the network. By this approach, which is a simple form of data augmentation, the dimension of the data set is reduced (actually, the data set is $\mathcal{D}={\left\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{(n)})\right\}}_{n=1}^N$ with ${\mathbf{y}}^{(n)}=\mathbf{F}({\mathbf{x}}^{(n)})+{\delta }_{\mathbf{y}}$).

2.2. Uncertainties

In our analysis, we are interested in estimating the uncertainty associated with the underlying processes. The quantity which exactly quantifies the model’s uncertainty is the predictive distribution of an unknown output $\mathbf{y}$ given an observable input $\mathbf{z}$, defined by

$$p(\mathbf{y}|\mathbf{z},\mathcal{D})=\int p(\mathbf{y}|\mathbf{z},\mathit{\omega })p(\mathit{\omega }|\mathcal{D})\mathrm{d}\mathit{\omega }.$$

(28)

If $p(\mathbf{y}|\mathbf{z},\mathcal{D})$ is known, the first two moments of the output $\mathbf{y}$ can be computed as

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y})& =\int \mathbf{y}p(\mathbf{y}|\mathbf{z},\mathcal{D})\mathrm{d}\mathbf{y},\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y}{\mathbf{y}}^T)& =\int \mathbf{y}{\mathbf{y}}^{T}p(\mathbf{y}|\mathbf{z},\mathcal{D})\mathrm{d}\mathbf{y},\hfill \end{array}$$

(30)

and the covariance matrix as

$$\mathrm{Cov}(\mathbf{y})=\mathbb{E}(\mathbf{y}{\mathbf{y}}^T)-\mathbb{E}(\mathbf{y})\mathbb{E}{(\mathbf{y})}^T.$$

(31)

From Equation (28), we see that the predictive distribution $p(\mathbf{y}|\mathbf{z},\mathcal{D})$ can be computed if the Bayesian posterior $p(\mathit{\omega }|\mathcal{D})$ is known. Unfortunately, computing the distribution $p(\mathit{\omega }|\mathcal{D})$ by means of Equation (13) is usually an intractable problem, because computing the evidence $p(\mathcal{D})=\int p(\mathcal{D}|\mathit{\omega })p(\mathit{\omega })\mathrm{d}\mathit{\omega }$ is not a trivial task. To address this problem, either:

• The Laplace approximation, which yields an approximate representation for the posterior $p(\mathit{\omega }|\mathcal{D});$ or
• A variational inference approach, which learns a variational distribution $q_{\mathit{\theta }}(\mathit{\omega })$ to approximate the posterior $p(\mathit{\omega }|\mathcal{D})$,

can be used. In the first case, we are dealing with deterministic (point estimate) neural networks, in which a single realization of the network parameters $\mathit{\omega }$ is learned, while in the second case, we are dealing with stochastic neural networks, in which a distribution over the network parameters $\mathit{\omega }$ is learned.

The Laplace approximation is of theoretical interest because it provides explicit representations for the different types of uncertainty. In Appendix A it is shown that under the Laplace approximation, the predictive distribution is given by [23,24,30]

$$\begin{array}{cc}\hfill p(\mathbf{y}|\mathbf{z},\mathcal{D})& \propto exp\left\{-\frac{1}{2}{[\mathbf{y}-\mathbf{f}(\mathbf{z},\widehat{\mathit{\omega }})]}^T{\mathcal{C}}_{\mathbf{y}}^{-1}(\mathbf{z},\widehat{\mathit{\omega }})[\mathbf{z}-\mathbf{f}(\mathbf{z},\widehat{\mathit{\omega }})]\right\},\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill {\mathcal{C}}_{\mathbf{y}}(\mathbf{z},\widehat{\mathit{\omega }})& ={\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\widehat{\mathit{\omega }})+{\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{z},\widehat{\mathit{\omega }}),\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill {\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{z},\widehat{\mathit{\omega }})& ={\mathsf{K}}_{\mathit{\omega }}(\mathbf{z},\widehat{\mathit{\omega }}){\mathsf{H}}^{-1}(\widehat{\mathit{\omega }}){\mathsf{K}}_{\mathit{\omega }}^T(\mathbf{z},\widehat{\mathit{\omega }}),\hfill \end{array}$$

(34)

where:

$${\mathsf{K}}_{\mathit{\omega }}(\mathbf{z},\mathit{\omega })=\frac{\partial \mathbf{f}}{\partial \mathit{\omega }}(\mathbf{z},\mathit{\omega })$$

(35)

is the Jacobian of $\mathbf{f}$ with respect to $\mathit{\omega }$ and:

$${\left[\mathsf{H}(\widehat{\mathit{\omega }})\right]}_{ij}=\frac{\partial E}{\partial {\mathit{\omega }}_i\partial {\mathit{\omega }}_j}(\widehat{\mathit{\omega }}),$$

(36)

is the Hessian matrix of the loss function. Equation (32) provides a Gaussian approximation to the predictive distribution $p(\mathbf{y}|\mathbf{z},\mathcal{D})$ with mean $\mathbf{f}(\mathbf{z},\widehat{\mathit{\omega }})$ and covariance matrix ${\mathcal{C}}_{\mathbf{y}}(\mathbf{z},\widehat{\mathit{\omega }})$. From Equation (33), we see that the covariance matrix ${\mathcal{C}}_{\mathrm{y}}(\mathbf{z},\widehat{\mathit{\omega }})$ has two components.

• The first component ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\widehat{\mathit{\omega }})$ is the covariance matrix in the distribution over the error in the output $\mathbf{y}$. This term, which is input dependent, describes the so-called aleatoric heteroscedastic uncertainty measured by $p(\mathbf{y}|\mathbf{z},\mathit{\omega })$. For a data model with output noise, we have (cf. Equation (11)) ${\mathcal{C}}_{\mathbf{y}}^{\delta }={\mathsf{C}}_{\mathbf{y}}^{\delta }$, and this term, which is input independent, describes the so-called aleatoric homoscedastic uncertainty measured by $p(\mathbf{y}|\mathbf{x},\mathit{\omega })$.
• The second component ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{z},\widehat{\mathit{\omega }})$ reflects the uncertainty induced in the weights $\mathit{\omega }$, also called epistemic uncertainty or model uncertainty. The sources of this uncertainty measured by $p(\mathit{\omega }|\mathcal{D})$ are for example: (i) non-optimal hyperparameters (the number of hidden layers, the number of units per layer, the type of activation function); (ii) non-optimal training parameters (the minimum learning rate at which the training is stopped, the learning rate decay factor, the batch size used for the mini-batch gradient descent); and (iii) a non-optimal optimization algorithm (ADAGRAD, ADADELTA, ADAM, ADAMAX, NADAM).

or model uncertainty, which refers to the fact that we do not know the model that best explains the given data. For NNs, this is uncertainty from not knowing the best values of weights in all the trainable layers. This is often referred to as reducible uncertainty, because we can theoretically reduce this type of uncertainty by acquiring more data.

Some comments can be made here.

• The heteroscedastic covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\delta }(\mathbf{z},\mathit{\omega })=\mathrm{diag}{\left[{\sigma }_j^2(\mathbf{z},\mathit{\omega })\right]}_{j=1}^{N_{\mathrm{y}}}$ can be learned from the data by minimizing the loss functions (26) and (27).
• The computation of the epistemic covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{z},\widehat{\mathit{\omega }})$ from Equation (34) requires the knowledge of the Hessian matrix $\mathsf{H}(\widehat{\mathit{\omega }})$. In general, this problem is practically insoluble because the matrix $\mathsf{H}$ is very huge, and so, is very difficult to compute. However, the problem can be evaded by using the diagonal Hessian approximation

  $${\left[\mathsf{H}(\widehat{\mathit{\omega }})\right]}_{ij}={\delta }_{ij}\frac{{\partial }^{2}E}{\partial {\mathit{\omega }}_i^2}(\widehat{\mathit{\omega }}),$$

  where ${\delta }_{ij}$ is the Kronecker delta. The diagonal matrix elements can be very efficiently computed by using a procedure which is similar to the back-propagation algorithm used for computing the first derivatives [31].
• The covariance matrix ${\mathcal{C}}_{\mathbf{y}}(\mathbf{z},\widehat{\mathit{\omega }})$ can be approximated by the conditional average covariance matrix $\mathbb{E}({\mathcal{C}}_{\mathbf{y}}|\mathcal{D})$ of all network errors $\epsilon =\mathbf{y}-\mathbf{f}(\mathbf{z},\widehat{\mathit{\omega }})$ over the data set $\mathcal{D}$, meaning that:

  $$\mathbb{E}({\mathcal{C}}_{\mathrm{y}}|\mathcal{D})=\frac{1}{N}\sum _{n=1}^N[{\epsilon }^{(n)}-\mathbb{E}(\epsilon )]{[{\epsilon }^{(n)}-\mathbb{E}(\epsilon )]}^T,$$

  where ${\epsilon }^{(n)}={\mathbf{y}}^{(n)}-\mathbf{f}({\mathbf{z}}^{(n)},\widehat{\mathit{\omega }})$ and $\mathbb{E}(\epsilon )=(1/N){\sum }_{n=1}^N{\epsilon }^{(n)}$. Note that this is a very rough approximation, because in contrast to ${\mathcal{C}}_{\mathbf{y}}(\mathbf{z},\widehat{\mathit{\omega }})$, $\mathbb{E}({\mathcal{C}}_{\mathbf{y}}|\mathcal{D})$ does not depend on $\mathbf{z}$.
• For exact data, we have ${\mathcal{C}}_{\mathbf{y}}(\mathbf{x},\widehat{\mathit{\omega }})=B{\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{x},\widehat{\mathit{\omega }})$. Thus, when learning a neural network with exact data, the predictive distribution $p(\mathbf{y}|\mathbf{x},\mathcal{D})$ is Gaussian with mean $\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$ and (epistemic) covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{x},\widehat{\mathit{\omega }})$.

2.3. Bayesian Networks

Stochastic neural networks are a type of neural networks built by introducing stochastic components into the network (activation or weights). A Bayesian neural network can be regarded as a stochastic neural network trained by using Bayesian inference [32]. This type of neural network provides a natural approach to quantify uncertainty in deep learning and allows to distinguish between epistemic and aleatoric uncertainties.

Bayesian neural networks equipped with variational inference bypass the computation of the evidence $p(\mathcal{D})=\int p(\mathcal{D}|\mathit{\omega })p(\mathit{\omega })\mathrm{d}\mathit{\omega }$, which determines the Bayesian posterior $p(\mathit{\omega }|\mathcal{D})$ via Equation (13), by learning a variational distribution $q_{\mathit{\theta }}(\mathit{\omega })$ to approximate $p(\mathit{\omega }|\mathcal{D})$, that is:

$$q_{\mathit{\theta }}(\mathit{\omega })\approx p(\mathit{\omega }|\mathcal{D}),$$

(37)

where $\mathit{\theta }$ are some variational parameters. These parameters are computed by minimizing the Kullback–Leibler (KL) divergence:

$$\mathrm{KL}(q_{\mathit{\theta }}(\mathit{\omega })|p(\mathit{\omega }|\mathcal{D}))=\int q_{\mathit{\theta }}(\mathit{\omega })log\left[\frac{q_{\mathit{\theta }}(\mathit{\omega })}{p(\mathit{\omega }|\mathcal{D})}\right]\mathrm{d}\mathit{\omega },$$

(38)

with respect to $\mathit{\theta }$. In fact, the KL divergence is a measure of similarity between the approximate distribution $q_{\mathit{\theta }}(\mathit{\omega })$ and the posterior distribution obtained from the full Gaussian process $p(\mathit{\omega }|\mathcal{D})$, and minimizing the KL divergence to be the same as minimizing the variational free energy defined by

$$\begin{array}{cc}\hfill F(\mathit{\theta },\mathcal{D})& =-\int q_{\mathit{\theta }}(\mathit{\omega })logp(\mathcal{D}|\mathit{\omega })\mathrm{d}\mathit{\omega }+\mathrm{KL}(q_{\mathit{\theta }}(\mathit{\omega })|p(\mathit{\omega }))\hfill \\ \hfill & =\int q_{\mathit{\theta }}(\mathit{\omega })\left[log{q}_{\mathit{\theta }}(\mathit{\omega })-logp(\mathit{\omega })-logp(\mathcal{D}|\mathit{\omega })\right]\mathrm{d}\mathit{\omega }.\hfill \end{array}$$

(39)

Considering the data model with output noise, assuming ${\mathsf{C}}_{\mathbf{y}}^{\delta }={\sigma }_{\mathbf{y}}^2\mathsf{I}$, and replacing $p(\mathit{\omega }|\mathcal{D})$ by $q_{\mathit{\theta }}(\mathit{\omega })$ in Equation (28), which gives an approximate predictive distribution:

$$\begin{array}{cc}\hfill p(\mathbf{y}|\mathbf{x},\mathcal{D})& \approx q_{\mathit{\theta }}(\mathbf{y}|\mathbf{x})=\int p(\mathbf{y}|\mathbf{x},\mathit{\omega })q_{\mathit{\theta }}(\mathit{\omega })\mathrm{d}\mathit{\omega }\hfill \end{array}$$

(40)

which can be approximated at test time by

$$q_{\mathit{\theta }}(\mathbf{y}|\mathbf{x})\approx \frac{1}{T}\sum _{t=1}^{T}p(\mathbf{y}|\mathbf{x},{\mathit{\omega }}_t),$$

(41)

where ${\mathit{\omega }}_t$ is sampled from the distribution $q_{\mathit{\theta }}(\mathit{\omega })$. As a result, for $p(\mathbf{y}|\mathbf{x},\mathit{\omega })=\mathcal{N}(\mathbf{y},\mathbf{f}(\mathbf{x},\mathit{\omega }),$${\mathsf{C}}_{\mathbf{y}}^{\delta }={\sigma }_{\mathbf{y}}^2\mathsf{I})$, the predictive mean and the covariance matrix of the output $\mathbf{y}$ given the input $\mathbf{x}$, can be approximated, respectively, by (Appendix B):

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y})& \approx \frac{1}{T}\sum _{t=1}^T\mathbf{f}(\mathbf{x},{\mathit{\omega }}_t),\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill \mathrm{Cov}(\mathbf{y})& \approx {\sigma }_{\mathbf{y}}^2\mathsf{I}+\frac{1}{T}\sum _{t=1}^T\mathbf{f}(\mathbf{x},{\mathit{\omega }}_t)\mathbf{f}{(\mathbf{x},{\mathit{\omega }}_t)}^T-\mathbb{E}(\mathbf{y})\mathbb{E}{(\mathbf{y})}^T.\hfill \end{array}$$

(43)

The first term ${\sigma }_{\mathbf{y}}^2\mathsf{I}$ in Equation (43) corresponds to the homoscedastic uncertainty (the amount of noise in the data), while the second part corresponds to the epistemic uncertainty (how much the model is uncertain in its prediction). The predictive mean (42) is known as model averaging, and is equivalent to performing T stochastic passes through the network and averaging the results. Note that for exact data, i.e., ${\sigma }_{\mathbf{y}}^2\approx 0$, the covariance matrix simplifies to:

$$\begin{array}{cc}\hfill \mathrm{Cov}(\mathbf{y})& \approx \frac{1}{T}\sum _{t=1}^T\mathbf{f}(\mathbf{x},{\mathit{\omega }}_t)\mathbf{f}{(\mathbf{x},{\mathit{\omega }}_t)}^T-\mathbb{E}(\mathbf{y})\mathbb{E}{(\mathbf{y})}^T\hfill \end{array}$$

(44)

and describes the epistemic uncertainty.

In this section, we present the most relevant algorithm that is used for Bayesian inference (Bayes-by-backprop), as well as two Bayesian approximate methods. For this purpose, we consider a data model with output noise and assume ${\mathsf{C}}_{\mathbf{y}}^{\delta }={\sigma }_{\mathbf{y}}^2\mathsf{I}$; in this case, $p(\mathcal{D}|\mathit{\omega })$ is given by Equation (15) in conjunction with Equations (21) or (22).

2.3.1. Bayes by Backpropagation

Bayes-by-backprop is a practical implementation of variational inference combined with a reparameterization trick [33]. The idea of the parameterization trick is to introduce a random variable $ϵ$, and to determine $\mathit{\omega }$ by a deterministic transformation $t(ϵ,\mathit{\theta })$, such that $\mathit{\omega }=t(ϵ,\mathit{\theta })$ follows the distributions $q_{\mathit{\theta }}(\mathit{\omega })$. If the variational posterior $q_{\mathit{\theta }}(\mathit{\omega })$ is a Gaussian distribution with a diagonal covariance matrix, i.e., $q_{\mathit{\theta }}(\mathit{\omega })=\mathcal{N}(\mathit{\omega };{\mathit{\mu }}_{\omega },\mathrm{diag}{\left[{\sigma }_{\mathit{\omega }j}^2\right]}_{j=1}^W)$, where ${\sigma }_{\mathit{\omega }j}={\left[{\sigma }_{\mathit{\omega }}\right]}_j$ and $W=dim(\mathit{\omega })=dim({\mathit{\mu }}_{\mathit{\omega }})=dim({\sigma }_{\mathit{\omega }})$, a sample of the weight $\mathit{\omega }$ can be obtained by sampling a unit Gaussian $ϵ\sim \mathcal{N}(\mathbf{0},\mathsf{I})$, scaling it by a standard deviation ${\sigma }_{\mathit{\omega }}$ and by shifting a mean ${\mathit{\mu }}_{\mathit{\omega }}$. Actually, to guarantee that ${\sigma }_{\mathit{\omega }}$ is always non-negative, the standard deviation can be parametrized pointwise as ${\sigma }_{\mathit{\omega }}={\sigma }_{\mathit{\omega }}({\rho }_{\mathit{\omega }})=log(1+exp({\rho }_{\mathit{\omega }}))$ or as ${\sigma }_{\mathit{\omega }}={\sigma }_{\mathit{\omega }}({\rho }_{\mathit{\omega }})=exp({\rho }_{\mathit{\omega }}/2)$ (i.e., ${\rho }_{\mathit{\omega }j}=log{\sigma }_{\mathit{\omega }j}^2$). Thus, the variational posterior parameters are $\mathit{\theta }=({\mathit{\mu }}_{\mathit{\omega }},{\rho }_{\mathit{\omega }})$. By using a Monte Carlo sampling with one sample, we compute the variational free energy (39) by using the relations:

$$\begin{array}{cc}\hfill ϵ& \sim \mathcal{N}(\mathbf{0},\mathsf{I}),\hfill \\ \hfill \mathit{\omega }& ={\mathit{\mu }}_{\mathit{\omega }}+{\sigma }_{\mathit{\omega }}({\rho }_{\mathit{\omega }})\circ ϵ,\hfill \\ \hfill F(\mathit{\theta },\mathcal{D})& =log{q}_{\mathit{\theta }}(\mathit{\omega })-logp(\mathit{\omega })-logp(\mathcal{D}|\mathit{\omega }),\hfill \end{array}$$

where ∘ denotes point-wise multiplication. In Ref. [33], the prior over the weights $p(\mathit{\omega })$ is chosen as a mixture of two Gaussian with zero mean but differing variances, meaning that:

$$p(\mathit{\omega })=\beta \mathcal{N}(\mathit{\omega };0,{\sigma }_1^2\mathsf{I})+(1-\beta )\mathcal{N}(\mathit{\omega };\mathbf{0},{\sigma }_2^2\mathsf{I}),$$

(45)

while in [34], it is shown that, for $q_{\mathit{\theta }}(\mathit{\omega })=\mathcal{N}(\mathit{\omega };{\mathit{\mu }}_{\mathit{\omega }},\mathrm{diag}\left[{\sigma }_{\mathit{\omega }j}^2\right])$ and $p(\mathit{\omega })=\mathcal{N}(\mathit{\omega };\mathbf{0},\mathsf{I})$, the KL divergence $\mathrm{KL}(q_{\mathit{\theta }}(\mathit{\omega })|p(\mathit{\omega }))$ can be analytically computed; the result is:

$$\begin{array}{cc}\hfill \mathrm{KL}(q_{\mathit{\theta }}(\mathit{\omega })|p(\mathit{\omega }))& =\int q_{\mathit{\theta }}(\mathit{\omega })\left[log{q}_{\mathit{\theta }}(\mathit{\omega })-logp(\mathit{\omega })\right]\mathrm{d}\mathit{\omega }\hfill \\ \hfill & =\frac{1}{2}\sum _{j=1}^W[{\mu }_{\mathit{\omega }j}^2+{\sigma }_{\mathit{\omega }j}^2-log({\sigma }_{\mathit{\omega }j}^2)-1],\hfill \end{array}$$

(46)

where ${\mu }_{\mathit{\omega }j}={\left[{\mathit{\mu }}_{\mathit{\omega }}\right]}_j$, ${\sigma }_{\mathit{\omega }j}={\left[{\sigma }_{\mathit{\omega }}\right]}_j$. Thus:

$$F(\mathit{\theta },\mathcal{D})=-logp(\mathcal{D}|\mathit{\omega })+\frac{1}{2}\sum _{j=1}^W[{\mu }_{\mathit{\omega }j}^2+{\sigma }_{\mathit{\omega }j}^2-log({\sigma }_{\mathit{\omega }j}^2)-1].$$

(47)

After the training stage, i.e., after the variational posterior parameters $\mathit{\theta }=({\mathit{\mu }}_{\omega },{\mathit{\rho }}_{\omega })$ have been learned, we consider the set of samples ${\left\{{\mathit{\omega }}_t\right\}}_{t=1}^T$, where ${\mathit{\omega }}_t={\mathit{\mu }}_{\omega }+{\mathit{\sigma }}_{\omega }({\mathit{\rho }}_{\omega })\circ {\mathit{ϵ}}_t$ and ${\mathit{ϵ}}_t$ is sampled from the Gaussian distribution $\mathcal{N}(\mathbf{0},\mathsf{I})$, and compute the predictive mean and covariance matrix according to Equations (42) and (43).

2.3.2. Dropout

Dropout, which was initially proposed as a regularization technique during the training [35,36], is equivalent to a Bayesian approximation. This result was proved in [37] by showing that the variational free energy $F(\mathit{\theta },\mathcal{D})$ has the standard form representation of the dropout loss function (as the sum of a square loss function and a $L_2$ regularization term). A simplified proof of this assertion is given in Appendix C. The term “dropout” refers to removing a unit along with all its connections. The choice of a dropped unit is random. In the case of dropout, the feed-forward operation of a standard neural network (2) and (3) becomes:

$$\begin{array}{cc}\hfill {\mathbf{u}}_l& ={\mathsf{W}}_l{\mathsf{Z}}_{l-1}{\mathbf{y}}_{l-1}+{\mathbf{b}}_l,\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill {\mathbf{y}}_l& =g_l({\mathbf{u}}_l),\hfill \end{array}$$

(49)

where:

$$\begin{array}{cc}\hfill {\mathsf{Z}}_{l-1}& =\mathrm{diag}{\left[z_{k,l-1}\right]}_{k=1}^{N_{l-1}},\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill z_{k,l-1}& \sim \mathrm{Bernoulli}(p).\hfill \end{array}$$

(51)

Essentially, the output of the unit k in layer $l-1,$$y_{k,l-1}$ is multiplied by the binary variable $z_{k,l-1}$ to create the new output $z_{k,l-1}{y}_{k,l-1}$. The binary variable $z_{k,l-1}$ takes the value 1 with probability p and the value 0 with probability $1-p$; thus, if $z_{k,l-1}=0$, the new output is zero and the unit k is dropped out as an input to the next layer l. The same values of the binary variable are used in the backward pass when propagating the derivatives of the loss function. In the test time, the weights ${\mathsf{W}}_l$ are scaled as $p{\mathsf{W}}_l$. Thus, we retain units with probability p at training time, and multiply the weights by p at test time. Alternatively, in order to maintain constant outputs at training time, we can scale the weights by $1/p$, and do not modify the weights at test time. The model parameters $\mathit{\omega }={\{{\mathsf{W}}_l,{\mathbf{b}}_l\}}_{l=1}^L$ are obtained by minimizing the loss function (21) with possibly an $L_2$ regularization term.

Uncertainty can be obtained from a dropout neural network [37]. For the set of samples ${\left\{{\mathit{\omega }}_t\right\}}_{t=1}^T$ , where ${\mathit{\omega }}_t$ corresponds to a realization from the Bernoulli distribution ${\mathsf{Z}}_{l-1}^{(t)}$ and ${\mathsf{W}}_l^{(t)}={\mathsf{W}}_l{\mathsf{Z}}_{l-1}^{(t)}$ for all $l=1,\dots ,L$, the predictive mean and covariance matrix are computed by means of Equations (42) and (43).

2.3.3. Batch Normalization

Ioffe and Szegedy [38] introduced batch normalization as a technique for training deep neural networks that normalizes (standardizes) the inputs to a layer for each mini-batch. This allows for higher learning rates, regularizes the model, and reduces the number of training epochs. Moreover, Teye et al. [39] showed that a batch normalized network can be regarded as an approximate Bayesian model, and it can thus be used for modeling uncertainty.

Let us split the data set $\mathcal{D}$ into $N_{\mathrm{b}}$ mini-batches, and let the nth mini-batch ${\mathcal{B}}^{(n)}$ contain M pair samples $({\mathbf{x}}^{(n,m)},{\mathbf{y}}^{(n,m)})$, meaning that:

$$\begin{array}{cc}\hfill \mathcal{D}& ={\cup }_{n=1}^{N_{\mathrm{b}}}{\mathcal{B}}^{(n)},\hfill \\ \hfill {\mathcal{B}}^{(n)}& ={\left\{({\mathbf{x}}^{(n,m)},{\mathbf{y}}^{(n,m)})\right\}}_{m=1}^M.\hfill \end{array}$$

In batch normalization, the input $u_{j,l}^{(n,m)}={\sum }_{i=1}^{N_{l-1}}{w}_{ji,l}{y}_{i,l-1}^{(n,m)}$ of the activation function $g_l$ corresponding to unit j, layer l, sample m, and mini-batch n, which is first normalized:

$${\tilde{u}}_{j,l}^{(n,m)}={\displaystyle \frac{u_{j,l}^{(n,m)}-{\mu }_{j,l}^{(n)}}{\sqrt{v_{j,l}^{(n)}+\epsilon }}},$$

(52)

and then scaled and shifted:

$${\widehat{u}}_{j,l}^{(n,m)}={\alpha }_{j,l}{\tilde{u}}_{j,l}^{(n,m)}+{\beta }_{j,l},$$

(53)

where:

$${\mu }_{j,l}^{(n)}={\displaystyle \frac{1}{M}}\sum _{m=1}^M{u}_{j,l}^{(n,m)}\mathrm{and}{v}_{j,l}^{(n)}={\displaystyle \frac{1}{M}}\sum _{m=1}^M{(u_{j,l}^{(n,m)}-{\mu }_{j,l}^{(n)})}^2$$

(54)

are the mean and variance of the activation inputs over the M samples (in unit j, layer l and mini-batch n), respectively, ${\alpha }_{j,l}$ and ${\beta }_{j,l}$ are learnable model parameters, and $\epsilon$ is a small number added to the mini-batch variance to prevent division by zero. By normalization (or whitening), ${\tilde{u}}_{j,l}^{(n,m)}$ becomes a random variable with zero mean and unit variance, while by scaling and shifting, we guarantee that the transformation (53) can represent the identity transform. In a stochastic framework, we interpret the mean ${\mu }_{j,l}^{(n)}$ and variance $v_{j,l}^{(n)}$, corresponding to the nth mini-batch ${\mathcal{B}}^{(n)}$, as realizations of the random variables ${\mu }_{j,l}$ and $v_{j,l}$, corresponding to the data set $\mathcal{D}$. The model parameters include the learnable parameters:

$$\mathit{\theta }=\{w_{ji,l},{\alpha }_{j,l},{\beta }_{j,l}|j=1,\dots ,N_l,i=1,\dots ,N_{l-1},l=1,\dots ,L\},$$

and the stochastic parameters:

$$\mathit{\omega }=(\mathit{\mu },\mathit{v})=\{({\mu }_{j,l},v_{j,l})|j=1,\dots ,N_{l}l=1,\dots ,L\},$$

where for the mini-batch ${\mathcal{B}}^{(n)}$:

$${\mathit{\omega }}^{(n)}=({\mathit{\mu }}^{(n)},{\mathit{v}}^{(n)})=\{({\mu }_{j,l}^{(n)},v_{j,l}^{(n)})|j=1,\dots ,N_{l}l=1,\dots ,L\}$$

is a realization of $\mathit{\omega }=(\mathit{\mu },\mathit{v})$. Optimizing over mini-batches of size M instead on the full training set, the objective function for the mini-batch ${\mathcal{B}}^{(n)}$ becomes [39]:

$$F^{(n)}(\mathit{\theta })=\frac{1}{2M}\sum _{m=1}^M||{\mathbf{y}}^{(n,m)}-{\mathbf{f}}_{{\mathit{\omega }}^{(n)}}({\mathbf{x}}^{(n,m)},\mathit{\theta }){||}^2+\mathsf{\Omega }(\mathit{\theta }),$$

(55)

where $\mathsf{\Omega }(\mathit{\theta })=\alpha {\sum }_{l=1}^L{∥{\mathsf{W}}_l∥}_2^2$ with ${\left[{\mathsf{W}}_l\right]}_{ji}=w_{ji,l}$ is the regularization term and the notation ${\mathbf{f}}_{{\mathit{\omega }}^{(n)}}(\mathbf{x},\mathit{\theta })$ indicates that the mean and variance ${\mathit{\omega }}^{(n)}=({\mathit{\mu }}^{(n)},{\mathit{v}}^{(n)})$ are used in the normalization step (52). At the end of the training stage, we obtain:

• The maximum a posteriori parameters $\widehat{\mathit{\theta }}={\mathit{\theta }}_{\mathrm{MAP}}$;
• The mean and variance realizations ${\{{\mathit{\omega }}^{(n)}=({\mathit{\mu }}^{(n)},{\mathit{v}}^{(n)})\}}_{n=1}^{N_{\mathrm{b}}}$ of the stochastic parameters $\mathit{\omega }=(\mathit{\mu },\mathit{v})$; and
• The moving averages of the mean and variance over the training set $\overline{\mathit{\omega }}=(\overline{\mathit{\mu }}=\mathbb{E}(\mathit{\mu }),\overline{\mathit{v}}=\mathbb{E}(\mathit{v}))$.

Some comments can be made here.

• A batch-normalized network samples the stochastic parameters $\mathit{\omega }$ once per training step (mini-batch). For a large number of epochs, the ${\mathit{\omega }}^{(n)}$ become independent and identical distributed random variables for each training example;
• The variational distribution $q_{\mathit{\theta }}(\mathit{\omega })$ corresponds to the joint distribution of the weights induced by the stochastic parameters $\mathit{\omega }$;
• The equivalence between a batch-normalized network and a Bayesian approximation was proven in [39] by showing that (cf. Equations (39) and (55)) $\partial \mathrm{KL}(q_{\mathit{\theta }}(\mathit{\omega })|p(\mathit{\omega }))/$$\partial \mathit{\theta }=(N/{\sigma }_{\mathbf{y}}^2)\partial \mathsf{\Omega }(\mathit{\theta })/\partial \mathit{\theta }$. The proof relies on the following simplified assumptions: (i) no scale and shift transformations; (ii) batch normalization applied to each layer; (iii) independent input features in each layer; and (iv) large N and M.

In the inference, the output for a given input $\mathbf{x}$ is ${\mathbf{f}}_{\overline{\mathit{\omega }}}(\mathbf{x},\widehat{\mathit{\theta }})$. To estimate the predictive mean and covariance matrix, we proceed as follows. For each $t=1\dots ,T$, we sample a mini-batch ${\mathcal{B}}^{(t)}$ from the data set $\mathcal{D}={\cup }_{n=1}^{N_{\mathrm{b}}}{\mathcal{B}}^{(n)}$, and for the corresponding mean and variance realization ${\mathit{\omega }}^{(t)}\in {\left\{{\mathit{\omega }}^{(n)}\right\}}_{n=1}^{N_{\mathrm{b}}}$, compute ${\mathbf{f}}_{{\mathit{\omega }}^{(t)}}(\mathbf{x},\widehat{\mathit{\theta }})$ and then the predictive mean and covariance matrix from Equations (42) and (43) with ${\mathbf{f}}_{{\mathit{\omega }}^{(t)}}(\mathbf{x},\widehat{\mathit{\theta }})$ in place of $\mathbf{f}(\mathbf{x},{\mathit{\omega }}_t)$.

3. Neural Networks for Atmospheric Remote Sensing

In this section, we design several neural networks for atmospheric remote sensing. To test the neural networks, we considered a specific problem, namely the retrieval of cloud optical thickness and cloud top height from radiances measured by the Earth Polychromatic Imaging Camera (EPIC) onboard the Deep Space Climate Observatory (DSCOVR). DSCOVR is placed in a Lissajous orbit about the Lagrange-1 point, and provides a unique angular perspective in an almost backward direction with scattering angles between ${168}^{\circ }$ and ${176}^{\circ }$. The EPIC instrument has 10 spectral channels ranging from the UV to the near-IR, which include four channels around the oxygen A- and B-bands; two absorption channels are centered at 688 nm and 764 nm with bandwiths of 0.8 nm and 1.0 nm, respectively, while two continuum bands are centered at 680 nm and 780 nm with bandwiths of 2.0 nm. These four channels are used for inferring the cloud parameters. To generate the database, we use a radiative transfer model based on the discrete ordinate method with matrix exponential [40,41] that uses several acceleration techniques, as for example, the telescoping technique, the method of false discrete ordinate [42], the correlated k-distribution method [43], and the principal component analysis [44,45,46]. The atmospheric parameters to be retrieved are the cloud optical thickness $\tau$ and the cloud top height H, while the forward model parameters are the solar and viewing zenith angles and the surface albedo. Specifically, we consider the fact that the cloud optical thickness varies in the range $4.0,\dots ,16.0$, the cloud top height in the range $2.0,\dots 10.0$ km, the solar and viewing zenith angles in the range $0^{\circ },\dots ,60{}^{\circ }$, and the surface albedo in the range $0.02,\dots ,0.2$ (only snow/ice free scenes are considered). The simulations are performed for a water-cloud model with a Gamma size distribution $p(a)\propto a^{\alpha }exp(-\alpha a/a_{\mathrm{mod}})$ with parameters $a_{\mathrm{mod}}=8$$\mathsf{\mu }$m and $\alpha =6$. The droplet size ranges between 0.02 and 50.0 $\mathsf{\mu }$m, the cloud geometrical thickness is 1 km and the relative azimuth angle between the solar and viewing directions is ${176}^{\circ }$. The ${\mathrm{O}}_2$ absorption cross-sections are computed using LBL calculations [47] with optimized rational approximations for the Voigt line profile [48]. The wavenumber grid point spacing is chosen as a fraction (e.g., $1/4$) of the minimum half-width of the Voigt lines taken from HITRAN database [49]. The Rayleigh cross-section and depolarization ratios are computed as in [50], while the pressure and temperature profiles correspond to the US standard model atmosphere [51]. The radiances are solar-flux normalized and are computed by means of the delta-M approximation in conjunction with the TMS correction. We generate $N=$ 20,000 samples by employing the smart sampling technique [52]. Among this data set, 18,000 samples were used for training and the other 2000 for prediction. The noisy spectra are generated by using the measurement noise ${\delta }_{\mathrm{mes}}\sim \mathcal{N}(\mathbf{0},\mathrm{diag}{\left[{\sigma }_{\mathrm{mes}j}^2\right]}_{j=1}^4)$, where for the jth channel, we use ${\sigma }_{\mathrm{mes}j}=0.1{\overline{I}}_j$ with ${\overline{I}}_j$ being the average of the simulated radiance over the N samples.

The neural network algorithms are implemented in FORTRAN by using a feed-forward multilayer perceptron architecture. The tool contains a variety of optimization algorithms, activation functions, regularization terms, dynamic learning rates, and stopping rules. For the present application, the number of hidden layers and the number of units in each layer are optimized by using 2000 samples from the training set for validation. To estimate the performances of different hyperparameter configurations, we used the holdout cross-validation together with a grid search over a set of three values for the number of hidden layers, i.e., $\{1,2,3\}$, and a set of eight values for the number of units, i.e., {25, 50, 75, 100, 125, 150, 175, 200}. The mini-batch gradient descent in conjunction with Adaptive Moment Estimation (ADAM) [53] is used as optimization tool, a mini batch of 100 samples is chosen, and a ReLU activation function is considered.

3.1. Neural Networks for Solving the Direct Problem

For a direct problem, the input $\mathbf{x}$ is the set of atmospheric parameters, the output $\mathbf{y}$ is the set of data, and the forward model $\mathbf{F}$ reproduces the radiative transfer model $\mathbf{R}$.

We consider a neural network trained with exact data. For the predictive distribution $p(\mathbf{y}|\mathbf{x},\mathcal{D})$ given by Equation (32), we assume that the epistemic covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{x},\widehat{\mathit{\omega }})$$(={\mathcal{C}}_{\mathbf{y}}(\mathbf{x},\widehat{\mathit{\omega }}))$ is computed from the statistics of $\epsilon =\mathbf{y}-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$, that is, ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}\approx \mathbb{E}({\mathcal{C}}_{\mathbf{y}}|\mathcal{D})$, where $\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$ is the network output. Furthermore, let ${\mathbf{y}}^{\delta }=\mathbf{y}+{\delta }_{\mathbf{y}}$ with ${\delta }_{\mathbf{y}}\sim \mathcal{N}(\mathbf{0},{\mathsf{C}}_{\mathbf{y}}^{\delta }),$ be the noisy data vector. Using the result:

$$p({\mathbf{y}}^{\delta }|\mathbf{y})\propto exp\left[-\frac{1}{2}({\mathbf{y}}^{\delta }-\mathbf{y}){}^T{({\mathsf{C}}_{\mathbf{y}}^{\delta })}^{-1}({\mathbf{y}}^{\delta }-\mathbf{y})\right],$$

(56)

we compute the predictive distribution for the noisy data $p({\mathbf{y}}^{\delta }|\mathbf{x},\mathcal{D})$ by marginalization, that is:

$$\begin{array}{cc}\hfill p({\mathbf{y}}^{\delta }|\mathbf{x},\mathcal{D})& =\int p({\mathbf{y}}^{\delta }|\mathbf{y})p(\mathbf{y}|\mathbf{x},\mathcal{D})\mathrm{d}\mathbf{y}\hfill \\ \hfill & =\int exp\left\{-\frac{1}{2}{[{\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})+\Delta \mathbf{y}]}^T{\left[{\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\widehat{\mathit{\omega }})\right]}^{-1}[{\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})+\Delta \mathbf{y}]\right\}\hfill \\ \hfill & \times exp\left[-\frac{1}{2}\Delta \mathbf{y}{}^T{({\mathsf{C}}_{\mathbf{y}}^{\delta })}^{-1}\Delta \mathbf{y}\right]\mathrm{d}\Delta \mathbf{y}\hfill \\ \hfill & \propto exp\left\{-\frac{1}{2}{[{\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})]}^T{\mathcal{C}}_{\mathbf{y}}^{-1}(\widehat{\mathit{\omega }})[{\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})]\right\},\hfill \end{array}$$

(57)

where (cf. Equation (33)) ${\mathcal{C}}_{\mathbf{y}}(\widehat{\mathit{\omega }})={\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\widehat{\mathit{\omega }})+{\mathsf{C}}_{\mathbf{y}}^{\delta }$ and $\Delta \mathbf{y}=\mathbf{y}-{\mathbf{y}}^{\delta }$.

In the next step, we solve the inverse problem ${\mathbf{y}}^{\delta }=\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$ by means of a Bayesian approach [54,55]. In this case, the posteriori density $p(\mathbf{x}|{\mathbf{y}}^{\delta },\mathcal{D})$ is given by

$$p(\mathbf{x}|{\mathbf{y}}^{\delta },\mathcal{D})=\frac{p({\mathbf{y}}^{\delta }|\mathbf{x},\mathcal{D})p(\mathbf{x})}{p(\mathbf{y})};$$

(58)

whence, by assuming that the state vector $\mathbf{x}$ is a Gaussian random vector with mean ${\mathbf{x}}_{\mathrm{a}}$ and the covariance matrix ${\mathsf{C}}_{\mathbf{x}}$, meaning that:

$$\begin{array}{cc}\hfill p(\mathbf{x})& =\mathcal{N}(\mathbf{x};{\mathbf{x}}_{\mathrm{a}},{\mathsf{C}}_{\mathbf{x}})\propto exp\left[-\frac{1}{2}(\mathbf{x}-{\mathbf{x}}_{\mathrm{a}}){\mathsf{C}}_{\mathbf{x}}^{-1}{(\mathbf{x}-{\mathbf{x}}_{\mathrm{a}})}^T\right],\hfill \end{array}$$

(59)

we obtain:

$$logp(\mathbf{x}|{\mathbf{y}}^{\delta },\mathcal{D})=-\frac{1}{2}V(\mathbf{x}|{\mathbf{y}}^{\delta })+C,$$

(60)

where:

$$\begin{array}{cc}\hfill V(\mathbf{x}|{\mathbf{y}}^{\delta })& =[{\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})]{\mathcal{C}}_{\mathbf{y}}^{-1}(\widehat{\mathit{\omega }}){[{\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})]}^T+(\mathbf{x}-{\mathbf{x}}_{\mathrm{a}}){\mathsf{C}}_{\mathbf{x}}^{-1}{(\mathbf{x}-{\mathbf{x}}_{\mathrm{a}})}^T\hfill \end{array}$$

(61)

is the a posteriori potential and C is constant. The maximum a posteriori estimator, defined by

$$\widehat{\mathbf{x}}={\mathbf{x}}_{\mathrm{MAP}}=arg\underset{\mathbf{x}}{max}logp(\mathbf{x}|{\mathbf{y}}^{\delta },\mathcal{D})=arg\underset{\mathbf{x}}{min}V(\mathbf{x}|{\mathbf{y}}^{\delta }),$$

(62)

can be computed by any optimization method, as for example, the Gauss–Newton method. If the problem is almost linear, the a posteriori density $p(\mathbf{x}|{\mathbf{y}}^{\delta },\mathcal{D})$ is Gaussian with the mean $\widehat{\mathbf{x}}$ and covariance matrix [54]:

$${\mathcal{C}}_{\mathbf{x}}(\widehat{\mathbf{x}},\widehat{\mathit{\omega }})={[{\mathsf{K}}_{\mathbf{x}}^T(\widehat{\mathbf{x}}){\mathcal{C}}_{\mathbf{y}}^{-1}(\widehat{\mathit{\omega }}){\mathsf{K}}_{\mathbf{x}}(\widehat{\mathbf{x}})+{\mathsf{C}}_{\mathbf{x}}^{-1}]}^{-1}.$$

(63)

It should be pointed out that we can train the neural network for a data set with output noise $\mathcal{D}={\left\{({\mathbf{x}}^{(n)},{\mathbf{y}}^{\delta (n)})\right\}}_{n=1}^N$, in which case, the covariance matrix ${\mathcal{C}}_{\mathbf{y}}(\widehat{\mathit{\omega }})$ can be directly computed from the statistics of $\epsilon ={\mathbf{y}}^{\delta }-\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$.

Essentially, the method involves the following steps:

• Train a neural network with exact data for simulating the radiative transfer model;
• Compute the epistemic covariance matrix from the statistics of all network errors over the data set;
• Solve the inverse problem by a Bayesian approach under the assumption that the a priori knowledge and the measurement uncertainty are both Gaussian;
• Determine the uncertainty in the retrieval by assuming that the forward model is nearly linear.

In Figure 1, we plot the histograms of the relative error over the prediction set:

$$\begin{array}{cc}\hfill {\epsilon }_x& =\frac{x_{\mathrm{pred}}-x}{x},\hfill \end{array}$$

where x stands for $\tau$ and H, and $x_{\mathrm{pred}}$ and x are the predicted and true values, respectively. Also shown here are the plots of $x_{\mathrm{pred}}$ and $x_{\mathrm{pred}}\pm 3{\sigma }_x$ versus x. The results demonstrate that the cloud optical thickness is retrieved with better accuracy than the cloud top height, and that for the cloud top height, the predicted uncertainty are especially unrealistic, because the condition:

$$x_{\mathrm{pred}}-3{\sigma }_x\le x\le x_{\mathrm{pred}}+3{\sigma }_x$$

is not satisfied. The reason for this failure might be that the forward model is not nearly linear.

3.2. Neural Networks for Solving the Inverse Problem

For an inverse problem, the input $\mathbf{x}$ includes the sets of data and forward model parameters (solar and viewing zenith angles, and surface albedo), while the output $\mathbf{y}$ includes the set of atmospheric parameters to be retrieved (cloud optical thickness and cloud top height).

3.2.1. Method 1

Let $\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$ be the output of a neural network trained with exact data, and assume that the predictive distribution $p(\mathbf{y}|\mathbf{x},\mathcal{D})$ given by Equation (32), and in particular, the epistemic covariance matrix ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}(\mathbf{x},\widehat{\mathit{\omega }})(={\mathcal{C}}_{\mathbf{y}}(\mathbf{x},\widehat{\mathit{\omega }}))$ are available. For the noisy input $\mathbf{z}=\mathbf{x}+{\delta }_{\mathbf{x}}$ with $p(\mathbf{z}|\mathbf{x})=\mathcal{N}(\mathbf{z};\mathbf{x},{\mathsf{C}}_{\mathbf{x}}^{\delta }={\sigma }_{\mathbf{x}}^2\mathsf{I})$ and under the assumption that the prior $p(\mathbf{x})$ is a slowly varying function as compared to $p(\mathbf{z}|\mathbf{x}),$ the predictive distribution of the network output can be approximated by [56]

$$\begin{array}{cc}\hfill p(\mathbf{y}|\mathbf{z},\mathcal{D})& =\int p(\mathbf{y}|\mathbf{x},\mathcal{D})p(\mathbf{x}|\mathbf{z})\mathrm{d}\mathbf{x}\hfill \\ \hfill & =\frac{1}{p(\mathbf{z})}\int p(\mathbf{y}|\mathbf{x},\mathcal{D})p(\mathbf{z}|\mathbf{x})p(\mathbf{x})\mathrm{d}\mathbf{x}\hfill \\ \hfill & \approx \int p(\mathbf{y}|\mathbf{x},\mathcal{D})p(\mathbf{z}|\mathbf{x})\mathrm{d}\mathbf{x}.\hfill \end{array}$$

(64)

Thus, if $p(\mathbf{x})$ varies much more slowly than $p(\mathbf{z}|\mathbf{x})=p(\mathbf{z}-\mathbf{x})$, we assume that $p(\mathbf{y}|\mathbf{z},\mathcal{D})$ is the convolution of the predictive distribution for an uncorrupted input $p(\mathbf{y}|\mathbf{x},\mathcal{D})$ with the input noise distribution $p(\mathbf{z}-\mathbf{x})$. In the noise-free case, that is, if ${\sigma }_{\mathbf{x}}\to 0$, we have ${lim}_{{\sigma }_{\mathbf{x}}\to 0}p(\mathbf{z}|\mathbf{x})=\delta (\mathbf{z}-\mathbf{x})$, yielding $p(\mathbf{y}|\mathbf{z},\mathcal{D})=p(\mathbf{y}|\mathbf{x},\mathcal{D})$. Using Equation (64), we obtain:

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y}|\mathbf{z})& =\int \mathbf{y}p(\mathbf{y}|\mathbf{z},\mathcal{D})\mathrm{d}\mathbf{y}\hfill \\ \hfill & =\int \left(\int \mathbf{y}p(\mathbf{y}|\mathbf{x},\mathcal{D})\mathrm{d}\mathbf{y}\right)p(\mathbf{z}|\mathbf{x})\mathrm{d}\mathbf{x}\hfill \\ \hfill & =\int \mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})p(\mathbf{z}|\mathbf{x})\mathrm{d}\mathbf{x}\hfill \end{array}$$

(65)

and:

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y}{\mathbf{y}}^T|\mathbf{z})& =\int \mathbf{y}{\mathbf{y}}^{T}p(\mathbf{y}|\mathbf{z},\mathcal{D})\mathrm{d}\mathbf{y}\hfill \\ \hfill & =\int \left(\int \mathbf{y}{\mathbf{y}}^{T}p(\mathbf{y}|\mathbf{x},\mathcal{D})\mathrm{d}\mathbf{y}\right)p(\mathbf{z}|\mathbf{x})\mathrm{d}\mathbf{x}\hfill \\ \hfill & =\int [{\mathcal{C}}_{\mathrm{y}}^{\mathrm{e}}(\mathbf{x},\widehat{\mathit{\omega }})+\mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})\mathbf{f}{(\mathbf{x},\widehat{\mathit{\omega }})}^T]p(\mathbf{z}|\mathbf{x})\mathrm{d}\mathbf{x}.\hfill \end{array}$$

(66)

Equations (65) and (66) show that in general $\mathbb{E}(\mathbf{y}|\mathbf{z})\ne \mathbf{f}(\mathbf{x},\widehat{\mathit{\omega }})$, and a noise process blurs or smooths the original mappings.

To compute the predictive mean $\mathbb{E}(\mathbf{y}|\mathbf{z})$ and the covariance matrix $\mathrm{Cov}(\mathbf{y}|\mathbf{z})$, the integrals in Equations (65) and (66) must be calculated. This can be done by Monte Carlo integration (by sampling the Gaussian distribution $p(\mathbf{z}|\mathbf{x})$) or by a quadrature method. In the latter case, we consider a uniform grid ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^{N_{\mathrm{int}}}$ in the interval, say $[\mathbf{z}-2{\sigma }_{\mathbf{x}},\mathbf{z}+2{\sigma }_{\mathbf{x}}]$, define the weights:

$$v_i=\frac{exp\left(-{\displaystyle \frac{1}{2{\sigma }_{\mathbf{x}}^2}}||\mathbf{z}-{\mathbf{x}}_i{||}^2\right)}{{\sum }_{i=1}^{N_{\mathrm{int}}}exp\left(-{\displaystyle \frac{1}{2{\sigma }_{\mathbf{x}}^2}}||\mathbf{z}-{\mathbf{x}}_i{||}^2\right)},$$

(67)

and use the computational formulas:

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y}|\mathbf{z})& =\sum _{i=1}^{N_{\mathrm{int}}}{v}_i\mathbf{f}({\mathbf{x}}_i,\widehat{\mathit{\omega }}),\hfill \end{array}$$

(68)

and:

$$\begin{array}{cc}\hfill \mathrm{Cov}(\mathbf{y}|\mathbf{z})& =\sum _{i=1}^{N_{\mathrm{int}}}{v}_i{\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}({\mathbf{x}}_i,\widehat{\mathit{\omega }})+\sum _{i=1}^{N_{\mathrm{int}}}{v}_i\mathbf{f}({\mathbf{x}}_i,\widehat{\mathit{\omega }})\mathbf{f}{({\mathbf{x}}_i,\widehat{\mathit{\omega }})}^T-\mathbb{E}(\mathbf{y}|\mathbf{z})\mathbb{E}{(\mathbf{y}|\mathbf{z})}^T.\hfill \end{array}$$

(69)

The neural network trained with exact data can be deterministic or stochastic, as for example, Bayes-by-backprop, dropout, and batch normalization. In this regard, for each noisy input $\mathbf{z}$, we consider a uniform grid ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^{N_{\mathrm{int}}}$ around $\mathbf{z}$, calculate the weights $v_i$ by means of Equation (67), and for each ${\mathbf{x}}_i$, compute ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}({\mathbf{x}}_i,\widehat{\mathit{\omega }})$ as follows:

• For a deterministic network, we approximate ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}({\mathbf{x}}_i,\widehat{\mathit{\omega }})$ by the conditional average covariance matrix $\mathbb{E}({\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}|\mathcal{D})$ of all network errors over, that is, ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}\approx \mathbb{E}({\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}|\mathcal{D})$, yielding ${\sum }_{i=1}^{N_{\mathrm{int}}}{v}_i{\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}({\mathbf{x}}_i,\widehat{\mathit{\omega }})=\mathbb{E}({\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}|\mathcal{D})$;
• For a Bayes-by-backprop and dropout networks, we compute ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}({\mathbf{x}}_i,\widehat{\mathit{\omega }})$ by means of Equation (44) with $\mathbb{E}(\mathbf{y})$ as in Equation (42);
• For a batch normalized network, we compute ${\mathcal{C}}_{\mathbf{y}}^{\mathrm{e}}({\mathbf{x}}_i,\widehat{\mathit{\omega }})$ as

  $$\begin{array}{cc}\hfill \mathrm{Cov}(\mathbf{y})& =\frac{1}{T}\sum _{t=1}^T{\mathbf{f}}_{{\mathit{\omega }}^{(t)}}({\mathbf{x}}_i,\widehat{\mathit{\theta }}){\mathbf{f}}_{{\mathit{\omega }}^{(t)}}{({\mathbf{x}}_i,\widehat{\mathit{\theta }})}^T-\mathbb{E}(\mathbf{y})\mathbb{E}{(\mathbf{y})}^T\hfill \end{array}$$

  with $\mathbb{E}(\mathbf{y})=(1/T){\sum }_{t=1}^T{\mathbf{f}}_{{\mathit{\omega }}^{(t)}}({\mathbf{x}}_i,\widehat{\mathit{\theta }})$.

Note that for a Bayes-by-backprop network, the output is $\mathbf{f}({\mathbf{x}}_i,\widehat{\mathit{\omega }}={\widehat{\mathit{\mu }}}_{\mathit{\omega }})$, for a dropout network, the output $\mathbf{f}({\mathbf{x}}_i,\widehat{\mathit{\omega }})$ is computed without dropout, and for a batch normalized network, the output is ${\mathbf{f}}_{\overline{\mathit{\omega }}}({\mathbf{x}}_i,\widehat{\mathit{\theta }})$.

In summary, the method uses:

• Deterministic and stochastic networks trained with exact data to compute the epistemic covariance matrix; and
• The assumption that the predictive distribution of the network output is the convolution of the predictive distribution for an uncorrupted input with the input noise distribution to estimate the covariance matrix.

Under the assumption that the noise process is Gaussian, the convolution integrals are computed by a quadrature method with a uniform grid around the noisy input.

The results for Method 1 with deterministic and stochastic networks are illustrated in Figure 2, Figure 3, Figure 4 and Figure 5.

3.2.2. Method 2

In order to model both heteroscedastic and epistemic uncertainties, we used the approach described in Ref. [57]. More precisely, we considered the data model with input and output noise, and used dropout to learn the heteroscedastic covariance matrix ${\mathcal{C}}_{\mathrm{y}}^{\delta }(\mathbf{z},\mathit{\omega })=\mathrm{diag}{\left[{\sigma }_j^2(\mathbf{z},\mathit{\omega })\right]}_{j=1}^{N_{\mathrm{y}}}$ from the data (see Section 2.1). The network has the output $[{\mu }_j(\mathbf{z},\mathit{\omega }),{\sigma }_j^2(\mathbf{z},\mathit{\omega })]\in {\mathbb{R}}^{2N_{\mathrm{y}}}$ and is trained to predict the log variance ${\rho }_j=log{\sigma }_j^2$, in which case, the likelihood loss function is given by Equations (26) and (27). Considering the set of samples ${\left\{{\mathit{\omega }}_t\right\}}_{t=1}^T$ , where ${\mathit{\omega }}_t$ corresponds to a realization of the Bernoulli distribution ${\mathsf{Z}}_{l-1}^{(t)}$ and ${\mathsf{W}}_l^{(t)}={\mathsf{W}}_l{\mathsf{Z}}_{l-1}^{(t)}$ for all $l=1,\dots ,L$, we compute the predictive mean and covariance matrix as [57]

$$\begin{array}{cc}\hfill \mathbb{E}(\mathbf{y})& \approx \frac{1}{T}\sum _{t=1}^T\mathit{\mu }(\mathbf{z},{\mathit{\omega }}_t),\hfill \end{array}$$

(70)

$$\begin{array}{cc}\hfill \mathrm{Cov}(\mathbf{y})& \approx \frac{1}{T}\sum _{t=1}^T\mathrm{diag}{\left[{\sigma }_j^2(\mathbf{z},{\mathit{\omega }}_t)\right]}_{j=1}^{N_{\mathrm{y}}}+\frac{1}{T}\sum _{t=1}^T\mathit{\mu }(\mathbf{z},{\mathit{\omega }}_t)\mathit{\mu }{(\mathbf{z},{\mathit{\omega }}_t)}^T-\mathbb{E}(\mathbf{y})\mathbb{E}{(\mathbf{y})}^T,\hfill \end{array}$$

(71)

for each noisy input $\mathbf{z}$. The first term in Equation (71) reproduces the heteroscedastic uncertainty, while the second and third terms reproduce the epistemic uncertainty.

Instead of a dropout network, a Bayes-by-backprop network can also be used to learn the heteroscedastic covariance matrix from the data. In this case, $F(\mathit{\theta },\mathcal{D})$ is given by Equation (47) with $logp(\mathcal{D}|\mathit{\omega })=-E_{\mathrm{D}}(\mathit{\omega })=-{\sum }_{n=1}^N{E}_{\mathrm{D}}^{(n)}(\mathit{\omega })$ and $E_{\mathrm{D}}^{(n)}(\mathit{\omega })$ as in Equation (27).

The results for Method 2 with a dropout and a Bayes-by-backprop network are illustrated in Figure 6 and Figure 7.

3.2.3. Method 3

Let $\widehat{\mathit{\omega }}={\{{\mathsf{W}}_l,{\mathbf{b}}_l\}}_{l=1}^L$ be the parameters of a dropout network trained with exact data. Further, assume that the input data are noisy, i.e., $\mathbf{z}=\mathbf{x}+{\delta }_{\mathbf{x}}$ with $p(\mathbf{z}|\mathbf{x})=\mathcal{N}(\mathbf{z};\mathbf{x},{\mathsf{C}}_{\mathbf{x}}^{\delta }=\mathrm{diag}{\left[{\sigma }_{\mathrm{x}k}^2\right]}_{k=1}^{N_{\mathrm{x}}})$. In order to compute the heteroscedastic uncertainty, we forward propagate the input noise through the network. This is done by using assumed density filtering and interval arithmetic.

• Assumed density filtering (ADF). This approach was applied to neural networks by Gast and Roth [58] to replace each network activation by probability distributions. In the following, we provide a simplified justification of this approach, while for a more pertinent analysis, we refer to Appendix D. For a linear layer ($g_l(x)=x)$, the feed-forward operation (without dropout) is:

  $$\begin{array}{cc}\hfill y_{k,l}& =\sum _{j=1}^{N_{l-1}}{w}_{kj,l}{y}_{j,l-1}+b_{k,l},k=1,\dots ,N_l,\hfill \end{array}$$

  (72)
  By straightforward calculation, we find:

  $${\mu }_{k,l}=\mathbb{E}(y_{k,l})=\sum _{j=1}^{N_{l-1}}{w}_{kj,l}\mathbb{E}(y_{j,l-1})+b_{k,l}=\sum _{j=1}^{N_{l-1}}{w}_{kj,l}{\mu }_{j,l-1}+b_{k,l},$$

  (73)

  and:

  $$\begin{array}{cc}\hfill \mathbb{E}(y_{k,l})\mathbb{E}(y_{k_1,l})& =\sum _{j=1}^{N_{l-1}}\sum _{j_1=1}^{N_{l-1}}{w}_{kj,l}{w}_{k_1{j}_1,l}{\mu }_{j,l-1}{\mu }_{j_1,l-1}\hfill \\ \hfill & +b_{k,l}{\mu }_{k_1,l}+b_{k_1,l}{\mu }_{k,l}-b_{k,l}{b}_{k_1,l},\hfill \end{array}$$

  (74)

  yielding:

  $$\begin{array}{cc}\hfill & \mathbb{E}(y_{k,l}{y}_{k_1,l})-\mathbb{E}(y_{k,l})\mathbb{E}(y_{k_1,l})\hfill \\ \hfill & =\sum _{j=1}^{N_{l-1}}\sum _{j_1=1}^{N_{l-1}}{w}_{kj,l}{w}_{k_1{j}_1,l}[\mathbb{E}(y_{j,l-1}{y}_{j_1,l-1})-\mathbb{E}(y_{j,l-1})\mathbb{E}(y_{j_1,l-1})].\hfill \end{array}$$

  (75)
  Assuming that the $y_{k,l}$ are independent random variables, in which case, the covariance matrix corresponding to the column vector ${[y_{1.l},\dots ,y_{N_{l}l}]}^T$ is diagonal, meaning that:

  $$\mathbb{E}(y_{k,l}{y}_{k_1,l})-\mathbb{E}(y_{k,l})\mathbb{E}(y_{k_1,l})={\delta }_{k{k}_1}\left[\mathbb{E}(y_{k,l}^2)-{\mathbb{E}}^2(y_{k,l})\right]={\delta }_{k{k}_1}{v}_{k,l},$$

  (76)

  we obtain $v_{k,l}={\sum }_{j=1}^{N_{l-1}}{w}_{kj,l}^2{v}_{j,l-1}$. In summary, the iterative process for a linear layer is:

  $$\begin{array}{cc}\hfill {\mu }_{k,0}& =x_k,v_{k,0}={\sigma }_{\mathrm{x}k}^2.\hfill \end{array}$$

  (77)

  $$\begin{array}{cc}\hfill {\mu }_{k,l}& =\sum _{j=1}^{N_{l-1}}{w}_{kj,l}{\mu }_{j,l-1}+b_{k,l},\hfill \end{array}$$

  (78)

  $$\begin{array}{cc}\hfill v_{k,l}& =\sum _{j=1}^{N_{l-1}}{w}_{kj,l}^2{v}_{j,l-1},l=1,\dots ,L.\hfill \end{array}$$

  (79)
  For a ReLU activation function $\mathrm{ReLU}(x)=max(0,x)$, it was shown that:

  $${\mu }_{\mathrm{ReLU}k,l}=\sqrt{v_{k,l}}\varphi (\alpha )+{\mu }_{k,l}\Phi (\alpha ),$$

  (80)

  $$v_{\mathrm{ReLU}k,l}={\mu }_{k,l}\sqrt{v_{k,l}}\varphi (\alpha )+({\mu }_{k,l}^2+v_{k,l})\Phi (\alpha )-{\mu }_{\mathrm{ReLU}k,l}^2,$$

  (81)

  where ${\mu }_{k,l}$ and $v_{k,l}$ are given by Equations (78) and (79), respectively, and:

  $$\begin{array}{cc}\hfill \alpha & =\frac{{\mu }_{k,l}}{\sqrt{v_{k,l}}},\hfill \\ \hfill \varphi (x)& \hfill =\frac{1}{\sqrt{2\pi }}exp\left(-\frac{1}{2}{x}^2\right),\\ \hfill \Phi (x)& ={\int }_{-\infty }^x\varphi (y)dy.\hfill \end{array}$$
• Interval Arithmetic (IA). Interval arithmetic is based on an extension of the real number system to a system of closed intervals on the real axis [59]. For the intervals X and Y, the elementary arithmetic operations are defined by the rule $X\oplus Y=\{x\oplus y|x\in X,y\in Y\}$, where the binary operation ⊕ can stand for addition, subtraction, multiplication, or division. This definition guarantees that $x\oplus y\in X\oplus Y$. Functions of interval arguments are defined in terms of standard set mapping, that is, the image of an interval X under a function f is the set $f(X)=\{f(x)|x\in X\}$. This is not the same as an interval function obtained from a real function f by replacing the real argument by an interval argument and the real arithmetic operations by the corresponding interval operations. The latter is called an interval extension of the real function f and is denoted by $F(X)$. As a corollary of the fundamental theorem of interval analysis, it can be shown that $f(X)\subseteq F(X)$. Interval analysis provides a simple and accessible way to assess error propagation. The iterative process for error propagation is (compared with Equations (77)–(79)):

  $$\begin{array}{cc}\hfill Y_{k,0}& =[x_k-{\sigma }_{\mathrm{x}k},x_k+{\sigma }_{\mathrm{x}k}],\hfill \end{array}$$

  (82)

  $$\begin{array}{cc}\hfill U_{k,l}& =\sum _{i=1}^{N_{l-1}}{w}_{kj,l}{Y}_{j,l-1}+b_{k,l},\hfill \end{array}$$

  (83)

  $$\begin{array}{cc}\hfill Y_{k,l}& =G_l(U_{k,l}),l=1,\dots ,L,\hfill \end{array}$$

  (84)

  while the output predictions ${\mu }_{k,L}$ and their standard deviations ${\sigma }_{k,L}=\sqrt{v_{k,L}}$ are computed as

  $$\begin{array}{cc}\hfill {\mu }_{k,L}& ={\displaystyle \frac{1}{2}}[{\underline{Y}}_{k,L}+{\overline{Y}}_{k,L}],\hfill \end{array}$$

  (85)

  $$\begin{array}{cc}\hfill {\sigma }_{k,L}& ={\displaystyle \frac{1}{2}}[{\overline{Y}}_{k,L}-{\underline{Y}}_{k,L}],k=1,\dots ,N_L,\hfill \end{array}$$

  (86)

  where $G_l(U)$ is the interval extension of the activation function $g_l$, and $\underline{X}$ and $\overline{X}$ are the left and right endpoints of the interval X, respectively, that is, $X=[\underline{X},\overline{X}]$.

By assumed density filtering and interval arithmetic, the forward pass of a neural network generates not only the output predictions ${\mu }_L={[{\mu }_{1,L},\dots ,{\mu }_{N_L,L}]}^T$ but also their variances ${\mathit{v}}_L={[v_{1,L},\dots ,v_{N_L,L}]}^T$. Following Ref. [2], we consider now a network with dropout, that is, in Equations (78), (79) and (83), we replace $w_{kj,l}$ by $w_{kj,l}{z}_{j,l-1}$, where $z_{j,l-1}\sim \mathrm{Bernoulli}(p)$ and the dropout probability p is the same as that used for training. For the set of samples ${\left\{{\mathit{\omega }}_t\right\}}_{t=1}^T$, where ${\mathit{\omega }}_t$ corresponds to a realization from the Bernoulli distribution ${\mathsf{Z}}_{l-1}^{(t)}$ and ${\mathsf{W}}_l^{(t)}={\mathsf{W}}_l{\mathsf{Z}}_{l-1}^{(t)}$ for all $l=1,\dots ,L$, we denote by ${\mu }_L(\mathbf{x},{\mathit{\omega }}_t)$ and ${\mathit{v}}_L(\mathbf{x},{\mathit{\omega }}_t)$ the outputs of the network for an input $\mathbf{x}$ corrupted by the noise ${\delta }_{\mathbf{x}}\sim \mathcal{N}(\mathbf{0},{\mathsf{C}}_{\mathbf{x}}^{\delta }=\mathrm{diag}{\left[{\sigma }_{\mathrm{x}k}^2\right]}_{k=1}^{N_{\mathrm{x}}})$, and compute the predictive mean and covariance matrix as (Appendix B)

$$\mathbb{E}(\mathbf{y}|\mathbf{x})\approx \frac{1}{T}\sum _{t=1}^T{\mu }_L(\mathbf{x},{\mathit{\omega }}_t),$$

(87)

$$\begin{array}{cc}\hfill \mathrm{Cov}(\mathbf{y}|\mathbf{x})& \approx \frac{1}{T}\sum _{t=1}^T\mathrm{diag}{\left[v_{k,L}(\mathbf{x},{\mathit{\omega }}_t)\right]}_{k=1}^{N_y}+\frac{1}{T}\sum _{t=1}^T{\mu }_L(\mathbf{x},{\mathit{\omega }}_t){\mu }_L{(\mathbf{x},{\mathit{\omega }}_t)}^T\hfill \\ \hfill & -\mathbb{E}(\mathbf{y}|\mathbf{x})\mathbb{E}{(\mathbf{y}|\mathbf{x})}^T.\hfill \end{array}$$

(88)

From Equations (87) and (88), it is obvious that the prediction ensemble is not generated by the output of the network $\mathbf{f}(\mathbf{x},{\mathit{\omega }}_t)$, but by the prediction ${\mu }_L(\mathbf{x},{\mathit{\omega }}_t)$. In summary, the algorithm involves the following steps:

• Transform a dropout network into its assumed density filtering or interval arithmetic versions (which does not require retraining);
• Propagate $(\mathbf{x},{\mathit{v}}_0)$ through the dropout network and collect T output predictions and variances;
• Compute the predictive mean and covariance matrix by means of Equations (87) and (88).

The results for Method 3 using assumed density filtering and interval arithmetic are illustrated in Figure 8 and Figure 9, respectively. The main drawback of this method is that it requires the knowledge of the exact input data $\mathbf{x}$. Because in atmospheric remote sensing that is not the case, Method 3 will be used for a comparison with the other two methods.

3.3. Summary of Numerical Analysis

In

Table 1. Average relative error $\mathbb{E}({\epsilon }_x)\pm \sqrt{\mathbb{E}({[{\epsilon }_x-\mathbb{E}({\epsilon }_x)]}^2)}$ and standard deviation $\mathbb{E}({\sigma }_x)$ over the prediction set for the methods used to solve direct and inverse problems. The parameter x stands for the cloud optical thickness $\tau$ and cloud top height H. In the case of the inverse problem, the results correspond to Method 1 with a deterministic network (1a), Bayes-by-backprop (1b), dropout (1c), and batch normalization (1d); Method 2 with dropout (2a) and Bayes-by-backprop (2b); and Method 3 with assumed density filtering (3a) and interval arithmetic (3b).

Method	x	Error	Std. Deviation
$\begin{array}{c}\mathrm{Direct}\\ \mathrm{Problem}\end{array}$	$\tau$	$-2.94\times {10}^{-2}\pm 6.67\times {10}^{-2}$	$2.04\times {10}^{-1}$
	H	$-5.87\times {10}^{-2}\pm 1.72\times {10}^{-1}$	$3.66\times {10}^{-1}$
1a	$\tau$	$-2.78\times {10}^{-2}\pm 1.30\times {10}^{-1}$	$8.92\times {10}^{-1}$
	H	$-1.96\times {10}^{-2}\pm 1.29\times {10}^{-1}$	$4.36\times {10}^{-1}$
1b	$\tau$	$3.39\times {10}^{-2}\pm 1.66\times {10}^{-1}$	$9.23\times {10}^{-1}$
	H	$1.83\times {10}^{-2}\pm 1.33\times {10}^{-1}$	$8.11\times {10}^{-1}$
1c	$\tau$	$-8.75\times {10}^{-3}\pm 2.25\times {10}^{-2}$	$3.23\times {10}^{-1}$
	H	$3.45\times {10}^{-3}\pm 4.11\times {10}^{-2}$	$2.31\times {10}^{-1}$
1d	$\tau$	$1.01\times {10}^{-2}\pm 2.41\times {10}^{-2}$	$3.54\times {10}^{-1}$
	H	$4.37\times {10}^{-3}\pm 2.73\times {10}^{-2}$	$2.29\times {10}^{-1}$
2a	$\tau$	$1.16\times {10}^{-2}\pm 4.05\times {10}^{-2}$	$8.24\times {10}^{-1}$
	H	$1.63\times {10}^{-2}\pm 4.21\times {10}^{-2}$	$6.72\times {10}^{-1}$
2b	$\tau$	$3.10\times {10}^{-2}\pm 1.38\times {10}^{-1}$	$9.63\times {10}^{-1}$
	H	$3.31\times {10}^{-2}\pm 4.57\times {10}^{-2}$	$4.88\times {10}^{-1}$
3a	$\tau$	$-6.67\times {10}^{-3}\pm 2.48\times {10}^{-2}$	$6.55\times {10}^{-1}$
	H	$8.18\times {10}^{-4}\pm 3.18\times {10}^{-2}$	$4.76\times {10}^{-1}$
3b	$\tau$	$-7.08\times {10}^{-3}\pm 2.53\times {10}^{-2}$	$7.82\times {10}^{-1}$
	H	$1.56\times {10}^{-3}\pm 3.33\times {10}^{-2}$	$6.53\times {10}^{-1}$

, we summarize the results of our numerical simulations by illustrating the average relative error and the standard deviation over the prediction set, $\mathbb{E}({\epsilon }_x)\pm \sqrt{\mathbb{E}({[{\epsilon }_x-\mathbb{E}({\epsilon }_x)]}^2)}$ and $\mathbb{E}({\sigma }_x)$, respectively, where x stands for the cloud optical thickness $\tau$ and the cloud top height H. The accuracy of a method is reflected by the bias of the error $\mathbb{E}({\epsilon }_x)$ and the interval about the mean with length $\sqrt{\mathbb{E}({[{\epsilon }_x-\mathbb{E}({\epsilon }_x)]}^2)}$, while the precision is reflected by the standard deviation $\mathbb{E}({\sigma }_x)$ (which determines the length of the uncertainty interval). Note that (i) $\sqrt{\mathbb{E}({[{\epsilon }_x-\mathbb{E}({\epsilon }_x)]}^2)}$ reproduces the square root of the diagonal elements of the conditional average covariance matrix $\mathbb{E}({\mathcal{C}}_{\mathbf{y}}|{\mathcal{D}}_{\mathrm{test}})$ of all network errors over the test set ${\mathcal{D}}_{\mathrm{test}}$; and (ii) roughly speaking, the epistemic (model) uncertainties are large if there are large variations around the mean.

The results in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 and Table 1 can be summarized as follows:

• For the direct problem, the neural network method used in conjunction with a Bayesian inversion method provides satisfactory accuracy, but does not correctly predict the uncertainty. The reason for this is that the forward model is not nearly linear, which is the main assumption for computing the uncertainty in the retrieval.
• For the inverse problem, the following features are apparent.

  (a)

  Method 1 using a deterministic and Bayes-by-backprop network yields a low accuracy, while the method using dropout and batch-normalized networks provides high accuracy;

  (b)

  Method 2 using a dropout network has an acceptable accuracy. For cloud top height retrieval, the method using a Bayes-by-backprop network has a similar accuracy, but for cloud optical thickness retrieval, the accuracy is low. Possible reasons for the loss of accuracy of a Bayes-by-backprop network can be (i) a non-optimal training and/or the use of the prior $p(\mathit{\omega })=\mathcal{N}(\mathit{\omega };\mathbf{0},\mathsf{I})$ instead of that given by Equation (45) (recall that for $p(\mathit{\omega })=\mathcal{N}(\mathit{\omega };\mathbf{0},\mathsf{I})$, the KL divergence $\mathrm{KL}(q_{\mathit{\theta }}(\mathit{\omega })|p(\mathit{\omega }))$ can be computed analytically).

  (c)

  Method 3 with assumed density filtering and interval arithmetic is comparable with Method 2 using a dropout network, that is, the method has a high accuracy.

  (d)

  From the methods with reasonable accuracy, by using a dropout network, Method 2 predicts similar uncertainties to Method 3 with assumed density filtering and interval arithmetic. In contrast, using dropout and batch normalized networks, Method 1 provides lower uncertainties, dominated by epistemic uncertainties. In general, it seems that Method 1 predicts a too small heteroscedastic uncertainty. Possibly, this is due to the fact that we trained a neural network with exact data, and for this retrieval example, the predictive distribution, represented as a convolution integral over the input noise distribution, does not correctly reproduce the aleatoric uncertainty.

  (e)

  Because $\sqrt{\mathbb{E}({[{\epsilon }_x-\mathbb{E}({\epsilon }_x)]}^2)}<\mathbb{E}({\sigma }_x)$, we deduce that the conditional average covariance matrix $\mathbb{E}({\mathcal{C}}_{\mathbf{y}}|{\mathcal{D}}_{\mathrm{test}})$ does not coincide with the predictive covariance matrix $\mathrm{Cov}(\mathbf{y})$, which reflects the uncertainty.

4. Conclusions

We presented several neural networks for predicting uncertainty in an atmospheric remote sensing. The neural networks are designed in a Bayesian framework and are devoted to the solution of direct and inverse problems.

• For solving the direct problem, we considered a neural network for simulating the radiative transfer model, computed of the epistemic covariance matrix from the statistics of all network errors over the data set, solving the inverse problem by a Bayesian approach, and determined the uncertainty in the retrieval by assuming that the forward model is nearly linear.
• For solving the inverse problem, two neural network methods, relying on different assumptions, were implemented:

  (a)

  The first method uses deterministic and stochastic (Bayes-by-backprop, dropout, and batch normalization) networks to compute the epistemic covariance matrix and under the assumption that the predictive distribution of the network output is the convolution of the predictive distribution for a noise-free input with the input noise distribution, estimates the covariance matrix;

  (b)

  The second method uses dropout and Bayes-by-backprop to learn the heteroscedastic covariance matrix from the data.

In addition, for solving the inverse problem, a third method that uses a dropout network and forward propagates the input noise through the network by using assumed density filtering and interval arithmetic was designed. Because this method requires the knowledge of the exact input data, it was used only for testing purposes.

Our numerical analysis has shown that a dropout network that is used to learn the heteroscedastic covariance matrix from the data is appropriate for predicting the uncertainty associated with the retrieval of cloud parameters from EPIC measurements. In fact, the strengths of a dropout network are (i) its capability to avoid overfitting and (ii) its stochastic character (the method is equivalent to a Bayesian approximation).

All neural network algorithms are implemented in FORTRAN and incorporated in a common tool. In the future, we intend to implement the algorithms in the high-level programming language Python and use the deep learning library PyTorch. This software library has a variety of network architectures that provide auto-differentiation and support GPUs to enable fast and efficient computation. The Python tool will be released through a public repository to make the methods available to the scientific community.