Entropy, Information Theory, Information Geometry and Bayesian Inference in Data, Signal and Image Processing and Inverse Problems ^†

Abstract

The main content of this review article is first to review the main inference tools using Bayes rule, the maximum entropy principle (MEP), information theory, relative entropy and the Kullback–Leibler (KL) divergence, Fisher information and its corresponding geometries. For each of these tools, the precise context of their use is described. The second part of the paper is focused on the ways these tools have been used in data, signal and image processing and in the inverse problems, which arise in different physical sciences and engineering applications. A few examples of the applications are described: entropy in independent components analysis (ICA) and in blind source separation, Fisher information in data model selection, different maximum entropy-based methods in time series spectral estimation and in linear inverse problems and, finally, the Bayesian inference for general inverse problems. Some original materials concerning the approximate Bayesian computation (ABC) and, in particular, the variational Bayesian approximation (VBA) methods are also presented. VBA is used for proposing an alternative Bayesian computational tool to the classical Markov chain Monte Carlo (MCMC) methods. We will also see that VBA englobes joint maximum a posteriori (MAP), as well as the different expectation-maximization (EM) algorithms as particular cases.

1. Introduction

As this paper is an overview and an extension of my tutorial paper in MaxEnt 2014 workshop [1], this Introduction gives a summary of the content of this paper.

The qualification Bayesian refers to the influence of Thomas Bayes [2], who introduced what is now known as Bayes’ rule, even if the idea had been developed independently by Pierre-Simon de Laplace [3]. For this reason, I am asking a question of the community if we shall change Bayes to Laplace and Bayesian to Laplacian or at least mention them both. Whatever the answer, we assume that the reader knows what probability means in a Bayesian or Laplacian framework. The main idea is that a probability law P(X) assigned to a quantity X represents our state of knowledge that we have about it. If X is a discrete valued variable, {P(X = x_n) = p_n; n = 1, ⋯, N} with mutually exclusive values x_n is its probability distribution. When X is a continuous valued variable, p(x) is its probability density function from which we can compute $P(a\le X<b)={\displaystyle {\int }_a^{b}p(x)\mathrm{d}x}$ or any other probabilistic quantity, such as its mode, mean, median, region of high probabilities, etc.

In science, it happens very often that a quantity cannot be directly observed or, even when it can be observed, the observations are uncertain (commonly said to be noisy), by uncertain or noisy, here, I mean that, if we repeat the experiences with the same practical conditions, we obtain different data. However, in the Bayesian approach, for a given experiment, we have to use the data as they are, and we want to infer it from those observations. Before starting the observation and gathering new data, we have very incomplete knowledge about it. However, this incomplete knowledge can be translated in probability theory via an a priori probability law. We will discuss this point later on regarding how to do this. For now, we assume that this can be done. When a new observation (data D) on X becomes available (direct or indirect), we gain some knowledge via the likelihood P(D|X). Then, our state of knowledge is updated combining P(D|X) and P(X) to obtain an a posteriori law P(X|D), which represents the new state of knowledge on X. This is the main esprit of the Bayes rule, which can be summarized as:

$$P(X|D)=P(D|X)P(X)/P(D).$$

(1)

As P(X|D) has to be a probability law, we have:

$$P(D)={\displaystyle \sum _{X}P(D|X)P(X).}$$

(2)

This relation can be extended to the continuous case. Some more details will be given in Section 2.

Associated with a probability law is the quantity of information it contains. Shannon [4] introduced the notion of quantity of information I_n associated with one of the possible values of x_n of X with probabilities P(X = x_n) = p_n to be $I_n=\ln \frac{1}{p_n}=-\ln p_n$ and the entropy H as the expected value

$$H=-{\displaystyle \sum _{n=1}^N{p}_n\ln p_n.}$$

(3)

The word entropy has also its roots in thermodynamics and physics. However, this notion of entropy has no direct link with entropy in physics, even if for a particular physical system, we may attribute a probability law to a quantity of interest of that system and then define its entropy. This information definition of Shannon entropy became the main basis of information theory in many data analyses and the science of communication. More details and extensions about this subject will be given in Section 3.

As we can see up to now, we did not yet discuss how to assign a probability law to a quantity. For the discrete value variable, when X can take one of the N values {x₁, ⋯, x_N} and when we do not know anything else about it, Laplace proposed the “Principe d’indifférence”, where $P(X=x_n)=p_n=\frac{1}{N},\forall n=1,\cdots ,N$, a uniform distribution. However, what if we know more, but not enough to be able to assign the probability law {p₁, ⋯, p_N} completely?

For example, if we know that the expected value is ${\displaystyle {\sum }_n{x}_n{p}_n=d}$, this problem can be handled by considering this equation as a constraint on the probability distribution {p₁, ⋯, p_N}. If we have a sufficient number of constraints (at least N), then we may obtain a unique solution. However, very often, this is not the case. The question now is how to assign a probability distribution {p₁, ⋯, p_N} that satisfies the available constraints. This question is an ill-posed problem in the mathematical sense of Hadamard [5] in the sense that the solution is not unique. We can propose many probability distributions that satisfy the constraint imposed by this problem. To answer this question, Jaynes [6–8] introduced the maximum entropy principle (MEP) as a tool for assigning a probability law to a quantity on which we have some incomplete or macroscopic (expected values) information. Some more details about this MEP, the mathematical optimization problem, the expression of the solution and the algorithm to compute it will be given in Sections 3 and 4.

Kullback [9] was interested in comparing two probability laws and introduced a tool to measure the increase of information content of a new probability law with respect to a reference one. This tool is called either the Kullback–Leibler (KL) divergence, cross entropy or relative entropy. It has also been used to update a prior law when new pieces of information in the form of expected values are given. As we will see later, this tool can also be used as an extension to the MEP of Jaynes. Furthermore, as we will see later, this criterion of comparison of two probability laws is not symmetric: one of the probability laws has to be chosen to be the reference, and then, the second is compared to this reference. Some more details and extensions will be given in Section 5.

Fisher [10] wanted to measure the amount of information that a random variable X carries about an unknown parameter θ upon which its probability law p(x|θ) depends. The partial derivative with respect to θ of the logarithm of this probability law, called the log-likelihood function for θ, is called the score. He showed that the first order moment of the score is zero, but its second order moment is positive and is also equivalent to the expected values of the second derivative of log-likelihood function with respect to θ. This quantity is called the Fisher information. It is also been shown that for the small variations of θ, the Fisher information induces locally a distance in the space of parameters Θ, if we had to compare two very close values of θ. In this way, the notion of the geometry of information is introduced [11,12]. We must be careful here that this geometrical property is related to the space of the parameters Θ for small changes of the parameter for a given family of parametric probability law p(x|θ) and not in the space of probabilities. However, for two probability laws p₁(x) = p(x|θ₁) and p₂(x) = p(x|θ₂) in the same exponential family, the Kullback–Leibler divergence KL [p₁ : p₂] induces a Bregman divergence B[θ₁ : θ₂] between the two parameters [13,14]. More details will be given in Section 8.

At this stage, we have almost introduced all of the necessary tools that we can use for different levels of data, signal and image processing. In the following, we give some more details for each of these tools and their inter-relations. Then, we review a few examples of their use in different applications. As examples, we demonstrate how these tools can be used in independent components analysis (ICA) and source separation, data model selection, in spectral analysis of the signals and in inverse problems, which arise in many sciences and engineering applications. At the end, we focus more on the Bayesian approach for inverse problems. We present some details concerning unsupervised methods, where the hyper parameters of the problem have to be estimated jointly with the unknown quantities (hidden variables). Here, we will see how the Kullback–Leibler divergence can help approximate Bayesian computation (ABC). In particular, some original materials concerning variational Bayesian approximation (VBA) methods are presented.

2. Bayes Rule

Let us introduce things very simply. If we have two discrete valued related variables X and Y, for which we have assigned probability laws P(X) and P(Y), respectively, and their joint probability law P(X, Y), then from the sum and product rule, we have:

$$P(X,Y)=P(X|Y)P(Y)=P(Y|X)P(X)$$

(4)

where P(X, Y) is the joint probability law, $P(X)={\displaystyle {\sum }_{Y}P(X,Y)}$ and $P(Y)={\displaystyle {\sum }_{X}P(X,Y)}$ are the marginals and $P(X|Y)=P(X,Y)/P(Y)$ and $P(Y|X)=P(X,Y)/P(X)$ are the conditionals. Now, consider the situation where Y can be observed, but not X. Because these two quantities are related, we may want to infer X from the observations on Y. Then, we can use:

$$P(X|Y)=\frac{P(Y|X)P(X)}{P(Y)}$$

(5)

which is called the Bayes rule.

This relation is extended to the continuous valued variables using the measure theory [15,16]:

$$p(x|y)=\frac{p(y|x)p(x)}{p(y)}$$

(6)

with:

$$p(y)={\displaystyle \int p(y|x)p(x)\mathrm{d}x}.$$

(7)

More simply, the Bayes rule is often written as:

$$p(x|y)\propto p(y|x)p(x).$$

(8)

This writing can be used when we want to use p(x|y) to compute quantities that are only dependent on the shape of p(x|y), such as the mode, the median or quantiles. However, we must be careful that the denominator is of importance in many other cases, for example when we want to compute expected values. There is no need for more sophisticated mathematics here if we want to use this approach.

As we mentioned, the main use of this rule is in particular when X can not be observed (unknown quantity), but Y is observed and we want to infer X. In this case, the term p(y|x) is called the likelihood (of unknown quantity X in the observed data y), p(x) is called a priori and p(x|y) a posteriori. The likelihood is assigned using the link between the observed Y and the unknown X, and p(x) is assigned using the prior knowledge about it. The Bayes rule then is a way to do state of knowledge fusion. Before taking into account any observation, our state of knowledge is represented by p(x), and after the observation of Y, it becomes p(x|y).

However, in this approach, two steps are very important. The first step is the assigning of p(x) and p(y|x) before being able to use the Bayes rule. As noted in the Introduction and as we will see later, we need other tools for this step. The second important step is after: how to use p(x|y) to summarize it. When X is just a scalar value variable, we can do this computation easily. For example, we can compute the probability that X is in the interval [a, b] via:

$$P(a\le X<b|y)={\displaystyle {\int }_a^{b}p(x|y)}\mathrm{d}x.$$

(9)

However, when the unknown becomes a high dimensional vectorial variable X, as is the case in many signal and image processing applications, this computation becomes very costly [17]. We may then want to summarize p(x|y) by a few interesting or significant point estimates. For example, compute the maximum a posteriori (MAP) solution:

$${\widehat{x}}_{\text{MAP}}=\underset{x}{\arg \max }\{p(x|y)\},$$

(10)

the expected a posteriori (EAP) solution:

$${\widehat{x}}_{\text{EAP}}={\displaystyle \int xp(x|y)\mathrm{d}x},$$

(11)

the domains of X which include an integrated probability mass of more than some desired value (0.95 for example):

$$\left[x_1,x_2\right]:{\displaystyle {\int }_{x_1}^{x_2}p(x|y)\mathrm{d}x=.95},$$

(12)

or any other questions, such as median or any α-quantiles:

$$x_q:{\displaystyle {\int }_{-\infty }^{x_q}p(x|y)\mathrm{d}x=(1-\alpha )}.$$

(13)

As we see, computation of MAP needs an optimization algorithm, while these last three cases need integration, which may become very complicated for high dimensional cases [17].

We can also just explore numerically the whole space of the distribution using the Markov chain Monte Carlo (MCMC) [18–26] or any other sampling techniques [17]. In the scalar case (one dimension), all of these computations can be done numerically very easily. For the vectorial case, when the dimensions become large, we need to develop specialized approximation methods, such as VBA and algorithms to do these computations. We give some more details about these when using this approach for inverse problems in real applications.

Remarks on notation used for the expected value in this paper: For a variable X with the probability density function (pdf) p(x) and any regular function h(X), we use indifferently:

$$\mathrm{E}\{X\}=E_p\{X\}=<X>=<X{>}_p={\displaystyle \int xp(x)\mathrm{d}x}$$

and:

$$\mathrm{E}\{h(X)\}=E_p\{h(X)\}=<h(X)>=<h(X){>}_p={\displaystyle \int h(x)p(x)\mathrm{d}x}.$$

As an example, as we will say later, the entropy of p(x) is noted indifferently:

$$H[p]=\mathrm{E}\{-\ln (p(X))\}={\mathrm{E}}_p\{-\ln p(X)\}=<-\ln p(X)>=<-\ln p(X){>}_p=-{\displaystyle \int p(x)\ln p(x)\mathrm{d}x}.$$

For any conditional probability density function (pdf) p(x|y) and any regular function h(X), we use indifferently:

$$\mathrm{E}\{X|y\}=\mathrm{E}{}_{p(x|y)}\{X\}=<X|y>=<X{>}_{p(x|y)}={\displaystyle \int xp(x|y)\mathrm{d}x}$$

and:

$$\mathrm{E}\{h(X)|y\}={\mathrm{E}}_{p(x|y)}\{h(X)\}=<h(X)|y>=<h(X){>}_{p(x|y)}={\displaystyle \int h(x)p(x|)\mathrm{d}x}.$$

As another example, as we will see later, the relative entropy of p(x) over q(x) is noted indifferently:

$$D[p|q]={\mathrm{E}}_p\left\{-\ln \frac{p(X)}{q(X)}\right\}=<-\ln \frac{p(X)}{q(X)}{>}_{p(x)}-{\displaystyle \int p(x)\ln \frac{p(x)}{q(x)}\mathrm{d}x}$$

and when there is not any ambiguity in the integration variable, we may omit it. For example, we may note:

$$D[p|q]={\mathrm{E}}_p\left\{-\ln \frac{p}{q}\right\}=<-\ln \frac{p}{q}{>}_p=-{\displaystyle \int p\ln \frac{p}{q}}.$$

Finally, when we have two variables X and Y with their joint pdf p(x, y), their marginals p(x) and p(y) and their conditionals p(x|y) and p(y|x), we may use the following notations:

$$\mathrm{E}\{h(X)|y\}={\mathrm{E}}_{p(x|y)}\{h(X)\}={\mathrm{E}}_{X|Y}\{h(X)\}=<h(X)|y>=<h(X){>}_{p(x|y)}={\displaystyle \int h(x)p(x|y)\mathrm{d}x}.$$

3. Quantity of Information and Entropy

3.1. Shannon Entropy

To introduce the quantity of information and the entropy, Shannon first considered a discrete valued variable X taking values {x₁, ⋯, x_N} with probabilities {p₁, ⋯, p_N} and defined the quantities of information associated with each of them as $I_n=\ln \frac{1}{p_n}=-\ln p_n$ and its expected value as the entropy:

$$\mathrm{H}[X]=-{\displaystyle \sum _{i=1}^N{p}_i\ln p_i}.$$

(14)

Later, this definition is extended to the continuous case by:

$$\mathrm{H}[X]=-{\displaystyle \int p(x)\ln p(x)\mathrm{d}x}.$$

(15)

By extension, if we consider two related variables (X, Y) with the probability laws, joint p(x, y), marginals, p(x), p(y), and conditionals, p(y|x), p(x|y), we can define, respectively, the joint entropy:

$$\mathrm{H}[X,Y]=-{\displaystyle \iint p(x,y)\ln p(x,y)\mathrm{d}x\mathrm{d}y},$$

(16)

as well as H[X], H[Y], H[Y|X] and H[X|Y].

Therefore, for any well-defined probability law, we can have an expression for its entropy. H[X], H[Y], H[Y|X], H[X|Y] and H[X, Y], which should better be noted as H[p(x)], H[p(y)], H[p(y|x)], H[p(x|y)] and H[p(x, y)].

3.2. Thermodynamical Entropy

Entropy is also a property of thermodynamical systems introduced by Clausius [27]. For a closed homogeneous system with reversible transformation, the differential entropy δS is related to δQ the incremental reversible transfer of heat energy into that system by δS = δQ/T with T being the uniform temperature of the closed system.

It is very hard to establish a direct link between these two entropies. However, in statistical mechanics, thanks to Boltzmann, Gibbs and many others, we can establish some link if we consider the microstates (for example, the number, positions and speeds of the particles) and the macrostates (for example, the temperature T, pressure P, volume V and energy E) of the system and if we assign a probability law to microstates and consider the macrostates as the average (expected values) of some functions of those microstates. Let us give a very brief summary of some of those interpretations.

3.3. Statistical Mechanics Entropy

The interpretation of entropy in statistical mechanics is the measure of uncertainty that remains about the state of a system after its observable macroscopic properties, such as temperature (T), pressure (P) and volume (V), have been taken into account. For a given set of macroscopic variables T, P and V, the entropy measures the degree to which the probability of the system is spread out over different possible microstates. In contrast to the macrostate, which characterizes plainly observable average quantities, a microstate specifies all atomic details about the system, including the position and velocity of every atom. Entropy in statistical mechanics is a measure of the number of ways in which the microstates of the system may be arranged, often taken to be a measure of “disorder” (the higher the entropy, the higher the disorder). This definition describes the entropy as being proportional to the natural logarithm of the number of possible microscopic configurations of the system (microstates), which could give rise to the observed macroscopic state (macrostate) of the system. The proportionality constant is the Boltzmann constant.

3.4. Boltzmann Entropy

Boltzmann described the entropy as a measure of the number of possible microscopic configurations Ω of the individual atoms and molecules of the system (microstates) that comply with the macroscopic state (macrostate) of the system. Boltzmann then went on to show that k ln Ω was equal to the thermodynamic entropy. The factor k has since been known as Boltzmann’s constant.

In particular, Boltzmann showed that the entropy S of an ideal gas is related to the number of states of the molecules (microstates Ω) with a given temperature (macrostate):

$$S=k\ln \mathrm{\Omega }$$

(17)

3.5. Gibbs Entropy

The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. Therefore, the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i and p_i is its probability that it occurs during the system’s fluctuations, then the entropy of the system is:

$$S=-k{\displaystyle \sum _{i=1}^N{p}_i\ln p_i}.$$

(18)

where k is again the physical constant of Boltzmann, which, like the entropy, has units of heat capacity. The logarithm is dimensionless. It is interesting to note that Relation (17) can be obtained from Relation (18) when the probability distribution is uniform over the volume Ω [28–30].

4. Relative Entropy or Kullback–Leibler Divergence

Kullback wanted to compare the relative quantity of information between two probability laws p₁ and p₂ on the same variable X. Two related notions have been defined:

• Relative Entropy of p₁ with respect to p₂:

  $$D[p_1:p_2]=-{\displaystyle \int p_1(x)\ln \frac{p_1(x)}{p_2(x)}}\mathrm{d}x$$

  (19)
• Kullback–Leibler divergence of p₁ with respect to p₂:

  $$\mathrm{KL}[p_1:p_2]=-D[p_1:p_2]={\displaystyle \int p_1(x)\ln \frac{p_1(x)}{p_2(x)}}\mathrm{d}x$$

  (20)

We may note that:

• KL [q : p] ≥ 0,
• KL [q : p] = 0, if q = p and
• KL [q : p₀] ≥ KL [q : p₁] + KL [p₁ : p₀].
• KL [q : p] is invariant with respect to a scale change, but is not symmetric.
• A symmetric quantity can be defined as:

  $$\mathrm{J}[q,p]=\frac{1}{2}(\mathrm{KL}[q:p]+\mathrm{KL}[p:q]).$$

  (21)

5. Mutual Information

The purpose of mutual information is to compare two related variables Y and X. It can be defined as the expected amount of information that one gains about Xif we observe the value of Y, and vice versa. Mathematically, the mutual information between X and Y is defined as:

$$\mathrm{I}[Y,X]=\mathrm{H}[X]-\mathrm{H}[X|Y]=\mathrm{H}[Y]-\mathrm{H}[Y|X]$$

(22)

or equivalently as:

$$\mathrm{I}[Y,X]=D[p(X,Y):p(X)p(Y)].$$

(23)

With this definition, we have the following properties:

$$\mathrm{H}[X,Y]=\mathrm{H}[X]+\mathrm{H}[Y|X]=\mathrm{H}[Y]+\mathrm{H}[X|Y]=\mathrm{H}[X]+\mathrm{H}[Y]-\mathrm{I}[Y,X]$$

(24)

and:

$$\begin{array}{l}\mathrm{I}[Y,X]={\mathrm{E}}_X\{D[p(Y|X)]:p(Y)\}\stackrel{\mathrm{\Delta }}{=}{\displaystyle \int D[p(y|x):p(y)]p(x)\mathrm{d}x}\\ =E_Y\{D[p(X|Y)]:p(X)\}\stackrel{\mathrm{\Delta }}{=}{\displaystyle \int D[p(x|y):p(x)]p(y)\mathrm{d}y}.\end{array}$$

(25)

We may also remark on the following property:

• I[Y, X] is a concave function of p(y) when p(x|y) is fixed and a convex function of p(x|y) when p(y) is fixed.
• I[Y, X] ≥ 0 with equality only if X and Y are independent.

6. Maximum Entropy Principle

The first step before applying any probability rules for inference is to assign a probability law to a quantity. Very often, the available knowledge on that quantity can be described mathematically as the constraints on the desired probability law. However, in general, those constraints are not enough to determine in a unique way that probability law. There may exist many solutions that satisfy those constraints. We need then a tool to select one.

Jaynes introduced the MEP [8], which can be summarized as follows: When we do not have enough constraints to determine a probability law that satisfies those constraints, we may select between them the one with maximum entropy.

Let us be now more precise. Let us assume that the available information on that quantity X is the form of:

$$E\left\{{\varphi }_k(X)\right\}=d_k,k=1,\dots ,K.$$

(26)

where ϕ_k are any known functions. First, we assume that such probability laws exist by defining:

$$\mathcal{P}=\left\{p(x):{\displaystyle \int {\varphi }_k(x)p(x)\mathrm{d}x=\mathrm{d}k,k=0,\dots ,K}\right\}$$

with ϕ₀ = 1 and d₀ = 1 for the normalization purpose. Then, the MEP is written as an optimization problem:

$$p_{ME}(x)=\underset{p\in \mathcal{P}}{\arg \max }\left\{H[p]=-{\displaystyle \int p(x)\ln p(x)\mathrm{d}x}\right\}$$

(27)

whose solution is given by:

$$p_{ME}(x)=\frac{1}{Z(\mathbf{\lambda })}\exp \left[-{\displaystyle \sum _{k=1}^K{\lambda }_k{\varphi }_k(x)}\right]$$

(28)

where Z(λ), called the partition function, is given by: $Z(\mathbf{\lambda })={\displaystyle \int \exp [-{\displaystyle {\sum }_{k=1}^K{\lambda }_k{\varphi }_k(x)}]\mathrm{d}x}$ and $\mathbf{\lambda }={\left[{\lambda }_1,\dots ,{\lambda }_K\right]}^{\prime }$ have to satisfy:

$$-\frac{\partial \ln Z(\mathbf{\lambda })}{\partial {\lambda }_k}=d_k,k=1,\dots ,K$$

(29)

which can also be written as −∇_λ ln Z(λ) = d. Different algorithms have been proposed to compute numerically the ME distributions. See, for example, [31–37]

The maximum value of entropy reached is given by:

$$H_{\max }=\ln Z(\mathbf{\lambda })+{\mathbf{\lambda }}^{\prime }\mathit{d}.$$

(30)

This optimization can easily be extended to the use of relative entropy by replacing H(p) by D[p : q] where q(x) is a given reference of a priori law. See [9,38,39] and [34,40–42] for more details.

7. Link between Entropy and Likelihood

Consider the problem of the parameter estimation θ of a probability law p(x|θ) from an n-element sample of data x = {x₁, ⋯, x_n}.

The log-likelihood of θ is defined as:

$$L(\mathit{\theta })=\ln {\displaystyle \prod _{i=1}^{n}p(x_i|\theta )}={\displaystyle \sum _{i=1}^n\ln p(x_i|\theta )}.$$

(31)

Maximizing L(θ) with respect to θ gives what is called the maximum likelihood (ML) estimate of θ.

Noting that L(θ) depends on n, we may consider $\frac{1}{n}L(\mathbf{\theta })$ and define:

$$\overline{L}(\mathit{\theta })=\underset{x\mapsto \infty }{\lim }\frac{1}{n}L(\mathit{\theta })=\mathrm{E}\{\ln p(x|\mathit{\theta })\}={\displaystyle \int p(x|\mathit{\theta }*)\ln p(x|\mathit{\theta })\mathrm{d}x},$$

(32)

where θ^* is the right answer and p(x|θ^*) its corresponding probability law. We may then remark that:

$$D[p(x|\mathit{\theta }*):p(x|\mathit{\theta })]=-{\displaystyle \int p(x|\mathit{\theta }*)\ln \frac{p(x|\mathit{\theta })}{p(x|\mathit{\theta }*)}\mathrm{d}x}=-{\displaystyle \int p(x|\mathit{\theta }*)\ln p(x|\mathit{\theta }*)\mathrm{d}x+\overline{L}(\mathit{\theta })}.$$

(33)

The first term in the right-hand side being a constant, we derive that:

$$\underset{\mathit{\theta }}{\arg \max }\{D[p(x|\mathit{\theta }*);p(x|\mathit{\theta })]\}=\underset{\mathit{\theta }}{\arg \max }\{\overline{L}(\mathit{\theta })\}.$$

In this way, there is a link between the maximum likelihood and maximum relative entropy solutions [24].

There is also a link between the maximum relative entropy and the Bayes rule. See, for example, [43,44] and their corresponding references.

8. Fisher Information, Bregman and Other Divergences

Fisher [10] was interested in measuring the amount of information that samples of a variable X carries about an unknown parameter θ upon which its probability law p(x|θ) depends. For a given sample of observation x and its probability law p(x|θ), the function L(θ) = p(x|θ) is called the likelihood of θ in the sample x. He called the score of x over θ the partial derivative with respect to θ of the logarithm of this function:

$$S(x|\theta )=\frac{\partial \ln p(x|\theta )}{\partial \theta }$$

(34)

He also showed that the first order moment of the score is zero:

$$\mathrm{E}\{S(X|\theta )\}=\mathrm{E}\left\{\frac{\partial \ln p(x|\theta )}{\partial \theta }\right\}=0$$

(35)

but its second order moment is positive and is also equivalent to the expected values of the second derivative of the log-likelihood function with respect to θ.

$$\mathrm{E}\{S^2(X|\theta )\}=\mathrm{E}\left\{{\left|\frac{\partial \ln p(x|\theta )}{\partial \theta }\right|}^2\right\}=\mathrm{E}\left\{\frac{{\partial }^2\ln p(x|\theta )}{\partial {\theta }^2}\right\}=F$$

(36)

This quantity is called the Fisher information [14].

It is also shown that for the small variations of θ, the Fisher information induces locally a distance in the space of parameters Θ, if we had to compare two very close values of θ. In this way, the notion of the geometry of information is introduced. The main steps for introducing this notion are the following: Consider D [p(x|θ^*) : p(x|θ^* + ∆θ)] and assume that ln p(x|θ) can be developed in a Taylor series. Then, keeping the terms up to the second order, we obtain:

$$D[p(x|\mathit{\theta }*):p(x|\mathit{\theta }*+\mathrm{\Delta }\mathit{\theta })]\simeq \frac{1}{2}\mathrm{\Delta }{\mathit{\theta }}^{\prime }\mathit{F}(\theta *)\mathrm{\Delta }\mathit{\theta }.$$

(37)

where F is the Fisher information:

$$\mathit{F}(\theta *)=\mathrm{E}\left\{\frac{{\partial }^2\ln p(x|\mathit{\theta })}{\partial {\mathit{\theta }}^{\prime }\partial \mathit{\theta }}{|}_{\mathit{\theta }=\mathit{\theta }*}\right\}.$$

(38)

We must be careful here that this geometry property is related to the space of the parameters Θ for a given family of parametric probability law p(x|θ) and not in the space of probabilities. However, for two probability laws p₁(x) = p(x|θ₁) and p₂(x) = p(x|θ₂) in the same exponential family, the Kullback–Leibler divergence KL [p₁ : p₂] induces a Bregman divergence B[θ₁|θ₂] between the two parameters [14,45–48].

To go further into detail, let us extend the discussion about the link between Fisher information and KL divergence, as well as other divergences, such as f-divergences, Rényi’s divergences and Bregman divergences.

• f-divergences:
  The f-divergences, which are a general class of divergences, indexed by convex functions f, that include the KL divergence as a special case. Let f: (0, ∞) ↦ R be a convex function for which f(1) = 0. The f-divergence between two probability measures P and Q is defined by:

  $$D_f[P:Q]={\displaystyle \int qf\left(\frac{p}{q}\right)}$$

  (39)
  Every f-divergence can be viewed as a measure of distance between probability measures with different properties. Some important special cases are:

  • – f(x) = x ln x gives KL divergence: $\mathrm{KL}[P:Q]={\displaystyle \int p\ln \left(\frac{p}{q}\right)}$.
  • – $f(x)=|x-1|/2$ gives total variation distance: $\mathrm{TV}[P,Q]={\displaystyle \int |p-q|/2}$.
  • – $f(x)={\left(\sqrt{x}-1\right)}^2$ gives the square of the Hellinger distance: $H^2[P,Q]={\displaystyle \int {\left(\sqrt{p}-\sqrt{q}\right)}^2}$.
  • – f(x) = (x − 1)² gives the chi-squared divergence: ${\chi }^2[P:Q]={\displaystyle \int \frac{{(p-q)}^2}{q}}$.
• Rényi divergences:
  These are another generalization of the KL divergence. The Rényi divergence between two probability distributions P and Q is:

  $$D_{\alpha }[P:Q]=\frac{1}{\alpha -1}\ln {\displaystyle \int p^{\alpha }{q}^{1-\alpha }}.$$

  (40)

  When α = 1, by a continuity argument, D_α[P : Q] converges to KL [P : Q].
  $D_{1/2}[P,Q]=-2\ln {\displaystyle \int \sqrt{pq}}$ is called Bhattacharyya divergence (closely related to Hellinger distance). Interestingly, this quantity is always smaller than KL:

  $$D_{1/2}[P:Q]\le \mathrm{KL}[P:Q].$$

  (41)
  As a result, it is sometimes easier to derive risk bounds with $D_{1/2}$ as the loss function as opposed to KL.
• Bregman divergences:
  The Bregman divergences provide another class of divergences that are indexed by convex functions and include both the Euclidean distance and the KL divergence as special cases. Let ϕ be a differentiable strictly convex function. The Bregman divergence B_ϕ is defined by:

  $$B_{\varphi }[\mathbf{x}:\mathbf{y}]=\varphi (\mathbf{x})-\varphi (\mathbf{y})-\langle \mathbf{x}-\mathbf{y},\nabla \varphi (\mathbf{y})\rangle$$

  (42)

  where $<\mathbf{x},\mathbf{y}>={\displaystyle {\sum }_j{x}_i{y}_i}$ here means the scalar product of x and y and where the domain of ϕ is a space where convexity and differentiability make sense (e.g., whole or a subset of R^d or an L_p space). For example, $\varphi (\mathbf{x})={\Vert \mathbf{x}\Vert }^2$ and R^d gives the Euclidean distance:

  $$B_{\varphi }[\mathbf{x}:\mathbf{y}]=\varphi (\mathbf{x})-\varphi (\mathbf{y})-\langle \mathbf{x}-\mathbf{y},\nabla \varphi (\mathbf{y})\rangle ={\Vert \mathbf{x}\Vert }^2-{\Vert \mathbf{y}\Vert }^2-\langle \mathbf{x}-\mathbf{y},2\mathbf{y}\rangle ={\Vert \mathbf{x}-\mathbf{y}\Vert }^2$$

  (43)

  and $\varphi (\mathbf{x})={\displaystyle {\sum }_j{x}_j\ln x_j}$ on the simplex in R^d gives the KL divergence:

  $$B_{\varphi }[\mathbf{x}:\mathbf{y}]={\displaystyle \sum _j{x}_j\ln x_j}-{\displaystyle \sum _j{y}_j\ln y_j}-{\displaystyle \sum _j(x_j-y_j)(1+\ln y_j)}={\displaystyle \sum _j{x}_j\ln \frac{x_j}{y_j}}=\mathrm{KL}[\mathbf{x}:\mathbf{y}]$$

  (44)

  where it is assumed ${\displaystyle {\sum }_j{x}_j}={\displaystyle {\sum }_j{y}_j}=1$.

Let X be a quantity taking values in the domain of ϕ with a probability distribution function p(x).

Then, E_p(x) {B_ϕ(X, m)} is minimized over m in the domain of ϕ at m = E {X}:

$$\widehat{m}=\underset{m}{\arg \max }\{B_{\varphi }(X,m)\}=\mathrm{E}\{X\}.$$

Moreover, this property characterizes Bregman divergence. When applied to the Bayesian approach, this means that, using the Bregman divergence as the loss function, the Bayes estimator is the posterior mean. This point is detailed in the following.

Links between all of these through an example:

Let us consider the Bayesian parameter estimation where we have some data y, a set of parameters x, a likelihood p(y|x) and a prior π(x), which gives the posterior $p(x|y)\propto p(y|x)\pi (x)$. Let us also consider a cost function $C[x,\tilde{x}]$ in the parameter space x ∈ X. The classical Bayesian point estimation of x is expressed as the minimizer of an expected risk:

$$\widehat{\mathbf{x}}=\underset{\tilde{\mathbf{x}}}{\arg \min }\left\{\overline{C}(\tilde{\mathbf{x}})\right\}$$

(45)

where:

$$\overline{C}(\tilde{\mathbf{x}})={\mathrm{E}}_{p(x|y)}\{C[\mathbf{x},\tilde{\mathbf{x}}]\}={\displaystyle \int C[\mathbf{x},\tilde{\mathbf{x}}]p(\mathbf{x}|\mathbf{y})\mathrm{d}x}$$

It is very well known that the mean squared error estimator, which corresponds to $C[\mathbf{x},\tilde{\mathbf{x}}]={\Vert \mathbf{x}-\tilde{\mathbf{x}}\Vert }^2$, is the posterior mean. It is now interesting to know that choosing $C[\mathbf{x},\tilde{\mathbf{x}}]$ to be any Bregman divergence $B_{\varphi }[x,\tilde{x}]$, we obtain also the posterior mean:

$$\widehat{\mathbf{x}}=\underset{\tilde{\mathbf{x}}}{\arg \min }\{{\overline{B}}_{\varphi }(\tilde{\mathbf{x}})\}={\mathrm{E}}_{p(\mathbf{x}|\mathbf{y})}\left\{{\displaystyle \int \mathbf{x}p(\mathbf{x}|\mathbf{y})\mathrm{d}\mathbf{x}}\right\}$$

(46)

where:

$${\overline{B}}_{\varphi }(\tilde{\mathbf{x}})={\mathrm{E}}_{p(\mathbf{x}|\mathbf{y})}\{D_{\varphi }[\mathbf{x},\tilde{\mathbf{x}}]\}={\displaystyle \int B_{\varphi }[\mathbf{x},\tilde{\mathbf{x}}]p(\mathbf{x}|\mathbf{y})\mathrm{d}\mathbf{x}}$$

Consider now that we have two prior probability laws π₁(x) and π₂(x), which give rise to two posterior probability laws p₁(x|y) and p₂(x|y). If the prior laws and the likelihood are in the exponential families, then the posterior laws are also in the exponential family. Let us note them as p₁(x|y; θ₁) and p₂(x|y; θ₂), where θ₁ and θ₂ are the parameters of those posterior laws. We then have the following properties:

• KL [p₁ : p₂] is expressed as a Bregman divergence B[θ₁ : θ₂].
• A Bregman divergence B[x₁ : x₂] is induced when KL [p₁ : p₂] is used to compare the two posteriors.

9. Vectorial Variables and Time Indexed Process

The extension of the scalar variable to the finite dimensional vectorial case is almost immediate. In particular, for the Gaussian case $p(\mathit{x})=\mathcal{N}(\mathit{x}|\mu ,\mathit{R})$, the mean becomes a vector μ = E{X}, and the variances are replaced by a covariance matrix: $\mathit{R}=\mathrm{E}\left\{(\mathit{X}-\mu ){\left(\mathit{X}-\mu \right)}^{\prime }\right\}$; and almost all of the quantities can be defined immediately. For example, for a Gaussian vector $p(\mathit{x})=\mathcal{N}(\mathit{x}|0,\mathit{R})$, the entropy is given by [49]:

$$H=\frac{n}{2}\ln (2\mathrm{\pi })+\frac{1}{2}\ln \left(|\det (\mathit{R})|\right)$$

(47)

and the relative entropy of $\mathcal{N}(x|\mathbf{0},\mathit{R})$$\mathcal{N}(\mathit{x}|0,\mathit{R})$with respect to $\mathcal{N}(\mathit{x}|0,\mathbf{S})$ is given by:

$$D=\frac{1}{2}\left(\mathrm{tr}(\mathit{R}{\mathit{S}}^{-1})-\log \frac{|\det (\mathit{R})|}{\det (\mathit{S})}-n\right).$$

(48)

The notion of time series or processes needs extra definitions. For example, for a random time series X(t), we can define p(X(t)), ∀t, the expected value time series $\overline{x}(t)=\mathrm{E}\left\{X(t)\right\}$ and what is called the autocorrelation function Γ(t, τ) = E {X(t) X(t + τ)}. A time series is called stationary when these quantities does not depend on t, i.e., $\overline{x}(t)=m$ and Γ(t, τ) = Γ(τ) [50]. Another quantity of interest for a stationary time series is its power spectral density (PSD) function:

$$S(\omega )=\mathrm{FT}\{\Gamma (\tau )\}={\displaystyle \int \Gamma (\tau )\exp [-j\omega \tau ]\mathrm{d}\tau }.$$

(49)

When X(t) is observed on times t = nΔT with ΔT = 1, we have X(n), and for a sample {X(1),⋯, X(N)}, we may define the mean μ = E {X} and the covariance matrix $\sum =\mathrm{E}\{(\mathit{X}-\mu ){(\mathit{X}-\mu )}^{\prime }\}$.

With these definitions, it can easily been shown that the covariance matrix of a stationary Gaussian process is Toeplitz [49]. It is also possible to show that the entropy of such a process can be expressed as a function of its PSD function:

$$\underset{n\to \infty }{\lim }\frac{1}{n}H(p)=\frac{1}{2\pi }{\displaystyle {\int }_{-\pi }^{\pi }\ln S(\omega )\mathrm{d}\omega }.$$

(50)

For two stationary Gaussian processes with two spectral density functions S₁(ω) and S₂(ω), we have:

$$\underset{n\to \infty }{\lim }\frac{1}{n}D[p_1:p_2]=\frac{1}{4\pi }{\displaystyle {\int }_{-\pi }^{\pi }\left(\frac{S_1(\omega )}{S_2(\omega )}-\ln \frac{S_1(\omega )}{S_2(\omega )}-1\right)}\mathrm{d}\omega$$

(51)

where we find the Itakura–Saito distance in the spectral analysis literature [50–53].

These definitions and expressions have often been used in time series analysis. In what follows, we give a few examples of the different ways these notions and quantities have been used in different applications of data, signal and image processing.

10. Entropy in Independent Component Analysis and Source Separation

Given a vector of time series x(t), the independent component analysis (ICA) consists of finding a separating matrix B, such that the components y(t) = Bx(t) are as independent as possible. The notion of entropy is used here as a measure of independence. For example, to find B, we may choose $D\left[p(y):{\displaystyle {\prod }_j{p}_j(y_j)}\right]$ as a criterion of independence of the components y_j. The next step is to choose a probability law p(x) from which we can find an expression for p(y) from which we can find an expression for $D\left[p(y):{\displaystyle {\prod }_j{p}_j(y_j)}\right]$ as a function of the matrix B, which can be optimized to obtain it.

The ICA problem has a tight link with the source separation problem, where it is assumed that the measured time series x(t) is a linear combination of the sources s(t), i.e., x(t) = As(t), with A being the mixing matrix. The objective of source separation is then to find the separating matrix B = A⁻¹.

To see how the entropy is used here, let us note y = Bx. Then,

$$p_Y(\mathbf{y})=\frac{1}{|\partial \mathbf{y}/\partial \mathit{x}|}{p}_X(\mathit{x})\to H(\mathbf{y})=-\mathrm{E}\{\ln p_Y(\mathbf{y})\}=\mathrm{E}\left\{\ln |\partial \mathbf{y}/\partial \mathrm{x}|\right\}-H(\mathit{x}).$$

(52)

H(y) is used as a criterion for ICA or source separation. As the objective in ICA is to obtain y in such a way that its components become as independent as possible, the separating matrix B has to maximize H(y). Many ICA algorithms are based on this optimization [54–65]

11. Entropy in Parametric Modeling and Model Selection

Determining the order of a model, i.e., the dimension of the vector parameter θ in a probabilistic model p(x|θ), is an important subject in many data and signal processing problems. As an example, in autoregressive (AR) modeling:

$$x(n)={\displaystyle \sum _{k=1}^K{\theta }_{k}x(n-k)+\epsilon (n)}$$

(53)

where θ = {θ₁⋯, θ_K}, we may want to compare two models with two different values of K.

When the order K is fixed, the estimation of the parameters θ is a very well-known problem, and there are likelihood based [66] or Bayesian approaches for that [67]. The determination of the order is however more difficult [68]. Between the tools, we may mention here the Bayesian methods [69–74], but also the use of relative entropy D [p(x|θ^*): p(x|θ)], where θ^* represents the vector of the parameters of dimension K^* and θ and the vector θ with dimension K ≤ K^*. In such cases, even if the two probability laws to be compared have parameters with different dimensions, we can always use the KL [p(x|θ^*): p(x|θ)] to compare them. The famous criterion of Akaike [75–78] uses this quantity to determine the optimal order. For a linear parameter model with Gaussian probability laws and likelihood-based methods, there are analytic solutions for it [68].

12. Entropy in Spectral Analysis

Entropy and MEP have been used in different ways in the spectral analysis problem. It has been an important subject of signal processing for the decades. Here, we are presenting, in a brief way, these different approaches.

12.1. Burg’s Entropy-Based Method

A classical one is Burg’s entropy method [79], which can be summarized as follows: Let X(n) be a stationary, centered process, and assume we have as data a finite number of samples (lags) of its autocorrelation function:

$$r(k)=\mathrm{E}\{X(n)X(n+k)\}=\frac{1}{2\pi }{\displaystyle {\int }_{-\pi }^{\pi }S(\omega )\exp [jk\omega ]\mathrm{d}\omega },k=0,\dots ,K.$$

(54)

The task is then to estimate its power spectral density function:

$$S(\omega )={\displaystyle \sum _{k=-\infty }^{\infty }r(k)\exp [-jk\omega ]}$$

As we can see, due to the fact that we have only the elements of the right-hand for k = −K, ⋯, +K, the problem is ill posed. To obtain a probabilistic solution, we may start by assigning a probability law p(x) to the vector $\underset{¯}{X}=[X(0),\dots ,X(N-1){]}^{\prime }$. For this, we can use the principle of maximum entropy (PME) with the data as constraints (54). As these constraints are the second order moments, the PME solution is a Gaussian probability law: $\mathcal{N}(\mathit{x}|0,\mathit{R})$. For a stationary Gaussian process, when the number of samples N → ∞, the expression of the entropy becomes:

$$H={\displaystyle {\int }_{-\pi }^{\pi }\ln S(\omega )\mathrm{d}\omega }.$$

(55)

This expression is called Burg’s entropy [79]. Thus, Burg’s method consists of maximizing H subject to the constraints (54). The solution is:

$$S(\omega )=\frac{1}{{\left|{\displaystyle \sum _{k=-K}^K{\mathrm{\lambda }}_k\exp [\mathrm{j}k\omega ]}\right|}^2},$$

(56)

where $\mathbf{\lambda }=[{\lambda }_0,\cdots ,{\lambda }_K{]}^{\prime }$, the Lagrange multipliers associated with the constraints (54), are here equivalent to the AR modeling of the Gaussian process X(n).

We may note that, in this particular case, we have an analytical expression for λ, which provides the possibility to give an analytical expression for S(ω) as a function of the data {r(k), k = 0,⋯, K}:

$$S(\omega )=\frac{\mathit{\delta }{\Gamma }^{-1}\mathit{\delta }}{e{\Gamma }^{-1}e},$$

(57)

where Γ = Toeplitz(r(0),⋯, r(K)) is the correlation matrix and δ and e are two vectors defined by δ = [1, 0,⋯, 0]′ and e = [1, e^−jω, e^−j2ω,⋯, e^−jKω]′.

We may note that we first used MEP to choose a probability law for X(n). With the prior knowledge that we have second order moments, the MEP results in a Gaussian probability density function. Then, as for a stationary Gaussian process, the expression of the entropy is related to the power spectral density S(ω), and as this is related to the correlation data by a Fourier transform, an ME solution could be computed easily.

12.2. Extensions to Burg’s Method

The second approach consists of maximizing the relative entropy D [p(x): p₀(x)] or minimizing KL [p(x) : p₀(x)] where p₀(x) is an a priori law. The choice of the prior is important. Choosing a uniform p₀(x), we retrieve the previous case [77].

However, choosing a Gaussian law for p₀(x), the expression to maximize becomes:

$$D[p(\mathit{x}):p_0(\mathit{x})]=\frac{1}{4\pi }{\displaystyle {\int }_{-\pi }^{\pi }\left(\frac{S(\omega )}{S_0(\omega )}-\ln \frac{S(\omega )}{S_0(\omega )}-1\right)}\mathrm{d}\omega$$

(58)

when N ↦ ∞ and where S₀(ω) corresponds to the power spectral density of the reference process p₀(x). Now, the problem becomes: minimize D [p(x): p₀(x)] subject to the constraints (54).

12.3. Shore and Johnson Approach

Another approach is to decompose first the process X(n) on the Fourier basis {cos kωt, sin kωt}, to consider ω to be the variable of interest and S(ω), normalized properly, to be considered as its probability distribution function. Then, the problem can be reformulated as the determination of the S(ω), which maximizes the entropy:

$$-{\displaystyle {\int }_{-\pi }^{\pi }S(\omega )\ln S(\omega )\mathrm{d}\omega }$$

(59)

subject to the linear constraints (54). The solution is in the form of:

$$S(\omega )=\exp \left[{\displaystyle \sum _{k=-K}^K{\mathrm{\lambda }}_k\exp [\mathrm{j}k\omega ]}\right].$$

(60)

which can be considered as the most uniform power spectral density that satisfies those constraints.

12.4. ME in the Mean Approach

In this approach, we consider S(ω) as the expected value Z(ω) for which we have a prior law μ(z), and we are looking to assign p(z), which maximizes the relative entropy D [p(z) : μ(z)] subject to the constraints (54).

When p(z) is determined, the solution is given by:

$$S(\omega )=\mathrm{E}\{Z(\omega )\}={\displaystyle \int Z(\omega )p(z)\mathrm{d}z}.$$

(61)

The expression of S(ω) depends on μ(z). When μ(z) is Gaussian, we obtain the Rényi entropy:

$$H={\displaystyle {\int }_{-\pi }^{\pi }{S}^2(\omega )\mathrm{d}\omega }.$$

(62)

If we choose a Poisson measure for μ(z), we obtain the Shannon entropy:

$$H=-{\displaystyle {\int }_{-\pi }^{\pi }S(\omega )\ln S(\omega )\mathrm{d}\omega },$$

(63)

and if we choose a Lebesgue measure over [0, ∞], we obtain Burg’s entropy:

$$H={\displaystyle {\int }_{-\pi }^{\pi }\ln S(\omega )\mathrm{d}\omega }.$$

(64)

When this step is done, the next step becomes maximizing these entropies subject to the constraints of the correlations. The obtained solutions are very different. For more details, see [39,79–85].

13. Entropy-Based Methods for Linear Inverse Problems

13.1. Linear Inverse Problems

A general way to introduce inverse problems is the following: Infer an unknown signal f(t), image f(x, y) or any multi-variable function f(r) through an observed signal g(t), image g(x, y) or any multi-variable observable function g(s), which are related through an operator $\mathscr{H}:f\mapsto g$. This operator can be linear or nonlinear. Here, we consider only linear operators $g=Hf$:

$$g(\mathbf{s})={\displaystyle \int h(\mathbf{r},\mathbf{s})f(\mathbf{r})\mathrm{d}\mathbf{r}}$$

(65)

where h(r, s) is the response of the measurement system. Such linear operators are very common in many applications of signal and image processing. We may mention a few examples of them:

• Convolution operations g = h * f in 1D (signal):

  $$g(t)={\displaystyle \int h(t-t^{\prime })f(t^{\prime })\mathrm{d}{t}^{\prime }}$$

  (66)

  or in 2D (image):

  $$g(x,y)={\displaystyle \iint h(x-x^{\prime },y-y^{\prime })f(x^{\prime },y^{\prime })\mathrm{d}{x}^{\prime }\mathrm{d}{y}^{\prime }}$$

  (67)
• Radon transform (RT) in computed tomography (CT) in the 2D case [86]:

  $$g(r,\varphi )={\displaystyle \iint \delta (r-x\cos \varphi -y\sin \varphi )f(x,y)\mathrm{d}x\mathrm{d}y}$$

  (68)
• Fourier transform (FT) in the 2D case:

  $$g(u,v)={\displaystyle \iint \exp [-j(ux+uy)]f(x,y)\mathrm{d}x\mathrm{d}y}$$

  (69)

  which arise in magnetic resonance imaging (MRI), in synthetic aperture radar (SAR) imaging or in microwave and diffraction optical tomography (DOT) [86–90].

No matter the category of the linear transforms, when the problem is discretized, we arrive at the relation:

$$\mathit{g}=\mathit{Hf}+\mathit{ϵ}.$$

(70)

where f = [f₁,⋯, f_n]′ represents the unknowns, g = [g₁,⋯, g_m]′ the observed data, ϵ = [ϵ₁,⋯, ϵ_m]′ the errors of modeling and measurement and H the matrix of the system response.

13.2. Entropy-Based Methods

Let us consider first the simple no noise case:

$$\mathit{g}=\mathit{Hf},$$

(71)

where H is a matrix of dimensions (M × N), which is in general singular or very ill conditioned. Even if the cases M > N or M = N may appear easier, they have the same difficulties as those of the underdetermined case M < N that we consider here. In this case, evidently the problem has an infinite number of solutions, and we need to choose one.

Between the numerous methods, we may mention the minimum norm solution, which consists of choosing between all of the possible solutions:

$$\mathrm{F}=\{\mathit{f}:\mathit{Hf}=\mathit{g}\}$$

(72)

the one that has the minimum norm:

$$\mathrm{\Omega }(\mathit{f})={\Vert \mathit{f}\Vert }_2^2={\displaystyle \sum _j{f}_j^2}.$$

(73)

This optimization problem can be solved easily in this case, and we obtain:

$${\widehat{f}}_{NM}=\underset{\mathit{f}\in ℱ}{\arg \min }\{\mathrm{\Omega }(\mathit{f})={\Vert \mathit{f}\Vert }_2^2\}={\mathit{H}}^{\prime }{(\mathit{H}{\mathit{H}}^{\prime })}^{-1}\mathit{g}.$$

(74)

In fact, we may choose any other convex criterion Ω(f) and satisfy the uniqueness of the solution. For example:

$$\mathrm{\Omega }(\mathit{f})=-{\displaystyle \sum _j{f}_j\ln f_j}$$

(75)

which can be interpreted as the entropy when f_j > 0 and ∑ f_j = 1, thus considering f_j as a probability distribution f_j = P (U = u_j). The variable U can correspond (or not) to a physical quantity. Ω(f) is the entropy associated with this variable.

If we consider f_j > 0 to represent the power spectral density of a physical quantity, then the entropy becomes:

$$\mathrm{\Omega }(\mathit{f})={\displaystyle \sum _j{f}_j\ln f_j}$$

(76)

and we can use it as criterion to select a solution to the problem (71).

As we can see, any convex criterion Ω(f) can be used. Here, we mentioned four of them with different interpretations.

• L₂ or quadratic:

  $$\mathrm{\Omega }(\mathit{f})={\displaystyle \sum _j{f}_j^2}$$

  (77)

  which can be interpreted as the Rényi’s entropy with q = 2.
• L_β:

  $$\mathrm{\Omega }(\mathit{f})={\displaystyle \sum _j{|f_j|}^{\beta }}$$

  (78)

  When β < 1 the criterion is not bounded at zero. When β ≥ 1 the criterion is convex.
• Shannon entropy:

  $$\mathrm{\Omega }(\mathit{f})=-{\displaystyle \sum _j{f}_j\ln f_j}$$

  (79)

  which has a valid interpretation if 0 < f_j < 1,
• The Burg entropy:

  $$\mathrm{\Omega }(\mathit{f})={\displaystyle \sum _j\ln f_j}$$

  (80)

  which needs f_j > 0.

Unfortunately, only for the first case, there is an analytical solution for the problem, which is $\widehat{\mathit{f}}={\mathit{H}}^{\prime }(\mathit{H}{\mathit{H}}^{\prime })g$. For all of the other cases, we may need an optimization algorithm to obtain a numerical solution [91–95].

13.3. Maximum Entropy in the Mean Approach

A second approach consists of considering f_j = E {U_j} or f = E {U} [41,41,42]. Again, here, U_j or U can, but need not, correspond to some physical quantities. In any case, we now want to assign a probability law $\widehat{p}(\mathit{u})$ to it. Noting that the data g = H f = HE {U} = E {HU} can be considered as the constraints on it, we may need again a criterion to determine $\widehat{p}(\mathit{u})$. Assuming then having some prior μ(u), we may maximize the relative entropy as that criterion. The mathematical problem then becomes:

$$\text{minimize}D[p(\mathit{u}):\mu (\mathit{u})]\text{subject}\mathrm{to}{\displaystyle \int \mathit{H}\mathit{u}p(\mathit{u})\mathrm{d}\mathit{u}=\mathit{g}}$$

(81)

The solution is:

$$\widehat{p}(\mathit{u})=\frac{1}{Z(\mathbf{\lambda })}\mu (\mathit{u})\exp [-{\mathbf{\lambda }}^{\prime }\mathit{H}\mathit{u}]$$

(82)

where:

$$Z(\mathbf{\lambda })={\displaystyle \int \mu (\mathit{u})\exp [-{\mathbf{\lambda }}^{\prime }\mathit{H}\mathit{u}]\mathrm{d}\mathit{u}}.$$

(83)

When $\widehat{p}(\mathit{u})$ is obtained, we may be interested in computing:

$$\widehat{f}=\mathrm{E}\left\{U\right\}={\displaystyle \int \mathit{u}\widehat{p}(\mathit{u})\mathrm{d}\mathit{u}}$$

(84)

which is the required solution.

Interestingly, if we focus on $\widehat{f}=\mathrm{E}\left\{U\right\}$, we will see that its expression depends on the choice of the prior μ(u). When μ(u) is separable: $\mu (\mathit{u})={\displaystyle {\prod }_j{\mu }_j(u_j)}$, the expression of $\widehat{p}(\mathit{u})$ will also be separable.

To go a little more into the details, let us introduce $\mathit{s}=\mathit{H}{\mathbf{\lambda }}^{\prime }$ and define:

$$G(\mathit{s})=\ln {\displaystyle \int \mu (\mathit{u})\exp [-{\mathit{s}}^{\prime }\mathit{u}]}\mathrm{d}\mathit{u}$$

(85)

and its conjugate convex:

$$F(\mathit{f})=\underset{\mathit{s}}{\sup }\{{\mathit{f}}^{\prime }\mathit{s}-G(\mathit{s})\}.$$

(86)

It can be shown easily that $\widehat{f}=\mathrm{E}\left\{U\right\}$ can be obtained either via the dual $\widehat{\mathbf{\lambda }}$ variables:

$$\widehat{\mathit{f}}=G^{\prime }(\mathit{H}\widehat{\mathbf{\lambda }})$$

(87)

where $\widehat{\mathbf{\lambda }}$ is obtained by:

$$\widehat{\mathbf{\lambda }}=\arg \min \{D(\mathbf{\lambda })=\ln Z(\mathbf{\lambda })+{\mathbf{\lambda }}^{\prime }g\},$$

(88)

or directly:

$$\widehat{\mathit{f}}=\underset{\{\mathit{f}:\mathit{Hf}=\mathit{g}\}}{\arg \min }\{F(\mathit{f})\}.$$

(89)

D(λ) is called the dual criterion and F (f) primal. However, it is not always easy to obtain an analytical expression for G(s) and its gradient G′(s). The functions F (f) and G(s) are conjugate convex.

For the computational aspect, unfortunately, the cases where we may have analytical expressions for Z(λ) or G(s) = ln Z or F (f) are very limited. However, when there is analytical expressions for them, the computations can be done very easily. In

Table 1. Analytical solutions for different measures μ(u)

$\mu (u)\propto \exp \left[{-}_2^1{\sum }_j{u}_j^2\right]$	$\widehat{\mathit{f}}={\mathit{H}}^{\prime }\mathbf{\lambda }$	$\widehat{\mathit{f}}={\mathit{H}}^{\prime }{(\mathit{H}{\mathit{H}}^{\prime })}^{-1}\mathit{g}$
$\mu (u)\propto \exp \left[-{\sum }_j|u_j|\right]$	$\widehat{\mathit{f}}=1./({\mathit{H}}^{\prime }\mathbf{\lambda }\pm 1)$	$\mathit{H}\widehat{\mathit{f}}=\mathit{g}$
$\mu (u)\propto \exp \left[-{\sum }_j{u}_j^{\mathrm{\alpha }-1}\exp \left[-\beta u_j\right]\right],u_j>0$	$\widehat{\mathit{f}}=\alpha 1./({\mathit{H}}^{\prime }\mathbf{\lambda }+\beta 1)$	$\mathit{H}\widehat{\mathit{f}}=\mathit{g}$

, we summarizes some of those solutions:

14. Bayesian Approach for Inverse Problems

In this section, we present in a brief way the Bayesian approach for the inverse problems in signal and image processing.

14.1. Simple Bayesian Approach

The different steps to find a solution to an inverse problem using the Bayesian approach can be summarized as follows:

• Assign a prior probability law p(ϵ) to the modeling and observation errors, here ϵ. From this, find the expression of the likelihood p(g|f, θ₁). As an example, consider the Gaussian case:

  $$p(\in )=\text{N}(\in |0,v_{\in }\mathit{I})\to p(\mathit{g}|\mathit{f}=\mathcal{N}(\mathit{g}|\mathit{H}\mathit{f},v_{\in }\mathit{I}).$$

  (90)

  θ₁ in this case is the noise variance v_ϵ.
• Assign a prior probability law p(f|θ₂) to the unknown f to translate your prior knowledge on it. Again, as an example, consider the Gaussian case:

  $$p(\mathit{f})=\mathcal{N}(\mathit{f}|0,v_f\mathit{I})$$

  (91)

  θ₂ in this case is the variance v_f.
• Apply the Bayes rule to obtain the expression of the posterior law:

  $$p(\mathit{f}|\mathit{g},{\mathit{\theta }}_1,{\mathit{\theta }}_2)=\frac{p(\mathit{g}|\mathit{f},{\mathit{\theta }}_1)p(\mathit{f}|{\mathit{\theta }}_2)}{p(\mathit{g}|{\mathit{\theta }}_1,{\mathit{\theta }}_2)}\propto p(\mathit{g}|\mathit{f},{\mathit{\theta }}_1)p(\mathit{f}|{\mathit{\theta }}_2),$$

  (92)

  where the sign ∝ stands for “proportionality to”, p(g|f, θ₁) is the likelihood, p(f|θ₂) the prior model, θ = [θ₁, θ₂]′ their corresponding parameters (often called the hyper-parameters of the problem) and p(g|θ₁, θ₂) is called the evidence of the model.
• Use p(f|g, θ₁, θ₂) to infer any quantity dependent of f.

For the expressions of likelihood in (90) and the prior in (91), we obtain very easily the expression of the posterior:

$$p(\mathit{f}|\mathit{g},v_{\in },v_f)=\mathcal{N}(\mathit{f}|\widehat{\mathit{f}},\widehat{\mathit{V}})\text{with}\widehat{\mathit{V}}={(\mathit{H}{\mathit{H}}^{\prime }+\frac{v_{\in }}{v_f}\mathit{I})}^{-1}and\widehat{\mathit{f}}=\widehat{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g}$$

(93)

When the hyper-parameters θ can be fixed a priori, the problem is easy. In practice, we may use some summaries, such as:

• MAP:

  $${\widehat{\mathit{f}}}_{\mathit{MAP}}=\underset{\mathit{f}}{\arg \max }\{p(\mathit{f}|\mathit{g},\mathit{\theta })\}$$

  (94)
• EAP or posterior mean (PM):

  $${\widehat{\mathit{f}}}_{EAP}={\displaystyle \int \mathit{f}p(\mathit{f}|\mathit{g},\mathit{\theta })}\mathrm{d}\mathit{f}$$

  (95)

For the Gaussian case of (91), the MAP and EAP are the same and can be obtained by noting that:

$${\widehat{\mathit{f}}}_{\text{MAP}}=\underset{\mathit{f}}{{\displaystyle \arg \min }}\{J(\mathit{f})\}\text{with}J(\mathit{f})={\Vert \mathit{g}-\mathit{Hf}\Vert }_2^2+\mathrm{\lambda }{\Vert \mathit{f}\Vert }_2^2,\text{where}\mathrm{\lambda }=v_{\in }/v_f.$$

(96)

However, in real applications, the computation of even these simple point estimators may need efficient algorithm:

• For MAP, we need optimization algorithms, which can handle the huge dimensional criterion J(f) = − ln p(f|g, θ). Very often, we may be limited to using gradient-based algorithms.
• For EAP, we need integration algorithms, which can handle huge dimensional integrals. The most common tool here is the MCMC methods [24]. However, for real applications, very often, the computational costs are huge. Recently, different methods, called approximate Bayesian computation (ABC) [96–100] or VBA, have been proposed [74,96,98,101–107].

14.2. Full Bayesian: Hyperparameter Estimation

When the hyperparameters θ have also to be estimated, a prior p(θ) is assigned to them, and the expression of the joint posterior:

$$p(\mathit{f},\mathit{\theta }|\mathit{g})=\frac{p(\mathit{g}|\mathit{f},{\mathit{\theta }}_1)p(\mathit{f}|{\mathit{\theta }}_2)p(\mathit{\theta })}{p(\mathit{g})}$$

(97)

is obtained, which can then be used to infer them jointly. Very often, the expression of this joint posterior law is complex, and any computation may become very costly. The VBA methods try to approximate p(f, θ|g) by a simpler distribution, which can be handled more easily. Two particular and extreme cases are:

• Bloc separable, such as q(f, θ) = q₁(f) q₂(θ) or
• Completely separable, such as $q(\mathit{f},\mathit{\theta })={\displaystyle {\prod }_j{q}_{1j}}(f_j){\displaystyle {\prod }_k{q}_{2k}(}{\theta }_k)$.

Any mixed solution is also valid. For example, the one we have chosen is:

$$q(\mathit{f},\mathit{\theta })=q_1(\mathit{f}){\displaystyle \prod _k{q}_{2k}({\theta }_k)}$$

(98)

Obtaining the expressions of these approximated separable probability laws has to be done via a criterion. The natural criterion with some geometrical interpretation for the probability law manifolds is the Kullback–Leibler (KL) criterion:

$$\mathrm{KL}[q:p]={\displaystyle \int q\ln \frac{q}{p}}={\langle \ln \frac{q}{p}\rangle }_q.$$

(99)

For hierarchical prior models with hidden variables z, the problem becomes more complex, because we have to give the expression of the joint posterior law:

$$p(\mathit{f},\mathit{z},\mathit{\theta }|\mathit{g})\propto p(\mathit{g}|\mathit{f},{\mathit{\theta }}_1)p(\mathit{f}|\mathit{z},{\mathit{\theta }}_2)p(\mathit{z}|{\mathit{\theta }}_3)p(\mathit{\theta })$$

(100)

and then approximate it by separable ones:

$$q(\mathit{f},z,\mathit{\theta }|\mathit{g})=q_1(\mathit{f})q_2(z)q_3(\mathit{\theta })\mathrm{or}q(\mathit{f},\mathit{\theta })={\displaystyle \prod _j{q}_{1j}(f_j|z_{f_j})}{\displaystyle \prod _j{q}_{2_j}(z_{f_j})}{\displaystyle \prod _k{q}_{3k}({\theta }_k)}$$

(101)

and then use them for estimation. See more discussions in [9,31,38,108–110]

In the following, first the general VBA method is detailed for the inference problems with hierarchical prior models. Then, a particular class of prior model (Student t) is considered, and the details of VBA algorithms for that are given.

15. Basic Algorithms of the Variational Bayesian Approximation

To illustrate the basic ideas and tools, let us consider a vector X and its probability density function p(x), which we want to approximate by q(x) = ∏_j q_j(x_j). Using the KL criterion:

$$\begin{array}{l}\mathrm{KL}[q:p]={\displaystyle \int q(x)\ln \frac{q(x)}{p(x)}\mathrm{d}x}={\displaystyle \int q(x)\ln q(x)\mathrm{d}x-}{\displaystyle \int q(x)\ln p(x)\mathrm{d}x}\\ ={\displaystyle \sum _j{\displaystyle \int q_j(x_j)\ln q_j(x_j)}\mathrm{d}{x}_j-{\langle \ln p(x)\rangle }_q}\\ ={\displaystyle \sum _j{\displaystyle \int q_j(x_j)\ln q_j(x_j)\mathrm{d}{x}_j}-}{\displaystyle \int q_j(x_j)<\ln p(x){>}_{q_{-j}}}d{x}_j\end{array}$$

(102)

where we used the notation: 〈ln p(x)〉_q = ∫ q(x) ln p(x) dx and q_−j(x) = ∏_i≠j q_i (x_i)

From here, trying to find the solution q_i, the basic method is an alternate optimization algorithm:

$$q_j(x_j)\propto \exp [<\ln p(\mathit{x}){>}_{q_{-j}}].$$

(103)

As we can see, the expression of q_j(x_j) depends on q (x_i), i ≠ j. It is not always possible to obtain analytical expressions for q_j(x_j). It is however possible to show that, if p(x) is a member of exponential families, then q_j(x_j) are also members of exponential families. These iterations then become much simpler, because at each iteration, we need to update the parameters of the exponential families. To go a little more into the details, let us consider some particular simple cases.

15.1. Case of Two Gaussian Variables

In the case of two variables x = [x₁, x₂]′, we have:

$$\{\begin{array}{c}{q}_1\left(x_1\right)\propto \exp \left[<\ln p(\mathit{x})>q_2(x_2)\right]\\ q_2\left(x_2\right)\propto \exp \left[<\ln p(\mathit{x})>q_1(x_1)\right]\end{array}$$

(104)

As an illustrative example, consider the case where we want to approximate p(x₁, x₂) by q(x₁, x₂) = q₁(x₁) q₂(x₂) to be able to compute the expected values:

$$\{\begin{array}{c}{m}_1=E\left\{x_1\right\}={\displaystyle \iint x_1}p\left(x_1,x_2\right)\mathrm{d}{x}_1\mathrm{d}{x}_2\\ m_2=\left\{x_2\right\}={\displaystyle \iint x_{2}p\left(x_1,x_2\right)}\mathrm{d}{x}_1\mathrm{d}{x}_2\end{array}$$

(105)

which need double integrations when p(x₁, x₂) is not separable in its two variables. If we can do that separable approximation, then, we can compute:

$$\{\begin{array}{c}{\tilde{\mu }}_1=\mathrm{E}\left\{x_1\right\}={\displaystyle \int x_1{q}_1\left(x_1\right)}\mathrm{d}{x}_1\\ {\tilde{\mu }}_2=\mathrm{E}\left\{x_2\right\}={\displaystyle \int x_2{q}_2\left(x_2\right)}d{x}_2\end{array}$$

(106)

which needs only 1D integrals. Let us see if $({\tilde{\mu }}_1,{\tilde{\mu }}_2)$ will converge to (m₁, m₂). To illustrate this, let us consider the very simple case of the Gaussian:

$$p(x_1,x_2)=\mathcal{N}\left(\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]|\left[\begin{array}{c}{m}_1\\ m_2\end{array}\right],\left[\begin{array}{c}{v}_1\\ \rho \sqrt{v_1{v}_2}\end{array}\begin{array}{c}\rho \sqrt{v_1{v}_2}\\ v_2\end{array}\right]\right).$$

(107)

It is then easy to see that $q_1(x_1)=\mathcal{N}(x_1|{\tilde{\mu }}_1,{\tilde{v}}_1)$ and $q_2(x_2)=\mathcal{N}(x_2|{\tilde{\mu }}_2,{\tilde{v}}_2)$ and that:

$$\{\begin{array}{c}{q}_1^{(k+1)}(x_1)=p(x_1)|x_2={\tilde{\mu }}_2^{(k)}=\mathcal{N}\left(x_1|{\tilde{\mu }}_1^{(k)},{\tilde{v}}_1^{(k)}\right)\\ q_1^{(k+1)}(x_2)=p(x_2|x_1={\tilde{\mu }}_1^{(k)}=\mathcal{N}\left(x_1|{\tilde{\mu }}_2^{(k)},{\tilde{v}}_2^{(k)}\right)\end{array}$$

(108)

with:

$$\{\begin{array}{lll}{\tilde{\mu }}_1^{(k+1)}\hfill & =\hfill & m_1+\rho \sqrt{v_1/v_2}\left({\tilde{\mu }}_2^{(k)}-m_2\right)\hfill \\ {\tilde{v}}_1^{(k+1)}\hfill & =\hfill & \left(1-{\rho }^2\right)v_1\hfill \\ {\tilde{\mu }}_2^{(k+1)}\hfill & =\hfill & m_2+\rho \sqrt{v_2/v_1}\left({\tilde{\mu }}_1^{(k)}-m_1\right)\hfill \\ {\tilde{v}}_2^{(k+1)}\hfill & =\hfill & (1-{\rho }^2)v_2\hfill \end{array}.$$

(109)

See [111] for details and where we showed that, initializing the algorithm with ${\tilde{\mu }}_1^{(0)}=0$ and ${\tilde{\mu }}_2^{(0)}=0$, the means converges to the right values m₁ and m₂, However, we may be careful about the convergence of the variances.

15.2. Case of Exponential Families

As we could see, to be able to use such an algorithm in practical cases, we need to be able to compute < ln p(x) >_q2(x2) and < ln p(x) >_q1(x1). Only for a few cases can we can do this analytically. Different algorithms can be obtained depending on the choice of a particular family for q_j(x_j) [103,112–120].

To show this, let us consider the exponential family:

$$p(\mathit{x}|\mathit{\theta })=g(\mathit{\theta })\exp [{\mathit{\theta }}^{\prime }\mathit{u}(\mathit{x})]$$

(110)

where θ is a vector of parameter and g(θ) and u(x) are known functions.

This parametric exponential family has the following conjugacy property: For a given prior p(θ) in the family:

$$p(\mathit{\theta }|\eta ,\mathit{\nu })=h(\eta ,\mathit{\nu })g{(\mathit{\theta })}^{\eta }\exp [{\mathit{\nu }}^{\prime }\mathit{\theta }]$$

(111)

the corresponding posterior:

$$\begin{array}{c}p(\mathit{\theta }|\mathit{x})\propto p(\mathit{x}|\mathit{\theta })p(\mathit{\theta }|\eta ,\mathit{\nu })\\ \propto g{(\mathit{\theta })}^{\eta +1}\exp [\mathit{\nu }+\mathit{u}(\mathit{x}){]}^{\prime }\mathit{\theta }]\\ \propto p(\mathit{\theta }|\eta +1,\mathit{\nu }+\mathit{u}(\mathit{x}))\end{array}$$

(112)

is in the same family.

For this family, we have:

$${\langle \ln p\left(\mathit{x}|\mathit{\theta }\right)\rangle }_q=\ln g\left(\mathit{\theta }\right)+{\mathit{\theta }}^{\prime }{\langle \mathit{u}\left(\mathit{x}\right)\rangle }_q.$$

(113)

It is then easy to show that:

$$q_j(x_j)\propto g(\mathit{\theta })\exp \left[{\mathit{\theta }}^{\prime }{\langle \mathit{u}\left(\mathit{x}\right)\rangle }_{q_{-j}}\right]$$

(114)

which are in the same exponential family. This simplifies greatly the computations, thanks to the fact that, in each iteration, we only need to compute $\tilde{\mathit{u}}(\mathit{x})=\langle \mathit{u}{\left(\mathit{x}\right)}_{q_{-j}}\rangle$ and update the parameters.

Now, if we consider:

$$p(\mathit{x}|\mathit{\theta })=g(\mathit{\theta })\exp [{\mathit{\theta }}^{\prime }\mathit{u}(\mathit{x})]$$

(115)

with a prior on θ:

$$p(\mathit{\theta }|\eta ,\mathit{\nu })=h(\eta ,\mathit{\nu })g{(\mathit{\theta })}^{\eta }\exp [{\mathit{\nu }}^{\prime }\mathit{\theta }]$$

(116)

and the joint p(x, θ|η, ν) = p(x|θ) p(θ|η, ν), which is not separable in x and θ, and we want to approximate it with the separable q(x, θ) = q₁(x) q₂(θ), then we will have:

$$\{\begin{array}{c}q(\mathit{\theta })=h(\tilde{\eta },\tilde{\mathit{\nu }})g{(\mathit{\theta })}^{\tilde{\eta }}\exp [{\tilde{\mathit{\nu }}}^{\prime }\mathit{\theta }]\\ q(\mathit{x})=g(\tilde{\mathit{\theta }})\exp [{\tilde{\mathit{\theta }}}^{\prime }\mathit{u}(x)]\end{array}\text{with}\{\begin{array}{lll}\tilde{\eta }\hfill & =\hfill & \eta +1\hfill \\ \tilde{\mathit{\nu }}\hfill & =\hfill & \mathit{\nu }+\tilde{\mathit{u}}(\mathit{x})\hfill \\ \tilde{\mathit{\theta }}\hfill & =\hfill & \tilde{\mathit{\nu }}\hfill \end{array}$$

(117)

where $\tilde{\mathit{\mu }}={\langle \mathit{u}\left(\mathit{x}\right)\rangle }_{q_1\left(\mathit{x}\right)}$.

16. VBA for the Unsupervised Bayesian Approach to Inverse Problems

Before going into the details and for similarity with the notations in the next sections, we replace x by f, such that now we are trying to approximate p(f, θ) = p(f|θ) p(θ) by a separable q(f, θ) = q₁(f) q₂(θ). Interestingly, depending on the choice of the family laws for q₁ and q₂, we obtain different algorithms:

• $q_1(\mathit{f})=\delta (\mathit{f}-\tilde{\mathit{f}})$ and $q_2(\mathit{\theta })=\delta (\mathit{\theta }-\tilde{\mathit{\theta }})$. In this case, we have:

  $$\{\begin{array}{c}{q}_1(\mathit{f})\propto \exp [<\ln p(\mathit{f},\mathit{\theta }){>}_{q_2}]\propto \exp \left[\ln p(\mathit{f},\tilde{\mathit{\theta }})\right]\propto p(\mathit{f},\mathit{\theta }=\tilde{\mathit{\theta }})\propto p(\mathit{f}|\mathit{\theta }=\tilde{\mathit{\theta }})\\ q_2(\mathit{\theta })\propto \exp [<\ln p(\mathit{f},\mathit{\theta }){>}_{q_1}]\propto \exp \left[\ln p(\tilde{\mathit{f}},\mathit{\theta })\right]\propto p(\mathit{f}=\tilde{\mathit{f}},\mathit{\theta })\propto p(\mathit{\theta }|\mathit{f}=\tilde{\mathit{f}})\end{array}$$

  (118)

  and so:

  $$\{\begin{array}{c}\tilde{\mathit{f}}=\arg {\max }_{\mathit{f}}\left\{p(\mathit{f},\mathit{\theta })=\tilde{\mathit{\theta }}\right\}\\ \tilde{\mathit{\theta }}=\arg {\max }_{\mathit{\theta }}\left\{p(\mathit{f}=\tilde{\mathit{f}},\mathit{\theta })\right\}\end{array}$$

  (119)

  which can be interpreted as an alternate optimization algorithm for obtaining the JMAPestimates:

  $$(\tilde{\mathit{f}},\tilde{\mathit{\theta }})=\underset{(\mathit{f},\mathit{\theta })}{\arg \max }\{p(\mathit{f},\mathit{\theta })\}.$$

  (120)

  The main drawback here is that the uncertainties of the f are not used for the estimation of θ and the uncertainties of θ are not used for the estimation of f.
• q₁(f) is free form and $q_2(\mathit{\theta })=\delta (\mathit{\theta }-\tilde{\mathit{\theta }})$ In the same way, this time we obtain:

  $$\{\begin{array}{l}<\ln p(\mathit{f},\mathit{\theta }){>}_{q_2(\mathit{\theta })}=\ln p(\mathit{f},\tilde{\mathit{\theta }})\hfill \\ <\ln p(\mathit{f},\mathit{\theta }){>}_{q_1(\mathit{f})}=<\ln p(\mathit{f},\mathit{\theta }){>}_{q_1(\mathit{f}|\tilde{\mathit{\theta }})}=Q(\mathit{\theta },\tilde{\mathit{\theta }})\hfill \end{array}$$

  (121)

  which leads to:

  $$\{\begin{array}{l}\begin{array}{l}{q}_1(\mathit{f})\propto \exp \left[\ln p(\mathit{f},\mathit{\theta }=\tilde{\mathit{\theta }})\right]\propto p(\mathit{f},\tilde{\mathit{\theta }})\hfill \\ q_2(\mathit{\theta })\propto \exp \left[Q(\mathit{\theta },\tilde{\mathit{\theta }})\right]\to \tilde{\mathit{\theta }}=\arg {\max }_{\mathit{\theta }}\left\{Q(\mathit{\theta },\tilde{\mathit{\theta }})\right\}\hfill \end{array}\hfill \end{array}$$

  (122)

  which can be compared the Bayesian expectation maximization (BEM) algorithm. The E-step is the computation of the expectation $Q(\mathit{\theta },\tilde{\mathit{\theta }})$ in (121), and the M-step is the maximization in (122). Here, the uncertainties of the f are used for the estimation of θ, but the uncertainties of θ are not used for the estimation of f.
• $q_1(\mathit{f})=\delta (\mathit{f}-\tilde{\mathit{f}})$ and q₂(θ) is free form. In the same way, this time we obtain:

  $$\{\begin{array}{l}<\ln p(\mathit{f},\mathit{\theta }){>}_{q_1(\mathit{f})}=\ln p(\mathit{f}=\tilde{\mathit{f}},\mathit{\theta })\hfill \\ <\ln p(\mathit{f},\mathit{\theta }){>}_{q_2(\mathit{\theta })}=<\ln p(\mathit{f},\mathit{\theta }){>}_{p(\mathit{\theta }|\mathit{f}=\tilde{\mathit{f}})}=Q(\tilde{\mathit{f}},\mathit{f})\hfill \end{array}$$

  (123)

  $$\{\begin{array}{l}{q}_2(\theta )\propto \ln p(\mathit{f}=\tilde{\mathit{f}},\mathit{\theta })=p(\mathit{\theta }|\mathit{f}=\tilde{\mathit{f}})\hfill \\ q_1(\mathit{f})\propto \exp \left[Q(\tilde{\mathit{f}},\mathit{\theta })\right]\to \tilde{\mathit{\theta }}=\arg {\max }_{\mathit{\theta }}\left\{Q(\mathit{f}=\tilde{\mathit{f}},\mathit{\theta })\right\}\hfill \end{array}$$

  (124)

  which can be compared with the classical EM algorithm. Here, the uncertainties of the f are used for the estimation of θ, but the uncertainties of θ are not used for the estimation of f.
• Both q₁(f) and q₂(θ) have free form. The main difficulty here is that, at each iteration, the expression of q₁ and q₂ may change. However, if p(f, θ) is in the generalized exponential family, the expressions of q₁(f) and q₂(θ) will also be in the same family, and we have only to update the parameters at each iteration.

17. VBA for a Linear Inverse Problem with Simple Gaussian Priors

As a simple example, consider the Gaussian case where $p\left(\mathit{g}|\mathit{f},{\theta }_1\right)=\mathcal{N}\left(\mathit{g}|\mathit{Hf},\left(1/{\theta }_1\right)\mathit{I}\right),p\left(\mathit{f}|{\theta }_2\right)=\mathcal{N}\left(\mathit{f}|0,\left(1/{\theta }_2\right)\mathit{I}\right)$ and $p\left({\theta }_1\right)=\mathcal{G}({\theta }_1|{\alpha }_{10},{\beta }_{10})p({\theta }_2)=\mathcal{G}({\theta }_2|{\alpha }_{20},{\beta }_{20})$, and so, we have:

$$\begin{array}{c}\ln p(\mathit{f},{\theta }_{1,}{\theta }_2|\mathit{g})=\frac{\mathrm{M}}{2}\ln {\theta }_1-\frac{{\theta }_1}{2}\Vert \mathit{g}-\mathit{Hf}\Vert {}_2^2+\frac{\mathcal{N}}{2}\ln {\theta }_2-\frac{{\theta }_2}{2}\Vert \mathit{f}\Vert {}_2^2\\ +({\alpha }_{10}-1)\ln {\theta }_1-{\beta }_{10}{\theta }_1+({\alpha }_{20}-1)\ln {\theta }_2-{\beta }_{20}{\theta }_2.\end{array}$$

(125)

From this expression J(f, θ₁, θ₂) = ln p(f, θ₁, θ₂|g), it is easy to obtain the equations of an alternate JMAP algorithm by computing the derivatives of it with respective to its arguments and equating them to zero:

$$\begin{array}{l}\frac{\partial J}{\partial \mathit{f}}=0\to \mathit{f}={({\mathit{H}}^{\prime }\mathit{H}+\mathrm{\lambda }\mathit{I})}^{-1}{\mathit{H}}^{\prime }\mathit{g}\mathrm{with}\mathrm{\lambda }=\frac{{\theta }_2}{{\theta }_1}\hfill \\ \frac{\partial J}{\partial {\theta }_1}=0\to {\theta }_1=\frac{{\tilde{\alpha }}_1}{{\tilde{\beta }}_1}\text{with}{\tilde{\alpha }}_1=({\alpha }_{10}-1)+\frac{\mathrm{M}}{2}\text{and}{\tilde{\beta }}_1={\beta }_{10}+\frac{1}{2}\Vert \mathit{g}-\mathit{Hf}\Vert {}_2^2\hfill \\ \frac{\partial J}{\partial {\theta }_2}=0\to {\theta }_1=\frac{{\tilde{\alpha }}_2}{{\tilde{\beta }}_2}\text{with}{\tilde{\alpha }}_2=({\tilde{\alpha }}_{20}-1)+\frac{\mathrm{M}}{2}\text{and}{\tilde{\beta }}_2={\beta }_{20}+\frac{1}{2}\Vert \mathit{f}\Vert {}_2^2\hfill \end{array}$$

(126)

From the expression of the joint probability law p(f, θ₁, θ₂|g), we can also obtain the expressions of the conditionals:

$$\{\begin{array}{l}\begin{array}{l}p(\mathit{f}|\mathit{g},{\theta }_1,{\theta }_2)=\mathcal{N}(\mathit{f}|\tilde{\mathit{f}},\tilde{\mathit{V}})\\ \text{with}\tilde{\mathit{V}}={({\mathit{H}}^{\prime }\mathit{H}+\mathrm{\lambda }\mathit{I})}^{-1},\tilde{\mathit{f}}=\tilde{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g},\mathrm{\lambda }=\frac{{\theta }_2}{{\theta }_1}\end{array}\hfill \\ \begin{array}{l}p({\theta }_1|\mathit{g},\mathit{f},{\theta }_2)=\mathcal{G}({\theta }_1|{\tilde{\alpha }}_1,{\tilde{\beta }}_1)\\ \text{with}{\tilde{\alpha }}_1=({\alpha }_{10}-1)+\frac{\mathrm{M}}{2},{\tilde{\beta }}_1={\beta }_{10}+\frac{1}{2}\Vert \mathit{g}-\mathit{Hf}\Vert {}_2^2\end{array}\hfill \\ \begin{array}{l}p({\theta }_2|\mathit{g},\mathit{f},{\theta }_1)=\mathcal{G}({\theta }_2|{\tilde{\alpha }}_2,{\tilde{\beta }}_2)\\ \text{with}{\tilde{\alpha }}_2=({\alpha }_{20}-1)+\frac{M}{2},{\tilde{\beta }}_2={\beta }_{20}+\frac{1}{2}\Vert \mathit{f}\Vert {}_2^2\end{array}\hfill \end{array}$$

(127)

However, obtaining analytical expressions of the marginals p(f|g), p(θ₁|g) and p(θ₂|g) is not easy. We can then obtain approximate expressions q₁(f|g), q₂(θ₁|g) and q₃(θ₂|g) using the VBA method. For this case, thanks to the conjugacy property, we have:

$$\{\begin{array}{l}\begin{array}{l}q(f)=\mathcal{N}(\mathit{f}|\tilde{\mathit{f}},\tilde{\mathit{V}})\\ \text{with}\tilde{\mathit{V}}={({\mathit{H}}^{\prime }\mathit{H}+{\lambda }^{\prime }\mathit{I})}^{-1},\tilde{\mathit{f}}=\tilde{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g},\tilde{\lambda }=\frac{<{\theta }_2>}{<{\theta }_1>};\end{array}\hfill \\ \begin{array}{l}q({\theta }_1=\mathcal{G}({\theta }_1|{\tilde{\alpha }}_1,{\tilde{\beta }}_1)\\ \text{with}{\tilde{\alpha }}_1=({\alpha }_{10}-1)+\frac{\mathrm{M}}{2},{\tilde{\beta }}_1={\beta }_{10}+\frac{1}{2}<\Vert \mathit{g}-\mathit{Hg}\Vert {}_2^2>\end{array}\hfill \\ \begin{array}{l}p({\theta }_2|\mathit{g},\mathit{f})=\mathcal{G}({\theta }_2|{\tilde{\alpha }}_2,{\tilde{\beta }}_2)\\ \text{with}{\tilde{\alpha }}_2=({\alpha }_{20}-1)+\frac{\mathcal{N}}{2},{\tilde{\beta }}_2={\beta }_{20}+\frac{1}{2}<\Vert \mathit{f}\Vert {}_2^2>\end{array}\hfill \end{array}$$

(128)

We can then compare the three algorithms in

Table 2. Comparision of three algorithms: JMAP, BEM and VBA

JMPA	BEM	VBA
$q(\mathit{f})=\delta (\mathit{f}-\tilde{\mathit{f}})$	$q(\mathit{f})=N(\mathit{f}|\tilde{\mathit{f}},\tilde{\mathit{V}})$	$q(\mathit{f})=N(\mathit{f}|\tilde{\mathit{f}},\tilde{\mathit{V}})$
$\tilde{\mathit{V}}={({\mathit{H}}^{\prime }\mathit{H}+\tilde{\lambda }\mathit{I})}^{-1}$	$\tilde{\mathit{V}}={({\mathit{H}}^{\prime }\mathit{H}+\tilde{\lambda }\mathit{I})}^{-1}$	$\tilde{\mathit{V}}={({\mathit{H}}^{\prime }\mathit{H}+\tilde{\lambda }\mathit{I})}^{-1}$
$\tilde{\mathit{f}}=\tilde{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g}$	$\tilde{\mathit{f}}=\tilde{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g}$	$\tilde{\mathit{f}}=\tilde{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g}$
$q({\theta }_1)=\delta ({\theta }_1-{\tilde{\theta }}_1)$	$q({\theta }_1)=\delta ({\theta }_1-{\tilde{\theta }}_1)$	$q({\theta }_1)=\mathcal{G}({\theta }_1|{\tilde{\alpha }}_1,{\tilde{\beta }}_1)$
${\tilde{\alpha }}_1=({\alpha }_{10}-1)+\frac{\mathrm{M}}{2}$	${\tilde{\alpha }}_1=({\alpha }_{10}-1)+\frac{\mathrm{M}}{2}$	${\tilde{\alpha }}_1=({\alpha }_{10}-1)+\frac{\mathrm{M}}{2}$
${\tilde{\beta }}_1={\beta }_{10}+\frac{1}{2}\Vert \mathit{g}-\mathit{Hf}\Vert {}_2^2$	${\tilde{\beta }}_1={\beta }_{10}+\frac{1}{2}<\Vert \mathit{g}-\mathit{Hf}\Vert {}_2^2>$	${\tilde{\beta }}_1={\beta }_{10}+\frac{1}{2}<\Vert \mathit{g}-\mathit{Hf}\Vert {}_2^2>$
${\tilde{\theta }}_1=\frac{{\tilde{\alpha }}_1}{{\beta }_1}$	${\tilde{\theta }}_1=\frac{{\tilde{\alpha }}_1}{{\tilde{\beta }}_1}$	${\tilde{\theta }}_1=\frac{{\tilde{\alpha }}_1}{{\tilde{\beta }}_1}$
$q({\theta }_2)=\delta ({\theta }_2-{\tilde{\theta }}_2)$	$q({\theta }_2)=\delta ({\theta }_2-{\tilde{\theta }}_2)$	$q({\theta }_2)=\mathcal{G}({\theta }_2|{\tilde{\alpha }}_2,{\tilde{\beta }}_2)$
${\tilde{\alpha }}_2=({\alpha }_{20}-1)+\frac{\mathrm{M}}{2}$	${\tilde{\alpha }}_2=({\alpha }_{20}-1)+\frac{\mathrm{M}}{2}$	${\tilde{\alpha }}_2=({\alpha }_{20}-1)+\frac{\mathcal{N}}{2}$
${\tilde{\beta }}_2={\beta }_{10}+\frac{1}{2}\Vert \mathit{f}\Vert {}_2^2$	${\tilde{\beta }}_2={\beta }_{20}+\frac{1}{2}<\Vert \mathit{f}\Vert {}_2^2>$	${\tilde{\beta }}_2={\beta }_{20}+\frac{1}{2}<\Vert \mathit{f}\Vert {}_2^2>$
${\tilde{\theta }}_2=\frac{{\tilde{\alpha }}_2}{{\tilde{\beta }}_2}$	${\tilde{\theta }}_2=\frac{{\tilde{\alpha }}_2}{{\tilde{\beta }}_2}$	${\tilde{\theta }}_2=\frac{{\tilde{\alpha }}_2}{{\tilde{\beta }}_2}$
$\tilde{\lambda }=\frac{{\tilde{\theta }}_2}{{\tilde{\theta }}_1}$	$\tilde{\lambda }=\frac{{\tilde{\theta }}_2}{{\tilde{\theta }}_1}$	$\tilde{\lambda }=\frac{{\tilde{\theta }}_2}{{\tilde{\theta }}_1}$

:

It is important to remark that, in JMAP, the computation of f can be done via the optimization of the criterion J(f, θ₁, θ₂) = ln p(f, θ₁, θ₂|g), which does not need explicitly the matrix inversion of $\tilde{\mathit{V}}={({\mathit{H}}^{\prime }\mathit{H}+{\lambda }^{\prime }\mathit{I})}^{-1}$ However, in BEM and VBA, we need to compute it due to the following requirements:

$$\begin{array}{l}<f{>}_q=\tilde{f},\\ <\Vert \mathit{f}\Vert {}^2{>}_q=\mathrm{tr}\left(<\tilde{\mathit{f}}{\tilde{\mathit{f}}}^{\prime }{>}_q\right)=\mathrm{tr}\left(\tilde{\mathit{f}}{\tilde{\mathit{f}}}^{\prime }+\tilde{\mathit{V}}\right)+\Vert \tilde{\mathit{f}}\Vert {}^2+\mathrm{tr}\left(\tilde{\mathit{V}}\right),\\ <f_j^2{>}_q={\left[\tilde{\mathit{V}}\right]}_{jj}+{\tilde{f}}_j^2,\\ <\Vert \mathit{g}-\mathit{Hf}\Vert {}^2{>}_q=\left[{\mathit{g}}^{\prime }\mathit{g}-2{\langle \mathit{f}\rangle }^{\prime }{}_q{\mathit{H}}^{\prime }\mathit{g}+{\mathit{H}}^{\prime }{\langle \tilde{\mathit{f}}\mathit{f}\rangle }_q\mathit{H}\right]\\ =\left[{\mathit{g}}^{\prime }\mathit{g}-2{\tilde{\mathit{f}}}^{\prime }\tilde{\mathit{H}}\mathit{g}+{\mathit{H}}^{\prime }\left(\tilde{\mathit{V}}+\tilde{\mathit{f}}{\tilde{\mathit{f}}}^{\prime }\right)\mathit{H}\right]\\ =\Vert \mathit{g}-\mathit{H}\tilde{\mathit{f}}\Vert {}^2+\mathrm{tr}\left({\mathit{H}}^{\prime }\tilde{\mathit{V}}\mathit{H}\right)\end{array}$$

(129)

For some extensions and more details, see [111].

18. Bayesian Variational Approximation with Hierarchical Prior Models

For a linear inverse problem:

$$\mathrm{M}:g=\mathit{Hf}+\in$$

(130)

with an assigned likelihood p(g|f, θ₁), when a hierarchical prior model p(f|z, θ₂) p(z|θ₃) is used and when the estimation of the hyper-parameters θ = [θ₁, θ₂, θ₃]′ has to be considered, the joint posterior law of all the unknowns becomes:

$$p(\mathit{f},\mathit{z},\mathit{\theta }|\mathit{g})=\frac{p(\mathit{f},\mathit{z},\mathit{\theta }|\mathit{g})}{p(\mathit{g})}=\frac{p(\mathit{g},{\mathit{\theta }}_1)p(\mathit{f}|\mathit{z},{\mathit{\theta }}_2)p(\mathit{z}|{\mathit{\theta }}_3)p(\mathit{\theta })}{p(\mathit{g})}$$

(131)

The main idea behind the VBA is to approximate this joint posterior by a separable one, for example: q(f, z, θ|g) = q₁(f) q₂(z) q₃(θ) and where the expressions of q(f, z, θ|g) are obtained by minimizing the Kullback–Leibler divergence (99), as explained in previous section. This approach can also be used for model selection based on the evidence of the model ln p(g) [121] where:

$$p(\mathit{g})={\displaystyle \iiint p(\mathit{f},\mathit{z},\mathit{\theta },\mathit{g})}\mathrm{d}f\mathrm{d}z\mathrm{d}\theta .$$

(132)

Interestingly, it is easy to show that:

$$\ln p(\mathit{g})=\mathrm{KL}\left[q:p\right]+ℱ(q)$$

(133)

where $ℱ\left(q\right)$ is the free energy associated with q defined as:

$$ℱ(q)={\langle \ln \frac{p(\mathit{f},\mathit{z},\mathit{\theta },\mathit{g})}{q(\mathit{f},\mathit{z},\mathit{\theta })}\rangle }_q$$

(134)

Therefore, for a given model $ℳ$, minimizing KL [q : p] is equivalent to maximizing $ℱ\left(q\right)$ and when optimized, $ℱ\left(q*\right)$ gives a lower bound for ln p(g). Indeed, the name variational approximation is due to the fact that $\ln p(g)\ge ℱ(q)$, and so, $ℱ\left(q\right)$ is a lower bound to the evidence ln p(g).

Without any other constraint than the normalization of q, an alternate optimization of $ℱ\left(q\right)$ with respect to q₁, q₂ and q₃ results in:

$$\{\begin{array}{l}{q}_1(\mathit{f})\propto \exp \left[-{\langle \ln p\left(\mathit{f},\mathit{z},\mathit{\theta },\mathit{g}\right)\rangle }_{q{(\mathit{z})}_q(\mathit{\theta })}\right],\hfill \\ q_2(\mathit{z})\propto \exp \left[-{\langle \ln p\left(\mathit{f},\mathit{z},\mathit{\theta },\mathit{g}\right)\rangle }_{q{(\mathit{f})}_q(\mathit{\theta })}\right],\hfill \\ q_2(\mathit{\theta })\propto \exp \left[-{\langle \ln p\left(\mathit{f},\mathit{z},\mathit{\theta },\mathit{g}\right)\rangle }_{q{(\mathit{f})}_q(\mathit{z})}\right],\hfill \end{array}$$

(135)

Note that these relations represent an implicit solution for q₁(f), q₂(z) and q₃(θ), which need, at each iteration, the expression of the expectations in the right hand of exponentials. If p(g|f, z, θ₁) is a member of an exponential family and if all of the priors p(f|z, θ₂), p(z|θ₃), p(θ₁), p(θ₂) and p(θ₃) are conjugate priors, then it is easy to see that these expressions lead to standard distributions for which the required expectations are easily evaluated. In that case, we may note:

$$q(\mathit{f},\mathit{z},\mathit{\theta })=q_1(\mathit{f}|\tilde{\mathit{z}},\tilde{\mathit{\theta }})q_2(\mathit{z}|\tilde{\mathit{f}},\tilde{\mathit{\theta }})q_3(\mathit{\theta }|\tilde{\mathit{f}},\tilde{\mathit{z}})$$

(136)

where the tilded quantities $\tilde{\mathit{z}}$, $\tilde{\mathit{f}}$ and $\tilde{\mathit{\theta }}$are, respectively, functions of ( $\tilde{\mathit{f}}$, $\tilde{\mathit{\theta }}$), ( $\tilde{\mathit{f}}$, $\tilde{\mathit{z}}$) and ( $\tilde{\mathit{f}}$, $\tilde{\mathit{z}}$). This means that the expression of $q_1(\mathit{f}|\tilde{\mathit{z}},\tilde{\mathit{\theta }})$ depends on ( $\tilde{\mathit{f}}$, $\tilde{\mathit{z}}$), the expression of $q_2(\mathit{z}|\tilde{\mathit{f}},\tilde{\mathit{\theta }})$ depends on ( $\tilde{\mathit{z}}$, $\tilde{\mathit{\theta }}$) and the expression of $q_3(\mathit{\theta }|\tilde{\mathit{f}},\tilde{\mathit{z}})$ depends on ( $\tilde{\mathit{f}}$, $\tilde{\mathit{z}}$). With this notation, the alternate optimization results in alternate updating of the parameters ( $\tilde{\mathit{z}}$, $\tilde{\mathit{\theta }}$) of q₁, the parameters ( $\tilde{\mathit{f}}$, $\tilde{\mathit{\theta }}$) of q₂ and the parameters ( $\tilde{\mathit{f}}$, $\tilde{\mathit{z}}$) of q₃. Finally, we may note that, to monitor the convergence of the algorithm, we may evaluate the free energy:

$$\begin{array}{l}ℱ(q)={\langle \ln p(\mathit{f},\mathit{z},\mathit{\theta },\mathit{g})\rangle }_q-{\langle \ln q(\mathit{f},\mathit{z},\mathit{\theta })\rangle }_q\\ ={\langle \ln p(\mathit{g}|\mathit{f},\mathit{z},\mathit{\theta })\rangle }_q+{\langle \ln p(\mathit{f}|\mathit{z},\mathit{\theta })\rangle }_q+{\langle \ln p(\mathit{z}|\mathit{\theta })\rangle }_q+{\langle \ln p(\mathit{\theta })\rangle }_q-{\langle \ln q(\mathit{f})\rangle }_q-{\langle \ln q(\mathit{z})\rangle }_q-{\langle \ln q(\mathit{\theta })\rangle }_q.\end{array}$$

(137)

Other decompositions for q(f, z, θ) are also possible. For example: q(f, z, θ) = q₁(f|z) q₂(z) q₃(θ) or even: $q(\mathit{f},\mathit{z},\mathit{\theta })={\displaystyle {\prod }_j{q}_{1j}(f_j)}{\displaystyle {\prod }_j{q}_{2j}(z_{fj}){\displaystyle {\prod }_l{q}_{3l}({\theta }_l)}}$. Here, we consider the first case and give some more details on it.

19. Bayesian Variational Approximation with Student t Priors

The Student t model is:

$$p(\mathit{f}|\nu )={\displaystyle \prod _j\mathcal{S}t}(f_j|\nu )\text{with}\mathcal{S}t(f_j|\nu )=\frac{1}{\sqrt{\pi \nu }}\frac{\Gamma ((\nu +1)/2)}{\Gamma (\nu /2)}{\left(1+f_j^2/\nu \right)}^{-(\nu +1)/2}$$

(138)

The Cauchy model is obtained when ν = 1. Knowing that:

$$\mathcal{S}t(f_j|\nu )={\displaystyle {\int }_0^{\infty }\mathcal{N}}(f_j|0,1/z_{f_j})\mathcal{G}(z_{f_j}|\nu /2,\nu /2)\mathrm{d}{z}_{f_j}$$

(139)

we can write this model via the positive hidden variables z_fj:

$$\{\begin{array}{l}p(f_j|z_{j_f})=\mathcal{N}(f_j|0,1/z_{f_j})\propto \exp \left[-\frac{1}{2}{z}_{f_j}{f}_j^2\right]\hfill \\ p(z_{f_j}|\alpha ,\beta )=\mathcal{G}(z_{j_j}|\alpha ,\beta )\propto z_{f_j}^{(\alpha -1)}\exp \left[-\beta z_{f_j}\right]\hfill \end{array}$$

(140)

Now, let us consider the forward model g = Hf + ϵ and assign a Gaussian law with unknown variance ${\upsilon }_{{\in }_i}$ to the noise ϵ_i, which results in $p(\in )=\mathcal{N}(g|0,{\mathrm{V}}_{\in })$ with ${\mathrm{V}}_{\in }=\text{diag}[{\upsilon }_e]$ with ${\upsilon }_{\in }=[{\upsilon }_{{\in }_1},\cdots ,{\upsilon }_{{\in }_M}]$, and so:

$$p(\mathit{g}|\mathit{f},{\mathbf{\upsilon }}_{\in })=\mathcal{N}(\mathit{g}|\mathit{Hf},{\mathit{V}}_{\in })\propto \exp \left[-\frac{1}{2}(\mathit{g}-\mathit{Hf}){\mathit{V}}_{\in }^{-1}(\mathit{g}-\mathit{Hf})\right].$$

(141)

Let us also note by $z_{{\in }_i}=1/{\upsilon }_{e_i}$, $z_{\in }=[z_{{\in }_1},\cdots ,z_{{\in }_M}]$and ${\mathrm{Z}}_{\in }=\text{diag}[z_{\in }]={\mathrm{V}}_{\in }^{-1}$ and assign a prior on it $p({\upsilon }_{{\in }_i}|{\alpha }_{{\in }_0},{\beta }_{\in }{}_0)=ℐ\mathcal{G}({\upsilon }_{{\in }_i}|{\alpha }_{{\in }_0},{\beta }_{{\in }_0})$ or equivalently:

$$p(z_{{\in }_i}|{\alpha }_{{\in }_0},{\beta }_{{\in }_0})=\mathcal{G}(z_{{\in }_i}|{\alpha }_{{\in }_0},{\beta }_{{\in }_0})andp(z_{\in }|{\alpha }_{{\in }_0},{\beta }_{{\in }_0})={\displaystyle \prod _i\mathcal{G}(z_{{\in }_i}|{\alpha }_{{\in }_0},{\beta }_{{\in }_0})}.$$

(142)

Let us also note ${\upsilon }_f=[{\upsilon }_{f{}_1},\cdots ,{\upsilon }_{f_N}]$, ${\mathrm{V}}_f=\text{diag}[{\upsilon }_f]$, $z_{f_i}=1/{\upsilon }_{f_i}$, ${\mathrm{Z}}_f=\text{diag}[z_f]={\mathrm{V}}_f^{-1}$ and note:

$$p(\mathit{f}|{\upsilon }_f)={\displaystyle \prod _j\mathcal{N}}(f_j|0,{\upsilon }_{f_j})={\displaystyle \prod _j\mathcal{N}(f_j|0,{\upsilon }_{f_j})}=\mathcal{N}(\mathit{f}|0,{\mathit{V}}_f)$$

(143)

and finally,

$$p({\upsilon }_f|{\alpha }_{f_0},{\beta }_{f_0})={\displaystyle \prod _j\mathcal{G}({\upsilon }_{f_i}|{\alpha }_{f_0},{\beta }_{f_0})}.$$

(144)

Then, we obtain the following expressions for the VBA:

$$\{\begin{array}{l}{q}_1(\mathit{f}|\tilde{\mathit{\mu }},{\tilde{\mathit{V}}}_f)=\mathcal{N}(\mathit{f}|\tilde{\mathit{\mu }},\tilde{\mathit{V}})\text{with}\tilde{\mathit{\mu }}=\tilde{\mathit{V}}{\mathit{H}}^{\prime }\mathit{g},\tilde{\mathit{V}}={({\mathit{H}}^{\prime }{\tilde{\mathit{V}}}_{\in }^{-1}\mathit{H}+{\tilde{\mathit{Z}}}_f)}^{-1};\hfill \\ q_{2j}(z_{f_j})=\mathcal{G}(z_{f_j}|{\tilde{\alpha }}_j,{\tilde{\beta }}_j)\text{with}{\tilde{\alpha }}_j={\alpha }_{00}+1/2,{\tilde{\beta }}_j={\beta }_{00}+<f_j^2>/2;\hfill \\ q_3(z_{{\in }_i})=\mathcal{G}(z{}_{{\in }_i}|{\tilde{\alpha }}_{{\in }_i},{\tilde{\beta }}_{{\in }_i})\text{with}{\tilde{\alpha }}_{{\in }_i}={\alpha }_{{\in }_0}+(\mathcal{N}+1)/2,{\tilde{\beta }}_{{\in }_i}={\beta }_{{\in }_0}+\frac{1}{2}<g_i-{[\mathit{Hf}]}_i{|}^2>;\hfill \end{array}$$

(145)

where:

$$\begin{array}{l}<{|g_i-{[\mathit{Hf}]}_i|}^2>={|g_i-\mathit{H}<\mathit{f}>{]}_i|}^2+{[{\mathit{H}}^{\prime }\tilde{\mathit{V}}\mathit{H}]}_{\mathit{ii}},\\ <\mathit{f}>=\tilde{\mathit{\mu }},<\mathit{f}{\mathit{f}}^{\prime }>=\tilde{\mathit{V}}+\tilde{\mathit{\mu }}{\tilde{\mathit{\mu }}}^{\prime },\\ <f_j^2>={[\tilde{\mathit{V}}]}_{\mathit{jj}}+{\tilde{\mu }}_j^2\end{array}$$

We have implemented these algorithms for many linear inverse problems [102], such as periodic components estimation in time series [122] or computed tomography [123], blind deconvolution [124], blind image separation [125,126] and blind image restoration [89].

20. Conclusions

The main conclusions of this paper can be summarized as follows:

• A probability law is a tool for representing our state of knowledge about a quantity.
• The Bayes or Laplace rule is an inference tool for updating our state of knowledge about an inaccessible quantity when another accessible, related quantity is observed.
• Entropy is a measure of information content in a variable with a given probability law.
• The maximum entropy principle can be used to assign a probability law to a quantity when the available information about it is in the form of a limited number of constraints on that probability law.
• Relative entropy and Kullback–Leibler divergence are tools for updating probability laws in the same context.
• When a parametric probability law is assigned to a quantity and we want to measure the amount of information gain about the parameters when some direct observations of that quantity is available, we can use the Fisher information. The structure of the Fisher information geometry in the space of parameters is derived from the relative entropy by a second order Taylor series approximation.
• All of these rules and tools are used currently in different ways in data and signal processing. In this paper, a few examples of the ways these tools are used in data and signal processing problems are presented. One main conclusion is that each of these tools has to be used in appropriate contexts. The example in spectral estimation shows that it is very important to define the problems very clearly at the beginning and to use appropriate tools and interpret the results appropriately.
• The Laplacian or Bayesian inference is the appropriate tool for proposing satisfactory solutions to inverse problems. Indeed, the expression of the posterior probability law represents the combination of the state of the knowledge in the forward model and the data and the state of the knowledge before using the data.
• The Bayesian approach can also easily be used to propose unsupervised methods for the practical application of these methods.
• One of the main limitation of those sophisticated methods is the computational cost. For this, we proposed to use VBA as an alternative to MCMC methods to propose realistic algorithms in huge dimensional inverse problems where we want to estimate an unknown signal (1D), image (2D), volume (3D) or even more (3D + time or 3D + wavelength), etc.