L1-Minimization Algorithm for Bayesian Online Compressed Sensing

Abstract

In this work, we propose a Bayesian online reconstruction algorithm for sparse signals based on Compressed Sensing and inspired by L1-regularization schemes. A previous work has introduced a mean-field approximation for the Bayesian online algorithm and has shown that it is possible to saturate the offline performance in the presence of Gaussian measurement noise when the signal generating distribution is known. Here, we build on these results and show that reconstruction is possible even if prior knowledge about the generation of the signal is limited, by introduction of a Laplace prior and of an extra Kullback–Leibler divergence minimization step for hyper-parameter learning.

1. Introduction

It has become commonplace to talk about the recent “information explosion” or “data deluge”. These expressions refer to a much faster growth in data production compared to all available data storage and to the even more evident divergence between the volume of data produced and the general data processing capacity. This state of things begs for more efficient ways of storing and analyzing data. Natural and man-made signals tend to be compressible, which means their innate redundancies allow them to be expressed as a small number of combinations of its components—they are sparse in some basis. In the last few decades a lot of effort has been directed to the development of compression techniques that allow the sampled data to be rewritten to a reduced number of bits. However, these techniques mean superfluous costs on the gathering and processing of the data. When talking about medical imaging that includes radiation, unnecessary data gathering almost certainly signifies collateral and unwanted health effects in the long run for the patients.

Compressed sensing (CS) is a framework for signal processing that presents itself as a successful resource that has produced many techniques for efficient information extraction [1]. CS utilizes the fact that real-world signals can typically be represented by a small combination of elemental components [2,3] to eliminate the data redundancy directly in its acquisition (in contrast with what occurs in a digital camera, for example, that captures millions of point measurements each time a picture is taken only to discard a large portion of this data immediately after its acquisition through a image compression algorithm). In a standard scenario, CS utilizes the sparsity property to enable the recovery of various signals from much fewer samples of linear measurements than required by the Nyquist–Shannon theorem [4,5]. The Nyquist rate [6,7] is a concept present in virtually all signal acquisition protocols used in consumer electronics and medical imaging devices, among others [2]. Unfortunately, it implies the necessity of a high sampling frequency, which means an especially high demand if we consider the now pervasive high definition registers. Between 2004 and 2006, Candès, Tao and Donoho [4,8,9] showed that sparse signals could be exactly reconstructed with below-the-Nyquist-rate sampling. Moreover (and perhaps what made it so intriguing), the reconstruction could be done in non-adaptive fashion through L1-minimization. These seminal works gave rise to CS [5]. It should be noted that “sampling” here has a twist, though, in CS, a sample of the signal is obtained by calculating its inner product against different test functions. All in all, the popularity of CS in the last few years made it pervasive through various disciplines, from Computer Science to Statistical Physics.

In this work, we consider the scenario of a signal generated by an unknown distribution and its recovery by means of a Bayesian online CS scheme. There is a vast literature on Bayesian schemes for CS (e.g., [10,11,12]) and their reconstruction have been thoroughly examined, not least by the Statistical Physics community [13,14,15]. Online reconstruction schemes can be invaluable in situations where real-time inference is needed (e.g., magnetic resonance imaging), when the data set is too large to be entirely stored on a hard disk (e.g., modern simulation science) or in the presence of time-variant signals (e.g., data streaming). Algorithms for Bayesian online learning are well-known in the neural networks community [16,17,18] and, in many fields of engineering such as optimal control theory, where they are known as Kalman filters [19,20,21]. In addition, an interesting compromise between online and offline learning is the so-called mini-batch learning [22], which has recently also found its way to the CS scenario [23]. In direct connection with this letter, previous work [24] has already established the possibility of online CS reconstruction, but it suffered from an important limitation, namely the necessity of an exact knowledge about the generating distribution of the signal.

This paper is organized as follows: in Section 2, the basic CS problem is defined, together with two reconstruction methods—the L1-minimization problem called LASSO (Least Absolute Shrinkage and Selection Operator) and the Bayesian scheme. Section 3 introduces the Bayesian online CS framework as presented in [24]. In Section 4, the main contribution of this work, an extension for the online framework in which imperfect knowledge about the signal generating distribution is allowed, is described. Section 5 presents the main results and Section 6 summarizes and discusses the main findings of this work.

2. Problem Setup

Let ${\mathit{x}}^0=\left(x_i^0\right)\in R^N$ be a real N-dimensional signal where each component is generated by a sparsity-inducing distribution $\varphi \left(x\right)=(1-\rho )\delta \left(x\right)+\rho g\left(x\right)$. It is assumed that $0<\rho <1$ and that $g\left(x\right)$ does not have a finite mass at the origin. Sequential linear projections of the signal with known independent random measurement vectors ${\mathit{A}}^{\mu }=\left(A_i^{\mu }\right)\in R^N$, with $A_i^{\mu }\sim \mathcal{N}(0,N^{-1})$ are obtained at instants $\mu =1,2,\dots$. Let these random projections $u^{\mu }\equiv {\mathit{A}}^{\mu }·{\mathit{x}}^0$ pass through a (possibly noisy) channel before their result becomes available—this way, the value of each measurement, $y^{\mu }$, will be distributed according to $P\left(y^{\mu }\right|u^{\mu })$ (Figure 1).

A common goal in Compressed Sensing is to accurately recover the signal ${\mathit{x}}^0$ based on the knowledge of $D^t=\{({\mathit{A}}^1,y^1),({\mathit{A}}^2,y^2),\dots ,({\mathit{A}}^t,y^t)\}$ and of the functional form of the channel $P\left(y\right|\mathit{A}·\mathit{x})$. Since the beginnings of CS, the staple technique in the field [5] has been the LASSO, an optimization problem with an L1-regularization term [25],

$${\widehat{\mathit{x}}}^t=\underset{\mathit{x}}{arg\; min}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2+\lambda {\parallel \mathit{x}\parallel }_1,$$

(1)

where $\mathit{y}={\left(y^{\mu }\right)}_{\mu =1}^t$, $\mathit{A}$ is a matrix with all vectors ${\mathit{A}}^{\mu }$ as rows and the Lp-norms are defined as ${\parallel \mathit{z}\parallel }_p=({\sum }_i|z_i{{|}^p)}^{1/p}$. The seminal works [4,8,9] were pivotal in proving that perfect signal reconstruction in absence of noise is possible by means of (1) even in subsampling scenarios (i.e., for $\rho <\alpha <1$, with $\alpha \equiv t/N$) . The Statistical Physics community is no stranger for these results, with typical behaviour analysis for large systems, where $N\to \infty$ [26,27] being accepted as guarantees for the validity of the method.

More recently, Bayesian techniques for signal reconstruction have also been proposed (e.g., [10,28]). They consist in estimating ${\mathit{x}}^0$ from the posterior distribution

$$P\left(\mathit{x}\right|D^t)=\frac{P\left(D^t\right|\mathit{x}){\prod }_{i=1}^N\varphi \left(x_i\right)}{\int d{\mathit{x}}^{\prime }P\left(D^t\right|{\mathit{x}}^{\prime }){\prod }_{i=1}^N\varphi \left(x_i^{\prime }\right)}.$$

(2)

It has been shown [11] that the Bayesian reconstruction typically performs better than the L1-minimization scheme for fewer measurements. In fact, the minimum mean squared error (mmse) estimator $\widehat{\mathit{x}}\left(D^t\right)=\int d\mathit{x}\mathit{x}P\left(\mathit{x}\right|D^t)$ is guaranteed [14] to minimize the mean square error $\mathsf{mse}$ defined for any possible estimate $\mathit{x}$ as:

$$\mathsf{mse}\equiv N^{-1}{〈\parallel \mathit{x}-{\mathit{x}}^0{\parallel }^2〉}_{D^t,{\mathit{x}}^0}.$$

(3)

3. Bayesian Online Compressed Sensing

In this work, reconstruction of the signal is done in an online manner. That is, measurements are used one at a time and discarded thereafter (in opposition to the offline/batch scenarios presented above where the whole dataset $D^t$ is available during the entire reconstruction). In addition, we would like to achieve this objective with limited knowledge about the generating distribution $\varphi \left(x\right)$.

Bayes’ Theorem (2) can be easily transformed into an online update recursion [16]. Since all measurements are independent by design, the likelihood can be factored as

$$P\left(D^t\right|\mathit{x})=\prod _{\mu =1}^{t}P\left(y^{\mu }\right|A^{\mu }·\mathit{x})=P\left(y^t\right|A^t·\mathit{x})\prod _{\mu =1}^{t-1}P\left(y^{\mu }\right|A^{\mu }·\mathit{x}).$$

(4)

This way, the posterior distribution $P\left(\mathit{x}\right|D^t)$ for the signal $\mathit{x}$ after t measurements can be written

$$P\left(\mathit{x}\right|D^t)\propto P\left(y^t\right|A^t·\mathit{x})P\left(\mathit{x}\right|D^{t-1}).$$

(5)

In general, distribution $P\left(\mathit{x}\right|D^t)$ can be complicated, so the calculation of $\widehat{\mathit{x}}\left(D^t\right)$ is potentially difficult. Consider for now the Compressed Sensing scenario where the prior distribution is exactly the same as the known signal generating distribution $\varphi \left(x\right)$. The $\delta \left(x\right)$ factor here adds a singularity to the prior $\varphi \left(x\right)$, so that the posterior should not approximate a Gaussian distribution even for $t\to \infty$, which was a crucial hypothesis in previous works [16] and would greatly simplify asymptotic calculations. In absence of such a simplification, a previous work [24] introduced the following mean-field approximation:

$$\tilde{P}\left(\mathit{x}\right|D_t)\simeq \prod _{i=1}^N\left(\frac{e^{-a_i^t{x}_i^2/2+h_i^t{x}_i}\varphi \left(x_i\right)}{Z(a_i^t,h_i^t)}\right)$$

(6)

for the posterior distribution $P\left(\mathit{x}\right|D_t)$ in Label (5) as a device to facilitate marginal computations and (as a consequence) their expected values. In this expression, $\left\{(a_i^t,h_i^t)\right\}$ is a set of natural parameters and

$$Z(a_i^t,h_i^t)=\int d{x}_i{e}^{-a_i^t{x}_i^2/2+h_i^t{x}_i}\varphi \left(x_i\right)$$

(7)

is the normalization of the marginal distribution ${\tilde{P}}_i\left(x_i\right|D_t)=\int d{\mathit{x}}_{\backslash i}\tilde{P}\left(\mathit{x}\right|D_t)$, where $d{\mathit{x}}_{\backslash i}$ denotes integration over all ${\left\{x_j\right\}}_{j\ne i}$. So as not to introduce any biases, we define $h_i^0=a_i^0=0,\forall i$, which results in ${\tilde{P}}_i\left(x_i\right|D^0\equiv ⌀)={\prod }_{i=1}^N\varphi \left(x_i\right)$. The mmse estimate of the signal and its variance can then be readily calculated from the approximate posterior as

$$\begin{array}{cc}\hfill m_i^t& =(\partial /\partial h_i^t)lnZ(a_i^t,h_i^t),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill v_i^t& =({\partial }^2/{\left(\partial h_i^t\right)}^2)lnZ(a_i^t,h_i^t),\hfill \end{array}$$

(9)

respectively. Equations (5) and (6) correspond to a sequence of update and projection steps—the update step adds new information obtained with the most recent measurement t, but transforms the posterior into a possibly intractable distribution; the projection step then returns this distribution to an easier exponential representation with independent components (Figure 2). The full scheme can be summarized as $2N$ update rules for the parameters $\left\{(a_i^t,h_i^t)\right\}$ [24]. Defining ${\Delta }^t\equiv {\sum }_{i=1}^N{A}_i^t{m}_i^{t-1}$ and ${\chi }^t\equiv {\sum }_{i=1}^N{\left(A_i^t\right)}^2{v}_i^{t-1}$, these rules are

$$\begin{array}{cc}\hfill a_i^{t+1}& =a_i^t-{\left(A_i^{t+1}\right)}^2\frac{{\partial }^2}{{\left(\partial {\Delta }^{t+1}\right)}^2}ln{\Omega }^t\left({\Delta }^{t+1}\right),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill h_i^{t+1}& =h_i^t+A_i^{t+1}\frac{\partial }{\partial {\Delta }^{t+1}}ln{\Omega }^t\left({\Delta }^{t+1}\right)-m_i^t{\left(A_i^{t+1}\right)}^2\frac{{\partial }^2}{{\left(\partial {\Delta }^{t+1}\right)}^2}ln{\Omega }^t\left({\Delta }^{t+1}\right),\hfill \end{array}$$

(11)

where ${\Omega }^t\left({\Delta }^{t+1}\right)=\int \mathcal{D}zP\left(y^{t+1}\right|{\Delta }^{t+1}+\sqrt{{\chi }^{t+1}}z)$ and $\mathcal{D}z=(dz/\sqrt{2\pi })exp(-z^2/2)$ is the standard Gaussian measure. It has been shown [24] that this online algorithm has an asymptotic error decay comparable to the offline reconstruction scheme when additive measurement noise is present. In the noiseless scenario, the offline Bayesian algorithm achieves zero error for finite t, while the online version decays exponentially.

4. Mismatched Priors and L1-Minimization Based Reconstruction

A limitation of the framework introduced above is the simple fact that, in a real-world scenario, one does not know the exact parameters of the generating distribution, and possibly not even some of its major characteristics except for the fact that the generated signals are somewhat sparse. Using this algorithm as a canvas, we propose a strategy based on a Bayesian formulation of the L1-regularized problem, relying on the use of the sparsity-inducing Laplace prior

$${\varphi }_{\lambda }\left(x\right)=(2/\lambda )exp(-\lambda |x\left|\right)$$

(12)

and a learning scheme for $\lambda$ concurrent with the measurements. The use of Laplace-like priors is not new in the literature [29,30,31], but, to the best of our knowledge, this has not been introduced as part of an online algorithm yet.

We observe that the L1-estimator (1) can also be written in a Bayesian manner as ${\widehat{\mathit{x}}}^t=\int d\mathit{x}\mathit{x}{lim}_{\beta \to \infty }{P}_{L1}\left(\mathit{x}\right|D^t;\beta ,\lambda )$, where

$$P_{L1}\left(\mathit{x}\right|D^t;\beta ,\lambda )=Z_{\beta ,\lambda }^{-1}exp\left[-\beta \left(\frac{1}{2{\sigma }_n^2}\sum _{\mu =1}^t{(y^{\mu }-{\mathit{A}}^{\mu }·\mathit{x})}^2+\lambda {\parallel \mathit{x}\parallel }_1\right)\right],$$

(13)

with $Z_{\beta ,\lambda }$ a normalization factor dependent on $\beta$ and $\lambda$. This formulation suggests the replacement of the generating distribution $\varphi \left(x\right)$ in Label (6) for ${\varphi }_{\lambda }\left(x\right)$, effectively altering the projection step of the Bayesian online algorithm to the similar expression

$${\tilde{P}}_{\lambda }\left(x\right|D_t)\simeq \prod _{i=1}^N\left(\frac{e^{-a_i^t{x}_i^2/2+h_i^t{x}_i-\lambda \left|x_i\right|}}{Z(a_i^t,h_i^t;\lambda )}\right),$$

(14)

where $Z(a,h;\lambda ):=\int dxexp\left(-a{x}^2/2+hx-\lambda \left|x\right|\right)$. The first and second moments of this alternate approximate distribution are, respectively,

$$\begin{array}{cc}\hfill m_i^t& =(\partial /\partial h_i^t)lnZ(a_i^t,h_i^t;\lambda ),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill v_i^t& =({\partial }^2/{\left(\partial h_i^t\right)}^2)lnZ(a_i^t,h_i^t;\lambda ),\hfill \end{array}$$

(16)

paralleling Equations (8) and (9). Note that the natural parameters update Equations (10) and (11) remain unchanged, since they are independent of our choice of prior, be it $\varphi \left(x\right)$, ${\varphi }_{\lambda }\left(x\right)$, or any other.

A common practical issue when making use of the L1-regularization scheme (1) is that the best value of the shrinkage parameter $\lambda$ is data-dependent and not always clear. Nevertheless, its value is highly influential on the reconstructed model—there is a direct relation between $\lambda$ and the number of non-zero coefficients in the estimate, which is reflected on the prediction accuracy of the model [32]. Therefore, since it is crucial to estimate accurate values of the shrinkage parameter, usual strategies in offline scenarios include calculation of its best value through cross-validation procedures [25] or straight calculation of an ideal number of non-zero covariates by adding one at a time in a recursive manner [32]. Unfortunately, these strategies are of little use in our present setting, where each measurement has to be used only once in an online manner.

Given any two distributions $q\left(x\right)$ and $r\left(x\right)$, the Kullback–Leibler divergence [33], $D_{KL}\left(q\right|\left|r\right)=\int dxq\left(x\right)ln[q\left(x\right)/r\left(x\right)]$ is a common measure for their dissimilarity. Also called relative entropy, it has the interesting property that $D_{KL}\left(q\right|\left|r\right)\ge 0$ always and it is zero if and only if $q=r$. Now, whenever distribution $r\left(x\right)$ is of the exponential family, i.e., $f\left(x\right)\propto {\sum }_k{\xi }_k{s}_k\left(x\right)$, minimization of $D_{KL}\left(q\right|\left|r\right)$ with respect to ${\xi }_k$ is equivalent to matching the moments $\int dxq\left(x\right)s_k\left(x\right)$ and $\int dxr\left(x\right)s_k\left(x\right)$. In particular, for the exact posterior P as defined in the left-hand side of Label (2), minimization of $D_{KL}\left(P\right||{\tilde{P}}_{\lambda })$ with respect to $\lambda$ after measurement t, i.e., finding ${\lambda }_t$ such that

$${\lambda }_t=\underset{\lambda }{arg\; min}{D}_{KL}(P\left|\right|{\tilde{P}}_{\lambda })$$

(17)

is the same as solving the implicit equation

$${\langle \left|\mathit{x}\right|\rangle }_P={\langle \left|\mathit{x}\right|\rangle }_{{\tilde{P}}_{\lambda }}$$

(18)

for $\lambda$. Here, ${\langle \left|\mathit{x}\right|\rangle }_P$ is the best possible estimate for the true sparsity of the original signal, $Q_0\equiv \int dx\left|x\right|\varphi \left(x\right)$. Our proposal is that ${\lambda }_t$ is, at any instant t, an optimal estimate for the shrinkage parameter. Moreover, it is not static. With the natural parameters $\left\{(a_i^t,h_i^t)\right\}$ evolving, ${\lambda }_t$ must change as well to maintain ${\langle \left|\mathit{x}\right|\rangle }_{{\tilde{P}}_{\lambda }}\simeq Q_0$.

Alas, the left-hand side of expression (18) is unknown. To overcome this difficulty, we make the approximation ${\langle \left|\mathit{x}\right|\rangle }_P\simeq \left|{\mathit{m}}^t\right|$. An argument can be made in its defense: as more and more measurements are acquired and the approximate posterior distribution (14) concentrates around the true values, two limits are obtained: (a) ${\langle \left|x\right|\rangle }_{{\tilde{P}}_{\lambda }}\underset{t\to \infty }{\to }{\langle \left|x\right|\rangle }_P$ and (b) ${\langle \left|x\right|\rangle }_{{\tilde{P}}_{\lambda }}\underset{t\to \infty }{\to }\left|{\langle x\rangle }_{{\tilde{P}}_{\lambda }}\right|$. The first limit has been proved in the large signal size limit $N\to \infty$ in [24] for an exact prior; here, we assume its validity also in the present scenario. As for the second limit, Minkowski inequality guarantees that ${\langle \left|\mathit{x}\right|\rangle }_{{\tilde{P}}_{\lambda }}\le \left|{\mathit{m}}^t\right|$; with the collapse of the distribution ${\tilde{P}}_i\left(x_i\right)$ around the true value $x_i^0$ at $t\to \infty$, no probability mass crosses the $x_i=0$ axis, so that the equality limit is met. Therefore, these results mean a good estimate for the solution of (18) is given by ${\widehat{\lambda }}_t$ that solves

$${\langle \left|\mathit{x}\right|\rangle }_{{\tilde{P}}_{\lambda }}\simeq \left|{\mathit{m}}^t\right|$$

(19)

for $\lambda$, where the right-hand side can be directly calculated from ${\tilde{P}}_{{\lambda }_{t-1}}\left(\mathit{x}\right|D^t)$. This implicit expression corresponds to an extra step in the online algorithm for the recovery of the signal, which is shown in full below (Algorithm 1).

Algorithm 1 Online $L1$-based signal recovery for CS.
1: Initialize $P_{\lambda }\left(\mathit{x}\right)$	▹$a_i^0,h_i^0=0(\forall i)$; ${\lambda }^0=\lambda$
2: while $t<t_{\max }$ do
3:   Obtain new measurement $y^t\sim P\left(y^t\right|A^t·x^0)$
4:   Update $\left\{(a_i^t,h_i^t)\right\}$	▹ Equations (10) and (11)
5:   Find ${\widehat{\lambda }}_t$	▹ Equation (19)
6:   Update ${\lambda }_t=(1-\gamma ){\lambda }_{t-1}+\gamma {\widehat{\lambda }}_t$
7:   Estimate signal means $m_i^t$ and variances $v_i^t$	▹ Equations (15) and (16)
8: end while
9: return ${\mathit{m}}^t$

5. Results and Discussion

We tested the algorithm on sparse signals generated by two different distributions of the form $\varphi \left(\mathit{x}\right)={\prod }_{i=1}^N\varphi \left(x_i\right)$, with $\varphi \left(x_i\right)=(1-\rho )\delta \left(x_i\right)+\rho g\left(x_i\right)$. The $\delta \left(x\right)$ factor induces sparsity by making every component equal to zero with probability $1-\rho$. The non-sparse components could either come from a Gaussian distribution $g\left(x\right)=(1/\sqrt{2\pi })exp(-x^2/2)$ or from a binary distribution $g\left(x\right)=\left(\delta \right(x+1)+\delta (x-1\left)\right)/2$. To better evaluate the accuracy losses incurred due to approximation (19), we also tested the algorithm for the ideal situation where $Q_0$ is known. To guarantee practical usefulness, all results correspond to the standard CS scenario $y^{\mu }=u^{\mu }+{\xi }^{\mu }$ with Gaussian noise ${\xi }^{\mu }\sim \mathcal{N}(0,{\sigma }_n^2)$, i.e., $P\left(y\right|u)=(1/\sqrt{2\pi {\sigma }_n^2})exp[-{(y-u)}^2/2{\sigma }_n^2]$. Its noiseless counterpart can be obtained by taking the limit ${\sigma }_n^2\to 0$, which leads to $P\left(y\right|u)=\delta (y-u)$. Figure 3 shows the average mean squared error normalized by $\parallel {\mathit{x}}^0{\parallel }_2^2$ resulting of simulations of the L1-based online CS algorithm with $\lambda$ learning. Simulations for finite signal length size N showed decreasing error for increasing N—in order to correct for finite size effects, the curves in Figure 3 were obtained by extrapolating the results for finite $N=200,500,1000$ and 2000 to $N\to \infty$ by means of a quadratic fit. As expected, approximation (19) induces a significant accuracy loss, but, for $\alpha \ge 1$, the reconstruction error decays approximately exponentially, proving the usefulness of the reconstruction scheme.

In all the examples shown, the solution to Equation (19) was obtained by numerically searching the value ${\widehat{\lambda }}_t$ which best approximates both sides, i.e., by finding ${\widehat{\lambda }}_t={arg\; min}_{\lambda }{[\langle |\mathit{x}|\rangle }_{{\tilde{P}}_{\lambda }}-|{\mathit{m}}^t{\left|\right]}^2$. The empirical results prove what could have been a reasonable assumption: Figure 4 shows that in the early stages when only a few measurements have been obtained, even for an exact prior, $|{\mathit{m}}^t|$ is not an excellent approximation of ${\langle \left|\mathit{x}\right|\rangle }_P$. It also shows that the approximation gets progressively better with more measurements. For this reason, we found that the use of a small damping factor $\gamma$ in the update of $\lambda$ provides faster and more accurate learning.

Finding an optimal value for the sparsity-inducing parameter $\lambda$ is not trivial. Ideally, one desires to find $\lambda$ such that the ${\tilde{P}}_{\lambda }\left(\mathit{x}\right)$ would represent the actual sparsity of the original signal $\langle \left|x_i^0\right|\rangle$. The limit $\lambda \to \infty$ is undesirable, since the soft constraints in ${\tilde{P}}_{\lambda }\left(\mathit{x}\right)$ due to the finite value of the natural parameters $\left\{(a_i^t,h_i^t)\right\}$ would not be enough to prevent the prior distribution to completely dominate the entire approximate posterior, leading all signal estimates to zero. At the same time, consider as a representative example the update Equations (10) and (11) for the noisy standard CS scenario, which can explicitly written as $a_i^{t+1}=a_i^t+{\left(A_i^{t+1}\right)}^2/({\sigma }_n^2+{\chi }^{t+1})$ and $h_i^{t+1}=h_i^t+A_i^{t+1}(y^{t+1}-{\Delta }^{t+1})/({\sigma }_n^2+{\chi }^{t+1})+m_i{\left(A_i^{t+1}\right)}^2/({\sigma }_n^2+{\chi }^{t+1})$. Taking the average of $d{a}_i^{\mu }=a_i^{\mu }-a_i^{\mu -1}$ and $d{h}_i^{\mu }=h_i^{\mu }-h_i^{\mu -1}$ over the measurement vectors ${\mathit{A}}^{\mu }$ results in

$$\left\{\begin{array}{c}{\langle d{a}_i^{\mu }\rangle }_{{\mathit{A}}^{\mu }}\simeq N^{-1}/({\sigma }_n^2+{\chi }^{\mu }),\hfill \\ {\langle d{h}_i^{\mu }\rangle }_{{\mathit{A}}^{\mu }}\simeq N^{-1}{m}_i/({\sigma }_n^2+{\chi }^{\mu }),\hfill \end{array}\right.$$

(20)

which means that all the natural parameters $a_i^t={\sum }_{\mu =1}^{t}d{a}_i^{\mu }$ typically grow with more and more measurements. Similarly, whenever $x_i^0$ is different from zero, any parameters $h_i^t={\sum }_{\mu =1}^{t}d{h}_i^{\mu }$ exhibit a growth in absolute value lead by the $m_i$ factor. For any fixed $\lambda$, these considerations mean that the regularizing effect of the prior ${\varphi }_{\lambda }\left(x\right)$ in Label (14) would vanish asymptotically in what would be a typical case of a prior being dominated by the likelihood. It is noteworthy that the introduction of a step for the minimization of the KL-divergence in the Bayesian online algorithm results in estimates ${\widehat{\lambda }}_t$ that typically increase with $\alpha$ as well, in a scale similar to the other natural parameters (Figure 5).

6. Conclusions

In this paper, we proposed an online algorithm for Compressed Sensing. Based on an earlier version that expected exact knowledge of the signal generating distribution [24], the present adaptation made use of a double-exponential prior (12) to get rid of this limitation. In order to keep the signal sparsity always in focus during the signal reconstruction, we introduced in this work an extra step where an optimal value for the sparsity-inducing parameter $\lambda$ can be estimated as a function of $\left\{(a_i^t,h_i^t)\right\}$ (i.e., a function of the data $D^t$). This was achieved through minimization of the KL-divergence between ${\tilde{P}}_{\lambda }\left(\mathit{x}\right)$ and the (unavailable) full posterior distribution (2), which is approximated by the signal estimate at the previous step.

It is clear that the method is strongly dependent on the validity of the mean-field approximation (6) for the posterior to obtain good reconstruction accuracy. This approximation was introduced in order to simplify the calculations of the moments and to reduce the number of parameters, which should be updated at every step (and, as a consequence, to reduce computational effort and memory requirements). Nevertheless, it is not essential and can be easily replaced by a full Gaussian distribution if needed. In all examples shown here, which made use of signals with fully independent components, the algorithm was shown to give accurate predictions for different signal generating distributions, with and without the presence of additive measurement noise.