Conjugate Gradient Hard Thresholding Pursuit Algorithm for Sparse Signal Recovery

Abstract

We propose a new iterative greedy algorithm to reconstruct sparse signals in Compressed Sensing. The algorithm, called Conjugate Gradient Hard Thresholding Pursuit (CGHTP), is a simple combination of Hard Thresholding Pursuit (HTP) and Conjugate Gradient Iterative Hard Thresholding (CGIHT). The conjugate gradient method with a fast asymptotic convergence rate is integrated into the HTP scheme that only uses simple line search, which accelerates the convergence of the iterative process. Moreover, an adaptive step size selection strategy, which constantly shrinks the step size until a convergence criterion is met, ensures that the algorithm has a stable and fast convergence rate without choosing step size. Finally, experiments on both Gaussian-signal and real-world images demonstrate the advantages of the proposed algorithm in convergence rate and reconstruction performance.

1. Introduction

As a new sampling method, Compressed Sensing (CS) has received broad research interest in signal processing, image processing, biomedical engineering, electronic engineering and other fields [1,2,3,4,5,6]. In particular in magnetic resonance image (MRI) processing, CS technology greatly improves the efficiency of MRI processing (Figure 1). By exploiting the sparse characteristics of signals, CS can accurately reconstruct sparse signals from significantly fewer samples than required in the Shannon-Nyquist sampling theorem [7,8].

Assume that a signal $\mathit{f}\in {\mathbb{R}}^N$ is s-sparse in some domain $\mathit{\psi }$. This means $\mathit{f}=\mathit{\psi }\mathit{x}$, and $\mathit{x}$ has at most $s(s\ll N)$ nonzero entries. This system can be measured by a sampling matrix $\mathbf{\Phi }\in {\mathbb{R}}^{M\times N}(M<N)$: $\mathit{y}=\mathbf{\Phi }\mathit{f}=\mathbf{\Phi }\mathit{\psi }\mathit{x}=\mathit{Ax}$, where $\mathit{y}$ is called measurement vector, $\mathit{A}$ is the so-called measurement matrix. The CS model is shown in Figure 2. If $\mathbf{\Phi }$ is incoherent with $\mathit{\psi }$, the coefficient vector $\mathit{x}$ can be reconstructed exactly from a few measurements by solving the undetermined linear system $\mathit{y}=\mathit{Ax}$ with constraint ${∥\mathit{x}∥}_0\le s$, i.e., solving the following ${\ell }_0$ norm minimization problem:

$$min{∥\mathit{x}∥}_{0}s.t.\mathit{y}=\mathit{Ax}.$$

(1)

As a combinatorial optimization problem, the above ${\ell }_0$ norm optimization is NP-hard [8]. One way to solve such problem is to transform it into a ${\ell }_1$ norm optimization problem:

$$min{∥\mathit{x}∥}_{1}s.t.\mathit{y}=\mathit{Ax}.$$

(2)

Since ${\ell }_1$ norm-based problem is convex, some methods such as basis pursuit (BP) [7] and LASSO [9] are usually employed to solve the ${\ell }_1$ norm minimization in polynomial time. The solutions obtained by these algorithms can well approximate the exact solution of (1). However, their computational complexity is too highly impractical for many applications. As a relaxation of the ${\ell }_0$ norm optimization, the ${\ell }_p$ norm for CS reconstruction has attracted extensive research interest, and the problem can be formulated as the following optimization problem:

$$min{∥\mathit{x}∥}_{p}s.t.\mathit{y}=\mathit{Ax},$$

(3)

where ${∥\mathit{x}∥}_p={\left({\displaystyle \sum _{i=1}^N}{\left|x_i\right|}^p\right)}^{1/\phantom{1p}p}$ denotes the ${\ell }_p$ norm of $\mathit{x}$. There are many methods have been developed to solve such problem [10,11,12,13]. Many studies have shown that using ${\ell }_p$ norm for $0<p<1$ requires fewer measurements and has much better recovery performance than using ${\ell }_1$ norm [14,15]. However, ${\ell }_p$ norm leads to a non-convex optimization problem which is difficult to solve efficiently.

Sparse signals can be quickly recovered by these algorithms while the measurement matrix satisfying the so-called restricted isometry property (RIP) with a constant parameter. A measurement matrix is said to satisfy the s-order RIP if for any s-sparse signal $\mathit{x}\in {\mathbb{R}}^N$

$$(1-{\delta }_s)\le {∥\mathit{Ax}∥}_2^2/\phantom{{∥\mathit{Ax}∥}_2^2{∥\mathit{x}∥}_2^2}{∥\mathit{x}∥}_2^2\le (1+{\delta }_s),$$

(4)

where $0\le {\delta }_s\le 1$. Commonly used sampling matrices include Gaussian matrices, random Bernoulli matrices, partial orthogonal matrices, etc. The measurement matrices consisting of these sampling matrices and orthogonal basis usually satisfy the RIP condition with a high probability [16].

The core ideas of this paper include: (1) by combining the steps of HTP and CGIHT in each iteration, a new algorithm called Conjugate Gradient Hard Thresholding Pursuit (CGHTP) is presented; (2) by alternatively selecting a search direction from the gradient direction and the conjugate direction to improve the convergence rate of the iterative procedure; (3) furthermore, an adaptive step size selection strategy similar to that in normalized iterative hard thresholding (NIHT) is adopted in CGHTP algorithm, which eliminates the effect of step size on the convergence of HTP algorithm.

The remainder of the paper is organized as follows. Section 2 reviews related work on iterative greedy algorithms for CS. Section 3 gives the key ideas and description of the proposed algorithm. The convergence analysis of the CGHTP algorithm is given in Section 4. Some simulation experiments to verify the empirical performance of the CGHTP algorithm is carried out in Section 5. Finally, the conclusion of this paper is presented in Section 6.

Notations: in this paper, ∅ represents an empty set. $〈\mathbf{a},\mathbf{b}〉$ denotes the inner product of the vector $\mathit{a}$ and $\mathit{b}$. $H_s\left(\mathit{x}\right)$ denotes an operator that sets all but the largest s absolute value of $\mathit{x}$. $\mathrm{supp}\left(\mathit{x}\right)$ is denoted to take the index set of nonzero entries of $\mathit{x}$. Let $T,\tilde{T}\subseteq \{1,2,\cdots ,N\}$. Let ${\mathit{x}}_T$ denote a subvector consisting of elements extracted from $\mathit{x}$ that indexed by the elements in T. ${\mathit{A}}_T$ denotes a submatrix that consists of the columns of $\mathit{A}$ with indices $i\in T$. ${\mathit{A}}^{\mathrm{T}}$ denotes the transpose of $\mathit{A}$. $\mathit{I}$ stands for a unit matrix whose size depends on the context. In addition, let $\tau =M/\phantom{MN}N$ denote the sampling ratio.

2. Literature Review

In the past decade, a series of iterative greedy algorithms has been proposed to directly solve Equation (1). These algorithms have been attracting increased attention due to their low algorithm complexity and good reconstruction performance. According to the support set selection strategy used in the iteration process, we can roughly classify the family of iterative greedy algorithms into three categories.

(1) orthogonal matching pursuit (OMP) [17]-based OMP-like algorithms, such as compressive sampling matching pursuit (CoSaMP), subspace pursuit (SP) [18], regularized orthogonal matching pursuit (ROMP) [19], generalized orthogonal matching pursuit (GOMP) [20], sparsity adaptive matching pursuit (SAMP) [21], stabilized orthogonal matching pursuit (SOMP) [22], perturbed block orthogonal matching pursuit (PBOMP) [23] and forward backward pursuit (FBP) [24]. A common feature of these algorithms is that in each iteration, a support set is determined to approximate the correct support set according to the correlation value between the measurement vector $\mathit{y}$ and the columns of $\mathit{A}$.

(2) iterative hard thresholding (IHT)-based IHT-like algorithms, such as NIHT, accelerated iterative hard thresholding (AIHT) [25], conjugate gradient iterative hard thresholding (CGIHT) [26,27]. Unlike the OMP-like algorithm, the approximate support set for the nth iteration of IHT-like algorithms is determined by the value of ${\mathit{A}}^{\mathrm{T}}\mathit{y}+(\mathit{I}-{\mathit{A}}^{\mathrm{T}}\mathit{A}){\mathit{x}}_{n-1}$, which is closer to the correct support set than using the values of ${\mathit{A}}^{\mathrm{T}}\mathit{y}$ [28].

(3) algorithms composed of OMP-like and IHT-like algorithms, such as Hard Thresholding Pursuit (HTP) [28,29], generalized hard thresholding pursuit (GHTP) [30], ${\ell }_0$ regularized HTP [31], Partial hard thresholding (PHT) [32] and subspace thresholding pursuit (STP) [33]. OMP-like algorithms use least squares to update sparse coefficients in each iteration, which plays a role in debiasing and makes it approach the exact solution quickly. However, it is well known that the least squares are time-consuming, which leads to a high complexity of single iteration. The advantage of IHT-like algorithms are the simple iteration operation and good support set selection strategy, but it needs more iterations. By combining the OMP-like algorithm and the IHT-like algorithm, these algorithms absorb the advantages of the above two kinds of algorithms, and have strong theoretical guarantees and good empirical performance [29].

Among the above algorithms, CoSaMP and SP do not provide a better theoretical guarantee than the IHT algorithm, but their empirical performance is better than IHT. The simple combination of CoSaMP and IHT algorithm lead to the HTP algorithm whose convergence requires fewer number of iterations than IHT algorithm. However, the reconstruction performance of HTP is greatly affected by the step size and an optimal step size is difficult to determine in practical applications, which may cause the HTP to converge slowly when selecting an inappropriate step size. Furthermore, although the HTP algorithm only requires a small number of iterations to converge to the exact vector $\mathit{x}$, it only uses a single gradient descent direction as the search direction in iterative progress, which may not be the best choice for general optimization problems. In [26], conjugate gradient method was used to accelerate the convergence of IHT algorithm due to its fast convergence rate. In this paper, inspired by the HTP algorithm and CGIHT algorithm, the operations of these two algorithms in each iteration is combined to take advantage of both.

In this paper, we first introduce the idea of the CGHTP algorithm, then analyze the convergence of the proposed algorithm, and finally validate the proposed algorithm with synthetic experiments and real-world image reconstruction experiments.

3. Conjugate Gradient Hard Thresholding Pursuit Algorithm

Firstly, we make a simple summary of the HTP algorithm and the CGIHT algorithm to facilitate an intuitive understanding of the proposed algorithm. The pseudo-codes of HTP and CGIHT are shown in Algorithms 1 and 2, respectively. In [20], the debiasing step in the OMP-like algorithms and the estimation step in the IHT-like algorithms are coupled in an algorithm to form the HTP algorithm. The HTP algorithm has excellent reconstruction performance as well as strong theoretical guarantee in terms of the RIP. Optimizing search direction in each iteration is the main innovation of HTP algorithm. It fully considers that in the iterative process, when the subspace determined by the support set is consistent, the conjugate gradient (CG) method [23] can be employed to accelerate the convergence rate. Two versions of CGIHT algorithms for CS are provided in [19], one of them, the algorithm called CGIHT restarted for CS is summarized in Algorithm 2. The iteration procedures of CGIHT and CGIHT restarted differ in their selection of search directions. CGIHT algorithm uses the conjugate gradient respect to ${\mathit{A}}_T^{\mathrm{T}}\mathit{A}$ as the search direction in all iteration, while CGIHT restarted determines the search direction according to whether the support set of the current iteration is equal to that of the previous iteration.

Inspired by the respective advantages of HTP and CGIHT algorithms, we present the CGHTP algorithm in this section. A simple block diagram of the CGHTP algorithm can be seen in Figure 3. The algorithm enters an iterative loop after inputting the initialized data. The loop process updates the ${\mathit{x}}_{\mathit{n}}$ and support set $T_n$ in two alternative ways, one of which is the combination of the gradient descent method and the adaptive step size selection method, and the other is the CG method. The selection criteria for these two ways is to determine whether the support sets of two adjacent iterations are equal. If the support set of the previous iteration is equal to that of the current iteration, then the gradient descent method is used. If not, the CG method is used [34]. All candidate solutions and candidate support sets obtained through these ways are finally updated by the least square method.

The main characteristic of the proposed CGHTP algorithm is that the support set selection strategy of HTP is combined with the acceleration strategy of CGIHT restarted algorithm in one algorithm. The main steps of CGHTP are listed in Algorithm 3. Similar to CGIHT, CGHTP is initialized with $\mathit{x}=0$, $T_{-1}=\mathrm{supp}\left(H_s\left({\mathit{A}}^{\mathrm{T}}\mathit{y}\right)\right)$. The main body of CGHTP mainly includes the following steps. In Step 1), the gradient direction of the cost function ${∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2^2$ [16] in the current iteration is solved to calculate the subsequent step size ${\alpha }_n$. In Step 2), if ${\tilde{\mathit{x}}}_{n+1}$ has a different support from the previous estimate, i.e., $T_n\ne T_{n-1}$, similar to the convergence criterion used in NIHT algorithm, $c{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2/\phantom{{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2{∥\mathit{A}({\mathit{x}}_{n+1}-{\mathit{x}}_n)∥}_2^2}{∥\mathit{A}({\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n)∥}_2^2$ is used to check whether the step size is smaller than it, where c is a small fixed constant and $0<c<1$. If this holds, continue to run the next steps. Otherwise the step size is continuously reduced by looping ${\alpha }_n\leftarrow {\alpha }_n/\phantom{{\alpha }_n\eta }\eta$ until the convergence criterion is met, here $0<\eta <1$. In Step 3), if $T_n=T_{n-1}$, the CG direction is used as the search direction. Finally, Step 4) updates ${\mathit{x}}_{n+1}$ by solving a least square problem.

Compared to the HTP algorithm, the CGHTP algorithm provides two search directions in the iterative process, including the gradient direction ${\mathit{g}}_n$ of the cost function ${∥\mathit{y}-{\mathit{Ax}}_n∥}_2^2$ and the CG direction ${\mathit{d}}_n$. According to whether the support sets of the previous and current iterations are equal, one of the two directions is selected as the search direction. In addition, then the support set is updated by solving a least square problem. In addition, CGHTP provides a way to choose the best step size while ensuring convergence.

The HTP algorithm uses the gradient direction as the search direction, which has a linear asymptotic convergence rate related to $(\kappa -1)/\phantom{(\kappa -1)(\kappa +1)}(\kappa +1)$, where $\kappa$ is the condition number of the matrix ${\mathit{A}}_{\mathrm{supp}\left({\mathit{x}}_n\right)}^{\mathrm{T}}{\mathit{A}}_{\mathrm{supp}\left({\mathit{x}}_n\right)}$. Owing to the superlinear asymptotic convergence speed with a rate given by $(\sqrt{\kappa }-1)/\phantom{(\sqrt{\kappa }-1)(\sqrt{\kappa }+1)}(\sqrt{\kappa }+1)$ [24], conjugate gradient method has been applied to accelerate the convergence rate in CGIHT algorithm. CGHTP draws on the advantages of the CGIHT algorithm. If $T_n=T_{n-1}$, it means that the correct support set is identified and the submatrix ${\mathit{A}}_{T_n}$ is no longer changed, CG method is applied to solve the overdetermined system ${\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}{\mathit{x}}_n={\mathit{A}}_{T_n}^{\mathrm{T}}\mathit{y}$, and the convergence rate may be superlinear with a rate given by $(\sqrt{\kappa }-1)/\phantom{(\sqrt{\kappa }-1)(\sqrt{\kappa }+1)}(\sqrt{\kappa }+1)$.

Algorithm 1 [20] Hard Thresholding Pursuit

Input: $\mathit{A}$, $\mathit{y}$, s  Initialization: ${\mathit{x}}_0=0$, $T_0=\varnothing$.  for each iteration $n\ge 1$ do   1) ${\tilde{\mathit{x}}}_n={\mathit{x}}_{n-1}+{\mathit{A}}^{\mathrm{T}}(\mathit{y}-\mathit{A}{\mathit{x}}_{n-1})$   2) $T_n=\mathrm{supp}\left(H_s\left({\tilde{\mathit{x}}}_n\right)\right)$   3) ${\mathit{x}}_n=arg{min}_{\mathit{z}\in {\mathbb{R}}^N}\{{∥\mathit{y}-\mathit{Az}∥}_2,\mathrm{supp}\left(\mathit{z}\right)\subseteq T_n\}$   4) update the residual ${\mathit{r}}_n=\mathit{y}-\mathit{A}{\mathit{x}}_n$   until the stopping criteria is met  end for

Algorithm 2 [19] Conjugate Gradient Iterative Hard Thresholding restarted for Compressed Sensing

Input: $\mathit{A}$, $\mathit{y}$, s  Initialization: ${\mathit{x}}_0=0$, $T_{-1}=\varnothing$, ${\mathit{d}}_{-1}=0$, $T_0=\mathrm{supp}\left(H_s\left({\mathit{A}}^T\mathit{y}\right)\right)$.  for each iteration $n\ge 0$ do   1) ${\mathit{g}}_n={\mathit{A}}^{\mathrm{T}}(\mathit{y}-\mathit{A}{\mathit{x}}_n)$ (compute the gradient direction)   2) if $T_n\ne T_{n-1}$ then       ${\beta }_n=0$     else      ${\beta }_n=-\frac{〈\mathit{A}{\mathit{g}}_{n{T}_n},\mathit{A}{\mathit{d}}_{n-1T_n}〉}{〈\mathit{A}{\mathit{d}}_{n-1T_n},\mathit{A}{\mathit{d}}_{n-1T_n}〉}$  (compute orthogonalization weight)     end   3) ${\mathit{d}}_n={\mathit{g}}_n+{\beta }_n{\mathit{d}}_{n-1}$  (compute conjugate gradient direction)   4) ${\alpha }_n=〈{\mathit{g}}_{n{T}_n},{\mathit{g}}_{n{T}_n}〉/\phantom{〈{\mathit{g}}_{n{T}_n},{\mathit{g}}_{n{T}_n}〉〈{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n},{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n}〉}〈{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n},{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n}〉$ (compute step size)   5) ${\mathit{x}}_{n+1}=H_s({\mathit{x}}_n+{\alpha }_n{\mathit{d}}_n)$, $T_{n+1}=\mathrm{supp}\left({\mathit{x}}_{n+1}\right)$   until the stopping criteria is met  end for

Algorithm 3 Conjugate Gradient Hard Thresholding Pursuit

Input: $\mathit{A}$, $\mathit{y}$, s,  Initialization: ${\mathit{x}}_0=0$, $T_{-1}=\varnothing$, ${\mathit{d}}_{-1}=0$, $T_0=\mathrm{supp}\left(H_s\left({\mathit{A}}^{\mathrm{T}}\mathit{y}\right)\right)$.  for each iteration $n\ge 0$ do   1) ${\mathit{g}}_n={\mathit{A}}^{\mathrm{T}}(\mathit{y}-\mathit{A}{\mathit{x}}_n)$ (compute gradient direction)   2) if $T_n\ne T_{n-1}$ then       ${\alpha }_n=〈{\mathit{g}}_{n{T}_n},{\mathit{g}}_{n{T}_n}〉/\phantom{〈{\mathit{g}}_{n{T}_n},{\mathit{g}}_{n{T}_n}〉〈{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n},{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n}〉}〈{\mathit{A}}_{T_n}{\mathit{g}}_{n{T}_n},{\mathit{A}}_{T_n}{\mathit{g}}_{n{T}_n}〉$       if ${\alpha }_n\ge c{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2/\phantom{{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2{∥\mathit{A}({\mathit{x}}_{n+1}-{\mathit{x}}_n)∥}_2^2}{∥\mathit{A}({\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n)∥}_2^2$ then       execute loop ${\alpha }_n\leftarrow {\alpha }_n/\phantom{{\alpha }_{n}2}\eta$ until (adaptively select step size)       ${\alpha }_n<{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2/\phantom{{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2{∥\mathit{A}({\mathit{x}}_{n+1}-{\mathit{x}}_n)∥}_2^2}{∥\mathit{A}({\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n)∥}_2^2$       ${\tilde{\mathit{x}}}_{n+1}=H_s({\mathit{x}}_n+{\alpha }_n{\mathit{d}}_n)$, $T_{n+1}=\mathrm{supp}\left({\tilde{\mathit{x}}}_{n+1}\right)$       end     end   3) if $T_n=T_{n-1}$ then       ${\beta }_n=-\frac{〈{\mathit{A}}_{{\mathit{T}}_{\mathit{n}}}{\mathit{g}}_{n{T}_n},{\mathit{A}}_{{\mathit{T}}_{\mathit{n}}}{\mathit{d}}_{n-1T_n}〉}{〈{\mathit{A}}_{{\mathit{T}}_{\mathit{n}}}{\mathit{d}}_{n-1T_n},{\mathit{A}}_{{\mathit{T}}_{\mathit{n}}}{\mathit{d}}_{n-1T_n}〉}$       ${\mathit{d}}_n={\mathit{g}}_n+{\beta }_n{\mathit{d}}_{n-1}$  (compute conjugate gradient direction)       ${\alpha }_n=〈{\mathit{g}}_{n{T}_n},{\mathit{g}}_{n{T}_n}〉/\phantom{〈{\mathit{g}}_{n{T}_n},{\mathit{g}}_{n{T}_n}〉〈{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n},{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n}〉}〈{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n},{\mathit{A}}_{T_n}{\mathit{d}}_{n{T}_n}〉$ (compute optimal step size)       ${\tilde{\mathit{x}}}_{n+1}=H_s({\mathit{x}}_n+{\alpha }_n{\mathit{d}}_n)$, $T_{n+1}=\mathrm{supp}\left({\tilde{\mathit{x}}}_{n+1}\right)$     end   4) ${\mathit{x}}_{n+1}=argmin\left\{{∥\mathit{y}-\mathit{Az}∥}_2,\mathrm{supp}\left(\mathit{z}\right)\subseteq T_{n+1}\right\}$  end for  Output: ${\mathit{x}}_{\mathit{n}+\mathbf{1}}$, $T_{n+1}$

4. Convergence Analysis of CGHTP

In this section, we analyze the convergence of the proposed algorithms. We give a simple proof that the CGHTP algorithm converges to the exact solution of (1) after a finite number of iterations when the measurement matrix $\mathit{A}$ satisfies some conditions.

In Algorithm 3, when $T_n\ne T_{n-1}$ and ${\alpha }_n\le c{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2/\phantom{{∥{\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n∥}_2^2{∥\mathit{A}({\mathit{x}}_{n+1}-{\mathit{x}}_n)∥}_2^2}{∥\mathit{A}({\tilde{\mathit{x}}}_{n+1}-{\mathit{x}}_n)∥}_2^2$, the iterative process of the CGHTP algorithm is equivalent to the HTP algorithm. Similar to Lemma 3.1 in [20], we obtain the following lemma:

Lemma 1.

The sequences defined by CGHTP eventually periodic.

Theorem 1.

The sequence$\left({\mathit{x}}_n\right)$generated by CGHTP algorithm converges to$\mathit{x}$in a finite number of iterations, where$\mathit{x}$denotes the exact solution of (1)

The proof of Theorem 1 is partitioned here into two steps. The first step is that when the current support set is not equal to the previous support set, the orthogonal factor $\beta$ is equal to 0. At this time, the CGHTP algorithm reduces to the HTP algorithm, so their convergence is the same. The second step is that when the two support sets are equal, the step size is related to the orthogonal factor and the CG direction. Detailed proof is given below.

Proof of Theorem 1.

When $T_n\ne T_{n-1}$, according to Algorithm 3, ${\tilde{\mathit{x}}}_{n+1}=H_s({\mathit{x}}_n+{\alpha }_n{\mathit{g}}_n)$ and $T_{n+1}=\mathrm{supp}\left({\tilde{\mathit{x}}}_{n+1}\right)$, since ${\mathit{x}}_{n+1}=argmin\left\{{∥\mathit{y}-\mathit{Az}∥}_2,\mathrm{supp}\left(\mathit{z}\right)\subseteq T_{n+1}\right\}$, one has

$$\begin{array}{l}{∥\mathit{y}-\mathit{A}{\mathit{x}}_{n+1}∥}_2^2-{∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2^2\le {∥\mathit{y}-\mathit{A}{\tilde{\mathit{x}}}_{n+1}∥}_2^2-{∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2^2\\ ={∥\mathit{y}-\mathit{A}{\mathit{x}}_n+\mathit{A}({\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1})∥}_2^2-{∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2^2\\ =2〈{\mathit{g}}_n,{\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1}〉+{∥\mathit{A}({\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1})∥}_2^2\end{array}$$

(5)

It is clear that ${\tilde{\mathit{x}}}_{n+1}$ is a better approximation to ${\mathit{x}}_n+{\alpha }_n{\mathit{g}}_n$ than ${\mathit{x}}_n$ is, so ${∥{\mathit{x}}_n+{\alpha }_n{\mathit{g}}_n-{\tilde{\mathit{x}}}_{n+1}∥}_2^2\le {∥{\alpha }_n{\mathit{g}}_n∥}_2^2$. By expanding the squares, we can obtain ${∥{\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1}∥}_2^2+2{\alpha }_n〈{\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1},{\mathit{g}}_n〉\le 0$, substituting this into (5), one has

$$\begin{array}{c}\hfill \begin{array}{c}{∥\mathit{y}-\mathit{A}{\mathit{x}}_{n+1}∥}_2^2-{∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2^2\le -\frac{1}{{\alpha }_n}{∥{\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1}∥}_2^2+{∥\mathit{A}({\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1})∥}_2^2\hfill \end{array}\end{array}$$

(6)

substituting ${\alpha }_n\le c{∥{\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1}∥}_2^2/\phantom{{∥{\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1}∥}_2^2{∥\mathit{A}({\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1})∥}_2^2}{∥\mathit{A}({\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1})∥}_2^2$ into inequality (6), one can show that

$$\begin{array}{c}\hfill \begin{array}{c}{∥\mathit{y}-\mathit{A}{\mathit{x}}_{n+1}∥}_2^2-{∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2^2\le \left(1-\frac{1}{c}\right){∥\mathit{A}({\mathit{x}}_n-{\tilde{\mathit{x}}}_{n+1})∥}_2^2\hfill \end{array}\end{array}$$

(7)

Since $0<c<1$, the right side of (7) is less than 0. This implies that the cost function is nonincreasing when $T_n\ne T_{n-1}$, hence it is convergent.

In the case of $T_n=T_{n-1}$, for the overdetermined linear system ${\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}{\mathit{x}}_n={\mathit{A}}_{T_n}^{\mathrm{T}}\mathit{y}$, since ${\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}$ is a positive definite matrix, CG method for solving such problem has the convergence rate as:

$$\begin{array}{c}\hfill {∥{\mathit{x}}_n-\mathit{x}∥}_{{\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}}\le 2{(\frac{\sqrt{\kappa }-1}{\sqrt{\kappa }+1})}^n{∥{\mathit{x}}_0-\mathit{x}∥}_{{\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}}\end{array}$$

(8)

where ${∥{\mathit{x}}_n-\mathit{x}∥}_{{\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}}={({\mathit{x}}_n-\mathit{x})}^{\mathrm{T}}{\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}({\mathit{x}}_n-\mathit{x})$. The submatrix ${\mathit{A}}_{T_n}$ satisfies the RIP condition: $(1-{\delta }_s){∥{\mathit{x}}_n-\mathit{x}∥}_2^2\le {∥{\mathit{A}}_{T_n}({\mathit{x}}_n-\mathit{x})∥}_2^2\le (1+{\delta }_s){∥{\mathit{x}}_n-\mathit{x}∥}_2^2$. It is easy to get that $\kappa \le \frac{1+{\delta }_s}{1-{\delta }_s}$. Then (8) can be rewritten as

$$\begin{array}{c}\hfill {∥{\mathit{x}}_n-\mathit{x}∥}_{{\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}}\le 2{\rho }^n{∥{\mathit{x}}_0-\mathit{x}∥}_{{\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}}\end{array}$$

(9)

where $\rho =\frac{\sqrt{1+{\delta }_s}-\sqrt{1-{\delta }_s}}{\sqrt{1+{\delta }_s}+\sqrt{1-{\delta }_s}}<1$. From Lemma 1, since the sequence $\left({\mathit{x}}_n\right)$ is eventually periodic, it must be eventually constant, which implies that $T_{n+1}=T_n$ and ${\mathit{x}}_{n+1}={\mathit{x}}_n$ for n large enough. In summary, the CGHTP algorithm is convergent. □

5. Numerical Experiments and Discussions

In this section, we verify the performance of the proposed algorithm by Gaussian-signal reconstruction experiments and real-world images reconstruction experiments. This section compares the reconstruction performance of HTP (with several different step sizes), CGIHT, and CGHTP with parameter $\eta =0.9$. All experiments are tested in MATLAB R2014b, running on a Windows 10 machine with Intel I5-7500 CPU 3.4 GHz and 8GB RAM.

5.1. Gaussian-Signal Reconstruction

In this subsection, we construct an $M\times N$ random Gaussian matrix as the measurement matrix $\mathbf{\Phi }$ with entries drawn independently from Gaussian distribution. In addition, we generate an s-sparse signal of length $N=256$, and each nonzero element of this signal is chosen at random and drawn from standard Gaussian distribution. Here, we define the subsampling rate as $\tau =M/\phantom{MN}N$. In each experiment, we let HTP execute with four different $\alpha$: $\alpha =1$, $\alpha ={\alpha }^{\ast }-0.5$, $\alpha ={\alpha }^{\ast }$ and $\alpha ={\alpha }^{\ast }+0.5$, where ${\alpha }^{\ast }$ denotes the step size with which HTP has the optimal empirical performance. In all the tests in this section, each experiment is repeated 500 times independently. To evaluate the performance of the tested algorithms, we use the relative reconstruction error as a criteria to measure the reconstruction accuracy for sparse signals, where the relative reconstruction error is defined as ${∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2/\phantom{{∥\mathit{y}-\mathit{A}{\mathit{x}}_n∥}_2{∥\mathit{y}∥}_2}{∥\mathit{y}∥}_2$. For all algorithms tested, we set a common stopping criterion: $n>200$ or the relative reconstruction error is less than ${10}^{-6}$.

The first issue to be verified concerns the number of iterations required by the CGHTP algorithm in comparison with HTP algorithm and CGIHT algorithm. Here we choose the sampling rate $\tau =0.5$. For each $1\le s\le 80$, we compute the average number of iterations of different algorithms and plot the curve in Figure 4, where we discriminated between successful reconstructions (mostly occurring for $1<s\le 40$) and unsuccessful reconstructions (mostly occurring for $40<s\le 80$). As can be seen from Figure 4, in the successful case, the CGHTP algorithm requires fewer iterations than the CGIHT algorithm and HTP algorithm with ${\alpha }^{\ast }$(in the case ${\alpha }^{\ast }=3.5$). In the unsuccessful case, the number of iterations of CGHTP algorithm is comparable to that of HTP algorithm with ${\alpha }^{\ast }$. This is because for the same measurement matrix $\mathbf{\Phi }$, when the sparsity level is small, the space formed by the columns of the ${\mathit{A}}_{T_n}$ is less correlated, so the condition number of ${\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}$ is small, the advantage of using CG method is obvious; as the sparsity level increase, the condition number of ${\mathit{A}}_{T_n}^{\mathrm{T}}{\mathit{A}}_{T_n}$ becomes larger, which has a bad influence on the convergence rate of CGHTP. The average time consumed by different algorithms is shown in Figure 5. It can be seen that CGHTP algorithm requires the least average reconstruction time than other algorithms, because it requires fewer iterations than other algorithms.

At the same time, to verify the advantages of the proposed algorithm in convergence speed, we select a fixed sparse signal and record the relative reconstruction errors of different algorithms under different number of iterations. As can be seen from Figure 6 and Figure 7, as the number of iterations increases, the relative reconstruction errors of the signals reconstructed by HTP and CGHTP decrease, and becomes stable at a certain error value where the convergence is reached. However, the relative reconstruction error of CGIHT algorithm decreases very slowly with the increase of iteration times. Figure 6 and Figure 7 show that the number of iterations for convergence of CGHTP algorithm is smaller than that of HTP algorithm with an optimal step size and CGIHT algorithm, which reflects that CGHTP algorithm converges faster than HTP algorithm and CGIHT algorithm. The acceleration strategy chosen by the gradient descent method and the conjugate gradient method results in a better performance of the CGHTP algorithm in convergence rate.

The recovery ability of the proposed algorithm is evaluated for different sparsity levels and different sampling rates. In each experiment, it is recorded as a successful reconstruction if ${∥{\mathit{x}}_n-\mathit{x}∥}_2<{10}^{-4}{∥\mathit{x}∥}_2$. Figure 8 describes the exact reconstruction rate of different algorithms with different sparsity levels at a fixed sampling rate of $\tau =0.5$ while Figure 9 depicts the recovery performance of these algorithms at different sampling rates at a fixed sparsity level $s=10$. Figure 8 and Figure 9 show that the CGHTP is outperforms other algorithms. In addition, for the same rate of exact reconstruction, CGHTP requires fewer samples than other algorithms.

5.2. Real-World Image Reconstruction

In this subsection, we investigated the performance of CGHTP to reconstruct natural images. Test images used in experiments consist of four natural images of size 512 × 512, as shown in Figure 10a, Figure 11a, Figure 12a and Figure 13a respectively. We choose the discrete wavelet transform(DWT) matrix as the sparse transform basis and the Gaussian random matrix as the measurement matrix. The sparsity level is set to one sixth of the number of samples, i.e., $s=M/\phantom{M6}6$. In all tests, each experiment is repeated 50 times independently and we set an algorithm-stopping criterion: $n>200$ or ${∥{\mathit{x}}_n-{\mathit{x}}_{n-1}∥}_2<{10}^{-4}{∥{\mathit{x}}_n∥}_2$. The reconstruction performance of images is usually evaluated by the Peak Signal to Noise Ratio (PSNR), which is defined as:

$$\mathrm{PSN}{\mathrm{R}}_{\mathrm{dB}}=10{log}_{10}\left(\frac{{255}^2}{\mathrm{MSE}}\right),$$

(10)

where MSE denotes the mean square error, and it can be calculated by:

$$\mathrm{MSE}=\frac{1}{ab}{\displaystyle \sum _{i=1}^a}{\displaystyle \sum _{j=1}^b}{\left|\mathit{X}(i,j)-\widehat{\mathit{X}}(i,j)\right|}^2,$$

(11)

where the matrices $\mathit{X}$ and $\widehat{\mathit{X}}$ of size $a\times b$ represent the original image and the reconstructed image. Meanwhile, we also consider using relative error (abbreviated as Rerr) to evaluate the quality of image reconstruction:

$$\mathrm{Rerr}={∥\mathit{X}-\widehat{\mathit{X}}∥}_2/\phantom{{∥\mathit{X}-\widehat{\mathit{X}}∥}_2{∥\widehat{\mathit{X}}∥}_2}{∥\widehat{\mathit{X}}∥}_2.$$

(12)

Table 1. Reconstruction results of Lenna, Pens, Pepper, and Baboon images at different sampling rates by HTP, CGIHT, and CGHTP. The optimal step size ${\alpha }^{\ast }$ for reconstructing Lenna, Pens, Pepper and Baboon images by HTP algorithm are ${\alpha }^{\ast }=5.5$, ${\alpha }^{\ast }=5$, ${\alpha }^{\ast }=6$ and ${\alpha }^{\ast }=4$, respectively. The abbreviations PSNR and Rerr in the table represent the Peak Signal to Noise Ratio and relative reconstruction error of the reconstructed image, respectively.

	$\mathit{\tau }\mathbf{=}\mathbf{0.2}$
	Algorithms	HTP	HTP(${\mathit{\alpha }}^{\ast }-\mathbf{0.5}$)	HTP(${\mathit{\alpha }}^{\ast }$)	HTP(${\mathit{\alpha }}^{\ast }+\mathbf{0.5}$)	CGIHT	CGHTP
PSNR (unit:dB)	Lenna	12.814	16.783	18.471	17.516	8.584	21.708
	Pens	13.491	14.512	15.450	14.926	9.843	17.471
	Pepper	11.092	17.266	18.293	17.566	8.273	20.492
	Baboon	10.344	12.605	13.313	12.298	9.120	14.863
Rerr	Lenna	0.877	0.281	0.217	0.244	0.610	0.102
	Pens	0.810	0.431	0.330	0.382	0.532	0.225
	Pepper	0.642	0.227	0.175	0.206	0.488	0.104
	Baboon	0.932	0.644	0.528	0.632	0.874	0.222
Average Recovery Time (unit:second)	Lenna	0.153	0.216	0.223	0.237	17.632	0.216
	Pens	0.144	0.197	0.212	0.228	17.537	0.205
	Pepper	0.145	0.221	0.240	0.246	17.682	0.231
	Baboon	0.140	0.192	0.211	0.196	17.45	0.203
	$\mathit{\tau }=\mathbf{0.4}$
	Algorithms	HTP	HTP(${\mathit{\alpha }}^{\ast }-\mathbf{0.5}$)	HTP(${\mathit{\alpha }}^{\ast }$)	HTP(${\mathit{\alpha }}^{\ast }+\mathbf{0.5}$)	CGIHT	CGHTP
PSNR (unit:dB)	Lenna	26.943	28.171	28.256	28.139	20.603	28.467
	Pens	23.042	24.004	24.655	24.142	18.597	25.110
	Pepper	25.604	27.216	28.471	27.318	20.363	28.537
	Baboon	17.614	18.237	18.333	18.249	14.621	18.364
Rerr	Lenna	0.021	0.017	0.016	0.017	0.045	0.015
	Pens	0.057	0.046	0.042	0.044	0.074	0.040
	Pepper	0.029	0.022	0.021	0.022	0.048	0.021
	Baboon	0.052	0.047	0.045	0.046	0.104	0.044
Average Recovery Time (unit:second)	Lenna	0.468	0.697	0.998	3.394	23.448	0.813
	Pens	0.446	0.556	0.641	0.695	23.454	0.640
	Pepper	0.459	0.725	1.110	4.321	23.637	0.812
	Baboon	0.409	0.556	0.553	0.599	21.726	0.531
	$\mathit{\tau }=\mathbf{0.6}$
	Algorithms	HTP	HTP(${\mathit{\alpha }}^{\ast }-\mathbf{0.5}$)	HTP(${\mathit{\alpha }}^{\ast }$)	HTP(${\mathit{\alpha }}^{\ast }+\mathbf{0.5}$)	CGIHT	CGHTP
PSNR (unit:dB)	Lenna	29.924	31.133	31.164	31.145	27.376	31.392
	Pens	26.566	27.957	28.319	28.273	24.331	28.633
	Pepper	29.207	30.867	31.028	30.891	27.363	31.297
	Baboon	19.527	19.544	19.584	19.496	17.351	20.257
Rerr	Lenna	0.013	0.010	0.010	0.010	0.019	0.009
	Pens	0.032	0.024	0.023	0.024	0.045	0.022
	Pepper	0.016	0.011	0.010	0.011	0.023	0.009
	Baboon	0.040	0.039	0.038	0.039	0.048	0.038
Average Recovery Time (unit:second)	Lenna	1.451	22.223	54.121	61.064	102.059	3.064
	Pens	1.436	1.823	2.980	19.836	109.281	3.186
	Pepper	1.559	24.761	56.593	62.299	94.708	3.140
	Baboon	1.342	1.753	1.896	2.135	102.794	1.814

shows the average reconstruction PSNR, Rerr, and reconstruction time of four reconstructed images by different algorithms at different sampling rates. Figure 9, Figure 10, Figure 11 and Figure 12 visually reflects the contrast between the original image and the reconstructed image, where subfigure (b–d) represents the reconstructed images obtained by HTP ($\alpha ={\alpha }^{\ast }$), CGIHT, and CGHTP, respectively.

It can be seen from Table 1 that the reconstruction performance of different algorithms is different for four tested images. In terms of reconstruction PSNR value, the lower the sampling rate, the more obvious the advantage of CGHTP algorithm over HTP algorithm and CGIHT algorithm. Especially in the case of low sampling rate (T = 0.2), the reconstructed PSNR value of the four tested images obtained by CGHTP algorithm is about 2 dB higher than that obtained by HTP algorithm with the best step size. Similarly, the reconstruction relative error of CGHTP is slightly lower than that of other algorithms. In addition, the average reconstruction time of CGHTP is comparable to that of HTP algorithm with an optimal step size. Due to the calculation of the step size $\alpha$ and the orthogonal factor $\beta$, the CGHTP algorithm requires slightly more computation than the HTP algorithm by approximately $O\left(Ns\right)$. However, in each iteration, the conjugate direction may be used as the search direction in CGHTP to accelerate the convergence rate, so they need fewer iterations than the HTP algorithm. Overall, CGHTP has an empirical performance comparable to the HTP algorithm with an optimal step size in natural image reconstruction.

6. Conclusions

In this paper, we introduce a new greedy iterative algorithm, termed CGHTP. Using conjugate gradient method to accelerate the convergence of the original HTP algorithm is one of the strong features of the algorithm, which makes the CGHTP algorithm faster than the HTP algorithm. Reconstruction experiments with Gaussian sparse signals and natural images illustrate the advantages of the proposed algorithm in convergence rate and reconstruction performance, especially in the case of low sparsity, CGHTP requires fewer iterations than HTP with the best step size and CGIHT algorithm. Although the proposed algorithm has better reconstruction performance, its theoretical guarantee is not very strong, our future works may focus on the research about stronger theoretical guarantee of this algorithm. In addition, we will study how to apply this method to more practical applications.