Stable Sparse Model with Non-Tight Frame

Abstract

Overcomplete representation is attracting interest in image restoration due to its potential to generate sparse representations of signals. However, the problem of seeking sparse representation must be unstable in the presence of noise. Restricted Isometry Property (RIP), playing a crucial role in providing stable sparse representation, has been ignored in the existing sparse models as it is hard to integrate into the conventional sparse models as a regularizer. In this paper, we propose a stable sparse model with non-tight frame (SSM-NTF) via applying the corresponding frame condition to approximate RIP. Our SSM-NTF model takes into account the advantage of the traditional sparse model, and meanwhile contains RIP and closed-form expression of sparse coefficients which ensure stable recovery. Moreover, benefitting from the pair-wise of the non-tight frame (the original frame and its dual frame), our SSM-NTF model combines a synthesis sparse system and an analysis sparse system. By enforcing the frame bounds and applying a second-order truncated series to approximate the inverse frame operator, we formulate a dictionary pair (frame pair) learning model along with a two-phase iterative algorithm. Extensive experimental results on image restoration tasks such as denoising, super resolution and inpainting show that our proposed SSM-NTF achieves superior recovery performance in terms of both subjective and objective quality.

1. Introduction

Sparse representation of signals in dictionary domains has been widely studied and has provided promising performance in numerous signal processing tasks such as image denoising [1,2,3,4,5], super resolution [6,7,8], inpainting [9,10] and compression [11,12]. It is well known that images are represented by a linear combination of certain atoms of a dictionary. Overcomplete sparse representation is the overcomplete system with a sparse constraint. Common overcomplete systems differ from the traditional bases, such as DCT, DFT and Wavelet, because they offer a wider range of generating elements; potentially, this wider range allows more flexibility and effectiveness in signal sparse representation. However, it is a severely under-constrained illposed problem to find the underlying overcomplete representation due to the redundancy of the systems. When the underlying representation is sparse and the overcomplete systems have stable properties, the ill-posedness will disappear [13]. Sparse models are generally classified into two categories: Synthesis sparse models and analysis sparse model [14]. The commonly referred to sparse models are synthesis sparse models. The analysis ones characterize the signal by multiplying it with an analysis overcomplete dictionary, leading to a sparse outcome. A variety of effective sparse models have been investigated and established such as the classical synthesis sparse model [9,15], the classical analysis sparse model [14], the nonlocal sparse model [16,17] and the 2D sparse model [18]. Unfortunately, these models ignore the stability recovery property which claims that once a sufficient sparse solution is found, all alternative solutions necessarily reside very close to it [9]. Recently, the stable recovery of sparse representation has drawn attention in signal processing theory. Generally speaking, stable recovery can be guaranteed by two properties: Sufficient sparsity and a favorable structure of the dictionary [19]. Donoho defines the concept of mutual incoherence of the dictionary and applies it to prove some possibility of stable recovery [19]. The authors of [20] proposea sparsity-based orthogonal dictionary learning method to minimize the mutual incoherence. The authors of [21] propose an incoherent dictionary learning scheme by integrating a low rank gram matrix of the dictionary into the dictionary learning model.

A more powerful stable recovery guarantee developed by Candes and Tao, termed Restricted Isometry Property (RIP), makes consequent analysis easy [22]. A matrix $\mathsf{\Phi }$ is said to satisfy the RIP of order k if there exists a constant ${\delta }_k\in (0,1)$ such that

$$(1-{\delta }_k){\parallel \mathbf{y}\parallel }_2^2\le {\parallel \mathsf{\Phi }\mathbf{y}\parallel }_2^2\le (1+{\delta }_k){\parallel \mathbf{y}\parallel }_2^2$$

(1)

holds for all k-sparse vectors $\mathbf{y}$. ${\delta }_k$ is defined as the smallest constant which satisfies the above inequalities and is called the restricted isometry constant of $\mathsf{\Phi }$.

Most RIP research substantially investigates applying RIP as a stablility analysis instrument [17,23,24] or finding optimal RIP constant [25,26] which are all theoretical analyses rather than practical applications. According to the research of [21], the intrinsic property of a dictionary has a direct influence on its performance. All familiar algorithms are staggeringly unstable with a coherent or degenerate dictionary [19]. Recognizing the gap between theoretical analyses and practical applications of RIP, this paper aims to build a stable sparse model satisfying the RIP.

Recently, the frame as a stable overcomplete system has drawn some attention in signal processing as the given signal can be represented by its canonical expansion in a manner similar to conventional bases under the frame. Some data-driven approaches are proposed in [1,27,28,29,30]. The authors of [27,29,30] utilize redundant tight frame in compressed sensing and [28] applies tight frame to few-view image reconstruction. Study [1] presents a data-driven method that the dictionary atoms associated with the tight frame are generated by filters. These approaches achieve much better image processing performance than previous methods, and meanwhile the tight frame condition which requires the frame almost-orthogonality will limit the flexibility in sparse representation. Study [31] derives stable recovery result for $l_1$-analysis minimization in redundant, possibly non-tight frames. Inspired by this result and the relationship between RIP and frame, we aim to establish a stable sparse model with RIP based on non-tight frame.

We call a sequence ${\left\{{\mathit{\varphi }}_i\right\}}_{i=1}^M\in \mathbf{H}$ a frame if and only if there exist two positive numbers A and B such that

$${A\parallel \mathbf{x}\parallel }_2^2\le \sum _{i=1}^M|<\mathbf{x},{\mathit{\varphi }}_i{>|}^2\le B{\parallel \mathbf{x}\parallel }_2^2\forall \mathbf{x}\in {\mathbf{H}}^N$$

(2)

Here, A and B are called the bounds of the frame. We find that every submatrix ${\mathsf{\Phi }}_k$ satisfied RIP is a non-tight frame with $(1-{\delta }_k)$ and $(1+{\delta }_k)$ as its frame bounds with a given k. Obviously, there is an essential connection between the non-tight frame and the RIP.

In this paper we focus on a stable sparse model and more specifically on the development of an algorithm that would learn a pair of non-tight frame based dictionaries from a set of signal examples. We propose a stable sparse model via applying the non-tight frame condition to approximate the RIP. This model shares the favorite overcomplete structures with the common sparse models, and meanwhile it contains RIP and closed-form sparse coefficient expression which ensure stable recovery. Recognizing that the optimal framebounds are essentially the maximum and minimum singular values of the frame, RIP is actually enforced on the dictionary pair (the frame and its dual frame) by constraining the singular values of them. We also formulate a dictionary pair learning model via applying the second-order truncated Taylor series to approximate the inverse frame operator. Then we present an efficient algorithm to learn the dictionary pair via a two-phase iterative approach. To summarize, this paper makes the following contributions:

• We propose a stable sparse model along with a dictionary pair learning model. Non-tight frame condition is utilized to develop a relaxation of RIP to guarantee stable recovery of sparse representation. Moreover, the sparse coefficients are also modeled, which leads to a more stable recovery especially for seriously noisy image.
• It is nearly impossible to solve the dictionary pair learning model in a straightforward way since the inverse frame operator is involved. We provide an effective way to modify the model via applying a second-order truncated Taylor series to approximate the inverse frame operator, and provide an efficient algorithm for the modified one.
• We present the stability analysis of the proposed model and demonstrate it on natural and synthetic image denoising, super resolution and image inpainting. The denoising results show that the proposed approach outperforms synthesis models such as the KSVD and the data-driven tight frame based methods for natural image case in terms of average PSNR. Moreover, it also gains comparable performance to the Analysis KSVD for a piecewise-constant (PWC) image in terms of average PSNR. The meaningful structures in the trained dictionary pairs for natural images and a PWC image are observed. The super resolution results show that the SSM-NTF produces better performance than the Bicubic interpolation method and the method in [32]. The inpainting results show that our model is able to eliminate text of fonts completely.

This paper is organized as follows: Section 2 reviews the related work on frame, synthesis sparse model and analysis sparse model. Section 3 presents our stable sparse model with non-tight frame SSM-NTF along with a dictionary pair learning model. Section 4 proposes the corresponding dictionary pair learning algorithm. Section 5 proposes the image restoration method of our proposed SSM-NTF model. In Section 6 we analyze the computational complexity of our proposed algorithm. In Section 7, we demonstrate the the effectiveness of our SSM-NTF model by analyzing the convergence of the corresponding algorithm, denoising natural and piecewise constant images, super resolution and image inpainting. Finally, Section 8 concludes this paper.

2. Related Work

In this section, we briefly review the related work on frame, synthesis sparse model and analysis sparse model.

Frame: A frame $\mathsf{\Phi }$ is called a tight frame if the frame bounds are equal in the Equation (2) [32]. There are two associated operators can be defined between the Hilbert space ${\mathbf{H}}^N$ and Square Integrable Space ${\mathbf{l}}_2^M(·)$ once a frame is defined. One is the analysis operator $\mathbf{T}$ defined by

$${\left(\mathbf{T}\mathbf{x}\right)}_i=<\mathbf{x},{\mathit{\varphi }}_i>,\forall \mathbf{x}\in {\mathbf{H}}^N$$

(3)

and the other is its adjoint operator ${\mathbf{T}}^{*}$ which is called the synthesis operator

$$\begin{array}{c}\hfill {\mathbf{T}}^{*}\mathbf{c}=\sum _{i=1}^M\mathbf{c}{\mathit{\varphi }}_i\forall \mathbf{c}={\left({\mathbf{c}}_i\right)}_{i\in J}\in {\mathbf{l}}_2^M\left(\mathbf{T}\right)\end{array}$$

(4)

then, the frame operator can be defined as the following canonical expansion

$$\begin{array}{c}\hfill \mathbf{Fx}={\mathbf{T}}^{*}\mathbf{T}\mathbf{x}=\sum _{i=1}^M<\mathbf{x},{\mathit{\varphi }}_i>{\mathit{\varphi }}_i\end{array}$$

(5)

In Euclidean space, a given frame $\mathsf{\Phi }$ can be represent in manner of matrix with its columns of it as the frame elements. Then one of its adjoint operator can be representated as ${\mathit{\psi }}_i={\mathbf{F}}^{-1}{\mathit{\varphi }}_i$ [32]. Let $\mathbf{x}\in {\mathbb{R}}^N$ be an arbitrary vector, a reconstruction function can be expressed as the following form

$$\begin{array}{c}\hfill \mathbf{x}=\sum _{i=1}^M<\mathbf{x},{\mathit{\psi }}_i>{\mathit{\varphi }}_i\end{array}$$

(6)

Synthesis sparse model: The conventional synthesis sparse model represents a vector $\mathbf{x}$ by the linear combination of a few atoms from a large dictionary $\mathsf{\Phi }$, denoted as $\mathbf{x}={\mathsf{\Phi }\mathbf{y},\parallel \mathbf{y}\parallel }_0\le L$, where L is the sparsity of $\mathbf{y}$. The computational techniques for approximating sparse coefficient $\mathbf{y}$ under a given dictionary $\mathsf{\Phi }$ and $\mathbf{x}$ includes greedy pursuit (e.g., OMP [9]) and convex relaxation optimization, such as Lasso [33] and FISTA [8]. In order to improve the performance of sparse representation, some modified models such as the nonlocal sparse model [16], the frame based sparse model [21], and the MD sparse model [18] are also investigated.

Analysis sparse model: The analysis sparse model is defined as: $\mathbf{y}={\mathsf{\Omega }\mathbf{x},\parallel \mathbf{y}\parallel }_0=p-l$ where $\mathsf{\Omega }\in {\mathcal{R}}^{p\times d}$ is a linear operator (also called as a dictionary), and l denotes the co-sparsity of the signal $\mathbf{x}$. The analysis representative vector $\mathbf{y}$ is sparse with l zeros. The zeros in $\mathbf{y}$ denote the low-dimensional subspace to which the signal $\mathbf{x}$ belongs. The analysis sparse coding [14] and dictionary learning [34] approach are also been proposed.

However, all these models ignore the stability recovery property which provides stable reconstruction of the signals in presence of noise.

Dictionary learning methods: The dictionaries include analytical dictionaries, such as DCT, DWT, curvelets and contourlets and learned dictionaries. Some dictionary learning method are proposed, such as the classical KSVD [9] algorithm, the efficient sparse coding which convert the original dictionary learning problem to two least squares problem by applying the Lagrange dual [3], the non-local sparse model [16] which learns a set of PCA sub-dictionaries by cluster the samples into K clusters using image nonlocal self-similarity prior and its improved version which using the $l_q$-norm to instead the $l_2$-norm in order to handle different image contents. With the realization of stability, some mutual-coherence based methods are proposed. In [20] a sparsity-based orthogonal dictionary learning method is proposed to minimize the mutual-coherence of the dictionary. The authors of [21] propose an in coherent dictionary learning scheme by integrating a low rank gram matrix of the dictionary into the dictionary learning model. However, these methods only concern the capability of the dictionary without modeling the sparse coefficients which still has some probability of instability.

3. The Proposed SSM-NTF

In this section, we present the stable sparse model with non-tight frame, (Section 3.1), the stability analysis of the proposed model, (Section 3.2) and the dictionary pair (the frame pair) learning model, (Section 3.3).

3.1. Stable Sparse Model with Non-Tight Frame

In this section, we derive our stable sparse model with non-tight frame where the non-tight frame condition serves as an approximation to the RIP.

According to [35], a k-th RIP constant can be express as

$${\delta }_k\left(\mathsf{\Phi }\right)=\frac{{\mathrm{\Gamma }}_k\left(\mathsf{\Phi }\right)-1}{{\mathrm{\Gamma }}_k\left(\mathsf{\Phi }\right)+1}$$

(7)

where

$${\mathrm{\Gamma }}_k\left(\mathsf{\Phi }\right)=\frac{{\theta }_{max}^k}{{\theta }_{min}^k}$$

(8)

$$\begin{array}{c}\hfill {\theta }_{max}^k=\underset{{\parallel \mathbf{y}\parallel }_0=k}{max}\frac{{\parallel \mathsf{\Phi }\mathbf{y}\parallel }_2^2}{{\parallel \mathbf{y}\parallel }_2^2},{\theta }_{min}^k=\underset{{\parallel \mathbf{y}\parallel }_0=k}{min}\frac{{\parallel \mathsf{\Phi }\mathbf{y}\parallel }_2^2}{{\parallel \mathbf{y}\parallel }_2^2}\end{array}$$

(9)

The Equation (7) provides a new perspective in integrating the RIP to sparse model via applying ${\theta }_{max}^k$ and ${\theta }_{min}^k$ instead of the RIP constant ${\delta }_k\left(\mathsf{\Phi }\right)$. The difficulty in building a stable sparse model decreases. However, the sparsity k varies with the noise level, and also, in a feasible numerical calculation method, it is impossible to sweep through all the samples satisfying ${\parallel \mathbf{x}\parallel }_0=k$ to pursue an unknown dictionary $\mathsf{\Phi }$.

Let $\mathbf{x}$ be a signal vector, the frame reconstruction function can be formulated as $\mathbf{x}=\mathsf{\Phi }{\mathsf{\Psi }}^T\mathbf{x}$ where $\mathsf{\Psi }$ is a dual frame of $\mathsf{\Phi }$. Adding a reasonable sparsity prior to the signal $\mathbf{x}$ over $\mathsf{\Psi }$ domain, we can derive

$$\frac{\parallel \mathsf{\Phi }\left({\mathsf{\Psi }}^T\mathbf{x}\right){\parallel }_2^2}{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}=\frac{{\parallel \mathbf{x}\parallel }_2^2}{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}=\frac{1}{\frac{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}{{\parallel \mathbf{x}\parallel }_2^2}}$$

(10)

Denoting the optimal frame bounds of $\mathsf{\Phi }$ as A and B, the frame condition of $\mathsf{\Psi }$ can be formulated as $\frac{1}{B}\le \frac{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}{{\parallel \mathbf{x}\parallel }_2^2}\le \frac{1}{A}$. Then a pair of bounds for Equation (10) can be obtained as $A\le \frac{\parallel \mathsf{\Phi }\left({\mathsf{\Psi }}^T\mathbf{x}\right){\parallel }_2^2}{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}\le B$. A formula similar to Equation (9) is derived as $B=\underset{\mathbf{x}\in \mathcal{J}}{max}\frac{\parallel \mathsf{\Phi }\left({\mathsf{\Psi }}^T\mathbf{x}\right){\parallel }_2^2}{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}$, $A=\underset{\mathbf{x}\in \mathcal{J}}{min}\frac{\parallel \mathsf{\Phi }\left({\mathsf{\Psi }}^T\mathbf{x}\right){\parallel }_2^2}{\parallel {\mathsf{\Psi }}^T{\mathbf{x}\parallel }_2^2}$ where $\mathcal{J}$ is the data set. Imitating Equation (7), we can obtain a RIP-like constant expression

$$\widehat{\delta }\left(\mathsf{\Phi }\right)=\frac{\widehat{\mathrm{\Gamma }}\left(\mathsf{\Phi }\right)-1}{\widehat{\mathrm{\Gamma }}\left(\mathsf{\Phi }\right)+1}$$

(11)

where $\widehat{\mathrm{\Gamma }}\left(\mathsf{\Phi }\right)=\frac{B}{A}$. Obviously, $\widehat{\delta }\left(\mathsf{\Phi }\right)$ can be regarded as an approximation of the RIP constant which benefits the computation due to the ignorance on sparsity degree. In a word, the RIP constraint can be satisfied by constraining the frame bounds. Thus, a stable overcomplete system with a sparsity prior can be established.

Now we discuss the characteristic of the frame bounds A and B. The Frame Condition (2) has a more compact form $\sqrt{A}\le {\eta }_{\mathsf{\Phi }}\le \sqrt{B}$ where ${\eta }_{\mathsf{\Phi }}$ denotes any singular value of $\mathsf{\Phi }$. More specifically, $\sqrt{A}={\eta }_{min}$, $\sqrt{B}={\eta }_{max}$ where ${\eta }_{max}$ and ${\eta }_{min}$ denote the maximum and minimum singular values of $\mathsf{\Phi }$, respectively. Then, we can obtain ${\eta }_{max}\ge {\theta }_{max}^k,{\eta }_{min}\le {\theta }_{min}^k$. It is easy to know that $\widehat{\delta }\left(\mathsf{\Phi }\right)\ge {\delta }_k\left(\mathsf{\Phi }\right)$. Obviously, $\widehat{\delta }\left(\mathsf{\Phi }\right)$ is a reasonable relaxation of ${\delta }_k\left(\mathsf{\Phi }\right)$ as $\widehat{\delta }\left(\mathsf{\Phi }\right)$ is slightly exceed ${\delta }_k\left(\mathsf{\Phi }\right)$ but resides very close to it as long as the data is not seriously degraded. Therefore, the RIP constraint can be enforced on the frames by limiting the maximum and minimum singular values.

In this paper, we integrate non-frame to traditional sparse model to establish a stable sparse model with RIP. Let $\mathbf{x}$ be a signal vector. Under the assumption of the sparsity prior of ${\mathsf{\Psi }}^T\mathbf{x}$, we apply a soft thresholding operator ${\mathcal{S}}_{\lambda }(·)$ (which shall be defined in the next subsection) on it such that

$$\mathbf{x}=\mathsf{\Phi }{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{x}\right)$$

(12)

where $\lambda$ is a vector with elements ${\lambda }_i$ corresponding to ${\mathit{\psi }}_i$, $i=1,2,\dots ,M$. Therefore, we propose the stable sparse model with non-tight frame (SSM-NTF) as follows

$$\begin{array}{cc}\hfill & \mathbf{y}={\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{x}\right),\mathbf{x}=\mathsf{\Phi }\mathbf{y},\hfill \\ \hfill & {\mathrm{s}.\mathrm{t}.\parallel \mathbf{y}\parallel }_0\le s.\hfill \end{array}$$

(13)

Here, the correlation between the frame $\mathsf{\Phi }$ and its dual frame $\mathsf{\Psi }$ is formulated as $\mathsf{\Psi }={\mathbf{F}}^{-1}\mathsf{\Phi }$. The frame operator $\mathbf{F}$ is formulated as $\mathsf{\Phi }{\mathsf{\Phi }}^T$ which is indeed a gram matrix of $\mathsf{\Phi }$. The singular values of $\mathsf{\Phi }$ are constrained by $\sqrt{A}\le {\eta }_{\mathsf{\Phi }}\le \sqrt{B}$ to satisfy the RIP. Actually, by constraining the singular values of $\mathsf{\Phi }$, the elements of the gram matrix are also bounded which meets the theory of mutual coherence.

In order to be consistent with the traditional sparse models, we refer to the frame $\mathsf{\Phi }$ and its dual frame $\mathsf{\Psi }$ as dictionary and its dual dictionary.

3.2. The Stability Analysis of the Proposed Model

In sparse representation problem, a given noiseless signal $\mathbf{x}$, can be formulated formulated as

$$\left({\mathit{P}}_0\right):\underset{\mathbf{x}}{min}{\parallel \mathbf{y}\parallel }_{0}s.t.\mathbf{x}=\mathsf{\Phi }\mathbf{y}$$

(14)

where $\mathsf{\Phi }$ is the sparse representation dictionary and $\mathbf{y}$ is the sparse coefficients. While $\mathbf{x}=\mathsf{\Phi }\mathbf{y}$ is an underdetermined linear system, the problem $\left({\mathit{P}}_0\right)$ has the unique solution ${\mathbf{y}}_0$ as soon as it satisfies the uniqueness property which is formulated as

$$\parallel \widehat{\mathbf{y}}{\parallel }_0<\frac{1}{2}(1+1/\mu )$$

(15)

where $\mathsf{\mu }$ is the mutual-coherence of $\mathsf{\Phi }$ [9]. However, the signals are usually acquired with noise, then the problem $\left({\mathit{P}}_0\right)$ should be relaxed to the problem $\left({\mathit{P}}_{ϵ}\right)$ which is expressed as

$$\left({\mathit{P}}_{ϵ}\right):\underset{\mathbf{x}}{min}{\parallel \mathbf{y}\parallel }_{0}s.t.{\parallel \mathbf{x}-\mathsf{\Phi }\mathbf{y}\parallel }_F^2\le ϵ$$

(16)

where $ϵ$ is an error-tolerant which exists due to the noise. The problem $\left({\mathit{P}}_{ϵ}\right)$ will no longer maintain the uniqueness of solution as $\mathbf{x}=\mathsf{\Phi }\mathbf{y}+ϵ$ is an inequality system. Thus, the notion of Uniqueness Property (15) is replaced by the notion of stability which claims that all the alternative solutions reside very close to the ideal solution. Under the stable guarantee, we can yet ensure that the recovery results of our methods produce meaningful solutions. Assume that ${\mathbf{y}}_0$ is the ideal solution to the problem $\left({\mathit{P}}_{ϵ}\right)$ and $\widehat{\mathbf{y}}$ is the candidate one, the traditional sparse model has a stability claim of the form [9]

$$\parallel \widehat{\mathbf{y}}-{\mathbf{y}}_0{\parallel }_2^2\le \frac{4{ϵ}^2}{1-(2s_0-1)\mu },$$

(17)

where $\mathsf{\mu }$ is the mutual coherence which is formulated as $\mu ={max}_{i\ne j}|<{\mathit{\varphi }}_i,{\mathit{\varphi }}_j>|,i,j=1,2,\cdots ,M$. Apparently, the error bound of Equation (17) can only be determined with given sparsity $s_0$ and the mutual coherence $\mathsf{\mu }$. However, the mutual coherence of an unknown dictionary is very difficult to calculate which lead to a result that we can not ensure the stability in the dictionary learning case. In contrast, we derive a similar stability claim of our proposed SSM-NTF model.

Defining $\mathbf{d}=\widehat{\mathbf{y}}-{\mathbf{y}}_0$ with ${\mathbf{y}}_0$ as the ideal solution to the model, we have that $\parallel \mathsf{\Phi }\widehat{\mathbf{y}}-\mathsf{\Phi }{\mathbf{y}}_0{\parallel }_2={\parallel \mathsf{\Phi }\mathbf{d}\parallel }_2\le 2ϵ$. From the previous subsection, we have know that the frame $\mathsf{\Phi }$ satisfies the RIP with the corresponding parameter $\widehat{\delta }\left(\mathsf{\Phi }\right)$. Thus, using this property and exploiting the lower-bound part in Equation (1), we get

$$(1-\widehat{\delta }\left(\mathsf{\Phi }\right)){\parallel \mathbf{d}\parallel }_2^2\le {\parallel \mathsf{\Phi }\mathbf{d}\parallel }_2^2\le 4{ϵ}^2$$

(18)

where $\widehat{\delta }\left(\mathsf{\Phi }\right)=\frac{\widehat{\mathrm{\Gamma }}\left(\mathsf{\Phi }\right)-1}{\widehat{\mathrm{\Gamma }}\left(\mathsf{\Phi }\right)+1}=\frac{\frac{B}{A}-1}{\frac{B}{A}+1}$. Thus, we get a stability claim of the form

$${\parallel \mathbf{d}\parallel }_2^2={\parallel \widehat{\mathbf{y}}-{\mathbf{y}}_0\parallel }_2^2\le \frac{4{ϵ}^2}{1-\frac{\frac{B}{A}-1}{\frac{B}{A}+1}}$$

(19)

Obviously, the error bound of the SSM-NTF is determined by $\frac{B}{A}$, the ratio of the upper bound to the lower bound of the frame, rather than the specific values of A and B. Thus, for the convenience of numerical experiments, we usually set A to a fixed value. A main advantage of standard orthogonal transformations is that they maintain the energy of the signals in the transform domain as its frame bounds A and B are equal to 1. However, the standard orthogonal basis is non-redundant that limits its performance in sparse representation. In order to make a trade off between the represent accuracy and the degree of redundant, we usually set the lower frame bound A to a value a little smaller than 1 but not over-small as A is the minimum singular value of $\mathsf{\Phi }$ which determines the condition number of $\mathsf{\Phi }$. Thus, once the tolerance error is given, the value of B can be easily calculated. Further, a pair of dictionaries conform to the given error can be obtained using the proposed SSM-NTF model. On the other hand, if the value of B is given by experience, the error bound of our model can be measured.

3.3. Learning Model of Dictionary Pair

Assuming $\mathbf{X}\in {\mathbb{R}}^{N\times L}$ is the training data with signal vectors ${\mathbf{x}}_i\in {\mathbb{R}}^N,i=1,2,\dots ,L$, as its columns. The dictionary pair learning model can be written as

$$\begin{array}{cc}\hfill & \underset{\mathsf{\Phi },\mathsf{\Psi },\lambda ,\mathbf{Y}}{min}{\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{Y}\parallel }_F^2+{\gamma }_1\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right){\parallel }_F^2+{\gamma }_2{\parallel \mathbf{Y}\parallel }_0+{\gamma }_3{\parallel \mathsf{\Psi }-{\mathbf{F}}^{-1}\mathsf{\Phi }\parallel }_F^2\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\sqrt{A}\le {\eta }_{\mathsf{\Phi }}\le \sqrt{B}\hfill \end{array}$$

(20)

However, the Problem (20) is difficult to solve. First, the inverse of the frame operator $\mathbf{F}$ has no closed-form explicit expression. Secondly, the thresholding operator is a highly nonlinear operator which makes the optimization with respect to $\lambda$ hard to optimize.

Apparently, the Problem (20) is difficult to solve as the existence of the inverse of $\mathbf{F}$. Fortunately, the matrix ${\mathbf{F}}^{-1}$ can be expressed as a convergent series [36] which is formulated as

$${\mathbf{F}}^{-1}=\frac{2}{A+B}\sum _{k=0}^{\infty }{(\mathbf{I}-\frac{2\mathbf{F}}{A+B})}^k$$

(21)

Here, we truncated the series at $k=1$ to make a tradeoff between computational complexity and approximation accuracy. It is formulated as

$${\mathbf{F}}^{-1}\approx \frac{2}{A+B}+\frac{2}{A+B}(\mathbf{I}-\frac{2\mathbf{F}}{A+B})=\frac{2}{A+B}(2\mathbf{I}-\frac{2\mathbf{F}}{A+B})$$

(22)

In this way, once the frame bounds are given, the inverse of $\mathbf{F}$ can be calculated easily. Then the optimization problem for training RIP-dictionary pair is formulated as

$$\begin{array}{cc}\hfill & \underset{\mathsf{\Phi },\mathsf{\Psi },\lambda ,\mathbf{Y}}{min}{\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{Y}\parallel }_F^2+{\gamma }_1\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right){\parallel }_F^2+{\gamma }_2{\parallel \mathbf{Y}\parallel }_0+{\gamma }_3{\parallel \mathsf{\Psi }-\frac{2}{A+B}(2\mathbf{I}-\frac{2\mathbf{F}}{A+B})\mathsf{\Phi }\parallel }_F^2\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\sqrt{A}\le {\sigma }_{\mathsf{\Phi }}\le \sqrt{B}\hfill \end{array}$$

(23)

where ${\mathcal{S}}_{\lambda }(·)$ is the elementwise thresholding operator. There are two basic thresholding methods: The hard thresholding method whose thresholding operator defines as ${\mathcal{S}}_{\lambda }(·)\to max\left(|·|-\lambda ,0\right)$ and the soft thresholding whose operator is defined as ${\mathcal{S}}_{\lambda }(·)\to sgn(·)max\left(|·|-\lambda ,0\right)$. Both of the two operator are are non-convex and highly discontinuous which lead to big challenges to solve Problem (23). The mean reason is the fact that the update of the thresholding values $\lambda$ causing non-smooth changes to the cost function. To solve this difficulty, we design an alternative direction method via global search and least square that will be introduce in Section 4.1.

4. Dictionary Pair Learning Algorithm

In this subsection, we propose the two-phase iterative algorithm for dictionary pair learning by dividing Problem (23) into two subproblems: The sparse coding phase which updates the sparse coefficients $\mathbf{Y}$ and thresholding values $\lambda$, and the dictionary pair update phase which computes $\mathsf{\Phi }$ and $\mathsf{\Psi }$.

4.1. Sparse coding phase

In this subsection, we discuss how to calculate the sparse coefficients $\mathbf{Y}$ and the threshold values $\lambda$ with given $\mathsf{\Phi }$ and $\mathsf{\Psi }$ under our SSM-NTF model.

Given a pair of dictionaries $\mathsf{\Phi }$ and $\mathsf{\Psi }$, calculating $\mathbf{Y}$ and $\lambda$ from $\mathbf{X}$ is formulated as:

$$\{\widehat{\mathbf{Y}},\widehat{\lambda }\}=\underset{\mathbf{Y},\lambda }{min}{\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{Y}\parallel }_F^2+{\gamma }_1\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right){\parallel }_F^2+{\gamma }_2{\parallel \mathbf{Y}\parallel }_0$$

(24)

We pursue the two variables alternatively. Firstly, with fixed $\lambda$, we obtain the sparse coefficients $\mathbf{Y}$ by solving Problem (24) through OMP [9] as it can be easily convert to the classical synthesis sparse expression ${min\parallel \mathbf{Z}-\mathbf{DY}\parallel }_F^2,s.t.{\parallel \mathbf{Y}\parallel }_0$ where $\mathbf{Z}=\left[\mathbf{X}{\sqrt{\gamma }}_1{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right)\right]$ and $\mathbf{D}=\left[\mathsf{\Phi }{\sqrt{\gamma }}_1\mathbf{I}\right]$.

Secondly, the pursue of $\lambda$ is equivalent to solving the following problem

$$\widehat{\lambda }=\underset{\lambda }{argmin}{\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right)\parallel }_F^2$$

(25)

which can be decomposed into M individual optimization problems

$${\widehat{\lambda }}_i=\underset{{\lambda }_i}{argmin}{\parallel {\overline{\mathbf{y}}}_i-{\mathcal{S}}_{{\lambda }_i}\left({\mathit{\psi }}_i^T\mathbf{X}\right)\parallel }_2^2,i=1,\dots ,M.$$

(26)

where ${\mathit{\psi }}_i$ is the column of $\mathsf{\Psi }$. From the definition of soft thresholding operator, we can know that the function of Problem (26) is discrete. By denoting the data indices set that remains intact after the thresholding as ${\mathcal{J}}_i$, we split the data $\mathbf{X}$ into two parts: ${\mathbf{X}}^{{\mathcal{J}}_i}$ and ${\mathbf{X}}^{{\widehat{\mathcal{J}}}_i}$ such that

$$\begin{array}{cc}\hfill {\mathcal{S}}_{{\lambda }_i}\left({\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}\right)& ={\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}-sgn\left({\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}\right){\lambda }_i\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\mathcal{S}}_{{\lambda }_i}\left({\mathit{\psi }}_i^T{\mathbf{X}}^{{\widehat{\mathcal{J}}}_i}\right)& =0.\hfill \end{array}$$

(28)

where ${\widehat{\mathcal{J}}}_i$ is a supplementary to the intact indices ${\mathcal{J}}_i$ which turn the all elements to zero. It is clear to know that the variables ${\mathcal{J}}_i$ and ${\widehat{\mathcal{J}}}_i$ are both functions of ${\lambda }_i$ without explicit expressions which leads to a large challenge in optimization.

In order to solve Problem (26), an intermediate variable ${\mu }_i$ is necessarily to introduced to separate the whole problem into two parts: The update of the indices ${\mathcal{J}}_i$ and ${\widehat{\mathcal{J}}}_i$ (determined by ${\mu }_i$) and the update of the explicit thresholding value ${\lambda }_i$. Then Problem (26) can be transformed to another optimization problem:

$$\begin{array}{cc}\hfill & \{{\widehat{\lambda }}_i,{\widehat{\mu }}_i\}=\underset{{\lambda }_i}{argmin}\parallel {\overline{\mathbf{y}}}_i^{{\widehat{\mathcal{J}}}_i}{\parallel }_2^2+\parallel {\overline{\mathbf{y}}}^{{\mathcal{J}}_i}-[{\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}-sgn\left({\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}\right){\lambda }_i]{\parallel }_2^2+\tau /2{\parallel {\lambda }_i-{\mu }_i\parallel }_2^2\hfill \end{array}$$

(29)

where ${\mathcal{J}}_i$ and ${\widehat{\mathcal{J}}}_i$ are two functions of the intermediate variable ${\mu }_i$.

At the k-th step, to obtain ${\mu }_i$, we solve Problem (29) with ${\lambda }_i$ fixed and denote the functions as $f\left({\mu }_i\right)+g\left({\mu }_i\right)+l\left({\mu }_i\right)$ where $f\left({\mu }_i\right)={\parallel {\overline{\mathbf{y}}}_i^{{\widehat{\mathcal{J}}}_i}\parallel }_2^2$, $g\left({\mu }_i\right)={\parallel {\overline{\mathbf{y}}}^{{\mathcal{J}}_i}-[{\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}-sgn\left({\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}\right){\lambda }_i]\parallel }_2^2$, $l\left({\mu }_i\right)=\tau /2{\parallel {\lambda }_i-{\mu }_i\parallel }_2^2$. Optimizing this expression is obviously non-trivial as the target function is non-convex and highly discontinuous. Actually, with ${\lambda }_i$ fixed, the minimization of $f\left({\mu }_i\right)+g\left({\mu }_i\right)$ can be globally solved due to its discrete finite nature. In another word, if a series of candidate terms of ${\mu }_i$ are given, the global search is guaranteed to succeed.

Once a ${\lambda }_i$ is given, the $f\left({\mu }_i\right)+g\left({\mu }_i\right)$ will be a piecewise constant function. It means that the function values remain unchanged within a series of intervals which are determined by $|{\mathit{\psi }}_i^T{\mathbf{x}}_i|,i=i_1,i_2,\dots ,i_l$. Therefore, $|{\mathit{\psi }}_i^T{\mathbf{x}}_i|,i=1,2,\dots ,L$ can be taken as a portion of candidate terms of ${\mu }_i$. For the function $\mathsf{ł}\left({\mu }_i\right)$, it is clear that it minimizes at ${\mu }_i={\lambda }_i$ and monotonically increases with the increasing distance between $\mathsf{ł}\left({\mu }_i\right)$ and the given $\mathsf{ł}\left({\lambda }_i\right)$. So, to minimize $\mathsf{ł}\left({\mu }_i\right)$, we only need to choose the closest point in the feasible region.

Without loss of generality, we assume that all the $|{\mathit{\psi }}_i^T{\mathbf{x}}_i|,i=1,2,\dots ,L$ are ascending ordered and the corresponding signals are in the same order. We compute all the possible values of $f\left({\mu }_i\right)+g\left({\mu }_i\right)$ by

$$\begin{array}{cc}\hfill & f\left({\mathit{\psi }}_i^T{\mathbf{X}}_{j+1}\right)=f\left({\mathit{\psi }}_i^T{\mathbf{X}}_j\right)+y_{i(j+1)}^2\hfill \\ \hfill & g\left({\mathit{\psi }}_i^T{\mathbf{X}}_{j+1}\right)={\parallel {\overline{\mathbf{y}}}_i-{\mathit{\psi }}_i^T\mathbf{X}+sgn\left({\mathit{\psi }}_i^T\mathbf{X}\right){\lambda }_i\parallel }_2^2-{\beta }_{k+1}\hfill \end{array}$$

(30)

where ${\beta }_{k+1}={\beta }_k+{[y_{i(k+1)}-{\mathit{\psi }}_i^T{\mathbf{x}}_{k+1}+sgn\left({\mathit{\psi }}_i^T{\mathbf{x}}_{k+1}\right){\lambda }_i]}^2,{\beta }_1={[y_{i1}-{\mathit{\psi }}_i^T{\mathbf{x}}_1+sgn\left({\mathit{\psi }}_i^T{\mathbf{x}}_1\right){\lambda }_i]}^2$. Sort $|{\mathit{\psi }}_i^T{\mathbf{x}}_i|,i=1,2,\dots ,L$ in descending order of $f\left({\mu }_i\right)+g\left({\mu }_i\right)$, and every two adjacent values form an interval on which the function value remains unchanged. In another word, the objective function $f\left({\mu }_i\right)+g\left({\mu }_i\right)+l\left({\mu }_i\right)$ is minimized at the point closest to ${\lambda }_i$ in the interval. Thus, compute all the minimizer values on every interval and the minimum must be the optimal result.

With ${\mu }_i$ fixed, we solve the following problem in order to pursue ${\lambda }_i$:

$$\begin{array}{cc}\hfill & \{{\widehat{\lambda }}_i\}=\underset{{\lambda }_i}{argmin}\parallel {\overline{\mathbf{y}}}^{{\mathcal{J}}_i}-[{\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}-sgn\left({\mathit{\psi }}_i^T{\mathbf{X}}^{{\mathcal{J}}_i}\right){\lambda }_i]{\parallel }_2^2+\tau /2{\parallel {\lambda }_i-{\mu }_i\parallel }_2^2\hfill \end{array}$$

(31)

This is a standard continuous convex function that can be easily sovled by least square.

We summarize our sparse coding method in Algorithm 1.

Algorithm 1 Sparse coding algorithm

Input and Initialization:     Training data $\mathbf{X}\in {\mathcal{R}}^{N\times L}$, iteration number r, initial value ${\lambda }_i=0$. Output:     Sparse coefficients $\mathbf{y}$, and thresholding values ${\lambda }_i$ 1: Compute the sparse coefficients $\mathbf{y}$ via Problem (24) according to the OMP algorithm. 2: Sort the columns of $\mathbf{X}$ and $\mathbf{y}$ in increasing order of $|{{\mathsf{\psi }}_i}^T\mathbf{X}|$. 3: For p=1:r For j=1:L Compute all the possible values for $f\left({\mu }_i\right)+g\left({\mu }_i\right)$ by $$\begin{array}{c}f\left({\mathsf{\psi }}_i^T{\mathbf{X}}_{j+1}\right)=f\left({\mathsf{\psi }}_i^T{\mathbf{X}}_j\right)+y_{i(j+1)}^2\hfill \\ g\left({\mathsf{\psi }}_i^T{\mathbf{X}}_{j+1}\right)={\parallel {\overline{\mathbf{y}}}_i-{\mathsf{\psi }}_i^T\mathbf{X}+sgn\left({\mathsf{\psi }}_i^T\mathbf{X}\right){\lambda }_i\parallel }_2^2-{\beta }_{k+1}\hfill \end{array}$$ where ${\beta }_{k+1}={\beta }_k+{[y_{i(k+1)}-{\mathsf{\psi }}_i^T{\mathbf{x}}_{k+1}+sgn\left({\mathsf{\psi }}_i^T{\mathbf{x}}_{k+1}\right){\lambda }_i]}^2,{\beta }_1={[y_{i1}-{\mathsf{\psi }}_i^T{\mathbf{x}}_1+sgn\left({\mathsf{\psi }}_i^T{\mathbf{x}}_1\right){\lambda }_i]}^2$. Denote them as a vector $\nu$. End for 4: Sort the elements of $|{{\mathsf{\psi }}_i}^T\mathbf{X}|$ in descending order of $\nu$. Denote the intervals bounded as ${\xi }_q,q=1,2,L-1$. 5: compute every ${\nu }_q+l\left({\widehat{\mu }}_i\right)$ where ${\widehat{\mu }}_i$ is the point closest to ${\lambda }_i$ in ${\xi }_{j-1}$. 6: ${\widehat{\mu }}_i=arg{min}_{{\mu }_i}{\nu }_q+l\left({\mu }_i\right)$. 7: Compute ${\lambda }_i$ via Problem (31). End for

4.2. Dictionary Pair Update Phase

To obtain $\mathsf{\Psi }$, we solve the following problem with all other variables fixed:

$$\widehat{\mathsf{\Psi }}=\underset{\mathsf{\Psi }}{argmin}\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right){\parallel }_F^2+\frac{{\gamma }_3}{{\gamma }_1}{\parallel \mathsf{\Psi }-{\mathbf{F}}^{-1}\mathsf{\Phi }\parallel }_F^2$$

(32)

Such problem is a highly nonlinear optimization due to the definition of ${\mathcal{S}}_{\lambda }$. Here we solve $\mathsf{\Psi }$ columnwisely by updating each column of $\mathsf{\Psi }$.

For each ${\mathit{\psi }}_i$, we solve the following subproblem:

$${\widehat{\mathit{\psi }}}_i=\underset{{\mathit{\psi }}_i}{argmin}\parallel {\overline{\mathbf{y}}}_i-{\mathcal{S}}_{{\lambda }_i}\left({\mathit{\psi }}_i^T\mathbf{X}\right){\parallel }_2^2+\frac{{\gamma }_3}{{\gamma }_1}{\parallel {\mathit{\psi }}_i-{\mathbf{F}}^{-1}{\mathit{\varphi }}_i\parallel }_2^2$$

(33)

We denote ${\mathcal{J}}_i$ and ${\widehat{\mathcal{J}}}_i$ as the indices set as before. Set the elements of ${\overline{\mathbf{y}}}_i$ corresponding to the indices ${\widehat{\mathcal{J}}}_i$ to be zeros and denote the new vector as ${\overline{\mathbf{z}}}_i$. This operation leads to a consequence that ${\mathit{\psi }}_i^T{\mathbf{X}}^{{\widehat{\mathcal{J}}}_i}$≈0. Then we solve the following quadratic optimization problem that is easy to solve with least squares.

$$\begin{array}{cc}\hfill & \widehat{{\mathit{\psi }}_i}=\underset{{\mathit{\psi }}_i}{argmin}\parallel {\overline{\mathbf{z}}}_i-{\mathit{\psi }}_i^T{\mathbf{X}\parallel }_2^2+\frac{{\gamma }_3}{{\gamma }_1}{\parallel {\mathit{\psi }}_i-{\mathbf{F}}^{-1}{\mathit{\varphi }}_i\parallel }_2^2\hfill \end{array}$$

(34)

The optimization problem to pursue $\mathsf{\Phi }$ is formulated as

$$\begin{array}{cc}\hfill \widehat{\mathsf{\Phi }}=\underset{\mathsf{\Phi }}{argmin}& {\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{Y}\parallel }_F^2+{\gamma }_3{\parallel \mathsf{\Psi }-{\mathbf{F}}^{-1}\mathsf{\Phi }\parallel }_F^2\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \sqrt{A}\le {\delta }_{\mathsf{\Phi }}\le \sqrt{B}\hfill \end{array}$$

(36)

where the frame operator $\mathbf{F}$ is given by $\mathsf{\Phi }{\mathsf{\Phi }}^T$ and ${\mathbf{F}}^{-1}$ is defined as Equation (22). The target function then becomes

$${\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{Y}\parallel }_F^2+{\eta }_3{\parallel \mathsf{\Psi }-\frac{2}{A+B}(2\mathbf{I}-\frac{2\mathsf{\Phi }{\mathsf{\Phi }}^T}{A+B})\mathsf{\Phi }\parallel }_F^2$$

(37)

which is denoted by $h\left(\mathsf{\Phi }\right)$. We apply the gradient descent method to unconstraint version of Problem (35) and then project the solution to the feasible space. The gradient is given by a very complicated form as follows

$$\begin{array}{cc}\hfill & \nabla h\left(\mathsf{\Phi }\right)=(\mathbf{X}-\mathsf{\Phi }\mathbf{Y}){\mathbf{Y}}^T-{\gamma }_3\{\frac{4}{\alpha }h\left(\mathsf{\Phi }\right)+\frac{4}{{\alpha }^2}[\mathsf{\Phi }{\mathsf{\Phi }}^{T}h\left(\mathsf{\Phi }\right)+\mathsf{\Phi }h{\left(\mathsf{\Phi }\right)}^T\mathsf{\Phi }+h\left(\mathsf{\Phi }\right)\mathsf{\Phi }{\mathsf{\Phi }}^T]\}.\hfill \end{array}$$

(38)

In order to reduce the complexity, the gradient can also be computed with the fixed $\mathbf{F}$ calculated in the previous step of the ADM. Then at the k-th iteration, the gradient can be written as

$$\nabla h\left({\mathsf{\Phi }}^k\right)=(\mathbf{X}-{\mathsf{\Phi }}^{k-1}\mathbf{Y}){\mathbf{Y}}^T-{\gamma }_3{{\mathbf{F}}^{(-\mathbf{1})}}^T({\mathsf{\Psi }}^k-{\mathbf{F}}^T{\mathsf{\Phi }}^{k-1})$$

(39)

where $\mathbf{F}={\mathsf{\Phi }}^{k-1}{{\mathsf{\Phi }}^{(k-1)}}^T$. The descent step length can be obtained by optimizing the problem $\underset{\theta }{min}h(\mathsf{\Phi }+\theta \nabla h\left(\mathsf{\Phi }\right))$ with fixed $\mathbf{F}$, which is given by

$$\begin{array}{c}\hfill \widehat{\theta }=\frac{<\mathbf{a},\mathbf{b}>+{\gamma }_3<\mathbf{c},\mathbf{d}>}{{\parallel \mathbf{a}\parallel }_F^2+{\gamma }_3{\parallel \mathbf{c}\parallel }_F^2}\end{array}$$

(40)

where $\mathbf{a}=\nabla h\left(\mathsf{\Phi }\right)\mathbf{Y},\mathbf{b}=\mathbf{X}-\mathsf{\Phi }\mathbf{Y},\mathbf{c}={\mathbf{F}}^{-1}\nabla h\left(\mathsf{\Phi }\right),\mathbf{d}=\mathsf{\Psi }-{\mathbf{F}}^{-1}\mathsf{\Phi }$. For the frame condition $\sqrt{A}\le {\delta }_{\mathsf{\Phi }}\le \sqrt{B}$, we apply a SVD decomposition $\mathsf{\Phi }=\mathbf{U}\Sigma {\mathbf{V}}^T$ and map the singular values to the interval $[\sqrt{A},\sqrt{B}]$ linearly. We denote the mapped singular matrix as $\widehat{\Sigma }$ and reconstruct $\mathsf{\Phi }$ by $\mathsf{\Phi }=\mathbf{U}\widehat{\Sigma }{\mathbf{V}}^T$.

We summarize our algorithm in Algorithm 2.

Algorithm 2 Dictionary pair learning algorithm

Input and Initialization:     Training data $\mathbf{X}$, frame bound A,B, iteration $num$, gradient descent iterations r.     Build frame $\mathsf{\Phi }\in {\mathbb{R}}^{N\times M}$ and $\mathsf{\Psi }\in {\mathbb{R}}^{N\times M}$, either by using random entries, or using M randomly chosen data. Output:     Frame $\mathsf{\Phi },\mathsf{\Psi }$, Sparse coefficients $\mathbf{Y}$, and thresholding values $\lambda$     For l=1:$num$     Sparse Coding Step: 1: Compute the sparse coefficients $\mathbf{Y}$ and the thresholding values $\lambda$ via Algorithm 1. Frame Update Step: 2: Update $\mathsf{\Psi }$ in column-wise. Compute $\mathbf{W}=\mathcal{S}\left({\mathsf{\Psi }}^T\mathbf{X}\right).$ For $i=1:M$ Denote ${\widehat{\mathcal{J}}}_i$ as the indices of zeros in the i-th column of W. Set ${\mathit{\psi }}_i^T{\mathbf{X}}^{{\widehat{\mathcal{J}}}_i}=0$. Compute ${\mathit{\psi }}_i$ via Equation (34). End For 3: Update $\mathsf{\Phi }$ via Gradient Descent and singular value map. For k=1:r Calculate Gradient via Equation (39) and calculate the descent step via Equation (40) End for 4: Apply SVD decomposition $\mathsf{\Phi }=\mathbf{U}\Sigma {\mathbf{V}}^T$, map $\Sigma$ to obtain $\widehat{\Sigma }$ and reconstruct $\mathsf{\Phi }=\mathbf{U}\widehat{\Sigma }{\mathbf{V}}^T$. End for

5. Restoration

The image restoration aims to reconstruct a high-quality image $\mathcal{I}$ from its degraded (e.g., noisy, blurred and/or downsampled) version $\mathcal{L}$, denoted by $\mathcal{L}=SH\mathcal{I}+\mathbf{n}$, where H represents a blurring filter, S the downsampling operator, and $\mathbf{n}$ is a noisy signal. For the signal satisfies the SSM-NTF, the restoration model based on SSM-NTF is formulated as

$$\begin{array}{cc}\hfill & \{\widehat{\mathcal{I}},\widehat{\mathbf{Y}},\widehat{\lambda \}}=\underset{\mathcal{I},{\left\{{\mathbf{y}}_i\right\}}_{i=1}^N,\widehat{\lambda }}{min}\parallel \mathcal{L}-SH\mathcal{I}{\parallel }_F^2+\gamma \sum _i\parallel {\mathbf{R}}_i\mathcal{I}-\mathsf{\Phi }{\mathbf{y}}_i{\parallel }_F^2+{\gamma }_1\sum _i\parallel {\mathbf{y}}_i-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T{\mathbf{R}}_i\mathcal{I}\right){\parallel }_F^2+{\gamma }_2\sum _i\parallel {\mathbf{y}}_i{\parallel }_0\hfill \end{array}$$

(41)

where ${\mathbf{R}}_i$ is an operator that extracts the i-th patch of the image $\mathcal{I}$ and ${\mathbf{y}}_i$ is the i-th column of $\mathbf{Y}$. $\lambda$ denotes a vector $[{\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_M]$ with ${\lambda }_j$ operating on the j-th element of ${\mathsf{\Psi }}^T{\mathbf{R}}_i\mathcal{I}$. On the right side of Equation (41), the first term is the global force that demands the proximity between the degraded image $\mathcal{L}$, and its high-quality version $\mathcal{I}$. The rest terms are the local constraints to make sure every patch at location i satisfies the SSM-NTF.

To solve Problem (41), we apply Algorithm 1 to obtain the sparse coefficients $\mathbf{Y}$ and the threshold values $\lambda$. We mainly state the iterative method to obtain $\mathcal{I}$. Assume the sign of ${\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^k$ will not change much between two steps, we set it in the k-th step by ${\mathbf{c}}^k=sign\left({\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^{k-1}\right)$. where $sign$ is the sign function. Denote ${\mathbf{d}}^k={\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^{k-1}$. We set ${\mathbf{O}}^k$ as an index set that satisfies $|{\mathbf{d}}_l|\le {\lambda }_l,l\in {\mathbf{O}}^k$. Set ${\mathbf{u}}^k\in {\mathcal{R}}^M$ as a vector with elements ${\mathbf{u}}_l=\left\{\begin{array}{cc}{\lambda }_l\hfill & l\in {\mathbf{O}}^k,\hfill \\ 0\hfill & otherwise.\hfill \end{array}\right.$ Then the non-convex and non-smooth threshold can be removed with the substitution that ${\mathbf{y}}_i-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^k\right)\approx {\mathbf{y}}_i+{\mathbf{c}}^k\odot \mathbf{u}-{\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^k$ Thus, in the k-th step, the problem needs to be solved is expressed as

$$\begin{array}{cc}\hfill & \{\widehat{{\mathcal{I}}^k}\}=\underset{{\mathcal{I}}^{k-1}}{min}\parallel \mathcal{L}-SH{\mathcal{I}}^{k-1}{\parallel }_F^2+\gamma \sum _i\parallel {\mathbf{R}}_i{\mathcal{I}}^{k-1}-\mathsf{\Phi }{\mathbf{y}}_i{\parallel }_F^2+{\gamma }_1\sum _i{\parallel {\mathbf{y}}_i+{\mathbf{c}}^k\odot {\mathbf{u}}^k-{\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^{k-1}\parallel }_F^2\hfill \end{array}$$

(42)

where ⊙ is point multiplication. This convex problem can be easily solved by gradient descent algorithm.

We summarized the restoration algorithm in Algorithm 3.

Algorithm 3 Restoration algorithm

Input     Training dictionaries $\mathsf{\Phi }$, $\mathsf{\Psi }$, iteration number r, a degraded image $\mathcal{L}$, set ${\mathcal{I}}_0=\mathcal{L}$. Output:     The high quality image $\widehat{\mathcal{I}}$ 1: Compute $\mathbf{Y}$ and $\lambda$ via the method in Algorithm 1. For k=1:r 2: compute ${\mathbf{d}}^k={\mathsf{\Psi }}^T{\mathbf{R}}_i{\mathcal{I}}^{k-1}$. Set ${\mathbf{c}}^k=sign\left({\mathbf{d}}^k\right)$. set ${\mathbf{O}}^k$ as an index set that satisfies $|{\mathbf{d}}_l^k|\le {\lambda }_l,l\in {\mathbf{O}}^k$. Set ${\mathbf{u}}_l^k=\left\{\begin{array}{cc}{\lambda }_l\hfill & l\in {\mathbf{O}}^k,\hfill \\ 0\hfill & otherwise.\hfill \end{array}\right.$. 3: Solving the Problem (42) via gradient descent algorithm. End for

6. Complexity Analysis

In this section, we discuss the computational complexity of our sparse coding and dictionary pair learning algorithms with regard to those of conventional sparse model counterparts.

We first analyze complexities of the main components of the sparse coding (SC) and dictionary updating (DU) algorithms. In terms of SC, given a set of training samples, $\mathbf{X}\in {\mathcal{R}}^{N\times L}$, the complexity of BtOMP of calculating $\widehat{\mathbf{Y}}=\underset{\mathbf{Y}}{min}{\parallel \mathbf{X}-\mathsf{\Phi }\mathbf{Y}\parallel }_F^2+{\gamma }_1\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right){\parallel }_F^2+{\gamma }_2{\parallel \mathbf{Y}\parallel }_0$ is $O\left(K^{2}ML\right)$ where K is the target sparsity and the complexity of threshold of calculating $\widehat{\lambda }=\underset{\lambda }{min}{\parallel \mathbf{Y}-{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{X}\right)\parallel }_F^2$ is $O\left(NML\right)$, which cost most of time in SC step at each iteration. The sparse coefficients $\mathbf{Y}\in N\times L$ and the the threshold values $\lambda$ are computed with fixed dictionaries $\mathsf{\Phi }\in {\mathcal{R}}^{N\times M}$ and $\mathsf{\Psi }\in {\mathcal{R}}^{N\times M}$. Correspondingly, the traditional sparse coefficients $\mathbf{B}\in N\times L$ is sparse approximated by dictionary $\mathbf{D}\in {\mathcal{R}}^{N\times M}$ and the computational complexity is $O\left(K^{2}ML\right)$.

In terms of DU, with the given training samples $\mathbf{X}\in {\mathcal{R}}^{N\times L}$, we learn a pair of dictionary $\mathsf{\Phi }\in {\mathcal{R}}^{N\times M}$ and $\mathsf{\Psi }\in {\mathcal{R}}^{N\times M}$. We update $\mathsf{\Psi }$ via Problem (34) with a computational complexity of $O\left(N^{2}L\right)$. In order to update $\mathsf{\Phi }$. we need to calculate the gradient via Equation (39) with a computational complexity of $O\left(NML\right)$ and the step size via Equation (40) with a computational complexity of $O\left(rNML\right)$ where r is the iteration number of the gradient descent. For the traditional dictionary learning, the corresponding training set is $\mathbf{X}\in {\mathcal{R}}^{N\times L}$ and the dictionary $\mathbf{D}\in {\mathcal{R}}^{N\times M}$ is updated by SVD decomposition of rank-1 with a computational complexity of $O\left(KML\right)$.

7. Experimental Results

We demonstrate the effectiveness of our SSM-NTF model by first discussing the convergence of our dictionary pair learning algorithm and then evaluating the performance on natural and piecewise constant image denoising, super resolution and image inpainting.

7.1. Convergence Analysis

The convergence of the presented dictionary pair learning algorithm is evaluated in Figure 1. Here, we train a pair of dictionaries $\mathsf{\Phi }$ and $\mathsf{\Psi }$ of size $100\times 200$ from 65,000 patches, which are of size 10 × 10 randomly sampled from six natural images. We apply the frame reconstruction function $\mathsf{\Phi }{\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{x}\right)$ to reconstruct the patches. The convergence of the presented dictionary pair learning algorithm is evaluated in Figure 1. The dictionary pair is illustrated in Figure 2. They exhibit that our dictionary pair learning method is able to capture the feature of the image along with the convergence property.

7.2. Image Denoising

In this subsection, we evaluate the performance of our proposed SSM-NTF model on image denoising. Benefitting from the concept of non-tight frame, the proposed SSM-NTF model contains a pair of dictionaries: The frame and its dual frame. As a result, our proposed SSM-NTF model contains an analysis system and a synthesis system. The analysis-like system is denoted as

$$\mathbf{y}={\mathcal{S}}_{\lambda }\left({\mathsf{\Psi }}^T\mathbf{x}\right),\parallel \mathbf{y}\parallel \le s$$

(43)

which analyzes the signals in $\mathsf{\Psi }$ domain. The synthesis system is denoted as

$$\mathbf{x}=\mathsf{\Phi }\mathbf{y},\parallel \mathbf{y}\parallel \le s$$

(44)

which reconstructs the analyzed signals. The two systems share the same sparse coefficients $\mathbf{Y}$.

Therefore, we compare our proposed SSM-NTF with synthesis and analysis models, respectively. It is well known that the synthesis sparse model has advantage in dealing with the natural image while the analysis sparse model is mostly used to address the piecewise constant image. Therefore, we respectively, perform the denoising experiments on natural images and piecewise images comparing with the most related approaches.

7.2.1. Natural Image Denoising

We now turn to present experimental results on six classical natural images named ‘Barbara’, ‘Boat’, ‘Couple’, ‘Hill’, ‘Lena’ and ‘Man’ which are shown in [1], to evaluate the performance of the training algorithm. The denoising problem which has been widely studied in sparse representation is used as the target application. We add Gaussian white noise to these images at different noise levels $\sigma =20,30,40,50,60,70,80,90,100$. Then we use the learned dictionary pair to denoise the natural images, with overlap of 1 pixel between adjacent patches of size $10\times 10$. The patch denoising stage is followed by weighted averaging the overlapping patch recoveries to obtain the final clean image. The parameters in our scheme are ${\gamma }_1=1.1$ and ${\gamma }_3=1.2{(L/M)}^2$ where L and M are the sample and dictionary size, respectively. We have stated in Section 3.2 that we usually set A to a positive number around but smaller than 1. In fact, we set A from 0.6 to 1 by a step of 0.03 to test the denoising performance to determine the specific value of it. Then, with fixed A, we set B from 1 to 4 with a step length of 0.3 to run experiments on every noise level to determine the values of B. The values of frame bounds A and B are shown in

Table 1. The valuesof A and B.

$\mathit{\sigma }$	20	30	40	50	60	70	80	90	100
A	0.8	0.8	0.8	0.8	0.8	0.8	0.8	0.8	0.8
B	1.8	1.8	1.8	2.4	3.0	3.3	3.3	3.6	3.6

. For example, when the noise level $\sigma =40$, A and B are set to be 0.8 and 1.8, respectively.

Table 2. PSNR (dB) for nature image denoising results.

$\mathit{\sigma }$	Image	Barbara	Boat	Couple	Hill	Lena	Man	Average
20	KSVD [9]	31.01	30.50	30.15	30.27	32.51	30.26	30.78
	[21]	31.03	30.45	30.16	30.30	32.52	30.30	30.79
	[1]	31.07	30.35	30.20	30.31	32.56	30.07	30.76
	SSM-NTF	31.06	30.56	30.32	30.42	32.60	30.33	30.88
30	KSVD [9]	28.75	28.60	28.07	28.51	30.59	28.43	28.83
	[21]	28.78	28.63	28.06	28.53	30.60	28.47	28.85
	[1]	29.07	28.48	28.22	28.64	30.60	28.26	28.88
	SSM-NTF	29.03	28.71	28.32	28.74	30.82	28.59	29.04
40	KSVD [9]	27.03	27.23	26.54	27.23	29.13	27.17	27.39
	[21]	27.05	27.18	26.59	27.21	29.10	27.12	27.38
	[1]	27.58	27.20	26.87	27.49	29.25	26.99	27.56
	SSM-NTF	27.52	27.49	27.07	27.65	29.43	27.41	27.76
50	KSVD [9]	25.71	26.05	25.42	26.29	27.92	26.18	26.26
	[21]	25.77	26.08	25.40	26.31	27.87	26.23	26.28
	[1]	26.45	26.15	25.84	26.63	28.15	26.09	26.55
	SSM-NTF	26.40	26.41	26.04	26.88	28.49	26.49	26.79
60	KSVD [9]	24.45	25.18	24.57	25.69	27.01	25.40	25.38
	[21]	24.45	25.20	24.50	25.65	27.03	25.42	25.38
	[1]	25.64	25.33	25.04	25.91	27.22	25.38	25.75
	SSM-NTF	25.52	25.55	25.22	26.11	27.41	25.74	25.93
70	KSVD [9]	23.40	24.46	23.90	25.10	26.18	24.77	24.63
	[21]	23.46	24.46	23.93	25.15	26.19	24.79	24.66
	[1]	24.88	24.67	24.36	25.30	26.44	24.78	25.07
	SSM-NTF	24.74	25.05	24.60	25.50	26.61	25.06	25.26
80	KSVD [9]	22.80	23.85	23.44	24.69	25.34	24.19	24.05
	[21]	22.87	23.93	23.47	24.76	25.40	24.27	24.09
	[1]	24.18	24.09	23.79	24.77	25.74	24.25	24.47
	SSM-NTF	24.11	24.42	24.02	24.91	26.02	24.58	24.68
90	KSVD [9]	22.31	23.34	22.98	24.31	24.96	23.78	23.61
	[21]	22.31	23.38	23.01	24.36	24.97	23.84	23.64
	[1]	23.50	23.56	23.27	24.30	25.14	23.79	23.93
	SSM-NTF	23.41	23.84	23.47	24.58	25.43	24.13	24.14
100	KSVD [9]	21.88	22.92	22.64	23.96	24.44	23.39	23.21
	[21]	21.89	22.95	22.67	23.99	24.47	23.43	23.23
	[1]	23.01	23.11	22.83	23.86	24.59	23.38	23.46
	SSM-NTF	22.90	23.46	23.05	24.16	24.90	23.68	23.69

shows the comparison results in terms of PSNR. There are three related image denoising methods involved, including the classical dictionary learning algorithm KSVD [9], the data-driven tight frame based denoising method [1] and the incoherent dictionary learning based method [21]. The patch size of KSVD [9] and the method in [21] are $8\times 8$ with stripe 1 and the dictionaries are of size $64\times 256$ at their optimal state according to the previous work. We point out that [1] works on filters of size $16\times 16$ instead of image patches and initialized by 64 3-level Harr wavelet filters in size $16\times 16$. All the three compared methods can achieve their best performance with 50 iterations.

Table 2 shows that the incoherent dictionary learning method [21] outperforms the KSVD [9] in average as the mutual incoherent of dictionary can provide stable recovery. That [1] outperforms [21] implies that the tight frame is a more stable system. Then our stable sparse model based method outperforms [1] in average suggests that applying non-tight frame to approximate RIP can provide even better and more stable reconstruction results. Figure 3 shows two exemplified visual results on images ‘Man’ and ‘Couple’ at noise levels $\sigma =50$ and $\sigma =40$, respectively. The proposed method shows much clearer and better visual results than the other competing methods.

7.2.2. Piecewise Constant Image Denoising

In this subsection, we demonstrate the analytical property of our SSM-NTF model using a synthetic image. The denoising problem which has been widely studied in sparse representation is used as the target application. We start with a piecewise constant of size $256\times 256$ contaminated by Gaussian white noise with noise level $\sigma =5$ and extract all possible $5\times 5$ image pathes. For the denoising we apply the dictionary pair learning algorithm with the parameters ${\gamma }_1=1.5,{\gamma }_3=1.2$ $L/M$, $A=0.8$ and $B=1.8$ in parallel with patch denosing with the synthesis KSVD [9] and the analysis KSVD [14]. We apply 100 iterations of the our dictionary learning method on this training set, and learning dictionary pair of size $25\times 50$. The experimental set of the synthesis KSVD [9] and the analysis KSVD [14] are at their optimal state according to the previous work.

The learned dictionary pair $\mathsf{\Phi }$ which exhibits much like the synthesis dictionary and $\mathsf{\Psi }$ which exhibits a high resemblance to the analysis dictionary are illustrated in Figure 4. The resulting PSNRs of the denoised images are 45.32 dB for Analysis KSVD, 43.60 dB for Synthesis KSVD, and 45.17 dB for our proposed algorithm. The figure shows that our dictionary pair learning method is able to capture the features of the piecewise constant image. Figure 5 shows the absolute difference images for each of the three methods. Note that these images are displayed in the dynamic range $[0,20]$ to make the differences more pronounced. Our proposed approach leads to a much better denoising result than the synthesis KSVD and is comparable with the analysis KSVD.

7.3. Super Resolution

We evaluate our SSM-NTF in comparison with two examplar-based scheme [37] for image Super Resolution (SR) Problem (41) with a bicubic filter. Figure 6 shows the 15 test natural images [18] with both rich texture and structure. All the schemes are applied to the illumination channel, where the scale factor is 3, we always use $3\times 3$ low-resolution patches with overlap of 1 pixel between adjacent patches, corresponding to $9\times 9$ patches with overlap of 3 pixels for the high-resolution patches. In these experiments, we have used the following parameters: $A=0.8,B=1.8$, ${\gamma }_1=1.1$ and ${\gamma }_3=1.2$ $L/M$ where L and M are the sample and dictionary size, respectively. In our scheme, dictionary learning is performed between HR and middle-level (MR) images which are the first-, and second-order derivatives of the upsampled version of one LR image by a factor of 2. The four $1D$ filters used to extract the derivatives are:

$$\begin{array}{cc}\hfill & {\mathbf{f}}_1=[-1,0,1],{\mathbf{f}}_2={\mathbf{f}}_1^T\hfill \\ \hfill & {\mathbf{f}}_3=[1,0,-2,0,1],{\mathbf{f}}_4={\mathbf{f}}_3^T\hfill \end{array}$$

(45)

We train two pairs of HR/LR dictionaries $\{{\mathsf{\Phi }}_h,{\mathsf{\Psi }}_h\}$ and $\{{\mathsf{\Phi }}_l,{\mathsf{\Psi }}_l\}$ from 100,000 HR/LR patch pairs $[{\mathbf{X}}_h,{\mathbf{X}}_l]$ randomly sampled from the collected natural images which are also used in [37] where ${\mathbf{X}}_h$ is sampled from the HR images and ${\mathbf{X}}_l$ is sampled from the four feature images. The feature images are obtained by applying the four filters to the upsampled LR image. Given $\mathsf{\Phi }$ and $\mathsf{\Psi }$ and the four MR feature images, the sparse coefficients $\mathbf{Y}$ and threshold value $\lambda$ can be calculated by Algorithm 1. With the theory in [37], the HR image can be recovered via Algorithm 3. In the experiment, our HR dictionary pair are of size $81\times 450$ and MR ones are of size $144\times 450$. The dictionary size of [37] is 81 × 1024 (HR) and 144 × 1024 (MR) at its best performance as stated in the paper. Thus, the dictionary size of [37] is larger than the sum of our dictionaries.

Table 3. PSNR (dB) for $3\times$ SR reconstructions results.

$Image$	1	2	3	4	5	6	7	8
$Bicubic$ [37]	32.69	26.32	26.04	28.39	28.70	25.81	37.21	24.40
$SC$ [37]	32.89	26.53	26.16	28.51	29.09	26.25	37.59	24.61
$SSM-NTF$	33.10	26.75	26.41	28.82	29.31	26.40	37.49	24.77
$Image$	9	10	11	12	13	14	15	$Average$
$Bicubic$ [37]	31.40	28.72	33.24	30.28	26.49	28.11	31.90	29.31
$SC$ [37]	31.73	29.35	33.50	30.62	26.75	28.74	32.13	29.63
$SSM-NTF$	31.90	29.49	33.66	30.99	26.95	29.02	32.19	29.82

shows the objective evaluation results of our proposed SSM-NTF compared with bicubic interpolation and [37]. On average, our SSM-NTF presents best in PSNR. Figure 7 presents the corresponding visual comparison of the illumination SR results of Image 12. We can observe that the result of bicubic interpolation is too smooth and the result of [37] suffers from obvious ringing artifact and noises. The HR reconstruction of our SSM-NTF method provides more clear details.

7.4. Image Inpainting

To illustrate the potential applicability of our proposed SSM-NTF model on image inpainting, we apply it to the applications of text removal. In these experiments, we have used the following parameters: $A=0.8,B=1.8$, ${\gamma }_1=1.1$ and ${\gamma }_3=1.2$ $L/M$ where L and M are the sample and dictionary size, respectively. We operate on the image ‘Adar’, ‘Lena’, ‘Couple’, ‘Hill’ with super-imposed text of various fonts.

In this experiment, we applied our SSM-NTF model to image inpainting in a way similar to the non-blind KSVD inpainting algorithm [9], which requires the knowledge of which pixels are corrupted and required inpainting. Actually, only the non-corrupted pixels are used to training the dictionary pair and inpainting the images. We operate our method on pathes of size $10\times 10$ that extract from the images with overlap of 1 pixel between adjacent. The trained dictionary pair are of size $100\times 200$. The KSVD algorithm in this experiment is dealing with patches of size $8\times 8$ that extract from the images with overlap of 1 pixel between adjacent. The dictionary size is $64\times 256$ at its best performance according to [9]. The patch inpainting stage is followed solving Problem (41).

Table 4. PSNR (dB) for image inpainting results.

$\mathbf{Image}$	Adar	Lena	Hill	Couple	Average
$DCT$ [9]	27.11	31.30	28.53	29.21	29.04
$KSVD$ [9]	28.01	31.69	28.90	29.50	29.53
$SSM-NTF$	28.95	32.02	29.31	29.87	30.04

shows the objective evaluation results of our proposed SSM-NTF compared with DCT and KSVD [9]. The visual comparisons are shown in Figure 8 and Figure 9. We find that the proposed SSM-NTF method is able to eliminate text of fonts completely while the KSVD is dull. Our SSM-NTF method achieves better performance in terms of both subjective and objective quality.

8. Conclusions

In this paper, we propose a stable sparse model with non-tight frame (SSM-NTF) and further formulate a dictionary pair learning model to stably recover the signals. We theoretically analyze the rationality of the approximation for RIP with the non-tight frame condition. The proposed SSM-NTF has RIP and the closed-form expression of the sparse coefficients that ensure the stable recovery especially for seriously noise images. The proposed SSM-NTF contains both a synthesis sparse and an analysis system which share the common sparse coefficients without taking into account the thresholding. We also propose an efficient dictionary pair learning algorithm via developing an explicit analytical expression of the inherent relation between the dictionary pair. The proposed algorithm is capable of approximating structures of signals via a pair of adaptive dictionaries. The effectiveness of our proposed SSM-NTF and its corresponding algorithms are demonstrated in image denoising, image super-resolution and image inpainting. The results of numerical experiments show that the proposed SSM-NTF achieves superior to the compared methods in objective and subjective quality on most of the cases.

On the other hand, our proposed SSM-NTF is actually a 1D sparse model. The 1D sparse model suffers from high memory as well as high computational costs especially when handling high dimensional data. MD frame can be expressed as the kronecker product of a series of 1D frames. Benefitting from this good characteristic, in future work, we will extend our stable sparse model to propose an MD stable sparse model. Moreover, the proposed SSM-NTF is not effective enough to remove other kinds of noise (e.g., salt and pepper noise) as the loss function of SSM-NTF is gaussian. We would like to improve the performance of our model by changing the loss function.