Multi-Color Channels Based Group Sparse Model for Image Restoration

Abstract

The group sparse representation (GSR) model combines local sparsity and nonlocal similarity in image processing, and achieves excellent results. However, the traditional GSR model and all subsequent improved GSR models convert the RGB space of the image to YCbCr space, and only extract the Y (luminance) channel of YCbCr space to change the color image to a gray image for processing. As a result, the image processing process cannot be loyal to each color channel, so the repair effect is not ideal. A new group sparse representation model based on multi-color channels is proposed in this paper. The model processes R, G and B color channels simultaneously when processing color images rather than processing a single color channel and then combining the results of different channels. The proposed multi-color-channels-based GSR model is compared with state-of-the-art methods. The experimental contrast results show that the proposed model is an effective method and can obtain good results in terms of objective quantitative metrics and subjective visual effects.

1. Introduction

Image restoration is one of the fundamental tasks in image processing and computer vision, and it aims to recover the original image $x$ from the observed degraded image $y$. Image restoration is an ill-posed inverse problem and can be chartered by the following Formula (1):

$$y=Hx+n$$

(1)

Here, $x$ and $y$ represent the original image and the degraded image respectively, $H$ denotes the degenerate operator, and $n$ denotes additive Gaussian noise. $H$ is modeled according to different image processing tasks. When $H$ is a unit matrix, Formula (1) represents an image denoising task [1,2]. When $H$ is a blurring operator, Formula (1) represents an image deblurring task [3]. When $H$ is a mask, Formula (1) represents image restoration [4,5]. When $H$ is a random projection matrix, Formula (1) represents image compressive sensing applications [6,7].

The image restoration task is studied in this paper. In order to overcome the ill-posed characteristic and recover a satisfied copy of the original image, image priors are used to regularize the minimal problem,

$$\arg \underset{x}{\min }\frac{1}{2}{\Vert Hx-y\Vert }_2^2+\lambda \mathsf{\Psi }\left(x\right)$$

(2)

where the first term $\frac{1}{2}{\Vert Hx-y\Vert }_2^2$ represent the data fidelity term, $\mathsf{\Psi }\left(x\right)$ represents the regularization term of the image a priori, and $\lambda$ is a regularization parameter. Image priors play an important role in image restoration, and many relevant algorithms are proposed, such as total-variation-based methods [8,9,10], sparse-representation-based methods [11,12,13,14], nonlocal self-similarity-based methods [1,2,15] and deep-convolutional-neural-network-based methods [16,17,18].

Sparse representation is an effective technique in image processing. Sparse-representation-based algorithms can be divided into two categories, i.e., patch sparse representation (PSR) [11,19] and group sparse representation (GSR) [12,13,15]. In the PSR model, the basic sparse representation unit is the image block. PSR assumes that each block of the image to be repaired can be perfectly modeled by a sparse linear combination of some fixed or learnable basic elements. These basic elements are called atoms and constitute a set called a dictionary. The PSR model not only does well in image denoising [20], but also can be applied to various other image processing and computer vision tasks [21]. However, the PSR model with an over-complete dictionary usually produces visual artifacts which are not expected in image restoration [22]. In addition, the PSR model assumes that the coded image blocks using sparse representation are independent and ignores the correlation among similar image blocks [13,23], which results in image degradation. To solve the above problems, the GSR model uses an image group as its basic processing unit to integrate the local sparsity with the nonlocal self-similarity of images. GSR shows great potential in various image processing tasks, and many literatures are provided. Zhang et al. [13] proposed the traditional GSR model for image restoration, which is essentially equal to the low rank minimization model [24]. Dong et al. [25] proposed an effective Gaussian scale mixing-based image restoration model under the GSR framework. Zha et al. [26] proposed a group-based adaptive dictionary, which connects the GSR model with the low rank minimization model. Zha et al. [14] combined the PSR and GSR models to reduce their shortcomings, respectively. Zha et al. [27] introduced the group sparse residual constraints to the GSR model to further define and simplify the problem of image restoration.

The traditional GSR model and the improved GSR models usually transform the RGB space of the image into YCbCr space and extract the Y (luminance) component for image processing. The Y channel in YCbCr space contains much valuable information about edge, shadow, object, texture and others. Moreover, human vision is sensitive to the change of the Y channel. These good characteristics make the GSR model and its improved models very successful in various image processing tasks. However, these methods only extract the Y channel, and cannot be loyal to each color channel of the color image. In this paper, the color channels refer to red (R), green (G), and blue (B) channels in RGB space. In order to exploit all the three channels, we propose a new GSR model based on multi-color channels. Compared with the traditional GSR model, which only extracts the Y channel in YCbCr space, the proposed model preserves as much image information as possible.

The main contribution of the paper is to propose the multi-color-channels-based GSR model for image restoration. In this proposed model, three channels are exploited simultaneously, and hence as much image color information as possible is retained. The paper is organized as follows. Section 1 introduces briefly the image restoration models and the development of the sparse-representation-based methods. Section 2 describes the traditional GSR model. The proposed multi-color-channels-based GSR model is provided in Section 3. Section 4 provides the visual and quantitative experimental results. Finally, conclusions are given in Section 5.

2. GSP Model

Compared with the patch sparse representation (PSR) model, the traditional group sparse representation (GSR) model uses the image group instead of a single image block as the basic unit of sparse representation, which produces satisfactory results in image processing tasks [13,28,29]. The model proposed in this paper is based on the traditional GSR model, which is briefly introduced here.

2.1. Image Group Construction

The construction of the image group in GSR model is illustrated in Figure 1. The image $x$ with the size of $\sqrt{N}\times \sqrt{N}$ is divided into $n$ overlapped patches with the size of $\sqrt{b}\times \sqrt{b}$. The overlapped patches are denoted by $x_i$ where $i=1, 2, \dots , n$. For each $x_i$, its $m$ similar matched patches are constructed using the Euclidean distance as the similarity metric in a training window with the size of $W\times W$. The similar matched patches form a set denoted by $S_i$. All the similar patches in $S_i$ form a matrix with the size of $b\times m$, denoted by $X_i\in R^{b\times m}$. Each element in $S_i$ is a column of the matrix $X_i$, i.e., $X_i=\left\{x_{i,1},x_{i,2},\dots ,x_{i,m}\right\}$. The column $x_{i,j}$ denotes the jth similar patch of the ith image group corresponding to $x_i$. The matrix $X_i$ includes all the similar patches as columns, and hence is referred to as a group.

2.2. Sparse Representation Model

After constructing the image group, a sparse representation model is established in the image group. In order to obtain the sparse representation model of the image group, a dictionary, denoted by $D_i$ is given, which is usually learned from the image group, such as the graph-based dictionary [30] and principal component analysis (PCA) dictionary [31]. The aim of the sparse representation is to present the image group using as few atoms of the dictionary as possible. That is, the task is to code the dictionary sparsely using a sufficiently sparse vector. Each sparse representation, denoted by ${\widehat{\beta }}_i$, corresponding to the image group $X_i$, is obtained via minimizing the $L_0$ norm using the following Formula (3).

$${\widehat{\beta }}_i=\arg \underset{{\beta }_i}{\min }\left(\frac{1}{2}{\Vert X_i-D_i{\beta }_i\Vert }_2^2+\lambda {\Vert {\beta }_i\Vert }_0\right), \forall i$$

(3)

Here, ${\beta }_i$ denotes the group sparse coefficient corresponding to the ith image group $X_i$. $\frac{1}{2}{\Vert X_i-D_i{\beta }_i\Vert }_2^2$ is the fitting term used to measure the similarity between the image to be repaired via the sparse representation model and the original image. $\lambda$ is a regularization parameter, and ${\Vert \Vert }_0$ denotes the $L_0$ norm used to calculate the no-zero terms of each column in ${\beta }_i$. Since $L_0$ norm minimization is NP-hard, $L_1$ norm minimization is used instead of $L_0$ norm minimization via the following Formula (4).

$${\widehat{\beta }}_i=\arg \underset{{\beta }_i}{\min }\left(\frac{1}{2}{\Vert X_i-D_i{\beta }_i\Vert }_2^2+\lambda {\Vert {\beta }_i\Vert }_1\right), \forall i$$

(4)

The original image $x$ is not available in the image restoration task, and the image groups are from the degraded image $y\in R^N$. The image $y$ is segmented into $n$ overlapped patches with the same size of $\sqrt{b}\times \sqrt{b}$, i.e., a vector with the length of $b$. The $m$ similar matched patches are found to generate the image groups corresponding to the $n$ overlapped patches. Each image group denoted by $Y_i\in R^{b\times m}$ is a matrix with the size of $b\times m$, which also can be written as $Y_i=\left\{y_{i,1}, y_{i,2}, \dots , y_{i,m}\right\}$. Now, the image restoration task is to restore $X_i$ from $Y_i$ using the GSR model via the following Formula (5).

$${\widehat{\beta }}_i=\arg \underset{{\beta }_i}{\min }\left(\frac{1}{2}{\Vert Y_i-D_i{\beta }_i\Vert }_2^2+\lambda {\Vert {\beta }_i\Vert }_1\right), \forall i$$

(5)

After obtaining ${\widehat{\beta }}_i$, $i=1, 2, \dots , n$, the restored whole image $\widehat{x}$ can be represented by a set of sparse codes ${\left\{{\widehat{\beta }}_i\right\}}_{i=1}^n$.

2.3. Learning a Dictionary

Sparse representation aims to obtain the sparsest representation of the original signal on the learned dictionary, and the signal is represented as a linear combination of a small number of dictionary atoms [32]. Therefore, in the framework of sparse representation, the construction of the dictionary is the key step of image restoration. However, in image restoration tasks, the original image $x$ is unknown, and the original image group $X_i$ cannot be obtained directly. The dictionary is only constructed according to the degraded image $y$. The image $y$ is first initialized using the bilinear interpolation algorithm. The following operations are similar to the procedures depicted in Section 2.1. Specifically, the estimated image group $r_i$ corresponding to the original image group $X_i$ is extracted from the initialized image $y$. Then the singular value decomposition is performed on the estimation group $r_i$ via the following Equation (6).

$$r_i=U_i{\displaystyle {\sum }_i{V}_{{}_i}^T}={\displaystyle {\sum }_{j=1}^{n_1}{\gamma }_{i,j}\left(u_{i,j}{v}_{i,j}^T\right)}$$

(6)

Here, ${\gamma }_i$ denotes $\left[{\gamma }_{i,1}, {\gamma }_{i,2}, \dots , {\gamma }_{i,n1}\right]$, ${\displaystyle {\sum }_i}$ denotes a diagonal matrix, ${\gamma }_{i,j}$ denotes the main diagonal elements of ${\displaystyle {\sum }_i}$, $d$ represents each atom of the dictionary $D_i$, and $n_1$ is defined as the minimum of $d$ and $m$. $u_{i,j}$ and $v_{i,j}^T$ denotes the column vectors of the matrixes $U_i$ and $V_i^T$, respectively. Each atom $d_{i\otimes j}\in R^{b\times m}$ of the adaptive dictionary $D_i$ is defined as follows,

$$d_{i\otimes j}=u_{i,j}{v}_{i,j}^T, j=1,2,3,\dots ,n_1$$

(7)

In this way, the adaptive dictionary $D_i$, corresponding to each image group can be represented as follows:

$$D_i=\left[d_{i\otimes 1},d_{i\otimes 2}\dots ,d_{i\otimes n_1}\right]$$

(8)

The singular value decomposition is only needed to be performed on each group once to learn the adaptive dictionary corresponding to the image group. After the dictionary is learned, the sparse representation model is updated and then the $L_1$ norm minimization is solved. Finally, the split Bregman iteration (SBI) algorithm [13] is used for iterative optimization to obtain the final restoration result.

3. Proposed Multi-Color Channels Based GSR Model

The image restoration algorithm based on the GSR model has achieved good results in applications. However, most image restoration algorithms based on the GSR model transform the image into YCbCr space when processing color images, and only process the Y channel. Finally, the processed Y channel image is combined with other unprocessed channels to obtain the final restoration image. Although the Y channel of the image is formed by the integration of the three channels of a RGB color image in a certain proportion, it cannot preserve the maximum fidelity of the original R, G and B channels. This paper proposes a new GSR model based on multi-color channels, which processes three color channels of a RGB color image at the same time.

3.1. Construction of the Proposed Model

3.1.1. Construction of Image Groups

For an input degraded image $y$, the image group $Y_i{}^{Ch}$, $Ch\in \left\{R, G, B\right\}$, is constructed based on the multi-color channel image, where $R$, $G$ and $B$ denote the corresponding image channels. The construction of each channel image group $Y_i{}^{Ch}$ is the same as that of $Y_i$ in Section 2.2. Given a RGB image, in each channel, $m$ similar patches of the repairing patch with the size of $\sqrt{b}\times \sqrt{b}$ are found in the training window with the size of $W\times W$. $Y_i{}^R$, $Y_i{}^G$, and $Y_i{}^B$ denote the image groups corresponding to the $R$, $G$ and $B$ channels, respectively. The three image groups form the total image group $Y_i{}^C$ as Formula (9).

$$Y_i{}^C=\left[\begin{array}{l}{Y}_{i,1}^R Y_{i,2}^R \dots Y_{i,m}^R\\ Y_{i,1}^G Y_{i,2}^G \dots Y_{i,m}^G\\ Y_{i,1}^B Y_{i,2}^B \dots Y_{i,m}^B\end{array}\right] \in R^{3b\times m}$$

(9)

$$Y_i=\left[Y_{i,1}^Y Y_{i,2}^Y \dots Y_{i,m}^Y\right]\in R^{b\times m}$$

(10)

The above Formula (10) denotes the image group constructed using the Y channel in a traditional GSR model. The traditional GSR model and its improved models transform the RGB space into YCbCr space and extract the Y component for processing. The advantage of traditional GSR model is that human vision is sensitive to the change of the Y component in YCbCr space, and the Y component requires fewer storage units than the three color channels in RGB space, which is conducive to transmission and processing. However, the Y component cannot be loyal to each color channel in image processing. In this paper, as shown in Equation (9), stacking the image groups of three different color channels R, G and B in a matrix make it possible to tackle the three channels simultaneously. The total image group $Y_i{}^C$ standing for the three color channels can be directly decomposed by singular value decomposition to obtain an adaptive dictionary and the sparse representation model. In the proposed model, the total image group is processed at the same time to avoid the loss of information caused by the transformation from RGB space to YCbCr space. Moreover, the rich information between different color channels can make up for each other [33]. On the other hand, there is no need to set and adjust the parameters of each color channel respectively when processing three color channels simultaneously, compared with separately processing the image group of each color channel to fuse different results to form the final combination results. Therefore, the proposed construction of image groups is more accurate that of the traditional GSR model. The advantages of the proposed multi-color-channels-based GSR model are low computational cost, low time cost and high efficiency.

3.1.2. Proposed Sparse Representation Model

After constructing a multi-color channel image group $Y_i{}^C$, a sparse representation model can be established based on the image group. By taking the traditional model Formula (5) and $Y_i{}^C$ in Formula (9) in the regularization framework of Formula (2), the group sparse representation model based on multi-color channels of the proposed model in this paper is obtained.

$$\widehat{\beta }=\arg \underset{\beta }{\min }\frac{1}{2}\Vert y-D^C{\beta }^C\Vert +\lambda {\displaystyle \sum _{i=1}^n{\Vert {\beta }_i^C\Vert }_1}$$

(11)

Here, $D^C$ denotes all the adaptive dictionaries obtained via performing singular value decomposition on ${\left\{Y_i^C\right\}}_{i=1}^n$, i.e., $D^C=\left[D_1^C,D_2^C,\dots ,D_n^C\right]$. ${\beta }^C$ represents the sparse coefficient of all multi-color channel image groups ${\left\{Y_i^C\right\}}_{i=1}^n$, i.e., ${\beta }^C=\left[{\beta }_1^C,{\beta }_2^C,\dots ,{\beta }_n^C\right]$. It can be seen from Formula (11) that the proposed model simultaneously processes the image groups of multi-color channels.

3.1.3. Adaptive Dictionaries

An important problem of the proposed multi-color-channels-based GSR model is the selection of the dictionary. The dictionary is usually learned from natural images. Compared with the dictionaries of traditional analysis and design [33], such as wavelet, curvelet curve and discrete cosine transform, the dictionary directly learned from images can better adapt to the local structure of images, enhance the sparsity, and hence obtain higher performance. For example, the well-known K-SVD over-complete dictionary not only preserves the local structure of images, but also provides excellent denoising results [20]. However, the sparse representation on the over-complete dictionary is very computationally expensive and easily leads to artifacts in the image restoration [34]. In order to better adapt to the local structure of images, an adaptive dictionary $D_i{}^C$ is learned for each group $Y_i{}^C$. The learning procedures of the adaptive dictionary $D_i{}^C$ are similar to those depicted in Section 2.3. Firstly, the bilinear interpolation algorithm is used to initialize the degraded image $y$, and the estimation group $r_i^C$ as in Formula (12) is obtained from the initialized image. Then singular value decomposition is performed on the estimation group, and the adaptive dictionary $D_i{}^C$ of each group is learned through the estimation group $r_i^C$.

$$r_i{}^C=\left[\begin{array}{l}{r}_{i,1}^R r_{i,2}^R \dots r_{i,m}^R\\ r_{i,1}^G r_{i,2}^G \dots r_{i,m}^G\\ r_{i,1}^B r_{i,2}^B \dots r_{i,m}^B\end{array}\right]\in R^{3b\times m}$$

(12)

3.2. Implementation of the Proposed Method

3.2.1. Solution of the Coefficients

In this paper, an alternating minimization algorithm [15] is used to solve the minimization problem of Equation (11). Solving the GSR model based on multi-color channels is to solve the ${\beta }_i$ minimization problem in Equation (11). This subsection provides implementation details of solving the ${\beta }_i$ minimization problem. Before solving the model proposed in this paper, the following theorem [13] is introduced.

Theorem 1. 

Let $x, y\in R^N$, and $X_i, Y_i\in R^{b\times m}$. $e\in R^N$ is the error vector, i.e., $e=x-y$. $e_j$ is the jth element of the error vector. Suppose that $e_j$ is independent and comes from a Gaussian distribution with zero mean and variance ${\sigma }_s^2$. Then for any $\varpi >0$, we have the following property to describe the relation between $\frac{1}{N}{\Vert x-y\Vert }_2^2$ and $\frac{1}{S}{{\displaystyle {\sum }_{i=1}^n\Vert X_i-Y_i\Vert }}_2^2$, that is

$$\underset{\begin{array}{l}N\to \infty \\ S\to \infty \end{array}}{\lim }P\left(\left|\frac{1}{N}{\Vert x-y\Vert }_2^2-\frac{1}{S}{{\displaystyle {\sum }_{i=1}^n\Vert X_i-Y_i\Vert }}_2^2\right|<\varpi \right)=1$$

(13)

where $P()$ represents the probability and $S=b\times m\times n$. (The proof is referred to in [13]).

Based on Theorem 1, Formula (11) is transformed into the following formul:

$${\left\{{\widehat{\beta }}_i{}^C\right\}}_{i=1}^n=\arg \underset{{\left\{{\beta }_i{}^C\right\}}_{i=1}^n}{\min }{\displaystyle \sum _{i=1}^n\left(\frac{1}{2}{\Vert Y_i{}^C-D_i{}^C{\beta }_i{}^C\Vert }_2^2+\tau {\Vert {\beta }_i{}^C\Vert }_1\right)}$$

(14)

where $\tau =\frac{\lambda S}{N}$, and $S=b\times m\times n$. Formula (14) may be tackled as the solution to the traditional GSR model, where the problem is divided into n sub-problems corresponding to each $Y_i{}^C$. In order to adapt to the local structure of the images, each $Y_i{}^C$ corresponds to a learning adaptive dictionary $D_i{}^C$. Formula (14) is transformed into

$$\begin{array}{ll}\left\{{\widehat{\beta }}_i{}^C\right\}& =\arg \underset{{\left\{{\beta }_i{}^C\right\}}_{i=1}^n}{\min }{\displaystyle \sum _{i=1}^n\left(\frac{1}{2}{\Vert Y_i{}^C-D_i{}^C{\beta }_i{}^C\Vert }_2^2+\tau {\Vert {\beta }_i{}^C\Vert }_1\right)}\\ & =\arg \underset{{\left\{{\beta }_i{}^C\right\}}_{i=1}^n}{\min }{\displaystyle \sum _{i=1}^n\left(\frac{1}{2}{\Vert r_i{}^C-D_i{}^C{\beta }_i{}^C\Vert }_2^2+\tau {\Vert {\beta }_i{}^C\Vert }_1\right)}\\ & =\arg \underset{{\left\{{\beta }_i{}^C\right\}}_{i=1}^n}{\min }{\displaystyle \sum _{i=1}^n\left(\frac{1}{2}{\Vert {\gamma }_i{}^C-{\beta }_i{}^C\Vert }_2^2+\tau {\Vert {\beta }_i{}^C\Vert }_1\right)}\end{array}$$

(15)

where $Y_i{}^C=H{r}_i{}^C$ and $r_i{}^C=D_i{}^C{\gamma }_i{}^C$. The minimization problem of the Formula (14) is simplified to the solution to the problem of Formula (15). Different from the traditional GSR model, the proposed method obtains the coefficients via a soft threshold [35] as Formula (16).

$${\widehat{\beta }}_i{}^C=soft\left({\gamma }_i{}^C,\sqrt{2\tau }\right), \forall i$$

(16)

3.2.2. Adaptive Parameters

In general, $\lambda$ is set to a fixed value empirically. However, a fixed value cannot guarantee the stability of the whole algorithm. In order to overcome this challenge, an adaptive parameter-setting scheme is used to make the proposed multi-color-channels-based GSR model more stable and practical. Using spatially adaptive Laplacian priors, the regularization parameters adaptively update in each iteration [36]:

$${\lambda }_i=\frac{2\sqrt{2}{\sigma }_w^2}{{\sigma }_i+\epsilon }$$

(17)

Here ${\sigma }_i$ is the ith local estimated variance, and $\epsilon$ is a sufficiently small constant to guarantee a no-zero denominator. The likelihood estimation [36] of ${\sigma }_i$ can be represented as

$${\widehat{\sigma }}_i=\sqrt{\max \left({\widehat{\gamma }}_i^2/m-{\sigma }_m^2,0\right)}$$

(18)

where ${\widehat{\gamma }}_i$ denotes the singular value of the estimation $r_i^C$, and $m$ is the number of similar patches. The proposed method adopts an iterative regularization strategy [8] to alternatively update the estimation of the noise variance ${\sigma }_w$ and the signal variance ${\sigma }_i$.

$${\widehat{\sigma }}_w^k=\kappa \sqrt{{\sigma }_w^2-{\Vert y-y^{k+1}\Vert }_2^2}$$

(19)

$${\widehat{\sigma }}_i{}^{k+1}=\sqrt{\max \left({\left({\widehat{\gamma }}_i^k\right)}^2/m-{\left({\sigma }_m^k\right)}^2,0\right)}$$

(20)

Here $\kappa$ is the scale factor controlling the noise variance, and $k$ is the number of iterations. The complete description of the proposed GSR model based on multi-color channels is shown in the following Algorithm 1.

Algorithm 1 The proposed multi-color-channels-based GSR model

Procedures of multi-color-channels-based GSR model

Initialization: set ${\widehat{x}}^{(0)}=y$, $\lambda ={\lambda }_0$; For k = 1, …, Iter do Construct image group $Y_i{}^C$. For Each group $Y_i{}^C$ do Construct $D_i{}^C$ via SVD Update $\lambda$ via (17) Update ${\widehat{\beta }}_i{}^C$ via (16) Update ${\widehat{X}}^C{}_i=D_i{}^C{\widehat{\beta }}_i{}^C$ end for end for Output: the final restored image $\widehat{x}$

4. Experimental Results

This section provides lots of experimental results to verify the effectiveness of the proposed GSR model based on multi-color channels. In the restoration experiments, four types of corrupted images corresponding to different pixel missing rates are used as examples. The selected four pixel missing rates are 80%, 70%, 60%, and 50%. The pixels are lost randomly under different missing rates. The size of image block, $\sqrt{b}\times \sqrt{b}$, is set to $5\times 5$, the overlap width between adjacent blocks is set to 1 pixel, the size of search window, $W\times W$, is set to $20\times 20$, the number of similar blocks, $m$, is set to $75$, and the constant, $\epsilon$, is set to $0.2$.

The quality of the restored images is evaluated according to objective quantitative metrics and subjective visual effects. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [37] are used as objective metrics. The PSNR is calculated as shown in Formulas (21) and (22),

$$MSE=\frac{1}{H\times W}{\displaystyle \sum _{i=0}^{H-1}{\displaystyle \sum _{j=0}^{W-1}{\Vert X(i,j)-Y(i,j)\Vert }^2}}$$

(21)

$$PSNR=10\cdot {\log }_{10}\left(\frac{{\left(2^n-1\right)}^2}{MSE}\right)$$

(22)

Here, $X$ and $Y$ denote the original image and the repaired image, respectively. $H$ and $W$ denote the height and the width of the of the image, respectively. Formula (21) is used to calculate the mean square variance (MSE) of the original image and the repaired image. Formula (22) is used for calculating PSNR, where $n$ is the number of bits per pixel. The larger the value of PSNR, the smaller the distortion. The calculation of SSIM is shown in Formulas (23)–(27),

$$l(X,Y)=\frac{2u_X{u}_Y+C_1}{u_X^2+u_Y^2+C_1}, c(X,Y)=\frac{2{\sigma }_X{\sigma }_Y+C_2}{{\sigma }_X^2+{\sigma }_Y^2+C_2}, s(X,Y)=\frac{2{\sigma }_{XY}+C_3}{{\sigma }_X{\sigma }_Y+C_3}$$

(23)

$$u_X=\frac{1}{H\times W}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^{W}X\left(i,j\right)}}$$

(24)

$${\sigma }_X^2=\frac{1}{H\times W-1}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^W{\left(X\left(i,j\right)-u_X\right)}^2}}$$

(25)

$${\sigma }_{XY}=\frac{1}{H\times W-1}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^W((X(i,j)-u_X)(Y(i,j)-u_Y))}}$$

(26)

$$SSIM(X,Y)=l(X,Y)\cdot c(X,Y)\cdot s(X,Y)$$

(27)

SSIM is measured according to brightness, $l$, contrast, $c$, and image structure, $s$, in Formula (23). $H$ and $W$ denote the height and the width of the original image $X$ and the repaired image $Y$. $u_X$ and $u_Y$ denote the mean value of $X$ and $Y$. ${\sigma }_X$ and ${\sigma }_Y$ denote the variance of $X$ and $Y$, respectively, and ${\sigma }_{XY}$ denotes the covariance of $X$ and $Y$. $C_1$, $C_2$ and $C_3$ are constants to be introduced to avoid the denominator being zero. The range of SSIM is $\left[0, 1\right]$. The SSIM metric is closer to a human’s subjective feelings. The larger SSIM value indicates that the two images are more similar and the restoration effect is better.

In this paper, 10 images as shown in Figure 2 below are selected as test images. The quantitative experimental results and visual restoration effects given in the subsection are obtained on these images. The names of the images are Bahoon, Butterfly, Cowboy, Flower, Girl, Lake, Lena, Mickey, Mural, and Nanna, respectively.

4.1. Objective Assessment

The proposed model in this paper is compared with 11 state-of-the-art models. These models used for comparison include SALSA [38], BPFA [4], GSR [13], NGS [39], TSLRA [40], JPG-SR [14], GSRC-NLP [27], MAR-LOW [41], HSSE [42], IR-CNN [17], and IDBP [43]. Among the comparison models, GSR [13], JPG-SR [14], GSRC-NLP [27], and HSSE [42] are based on traditional GSR or improved GSR models, which are the same type as the proposed multi-color-channels-based GSR model in this paper. In order to comprehensively evaluate the performance of the proposed model for image restoration, the proposed model is also compared with non-GSR-based image restoration models. The BPFA [4] model uses the nonparametric Bayesian dictionary learning method for image sparse representation. The SALSA [38] model proposes an algorithm belonging to the extended Lagrangian method family to deal with constrained problems. The NGS [39] model proposes a new gradient regularization framework using nonlocal similarity. The TSLRA [40] and MAR-LOW [41] are the type of low rank minimization methods. The MAR-LOW [41] combines spatial autoregression with low rank minimization methods and achieves excellent results. The IR-CNN [17] and IDBP [43] are deep-learning-based methods.

Table 1. PSNR values of the proposed method and some state-of-the-art methods with pixel missing rates 80% and 70%.

Pixel Missing Rate	Methods	Bahoon	Butterfly	Girl	Lena	Mural	Nanna	Average
80%	SALSA	24.41	21.91	23.03	27.32	22.45	23.20	23.72
	BPFA	23.25	21.06	22.47	27.56	21.10	22.38	22.97
	GSR	24.57	26.03	25.50	31.41	26.01	25.24	26.46
	NGS	24.89	23.85	24.27	28.87	23.78	24.58	25.04
	TSLRA	25.49	25.32	25.15	30.58	25.21	25.55	26.22
	JPG-SR	25.40	26.58	25.55	31.25	26.29	25.92	26.83
	GSRC-NLP	25.55	26.78	26.02	30.51	26.56	26.17	26.93
	MAR-LOW	26.40	30.86	29.21	33.83	29.66	29.68	29.94
	HSSE	25.57	26.61	26.12	31.83	26.23	26.27	27.11
	Proposed	26.43	30.90	29.28	33.85	29.67	29.70	29.97
70%	SALSA	25.71	24.85	24.99	29.62	24.69	25.34	25.87
	BPFA	24.57	23.95	24.71	30.37	23.34	24.47	25.24
	GSR	26.17	28.92	27.86	33.54	28.46	27.89	28.81
	NGS	26.08	26.36	26.18	30.77	26.29	26.35	27.01
	TSLRA	26.72	27.76	27.09	32.64	27.22	27.32	28.13
	JPG-SR	26.80	29.24	27.96	33.40	28.50	28.24	29.02
	GSRC-NLP	26.98	29.47	28.20	33.85	28.71	28.51	29.29
	MAR-LOW	28.70	33.82	32.14	36.21	32.26	32.75	32.65
	HSSE	26.87	29.29	28.25	33.86	28.57	28.49	29.22
	Proposed	28.72	33.90	32.26	36.23	32.30	32.77	32.70

and

Table 2. PSNR values of the proposed method and some state-of-the-art methods with pixel missing rates 60% and 50%.

Pixel Missing Rate	Methods	Bahoon	Butterfly	Girl	Lena	Mural	Nanna	Average
60%	SALSA	26.78	27.02	26.79	31.42	26.50	26.97	27.58
	BPFA	25.82	26.06	26.68	32.38	25.17	26.14	27.04
	GSR	27.74	31.09	29.54	35.80	29.98	30.13	30.71
	NGS	27.29	28.37	27.83	32.81	27.99	28.06	28.73
	TSLRA	27.92	29.42	28.91	34.26	29.07	29.17	29.79
	JPG-SR	28.14	31.15	29.87	35.44	30.15	30.37	30.85
	GSRC-NLP	28.31	31.46	30.17	35.95	30.33	30.59	31.14
	MAR-LOW	30.78	36.33	34.58	38.30	34.64	35.44	35.01
	HSSE	28.25	31.54	30.20	35.93	30.31	30.53	31.13
	Proposed	30.80	36.41	34.70	38.34	34.70	35.55	35.08
50%	SALSA	27.98	29.03	28.32	33.26	28.17	28.57	29.22
	BPFA	27.13	28.16	28.46	34.15	27.20	28.17	28.88
	GSR	29.42	32.78	31.93	37.64	31.73	32.16	32.61
	NGS	28.49	30.28	29.60	34.56	29.88	29.71	30.42
	TSLRA	29.15	31.01	30.48	35.52	30.62	30.87	31.28
	JPG-SR	29.61	32.83	31.77	37.18	31.72	32.21	32.55
	GSRC-NLP	29.75	33.02	31.95	37.64	31.88	32.36	32.77
	MAR-LOW	32.88	38.70	37.01	40.24	36.98	38.22	37.34
	HSSE	29.79	33.29	32.19	37.84	31.95	32.50	32.93
	Proposed	32.91	38.80	37.15	40.30	37.10	38.37	37.44

show the PSNR values of the restoration results on different images under the four pixels missing rates of 80%, 70%, 60% and 50% of the proposed model and the comparison models. In the tables, the best results are highlighted in bold.

Table 3. PSNR and SSIM values of the proposed method and two deep-learning-based methods with four different pixel missing rates.

Images	Pixel Missing Rate 80%			Pixel Missing Rate 70%			Pixel Missing Rate 60%			Pixel Missing Rate 50%
	IR-CNN	IDBP	Proposed	IR-CNN	IDBP	Proposed	IR-CNN	IDBP	Proposed	IR-CNN	IDBP	Proposed
Cowboy	25.47	24.43	29.38	27.54	27.15	32.31	29.95	28.97	35.36	31.96	31.40	38.12
	0.8622	0.8400	0.9343	0.9163	0.8955	0.9621	0.9479	0.9273	0.9770	0.9660	0.9522	0.9854
Girl	25.36	25.03	29.28	27.78	27.45	32.26	30.10	29.32	34.70	32.10	31.09	37.15
	0.8135	0.7999	0.9212	0.8910	0.8698	0.9549	0.9327	0.9102	0.9720	0.9564	0.9371	0.9823
Flower	28.41	28.12	32.14	30.77	30.49	35.11	32.97	32.16	37.59	35.28	34.15	40.00
	0.8586	0.8413	0.9381	0.9140	0.8943	0.9652	0.9460	0.9264	0.9786	0.9661	0.9508	0.9864
Lake	25.24	25.39	28.86	27.59	27.87	31.33	29.75	29.53	33.76	31.80	31.47	35.84
	0.8278	0.8203	0.9115	0.8914	0.8805	0.9433	0.9302	0.9155	0.9623	0.9535	0.9421	0.9743
Mickey	26.45	25.40	29.57	29.66	28.69	32.24	31.82	31.18	34.51	34.25	33.14	36.62
	0.8671	0.8436	0.9260	0.9220	0.9023	0.9532	0.9474	0.9319	0.9687	0.9637	0.9517	0.9787
Mural	25.75	25.26	29.67	28.73	27.68	32.30	30.58	29.79	34.70	32.20	31.35	37.10
	0.7923	0.7694	0.8960	0.8721	0.8388	0.9393	0.9091	0.8824	0.9633	0.9361	0.9148	0.9780
Average	26.11	25.61	29.82	28.68	28.22	32.59	30.86	30.16	35.10	32.93	32.10	37.47
	0.8369	0.8191	0.9211	0.9011	0.8802	0.9530	0.9356	0.9156	0.9703	0.9570	0.9415	0.9808

shows the SSIM values of the results on different images under the four pixel missing rates of the proposed model and the comparison models, and the best results are highlighted in bold. Table 1 and Table 2 show the comparison results between the proposed model and the traditional models, and Table 3 shows the comparison results between the proposed model and the deep-learning-based models. In order to reduce the burden of experiments, the result data of NGS [39], TSLRA [40], and HSSE [42], provided in HSSE [42], are used here directly to avoid repetitive implementations. The experimental data of the deep learning methods IR-CNN [17] and IDBP [43] are directly from JPG-SR [14] and GSRC-NLP [27].

In addition to PSNR and SSIM values of each test image, the average PSNR and SSIM values of all test images under different pixel missing rates corresponding to different models are calculated for overall comparison. It can be seen from Table 1, Table 2 and Table 3 that the proposed multi-color-channels-based GSR model in this paper obtains better results than other competitive models. Firstly, the proposed model is compared with the non-GSR-based image restoration models. In terms of the average metric values, the proposed model is 7.20 dB higher than the SALSA model, 7.77 dB higher than the BPFA model, 6.00 dB higher than the NGS model, 4.94 dB higher than the TSLRA model, and 0.06 dB higher than the MAR-LOW model. Among the GSR based models, the JPG-SR method combines the PSR model with the GSR model, and it not only preserves the advantages of the two models, but also reduces their disadvantages, and therefore the sparse representation model is unified. The GSRC-NLP method introduces group sparse residual constraints to the traditional GSR model to approximate the group sparsity of the original group and improve the quality of image restoration. The HSSR model combines the external GSR model with the internal GSR model using the NSS prior knowledge of degraded images and external data sets to optimize the image. However, the above three GSR-based-models and many other improved models not compared in this paper transform the color image into YCbCr space, and only extract the Y channel for processing. In this way, image processing cannot be loyal to each color channel of the original color image, which makes the effect of image restoration poor. The proposed method utilizes three color channels in image restoration to obtain better results, and Table 1 and Table 2 prove this conclusion. Compared with the aforementioned three GSR-based models in terms of average PSNR metric values, the proposed model is 4.15 dB higher than the traditional GSR model, 3.99 dB higher than the JPG-SR model, 3.77 dB higher than the GSRC-NLP model, and 3.70 dB higher than the HSSE model. It can be concluded from Table 1 and Table 2 that the PSNR of the proposed model in this paper on the images with 80%, 70%, 60%, and 50% pixel missing rates are higher than the nine state-of-the-art comparison methods.

Table 3 provides the comparison results of the proposed model with deep-learning-based IR-CNN and ISBP models. In terms of average metric values, the PSNR and SSIM values of the proposed model are 4.1 dB and 0.049 higher than those of the IR-CNN model, and 4.72 dB and 0.067 higher than those of the IDBP model, respectively. It can be seen from the PSNR and SSIM values in Table 3 that the proposed model based on multi-color channels is better than the IR-CNN and IDBP models. The experimental results in Table 1, Table 2 and Table 3 show that the model proposed in this paper is effective and can obtain good restoration results compared with the state-of-the-art models.

4.2. Subjective Assessment

In order to further demonstrate the effectiveness of the proposed multi-color-channels-based GSR model, Figure 3 and Figure 4 respectively show the visual butterfly and girl restoration results of the proposed model and some state-of-the-art models with an 80% pixel missing rate. As can be seen from Figure 3 and Figure 4, the SALSA model cannot restore sharp edges and fine details. The BPFA model can have better visual effects than the salsa model, but it still has undesirable visual artifacts, such as ringing artifacts in Figure 4d. The GSR, JPG-SR and GSRC-NLP models produce over-smoothing effects. Compared with the model proposed in this paper, the MAR-LOW has a similar visual effect, which is not easy to distinguish with the naked eye. However, according to the experimental results in Table 1, Table 2 and Table 3, the proposed model has better objective evaluation results. In general, the visual effects of Figure 3 and Figure 4 show that the GSR model based on multi-color channels proposed in this paper performs best in image restoration. It not only preserves fine details, but also effectively removes visual artifacts.

5. Conclusions

In order to improve the performance of the traditional GSR model in image restoration, the paper extends the GSR model to a new multi-color-channels-based case, and process R, G, and B color channels of an image simultaneously. The problem of image restoration is transformed into how to use multi-color channels for sparse representation. The alternating minimization method with an adaptive parameter setting strategy is used to solve the proposed multi-color-channels-based model. The proposed method is compared with state-of-the-art methods. The experimental results show that the proposed model can obtain good results in terms of visual effects and quantitative metrics.