Multi-Frame Blind Super-Resolution Based on Joint Motion Estimation and Blur Kernel Estimation

Abstract

Multi-frame super-resolution makes up for the deficiency of sensor hardware and significantly improves image resolution by using the information of inter-frame and intra-frame images. Inaccurate blur kernel estimation will enlarge the distortion of the estimated high-resolution image. Therefore, multi-frame blind super resolution with unknown blur kernel is more challenging. For the purpose of reducing the impact of inaccurate motion estimation and blur kernel estimation on the super-resolved image, we propose a novel method combining motion estimation, blur kernel estimation and super resolution. The confidence weight of low-resolution images and the parameter value of the motion model obtained in image reconstruction are added to the modified motion estimation and blur kernel estimation. At the same time, Jacobian matrix, which can better describe the motion change, is introduced to further correct the error of motion estimation. Based on the results acquired from the experiments on synthetic data and real data, the superiority of the proposed method over others is obvious. The reconstructed high-resolution image retains the details of the image effectively, and the artifacts are greatly reduced.

1. Introduction

People increasingly need high-resolution (HR) images in order to obtain better visual effects or for subsequent image processing. However, improving image resolution in hardware is not easy because of the limitation of the sensor manufacturing process and the high cost of it. In recent years, super-resolution (SR) technology without changing the hardware environment has developed rapidly. It can fall into two categories: non-blind super-resolution or blind super-resolution, according to whether the blur kernel is known or not.

Multi-frame super-resolution (MFSR) refers to the reconstruction of the HR image from multi-frame low-resolution (LR) images in the same scene. This paper mainly studies multi-frame blind super-resolution. It is often applied in satellite remote sensing [1], low-quality video surveillance [2] and mobile phone videos [3], etc.

Influenced by unknown motion, blur and noise, the quality of super-solved HR images using existing multi-frame blind super-resolution methods is often not ideal in practical applications. It is difficult to accurately fuse scene information observed in multi-frame LR images due to noise and blur. In addition, the irregular pixel movement has an impact on the estimated HR image directly.

Coping with the above-mentioned problems, a novel method of multi-frame blind super resolution for joint motion estimation, blur kernel estimation and noise suppression, is proposed. We combine the parameters of image reconstruction, blur kernel estimation and motion estimation while estimating the high-resolution image, blur kernel and motion matrix as accurately as possible. In addition, in order to avoid interference from external data, we directly use the information provided by successive frames to boost the resolution and quality of the image without training and learning in deep learning methods. In the algorithm we propose, two major contributions are made for improving the quality of the estimated HR image:

(1) We combine the confidence weight in the image reconstruction stage to correct the motion estimation error in the iterative process of motion estimation. At the same time, we introduce the Jacobian matrix of motion parameters to reduce the error and register the LR images as much as possible.

(2) We combine the confidence weight in the image reconstruction stage and the corrected motion estimation to correct the blur estimation error in the iterative process of blur kernel estimation. More accurate blur kernel estimation has a direct impact on the effect of image reconstruction.

The paper is organized in the following manner. We introduce the related work of MFSR in Section 2. The observation model of multi-frame blind super-resolution is constructed in Section 3. We specifically describe the proposed algorithm in Section 4 and present the experimental results on synthetic data and real data in Section 5. The research is summarized in Section 6.

2. Related Work of MFSR

Non-blind super-resolution Non-blind super-resolution methods include interpolation-based methods [4], reconstruction-based methods [5,6,7] and learning-based methods [8,9,10]. Michel Bätz et al. [4] designed an extended dual weighting scheme. This method was based on Voronoi tessellation which relied on motion confidence weights and distance weights. X. Liu et al. [6] developed an adaptive bilateral total variation (ABTV) regularization method and used the half-quadratic estimation to take error norm adaptively. T. Nascimento et al. [7] proposed a method by combining Demons registration and regularized Bayesian reconstruction. S. Lu et al. [8] reconstructed the HR image by learning texture details using local linear embedding. T. Kato et al. [9] proposed a method based on sparse coding by optimizing a single objective function. K. Ning et al. [10] proposed a method to obtain more high-frequency detail information. This method fused adjacent frames by combining the registration module based on Generative Adversarial Networks (GAN). Non-blind super-resolution methods are effective when the blur kernel is known, or the blur is not considered. When the blur kernel does not match, the quality of the super-resolved image is very poor. Figure 1 shows the influence of the blur kernel mismatch on the SR result, where ${\mathsf{\sigma }}_{\mathit{blur}}$ denotes the blur kernel width. If the blur kernel width is smaller than the real value, the SR result is over-smoothing, and the high frequency information is blurred. If the blur kernel width is larger than the real value, the SR result shows obvious ringing artifacts due to over-enhancing of the high-frequency edges.

Blind super-resolution Multi-frame blind super-resolution can fall into two categories according to whether reconstruction of the HR image and the deblurring are separated independently. E. Faramarzi et al. [11] proposed a method to first estimate the blur from some enhanced edges iteratively and then estimated the reconstructed HR image by the non-blind super-resolution method using the estimated blur kernel. Q. Qian and B.K. Gunturk [12] proposed a blind super-resolution method, which was first to deblur each LR image by using the estimated blur kernel and then to apply MFSR to the deblurred images. A. Buades et al. [13] proposed a method for MFSR with two steps. In the first stage, a non-linear filtering method was proposed to produce an unsampled but still blurry image using the inter-frame motion and exploiting spatiotemporal redundancy. In the second stage, a variational method for single image deblurring was proposed. L. Huang and Y. Xia [14] first proposed an accurate blur kernel estimation-based matrix decomposition method and then presented a matrix-variable optimization method for blind super resolution. Such methods that reconstruct the HR image by using the first estimated blur kernel have a simple operation flow and the blur kernel is estimated only one. However, they make it easy to magnify the error caused by inaccurate estimation of the blur kernel, which leads to the unsatisfactory effect of the reconstructed HR image.

Another kind of multi-frame blind super-resolution is to estimate the blur kernel and reconstruct the HR image simultaneously. H. Zhang et al. [15] proposed a method to perform image alignment, deblurring and MFSR by an assumption of projective motion path. Ce Liu et al. [16] proposed a Bayesian method for estimating the motion, blur kernel and noise simultaneously while reconstructing the original high-resolution frame. Z. Ma et al. [17] proposed an expectation maximization algorithm to guide motion blur estimation and high-resolution image reconstruction. Zhen Lv et al. [18] proposed a novel method of joint image registration and Point Spread Function (PSF) estimation so as to produce the reconstructed HR image. This method can be formulated as a convex optimization problem. Z. Shi et al. [19] proposed a blind MFSR algorithm based on combining ANN learning and non-subsampled Contourlet directional image representation. T. Honda et al. [20] proposed a joint deblurring, denoising and super-resolution method for multi-frame RGB/NIR imaging. Such methods are suitable for complex environments. However, they are not easy to converge, and the quality of the reconstructed HR image is not ideal because there are many uncertain factors to be considered at the same time. In order to accelerate the convergence speed of the algorithm and obtain better image quality, we propose the confidence weight of LR images obtained in image reconstruction, and the parameter value of the motion model are added to the modified motion estimation and blur kernel estimation.

3. Observation Model of Multi-Frame Blind Super-Resolution

Classical MFSR methods assume that these low-resolution images are obtained from an unknown high-resolution image after warping, blurring, down sampling and noise. Therefore, the observation model of multi-frame blind super-resolution can be expounded as

$${\mathit{y}}_{\mathit{n}}=\mathit{D}{\mathit{K}}_{\mathit{n}}{\mathit{F}}_{\mathit{n}}\mathit{x}+{\mathit{\epsilon }}_{\mathit{n}},$$

(1)

where ${\mathit{y}}_{\mathit{n}}$ represents the $nth$ LR image, $n=1,2,\dots ,N$. $N$ is the number of LR images. $\mathit{D}$ represents the down sampling matrix. ${\mathit{K}}_{\mathit{n}}$ represents the blur matrix of the $nth$ LR image. ${\mathit{F}}_{\mathit{n}}$ represents the sub-pixel displacement motion matrix of the $nth$ LR image. $\mathit{x}$ is the HR image. ${\mathit{\epsilon }}_{\mathit{n}}$ is the noise of the $nth$ LR image. The observation model of MFSR is shown in Figure 2. Mathematically, $\mathit{x}$ needs to be calculated, ${\mathit{y}}_{\mathit{n}}$ and $\mathit{D}$ are known, ${\mathit{K}}_{\mathit{n}}$, ${\mathit{F}}_{\mathit{n}}$ and ${\mathit{\epsilon }}_{\mathit{n}}$ are unknown in Equation (1). The solution is unstable and not unique. Therefore, MFSR is an ill-posed inverse problem. It is unlikely that the calculated HR image is completely consistent with the real HR image, due to various unknown factors. Super resolution reconstruction can only make the reconstructed HR image quality as good as possible.

${\mathit{K}}_{\mathit{n}}$, ${\mathit{F}}_{\mathit{n}}$, $\mathit{x}$ can be estimated in the Bayesian MAP framework as

$$\begin{gathered} \left\{{\displaystyle \hat{\mathit{K}}},{\displaystyle \hat{\mathit{F}}},{\displaystyle \hat{\mathit{x}}}\right\}=\underset{\mathit{K},\mathit{F},\mathit{x}}{\mathrm{argmax}}p(\mathit{K},\mathit{F},\mathit{x}|{\mathit{y}}_{\mathbf{1}},{\mathit{y}}_{\mathbf{2}},\dots ,{\mathit{y}}_{\mathit{N}}) \\ =\underset{\mathit{K},\mathit{F},\mathit{x}}{\mathrm{argmax}}p\left({\mathit{y}}_{\mathbf{1}},{\mathit{y}}_{\mathbf{2}},\dots ,{\mathit{y}}_{\mathit{N}}|\mathit{K},\mathit{F},\mathit{x}\right)p\left(\mathit{x}\right)p\left(\mathit{K}\right)p\left(\mathit{F}\right), \end{gathered}$$

(2)

where $\mathit{p}\left(\mathit{x}\right)$ represents the image prior, $\mathit{p}\left(\mathit{K}\right)$ represents the blur prior, $\mathit{p}\left(\mathit{F}\right)$ represents the motion prior.

4. Proposed Multi-Frame Blind Super-Resolution Algorithm

We employ the alternating minimization (AM) method for reconstructing the HR image $\mathit{x}$, motion matrix $\mathit{F}$ and the blur kernel matrix $\mathit{K}$ iteratively. To overcome the poor convergence of a simple alternating minimization method, we propose a method of combining the estimation of motion parameters, blur kernel estimation and parameters in image reconstruction, and finally outputting the estimated high-resolution image. In the image reconstruction stage, a coarse-to-fine strategy is used to enlarge to the required scale factor. We modify the motion matrix by combining the confidence weight in the image reconstruction stage and the Jacobian matrix of motion parameters. We use the modified motion matrix and the confidence weight of each LR image in the image reconstruction stage to modify the blur matrix.

4.1. Preparation and Initial Settings

The middle frame of LR images is selected as the reference frame. We use the inter-frame information to calculate the relationship between each frame and the reference frame. The initial values of blur kernel, motion estimation and the HR image are set first. Bicubic interpolation is used for the initialization of the estimated HR image. Enhanced correlation coefficient (ECC) optimization [21] is employed for the initialization of motion estimation. The blind image deblurring method [22] is used for the initialization of the blur kernel estimation.

4.2. Image Reconstruction

$\mathit{K}$ and $\mathit{F}$ are fixed in this image reconstruction step, the estimated HR image $\mathit{x}$ can be computed in the following way:

$${\displaystyle \hat{\mathit{x}}}=\underset{\mathit{x}}{\mathrm{argmax}}p({\mathit{y}}_{\mathbf{1}},{\mathit{y}}_{\mathbf{2}},\dots ,{\mathit{y}}_{\mathit{N}}|\mathit{K},\mathit{F},\mathit{x})p\left(\mathit{x}\right),$$

(3)

We adopt the regularization technique to estimate the HR image $\mathit{x}$ which can be expounded as

$${\displaystyle \hat{\mathit{x}}}=\underset{\mathit{x}}{argmin}\left\{{{\displaystyle \sum }}_{n=1}^N\mathit{W}·h(\mathit{D}{\mathit{K}}_{\mathit{n}}{\mathit{F}}_{\mathit{n}}\mathit{x}-{\mathit{y}}_{\mathit{n}})+\lambda \gamma \left(\mathit{x}\right)\right\},$$

(4)

where $\mathit{W}$ is the confidence weights matrix, $h\left(\mathit{r}\right)$ is the Huber loss function as the data fidelity term, $\gamma \left(\mathit{x}\right)$ is the regularization term with respect to $\mathit{x}$, $\lambda$ is the trade-off parameter between the fidelity term and the regularization term. We use cross validation [5] to determine the value of $\lambda$.

The larger the observation error, the smaller the corresponding confidence weight. This paper estimates the confidence weight matrix $\mathit{W}={\left({\mathit{\beta }}_{\mathbf{1}},\dots ,{\mathit{\beta }}_{\mathit{n}},\dots ,{\mathit{\beta }}_{\mathit{N}}\right)}^T$ by using the method proposed in our previous article [23]. ${\mathit{\beta }}_{\mathit{n}}$ represents the confidence weight of the $nth$ LR image. ${\mathit{\beta }}_{\mathit{n}}$ can be expounded as

$${\mathit{\beta }}_{\mathit{n}}=\{\begin{array}{c}\frac{\mathrm{mean}\left({\mathit{r}}_{\mathit{n}}\right)}{{\mathit{r}}_{\mathit{n}}} if \left|r_{n,i}\right|\le c{\sigma }_f^t\\ \frac{\mathrm{mean}\left({\mathit{r}}_{\mathit{n}}\right)}{{\mathit{r}}_{\mathit{n}}} ·\frac{c{\sigma }_f^t}{\left|r_{n,i}\right|} otherwise, \end{array}$$

(5)

where ${\mathit{r}}_{\mathit{n}}=\mathit{D}{\mathit{K}}_{\mathit{n}}{\mathit{F}}_{\mathit{n}}\mathit{x}-{\mathit{y}}_{\mathit{n}}$ represents the observation error of the nth LR image, $\mathrm{mean}\left({\mathit{r}}_{\mathit{n}}\right)$ is the mean value of ${\mathit{r}}_{\mathit{n}}$. $c$ represents a positive constant. We set $c=2$. ${\sigma }_f^t$ represents the scale parameter of iteration $t$. For discriminating inliers and outliers adaptively, ${\sigma }_f^t$ can be estimated by using the median absolute deviation (MAD) [24]. The equation can be expounded as

$${\sigma }_f^t={\sigma }_0·MAD\left(r^{t-1}|{\beta }^{t-1}\right),$$

(6)

where ${\sigma }_0=1.4826$.

Huber loss function $h\left(\mathit{r}\right)$ is used for the fidelity term. The Huber loss function $h\left(\mathit{r}\right)$ can be defined as

$$h\left(\mathit{r}\right)=\{\begin{array}{c}{r}^2 if \left|r\right|\le \delta \\ 2\delta \left|r\right|-{\delta }^2 otherwise, \end{array}$$

(7)

Bilateral Total Variation (BTV) [25] is used for the regularization term $\gamma \left(\mathit{x}\right)$. The BTV with the image sparsity is used to handle outliers. The equation can be expounded as

$$\gamma \left(\mathit{x}\right)={{\displaystyle \sum }}_{n=-P}^P{{\displaystyle \sum }}_{m=-P}^P{\alpha }^{\left|m\right|+\left|n\right|}\parallel \mathit{x}-S_x^n{S}_y^m\mathit{x}{\parallel }_1,$$

(8)

where $\alpha$ represents the scaled weight parameter. $\alpha$ can be computed by using the method in [19]. $P$ is the size of the sliding window. $S_x^n$ represents moving $n$ pixels in the x direction. $S_y^m$ represents moving $m$ pixels in the y direction.

Scaled Conjugate Gradient (SCG) [26] is employed for the purpose of solving the problem in the Equation (4).

4.3. Motion Estimation

$\mathit{K}$ and $\mathit{x}$ are fixed in this motion estimation step, the motion matrix $\mathit{F}$ can be computed as

$${\displaystyle \hat{\mathit{F}}}=\underset{\mathit{F}}{\mathrm{argmax}}p({\mathit{y}}_{\mathbf{1}},{\mathit{y}}_{\mathbf{2}},\dots ,{\mathit{y}}_{\mathit{N}}|\mathit{K},\mathit{F},\mathit{x})p\left(\mathit{F}\right),$$

(9)

We adopt the affine transformation model described by the parameter ${\mathit{\phi }}_{\mathit{n}}={\left(a_1,a_2,a_3,a_4,t_x,t_y\right)}^T$ with the translation $\mathit{t}={(t_x,t_y)}^T$ and the parameter $a_i\left(i=1,2,3,4\right)$, which represents the rotation, scaling and other changes for the $nth$ LR image ${\mathit{y}}_{\mathit{n}}$.

In object motion, the Jacobian matrix is an important part of instantaneous motion. In order to better describe the sub-pixel motion model ${\mathit{\phi }}_{\mathit{n}}$, we introduce the Jacobian matrix $\mathit{J}\left({\mathit{\phi }}_{\mathit{n}}\right)$ of motion parameters into the $nth$ LR image. $\mathit{J}\left({\mathit{\phi }}_{\mathit{n}}\right)$ can be expounded as

$$\mathit{J}\left({\mathit{\phi }}_{\mathit{n}}\right)=\mathit{D}{\mathit{K}}_{\mathit{n}}\frac{\mathbf{\partial }{\mathit{F}}_{\mathit{n}}}{\mathbf{\partial }{\mathit{\phi }}_{\mathit{n}}},$$

(10)

The derivative of ${\mathit{F}}_{\mathit{n}}$ w. r. t. ${\mathit{\phi }}_{\mathit{n}}$ is computed using the bilinear interpolation [27].

The motion matrix ${\mathit{F}}_{\mathit{n}}$ describe the sub-pixel motion of the $nth$ LR image according to the motion parameter ${\mathit{\phi }}_{\mathit{n}}$ relative to $\mathit{x}$. ${\mathit{F}}_{\mathit{n}}$ can be expressed as

$$\widehat{{\mathit{F}}_{\mathit{n}}}=\underset{{\mathit{F}}_{\mathit{n}}}{argmin} ·\parallel \mathit{J}{\left({\mathit{\phi }}_{\mathit{n}}\right)}^T{\mathit{\beta }}_{\mathit{n}}\mathit{J}\left({\mathit{\phi }}_{\mathit{n}}\right)-\mathit{J}{\left({\mathit{\phi }}_{\mathit{n}}\right)}^T{\mathit{\beta }}_{\mathit{n}}{\left(\mathit{D}{\mathit{K}}_{\mathit{n}}{\mathit{F}}_{\mathit{n}}\mathit{x}-{\mathit{y}}_{\mathit{n}}\right)\parallel }_2^2,$$

(11)

where ${\mathit{\beta }}_{\mathit{n}}$, which is the confidence weight of the $nth$ LR image, is calculated in the image reconstruction stage. $\mathit{J}\left({\mathit{\phi }}_{\mathit{n}}\right)$ is the Jacobian matrix of the $nth$ LR image. We use the Conjugate Gradient (CG) method to work out the Equation (11) in this paper.

4.4. Blur Kernel Estimation

$\mathit{F}$ and $\mathit{x}$ are fixed in this blur kernel estimation step, the blur matrix $\mathit{K}$ can be determined as follows

$$\hat{\mathit{K}}=\underset{\mathit{K}}{\mathrm{argmax}}p({\mathit{y}}_{\mathbf{1}},{\mathit{y}}_{\mathbf{2}},\dots ,{\mathit{y}}_{\mathit{N}}|\mathit{K},\mathit{F},\mathit{x})p\left(\mathit{K}\right),$$

(12)

We estimate the blur kernel by using the estimated HR image $\mathit{x}$ and motion estimation $\mathit{F}$, the blur matrix ${\mathit{K}}_{\mathit{n}}$ of the $nth$ LR image can be expounded as

$$\widehat{{\mathit{K}}_{\mathit{n}}}=\underset{{\mathit{K}}_{\mathit{n}}}{argmin}\left\{{\mathit{\beta }}_{\mathit{n}}·\parallel \mathit{D}{\mathit{K}}_{\mathit{n}}{\mathit{F}}_{\mathit{n}}\mathit{x}-{\mathit{y}}_{\mathit{n}}{\parallel }_1+\mathbf{\beth } \parallel \mathbf{\nabla }{\mathit{K}}_{\mathit{n}}{\parallel }_1\right\},$$

(13)

where ${\mathit{\beta }}_{\mathit{n}}$, which is the confidence weight of the $nth$ LR image, is calculated in the image reconstruction stage. We set

$${\mathit{A}}_{\mathit{n}}={\mathit{F}}_{\mathit{n}}\mathit{x},$$

(14)

Equation (13) can be rewritten as

$$\widehat{{\mathit{K}}_{\mathit{n}}}=\underset{{\mathit{K}}_{\mathit{n}}}{argmin}\left\{{\mathit{\beta }}_{\mathit{n}}·\parallel \mathit{D}{\mathit{K}}_{\mathit{n}}{\mathit{A}}_{\mathit{n}}-{\mathit{y}}_{\mathit{n}}{\parallel }_1+\mathbf{\beth } \parallel \mathbf{\nabla }{\mathit{K}}_{\mathit{n}}{\parallel }_1\right\},$$

(15)

Equation (15) can be worked out through solving the following linear system by the Iterated Reweighted least squares (IRLS) method [28] iteratively:

$$\left[{\mathit{A}}_{\mathit{n}}^{\mathit{T}}{\mathit{D}}^{\mathit{T}}{\mathit{\beta }}_{\mathit{n}}\mathit{D}{\mathit{A}}_{\mathit{n}}+\mathbf{\beth }\left({\mathit{I}}_{\mathit{x}}^{\mathit{T}}{\mathit{W}}_{\mathit{s}}{\mathit{I}}_{\mathit{x}}+{\mathit{I}}_{\mathit{y}}^{\mathit{T}}{\mathit{W}}_{\mathit{s}}{\mathit{I}}_{\mathit{y}}\right)\right]{\mathit{K}}_{\mathit{n}}={\mathit{A}}_{\mathit{n}}^{\mathit{T}}{\mathit{D}}^{\mathit{T}}{\mathit{\beta }}_{\mathit{n}}{\mathit{y}}_{\mathit{n}},$$

(16)

where the matrices ${\mathit{I}}_{\mathit{x}}$ and ${\mathit{I}}_{\mathit{y}}$ represent the $x$- and $y$- derivative filters, respectively. the diagonal weight matrix ${\mathit{W}}_{\mathit{s}}$ can be expressed as

$${\mathit{W}}_{\mathit{s}}=diag\{{[{\left(\mathbf{\nabla }{\mathit{K}}_{\mathit{n}}\right)}^2+ϵ]}^{-\frac{1}{2}}\},$$

(17)

where $ϵ$ is a tuning parameter, which is set as $ϵ=0.0001$.

The corrected blur matrix and motion matrix are used to return to the image reconstruction stage to calculate the super-resolved HR image iteratively. The number of iterations is set to five in this paper. The final algorithm of multi-frame blind super-resolution is demonstrated in Algorithm 1.

Algorithm 1. The final algorithm of multi-frame blind super-resolution.

Input: LR images ${\mathit{y}}_{\mathbf{1}},{\mathit{y}}_{\mathbf{2}},\dots ,{\mathit{y}}_{\mathit{N}}$ and upsampling factor s

Initialize ${\mathit{x}}^{\left(0\right)}={\mathit{y}}_{\left(\mathit{N}/\mathbf{2}\right)}$↑s  % bicubic interpolation,

${\mathit{F}}^{\left(0\right)}$  % Enhanced correlation coefficient optimization [17],

${\mathit{K}}^{\left(0\right)}$  % Blind image deblurring method [18]

for   $i:=1 to I$ do  % $I$ is the number of iterations

%step1: image reconstruction

while not satisfy SCG stopping criterion do

Compute the confidence weights matrix $\mathit{W}$ from Equation (5)

Estimate the HR image ${\mathit{x}}^{\left(i\right)}$ by solving Equation (4)

end while

%step2: motion estimation

while not satisfy CG stopping criterion do

Compute the Jacobian matrix $\mathit{J}\left({\mathit{\phi }}_{\mathit{n}}\right)$ from Equation (10)

Correct the motion matrix ${\mathit{F}}^{\left(i\right)}$ by solving Equation (11)

end while

%step3: blur kernel estimation

while not satisfy stopping criterion do

Compute the diagonal weight matrix ${\mathit{W}}_{\mathit{s}}$ from Equation (17)

Corrected blur matrix ${\mathit{K}}^{\left(i\right)}$ by solving Equation (16)

end while

$i=i+1$

end for

Output: $\mathit{x}$ = ${\mathit{x}}^{\left(i\right)}$

5. Experimental Results and Analysis

Synthetic data and real data are used to test the proposed multi-frame blind super-resolution algorithm. Our algorithm is compared to BayesSR [16], SRB [17], DeepSR [29] and FBSR [14]. Simulation results are conducted in MATLAB (R2018b) on a notebook computer with Intel(R) Core (TM) i7-8650U CPU and 16 GB RAM. We first transfer the input RGB LR images to YCbCr color space. The chromatic components Cb and Cr are super-resolved by bicubic interpolation. Our algorithm uses only the Y component containing the luminance or geometry of the image. The final estimated HR image is composed of the processed Y component and the chromatic components.

5.1. Experiments on Synthetic Data

The ground truth HR images are available on synthetic data, so our proposed algorithm and comparison methods are first measured on synthetic data. In order to evaluate image quality, we employ peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and information fidelity criterion (IFC).

Thirty-one LR images are created from one HR image. Random motion parameters, blur kernels, mixed noises and down sampling are added to the HR image. Random translations range from −3 to +3 pixels, while random rotation angles range from −3° to +3°. The random Gaussian blur kernel ${\sigma }_{blur}$ of each LR image ranges from $0.5 \mathrm{to} 3.5$, and the size of the blur kernel is $15\times 15$ in this paper. We put in mixed noises in the form of random Gaussian noise and Poisson noise, and random standard deviation of Gaussian noise, ${\sigma }_{noise}$, ranges from $0.005 \mathrm{to} 0.035$. Synthetic LR images are generated by using commonly natural images shown in Figure 3.

We find that the SR result becomes better gradually by correcting the blur kernel and motion estimation from Figure 4. The artifacts caused by inaccurate estimation of blur kernel disappear gradually in the iterative process.

PSNR, SSIM and IFC results on synthetic LR images are shown in

Table 1. PSNR, SSIM and IFC results of our algorithms on synthetic LR images when the magnification factor is 4.

Methods	Baby			Butterfly			Head			Lena
	PSNR	SSIM	IFC	PSNR	SSIM	IFC	PSNR	SSIM	IFC	PSNR	SSIM	IFC
Bicubic interpolation	22.3861	0.6307	1.4729	20.9843	0.6547	2.0827	22.7681	0.4478	0.6573	22.1486	0.6142	1.6548
BayesSR	24.6479	0.6872	1.5934	21.3802	0.7260	2.3091	24.8879	0.4927	0.7124	24.6742	0.6841	1.7259
SRB	25.7700	0.7081	1.7981	22.1211	0.7773	2.6048	25.6746	0.5574	0.8692	25.7937	0.7285	1.8423
DeepSR	25.5273	0.6850	1.7548	22.8698	0.7261	2.5813	25.7694	0.5473	0.8827	25.7849	0.7746	1.8948
FBSR	26.2364	0.7148	1.8072	22.8971	0.7809	2.6543	26.0814	0.6072	0.9089	26.4375	0.7901	1.9024
Ours	27.3467	0.7776	1.9230	23.2113	0.8023	2.7421	27.7681	0.6783	0.9826	27.3868	0.8138	2.0442

. Visual comparison results of the Baby image and the Butterfly image are shown in Figure 5 and Figure 6, respectively.

We can see that the values of PSNR, SSIM and IFC have been improved on synthetic data in Table 1. The values of PSNR, SSIM and IFC increased by 3.92%, 6.56% and 6.32%.

We can see from Figure 5, when the noise level is high, the noise outliers of the simple bicubic interpolation are particularly obvious. There are still slight noise points in the BayesSR and SRB methods. The noise level of the DeepSR method is much better, but the image is blurred. The FBSR method is worse than our proposed method in the details of the image. In comparison, the proposed method can preserve the specific details of the image better and suppress the noise as well. The main reason for this is that our algorithm suppresses noise in the manner of iteratively updating the explicit detection of the confidence weight of each low-resolution image and implicit detection using Huber loss function. In addition, motion estimation and blur kernel estimation are modified in the iteration, while the visual effect of the reconstructed high-resolution image is also better.

Regarding the low-level noise, the influence of blur and motion on the super-resolution reconstruction should be considered here. We can see from Figure 6 that the high-resolution images reconstructed by the bicubic interpolation, BayesSR and DeepSR methods have been blurred. The high-resolution image estimated by the SRB method has obvious point artifacts. The FBSR method gives some ringing artifacts. Our proposed algorithm can better find a balance between deblurring and artifact removal, so the reconstructed high-resolution image has higher quality, which owes much to the combination of motion estimation, blur kernel estimation and image reconstruction.

5.2. Experiments on Real Data

The proposed algorithm is tested on real data which comes from [17], except for our own synthetic data. Each group of images in the real dataset has 31 low resolution images with various motions and blurring in the same scene. The quality of real images cannot be evaluated by image evaluation criteria such as PSNR, SSIM and IFC. In order to test our method, we use no-reference assessment metrics, such as Natural Image Quality Evaluator (NIQE) [30], Perception-Based Image Quality Evaluator (PIQE) [31] and No-Reference Quality Metric (NRQM) [32] for super-resolved images. The results of NIQE, PIQE and NRQM on real data are presented in

Table 2. NIQE, PIQE and NRQM results of our algorithms on real data when the magnification factor is 5.

Images	Assessment Metrics	Bicubic Interpolation	BayesSR	SRB	DeepSR	FBSR	Ours
IMG_0659	NIQE	8.602	8.9946	8.0072	7.7975	6.8901	6.0805
	PIQE	90.0698	75.6895	66.9028	58.9616	43.2765	38.3211
	NRQM	3.9608	3.154	3.944	2.9465	4.0102	4.408
IMG_0663	NIQE	39.8007	39.8026	39.8	39.8005	39.7992	39.796
	PIQE	74.6022	65.3371	59.1997	77.0891	56.9802	49.5662
	NRQM	3.7656	4.4025	4.7599	4.2321	4.9025	5.5618
IMG_0667	NIQE	11.3632	9.3836	8.9293	8.2567	7.9208	7.2685
	PIQE	90.1674	66.6976	69.7264	77.3032	60.2489	58.4445
	NRQM	3.401	3.6383	3.0631	2.6799	3.8943	4.0978
IMG_0687	NIQE	10.4796	8.9724	8.3236	9.7315	8.7233	8.0351
	PIQE	90.9179	69.031	67.1065	53.1181	50.2467	42.425
	NRQM	2.8295	2.6297	2.9789	2.5811	3.0874	3.8819
IMG_1480	NIQE	8.4942	6.1062	6.0722	7.7383	6.0002	5.9913
	PIQE	80.8595	74.114	59.0136	72.4327	56.3024	54.7688
	NRQM	4.4241	4.5514	2.0487	2.8916	4.8912	5.4116
IMG_1481	NIQE	9.2407	8.7282	8.8947	6.7134	6.6962	6.7838
	PIQE	92.4789	74.5397	68.7935	88.9525	50.5608	41.6472
	NRQM	4.4951	3.0268	2.5334	2.3857	5.4580	7.0695

. Visual comparison results of the IMG_0663 image and the IMG_1480 image are shown in Figure 7 and Figure 8.

We can see that the values of NIQE, PIQE and NRQM have been improved on real data from Table 2. The values of NIQE, PIQE and NRQM increase by 4.45%, 10.56% and 15.75%.

It can be seen from Figure 7 that bicubic interpolation, BayesSR and DeepSR have different degrees of blur. SRB and FBSR have obvious ringing artifacts. Figure 7 and Figure 8 show that the proposed algorithm can produce better results on real data. The estimated HR image can preserve the details of the image and has no obvious artifacts.

It can be drawn from the experimental results on synthetic data and real data that our proposed method is superior to the previous multi-frame blind super-resolution methods. The reason for this is that we combine the parameters of image reconstruction, blur kernel estimation and motion estimation. The error of each estimation can be further reduced through iteration. On the one hand, this method can quickly correct the error of motion estimation and blur estimation. On the other hand, it can accelerate the convergence of our algorithm and the estimated HR image is very stable.

6. Conclusions

In MFSR practical applications, the reconstructed HR images are greatly affected by motion, blur and noise. Currently, it is assumed that the blur kernel is known in most of the super-resolution methods. Multi-frame blind super-resolution under an unknown blur kernel is challenging. We develop a novel multi-frame blind super-resolution method in this paper. In contrast to deep learning methods, our method deals with multi frame super-resolution issues, but not through training and learning. We first calculate the motion estimation and blur kernel estimation of the image independently. Then, we use the result and the confidence weight of multi-frame blind super-resolution to correct the motion estimation and blur kernel estimation. In the iterative motion estimation, the Jacobian matrix of motion parameters is introduced to estimate the motion more accurately. The final estimated HR image is gradually obtained in the iterative correction process. This method of combining the estimation of motion parameters, blur kernel estimation and parameters in image reconstruction overcomes the poor convergence of the simple alternating minimization method.

Our proposed method can better preserve the details of the image without obvious artifacts compared to the existing methods. We mainly study the MFSR of natural images. We assume that the next task for us to perform is to study the MFSR of special images or specific domains, such as low light and multi-modal MFSR. Our future work will be directed to the development of new priors for special applications.