Super-Resolution Reconstruction and Its Application Based on Multilevel Main Structure and Detail Boosting

Abstract

Vivid main structure and rich texture detail are important factors with which to determine the quality of high-resolution images after super-resolution (SR) reconstruction. Owing to the loss of high-frequency information in the process of SR reconstruction and the limitation of the accurate estimation of the unknown information in the inversion process, a gap still exists between the high-resolution image and the real image. The main structure can better preserve the edge structure of the image, and detail boosting can compensate for the missing high-frequency information in the reconstruction process. Therefore, a novel single remote-sensing image SR reconstruction method based on multilevel main structure and detail boosting (MMSDB-SR) is put forward in this paper. First, the multilevel main structure was obtained based on the decomposition of the remote-sensing image through use of the relative total variation model. Subsequently, multilevel texture detail information was obtained by a difference process. Second, the multilevel main structure and texture detail were reconstructed separately. The detail-boosting function was used to compensate for the missing high-frequency details in the reconstruction process. Finally, the high-resolution remote-sensing image with clear edge and rich texture detail can be obtained by fusing the multilevel main structure and texture-detail information. The experimental results show that the reconstructed high-resolution image has high clarity, high fidelity, and multi-detail visual effects, and the objective evaluation index exhibits significant improvement. Actual results show an average gain in entropy of up to 0.34 dB for an up-scaling of 2. Real results show an average gain in enhancement measure evaluation of up to 2.42 for an up-scaling of 2. The robustness and universality of the proposed SR method are verified.

1. Introduction

Super-resolution (SR) reconstruction is the technology of obtaining high-resolution (HR) images or sequences from one or more low-resolution (LR) observation images by means of signal processing [1]. SR reconstruction is widely used in remote sensing, video surveillance, medical diagnosis, military reconnaissance, and other civil and military fields [2]. This research has important theoretical significance and application value. In recent years, it has become one of the most active research topics in the field of image reconstruction. In the field of remote sensing, with the rapid development of optical remote-sensing satellite technology, the amount of remote-sensing data in China is becoming increasingly abundant, but a gap still exists regarding the resolution of foreign optical satellite images. Therefore, the resolution of optical satellite images can be improved with the help of SR reconstruction technology at a small economic cost to meet the increasingly more refined applications of remote-sensing images [3,4,5]. At present, in terms of the amount of image data involved in SR reconstruction, the SR reconstruction method should be divided into SR reconstruction technology based on single image and that based on multiple images.

The multiple-image-based SR methods mostly exploit the differences between LR images to generate HR images. Owing to the fact that the complementary information contained in multiple images can make up for the high-frequency information lost in the reconstruction process, it has received the attention of many researchers and promising results have been forthcoming. In the remote sensing field, the satellite has the return visit period, in which a multi-sensor can achieve multi-angle observation, and different satellites can acquire images covering the same area with similar exposures. That is to say, image data is easy to obtain. For example, three-line-array images captured seconds apart in the same satellite overpass are shown in Figure 1. As shown in Figure 1, despite the fact that the images are acquired easier, the television tower is different in the images due to the different observation angles. However, beyond that, image distortion, terrain undulation, and shadow existence, are much more challenging for multiple-image-based SR methods. These factors can exert a significant influence on the quality of the reconstructed image. In addition, high-precision sub-pixel registration accuracy of homologous or heterogenous images is still a bottleneck.

By contrast, single-image SR is a valuable and efficient method. However, the key issues remain to be solved urgently, which are how to recover clear edges and finer texture details. Key challenges in single-image SR reconstruction, then, are to fully exploit the information contained in the image and to retain the clarity of the reconstructed image edge, with an aim toward reconstructing the HR image, which can provide more detailed information about the target scene. In summary, to reconstruct a high-resolution remote-sensing image with high clarity, high fidelity, and multiple textures, a flexible single-remote-sensing-image SR reconstruction method based on multilevel main structure and detail boosting (MMSDB-SR) is presented. One embodying a simple and yet effective method of achieving retention of the edge structure of the reconstructed image and restoration of the high-frequency texture details.

2. Related Work

The existing single-image SR reconstruction methods can be divided into four categories [6,7]: SR models based on interpolation, based on reconstruction, based on deep learning, and based on information enhancement.

SR models based on interpolation comprise the most basic SR reconstruction method [8,9,10], and they are widely used as a pre-treatment method for other SR reconstructions in practical applications. Interpolation-based SR models have the performance advantages of simple operation and fast speed, but image edge structure is not taken into account in the process of SR reconstruction using them. This may lead to image edges spreading in different degrees and detail information blurring in the reconstructed image. SR models based on reconstruction [11,12,13] are usually used to incorporate some prior distribution of the corresponding HR image to constrain the constructed result. Many models have been introduced in reconstruction-based SR models, such as the iteration back-projection (IBP) [14], maximum a posteriori (MAP) [15], and the total variation (TV) [12] models, among others. These models aim to solve the difficult problem of SR reconstruction and obtain a stable and relative optimal solution. Although a SR model based on reconstruction is simple to realize, it has the drawback of losing high-frequency information and blurring the edges. Moreover, prior-knowledge-provided accuracy is still a difficult problem in remote-sensing images with complex topography [16,17]. In recent years, a SR reconstruction method based on deep learning was developed [18,19] in which a large number of convolution templates were utilized to learn the features of images. The high-level structural features are learned from patches between LR and HR images in deep-learning-based SR models. On the basis of the learning features, the quality of reconstructed images can be improved effectively. In addition, other state-of-the-art SR methods have been proposed. Image SR using deep convolutional networks (SRCNN) was put forward by Dong et al. [20]. Accurate image SR using very deep convolutional networks (VDSR) was introduced by Kim et al. [21]. These approaches have shown great promise, but a large-capacity training-sample library is needed and the training process is very time-consuming. Such methods can only realize the designated up-scaling factor reconstruction under the training model, and thus a certain lack of flexibility exists. When the convolutional layer is less, there are still some deficiencies in the reconstruction of high-frequency information. SR models based on deep learning often lack high-frequency details and are perceptually unsatisfying in that they fail to match the fidelity expected at the higher remote-sensing image resolutions. SR models based on information enhancement [22,23,24] comprise a kind of method based on image pre-processing. A level of image-enhancement technology has been introduced in some SR models to compensate the loss of high-frequency information in the reconstruction process [25,26]. The problem of SR reconstruction model complexity is solved in such information-enhancement-based SR models. Nevertheless, how to balance the scale of image enhancement and to avoid reconstruction images appearing with excessively sharp edges are the difficult problems introduced in their use.

In summary, a method of improving the visibility and texture details of a reconstructed image from a LR remote-sensing image with clear edge structure and rich texture details based on the multilevel main structure and detail boosting is presented in this paper, the rest of which is organized as follows. We describe the SR reconstruction method and objective evaluation indexes are introduced in Section 3. Experimental analysis and discussion of the SR reconstruction are provided in Section 4. A review of SR applications in Section 5, and final remarks in Section 6.

3. Method

Here, contrary to previous works, a SR reconstruction method based on multilevel main structure and detail boosting is put forward. This method aims to solve the existing problems of edge blurring and lack of detail information in single-image SR reconstruction methods. Motivated by this, a novel and flexible single-image SR method flow is introduced. First, the relative total variation (RTV) algorithm is used to extract multilevel main structure information, and multilevel detail information is obtained by a difference method. Next, the multilevel main structure and texture-detail information are up-sampled separately. Finally, a detail-boosting function is utilized to enhance the texture-detail information in the HR image. Under the premise of controlling excessive edge sharpening, the multilevel main structure and texture-detail information are fused. The complete single remote-sensing image SR reconstruction flow is shown in Figure 2.

3.1. RTV Algorithm

For remote-sensing images, the meaningful structure of images is the main data of the human visual perception system, and thus the main structure forms an important basis for acquiring image information. Therefore, it is valuable to extract meaningful structural data and texture details from the image; it can also be powerful support for SR reconstruction based on a single remote-sensing image. The RTV algorithm was proposed in the literature [27], and it can effectively decompose the main structural and texture details of an image whether the texture is regular or symmetric. In other words, this algorithm is general and arbitrary and is applicable to non-uniform or anisotropic textures. The objective function of the RTV model is finally expressed as

$$\begin{array}{c}\xi (\epsilon ,\lambda ,I,S)={\displaystyle \sum _P{(S_P-I_p)}^2}+\lambda \left(\frac{{\mathcal{D}}_x(p)}{{\mathcal{L}}_x(p)+\epsilon }+\frac{{\mathcal{D}}_y(p)}{{\mathcal{L}}_y(p)+\epsilon }\right)\\ S=\arg \underset{S}{\min }\xi (\epsilon ,\lambda ,I,S)\end{array}$$

(1)

where I is the input image, p the pixel index of the two-dimensional image, S the output structure image, $\lambda$ a weight value that controls the degree of separation between the main structure and texture detail of the image, and $\epsilon$ a small positive value introduced to avoid the case of the denominator appearing to be 0, the fixed value of which is 0.001. A general pixel-wise windowed total variation measure is contained in this method. ${\mathcal{D}}_x(p)$ and ${\mathcal{D}}_y(p)$ are windowed total variations in the x and y directions, respectively, for pixel p. To help distinguish prominent structures from the texture elements, a novel windowed inherent variation is defined. ${\mathcal{L}}_x(p)$ and ${\mathcal{L}}_y(p)$ are windowed inherent variations in the x and y directions, respectively, for pixel p. expressed as

$$\begin{array}{l}{\mathcal{D}}_x(p)={\displaystyle \sum _{q\in R(p)}{g}_{p,q}·\left|{({\partial }_{x}S)}_q\right|}\\ {\mathcal{D}}_y(p)={\displaystyle \sum _{q\in R(p)}{g}_{p,q}·\left|{({\partial }_{y}S)}_q\right|}\end{array},\begin{array}{l}{\mathcal{L}}_x(p)=|{\displaystyle \sum _{q\in R(p)}{g}_{p,q}·{({\partial }_{x}S)}_q}|\\ {\mathcal{L}}_y(p)=|{\displaystyle \sum _{q\in R(p)}{g}_{p,q}·{({\partial }_{y}S)}_q}|\end{array}$$

(2)

where q belongs to R(p), the window centered at pixel p. ${\partial }_x$ and ${\partial }_y$ are the partial derivatives in two directions. $g_{p,q}$ is a weighting function, expressed as

$${\mathrm{g}}_{p,q}\propto \exp \left(-\frac{{(x_p-x_q)}^2+{(y_p-y_q)}^2}{2{\delta }^2}\right),$$

(3)

where δ controls the spatial scale of the window, the size of δ is an important parameter in the process of image structure and texture separation, the range of empirical values is between 0 and 8, and the δ increase can suppress the texture very well and retain the effect of sharpening the edge of the main structure. Next, we provide an example to introduce the solution process of Equation (3) in the x direction:

$$\begin{array}{ll}{\displaystyle \sum _p\frac{{\mathcal{D}}_x(p)}{{\mathcal{L}}_x(p)+\epsilon }}& ={\displaystyle \sum _q{\displaystyle \sum _{p\in R(q)}\frac{g_{p,q}}{|{\displaystyle \sum _{q\in R(p)}{g}_{p,q}·{({\partial }_{x}S)}_q}|+\epsilon }}}|{({\partial }_{x}S)}_q|\\ & \approx {\displaystyle \sum _q{\displaystyle \sum _{p\in R(q)}\frac{g_{p,q}}{{\mathcal{L}}_x(x)+\epsilon }\frac{1}{\left|{({\partial }_{x}S)}_q\right|+{\epsilon }_s}}}{({\partial }_{x}S)}_q^2\\ & ={\displaystyle \sum _q{u}_{xq}{w}_{xq}}{({\partial }_{x}S)}_q^2.\end{array}$$

(4)

Actually, the second line of Equation (4) is an approximate calculation, since the small value ${\epsilon }_s$ is introduced. Here, the quadratic term ${({\partial }_{x}S)}_q^2$ is reconstructed and the nonlinear parts $u_{xq}$, $w_{xq}$, $u_{xq}$, $w_{xq}$ are expressed as follows:

$$\begin{array}{l}{u}_{xq}={{\displaystyle \sum _{p\in R(q)}\frac{g_{p,q}}{{\mathcal{L}}_x(p)+\epsilon }=\left(G_{\delta }\ast \frac{1}{\left|G_{\delta }\ast {\partial }_{x}S\right|+\epsilon }\right)}}_q,\\ w_{xq}=\frac{1}{\left|{({\partial }_{x}S)}_q\right|+{\epsilon }_s}.\end{array}$$

(5)

Here, $G_{\delta }$ is the Gaussian kernel function of standard deviation δ and * a convolution symbol. The calculation in the y direction is similar to that in the x direction.

By deduction, we can transform Formula (1) into the following matrix form:

$$\xi (\epsilon ,\lambda ,I,S)={(v_S-v_I)}^T(v_S-v_I)+\lambda (v_S^T{C}_x^T{U}_x{W}_x{C}_x{v}_S+v_S^T{C}_y^T{U}_y{W}_y{C}_y{v}_S),$$

(6)

where $v_S$ and $v_I$ represent the images are vectorized into column vectors, respectively; $C_x$ and $C_y$ are the Toeplitz matrices from the discrete gradient operators with forward difference. $U_x$, $U_y$, $W_x$, and $W_y$ are diagonal matrices, the diagonal values of which are, respectively, $U_x[i,j]=u_{xi}$, $U_y[i,j]=u_{yi}$, $W_x[i,j]=w_{xi}$, and $W_y[i,j]=w_{yi}$.

The form in Equation (6) enables a special iterative optimization procedure. A numerically stable approximation is naturally obtained, the optimization is as follows. First, the values of u and w are straightforward to calculate based on Equation (5). Second, the values of U_x, U_y, W_x and W_y are used, minimization boils down to solving a linear system in each iteration as

$$v_S^{t+1}={\left(1+\lambda L^t\right)}^{-1}{v}_I.$$

(7)

Here, 1 is a unit matrix, $L^t=C_x^T{U}_x^t{W}_x^t{C}_x+C_y^T{U}_y^t{W}_y^t{C}_y$ the weight matrix, and $\left(1+\lambda L^t\right)$ the symmetric positive-definite Laplacian matrix and an invertible matrix. We can directly solve the inverse operation of the matrix or use a conjugate gradient algorithm to solve it. The result is obtained after t-th iteration. The whole optimization process is summarized in Reference [27].

3.2. Multilevel Main Structure and Detail-Information Extraction

From the psychological point of view, the overall structure of the image is the main source of human visual perception. Definition of the edge main structure is one of the important indicators that determine the resolution of an image. Owing to the RTV algorithm being applicable to non-uniform or anisotropic textures, it can be used to extract the main structure of edges from remote-sensing images with complex textures. Therefore, the RTV algorithm is used to decompose the main structure and texture detail in single remote-sensing-image SR reconstruction. The accuracy of HR-image-structure estimation can be improved based on edge main structure reconstruction. Multilevel main structural information is obtained by adjusting the parameters of the RTV algorithm. For example, the target of multilevel decomposition is to acquire a set of main structure information $v_{S_j}$ (scale j = 0, 1, …, m). Suppose that the input original remote-sensing image is decomposed into an m-layer main structure. The image must then be decomposed at m + 1 levels. When the main structure level is j = 0, we set $v_{S_0}=I$. Then, the RTV algorithm is iteratively applied to the input image and a series of multilevel edge main structures $b_1,\cdots ,b_j$ are obtained. The progressive main structure is mainly structured by the parameter increase at each level j. We set the initial parameters λ_j=1 = 1 × 10⁻² and ${\delta }_{j=1}=0.5$, and then set ${\lambda }_j=j*{\lambda }_{j=1}$ and ${\delta }_j=j*{\delta }_{j=1}$ for all $j\ge 1$. The multilevel main structure information can be described as $b_1,\cdots ,b_j$, and the mathematical description of multilevel main structure extraction is expressed as follows:

$$v_{s_j}^{t+1}=\{\begin{array}{ll}RTV(v_{s_0},{\lambda }_j,{\delta }_j)& ifj=1\\ RTV(v_{s_{j-1}},{\lambda }_j,{\delta }_j)& otherwise\end{array}.$$

(8)

To fully extract the information carried by a single image, multilevel texture detail is acquired through multilevel main structure difference processing. The specific computational process of extracting different levels of texture-detail information is as follows.

$$d_1=I-v_{S_1},d_2=v_{S_1}-v_{S_2},\dots ,d_i=v_{S_{j-1}}-v_{S_j}(i=1,2,\dots ,m).$$

(9)

An edge-guided image-interpolation algorithm via directional filtering and data fusion [10] is used as an up-sampling method in this paper. Up-sampling the multilevel main structure and detail information. Related information of multiple-direction known pixel points is taken into consideration in this interpolation algorithm. Clear and saliency edge main structure can be retained in the reconstruction result. The multilevel main structure $v_{S_j}$ was reconstructed $V_{S_j}$, the multilevel detail information d_i was reconstructed D_i.

3.3. Detail Boosting and Fusion

Compared with a SR reconstruction based on multiple images, it is difficult for a single image to provide complementary information and compensate the lost detail information in the reconstruction process. To solve this problem, we converted the image into a signal. The middle part of the signal is the texture detail of the image and the two sides are mostly the edge main structure of the image. To make the reconstruction of HR image restore more vivid texture detail, a flexible detail-boosting function is proposed in this paper, one that can compensate for the lost high-frequency detail in the SR reconstruction process. The mathematical description is:

$$F(D_i;\alpha ,{\beta }_i)=(1-{\beta }_i\mathrm{sign}(\frac{1-e^{-2D_i\alpha }}{1+e^{-2D_i\alpha }}))·\frac{1-e^{-2D_i\alpha }}{1+e^{-2D_i\alpha }},$$

(10)

where $D_i$ is the reconstructed multilevel texture detail information, i refers to the range of samples of D_i, $\alpha$ controls the amount of detail boosting, $\beta$ controls the high-frequency range compression, and sign is better known as signum function. The value of $\alpha$ typically varies in a small range (1,10], ${\beta }_i(i=1,\cdots ,m)$ are usually fixed to 0.5, 0.25 and 0.25, respectively.

The function has the task of boosting the texture detail of the image and of suppressing the reconstruction image oversaturation. We set function f equal to $1-e^{-2D_i\alpha }/1+e^{-2D_i\alpha }$, where f belongs to the hyperbolic tangent function and is a sigmoid function. A sigmoid function is an S-shaped function commonly seen in biology, and is also known as an S-shaped growth curve. A sigmoid function is often used as a threshold function in deep-learning convolution neural networks. In information science, the sigmoid function has the inverse function. A sigmoid function can map variables to [0,1] and the function-value range is [−1,1]. For multiple-scale SR experiments, different detail-boosting parameters were applied to the multilevel detail information and a series of differently enhanced texture-detail levels D₁, … ,D_i obtained. The progressive detail-boosting information was realized by increasing the parameter α. We set ${\alpha }_{\times \eta }=2(j-1){\alpha }_{\times 2}$, set the initial parameter ${\alpha }_{\times 2}=2$, the subscript ×$\eta$ is an up-scaling factor of SR reconstruction. The detail-boosting function is used to activate the details of the middle texture. The detail-boosting information can be exploited to infer high-frequency components. While D_i is acquired by difference processing from the reconstructed multilevel main structure, D_i is mixed with a certain marginal main structure. When the texture-detail information D_i is improved, the edge main structure information may be promoted simultaneously. This may lead to whole-image oversaturation. Consequently, the symbolic function and the parameter control the high-frequency range compression.

Finally, the multilevel main structure and texture-detail information are fused. An HR image is generated that contains the clear edge and rich texture detail. The fusion process is expressed as Equation (11) and the proposed SR reconstruction (SRR) method is outlined in Algorithm 1:

$$\mathrm{HR}={\displaystyle \sum _{i=1}^{m}F(D_i;\alpha ,{\beta }_i)}+{\displaystyle \sum _{j=1}^m{v}_{S_j}}/m.$$

(11)

Algorithm 1. Multilevel Main Structure and Detail-Boosting SRR
1:	Input: image I, up-scaling factor $\eta$,
2:	Initialization
3:	For band=1: n do
4:	Decompose and obtain multilevel main structure, $v_{S_j}=I$, $RTV(v_{s_{j-1}},{\lambda }_j,{\delta }_j)$
5:	Calculate multilevel detail information, $d_i=v_{S_{j-1}}-v_{S_j}$
6:	Up-sample multilevel main structure and texture detail, and get $V_{S_j}$ and Dj
7:	Detail boosting with Equation (10)
8:	Fusion with Equation (11)
9:	End for
10:	Output: HR image

3.4. Objective Evaluation

How to evaluate quality improvement is also one of the key points of SR reconstruction. Referring to the evaluation method of most SR methods in the literature, the full-reference-image quality-assessment index Peak Signal-to-Noise Ratio (PSNR) [28] and Structural Similarity (SSIM) [29] were used in the simulation experiment. The full-reference-image quality-assessment algorithm needs a reference image that is the same size as the evaluated image. For remote-sensing satellites, a complete reference image cannot be obtained, which is also the significance of studying SR reconstruction of remote-sensing images. Under the circumstances, entropy [30] and enhancement measure evaluation (EME) [31] are used to evaluate the quality of SR reconstruction results in terms of whether they are satisfactory or not.

Peak Signal-to-Noise Ratio (PSNR). PSNR is one of the commonly quantitative evaluation methods and mainly used to evaluate the SR reconstructed image quality by calculating the mean square error between the real HR image and the reconstructed HR image. The higher PSNR value is, the better reconstruction image will be. The description of this index can be expressed as follows:

$$PNSR=10lg\frac{L^{2}mn}{{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{[I_{HR}(i,j)-I_{SR}(i,j)]}^2}}},$$

(12)

where $I_{HR}(i,j)$ is the real HR image and $I_{SR}(i,j)$ is the SR reconstruction image, m and n represent the line number and column number of the image, respectively, and L generally represents the gray distribution range of image.

Structural Similarity index (SSIM). SSIM is widely used in the SR reconstruction quality evaluation when the HR image exist. The mathematical description of the SSIM index is defined as:

$$SSIM=\frac{(2{\mu }_x{\mu }_y+C_1)(2{\delta }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\delta }_x^2+{\delta }_y^2+C_2)},$$

(13)

where μ_x, μ_y is the average of x and y respectively, ${\delta }_x^2$, ${\delta }_y^2$ are the variance of x and y respectively, ${\delta }_{xy}$ is the covariance of x and y, C₁, C₂ are the constants.

Entropy. It is used to represent the degree of uniform distribution of any energy in space. The better the energy distribution, the larger the entropy value. The entropy of image information can be generally expressed as:

$$\mathrm{Entropy}=-{\displaystyle \sum _{i=0}^{n}P(i){\log }_{2}P(i)},$$

(14)

where P(i) is the probability of a pixel value i appearing in the image and i the grayscale range of the image. The larger the entropy value is, the larger the peak area of the image histogram. When the probability of all of the grayscale values tends to be equal, more image information will be carried, and the more abundant the information that will be contained.

Enhancement Measure Evaluation (EME). The principle of EME is to calculate the maximum and minimum ratios of the gray level in the sub-region, which is obtained by dividing the evaluated image into k₁ × k₂ sub-regions. The logarithm of the ratios is the evaluation result of the image detail. This evaluation index represents the degree of gray change of the local image. The larger the EME value, the richer the detail information in the image. The EME value is mathematically expressed as:

$$\mathrm{EME}=\frac{1}{k_1{k}_2}{\displaystyle \sum _{l=1}^{k_2}{\displaystyle \sum _{k=1}^{k_1}20\log \frac{I_{\max ;k,l}^w}{I_{\min ;k,l}^w}}},$$

(15)

where $I_{\max ;k,l}^w$ and $I_{\min ;k,l}^w$ denote the maximum and minimum values of the local image blocks w_k,_l, respectively.

4. Experimental Analysis and Discussion

To verify the reliability and universality of the proposed SR reconstruction method based on multilevel main structure and detail boosting, the panchromatic and multispectral images of ZiYun-3 (ZY-3), GaoFen-2 (GF-2), and WorldView-2 were selected as experimental data. SR reconstruction experiments were carried out on a machine with an Intel(R) CPU E3-1505M v6 @3.00 GHz processor running a 64-bit operating system and the MATLAB R2016a platform (MathWorks, USA, Natick).

4.1. Simulation Image SR Experiment

As the real HR remote sensing image under the same sensor is difficult to obtain, the effectiveness of the proposed SR reconstruction method was verified by simulation experiments. Simulation images were obtained from Resources satellite three (ZY-3) and Gaofen-2 (GF-2). The parameters of the simulation experimental images are shown in

Table 1. Parameters of simulation experimental imagery.

No.	Figure	Satellite	View/Spectral Mode	Image Size	Number of Bands	GSD (m)	Acquisition Date
1	3a	ZY3	panchromatic	500 × 500	1	2.1	6 June 2016
2	3b	ZY3	panchromatic	870 × 870	1	2.1	12 February2015
3	3c	ZY3	multi-spectral	500 × 500	3	5	10 January 2017
4	3d	GF-2	multi-spectral	500 × 500	3	3.2	11 November 2017

and the original HR images are shown in Figure 3.

The remote sensing image obtained from the satellite was treated as the real HR image, the corresponding LR image was obtained through the simulation model Y = HK ∗ X + N, where Y represents the simulated LR image, X is the original remote sensing image, N represents the noise, K the fuzzy convolution kernel, H is the down-sampling matrix, * representative convolution operation. The type of subsampling is bilinear that was used to decrease the size of the original images. In this experiment, the scaling factor $\eta$ = 2, three layers were used in the algorithm to decompose the images. The result based on the typical and state-of-the-art SR reconstruction methods were compared with our SR method. Due to a limitation of space, the simulation experimental results with random noise with the range of [−2, 2] were shown in Figure 4.

From the subjective perspective, the edge structure is blurry, the sawing phenomenon is also very serious in Figure 4a. The main reason is the high-frequency information is lost. The IBP results in Figure 4b are better than Bicubic. One important factor is that the real HR image is involved in the SRR process. The overall effect of the experiment result based on the SRCNN and VDSR method is better than the traditional methods. Nevertheless, compared with the real HR image, the result is a little blurry, and the detail information is still not enough. The reconstructed result in Figure 4e is too saturated, the essential information of the image has been changed, and the result deviates from the real HR image. In Figure 4f, the proposed method can reconstruct rich detail information and retain a better edge structure. The high-frequency information is promoted in this method. In order to evaluate the reconstruction results more objectively, the quality evaluation method was adopted in this paper, and the objective evaluation index results for different amounts of noise are shown in

Table 2. The objective evaluation index results with random noise, with the range of [−2,2].

	Method	Bicubic	IBP	SRCNN [20]	VDSR	HE	MMSDB-SR
Image Data
Exp_1		PSNR: 25.53	PSNR: 26.17	PSNR: 26.36	PSNR: 26.47	PSNR: 13.57	PSNR: 26.59
		SSIM: 0.82	SSIM: 0.83	SSIM: 0.85	SSIM: 0.86	SSIM: 0.63	SSIM: 0.88
Exp_2		PSNR: 20.65	PSNR: 20.71	PSNR: 20.76	PSNR: 20.94	PSNR: 11.69	PSNR: 21.02
		SSIM: 0.82	SSIM: 0.83	SSIM: 0.85	SSIM: 0.87	SSIM: 0.62	SSIM: 0.89
Exp_3		PSNR: 32.09	PSNR: 32.13	PSNR: 33.05	PSNR: 33.08	PSNR: 13.39	PSNR: 33.11
		SSIM: 0.81	SSIM: 0.85	SSIM: 0.94	SSIM: 0.95	SSIM: 0.69	SSIM: 0.89
Exp_4		PSNR: 29.94	PSNR: 30.01	PSNR: 30.15	PSNR: 30.16	PSNR: 13.47	PSNR: 30.17
		SSIM: 0.85	SSIM: 0.89	SSIM: 0.90	SSIM: 0.91	SSIM: 0.64	SSIM: 0.92

and

Table 3. The objective evaluation index results with random Noise with the range of [−5,5].

	Method	Bicubic	IBP	SRCNN [20]	VDSR	HE	MMSDB-SR
Image Data
Exp_1		PSNR: 24.16	PSNR: 24.81	PSNR: 25.08	PSNR: 25.17	PSNR: 13.49	PSNR: 25.21
		SSIM: 0.76	SSIM: 0.78	SSIM: 0.81	SSIM: 0.83	SSIM: 0.55	SSIM: 0.85
Exp_2		PSNR: 20.52	PSNR: 20.58	PSNR: 20.63	PSNR: 20.46	PSNR: 11.67	PSNR: 20.53
		SSIM: 0.81	SSIM: 0.82	SSIM: 0.84	SSIM: 0.86	SSIM: 0.59	SSIM: 0.89
Exp_3		PSNR: 28.38	PSNR: 28.49	PSNR: 29.17	PSNR: 29.19	PSNR: 13.37	PSNR: 29.23
		SSIM: 0.76	SSIM: 0.79	SSIM: 0.83	SSIM: 0.85	SSIM: 0.66	SSIM: 0.87
Exp_4		PSNR: 27.92	PSNR: 28.36	PSNR: 28.84	PSNR: 28.87	PSNR: 13.46	PSNR: 28.92
		SSIM: 0.72	SSIM: 0.74	SSIM: 0.77	SSIM: 0.81	SSIM: 0.59	SSIM: 0.82

.

To further test the performance of the MMSDB-SR method, simulative panchromatic and multi-spectral images were used to experiment for different scale factors. Similar to 2×-up-scaling-factor SR reconstruction, 2×-, 2.5×-, 3×-, 3.5×- and 4×-upscaling-factor SRR experiments results are shown in Figure 5.

In the different scale factors SRR experiment, SRR based on Bicubic, IBP and HE were compared with MMSDB-SR method. This is because SRR based on SRCNN and VDSR directly learn an end-to-end mapping between the low/high-resolution images. The deep-learning methods were not trained for each different scale, and these methods were trained and rebuilt on a set scale at the same time. As can be clearly seen in Figure 5, the overall outline of the SRR result based on Bicubic is not very clear. The SRR result based on IBP is better than Bicubic, but there was no qualitative improvement. The reconstruction image distribution of the SRR result based on HE is too saturated. The detail information of the reconstruction image is not really improved. Through the comparison, it can be shown that MMSDB-SR method resulted in a clear edge structure. In order to analyze the experimental results more objectively, the objective evaluation indexes are listed in

Table 4. Objective evaluation indexes of different-scale reconstruction results.

	Index	Method	×2	×2.5	×3	×3.5	×4
Exp_1	PNSR	Bicubic	25.53	25.27	22.97	21.19	21.04
		IBP	26.17	25.28	23.93	23.37	23.08
		HE	13.57	13.12	12.36	12.15	12.12
		MMSDB-SR	26.59	25.35	24.36	24.05	23.93
	SSIM	Bicubic	0.82	0.82	0.74	0.61	0.57
		IBP	0.83	0.73	0.71	0.64	0.54
		HE	0.63	0.62	0.54	0.46	0.40
		MMSDB-SR	0.88	0.88	0.84	0.83	0.81
Exp_4	PNSR	Bicubic	29.94	24.60	23.84	22.42	22.22
		IBP	30.01	25.23	24.31	23.54	23.48
		HE	13.47	13.37	13.26	13.06	13.04
		MMSDB-SR	30.17	26.39	25.05	24.98	24.90
	SSIM	Bicubic	0.85	0.71	0.61	0.47	0.45
		IBP	0.89	0.76	0.61	0.54	0.48
		HE	0.64	0.52	0.44	0.35	0.32
		MMSDB-SR	0.92	0.85	0.78	0.76	0.75

.

It can also be seen from the objective evaluation results, that the MMSDB-SR result is better than other methods. IBP is better is obviously better than the Bicubic method because the real HR images participate in the reconstruction process. Nevertheless, when the real HR image does not exist, the IBP method will not work well. Meanwhile, when the scale factor is too big, the quality of the results of all SRR methods will also decline because of the reference information limited. Therefore, it is still necessary to further explore and study the goal which retains all kinds of detail information of the original image.

4.2. Real Remote-Sensing Image SR Experiment

Unlike the experimental data selected in most of the published literature, in this paper real remote-sensing images were used as SR reconstruction experimental data. Instead of using simulated degraded images, urban areas, villages and towns, roads, water areas, mountains, and other topographies were selected for the SR reconstruction experiments. The parameters of the experimental images are shown in

Table 5. Parameters of experimental imagery.

No.	Figure	Satellite	View/Spectral Mode	Image Size	GSD (m)	Acquisition Date
1	4a	ZY3	panchromatic	2000 × 2000	2.1	10 July 2013
2	4b	GF-2	panchromatic	1024 × 1024	1	1 September 2016
3	4c	WorldView-2	panchromatic	500 × 500	0.46	16 October 2017
4	4d	ZY3	multi-spectral	1024 × 1024	5	9 March 2013
5	4e	GF-2	multi-spectral	1024 × 1024	4	19 May 2016
6	4f	WorldView-2	multi-spectral	500 × 500	1.8	1 June 2016

and the experimental images themselves are shown in Figure 6.

The actual remote-sensing images were super-resolved with a 2× up-scaling factor. The reconstructed results were compared with those obtained from the bicubic, IBP, SRCNN, VDSR, and histogram-equalization (HE) methods. In the case of multispectral images, the three multi spectral channels, including red, green and blue, were used to apply the SRR process. Subjective and objective evaluation methods were utilized to evaluate and analyze all the SR reconstruction results. The first three groups of images were panchromatic images and the last three groups were multispectral images; the experimental results are shown in Figure 7.

Taking all of the reconstructed results together, reconstruction of the image using the bicubic method did not perform well. The edge structure was fuzzy, the amount of information was not increased by the SRR process, and the texture details were not rich enough. SRR results based on the IBP method were better than those based on the bicubic method. However, the improvement of the results was not visually obvious. The improvement of reconstructed image quality is evidently limited by not having real HR images as references. Compared to the state-of-the-art SR methods in the field of deep learning, such as the SRCNN and VDSR methods, the quality of the reconstructed image by those methods was better than that obtained using the bicubic and IBP methods. The definition of the reconstructed image edge structure was improved, but the information on the complex geomorphology did not substantially change. To further illustrate that the contribution of this paper is not simple contrast enhancement, HE was used in the SR experiments to facilitate comparative analysis. It is well known that HE can improve the contrast of images visually. The detail information was not improved, and the edge structure not clear enough. Through the comparison and analysis of the reconstruction results, it can be shown that the SRR method proposed in this paper resulted in a clear edge structure. Meanwhile, the HR image contained rich topographic feature information and more prominent texture details. To make a quantitative analysis of the reconstruction results obtained by using different reconstruction methods, the reconstruction images were evaluated objectively through entropy and the EME index. The objective evaluation indexes of different SRR results are listed in

Table 6. Objective evaluation indexes of different reconstruction methods in real experiments.

	Index	Bicubic	IBP [14]	SRCNN [20]	VDSR [21]	HE	MMSDB-SR
Exp_1	Entropy	6.18	6.26	6.28	6.29	6.11	6.66
	EME	5.93	6.05	6.17	6.54	6.80	9.69
Exp_2	Entropy	7.55	7.60	7.67	7.68	7.26	7.94
	EME	10.23	10.51	11.70	11.71	9.18	12.45
Exp_3	Entropy	6.98	7.01	7.03	7.02	6.74	7.21
	EME	5.90	11.75	11.85	12.26	9.03	13.02
Exp_4	Entropy	7.52	7.53	7.54	7.54	6.99	7.60
	EME	14.08	15.32	15.70	15.81	14.06	15.99
Exp_5	Entropy	7.64	7.65	7.66	7.67	6.98	7.76
	EME	11.39	12.18	13.73	14.73	9.68	15.07
Exp_6	Entropy	7.39	7.42	7.47	7.52	5.99	7.99
	EME	20.02	20.10	21.65	21.85	20.75	23.62

.

In the SRR experiments, entropy and the EME index were used to evaluate the reconstructed HR images, because the real HR remote-sensing images acquired by the same sensor cannot be obtained. For the objective evaluation indicators used in this paper, the entropy represents the degree of uniform distribution of any energy in space, the higher the objective evaluation index, the more information of the reconstructed image. Statistical results show that the proposed multilevel main structure and detail-boosting method reflect significant advantages in objective evaluation indicators. The entropy indexes of the reconstructed image increased by 0.32, 0.28, 0.25, 0.24, and 0.85 dB for the Bicubic, IBP, SRCNN, VDSR, and HE methods, respectively. Compared with other SR methods, the EME index of the proposed SR method was also significantly improved, and the reconstructed HR image is superior to that obtained using the classical and state-of-the-art SR methods. To further test the complexity of the proposed algorithm compared with other SRR methods, the running time of different SRR methods were calculated during the experiment. The number of iterations was set to 10 in IBP method. For deep learning methods, 500 and 290 images were used for the training process in the SRCNN and VDSR method, and training takes roughly 12 and 4 h on GTX 1060, respectively. Different number of images were used in deep learning methods, because the fewer the layers, the more difficult the convergence. The execution time is shown in

Table 7. The running time (s) of different SRR methods with 2× up-scaling factor.

	Bicubic	IBP [14]	SRCNN [20]	VDSR [21]	HE	MMSDB-SR
Exp_1	0.08	2.90	306.92	97.75	2.02	101.40
Exp_2	0.26	2.14	175.06	43.14	1.37	65.03
Exp_3	0.23	1.41	41.12	22.22	1.17	25.87
Exp_4	0.14	3.17	170.77	104.88	2.33	131.13
Exp_5	0.14	3.17	171.75	103.06	2.36	130.16
Exp_6	0.08	1.74	40.87	10.49	1.85	33.25

.

It can be seen from Table 7, Bicubic, IBP and HE methods with relatively short running time as their computation complexity are relatively low, the decrease of image size is accompanied by the loss of information. Compared with the state-of-the-art SR methods, the execution time of MMSDB-SR is shorter than SRCNN, and longer than VDSR. The reconstruction process of the MMSDB-SR method focuses on detail information, and the reconstruction results are superior to VDSR in visual and subjective evaluation. The sacrifice of time is worthwhile.

To further test the performance of the proposed SR method, panchromatic and multi-spectral images were again used as experimental objects. Similar to 2×-up-scaling-factor SR reconstruction, 2×-, 2.5×-, 3×-, 3.5×- and 4×-upscaling-factor SRR experiments were carried out separately. The experimental results are shown in Figure 8.

Compared with the experimental results, it can be seen that the proposed SRR performed better than that based on the interpolation method, and the high-frequency edge information was blurred owing to the effect of the interpolation algorithm. The method proposed in this paper not only has the same flexibility as the interpolation method, but can also restore the high-frequency information in the reconstructed image. When the multiples of reconstruction increase, the edge blurring of the traditional algorithms become increasingly more serious, and the detail distortion is more obvious. The SR algorithm proposed in this paper restores the high-frequency information in the reconstruction process. The reconstruction image has more feature information, clear edge contours, and richer texture information. It not only meets the needs of human vision, but also reflects better reconstruction performance. According to the results of the state-of-the-art SR methods based on deep learning, a training model cannot used for all scales at the same time. To handle multiple scales, existing methods require multiple networks. Therefore, in multiscale-factor SRR experiments the state-of-the-art SR methods were not included. To evaluate the multiscale-factor SR experimental results objectively, the objective evaluation indexes required are listed in

Table 8. Objective evaluation indexes of different-scale reconstruction results.

	Index	Method	×2	×2.5	×3	×3.5	×4
Exp_1	Entropy	Bicubic	6.98	5.48	5.48	5.48	5.48
		IBP	7.01	7.02	7.02	7.02	7.02
		HE	6.74	5.42	5.42	5.42	5.42
		MMSDB-SR	7.21	7.20	7.20	7.20	7.20
	EME	Bicubic	5.90	4.54	4.34	3.89	3.54
		IBP	11.75	6.24	6.06	5.35	4.83
		HE	9.03	4.49	4.42	3.36	3.31
		MMSDB-SR	13.02	10.57	9.51	8.98	8.18
Exp_4	Entropy	Bicubic	7.64	6.64	6.64	6.64	6.64
		IBP	7.65	6.66	6.65	6.65	6.65
		HE	6.98	5.99	5.99	5.99	5.99
		MMSDB-SR	7.76	7.57	7.55	7.55	7.55
	EME	Bicubic	11.39	11.08	6.36	6.07	5.09
		IBP	12.18	11.75	10.94	9.22	8.37
		HE	9.68	8.71	7.97	7.81	7.77
		MMSDB-SR	15.07	14.89	12.61	11.42	10.41

.

It can be seen directly from Table 4 that the reconstructed image carried more information after being processed using the proposed SR method. In other words, the SR method in this paper can provide the high-frequency information in multiple SRR processes. By comparison, the information based on the bicubic method remained basically unchanged. When multiple scales were too high, there was little detailed information and the edge ambiguity was even more serious.

4.3. Discussion

The SR method proposed in this paper can provide abundant texture-detail information in 2×-up-scaling-factor and multiple SRR experiments. Based on the results of comprehensive subjective evaluation and objective evaluation analysis, the reconstruction effect is better than that of the classical and state-of-the art SR methods. The amount of information in an SRR image reconstructed based on HE cannot be increased. In addition, the image contrast is promoted in SRR under the circumstance of normal light and radiation, which may eventually result in poor visual effects in the reconstructed image. Compared with the HE method, it can be proved that the proposed SR reconstruction method is not simple contrast enhancement. The idea of multilevel decomposition is different from the approach of wavelets. The main difference is the size of the image decreases during the decomposition process in the wavelets. When the size of the image decreases, the information of the image is also lost. The multi-scale decomposition in this paper can better retain the image information and be better used for SR reconstruction. The information in a single remote-sensing image is fully extracted to complete SRR, and the problem of high-precision registration of multiple remote-sensing images is avoided in the proposed method. According to the results of previous experiments, the proposed SR method can restore an HR image without requiring any training on it. Moreover, the proposed SR reconstruction method is flexible. Generally, however, once a deep-learning network is well trained, it can only reconstruct the designated up-scaling-factor SR experiment.

The contributions of the SRR method proposed in this paper are mainly the following:

(1)

To fully extract the information in a single remote-sensing image, a multilevel decomposition model is proposed to extract multilevel main structure and texture detail.

(2)

A novel detail-boosting function is put forward to improve the multilevel detail information.

(3)

A flexible SRR method is realized using a single LR image without any auxiliary information.

5. Applications

We can obtain HR remote-sensing images through use of an SRR method rather than via the observation images acquired by a satellite. Clearly, targets and rich topographies can be provided in HR images. Therefore, HR images have a wide range of economic and military applications. Thus, fast and effective feature extraction from HR remote-sensing images is a hot topic in the photogrammetry and remote-sensing fields. To illustrate the performance of SRR of remote-sensing images with different resolutions for feature extraction, feature-extraction experiments were carried out. It is proven that the edge of an HR image is clearer, and the texture features are recognized more easily. The SIFT operator [32] and LSD algorithm [33] were used to extract the point and line features, respectively. The imagery was acquired from the ZY-3 satellite. The point- and line-feature-extraction results for the original image and different SRR images are shown in Figure 9 and Figure 10, respectively.

Two conclusions can be drawn intuitively from examination of Figure 9 and Figure 10. First, feature point and line extraction in HR images is greater than those in LR images. Meanwhile, our SR method can extract more point and line features. Second, the reconstructed image can extract the line features that are not found in the original image. In particular, for feature-line-extraction results, the consistency of some line features is better in the proposed SRR. Combined with this result, it can be further concluded that the proposed method can reconstruct more feature information than the interpolation-based and state-of-the-art SR methods. It can also be proven from these results that the proposed SRR can not only improve the spatial resolution of the image, but it can also advance the recognition degree of the remote-sensing-image information.

6. Conclusions

In this paper, a novel single remote-sensing-image SRR method was put forward. The information in a single-remote-sensing image is fully extracted using a multilevel decomposition model. The proposed SR method can realize multiple SRRs and the edge main structure of the reconstruction image is clear and sharp. To restore the lost high-frequency information in multiple SRRs, an optimized and flexible detail-boosting function was proposed for our SRR that effectively improves the problem of reconstructed image-edge overflow. As a result, the reconstructed image is rich in texture landform information. Our SRR method solves the problem of how to regenerate HR images with newer and more useful high-frequency detail. Actual results show an average gain in entropy of up to 0.34 dB for 2× up-scaling. Actual image results show an average gain in EME of up to 2.42 for 2× up-scaling. Experimental results show that the definition of the reconstructed HR image and the expression of the texture detail are better using the proposed SRR than those obtained using the bicubic, IBP, and other state-of-the art SRR methods. They further demonstrate that our SR method is not a simple contrast enhancement method. The proposed SR method can actually improve the details in the reconstructed HR image. Moreover, the complexity of the SRR model is simplified in the proposed method. Finally, the reconstructed HR image can be used for feature extraction and image fusion applications. From such applications, it can be seen that SRR is very meaningful research technology. In the future, we plan to further accelerate the proposed SRR method and improve the SR algorithm, at which time we plan to extend the proposed method to video remote-sensing-image SRR.