Deep Residual Dense Network for Single Image Super-Resolution

Abstract

In this paper, we propose a deep residual dense network (DRDN) for single image super- resolution. Based on human perceptual characteristics, the residual in residual dense block strategy (RRDB) is exploited to implement various depths in network architectures. The proposed model exhibits a simple sequential structure comprising residual and dense blocks with skip connections. It improves the stability and computational complexity of the network, as well as the perceptual quality. We adopt a perceptual metric to learn and assess the quality of the reconstructed images. The proposed model is trained with the Diverse2k dataset, and the performance is evaluated using standard datasets. The experimental results confirm that the proposed model exhibits superior performance, with better reconstruction results and perceptual quality than conventional methods.

1. Introduction

Single image super-resolution (SISR) is used to reconstruct a high-resolution image (HR) from a single low-resolution (LR) input image with better visual quality [1,2]. For instance, in the sharing and research of multimedia content, access to the original data is non-existent, and the quality of the received image cannot be estimated. If the image quality is poor, then it becomes difficult to restore the information. Therefore, it is important to restore the original image in super-resolution. This is currently being implemented in many applications, such as closed-circuit television surveillance [3] and security systems [4], satellite remote sensing [5], medical imaging [6,7] atmospheric monitoring [8], and robotics [9].

Super-resolution methods can be broadly classified into two main categories: conventional [10] and deep learning methods [11]. Conventional approaches in computer vision for super-resolution are interpolation-based methods such as bicubic, nearest-neighbor [12], and bilinear interpolation [13,14,15]. Deep learning methods yield better performances than conventional methods. We categorized the network architecture as based on Saeed et al. [11]. In this reference, they are categorized as single image super resolution models based on their structures. Furthermore, the deep learning methods of single image super-resolution can be categorized into two types: peak signal-to-noise ratio (PSNR) and perception-oriented methods. In PSNR-oriented methods, deep neural networks [16,17,18] provide significantly improved performance measures in terms of the PSNR and structural similarity index (SSIM) [19]. In SISR problems, deep learning models are implemented using basic linear networks [20], which are simple structures involving one path and the signal flows sequentially from the start layer to the end layer. By contrast, residual networks [21] use dense and skip connections [22] as well as multiple branches for residual learning. Residual learning with deep networks yields better performance. We will briefly discuss recent deep residual-based [23,24,25] methods used to improve visual quality that are relevant to our current study. One of them includes the enhanced deep residual network (EDSR) [26]. This network was derived from a super-resolution residual network (SRResNet) [27]. In an EDSR, the network is designed to improve the high-frequency details that can improve the visual quality. Other residual methods, such as the very deep super-resolution network (VDSR) [28,29] and cascading residual network (CARN) [30], enhance the performance using different types of architectures and parameters.

Additionally, deep learning methods involving single image super-resolution include perceptual-oriented methods. This study focuses on enhancing perception-based [31,32] image quality. Methods related to perceptual image quality include generative adversarial networks (GANs) [33], where the deep spatial feature transforms GANs [34] and focuses on recovering realistic textures using spatial feature transformation in GAN architectures. Enhanced super-resolution GANs [35] are designed to enhance visual quality in a realistic and natural manner. In general, GAN models pose generalization problems, such as training unstable and complex structures that consume excessive memory. To mitigate issues of training instability, computational complexity, and memory usage, it replaces the residual in residual dense block (RRDB) instead of that in the residual block. The proposed model is a simple sequential structure for improving the perceptual quality of reconstructed images using various depth models.

We propose a deep residual dense network (DRDN) for super-resolution in this paper. The proposed model consists of the residual network and parts of the RRDB based on the EDSR training parameters. Therefore, the proposed network is designed with local and global residual learning which includes a deep residual layers along with the dense and skip connections. Finally, it is implemented with a new structure as a DRDN, which is designed for perceptual quality, fewer parameters, and low computational cost. In the proposed network, the experimental results show that the model yielded better reconstruction results and perceptual quality when evaluated using perceptual metrics. The remainder of this paper is organized as follows. Section 2 provides a brief background of studies pertaining to single image super-resolution. Section 3 provides the details of the proposed approach for a perceptual visual quality network. In Section 4, the experimental process and the results of the proposed method on various datasets are presented. Additionally, comparisons of the proposed method with reference methods in terms of perceptual quality, universal image quality index (UQI), PSNR, and SSIM are reported. Section 5 presents the conclusions.

2. Related Works

The era of deep learning began with cybernetics [36] based on biological learning. Subsequent research advances were realized using the perceptron network [37] for image recognition, and artificial neural networks [38] were developed to achieve intelligent behavior. Finally, researchers performed deeper training in networks, and named it as deep learning [39]. Deep learning is a leading technology in the field of single image super-resolution for reconstructing high-resolution images. The evolution of single image super-resolution for deep learning can be categorized based on the network architecture, such as linear networks, residual networks, recursive networks, progressive reconstruction designs, densely connected networks, multi-branch designs, attention-based networks, and GAN models.

CARN: The cascading residual network (CARN) is a residual network that learns locally and globally with cascading connections. Strategy learning from global cascading to local cascading is efficient for high-frequency details.

VDSR: This is a deep neural network for residual-based learning that solves the problems of super-resolution in convolution neural networks [40]. In addition, it provides solutions for context, convergence, and scale factor issues. This network generates residual maps and combines them with a HR image to form a super-resolution image.

SRResNet: This is one of the residual networks that exhibits better performance than conventional computer vision problems in low-level resolution. It is implemented using basic residual networks.

EDSR: The EDSR is a recent state-of-the-art super-resolution method that is a modified network based on SRResNet, whose batch normalization (BN) process consumes excessive memory while training the model using the graphical processing unit. In the EDSR network, BN layers are removed, thereby improving image reconstruction.

AGTM: The aggregated residual transformation method (AGTM) is a modified version of the EDSR for reducing the number of parameters and time complexity. The aggregated residual transformation method [41,42,43] achieves the same level of performance as the EDSR.

PM-DAN: The perceptual-metric-guided deep attention network (PM-DAN) [44] is an attention-based decoder–encoder network that focuses on the visual quality of newly constructed images. In this case, the residual spatial attention unit captures key information.

ESRGAN: This network is a family of GANs that comprises two main building components: a discriminator and a generator. ESRGAN is a modified version that improves the visual quality of the reconstructed image. In this study, the RRDB was introduced instead of the residual block with a perceptual loss function.

3. Proposed Methods

The EDSR performed better results in single image super-resolution, in which the perception quality improved when it is compared with other models. To improve the perceptual quality, the modified residual block was replaced with the RRDB in the EDSR structure. The RRDB was designed for the discriminator of the GAN model. Finally, the RRDB was built in the residual network of the proposed model. A comparison between the residual block of the EDSR and the proposed model is shown in Figure 1. The original EDSR comprised two convolutional layers and a rectified linear unit (ReLU), as shown in Figure 1a. The convolution layer was used for feature extraction, whereas the ReLU was used to activate the network. The main difference between the EDSR and the proposed model is evident in the RRDB layers. Depending on the depth, we implemented two models, RRDB_20 and RRDB_28.

If the model has a depth of 20, then the number of RRDB layers in the model is 20. If the model has a depth of 28, then the number of RRDB layers in the model is 28. The proposed model comprised three residual dense blocks (RDBs) in the RRDB, as shown in Figure 1b. In addition, it combined multiple RDBs that were concatenated with each other. The RDB comprised a dense block with its own convolutional layers, dense connections that are stacked together (as shown in Figure 2), and a leaky rectified linear unit (LReLU) to rectify the network. After modifying the residual block, the appearance of the proposed model is as illustrated in Figure 3.

The process of this model begins from the low-resolution image as the input $x_{lr}$ and $y_{sr}$ as the output of the proposed model. The first convolutional layer extracts the features $F_0$ from the low-resolution input, as shown in (1) [45,46].

$$F_0=H_{fe}\left(x\right)$$

(1)

where $H_{fe}\left(.\right)$ denotes the convolution operation for the first convolutional layer, and $F_0$ is the feature extraction used as input to the RRDB. Suppose we have $rd$ residual in residual dense blocks, and the output $F_{RD}$ the rdth RRDB can be obtained by (2).

$$F_{RD}=H_{RRDB,\hspace{0.17em}rd}\left(F_{rrdb-1}\right)$$

(2)

where $H_{RRDB,\hspace{0.17em}rd}$ denotes the operations of the rdth RRDB. The inner layers of the RRDB comprised three RDBs, and the features of the RRDB can be calculated using (3).

$$F_{RRDB}=H_{RDB,\hspace{0.17em}rd}\left(\left(H_{RDB,\hspace{0.17em}\hspace{0.17em}rd-1}\left(F_{rd-1}\right)\right)\left(\dots \dots .\right)\left(H_{RDB,\hspace{0.17em}\hspace{0.17em}rd2}\left(F_{rd2}\right)\right)\right)+F_{rrl}$$

(3)

where $H_{RDB,\hspace{0.17em}rd}$ represents the operations of the rdth RDB. As $H_{RDB,\hspace{0.17em}rd}$ is a combined operation of the convolution and LReLU. As $F_{RRDB}$ is obtained using all RDBs and residual in residual learning $\left(F_{rrl}\right)$. The output of the dth RDB can be obtained using the input of $F_{d-1}$ as shown in (4).

$$F_{rd}=H_{DB,\hspace{0.17em}d}\left(F_{d-1}\right)$$

(4)

where, $F_{rd}$ represents the local features of the RDB. As $F_{rd}$ is obtained using all convolutional layers and the LReLU, the inner layers of the dense block are formulated using (5).

$$F_{d,c}=\sigma \left(W_{d,c}\hspace{0.17em}\left[F_{d-1},\hspace{0.17em}\hspace{0.17em}{F}_{d,1},\hspace{0.17em}\dots ..\hspace{0.17em},\hspace{0.17em}{F}_{d,c-1}\right]\right)$$

(5)

where $\sigma$ denotes the LReLU activation function. $W_{d,\hspace{0.17em}c}$ the weights of the c-th convolutional layer, and $F_{d-1},\hspace{0.17em}\hspace{0.17em}{F}_{d,1},\hspace{0.17em}\dots ..\hspace{0.17em},\hspace{0.17em}{F}_{d,c-1}$ the concatenation of the feature-maps yielded by the (d−1)th RDB, convolutional layers 1,…, (c−1) in the dth RDB. Furthermore, the residual in residual dense fusion (RRDF) is shown in (6).

$$F_{RRDF}=F_0+F_{GRL}$$

(6)

where $F_{GRL}$ is the global residual learning, and $F_0$ is the feature extraction of the first convolution layer. Final output of the DRDN can be obtained by the (7).

$$y_{sr}=H_{DRDN}\left(x_{lr}\right)$$

(7)

The residual blocks of the EDSR and the proposed model are shown in Figure 4. Figure 4a shows the inner convolutional layers (as shown in Figure 1a) of the EDSR residual block as a building block. In the EDSR, the residual block consists of two convolutional layers, which are the same size as the 256-d input and output with a 3 × 3 kernel. Figure 4b represents the inner convolutional layers (as shown in Figure 2) of the proposed RRDB as a building block. Figure 4b shows the image transformation steps on one block. In the proposed model, the residual block (RRDB) has five convolutional layers to extract the features that begin with the 64-d input to the first convolutional layer, and its output is 32-d with the 3 × 3 kernel.

Likewise, the second to fifth convolutional layers are followed by different input and output image sizes. Finally, reconstruction of the super-resolution image was performed. In addition, the proposed models were implemented at various depths, such as RRDB_20 and RRDB_28. These two models were distinguished by their depths, and the number of parameters involved is shown in the

Table 1. Parameters of EDSR and proposed models.

Model		No. of Residual Blocks	Total no. of Parameters	Residual Scaling	Loss Function
EDSR		32	~45,450 K	–	L1
DRDN (ours)	RRDB_28	28	~20,627 K	0.2	L1
	RRDB_20	20	~14,872 K	0.2	L1

. Furthermore, Table 1 provides the model type, number of residual blocks, total number of parameters, residual scaling, and loss function. Based on to the performances of the three models, RRDB_28 and RRDB_20 exhibited performances comparable to the EDSR in terms of the PSNR and SSIM. In terms of perceptual quality, the RRDB_20 model outperformed the EDSR.

4. Experimental Results

In this section, we present the experimental procedures and results of the proposed model. A comparison with a state-of-the-art method is presented as well. For the experimental evaluation, the Diverse2k resolution (Div2K) training dataset [47] was used, and a quantitative evaluation was performed using a public benchmark dataset. Finally, perceptual quality was evaluated using the perception-based image quality evaluator (PIQE).

4.1. Training Datasets

In the training, the Div2K dataset [47] was used, which is a 2k-resolution high-image-quality dataset comprising 800 training images, 100 validation images, and 100 test images. In addition, low-resolution (LR) bicubic images were available in ×2, ×3, ×4, ×8 factors for training and evaluating the proposed model.

Evaluation on Benchmark Datasets

To evaluate the performance of the model in terms of quantitative measures, we compared our model with publicly available benchmark datasets, such as set 5 [48], set 14 [13], BSD100 [49], and Urban100 [50]. The main purpose of these datasets are testing and predicting the performance of the newly designed network architectures. It is also easy to compare with the existing conventional models.

Set 5 [48]: This is a standard dataset with five test images of a baby, bird, butterfly, head, and woman.

Set 14 [13]: More categories were compared with set 5. However, the number of images used was 14, which included a baboon, Barbara, bridge, coastguard, comic, face, flowers, foreman, lenna, man, monarch, pepper, ppt3, and zebra.

BSD100 [49]: It is one of the classical datasets, consisting of 100 test images. The dataset consists of images ranging from nature to individual objects such as plants, people, food, animals, and devices, etc.

Urban100 [50]: It is also a classical dataset composed of the same 100 images as BSD100. The dataset focuses on artificial structures which are made by humans, such as urban scenes.

4.2. PIQE

The PIQE [51] is a metric used for human perceptual quality assessment in the field of super-resolution image reconstruction. The mathematical expression for the PIQE is as follows:

$$PIQE=\frac{\left({\displaystyle \sum _{k=1}^{N_{SA}}{D}_{sk}}\right)+\hspace{0.17em}\hspace{0.17em}{C}_1}{\left(N_{SA}+C_1\right)}$$

(8)

Here, $N_{SA}$ represents the number of spatially active blocks in an image, $C_1$ a positive constant, and $D_{SK}$ the distorted block. The PIQE metric returns a positive scalar in the range of 0–100. The PIQE score is the individual image quality score, which is inversely correlated with the perceived quality of an image. A low score indicates good perceptual quality, whereas a high score indicates otherwise. The PIQE metric was evaluated based on benchmark datasets, such as set 5 and set 14, with scale factors ×2, ×3, ×4, and ×8 presented in

Table 2. Perceptual evaluation based on benchmark datasets (perception-based image quality evaluator (PIQE)).

Dataset	Scale	VDSR	EDSR	DRDN (ours)
				RRDB_28	RRDB_20
Set 5	×2	86.8745	56.3733	56.7676	56.3124
	×3	84.7418	74.1060	73.2497	66.4755
	×4	80.2138	82.0745	79.9208	75.7457
	×8	95.8925	82.2949	80.9344	76.2772
Set 14	×2	86.6052	48.5839	48.1912	48.1648
	×3	84.7468	75.5955	67.7200	71.9378
	×4	80.7410	79.5800	79.7969	77.4722
	×8	96.4151	83.9043	80.3520	79.8502
BSD100	×2	41.9925	40.5223	37.8143	37.7747
	×3	65.0.338	64.3396	63.6483	63.2722
	×4	81.0542	80.1188	74.1877	78.6450
	×8	89.3395	87.5180	85.6520	84.6193
Urban100	×2	53.3344	50.4255	50.2638	50.4026
	×3	67.8616	65.6758	63.8491	65.2255
	×4	77.0920	74.5159	67.6317	68.6164
	×8	83.4780	78.1662	72.8245	73.1051

. As shown, ×2 of the RRDB_20, set 5 and set 14 PIQE exhibited a lower value compared with that of the EDSR and RRDB_28, which were 56.3124 and 48.1648, respectively. Hence, RRDB_20 demonstrated better perceptual quality. Similarly, other scale models of set 5 and set 14, such as ×3, ×4, and ×8, yielded values of 66.4755, 75.7457, 76.2772 and 71.9378, 77.4722, 79.5802 respectively. In addition, RRDB_28 of the set 14 (×2), BSD100 (×3), and urban100 (×2, ×3, ×4, ×8) are higher PIQE values than conventional methods. It is evident that the proposed models exhibited better perceptual quality than the EDSR.

4.2.1. UQI

The image quality measures for the human visual system (HVS) were calculated using the universal image quality index (UQI) [52,53] metrics. In particular, UQI models the correlation loss, luminance distortion, and contrast distortion. The UQI metric is expressed as in (9).

$$UQI=\frac{1}{M}{\displaystyle \sum _{j=1}^M{Q}_j}$$

(9)

where UQI is universal image quality index, $Q_j$ is the local quality index, and M is the total number of steps. The UQI metric was used to evaluate the image quality for the HVS based on benchmark datasets such as set 5, set 14, and BSD100, and urban100 with scale factors of ×2, ×3, ×4, and ×8 are presented in

Table 3. Universal image quality index (UQI) evaluation based on benchmark datasets.

Dataset	Scale	VDSR	EDSR	DRDN (ours)
				RRDB_28	RRDB_20
Set 5	×2	0.9936	0.9949	0.9950	0.9951
	×3	0.9814	0.9844	0.9928	0.9929
	×4	0.9819	0.9868	0.9869	0.9870
	×8	0.9262	0.9640	0.9632	0.9666
Set 14	×2	0.9886	0.9906	0.9919	0.9920
	×3	0.9754	0.9792	0.9888	0.9889
	×4	0.9736	0.9753	0.9819	0.9820
	×8	0.9432	0.9667	0.9662	0.9673
BSD100	×2	0.9840	0.9942	0.9944	0.9947
	×3	0.9882	0.9919	0.9923	0.9925
	×4	0.9794	0.9807	0.9842	0.9845
	×8	0.9714	0.9722	0.9717	0.9731
Urban100	×2	0.9779	0.9897	0.9898	0.9902
	×3	0.9656	0.9825	0.9833	0.9834
	×4	0.9687	0.9736	0.9744	0.9743
	×8	0.9467	0.9469	0.9475	0.9479

. As shown, ×2 of the RRDB_20 set 5, UQI exhibited a higher value compared with that of VDSR, RRDB_28, and EDSR was 0.9951. Additionally, set 14 was 0.9920. Hence, RRDB_20 shows improved HVS image quality. Similarly, other scale models of set 5, such as ×3, ×4, and ×8, yielded values of 0.9929, 0.9870, 0.9666, and set 14, yielded 0.9889, 0.9820, and 0.9673, respectively. In addition, RRDB_28 of urban100 (×3) is higher value than the conventional methods in the Table 3. Hence, the proposed model shows improved image quality when compared to other models.

4.2.2. Training Details

For training, we followed the training parameters given in Bee et al. [26]. We also used the RGB input patches of size 48 × 48 from the LR image using the corresponding HR patches. We trained our model using the Adam optimizer by setting$\hspace{0.17em}{\beta }_1=0.9$$\hspace{0.17em}{\beta }_2=0.999$, and$\hspace{0.17em}\epsilon ={10}^{-8}$. We set the mini-batch to 16. The learning rate was initialized as 10⁻⁴, and the learning decay varying at boundaries (70,000, 100,000) were 5 × 10⁻⁵ and 1 × 10⁻⁵ respectively. We implemented the proposed model using the tensor flow framework and trained it using NVIDIA GeForce GTX GPUs device. The proposed model architectures required 10 days to train.

4.3. PSNR (dB)/SSIM Evaluation

The mean PSNR and SSIM of the super-resolution methods evaluated on benchmark datasets such as set 5, set 14, BSD100, and urban100 for super-resolution factors ×2, ×3, ×4, and ×8 of the proposed models were compared with those of the simulated EDSR, as shown in

Table 4. Public benchmark test results (peak signal-to-noise ratio (PSNR) (dB)/structural similarity index (SSIM)).

Dataset	Scale	VDSR [54]	EDSR	DRDN (ours)
				RRDB_28	RRDB_20
Set 5	×2	37.53/0.9559	38.16/0.9550	38.03/0.9546	38.08/0.9548
	×3	33.67/0.9210	35.29/0.9332	35.19/0.9320	35.22/0.9326
	×4	31.35/0.8830	32.31/0.8829	32.35/0.8835	32.30/0.8828
	×8	25.93/0.7240	26.94/0.7461	26.80/0.7388	26.84/0.7402
Set 14	×2	33.05/0.9130	33.93/0.9122	33.69/0.9100	33.74/0.9101
	×3	29.78/0.8320	31.13/0.8487	31.09/0.8480	31.11/0.8483
	×4	28.02/0.7680	28.80/0.7693	28.83/0.7704	28.79/0.7690
	×8	24.26/0.6140	25.16/0.6200	25.07/0.6163	25.05/0.6164
BSD100	×2	31.90/0.8960	33.85/0.9196	33.76/0.9188	33.77/0.9190
	×3	28.83/0.7976	29.74/0.8075	29.72/0.8075	29.73/0.8074
	×4	27.29/0.7252	28.60/0.7480	28.59/0.7482	28.60/0.7483
	×8	24.49/0.5830	25.44/0.5916	25.38/0.5867	25.40/0.5903
Urban100	×2	30.77/0.9141	32.59/0.9263	32.19/0.9225	32.29/0.9237
	×3	27.14/0.8279	29.73/0.8719	29.67/0.8716	29.61/0.8702
	×4	25.18/0.7525	26.40/0.7805	26.34/0.7796	26.33/0.7790
	×8	21.70/0.5710	22.63/0.5993	22.52/0.5926	22.51/0.5928

. The simulated performance of the EDSR indicated better PSNR and SSIM scores when compared with those of the reference EDSR [26]. By comparison, the PSNR and SSIM scores were measured on the y-channel. The validation of the PSNR and SSIM scores of the EDSR and proposed models on the Div2K dataset are shown in Figure 5. In addition, visual comparisons of the super-resolution images from set 5 (×2 and ×4) and set 14 (×3 and ×8) are shown in Figure 6 and Figure 7, respectively. As shown by the results in Table 3, the proposed model performed comparably with the EDSR in terms of reduced parameters and better perceptual quality during the reconstruction of super-resolution images. In addition, RRDB_20 yielded better average perceptual quality than RRDB_28, and its performance varied with individual images, as shown in Figure 6 and Figure 7.

5. Conclusions

In this paper, we proposed a DRDN for single image super-resolution. The proposed models were designed based on RRDBs and it required fewer parameters and demonstrated better stability and perceptual quality. The performance of the proposed model is limited with the increased number of layers, and to train comfortably, the network needs to be optimized. Furthermore, we need to investigate the cascaded multiscale deep network to improve the visual textures with a lower computational cost. The experimental results confirmed that the proposed model achieved better reconstruction results and perceptual quality than the conventional methods.