Single-Image Super-Resolution Challenges: A Brief Review

Abstract

Single-image super-resolution (SISR) is an important task in image processing, aiming to achieve enhanced image resolution. With the development of deep learning, SISR based on convolutional neural networks has also gained great progress, but as the network deepens and the task of SISR becomes more complex, SISR networks become difficult to train, which hinders SISR from achieving greater success. Therefore, to further promote SISR, many challenges have emerged in recent years. In this review, we briefly review the SISR challenges organized from 2017 to 2022 and focus on the in-depth classification of these challenges, the datasets employed, the evaluation methods used, and the powerful network architectures proposed or accepted by the winners. First, depending on the tasks of the challenges, the SISR challenges can be broadly classified into four categories: classic SISR, efficient SISR, perceptual extreme SISR, and real-world SISR. Second, we introduce the datasets commonly used in the challenges in recent years and describe their characteristics. Third, we present the image evaluation methods commonly used in SISR challenges in recent years. Fourth, we introduce the network architectures used by the winners, mainly to explore in depth where the advantages of their network architectures lie and to compare the results of previous years’ winners. Finally, we summarize the methods that have been widely used in SISR in recent years and suggest several possible promising directions for future SISR.

1. Introduction

Single-image super-resolution is an important task in image processing, aiming to reconstruct high-resolution images from low-resolution images and optimize both details and textures to improve the quality of visual perception. It is currently used in a wide range of real-life scenarios [1,2,3,4], including security surveillance [5,6,7], remote sensing [8,9,10], medical imaging [11,12,13], etc., while contributing to other advanced computer vision tasks [14,15,16,17,18,19], and is therefore of wide interest to academia and industry [20,21,22,23,24].

With the rapid development of deep learning [25,26,27,28,29,30], deep-learning-based SISR models have achieved state-of-the-art performance on various benchmarks, but SISR remains a challenging problem as a severely discomforting computer vision problem. This discomfort can become more severe as the scale factor changes, so there are still many aspects of SISR that need to be improved.

To facilitate the development of SISR, challenges regarding image super-resolution have emerged. Among them, NTIRE, PIRM, and AIM are the three most popular challenges. In this paper, we overview the recent progress of deep-learning-based SISR in addressing its top challenge. Although there have been some previous surveys on SISR [31,32,33,34,35,36], our survey differs from them in that we focus on the performance and progress of SISR techniques that address their top challenges. Unlike earlier works that mostly investigated traditional SISR algorithms or focused on a particular class of SISR techniques, this survey systematically and comprehensively reviews the development of SISR as its top challenge during 2017–2022.

NTIRE: The New Trends in Image Recovery and Enhancement (NTIRE) challenge was combined with CVPR [37]. For the single-image super-resolution challenge, the challenge tasks include efficient super-resolution, extreme super-resolution, real-world super-resolution, and classic super-resolution, intending to reconstruct a degraded and resulting low-resolution image into a new high-resolution image at a target multiple, and the challenge promotes the development of SR research in the ideal case or the real-world case.

PIRM: The Perceptual Image Recovery and Manipulation (PIRM) challenge was held in conjunction with ECCV and includes multiple tasks [38]. This challenge focuses on generating high-resolution images with both accuracy and perceptual quality. It is well known that when you choose to generate high-resolution images with higher accuracy, you tend to obtain poor visual perception, and when you choose to generate high-resolution images with higher perceived quality, the image quality is often not good enough.

AIM: The Advances in Image Manipulation (AIM) challenge was combined with ICCV [39,40]. The AIM challenge includes the following main tasks: to train SISR models that can be applied to real-world scenarios, to improve the efficiency of SISR models, to increase the speed of the models, and to reduce the memory needed to run them given a benchmark, etc.

This paper mainly reviews the content of the challenges and the superior methods on single-image super-resolution in NTIRE, PIRM, and AIM during 2017–2022. The rest of this paper is organized as follows: Section 2 presents the datasets used in the above challenge; Section 3 presents the various IQAs proposed and used in the challenge; Section 4 presents the models that won the above challenge during 2017–2022, focusing on the deep feature extraction part; Section 5 concludes and discusses the possible future directions of SISR.

2. Background

Among the SISR tasks, we can express the degradation process of high-resolution images to low-resolution images using the following formula:

$$y=\phi \left(x,{\theta }_{\eta }\right)$$

(1)

where $y$ denotes a low-resolution image, $x$ denotes a high-resolution image, $\phi$ is a function representing the degradation process, and ${\theta }_{\eta }$ is various parameters in the degradation process, including noise and downscaling kernels. And the SISR task is to try to predict and reconstruct a high-resolution image $\widehat{x}$ from the degraded obtained low-resolution image, and the process can be expressed as follows:

$$\widehat{x}={\phi }^{-1}\left(y,{\theta }_{\varsigma }\right)$$

(2)

where $\widehat{x}$ denotes the reconstructed high-resolution image, $y$ denotes the input low-resolution image, ${\phi }^{-1}$ denotes the function solved backward from the degradation process, and ${\theta }_{\varsigma }$ is the various parameters in the function solved backward. The image degradation process in SISR tasks is often unknown and complex and is affected by various factors, such as noise, blur, compression, and artifacts, so the most challenging task in SISR tasks is how to construct the inverse solution function ${\phi }^{-1}$. In the field of SISR, most researchers have modeled the degenerate function φ in Equation (1) as follows:

$$y=\left(x\otimes k\right){\downarrow }_s+n$$

(3)

where $y$ denotes a low-resolution image, $x$ denotes a high-resolution image, $\otimes$ denotes a convolution operation, $k$ denotes a blurring kernel that makes the image blurred, ${\downarrow }_s$ denotes a downscaling operation that reduces the size of the image by a factor of s, and $n$ denotes an additive Gaussian white noise with kernel width $\sigma$.

3. Dataset

To compare the strengths and weaknesses of the SISR model, it is necessary to train as well as test the validation on the same dataset. The datasets used until 2017 are Train 91 [41] proposed by Yang et al. and Set5 [42], Set14 [43], BSD100 [44], and Urban100 [45] proposed and merged by Timofte et al. [46].

With the development of SISR networks, the size of the previously proposed dataset is not sufficient for training complex neural networks, and the training of SISR networks requires more a priori information, so the size of the dataset gradually increases.

First presented at NTIRE 2017, the DIV2K [47] dataset features 1000 images collected from the Internet and covers a variety of content, including people, environments, animals, and more. Each image in this dataset has 2K pixels, i.e., they have 2K pixels in at least one axis (horizontal or vertical) direction with a much higher resolution than the images in the dataset presented above.

The challenging task of NTIRE 2019 was to achieve single-image super-resolution in the real world. At that time, most of the LR images in the dataset were HR images obtained via simple bicubic downscaling, while the image degradation in the real world was much more complex than that, so the SISR method at that time did not perform well on the real-world images. The dataset used in NTIRE 2019 is RealSR [48], proposed by J. Cai et al. This dataset was captured using a digital camera, and an image alignment algorithm was developed to gradually align image pairs at different resolutions to obtain LR-HR image pairs of the same scene by adjusting the focal length. In addition, for the 2020 AIM Real-World Super-Resolution Challenge, Wei et al. proposed the DRealSR [49] dataset, which has more numbers and diversity than the RealSR, and the DPED [50] dataset, which consists of three different mobile phones and one high-end camera, was used for the challenge. The dataset consists of real photos taken on three different cell phones and a high-end camera.

Due to the resolution limitation of the dataset, scaling of larger factors was difficult to achieve with the then-available datasets, so the DIV8K [51] dataset was proposed at the 2019 AIM Extreme Super-Resolution Challenge, which is suitable for scaling factors of 32 and above. The dataset has 1504 high-resolution images, of which the validation set and the test set have one hundred images each. The horizontal pixel resolution of the images in the validation set, test set, and part of the training set is not less than 7680, and the horizontal resolution of the remaining images in the training set is not less than 5760. In

Table 1. Image representation of the SISR challenge datasets 2017–2022.

Dataset	Amount	Format	Short Description
Train 91	91	PNG	Images for training, including a car, flower, fruit, etc.
Set5	5	PNG	Images for testing, including a baby, bird, butterfly, head, and woman.
Set14	14	PNG	Images for testing, including humans, animals, insects, etc.
BSD100	100	PNG	Images for testing, including animals, buildings, food, etc.
Urabn100	100	PNG	Images for testing, including a city, urban, structure, etc.
DIV2K	1000	PNG	Each image in this dataset has 2K pixels, including the environment, flora, fauna, handmade object, etc.
RealSR	595	PNG	The dataset was built via two cameras (Cannon 5D3 and Nikon D810) and used an image alignment algorithm to obtain LR-HR pairs for the real-world SISR challenge.
DRealSR	2507	PNG	Compared to RealSR, it has more diversity and more data volume.
DPED	6000	PNG	The authors used three cell phones and a DSLR to photograph an object simultaneously to form a new database of 6000 photographs for this study.
DIV8K	1504	PNG	Images are suitable for scaling factors of 32 and above.

, we list a number of datasets commonly used in the SISR challenges. In Figure 1, We show a selection of images from a commonly used dataset.

4. Evaluation Method

Typically, image quality is assessed using both subjective human perception methods (i.e., whether the image looks realistic) and objective methods. SISR aims to generate images that match human perception and are of high image quality. Since subjective human perception methods take a lot of time to evaluate, the prevailing method is the objective method. Since objective methods do not reflect human perception of images, the results obtained using subjective and objective methods sometimes differ significantly, and we next describe the subjective and objective methods used in the SISR challenge in 2017–2022.

4.1. Peak Signal-to-Noise Ratio (PSNR)/Structural Similarity Index (SSIM)

Given a high-resolution image as opposed to low-resolution images $I$ with $N$ pixels and a super-resolution image $\widehat{I}$, $L$ is typically 255, and PSNR [52] is defined based on MSE.

MSE is defined as follows:

$$\mathrm{MSE}=\frac{1}{N}{‖I-\widehat{I}‖}^2$$

(4)

PSNR is defined as follows:

$$\mathrm{PSNR}=10{\mathit{log}}_{10}\left(\frac{L^2}{\mathrm{MSE}}\right)$$

(5)

SSIM [52] is defined as follows:

$$SSIM\left(I,\widehat{I}\right)=\frac{\left(2{\mu }_I{\mu }_{\widehat{I}}+C_1\right)\left({\sigma }_{I\widehat{I}}+C_2\right)}{\left({\mu }_I^2+{\mu }_{\widehat{I}}^2+C_1\right)\left({\sigma }_I^2+{\sigma }_{\widehat{I}}^2+C_2\right)}$$

(6)

where ${\mu }_I$ and ${\sigma }_I^2$ are the mean and variance of $I$, ${\sigma }_{I\widehat{I}}$ is the covariance between $I$ and $\widehat{I}$, and $C_1$ and $C_2$ are the constant terms.

4.2. Perception Index (PI)

The previous IQA can only reflect the quality of the picture, which is difficult to reflect the effect of human visual perception of the picture. The PI is proposed in the 2018 PIRM Challenge on Perceptual Image Super-resolution to reflect the perceived quality of the picture.

The No-Reference Quality Metric (NRQM) [53] is a learning-based no-reference metric that trains a regression network for evaluating the perceptual quality of SR images by learning a large number of SR images and the corresponding perceptual scores.

Natural Image Quality Evaluator (NIQE) [54] is a natural scene statistic (NSS) model based on which the quality of the test image is expressed as the distance between the multivariate Gaussian (MVG) fit of the NSS features extracted from the test image and the MVG model of the perceptual quality features extracted from the natural image. PI uses reference-free image quality assessment methods such as NRQM and NIQE to achieve the following:

$$\mathrm{PI}=\frac{1}{2}\left(\left(10-\mathrm{NRQM}\right)+\mathrm{NIQE}\right)$$

(7)

4.3. Learned Perceptual Image Patch Similarity (LPIPS)

LPIPS [55] is used to measure the difference between two images and is more consistent with human perception than the traditional SSIM and PSNR.

$$d\left(x,x_0\right)=\sum _l\frac{1}{H_l{W}_l}\sum _{h,w}{‖w_l\odot \left({\widehat{y}}_{hw}^l-{\widehat{y_0}}_{hw}^l\right)‖}_2^2$$

(8)

where d denotes the distance from $x_0$ to $x$. The feature stack is extracted from the l layer and unit-normalized in the channel dimension. The vector $W_l$ is used to deflate the number of activated channels, and finally, the $L_2$ distance is calculated. Finally, it is averaged over the space and summed by channel.

4.4. Mean Opinion Score (MOS)/Mean Opinion Rank (MOR)

MOS refers to the scoring of the generated image against a relative reference image with six ratings.

$$\mathrm{MOS}=\underset{x\in \left[\mathrm{0,1},\mathrm{2,3},\mathrm{4,5}\right]}{\sum x·p\left(x\right)}$$

(9)

MOR means that the study participants are asked to rank the images obtained using the different methods without seeing the reference image. In addition, the IQA-Rank was calculated using the average of four evaluation methods: NIQE, Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [56], Perception-based Image Quality Evaluator (PIQE) [57], and NRQM. BRISQUE means extracting the mean subtracted contrast normalized (MSCN) features from the image and fitting them to an asymmetric generalized Gaussian distribution (AGGD), extracting the fitted Gaussian distribution features and inputting them to a vector machine (SVM) to predict the result of the image quality evaluation. PIQE is more concerned with extracting local features and predicting the overall image quality score from the local image quality score. Similar to BRISQUE, PIQE also calculates MSCN coefficients first and then calculates the quality fraction of the whole picture according to the formula based on distortion, etc. Then, the IQA-Rank of each method and the ranking given by the participants were used to calculate the average to obtain the MOR.

4.5. Parameters Used to Measure Efficiency

To improve the operational efficiency of the network, several parameters used to measure efficiencies, such as runtime, parameter calculation, FLOPs, activation, and GPU memory usage, are used in a series of efficient super-resolution challenges. In

Table 2. The subjective and objective methods used in the SISR challenge in 2017–2022.

Evaluation Method	Full-/Non-Reference	Short Description
Peak Signal-to-Noise Ratio (PSNR)	Full-Reference	The image quality reference value between the maximum signal and the background noise is calculated to measure whether the image is distorted or not. Higher PSNR values indicate higher quality of the generated images.
Structural Similarity Index (SSIM)	Full-Reference	Calculate whether the structure of two images is similar from the point of view of brightness, contrast, and image structuring. The larger the SSIM, the more similar the images are.
Perception Index (PI)	Non-Reference	It is used to calculate the perceived quality of the image, and often the lower the value, the better the perceived quality of the image.
Learned Perceptual Image Patch Similarity (LPIPS)	Full-Reference	The perceptual similarity is calculated, which is more in line with human perception than traditional methods (PSNR and SSIM). The lower value indicates that the two images are more similar.
Mean Opinion Score (MOS)	Non-Reference	The perceived quality of the images is evaluated via human ratings.
Mean Opinion Rank (MOR)	Non-Reference	Similar to MOS, the perceived quality of the images is evaluated via human ratings of the images.

, we show the subjective and objective methods used in the SISR challenge in 2017–2022.

5. Superior Method

Depending on the task of the challenge, it can be broadly classified into four categories: classic SISR, efficient SISR, perceptual extreme SISR, and real-world SISR. In this section, the network architecture of the winning approach in the 2017–2022 SISR challenge will be presented.

5.1. Classic SISR

Classic SISR refers to the reconstruction of LR images obtained by bicubic downsampling or unknown degradation into images with magnification factors of ×2, ×3, ×4. The classic SISR challenge has two tracks: one track is to obtain the LR image corresponding to each HR using the classic bicubic downsampling and degradation factor; the other track is to obtain the LR image using an unknown degradation. The goal of both tracks is to reconstruct the original HR image from LR separately [37,47,58,59]. In

Table 3. Classic SISR challenge track.

Challenge	Track	Winner
NTIRE 2017	Track 1: Degradation is achieved using bicubic downscaling with degradation factors ×2, ×3, ×4	EDSR
	Track 2: Degradation is achieved using an unknown method with degradation factors ×2, ×3, ×4
NTIRE 2018	Track 1: Degradation is achieved using bicubic downscaling with a degradation factor of ×8	DBPN
	Track 2: Degradation is achieved using an unknown method with a degradation factor of ×4. Track 3: Similar to Track 2, except that the degradation is more complex, with a degradation factor of ×4. Track 4: Similar to Track 2 and 3, the degradation factor is ×4, and the degradation mode is different between images, and four LR images are generated for each HR image.	WDSR
AIM 2022	Implement a ×4 super-resolution for JPEG images compressed using python code with a quality factor of 10.	TCIR

, we show the classic SISR challenge track and winner for the period 2017–2022. In

Table 4. Results of the classic SISR challenge winners (×2, ×3, and ×4) in 2017–2022.

Challenge Category	SISR Networks	×2		×3		×4
		PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
classic	EDSR	34.93	0.948	31.13	0.889	29.09	0.837
blind	EDSR	34.00	0.934	30.78	0.881	28.77	0.826

, we show the results of the classic SISR challenge winners with amplification factors of 2, 3, and 4 for 2017–2022 and show the winner with a factor of 8 in

Table 5. Results of the classic SISR challenge winners (×8) in 2017–2022.

Challenge Category	SISR Networks	×8
		PSNR	SSIM
classic	DBPN	25.455	0.7088
blind	WDSR (Mild)	23.631	0.6316
	WDSR (Difficult)	22.329	0.5721
	WDSR (Wild)	23.080	0.6038

.

5.1.1. EDSR (Winner of NTIRE 2017)

The EDSR [60] proposed by the SNU CVLab team won in both tracks. The SNU CVLab team has made a series of improvements to the EDSR based on SRResNET [61], and the specific network structure is shown in Figure 2. They removed the Batch Normalization (BN) from the residual module because the BN [62] layer would have eliminated the flexibility of the network due to the normalization feature, and this operation was effective in improving the PSNR. The training process becomes unstable as the depth of the network increases. To solve this problem, EDSR uses residual scaling. A residual scaling layer (scaling using constant multiplication) is also added after the second convolution, which is experimentally found to effectively stabilize the learning process when C = 0.1. The model consists of 36 such residual modules. In Figure 3 we compare the original residual block with the residual block in EDSR. The EDSR only has upsampling modules that differ by the scale factor. The EDSR architecture further optimizes SRResNET by removing the BN layer from the SISR network to improve the network performance, allowing one to train a larger model under limited conditions, and also giving rise to the exploration of batch normalization layers in SISR networks.

5.1.2. DBPN/WDSR (Winner of NTIRE 2018)

The DBPN [63] proposed by Muhammad Haris et al. achieved superiority on track 1, and the specific network structure is shown in Figure 4. DBPN is a kind of back-projection network, the previous networks are more feed-forward to predict the results of SR, and each layer is basically based on the results of the previous layer to obtain. We show the upper and lower projection units in DBPN in Figure 5. DBPN connects the features of up- and downsampling together. Each instance of upsampling or downsampling reconstruction uses all of the previous LR or HR image features. Finally, splicing the depth features of all the HR images were obtained from upsampling to reconstruct the final HR image. Unlike previous feed-forward networks, DBPN proposes an iterative mapping projection network that fully exploits the relationship between low-resolution images and high-resolution images, uses the error between the upper and lower projections to guide the reconstruction of images, and stitches the feature maps of all the high-resolution images obtained via upsampling to reconstruct high-resolution images. DBPN also achieves a super-resolution network with a large magnification factor.

The WDSR [64] proposed by Yu et al. achieved superiority on track 2, and the specific network structure is shown in Figure 6. WDSR has improved the Residual block in EDSR by increasing the number of channels of the feature map before the ReLU function, which can activate the network better and obtain better performance without increasing the computational overhead; In addition, a large convolutional kernel after the ReLU function is split into two small convolutional kernels, which can effectively reduce the number of computations while ensuring the same perceptual field; in addition, WDSR replaces the BN layer with the Weight Normalization (WN) [65] layer to increase the training speed and speed up the convergence of the neural network.

In the overall network architecture, WDSR removes the redundant convolutional layers from the EDSR and does not insert convolutional blocks after the upsampling layer. We show that in Figure 7. This operation can effectively improve the operation speed and reconstruction effect of the network.

5.1.3. TCIR (Winner of AIM 2022)

Compression plays an important role in the efficient transmission of images on broadband-limited Internet, but compression can lead to image artifacts and degrade image quality. Therefore, AIM 2022 proposed a super-resolution challenge for compressed images using the DIV2K dataset, and the TCIR proposed by the VUE Team won the year, with the specific network structure shown in Figure 8.

They divided the network into two stages: the first using a hybrid network of Transformer and CNN to remove artifacts and the second using a modified RRDBNet to achieve ×4 super-resolution.

The improvements made by the team to SwinIR [66] are outlined below. Firstly, they downsample the image with a convolution of step 2 to shrink the image by a factor of two. Since the image itself is compressed with a quality factor of 10, this operation does not affect the performance of TCIR, saves GPU memory, and accelerates the model. And then they use the new Swinv2 transformer module to replace the STL module in SwinIR to greatly improve the performance of the network. Thirdly, they added three RRDB modules to the RTCB (the basic module of the network) in TCIR, which can take advantage of both CNN and the Transformer. The network combines CNN and the Transformer together and achieves excellent results, proving that the combination of CNN and the Transformer has a good development prospect.

5.2. Efficient SISR

The goal of the Efficient SISR Challenge is to increase the efficiency of super-resolution networks with amplification factors of ×2, ×3, and ×4 as much as possible. Factors that affect the efficiency of SISR networks include runtime, number of parameters, FLOPS, activation, and memory consumption. Therefore, the efficiency of the SISR network is evaluated from different aspects. The Efficient SISR Challenge is a SISR challenge that optimizes other metrics as much as possible while limiting one parameter, aiming to improve the operational efficiency of SISR networks and advance the lightweight and efficiency of networks [39,67,68]. In

Table 6. Efficient SISR Challenge track in 2017–2022.

Challenge	Track	Winner
AIM 2019	Track 1: Degradation is achieved using bicubic downscaling with degradation factors ×2, ×3, ×4 Track 2: Degradation is achieved using an unknown method with degradation factors ×2, ×3, ×4	IMDN
	Track 3: Fidelity is used to design networks with high fidelity under the premise of guaranteeing the PSNR and running time of MSRResNet.	BaiDu-NAS
AIM 2020	The goal of this challenge is to design a network that reduces one or more aspects, such as runtime, parameters, FLOP, activation, and memory consumption while maintaining at least the PSNR of MSRResNet.	RFDN
NTIRE 2022	Main Track: Designing networks with short run times.	RLFN
	Sub Track 1: Designing networks with few model parameters and FLOPS.	BSRN
	Sub Track 2: Combines runtime, parameters, FLOPs, activation, and memory consumption.	EFDN

, we show the efficient SISR challenge track and winner for the period of 2017–2022, and, in

Table 7. Results of the Efficient SISR Challenge winners. ‘Params’ denotes the total number of parameters in 2017–2022. ‘FLOPs’ is the abbreviation for floating point operations. ‘Acts’ measures the number of elements of all outputs of convolutional layers. ‘GPU Mem’ represents maximum GPU memory consumption.

SISR Networks	PSNR [dB]	Ave. Time [ms]	Params [M]	FLOPs [G]	Acts [M]	GPU Mem. [M]
IMDN	28.78	50.86	0.893	58.63	154.14	120
BaiDu NAS	28.84	-	1.461	-	-	-
RFDN	28.75	41.97	0.433	27.10	112.03	200
RLFN	28.72	27.11	0.317	19.70	80.05	377.91
BSRN	28.69	140.47	0.156	9.50	65.76	729.94
EFDN	28.71	29.97	0.272	16.86	79.59	575.99
MSRResNet	28.70	-	1.517	166.36	292.55	610

, we show the results of efficient SISR challenge winners for 2017–2022.

5.2.1. IMDN (Winner of AIM 2019)

AIM 2019 presents a challenge of image super-resolution performed under constraints, which is based on MSRResNet [69] as a baseline with three tracks. The aim is to optimize the remaining one parameter under two of the conditions limiting the number of parameters, runtime, and PSNR, respectively.

The winner of the year was the IMDN (Information Distillation Network) [70] proposed by the Rainbow Team, the specific network structure of which is shown in Figure 9. The main idea is to use the IMDB module to replace the 16 residual modules in MSRResNet, where the IMDB module is shown in Figure 10. This module can divide the intermediate features into two parts by channels, one part is retained, and the other part is further processed via a 3 × 3 convolutional layer, and a 1 × 1 convolution is used to combine them at the end. This operation can effectively preserve information and greatly improve the performance of the SISR network with only a small increase in parameters. The final upsampling module simply employs a sub-pixel convolution to preserve as many parameters as possible. IMDB uses a Contrast-aware Channel Attention layer (CCA) to enhance image details and improve the accuracy of SISR. Due to the split channel operation in extracting features, the number of input channels is reduced, and an excellent balance between the number of parameters, running time, and PSNR at runtime is achieved. The information distillation model proposed via IMDN is one of the most advanced methods for lightweight networks, which effectively guides the development of lightweight networks.

5.2.2. RFDN (Winner of AIM 2020)

The 2020 AIM presents an efficient single-image super-resolution challenge with a magnification factor of ×4. The goal is to design a network that reduces one or more aspects of runtime parameter computation, flops, activation, and memory consumption while guaranteeing the PSNR of MSRResNet.

The RFDN [71] proposed by the NJU MCG team achieved superiority, and the specific network structure is shown in Figure 11. The team proposes the FDC module to make the network lighter and more accurate and proposes the SRB module to enable the network to harvest the most from the residual learning.

The NJU MCG team found that the feature extraction operation is implemented with a 3 × 3 convolution, but similar to other CNN models, it is more efficient to use a 1 × 1 convolution for channel separation. To ensure the spatial context and better refine the features, the 3 × 3 convolution is still used on the right convolutional theme, which is the FDC block proposed by the team.

The team also introduced a smaller range of residual learning in the network by designing a shallow residual block (SRB), which consists of a 3 × 3 convolution, a connection, and an activation unit, as shown in Figure 12. This block can benefit from residual learning without introducing any additional parameters; the residual linking in IMDB is too coarse to take advantage of residual linking. In contrast, the SRB block allows a lightweight network to take advantage of residual learning as well. In addition, the authors believe that using spatial attention in a shallow SR model is more effective than using channel attention and, therefore, replaced the CCA layer in the RFDB with the ESA layer in the network model that participated in the competition.

5.2.3. RLFN (Winner of NTIRE 2022)

In 2022, NTIRE proposed the Single-Image Efficient Super-Resolution Challenge, which has three tracks [67]: a main track for runtime, a sub-track for model complexity, and a sub-track for overall performance. The RLFN [72] proposed by ByteESR won in the main track, and the specific network structure is shown in Figure 13.

The team rethought RFDB and proposed the RLFB module as the basic module of their network. They use three layers of convolution to achieve local feature learning of residuals, simplifying the feature aggregation operation. They believed that although the feature extraction connection achieved using RFDB via 1 × 1 convolutional operations and cascade operations could effectively reduce the number of parameters, it would seriously slow down the inference speed. Therefore, they forgo multiple feature extraction connections and use several 3 × 3 conv and ReLU layers for local feature extraction. And they add the final output features to the shallow extracted features from the very first input after multiple local feature extraction. The obtained features are next passed through the 1 × 1 convolutional layer and its subsequent ESA module to obtain the final output of RLFB, as shown in Figure 14. In addition, to further reduce the runtime, the number of convolutional layers in each ConvGroups in ESA is reduced to one, which not only prevents performance degradation but also optimizes the inference time and model parameters.

5.3. Perceptual Extreme SISR

There is a balance between the two types of evaluation metrics, one focusing on picture quality and the other on perceptual quality, and there is no method yet to achieve the best picture quality while at the same time achieving optimal perceptual quality. Existing SISR methods tend to focus on the image quality of the reconstructed images; however, the perceived quality of the images is also an important indicator of the merit of the reconstructed images. In addition, the problem of SISR regarding the magnification factor at large scales has received little attention. Therefore, it is a worthwhile problem to realize large-scale SISR and to reconstruct images with excellent perceptual quality. The perceptual extreme super-resolution challenge aims to achieve super-resolution reconstruction with very large magnification factors, similar to ×16 [38,68,73], as well as to achieve super-resolution reconstruction of images with high perceptual quality. In

Table 8. Perceptual extreme SISR challenge track in 2017–2022.

Challenge	Track	Winner
PIRM 2018	The task divides the perceptual distortion plane into three regions in terms of RMSE, and the participant’s goal is to obtain the best average perceptual quality on each perceptual plane region. Region 1 (low RMSE, high PSNR)	EPSR
	Region 2 (middle RMSE, middle PSNR)	DBPN
	Region 3 (high RMSE, low PSNR)	ESRGAN
AIM 2019	Track 1: The aim is to generate high-fidelity results with an amplification factor of ×16.	DSSR
	Track 2: Designed to generate high perceptual quality results with a magnification factor × 16.	MGBPv2
NTIRE 2020	Realization of amplification factor × 16	RFB-SRGAN

, we show the perceptual extreme SISR challenge track and winner for the period of 2017–2022, and in

Table 9. Results of the perceptual extreme SISR challenge winners.

SISR Networks	PSNR	SSIM	LPIPS	PI	RMSE	TIME
EPSR	-	-	-	2.709	11.48	-
DBPN	-	-	-	2.199	12.40	-
ESRGAN	-	-	-	1.978	15.30	-
DSSR	26.79	0.7289	-	-	-	30
MGBPv2	25.44	0.6551	-	-	-	47.11
RFB-SRGAN	23.38	0.5504	0.348	3.977	-	8.1

, we show the results of perceptual extreme SISR challenge winners for 2017–2022.

5.3.1. EPSR/DBPN/ESRGAN (Winner of PIRM2018)

The 2018 PIRM challenge was to achieve factor × 4, the super-resolution of a single image with bicubic downsampling. Unlike in the past, this challenge aims to reconstruct perceptually good quality images. The task divides the perceptual distortion plane into three regions in terms of RMSE, and the participant’s goal is to obtain the best average perceptual quality on each perceptual plane region. The basic architecture of the networks used by the winners of this challenge is all GAN networks, where the ESRGAN [69] proposed by Wang et al. achieves the best average perceptual quality over region three, as shown in Figure 15, for the specific network structure. The EPSR [74] proposed by Vasu, S. et al. obtained the best average perceptual quality over Region I. The specific network structure is shown in Figure 16. The network is trained using a combination of mean squared error loss, perceptual loss, and adversarial loss using EDSR as the generator.

ESRGAN removes all BN layers compared to SRGAN [61] to train deeper networks and replaces the basic blocks in SRRESNet with RRDB to make the network easier to train. Basic blocks consist of residual modules and tight junctions, allowing more layers in the network and improving performance effectively. Meanwhile, ESRGAN uses a relative discriminator, which is no longer the probability of true and false in SRGAN, but the probability of judging the true image to be truer than the false one. And this design helps guide the generator to generate reconstructed images with more realistic texture details.

5.3.2. DSSR/MGBPv2 (Winner of AIM 2019)

AIM 2019 also presents an extreme super-resolution challenge, using the DIV8K dataset to achieve a factor × 16 super-resolution. The competition has two tracks: the first aims to generate high-fidelity results, and the second aims to generate high-perceptual-quality results.

The DSSR [75] proposed by NUAA-404 achieves superiority in generating a high-fidelity track, and the specific network structure is shown in Figure 17. The network connects two × 4 networks to achieve the target result of ×16. DSSR consists of two parts, i.e., SKIP and BODY. SKIP is a sub-pixel convolution module that uses the low-frequency information in LR images to reconstruct HR images. BODY consists of two networks with an amplification factor of ×4. The first network consists of a feature extraction layer, an ADRU layer, a GFF layer, and an AFSL layer. The second network consists of a feature extraction layer, an ADRB layer, and an AFSL layer. The BODY layer is used to reconstruct the HR image using the high-frequency information from LR, and finally, the HR image is obtained by combining the results of BODY with the results of SKIP. The ADRU module consists of four ADRB modules that are tightly connected, and the obtained features are merged via GFF. The first segment of the network consists of four ADRU modules tightly connected with the features obtained via fusion with the LFF layer. The convolution unit in ADRB consists of two wide convolutions and a Leaky ReLU, similar to WDSR, as shown in Figure 18.

The network proposes a new reconstruction model, AFSL, which uses more parameters and more computation than the commonly used subpixel convolution, and also brings better results.

The MGBPv2 proposed by BOE-IOT-AIBD achieves superiority in generating a high sensory quality track with the specific network structure. This method combines MutiGrid (MG) and BackProjections (BP) to provide feasibility for extreme SISR tasks. Although MGBP has good results, MGBP does not work on super-resolution issues. This is mainly due to the poor quality of the reconstructed images due to the small number of parameters and the recursive network structure that causes the number of network features to remain constant along the scale. Compared to the MGBP they proposed in 2018, they made the following improvements.

The MGBPv2 uses recursive networks at the beginning of the network. And then, BOE-IOT-AIBD proposed a strategy to merge patches in inference as a way to handle large-scale images. And they simplify the main module by allowing each instance in the network to use different parameters. In addition, the team proposes a multiscale training strategy that combines distortion or perceptual loss of the output image with a reduced scale output image.

5.3.3. RFB-SRGAN (Winner of NTIRE 2020)

The extreme super-resolution challenge is presented in NTIRE 2020, using a dataset of DIV8K, which aims to achieve a super-resolution with a magnification factor of ×16. The winning model for that year is the RFB-SRGAN [76] proposed by OPPO-Research based on ESRGAN, and the specific network structure is shown in Figure 19. The network consists of five modules: shallow feature extraction module, Trunk-A, Trunk-RFB, upsampling module, and reconstruction module. Among them, the Trunk-A module consists of 16 RRDBs, and Trunk-RFB consists of 8 RFB-RDBs.

For the perceptually extreme super-resolution challenge task, multi-scale features are needed to reconstruct the details. So, the team introduced the RFB module. The RFB module can utilize a multi-branch pool with different kernels corresponding to different sizes of receptive fields, apply its extended convolutional layers to control their eccentricity, and finally reconstruct to generate the final result. The RFB module in RFB-SRGAN uses a combination of 1 × 1, 1 × 3, and 3 × 1 convolutional kernels instead of large convolutional kernels such as 3 × 3, 5 × 5, and so on, as shown in Figure 20. This method effectively reduces the time and parameters needed for the computation, in addition to better extracting detailed features. The important reason for the team to use RFB is the ability to extract very detailed features that can effectively reconstruct the image.

The team also made adjustments in the upsampling section, using not only nearest interpolation (NNI) [77] or subpixel convolution (SPC) but alternating them. RFB after NNI can make the NNI transform from space to depth fully affects the depth, and RFB after SPC can make the SPC transform from depth to space fully affects space, alternating them to effectively make information exchange between space and depth. In addition to this, the use of SPC can effectively reduce the number of parameters and the running time. In Figure 21, we show upsampling method used in RFB-SRGAN.

5.4. Real-World SISR

Previous SISR network training relies on pairs of low-resolution images and high-resolution images, and the trained networks often have difficulty performing well in the real world. Since real-world LR images are degraded differently from LR images in datasets, resulting in existing SISR methods that often perform poorly on real-world images, the Real-World Image SISR Challenge aims to advance SISR models that can be used in the real world [78,79,80]. The challenge aims to train a network model to achieve super-resolution of natural images without paired high- and low-resolution images. In

Table 10. Real-world SISR challenge track in 2017–2022.

Challenge	Track	Winner
NTIRE 2019	Track: Realization of real-world SISR.	UDSR
AIM 2019	Track 1: Designed to generate SR images with LR features, magnification factor ×4.	DSGAN
	Track 2: Designed to generate clean, high-quality HR images with a magnification factor of ×4.
AIM 2020	Track: The aim is to obtain images with high quality and high fidelity with magnification factors ×2, ×3, ×4.	Baidu NAS
NTIRE2020	Track 1: An unknown degradation factor is used to obtain an approximation to the real-world LR, from which the SR network is trained.	Real-SR
	Track 2: The images taken with the iPhone 3 in the OPED dataset were used as LR, from which the SR network was trained.

, we show the real-world SISR challenge track and winner for the period of 2017–2022, and in

Table 11. Results of the real-world SISR challenge winners 2017–2022.

SISR Networks	PSNR	SSIM	LPIPS	MOS
UDSR	29.00	0.84	-	-
DSGAN [Track 1]	22.65	0.48	0.36	2.22
DSGAN [Track 2]	20.72	0.52	0.40	2.34
Baidu NAS [×2]	33.460	0.9237	-	-
Baidu NAS [×3]	30.950	0.876	-	-
Baidu NAS [×4]	31.396	0.875	-	-
Real SR	24.67	0.683	0.232	2.195

, we show the results of real-world SISR challenge winners for 2017–2022.

5.4.1. UDSR (Winner of NTIRE 2019)

The task of this challenge was to reconstruct real-world images, using the dataset RealSR, and the winner was UDSR, proposed by the SuperRior team.

In UDSR, the depth feature map is obtained from the input image via the convolution layer. The low-resolution image feature map is obtained via the residual block, and the low-resolution image feature map is used as the input. The first path processes the feature map via the residual block, the second path downsamples the feature map after going via the residual block, and the third path downsamples the feature map again. And then, the fourth path upsamples the obtained feature map and applies the residual block and convolution block. In addition, they output the highest-resolution feature maps as residual images and add them to the input images The high-resolution images obtained from the three paths are combined with the input image to achieve the final output image. We show the Network architecture of UDSR in Figure 22.

A tandem structure is used for training UDSR, where the HR image is first restored to its original size after quadruple downsampling the HR image of the first segment to calculate the loss function. The output of the first segment is used as the input of the second segment, and the HR image is restored to its original size after quadruple downsampling the HR image of the second segment to calculate the loss function. And the output of the second segment is used as the input of the third segment to calculate the loss function using the original HR image. After this three-stage training, the network can recover the LR image to the HR image. We show the training method of UDSR in Figure 23.

5.4.2. DSGAN (Winner of AIM 2019)

AIM 2019 presents the challenge of super-resolution in the real world with two tracks. One is to reconstruct high-resolution images with guaranteed low-quality picture features. And the other is to provide a set of unrelated images of the same quality as the target, with the learning goal of generating clean, high-quality HR images. The magnification factors of the target images are all ×4.

The DSGAN [81] proposed by the Mad Demon team achieves superiority in both tracks. In Figure 24, we show the network structure. The network is divided into two phases. The first phase is to generate LR images with real-world LR image features. The second phase is to train the network in the form of supervised LR-HR pairs formed in the first phase.

In the first stage, the HR image $y$ is bicubic downsampled to obtain $x_b$, and $x_b$ is used as the input to obtain the LR image $x_d$ via the generator. The discriminator is used to determine which of $x_d$ and $z$ is the synthetic LR image and which is the real LR image. In the second stage, the SR network is trained based on the obtained image pairs. During the training, the generated SR images are separated from the low and high frequencies using filters. The low frequencies use $L_1$ loss to focus on the recovery of image content, and the high frequencies use the adversarial loss to focus on the recovery of image details. In addition to obtaining better perceptual quality and also better combining low- and high-frequency information, a perceptual loss function is introduced; the perceptual loss function also allows better recovery of the image content. The SR network used is ESRGAN.

5.4.3. Baidu NAS (Winner of AIM 2020)

The 2020 AIM presented a real-world super-resolution challenge using the DRealSR dataset, and the winner that year was Baidu’s proposed GP-NAS-based design for super-resolution in search space [82]. Baidu focused more on the macroscopic network structure, using the GP-NAS method to search for parameters of key network structures and generate multiple alternative models. In Figure 25, we show overall architecture of the GP-NAS-based network model.

The backbone model of this method is DRBN, and the whole network consists of DRB except for the shallow feature extraction convolution and the end upsampling module. The shallow feature extraction convolution converts the input three-channel image into F-channel shallow features. Each DRB consists of L double-layer convolutions, and the L outputs in the DRB are connected at the end with a 1 × 1 convolution and passed through a channel attention module. There are two types of jump connections in each DRB, intra-block jump connections and extra-block jump connections. There are three key hyperparameters in this network: F is the number of channels, D is the number of DRB layers in the network, and L is the number of bilayer convolutions in each DRB.

While previous works have often used expertise or experience to make choices about these hyperparameters, the Baidu team used a neural architecture search based on a Gaussian process to determine these hyperparameters as a way to obtain a network architecture with optimal performance. The method combines AI and super-resolution networks, offering new possibilities for the development of super-resolution networks.

5.4.4. Real SR (Winner of NTIRE 2020)

NTIRE 2020 proposes the real-world resolution challenge, which is divided into two tracks, one using an unknown degradation factor to obtain LR and the other using iPhone 3 images from the OPED dataset, both aiming to obtain the best perceptual quality images.

The model proposed by Impressionism achieved superiority in both tracks [83]. In Figure 26, we show the network used by RealSR. The team designs a new degradation model for real-world images by estimating degradation kernels and blur kernels and proposes a new real-world super-resolution model with the aim of better perceptual quality.

The degradation model uses kernel estimation similar to Kernel GAN [84] to estimate degradation kernels from real-world images. To make the degraded image and the source image have the same noise distribution, the team obtains noise directly from the real-world image, and the team constructs a degradation pool from the degradation kernels and the obtained noise. To obtain more HR images, the team bicubic downsamples the real-world images to remove the noise to obtain clean images, degrades the obtained clean images with randomly selected blur kernels from the degradation pool to obtain LR images with noise and blur similar to the real-world ones, and finally trains the SR network.

The team designed the SR model based on ESRGAN. The generator adopts the structure of RRDB, and the loss function uses the weighted sum of L1, perceptual loss, and adversarial loss. The perceptual loss uses the inactive features of VGG-19 to enhance the low-frequency features; the adversarial loss is used to enhance the details of the image to make the image look more realistic. In addition, the discriminator uses a patch discriminator instead of VGG-128 for two reasons.

Firstly, VGG-128 can only discriminate images of size 128, which does not perform well on multi-scale tasks. Secondly, VGG-128 has a deeper network, which focuses more on global features and ignores local features, while the patch discriminator has a fixed perceptual field due to its fully convolutional structure, and its output values are only related to the local part, and the local loss is fed back to the generator to optimize the local details. To ensure overall consistency, the final error uses the average of all local errors.

6. Conclusions

In this paper, we present an overview of the challenge tasks on SISR for the period of 2017–2022. In Section 2, we discuss the datasets used in previous years’ challenges, using different datasets to meet the requirements of each challenge task and to provide enough a priori information to improve the efficiency of the network. Section 3 introduces the IQA methods commonly used in previous years’ competitions. Section 4 shows the challenge tasks and the network architectures of the winners in previous years. The challenge tasks in previous years can be broadly classified into four categories: 1. Classic SISR challenge, including two tracks of known bicubic downsampling degradation and unknown degradation factor super-resolution; 2. Perceptual Extreme SISR challenge, mainly to achieve super-resolution with special amplification factor × 16; 3. Efficient SISR challenge, aiming to achieve reduced network inference time and a number of computations; 4. Real-world SISR challenge, which aims to advance the development of networks with super-resolution that also work in the real world. Among these challenges, many effective approaches have been proposed and applied to their network architectures to improve network performance, such as attention modules, back-projection networks, tight junctions, residual networks, information distillation, recursive networks, and the recently acclaimed Transformer. Despite the advancements in SISR with the involvement of deep-learning-based methods, there are still a number of challenges that need to be considered. We present an outlook of future work in the following items.

• Normalization layer: During 2017–2022, many superior networks used different normalization layers to improve network performance, such as EDSR, to remove the BN layer, TCIR to use the LR layer, etc. The BN layer normalizes the same batch of data, which can accelerate network convergence, control overfitting, allow the use of larger learning rates, and is more applicable to scenarios with larger batch sizes. The LN layer normalizes the data of the whole layer and is insensitive to the size of the batch, in addition to inheriting the advantages of the BN layer. Therefore, it is often necessary to select the appropriate normalization layer by experience when designing the network. Switchable Normalization (SN) [85] was proposed in 2018, combining various operations of IN, LN, and BN to select the appropriate normalization layer for the network when targeting different vision tasks. This may become one of the normalization methods often used in SISR tasks in the future.
• More efficient or lighter networks: Using CNN networks to implement SR is fast and occupies less memory, but some edge information will be lost; using Transformer networks to implement SR can be achieved using full-text information to reconstruct images, but it is slower and occupies more memory. In addition, although CNN networks have advantages in local feature extraction, they are still inadequate for global feature representation. Transformer, on the other hand, has a good sense of global features but ignores local feature details. In recent years, many networks combining CNN and Transformer have been proposed. TCIR is a typical network combining CNN and Transformer. This network added several RRDB modules to the basic module of TCIR, which combined the advantages of both CNN and Transformer and achieved first place in the AIM 2022 compressed image super-resolution challenge. So, further research can be conducted in this direction to design networks to better combine the advantages of both.
• The need for more accurate and effective IQA: Existing IQA methods are difficult to balance perceptual quality and image quality, and images that score high in image quality often do not score high in perceptual quality. Therefore, we need a more suitable IQA method to evaluate both perceptual quality and image quality.
• Unsupervised real-world image super-resolution network model training: The currently proposed super-resolution challenge superiority methods on real-world images are based on learning degradation methods in the real world, from which LR images corresponding to HR are obtained, and then pairwise supervised network training is performed to obtain the network model. The performance of the obtained SISR network depends more on the ability to generate LR images with similar blurring as real-world LR images for training. Such networks are also often not strongly generalizable due to the various reasons for the blurring of real-world images. Therefore, how to implement unsupervised super-resolution training on real-world images is a direction for future development.