BSRT++: Improving BSRT with Feature Enhancement, Weighted Fusion, and Cyclic Sampling

Abstract

Multi-frame super-resolution (MFSR) generates a super-resolution (SR) image from a burst consisting of multiple low-resolution images. Burst Super-Resolution Transformer (BSRT) is a state-of-the-art deep learning model for MFSR. However, in this study, we show that there is room for further improvement of BSRT in the feature extraction and fusion process. Then, we propose a feature enhancement module (FEM), a cyclic sampling module (CSM), and a feature reweighting module (FRM) and integrate them into BSRT. Finally, we demonstrate that the modules can help recover the high-frequency information well, enhance inter-frame communication, and suppress misaligned features, thus significantly improving the SR performance and producing more visually plausible and pleasant results compared to other MFSR methods, including BSRT. On the SyntheticBurst and RealBurst datasets, the improved BSRT with the modules, dubbed BSRT++, achieved higher PSNR values of 1.15 dB and 1.31 dB than BSRT, respectively.

1. Introduction

Super-resolution (SR) involves the enhancement of a low-resolution (LR) image by increasing its resolution to reveal the underlying high-resolution (HR) image. This process finds applications in diverse fields such as surveillance, forensics, microscopy, and remote sensing [1]. However, the nature of SR is inherently ill-posed due to the presence of multiple HR images that can give rise to a single LR image.

The approach to SR is divided into single image super-resolution (SISR) and multi-frame super-resolution (MFSR) depending on the number of LR images used as input. MFSR approaches are able to aggregate subpixel information from multiple frames or shifted images of the same scene, alleviating the ill-posed nature of SR and generating a higher quality image compared to SISR approaches. However, MFSR imposes new challenges such as the alignment of input images (captured from different viewpoints) with subpixel accuracy and fusion of aligned images. Misaligned pixels act as noise during the fusion process and consequently become a major factor in deteriorating SR performance. Even if the input images are well aligned, the quality of the SR image varies depending on the fusion method because the amount or quality of information of each image required to generate the final SR image is different.

As a representative deep learning approach to MFSR, DBSR [2] introduced an innovative attention-based fusion module to address noisy RAW snapshots captured from handheld cameras. It utilized dense pixel-wise optical flow estimation to explicitly align the deep features of the input images. The aligned representations were subsequently merged through the computation of element-wise fusion weights. This approach allowed for the adaptive selection of reliable and informative content from each image. Similarly, BSRT [3] introduced a pyramid flow-guided deformable convolution network (Pyramid FG-DCN), which allows for a more accurate and flexible alignment of features through pyramid flow estimation and deformable convolution. In addition, it introduced the Swin Transformer [4] to capture both global and local contexts of images, significantly improving SR performance and achieving state-of-the-art performance. Several MFSR methods have since been proposed, but BSRT is still one of the best MFSR methods.

However, we found some important points that were overlooked during the feature extraction and fusion of BSRT. First, the input images (or their features) have different high-frequency information, and this high-frequency information can contribute to making the final SR image have rich details. However, during the BSRT fusion process, this high-frequency information is weakened. Second, the fusion layer using 1 × 1 convolution is not effective for inter-frame communication (especially between temporally spaced images), which may degrade SR performance. Third, the features extracted from each input image are of different importance, but BSRT fuses all the features with the same weight. In addition, BSRT does not take into account misaligned features. To address these issues, we propose three modules and integrate them into BSRT. The modified BSRT is dubbed BSRT++ hereafter. First, a feature enhancement module (FEM) further enhances the high-frequency information of each feature before the fusion process, helping to generate more detailed SR images. Second, a cyclic sampling module (CSM) cyclically samples neighboring frames and then hierarchically fuses their features so that inter-frame communication was significantly enhanced. Third, a feature reweighting module (FRM) calculates the importance of features to be used as weights in the fusion process, which helps suppress misaligned features.

Notice that most MFSR methods use LR-HR paired datasets for supervision, but it is too difficult to obtain paired datasets in real environments, and synthetically generated datasets cannot completely simulate real-world conditions such as noises and blurs. For this reason, self-supervised methods [5] have been proposed that do not require paired datasets. However, in this study, assuming that the paired datasets are given, we focus on developing an improved supervised MFSR method.

2. Related Work

2.1. Single Image Super-Resolution

SISR is a technology that generates an HR image from a single LR image. Traditionally, local patch-based SR techniques using linear mapping have been widely studied to produce high-quality HR images with relatively low complexity and computational power. However, these techniques have limited performance due to the nonlinear nature between LR and HR images.

Recently, research on deep learning-based SR methods has been actively conducted, showing higher performance than traditional methods. SRCNN [6], which is the first network to introduce deep learning to an SISR problem, is a three-layer fully convolutional network that improves the image quality of images enlarged by bicubic interpolation. This demonstrates that deep learning significantly improves SR performance. Since then, various methods based on deep learning have been proposed to deal with the SISR problem. VDSR [7] introduced a very deep convolutional network and showed that increasing network depth via residual learning can significantly improve SR performance, and ESPCN [8] proposed an efficient subpixel convolutional layer that converts image upsampling to pixel shuffling. EDSR [9] is designed to learn more stably by dividing the deep network into residual blocks and connecting each using a skip connection. In addition, by excluding the batch normalization process, it was possible to reduce the blur effect and memory problems. RCAN [10] also bypasses abundant low-frequency information through various skip connections, allowing the network to focus on learning high-frequency information, and it suggests a channel attention mechanism to readjust channel-specific features taking into account the dependence between channels, showing a better SR image quality improvement. Since then, Transformer has begun to be applied to SISR, and SwinIR [4] is a pioneering example. Unlike previous convolutional neural network (CNN)-based models, Transformer-based models can effectively model long-range dependencies between pixels, but since a lot of training data and GPU memory are required to train the model, research to develop efficient models is being actively conducted. To this end, ESRT [11], a hybrid model combining CNN and Transformer, has been proposed, alleviating the high computational burden of Transformer and showing the high potential of using Transformer in SISR. LBNet [12] is a lightweight model that combines a symmetric CNN for local feature extraction and a recursive Transformer for learning the global context of images, showing better performance in performance, size, execution time, and GPU memory consumption. Most recently, diffusion models [13] have shown immense promise in SR by generating SR images that are even more aesthetically pleasing and realistic. However, diffusion models also come with various challenges such as high computational complexity and color shift. To this end, a number of follow-up studies are underway.

In summary, despite advances in CNN, Transformer, and diffusion models for SISR, developing SISR methods with high accuracy and computational efficiency remains a challenge due to the limited information of the single-input image and the high complexity of deep learning models.

2.2. Multi-Frame Super-Resolution

MFSR is a technology that generates HR images from multiple LR images (frames), and it is possible to generate higher quality images by extracting information contained in multiple LR images and complementarily fusing them. However, in order to fuse multiple image information, the process of accurately aligning the images is essential, and the aligned features must be effectively fused. Therefore, the problem of alignment and fusion is considered the most important issue in MFSR.

The MFSR problem was first solved by modeling the relationship between input images in the frequency domain [14]. However, there was a problem in that visual artifacts occurred when multiple frames were processed in the frequency domain. Afterwards, an iterative backprojection algorithm [15] has been proposed to refine SR images while repeatedly minimizing the reconstruction errors between LR images obtained by downsampling the initial estimated SR images and input LR images, and an approach [16] that minimizes alignment errors between LR images through region-based matching methods was proposed as well.

Recently, deep learning-based MFSR methods have been proposed to take advantage of deep learning technology. For example, TDAN [17] introduced a deformable convolution network (DCN) to solve the problem of alignment between images. DCN is a network that expands the existing convolution operation to extract features considering the deformation of the area and shows excellent performance when there is movement or shape deformation within the area [18]. TDAN used this DCN to perform flexible alignment between images. EDVR [19] proposed a pyramid, cascading, and deformable convolution (PCD) module that can also handle the alignment of images with relatively large movements.

The image (or feature) alignment methods for MFSR are largely divided into implicit alignment and explicit alignment. Implicit alignment is a method to indirectly learn the relationship between images in the learning process without directly performing alignment between input images, whereas explicit alignment pre-aligns the input images for use in subsequent SR processes. In the case of implicit alignment, the learning process is relatively simple because prior knowledge of alignment is not required, but the MFSR method that uses implicit alignment has the disadvantage of being sensitive to unexpected changes or noise between images because the relationship between frames must be learned by itself. In contrast, explicit alignment requires pre-alignment but allows the MFSR method to use explicitly aligned images in SR processes, resulting in more stable and reliable SR results.

BIPNet [20] introduced an implicit alignment module based on deformable convolution and a pseudo-feature fusion module to enable flexible information exchange between images. AFCNet [21], based on BIPNet, improved the feature extraction process by integrating Restormer [22], which is a Transformer-based image restoration model. Burstormer [23] extracted features with reduced noise by applying a transformer-based attention module prior to alignment in the AFCNet framework. In addition, in the feature fusion process, neighboring frames were cyclically sampled and then hierarchically fused through a Transformer-based attention module so that inter-frame communication was not limited to adjacent frames.

DBSR [2] showed excellent MFSR results by combining an explicit alignment module using PWC-Net [24] with an attention-based fusion module. However, DBSR may be effective when the movement between LR images is small but not when the movement is relatively large. EBSR [25] proposed an improved PCD module, which allows for a more accurate and flexible alignment of images with large movements through deformable convolution operations. In addition, a non-local fusion module was proposed that effectively fuses information extracted from different areas within each image to further improve SR performance. As a Transformer-based MFSR model that uses explicit alignment, BSRT introduced the Pyramid FG-DCN that uses SpyNet [26] to obtain pyramid flows between multiple frames and DCNs [18] to more accurately align multi-scale features. BSRT further improved SR performance by capturing both global and local contexts for long-distance dependency modeling using Swin Transformer as the backbone. BSRT is described in more detail in the next section. FBAnet [27] warped each frame using the homography obtained by taking the first frame as a reference and fused the aligned deep features using weights computed from the differences in their affinity maps. FBAnet fed the fused features to two hourglass Transformer blocks to model long-range pixel dependencies.

2.3. BSRT

Given multiple noisy RAW images as input, BSRT aims to generate a denoised, demosaicked, and super-resolved RGB image as output. The process pipeline of BSRT is shown in Figure 1. The input images $x_k(k=1,\dots N)$ are N 4-channel RAW images. First, they are 2× upscaled to 1-channel ‘RGGB’ images by pixel shuffling and converted to 3-channel RGB images by a 3 × 3 convolution. Letting one of them be the reference frame, the optical flow between the reference frame and each frame is obtained by a pre-trained SpyNet. Meanwhile, the original RAW images are sent to several Swin Transformer blocks to extract features. Then, the feature maps are 2× upscaled by pixel shuffling as well. The feature maps are aligned to the reference frame’s feature map using the Pyramid FG-DCN alignment module with the preacquired optical flows. After that, these aligned feature maps $y_k$ are fused by a 1 × 1 convolution and passed through several Swin Transformer groups to reconstruct the HR image.

The main components of BSRT are Pyramid FG-DCN, a pyramid flow-based alignment module, and Swin Transformer-based feature extraction and reconstruction modules. Pyramid FG-DCN can predict fine distortion and offset in the alignment process, allowing a more precise alignment of feature maps, and the pyramid structure improves alignment accuracy by considering feature maps of various resolutions and sizes in the alignment process. In the feature extraction and image reconstruction process, Swin Transformer blocks and groups can capture long-range dependencies to aggregate correlated high-frequency information, enabling the extraction of useful features and reconstruction of high-quality images. For these advantages, BSRT won first place on the real-world track of the NTIRE 2022 Bust Super-Resolution Challenge, and SR results on synthetic and real-world datasets demonstrate that BSRT achieves state-of-the-art performance and produces more visually plausible and pleasant results.

However, BSRT has limited performance by overlooking the following points, so we want to further improve its performance in this study.

-

The input images contain complementary high-frequency information, which must be fully reconstructed (the solution will be given in Section 3.1).

-

In MFSR, flexible information exchange between long-range images (frames) should be achieved [20,23] (the solution will be given in Section 3.2).

-

The features extracted from each input image are of different importance and they cannot be completely aligned by the alignment process (the solution will be given in Section 3.3).

3. BSRT++

BSRT++ is an improvement to BSRT to produce a high-quality SR images from a burst consisting of multiple LR RAW images. BSRT++ has three additional modules compared to BSRT: FEM, CSM, and FRM. The process pipeline of BSRT++ is shown in Figure 2.

3.1. Feature Enhancement Module

Each input image contains complementary high-frequency information that plays an important role in enhancing the clarity and detail of the SR image. To ensure that high-frequency information is well recovered, FEM extracts and enhances high-frequency information from feature maps aligned by the Pyramid FG-DCN alignment module.

Inspired by [20], the aligned feature maps $y_k$ are enhanced as

$${\tilde{y}}_k=y_k+Conv(y_k-\overline{y}),k=1,\dots ,N.$$

(1)

Here, $\tilde{y}$ is the enhanced feature map and N is the number of input frames. $\overline{y}$ is the feature map of the reference frame. The residual feature map between each frame and the reference frame is computed, enhanced by a convolution operation, and finally added to the feature map of each frame. This enhances the detailed texture information unique to each frame, allowing the detailed texture information to be better reconstructed in the SR results. In contrast, FEM may boost the noise or misalignment errors of the feature maps. However, we assume that the feature maps are free of noise or misalignment at this stage because they were refined and aligned through Swin Transformers and the Pyramid FG-DCN alignment module. The noise and misalignment errors will be addressed in the FRM (Section 3.3).

3.2. Cyclic Sampling Module

Unlike BSRT, which simply concatenates feature maps before performing the 1 × 1 convolution operation for fusion, feature maps are rearranged and hierarchically concatenated. Inspired by [23], neighboring feature maps are sampled in duplicate, pairwise, and cyclically, and then the pairwise concatenations continue to be concatenated, as shown in Figure 2.

$$f=Concat(c_1,\dots ,c_{N/2},d_1,\dots ,d_{N/2}),$$

(2)

where

$$\begin{array}{c}\hfill c_i=Concat\left(\mathcal{W}({\tilde{y}}_{2i-1},{\tilde{y}}_{2i})\right),i=1,\dots ,{\displaystyle \frac{N}{2}},\\ \hfill d_i=Concat\left(\mathcal{W}({\tilde{y}}_{2i},{\tilde{y}}_{2i+1\left(\mathrm{mod}N\right)})\right),i=1,\dots ,{\displaystyle \frac{N}{2}}.\end{array}$$

Here, $Concat$ is the concatenation operation with $dim=1$; $\mathcal{W}$ denotes FRM that will be explained in Section 3.3. The final feature map f is sent to the 1 × 1 convolution operation for fusion. This sampling module facilitates information exchange between neighboring frames and enables effective interaction between distant frames at the same time, allowing long-range context to be used in the image reconstruction process. Furthermore, it is significantly more efficient compared to the approach of using a pseudo-feature set produced by channel-wise concatenating all feature maps [20].

3.3. Feature Reweighting Module

In BSRT, the feature maps are equally weighted in the fusion process. However, in our BSRT++, feature maps are reweighted based on their similarity to the reference feature map. Furthermore, since feature maps are paired in the CSM and then fused, the reweighting process $\mathcal{W}$ is performed in pairs. That is,

$${\widehat{y}}_{l/m},{\widehat{y}}_{m/l}=\mathcal{W}({\tilde{y}}_l,{\tilde{y}}_m).$$

(3)

Here, ${\widehat{y}}_{l/m}$ and ${\widehat{y}}_{m/l}$ represent the reweighted feature maps and are computed as

$${\widehat{y}}_{l/m}={\displaystyle \frac{{\psi }_m}{{\psi }_l+{\psi }_m}}{\tilde{y}}_l,{\widehat{y}}_{m/l}={\displaystyle \frac{{\psi }_l}{{\psi }_l+{\psi }_m}}{\tilde{y}}_m,$$

(4)

where

$$\begin{array}{c}\hfill {\psi }_l=\sum _j\left|{\tilde{y}}_l\left(j\right)-\overline{y}\left(j\right)\right|,{\psi }_m=\sum _j\left|{\tilde{y}}_m\left(j\right)-\overline{y}\left(j\right)\right|.\end{array}$$

Here, ${\psi }_l$ is the sum of the pixel differences (errors) between the l-th feature map and the reference feature map. $y\left(j\right)$ indicates the j-th pixel value of y. The larger the sum of errors, the more likely it is not accurately aligned or contains noise, so a relatively small weight is given. This reweighting process helps to reconstruct high-quality images by suppressing alignment errors and noise while ensuring that the details present in the reference frame are fully restored. Furthermore, the pairwise process helps to prevent misalignment errors or noise in a feature map from propagating to distant feature maps.

4. Experimental Results and Discussion

To verify the performance of the proposed method, we conducted an experiment by super-resolving an input image with a upscale factor of 4 on the SyntheticBurst and RealBurst dataset provided by the NTIRE 2022 Burst Super-Resolution Challenge [28]. The SyntheticBurst dataset consists of 46,839 sRGB images with a size of 448 × 448 that are used to synthesize RAW burst sequences for training and evaluation. Every sequence comprises 14 LR RAW images, each with a spatial resolution of $p\times p$ pixels ($p=48$ for evaluation), generated synthetically from a single sRGB image in the following manner. Initially, the provided sRGB image undergoes transformation into RAW space using the inverse camera pipeline [29]. Subsequently, random rotations and translations are implemented on this RAW image to create the HR burst sequence. The HR burst is then downsampled into an LR RAW burst sequence with the application of Bayer mosaicking and random noise addition. The RealBurst dataset consists of 200 RAW burst sequences comprising 14 LR RAW images. The LR RAW images were taken with a Samsung smartphone camera, and the HR images were taken with a DSLR camera. From the sequences, 882 patches with a size of 80 × 80 were extracted for evaluation.

PyTorch (version 1.6.0) was used to implement the proposed SR model, and training and evaluation were performed on a PC with a GTX 2080 Ti GPU. The learning rate was set to ${10}^{-4}$, and the burst size was set to 14. Due to equipment limitations, the batch size and the number of epochs were set to 1 and 50, respectively. In order to objectively evaluate the performance of the model applying the three proposed modules, the qualitative and quantitative image quality of the SR images was compared with the baseline BSRT model (BSRT has two models, BSRT-Small and BSRT-Large, and BSRT-Small has 5,026,321 and BSRT-Large has 20,824,561 model parameters [3]. In this study, BSRT-Small, a light model, was used as the baseline model). For the image quality evaluation of the SR images, peak signal-to-noise ratio (PSNR), structural similarity index map (SSIM), and learned perceptual image patch similarity (LPIPS) values were measured.

Figure 3 shows the validation accuracy of BSRT and BSRT++ according to the number of epochs. As the number of epochs increased, the validation accuracy of BSRT and BSRT++ continued to increase, and the difference in accuracy between BSRT and BSRT++ remained almost unchanged. This indicates that the models were reliably trained with good generalization ability and that BSRT++ will outperform BSRT even if the epoch number is further increased without loss of generality.

4.1. Ablation Study

Table 1. Performance evaluation of BSRT++ with different modules when $epochs=15$. The best results are in bold.

FEM	FRM	CSM	PSNR ↑	SSIM ↑	LPIPS ↓	Time (s)
-	-	-	39.23	0.948	0.055	0.338
√	-	-	40.17	0.956	0.052	0.340
-	-	√	40.17	0.955	0.054	0.350
√	-	√	40.45	0.957	0.053	0.352
√	√	√	40.57	0.959	0.052	0.354

shows the PSNR, SSIM, and LPIPS results of BSRT++ with different sets of proposed modules on the SyntheticBurst dataset. Each module contributed to the improvement in BSRT++ performance. The PSNR value increased by 0.94 dB when FEM and CSM were added separately, and it increased by 1.22 dB when both were added. BSRT++ performed best with all modules (improved by 1.34 dB compared to BSRT with none of them). In terms of LPIPS, adding FEM alone showed the same performance as adding all modules. However, the increase in inference time due to the added modules was about 0.016 s, and there was no significant difference. We believe that BSRT++ is still highly efficient because it operates fairly quickly in our low-spec experimental environment. For experimental convenience, the results were obtained when $epochs=15$, but from Figure 3, we can predict that the improvement will be consistent even if the number of epochs increases. In Figure 4, it is shown that FEM contributed to the recovery of richer texture in the SR images, while CSM and FRM contributed to the reduction of visual artifacts or distortions due to misalignment. When all the modules were applied, the SR images of BSRT++ had the smallest distortion without visual artifacts and were closest to the ground truth. On the contrary, the SR images of BSRT failed to properly restore the high-frequency information related to local textures with severe distortion and visual artifacts (e.g., the straight lines and texture were broken or crushed; ghost textures were restored).

4.2. Comparison with Other Methods

BSRT++ was also compared with other MFSR methods (actually, we tried to add more recent methods for comparison, but some methods did not provide the source codes and some required too large resources to train and test): HighResNet, DBSR, and Burstormer. The results are shown in

Table 2. Comparison of BSRT++ with other burst SR methods when $epochs=50$. The best results are in bold.

Method	SyntheticBurst			RealBurst			Time (s)
	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
Single Image [2]	36.86 a	0.919 a	0.113 a	44.02	0.972	0.051	0.400
HighRes-net [30]	37.45 a	0.924 a	0.106 a	43.99	0.972	0.051	0.463
DBSR b [2]	39.09	0.945	0.084	45.17	0.978	0.037	0.431
BSRT [3]	40.27	0.951	0.047	46.92	0.982	0.035	0.342
Burstormer [23]	41.13	0.967	0.108	47.77	0.983	0.039	0.201
BSRT++	41.42	0.968	0.046	48.23	0.984	0.024	0.374

^a The results were brought from [3] and obtained when $epochs=100$. ^b The results were obtained when $epochs=100$.

.

All MFSR methods perform better than the SISR method. BSRT outperformed DBSR and HighRes-net with 1/2 $epochs$ on all metrics. However, Burstormer had higher PSNR and SSIM values than BSRT with shorter processing times. Burstormer performed worse in terms of LPIPS, indicating that the resulting SR images may have perceptually inferior image quality. On the other hand, BSRT++ generated SR results with better perceptual and quantitative image quality than BSRT or Burstormer. As mentioned in the ablation study, the processing time of BSRT++ was slightly longer than that of BSRT due to the added modules. In Figure 5, where SR images produced by different MFSR methods were shown, it is shown that DBSR and BSRT suffered from visual artifacts or distortion and failed to recover fine texture and details (e.g., the straight lines were blurred or curved and ghost lines were restored). Burstormer produced visually more plausible SR images with reduced visual artifacts and distortion. However, BSRT++ best recovered the fine texture and details of the ground truth with little visual artifacts and distortion.

5. Conclusions

In this study, we proposed BSRT++, which improves the feature extraction and fusion process of BSRT with three modules: FEM, CSM, and FRM. The modules recovered the complementary high-frequency information of each frame, enhanced inter-frame communication, and suppressed misaligned features, respectively. In the experimental results, all modules contributed to improving the perceptual and quantitative quality of the SR images. When all three modules were applied, BSRT++ performed best and outperformed other state-of-the-art MFSR methods, including BSRT.

In this study, we assumed that the feature maps from multiple frames are well aligned through the Pyramid FG-DCN alignment module. However, these assumptions are often not established. In this study, the misalignment was partially resolved using the FRM module, but not completely. Therefore, in a future study, we plan to improve the alignment module using a network or approach that is more effective for flow calculation or feature alignment.

In this study, the effectiveness of BSRT++ was demonstrated on the SyntheticBurst and RealBurst datasets. However, further evaluation on other datasets will provide more comprehensive evidence of its robustness and generalizability across different scenarios and image types, leaving it as another future study.