Inf-OSRGAN: Optimized Blind Super-Resolution GAN for Infrared Images

Abstract

With the widespread application of infrared technology in military, security, medical, and other fields, the demand for high-definition infrared images has been increasing. However, the complexity of the noise introduced during the imaging process and high acquisition costs limit the scope of research on super-resolution algorithms for infrared images, particularly when compared to the visible light domain. Furthermore, the lack of high-quality infrared image datasets poses challenges in algorithm design and evaluation. To address these challenges, this paper proposes an optimized super-resolution algorithm for infrared images. Firstly, we construct an infrared image super-resolution dataset, which serves as a robust foundation for algorithm design and rigorous evaluation. Secondly, in the degradation process, we introduce a gate mechanism and random shuffle to enrich the degradation space and more comprehensively simulate the real-world degradation of infrared images. We train an RRDBNet super-resolution generator integrating the aforementioned degradation model. Additionally, we incorporate spatially correlative loss to leverage spatial–structural information, thereby enhancing detail preservation and reconstruction in the super-resolution algorithm. Through experiments and evaluations, our method achieved considerable performance improvements in the infrared image super-resolution task. Compared to traditional methods, our method was able to better restore the details and clarity of infrared images.

1. Introduction

With the rapid development of computer vision and digital image processing, super-resolution technology has garnered significant interest and served as an important image enhancement method. Such technology aims to reconstructing high-resolution images from their low-resolution counterparts, improving the quality and clarity of the images. It holds significant importance across diverse areas of image processing, such as surveillance video enhancement, remote monitoring, and medical image analysis [1,2,3,4,5]. On the other hand, infrared images, as a special type of image, are extensively utilized and valued in military, security, and medical fields. Their unique physical properties and information content offer enhanced thermal and concealed insights, critical for applications such as target recognition, night vision, and thermal imaging. Owing to their unique physical properties and information sources, infrared images offer a wealth of concealed and thermal data, thereby playing a pivotal role in areas such as target recognition, night vision, and thermal imaging [6,7,8,9]. However, low-resolution infrared images lack detail, which limits their effectiveness in practical applications. Therefore, developing effective methods to enhance the resolution of infrared images is imperative for recovering lost details and improving their utility in critical applications.

Compared to visible light images, infrared images usually have fewer textures and less detail. This poses a challenge in accurately reconstructing the missing details in infrared super-resolution tasks. Currently, most super-resolution datasets [10,11,12,13] focus on the visible light domain. The degradation processes of infrared images are more varied than, and are distinct from, those affecting visible light images. Classical degradation models for visible light images, which include blur, downsampling, noise, and JPEG compression [14,15], are relatively straightforward and fail to accurately simulate the complex degradation processes of infrared images. Practical degradation processes [16,17] are more complex but relatively singular, often overlooking the potential for simpler degradation scenarios. This implies that direct application of these models to infrared images may not adequately address degradation issues, thereby impacting super-resolution performance.

Infrared images typically exhibit less structural clarity compared to visible light images, because of increased blurriness and discontinuity. Consequently, models must accurately capture and reconstruct the structural information in infrared images. Deep-learning-based image super-resolution networks encompass convolutional neural networks, deep residual networks, and generative adversarial networks. The commonly employed loss functions in the super-resolution domain, including pixel-level loss [18,19,20] and perceptual loss [21,22], fail to differentiate between structural and appearance attributes. Furthermore, applying RGB three-channel super-resolution models, originally designed for visible light images, to infrared image processing can lead to substantial redundant computation, and such redundant computations do not yield additional useful information pertinent to infrared images.

To address these issues, this paper proposes a deep-learning-based super-resolution method for infrared images. In order to effectively tackle the limitations and challenges associated with infrared image processing, several key strategies were adopted. First, we constructed a large-scale dataset dedicated to infrared image super-resolution research, which contains scene-rich high-resolution infrared image samples. This dataset provided ample data support and served as a solid foundation for our study. It enabled us to train and evaluate our method on diverse and realistic datasets. Second, for the common degradation factors in infrared images, such as noise and blur, we enhanced the degradation process to accurately restore the intricate details and textures present in infrared images. In the design of the loss function, we introduced a new spatially correlative loss [23] function to improve the spatial consistency and retention of detail between the generated image and the real image. The incorporation of this innovative loss function into our method significantly improved the visual quality and realism of the generated images.

The main contributions of this paper are as follows:

• We constructed a high-quality super-resolution dataset for the infrared image super-resolution task.
• We enrich the degradation process by adding a gate mechanism and random shuffle strategy to the degradation process.
• We introduce a spatially correlative loss function to improve the model’s ability to retain structural texture.

2. Related Works

Infrared image super-resolution algorithms aim to construct real and reliable high-resolution infrared images from low-resolution ones for visual analysis in the real world. Real and reliable means that the super-resolved images should perform well in detail recovery, denoising, deblurring, artifact removal, and subjective perceptual quality.

2.1. Traditional Algorithms

Traditional image super-resolution algorithms rely on the statistical information of the image and fixed pixel information to generate high-resolution images. These methods include statistical-based [24], edge-based [25,26], patch-based [27,28], prediction-based [29,30], and sparse representation techniques [31]. Although these methods have been widely applied in practice, it is worth noting that they cannot eliminate issues such as noise, blurring, and visual artifacts.

2.2. Deep Learning Methods

Advances in deep neural network technology and enhanced computing power have led to deep-learning-based image super-resolution algorithms that surpass the traditional methods in terms of both objective metrics and subjective perception. Initially, these methods primarily employed convolutional neural networks for end-to-end mapping on artificially paired data, exemplified by the SRCNN [32] model, which pioneered the use of neural networks in the super-resolution task. Subsequently, researchers have made further improvements to network model design, proposing network architectures such as FSRCNN [32], ESPCNN [33], VDSR [34], SRResNet [35], RDN [36], SRDenseNet [37], DRCN [38], DRRN [39], RCAN [40], etc. These designs incorporated ideas including aspects of network depth, feature dimensions, residual information, dense stacking, and recursive structures, and effectively improved the super-resolution performance. In practice, task-based designs that rely on network training often yield more significant performance improvements. Constraints based solely on pixel loss in end-to-end mapping can result in images that appear blurry and overly smooth. However, generative models based on generative adversarial networks (GANs) [41] can mitigate this limitation. Through adversarial training between a generator and a discriminator, GAN-based models can progressively acquire the ability to generate realistic details.

The SRGAN [35] model was the first to introduce GAN networks into the super-resolution task and proposed perceptual loss, which effectively addressed the issue of the low perceptual quality in reconstructed images, thereby enhancing the realism of the generated images. The ESRGAN model [42] introduced the RRDBNet generator network structure. However, the exclusive focus on the classic degradation process restricts its applicability in real-world scenarios. In comparison to the kernel-based degradation estimation in RealSR [43], Real-ESRGAN [44] introduced higher-order degradation processes that more accurately reflect real-world conditions. Subsequently, the BSRGAN [16] model and GD model [17] independently proposed the shuffle strategy and a gate mechanism to diversify the degradation space. However, these operations merely capture non-combinations within the degradation space and fail to establish a unified baseline degradation model.

In the quest to enhance image details, researchers initially concentrated on optimizing the discriminator’s role. The SRGAN [35] model employs a discriminator that directly assesses the authenticity of generated images, whereas the ESRGAN [42] model refines the discriminator to gauge relative realism rather than absolute values. Real-ESRGAN [44] further employs a U-Net [45] structure to discriminate the realism of images at the pixel level, effectively improving the model’s generalization and performance. The AESRGAN [46] model adopts a multi-scale U-Net discriminator to further sharpen details. However, in infrared super-resolution tasks that prioritize texture restoration, noise interference can exacerbate artifacts. Therefore, it is necessary to further constrain texture and structural information through loss functions, ensuring that the generative model converges effectively within the degradation space and demonstrates generalization in real-world scenarios.

2.3. Open-Source Infrared Datasets

High-quality datasets are crucial for deep neural network-based super-resolution algorithms. The diversity of images is essential for learning generalized representations, which in turn significantly enhances the model’s generalization capabilities. In the realm of visible light image algorithms, a plethora of high-quality open-source datasets are readily accessible. For example, commonly used datasets like DPED [13] and DF2K [11,14] provide unified training and evaluation benchmarks for algorithms. However, in the field of infrared images, the availability of datasets is limited, with examples including [47,48,49] or datasets derived from self-taken images [50]. However, the quality of these images varies significantly, and there is a notable lack of scene diversity. For instance, images extracted from the same video often exhibit considerable similarities, thereby diminishing the diversity of the dataset. Additionally, the majority of infrared images obtained via web crawlers, which typically have very low resolutions, are not conducive to algorithm development.

Therefore, we constructed a multi-scene infrared super-resolution image dataset [51]. This dataset contains a total of 7224 images across nine distinct scenes, including seascape images, animal images, industrial scenes, urban landscapes, natural landscapes, indoor images, portraits, surveillance perspectives, and vehicle perspectives, as shown in Figure 1. The majority of images have resolutions of (1280 × 1024, 1920 × 1080), with a minority of (640 × 512), providing rich details and texture information with high resolution and quality. This dataset facilitates the development of more powerful and accurate super-resolution algorithms using real-world high-quality infrared image data, thus promoting the application of infrared super-resolution technology across diverse fields.

3. Method

This article builds upon the foundation of the Real-ESRGAN [44] model and proposes improvements to it.

3.1. Degradation Model

3.1.1. Prior Research

Blind super-resolution (Blind SR) represents a complex ill-posed inverse problem, positing that high-resolution (HR) images suffer from various forms of degradation. Typically, classic degradation models [52,53] are used to synthesize low-resolution inputs:

$$\mathit{x}=\mathcal{D}\left(\mathbf{y}\right)={[(\mathit{y}\otimes \mathit{k}){\downarrow }_r+\mathit{n}]}_{\mathrm{JPEG}}$$

(1)

where $\mathcal{D}$ represents the degradation process. In general, the high-resolution image $\mathit{y}$ is convolved with a blur kernel $\mathit{k}$, followed by a downsampling operation with a scale factor $\mathit{r}$ and the addition of noise $\mathit{n}$. Finally, the process concludes with the application of a commonly used JPEG compression, resulting in a low-resolution image $\mathit{x}$.

Traditional degradation models, although simple and mathematically convenient, often fall short in representing the spectrum of degradation present in actual low-resolution (LR) images. LR images may be affected by degradation such as blurring, downsampling, and various levels of noise with different orders. Therefore, a random shuffle strategy [16] is proposed to address this issue.

The degraded sequence $\{{\mathbf{B}}_{iso},{\mathbf{B}}_{aniso},{\mathbf{D}}^s,{\mathbf{N}}_G,{\mathbf{N}}_{JPEG},{\mathbf{N}}_s\}$ is shuffled randomly, where ${\mathbf{D}}^s$ denotes performing a downsampling operation with scale factor s randomly selected from the set $\{{\mathbf{D}}_{nearset}^s,{\mathbf{D}}_{bilinear}^s,{\mathbf{D}}_{bicubic}^s,{\mathbf{D}}_{down-up}^s\}$. In particular, ${\mathbf{D}}_{down-up}^s$ sequences of ${\mathbf{D}}_{down-up}^{s/a}$ and ${\mathbf{D}}_{up}^s$ can be inserted into other degradations.

To obtain more realistic degradation, ref. [44] enhanced the classic degradation model by extending it to high-order degradation processes. Classical degradation models, which can be considered as first-order modeling, only contain a limited number of degradations. In real-world scenarios, the degradation is varied and often involves multiple steps. These complex degradation processes cannot be adequately captured by a classic degradation model. Instead, a more sophisticated high-order degradation model is needed, which incorporates n repetitive degradation processes. Each degradation process within the model is described using a classical degradation model, as demonstrated:

$$\mathit{x}={\mathcal{D}}^n\left(\mathit{y}\right)=({\mathcal{D}}_n\circ \cdots \circ {\mathcal{D}}_2\circ {\mathcal{D}}_1)\left(\mathit{y}\right)$$

(2)

High-order degradation uses the same procedure but with varying hyperparameters. The term “higher-order” in this context refers to the execution time of the operation. The key aspect is the repeated execution of the operation with different parameters.

While a random shuffle strategy and high-order degradation model are effective in managing complex degradation scenarios by encompassing all basic degradation types, they often overlook important extreme cases prevalent in the real world. These extreme cases involve combinations of different subsets of basic degradation types. In order to tackle this issue, ref. [17] introduced a degradation model that utilizes a gate mechanism to select the basic degradation types that should be randomly incorporated into the degradation process:

$$I^{LR}={\mathcal{D}}_g\left(I^{HR}\right)=({\sigma }_g\left({\mathbf{D}}_1\right)\circ {\sigma }_g\left({\mathbf{D}}_2\right)\circ \cdots \circ {\sigma }_g\left({\mathbf{D}}_m\right))\left(I^{HR}\right)$$

(3)

where ${\mathcal{D}}_g$ represents the gated degradation process, and ${\mathbf{D}}_i\in \{{\mathbf{D}}_k,{\mathbf{D}}_n,{\mathbf{D}}_j,\dots \},i\in \{1,\dots ,m\}$ denotes the basic degradation type. ${\sigma }_g$ is the gate controller that determines whether a specific degradation type ${\mathbf{D}}_i$ should be included in the degradation process, given by

$${\sigma }_g\left({\mathbf{D}}_i\right)\left(I^d\right)=\left\{\begin{array}{cc}{\mathbf{D}}_i\left(I^d\right),\hfill & \hfill g=1,\\ I^d,\hfill & \hfill g=0,\end{array}\right.$$

(4)

where $I^d$ represents the input high-resolution (HR) image. When all gates are turned on, the gated degradation model becomes the same as the complex degradation model, while when all gates are shut off, it is equivalent to the traditional non-blind SR model. By manipulating the gate controller, a variety of combinations of basic degradation types can be generated, resulting in a model that covers non-blind SR, classic blind SR, and practical blind SR.

3.1.2. The Proposed Degradation Model

The proposed degradation model aims to reduce computational complexity, while enriching the degradation space. In the degradation process, a gate mechanism and random shuffle strategy are introduced into the second-order degradation model of Real-ESRGAN. Additionally, based on the characteristics of the experimental data and infrared devices, JPEG compression was not performed. This decision was made because the experimental data were stored in a single-channel format, while JPEG compression was primarily used for compressing color information in RGB images. Moreover, in practical scenarios, infrared devices can always export in lossless BMP format, thereby minimizing the need for JPEG compression of their images. Consequently, JPEG compression was excluded from the degradation process. The architecture of the proposed degradation model is illustrated in Figure 2.

The high-resolution (HR) image undergoes a two-stage degradation process to obtain the low-resolution (LR) image. In Equation (5), ${\mathcal{D}}_1$ and ${\mathcal{D}}_2$ represent the first-order and second-order degradation processes, respectively. The LR image can be obtained by

$$I^{LR}=({\mathcal{D}}_1\circ {\mathcal{D}}_2)\left(I^{HR}\right).$$

(5)

In each round of degradation, we perform a random shuffling of the degradation sequence $\{Blur,Noise,Resize\}$. We use ${\mathbf{D}}_n$ to represent the basic degradation, where n represents the corresponding sequence. Here, in Equation (6), $I^{IN}$ denotes the input image, and $I^{OUT}$ represents the output image.

$$I^{OUT}=\mathbf{D}\left(I^{IN}\right)=({\sigma }_1\left({\mathbf{D}}_1\right)\circ {\sigma }_2\left({\mathbf{D}}_2\right)\circ {\sigma }_3\left({\mathbf{D}}_3\right)\circ \cdots \circ {\sigma }_n\left({\mathbf{D}}_n\right))\left(I^{IN}\right).$$

(6)

For example, for a specific shuffling order $\{3,1,2\}$, the proposed degradation model processes the image as follows:

Step 1: Resize-${\mathbf{D}}_3$, $I^{OUT1}={\sigma }_1\left({\mathbf{D}}_3\right)I^{IN}$

Step 2: Blur-${\mathbf{D}}_1$, $I^{OUT2}={\sigma }_2\left({\mathbf{D}}_1\right)I^{OUT1}$

Step 3: Noise-${\mathbf{D}}_2$, $I^{OUT}={\sigma }_3\left({\mathbf{D}}_2\right)I^{OUT2}$

Each order generates a random list $\{{\sigma }_1,{\sigma }_2,\cdots ,{\sigma }_n\}$, which determines whether is ${\mathbf{D}}_i$ executed. If the value ${\sigma }_i$ is 0, the corresponding degradation process is not performed. If the ${\sigma }_i$ is 1, the degradation process is executed. To ensure the complexity of degradation, in our experiments the random list $\{{\sigma }_1,{\sigma }_2,\cdots ,{\sigma }_n\}$ was not all filled with 0 s, meaning that at least one degradation process was performed.

The blur degradation uses both isotropic and anisotropic Gaussian noise. The noise degradation applies Gaussian noise and Poisson noise. For the resize degradation, the image is randomly resized using area, bilinear, and bicubic interpolation.

By combining a gate mechanism and random shuffle strategy, the proposed degradation model can flexibly handle different basic degradation types and generate diverse degradation patterns. Such a strategy can better simulate diverse degradation situations in the real world and improve the adaptability and robustness of the model.

3.2. Network Structure

We adopted the same generator, RRDBNet, as the Real-ESRGAN [44] model, as well as a U-Net discriminator with skip connections. RRDBNet is a deep neural network architecture designed for image super-resolution tasks. It is based on the RRDB structure, standing for residual-in-residual dense blocks, that is formed by cascading three residual dense blocks (RDB) with an additional residual connection added. Our implementation of RRDBNet integrates 23 RRDBs to formulate a robust and efficacious model tailored for augmenting the resolution of images. The complete network structure of Inf-OSRGAN is shown in Figure 3.

3.3. Spatially Correlative Loss

To maintain the accuracy and perceptual quality of the reconstructed images within the degradation space described in Section 3.1, we introduced a spatially correlative loss function, namely F-LSeSim [23], that constrains the reconstruction outcomes by calculating the correlation between feature maps, paying more attention to maintaining the structural consistency and detail information to improve the accuracy and realism of the generated images.

F-LSeSim is designed for image-to-image (I2I) translation tasks requiring strict constraints on the structural consistency before and after image translation. FSeSim [23] employs pre-trained networks like VGG16 [54] to extract features, and then constructs spatial correlation maps at the patch level [55,56,57], encouraging structural consistency in the corresponding regions. Building upon this, LSeSim [23] discards the use of pre-trained networks and optimizes a new feature extraction network specifically for each task through self-supervised learning. This enables the network to provide more accurate spatial correlation maps for a specific task.

F-LSeSim

We introduced the constraint concept of F-LSeSim into the super-resolution task. The computation process is detailed as follows:

We generate contrastive maps at the patch level, following PatchNCE [58].

Let $v=S_{x_i}\in {\mathbb{R}}^{1\times N_p}$ denote the spatially correlative map associated with the “query” patch in the ground-truth image. $v^{+}=S_{{\widehat{x}}_i}\in {\mathbb{R}}^{1\times N_p}$ is a positive sample at the same location as the query region in the generated image $G\left(x\right)$. $v^{-}\in {\mathbb{R}}^{K\times N_p}$ is negative samples.

The query patches are positively matched with patches located at identical positions in the generated image $G\left(x\right)$ and negatively matched with patches taken from different positions within in $G\left(x\right)$ or from other images. A total of $K=255$ negative patches are utilized. The LSeSim contrast loss is given Equation (7):

$${\mathcal{L}}_c=-log{\displaystyle \frac{e^{sim(\mathit{v},{\mathit{v}}^{+})/\tau }}{e^{sim(\mathit{v},{\mathit{v}}^{+})/\tau }+{\sum }_{k=1}^K{e}^{sim(\mathit{v},{\mathit{v}}_k^{-})/\tau }}}$$

(7)

In Equation (7), $sim(\mathit{v},{\mathit{v}}^{+})={\mathit{v}}^T{\mathit{v}}^{+}/\parallel \mathit{v}\parallel \parallel {\mathit{v}}^{+}\parallel$ denotes the cosine similarity of the two spatially correlated patches and $\tau$ act as the temperature parameter. To minimize the loss, LSeSim promotes the closeness of corresponding patches that share similar structures, even if they have very different visual appearances, maintaining the structural coherence of the images. The loss in Equation (7) is exclusively employed to optimize the structural representation network.

FSeSim generates spatially correlative maps by calculating the self-similarity between features extracted from LSeSim. These maps elucidate the spatial relationships among various structures within the images, rather than their visual appearance.

Let x be the generated image and $\widehat{y}$ be the corresponding ground-truth image, respectively. Using LSeSim, we can extract features from the two input images, denoted as $f_x$ and $f_{\widehat{y}}$. For each query point in the image, the self-similarity between this feature and the features of other points in the image is computed, creating what is known as a spatially correlative map, formally represented as Equation (8):

$$S_{x_i}={\left(f_{x_i}\right)}^T\left(f_{x_{\ast }}\right)$$

(8)

where $f_{x_i}^T\in {\mathbb{R}}^1$ represents the feature of a particular query point $x_i$, and the corresponding features contained within a patch composed of $N_p$ points are included in $f_{x_{\ast }}\in {\mathbb{R}}^{N_p}$. By combining self-similarity maps of multiple query points $S_x=[S_{x_1};S_{x_2};\dots ;S_{x_s}]\in {\mathbb{R}}^{N_s\times N_p}$, a spatially correlative map of the entire image is formed, where $N_s$ stands for the number of sampled patches. These maps show the spatial relationships between different regions in the image. Subsequently, a comparison is made between several structural similarity maps, focusing on the ground-truth $\widehat{y}$ and generated image x, as Equation (9) shows below:

$${\mathcal{L}}_s=d(S_x,S_{\widehat{y}})$$

(9)

In this equation, the spatially correlative map corresponding to the ground-truth images is signified by $S_{\widehat{y}}$. Two different forms of $d(·)$ are considered: the $L1$ distance $\Vert S_x-S_{\widehat{y}}{\Vert }_1$ and the cosine distance $\Vert 1-cos(S_x,S_{\widehat{y}})\Vert$. The $L1$ distance promotes the spatial similarity across all points within a patch that is uniform, emphasizing the similarity in magnitude. Meanwhile, the cosine distance supports pattern correlation, irrespective of magnitude differences, focusing more on the overall pattern rather than specific magnitudes. In the generator, the spatially correlative loss is consistently defined by the loss in Equation (9).

By incorporating spatially correlative loss, we can constrain the reconstruction outcomes by calculating the correlation between features. This method emphasizes maintaining the structural consistency and capturing fine details of the image, thereby improving the accuracy and realism of the reconstructed image.

4. Experiments

4.1. Experiment Setup

4.1.1. Implementation Details

To demonstrate the advantages of the proposed model, we trained six 4× super-resolution networks using the widely used Real-ESRGAN generator, namely RRDBNet, and executed comparative experiments on diverse datasets. To reduce redundant computations and adapt to infrared images, we modified the input and output channels to be single-channel and used LR/HR synthetic pairs generated by the proposed degradation model for training. Firstly, we trained the first-stage model, Inf-OSRNet, using $L1$ loss with a learning rate of $2^{-4}$. Then, we trained a Inf-OSRGAN model, which combined $L1$ loss, perceptual loss, GAN loss, and spatially correlative loss with weights $\{1,1,0.1,1\}$ and a learning rate of $1^{-4}$. Both stages of training were iterated 100,000 times, using the Adam optimizer.

4.1.2. Degradation Details

During the two-stage training process, we employed the degradation process described in Section 3.1, which incorporated a shuffle strategy and gate mechanism into the second-order degradation process of Real-ESRGAN, excluding JPEG compression. The second-order degradation process involves three primary operations: blur, noise, and resize, which can be combined in six different sequences due to the shuffle strategy. The gate mechanism was employed to selectively execute these operations, ensuring a controlled application of degradation processes in the pipeline. For the blur operation, three types of kernels were utilized: Gaussian kernels with a probability of 0.7, generalized Gaussian kernels with a probability of 0.15, and plateau-shaped kernels with a probability of 0.15. The Gaussian kernels had a size range of 7 to 21 and a standard deviation varying from 0.2 to 3 for first-order degradation and 0.2 to 1.5 for second-order degradation. The generalized Gaussian kernels featured a shape parameter $\beta$ ranging from 0.5 to 4, while the plateau-shaped kernels had a $\beta$ value between 1 and 2. The noise operation consisted of Gaussian noise and Poisson noise, each with a probability of 0.5. The standard deviation for Gaussian noise ranged from 1 to 30 for the first-order and 1 to 25 for the second-order degradation. Poisson noise scaled from 0.05 to 3 in the first order and 0.05 to 2.5 in the second order. Resizing operations considered area, bilinear, and bicubic interpolation methods, applicable for both upsampling and downsampling, with a scaling range of 0.15 to 1.5. After completing the degradation process in the two stages, all images had to be resized to one-quarter of the size of the high-resolution (HR) images. Subsequently, the images underwent processing through a sinc filter, which was applied in a random manner.

4.1.3. Datasets

We adopted 6498 images from the training set of the infrared super-resolution dataset [51] mentioned in Section 2.3 for the experiment. We downsampled HR images to obtain several images with different scales and generated multi-scale versions with different scale factors (0.75, 0.5, 1/3). When evaluating our model, we used high-resolution images from the infrared super-resolution test set mentioned in Section 2.3 and other low-resolution images from open-source infrared datasets [59,60].

4.1.4. Evaluation Metrics

We evaluated different models using mean peak signal to noise ratio (PSNR), structural similarity (SSIM), and Fréchet inception distance (FID).

PSNR evaluates the quality of reconstructed images by quantifying the difference between the original and the reconstructed signal by calculating the ratio of the maximum possible power of a signal to the power of the noise, determining the ratio of the maximum possible power of a signal to the noise power. A higher PSNR value signifies that the reconstructed image closely resembles the original, indicating better quality. PSNR is usually measured in decibels (dB), calculated as in Equation (10):

$$\mathrm{PSNR}=10·{log}_{10}\left({\displaystyle \frac{R^2}{\mathrm{MSE}}}\right)$$

(10)

where $\mathit{R}$ is the maximum possible pixel value of the image and $\mathrm{MSE}$ is the mean squared error.

SSIM, examining the similarity of two images, takes into account their structural information, luminescence, and contrast. The SSIM index, which varies from −1 to 1, has a value of 1 when images perfectly match. The SSIM index can be broken down into three components: luminance (L), contrast (C), and structure (S), with the overall formula given by Equation (11):

$$\mathrm{SSIM}(I,K)={\displaystyle \frac{(2{\mu }_I{\mu }_K+C_1)(2{\sigma }_{IK}+C_2)}{({\mu }_I^2+{\mu }_K^2+C_1)({\sigma }_I^2+{\sigma }_K^2+C_2)}}$$

(11)

where ${\mu }_I$ and ${\mu }_K$ are the mean luminance of images $\mathit{I}$ and $\mathit{K}$; ${\sigma }_I^2$ and ${\sigma }_K^2$ are the variances of the respective images; and ${\sigma }_{IK}$ is the covariance between the two images. $C_1$ and $C_2$ are constants used for stability in the calculations.

FID gauges the resemblance between two image sets by computing the distance between their feature representations, with lower FID values suggesting a closer resemblance between the actual and generated images. The calculation of FID involves the following steps. We extracted features from real and generated images, then calculated the mean and covariance of the features for both sets, including the mean and covariance of the real image features $({\mu }_r,{\Sigma }_r)$, and the mean and covariance of the generated image features $({\mu }_g,{\Sigma }_g)$. The FID is calculated as in Equation (12):

$$\mathrm{FID}=\parallel {\mu }_r-{\mu }_g{\parallel }^2+\mathrm{Tr}({\Sigma }_r+{\Sigma }_g-2{\left({\Sigma }_r{\Sigma }_g\right)}^{1/2})$$

(12)

where Tr denotes the trace operator.

4.1.5. Training Curves

We present the training loss curve in Figure 4, where the horizontal axis represents the number of training iterations and the vertical axis represents the loss value. It can be observed from the graph that the loss value exhibited a consistent downward trend as the training progressed. This indicates that our algorithm gradually optimized the model parameters, resulting in an improved alignment between the predicted values and the ground truth. The decreasing trend of the loss curve reflects the learning capacity and optimization effectiveness of our model, providing strong support for our research.

We include the PSNR (peak signal-to-noise ratio) curve in Figure 5, where the horizontal axis represents the training iterations and the vertical axis represents the PSNR value.

In the first stage of training, we observed a consistent increase in the PSNR as the training progressed. This indicates that our algorithm successfully improved the quality of the reconstructed images at the pixel level, resulting in higher PSNR values. However, in the second stage of optimization, the objective was not solely focused on pixel-level accuracy. Instead, it aimed to enhance the visual quality of the images. As a result, we noticed a slight decrease in the PSNR values. This can be attributed to the fact that the optimization process prioritized perceptual aspects, such as enhancing texture details and reducing artifacts, rather than solely maximizing the PSNR metric. Despite the decrease in PSNR, it is important to note that the visual quality of the images significantly improved.

4.2. Comparisons with Prior Works

We compared our proposed Inf-OSRGAN model with several state-of-the-art super-resolution models, including RCAN [40], SRGAN [35], RealSR [43], and Real-ESRGAN [44]. We utilized low-resolution (LR) images obtained by downscaling the images from the test set by a factor of four. The experiments were conducted on a Linux (Ubuntu 20.04 LTS) system with an NVIDIA RTX A6000 GPU (Nvidia, Santa Clara, CA, USA).

Table 1. The various models along with their corresponding number of parameters and the time taken to process a single image. All models were tested on a Linux (Ubuntu 20.04 LTS) server equipped with an NVIDIA RTX A6000 GPU.

Model Name	Parameters (M)	Processing Time Per Image (s)
RCAN	1.67	0.2
SRGAN	1.47	0.2
RealSR	16.70	0.6
Real-ESRGAN	16.70	0.6
Inf-OSRGAN	16.70	0.6

presents a comparison of our method with prior works in terms of runtime and number of parameters.

While our approach, along with Real-ESRGAN (our baseline), exhibited a larger model size and slightly longer runtime, it compensated for this with superior visual quality. As illustrated in Figure 6, our method outperformed prior works in both overall aesthetic appeal and detail restoration, demonstrating a marked improvement in the fidelity of the images produced. Notably, Real-ESRGAN also exhibited better visual results compared to the other existing methods.

The primary focus of our optimization efforts was to enhance the super-resolution performance and visual quality, rather than to prioritize processing speed. The decision to sacrifice some processing time in favor of significant improvements in visual outcomes is a worthwhile trade-off. This strategy aligns with our goal of delivering high-quality images that meet the demands of various applications where visual fidelity is of utmost importance. By prioritizing these aspects, we are confident that the benefits of enhanced visual performance substantially outweigh the costs associated with increased processing time.

4.3. Ablation Studies

To illustrate the efficacy of our proposed degradation and spatially correlative loss function, this chapter presents ablation studies conducted on the baseline network of Real-ESRGAN. In the ablation studies, spatially correlative loss was first embedded into the baseline network without modifying the degradation process. For the degradation component, a gate mechanism and random shuffle strategy were individually integrated into the baseline network for separate evaluations, and then the degradation methods of the gate mechanism and random shuffle strategy were jointly embedded into the baseline network. Finally, both the spatially correlative loss and our degradation were embedded into the baseline network, resulting in the Inf-OSRGAN model.

The experimental results are shown in Figure 7, with the best results highlighted with stars. The ablation studies followed the same experimental settings as the main experiment.

It can be observed in Figure 7 that, compared to the baseline, the proposed Inf-OSRGAN led to an improvement, particularly in the degradation. Specifically, when only the random shuffle was added, the PSNR increased from 27.89 to 27.92. When only the gate mechanism was added, the PSNR increased from 27.89 to 29.70. When both the gate mechanism and random shuffle were added, the PSNR improved to 29.95. The optimization of metrics could also be observed in terms of SSIM and FID.

Low-resolution (LR) images were obtained through interpolation from the original images. When there was a rich degradation space, the super-resolution algorithms could recover more high-frequency details and textures from the low-resolution inputs, thereby improving the image quality. The improvement in super-resolution effect could be reflected through the metrics considering pixel-level, feature distribution, brightness, and contrast. Therefore, these metrics could effectively evaluate the improvement in super-resolution algorithms resulting from a rich degradation space.

Upon the integration of the spatially correlative loss function, there was a marginal reduction in PSNR, FID, and SSIM values. The introduction of spatially correlative loss in super-resolution tasks enhances the structural integrity of the reconstructed images, leading to a more uniform and visually cleaner output. Since the optimization goal of spatially correlative loss is not entirely consistent with traditional metrics, they may decrease. Spatially correlative loss prioritizes the optimization of image structure rather than pixel-level similarity. This approach aids in preserving the structural consistency of the image, emphasizing enhancing details and textures. Consequently, it yields a clearer and more visible image structure but also tends to smooth out the noise signal.

The traditional metrics primarily focus on the pixel-level similarity and statistical features of an image, rather than maintaining the structural consistency. Therefore, when introducing spatially correlative loss, the reconstructed image’s structure may be more pronounced, but a decrease in comparison with traditional metrics may be observed. This does not imply a decline in the actual visual quality and enhancement of details. In fact, enhancing spatial consistency can improve the structure and details of an image, leading to improved perceptual quality. Spatially correlative loss can effectively handle texture details and high-frequency information. Super-resolution tasks often involve the recovery of textures and edges. Traditional loss functions may struggle to capture these intricate patterns and tend to amplify artifacts of the generated images. In contrast, spatially correlative loss incorporates spatial information, enabling the network to better preserve and reconstruct these high-frequency components. This leads to sharper and more accurate representations of the underlying structures in the super-resolved images.

4.4. Visual Results

4.4.1. Results on Our Datasets

Metrics like PSNR and SSIM can be used to quantify the variances between high-resolution images and their reconstructed counterparts. However, these metrics only offer a partial assessment and cannot fully represent real perception. Due to the high sensitivity and subjectivity of human vision regarding image quality, sole reliance on objective metrics falls short in providing a comprehensive evaluation of visual quality. Visual comparison, however, can account for perceptual attributes such as details, textures, clarity, and naturalness.

Figure 8 displays the results of the super-resolution model on high-resolution images (1280 × 1024) from the test set proposed in Section 2.3, with the bicubic upsampled LR images used as references.

Compared to the baseline, after employing our degradation, the model was better able to enhance details and preserve more textures. Concurrently, it risked over-emphasizing non-existent details, thereby introducing supplementary noise and artifacts. The buildings depicted in the final set of images serve as a case in point.

Incorporation of spatially correlative loss accentuates the structural integrity of an image, yielding a more uniform visual outcome. However, this approach risks over-smoothing, which can result in the attenuation of subtle textures and the distortion of intricate details, as exemplified by the trunk images in the sixth set. Conversely, the absence of spatially correlative loss can lead to an increase in artifacts and a less pronounced structural clarity in the generated images. The proposed Inf-OSRGAN combines both approaches, effectively reducing noise, while preserving sharp edges and rich details, thus achieving a more realistic visual effect.

4.4.2. Results on Other Datasets

Figure 9 shows the results of the previous methods applied to low-resolution infrared images (256 × 192, 384 × 288) from other publicly available datasets [59,60,61,62,63], an infrared security target identification dataset and an infrared image denoising dataset.

By processing these low-resolution data, our model was able to reconstruct detailed textures, thereby enhancing their clarity and naturalness. Additionally, the model effectively suppressed noise and artifacts, enhancing the visual quality of the images. For the sixth set of images, the result of the baseline exhibited a lack of clarity in the main structures, with fine hair strands appearing indistinct. When our degradation model was integrated, the hair textures became more pronounced, but this came at the cost of introducing artifacts that cluttered the image. The application of the spatially correlative loss function improved the clarity of the main structures but resulted in an over-smoothing of weaker textures. The Inf-OSRGN model successfully emphasized the main structures while maintaining the hair textures and did not suffer from the issue of artifacts.

This set of low-resolution experimental data differed significantly from the test set mentioned earlier in terms of resolution and capture devices. This demonstrates the strong generalization capability of the Inf-OSRGAN model when applied to diverse datasets.

4.5. Discussion

Our model has a few limitations that are worth mentioning. Firstly, due to the diverse range of degradation modes present in infrared images, the convergence of our model requires a longer training time. This is because the model needs to learn and adapt to various degradation patterns, which leads to a more complex optimization process. Additionally, our model does not specifically simulate certain degradation models unique to infrared images, such as blind pixels or elemental stripes. These degradation modes tend to result in overcompensation, leading to worse results. This limitation arises from the fact that the training data and the designed loss function may not fully capture the complexities associated with these specific degradation modes.

It is important to acknowledge that these limitations suggest potential areas for future research and improvement. Future work could focus on developing more robust models that can handle a wider variety of degradation modes and specifically address the challenges posed by unique infrared degradation patterns. By addressing these limitations, we can further enhance the performance and applicability of our proposed model. Additionally, we plan to explore the applicability of the optimization techniques proposed in this paper to other super-resolution models. By applying these optimization methods to different models, we aim to assess their effectiveness and potential for improving the performance of various super-resolution approaches. Furthermore, we intend to investigate the transferability of these optimization strategies across different domains, such as medical imaging, satellite imagery, or low-light photography. This will involve conducting a series of experiments to evaluate how well these techniques perform when applied to datasets specific to these domains, which often present unique challenges, such as the varying anatomical structures in medical images. By evaluating the performance of the proposed techniques in these diverse domains, we will gain insights into their adaptability and potential for wider adoption. In addition, if there is sufficient demand, we may develop a software interface for the Inf-OSRGAN model, to enhance the user experience and accessibility.

5. Conclusions

To overcome the constraints of current infrared image super-resolution algorithms, this paper proposed an optimized method for super-resolution in the infrared image domain. Firstly, we constructed a high-quality infrared image dataset enriched with detailed textures, serving as the basis for evaluating infrared image super-resolution models. In terms of degradation, we incorporated a gate mechanism and random shuffle strategy to encompass a broad spectrum of realistic degradation encountered in real-world situations. Furthermore, in the training process of the Inf-OSRGAN network, we integrated a spatially correlative loss function to measure feature correlations, to ensure the preservation of structural consistency and detailed information, thereby enhancing both the accuracy and perceptual fidelity of the reconstructed images.

We validated the proposed method on the constructed high-resolution test set and various publicly available low-resolution infrared image datasets. Our method achieved outstanding performance on both the high-resolution test set and a diverse array of publicly available datasets. The experimental results demonstrated the superior capabilities of our method for detail restoration, with a particular emphasis on the recovery of nuanced textures. Specifically, these experiments substantiated the method’s reliability and practicality, showing its versatility across various scenes and scenarios, as well as its robust generalization across images of varying resolutions. The findings of this study offer a potent approach to infrared image enhancement.