Low-Light Image Enhancement Using CycleGAN-Based Near-Infrared Image Generation and Fusion

Abstract

Image visibility is often degraded under challenging conditions such as low light, backlighting, and inadequate contrast. To mitigate these issues, techniques like histogram equalization, high dynamic range (HDR) tone mapping and near-infrared (NIR)–visible image fusion are widely employed. However, these methods have inherent drawbacks: histogram equalization frequently causes oversaturation and detail loss, while visible–NIR fusion requires complex and error-prone images. The proposed algorithm of a complementary cycle-consistent generative adversarial network (CycleGAN)-based training with visible and NIR images, leverages CycleGAN to generate fake NIR images by blending the characteristics of visible and NIR images. This approach presents tone compression and preserves fine details, effectively addressing the limitations of traditional methods. Experimental results demonstrate that the proposed method outperforms conventional algorithms, delivering superior quality and detail retention. This advancement holds substantial promise for applications where dependable image visibility is critical, such as autonomous driving and CCTV (Closed-Circuit Television) surveillance systems.

1. Introduction

The development of effective surveillance systems and driver assistance technologies has become increasingly critical in recent years, particularly under challenging visual conditions. Ensuring clear visibility and accurate object detection in low-light adverse environments remains a significant challenge. To enhance image quality, techniques such as multi-exposure imaging or multi-band imaging (e.g., using IR cut switching during the day and auxiliary IR LED for night images) are often employed [1,2]. However, these approaches face several limitations, including high manufacturing costs, lengthy image acquisition times, and technical challenges such as multi-image alignment [3].

Numerous algorithms have been proposed to address these challenges. The iCAM06 algorithm, for instance, enhances HDR images through a systematic workflow involving XYZ color space conversion, chromatic adaptation, IPT color space transformation, and tone mapping with color correction [4]. Despite its effectiveness, iCAM06 often struggles with performance consistency under varying lighting conditions and may introduce artifacts in extreme scenarios.

Similarly, the L1LO algorithm employs a frequency-domain approach by decomposing images into multiple frequency bands and applying adaptive linear operators [5]. While computationally efficient, this method frequently fails to preserve fine details in complex scenes and struggles in regions with dramatic illumination changes.

Advanced methods such as KWON’s contrast sensitivity-based HDR image decomposition address these limitations by employing CSF-based multiscale decomposition to multiple base and detail layers [6]. This sophisticated processing pipeline, which includes Gaussian pyramid down-sampling and the application of CSF-based weight, excels at preserving fine details and minimizing halo artifacts. However, it demonstrates limitations in high-contrast regions, where its performance can degrade.

The Reinhard algorithm, widely recognized for HDR tone mapping, supports both global and local approaches [7,8]. The global method relies on log-average luminance calculations and scale factors for luminance compression, while the local version uses Gaussian-weighted averages for adaptive tone mapping. This versatility makes it particularly effective for general-purpose HDR processing. However, it can struggle to maintain local contrast in complex scenes.

Modern deep learning-based approaches have also emerged as promising solutions. RetinexNet, which leverages Retinex theory within a neural network framework [9], enhances low-light images by decomposing them into illumination and reflectance components for targeted enhancements. Its unsupervised learning approach allows for natural low-light enhancement without the need for paired training data.

Kim et al.’s method combines Retinex-based image enhancement techniques with contrast-limited adaptive histogram equalization (CLAHE) to improve both global and local contrast while preserving natural colors [10]. While effective in enhancing low-light or unevenly illuminated images, it causes slight color desaturation in dark regions.

Despite these advancements, challenges remain in overcoming the limitations of traditional image enhancement techniques in low-light conditions. While previous methods focused on converting saturated areas into unsaturated areas using images taken at different exposures, they often degrade certain image areas while improving others. This study proposes using a novel approach that leverages infrared images for training the model. The NIR wavelength band, which ranges from 800 to 1500 nm, offers better light penetration, diffraction capabilities, and reduced light saturation. In NIR images, object boundaries and textures are defined with strong contrast and sharpness, making them particularly useful for edge detection. In contrast, visible images are advantageous for expressing color components, and their fusion with NIR images allows for more accurate object identification, as NIR images emphasize the boundaries and textures of the entire image [11].

This work proposes a complementary cycle-consistent generative adversarial network (CycleGAN)-based training method, utilizing both visible and NIR images to create a sophisticated model. Given the difficulty in obtaining paired visible–NIR images for training, a sequential training strategy combining unpaired and paired image data within the CycleGAN framework is implemented. This innovative approach enables the generation of effective NIR training datasets without the need for extensive paired data collection. Then, the framework performs a two-stage mixing process between low-light visible images and virtually transformed NIR images. In the first stage, the difference in image between the visible and NIR images is calculated and used to blend the images by combining the NIR image weighted by the inverse of the difference and the visible image weighted by the difference itself. In the second stage, a bilateral filter is applied to the difference image obtained from the first stage [12], which is used to weight the visible image by its inverse and the synthesized image from the first stage by its difference [13]. This blending process is followed by additional contrast-limited adaptive histogram equalization for final image enhancement. During the fusion process between NIR and visible images, imbalances can arise between the modified luminance channel and the color channels (a and b), potentially resulting in unnatural color representation. To address this, a color adjustment method is proposed, ensuring the restoration of natural color fidelity while preserving the enhanced luminance information.

By fusing NIR and visible images, this method can significantly improve image clarity in low-light conditions [14], reducing oversaturation in low-illumination areas. Experimental results show substantial improvements in image quality under challenging lighting conditions, making this approach particularly valuable for surveillance systems and driver-assistance technologies, where reliable visibility is crucial.

The main contributions of this study are as follows:

▶

To address the lack of paired visible–NIR image datasets and improve training quality, two-stage training is adopted. The first stage uses CycleGAN to generate fake visible images from unpaired data, while the second stage fine-tunes the network with paired fake visible and real NIR images, enhancing domain translation and structural consistency;

▶

The purpose of the two-stage image fusion is to blend luminance information from visible and NIR images effectively. In the first stage, luminance differences are calculated and combined to emphasize details, while the second stage applies a bilateral filter to suppress noise and refine blending, followed by gamma correction to enhance global tone;

▶

Additional local tone processing is achieved using CLAHE to improve local contrast, while color adjustment restores natural chromatic balance by compensating for distortions caused during luminance enhancement.

The remainder of this paper is organized as follows: Section 2 reviews related work in deep learning-based approaches and image fusion. Section 3 details the proposed methodology, including the CycleGAN training strategy, and visible–NIR image blending. Section 4 presents experimental results and comparative analysis. Finally, Section 5 concludes the paper and outlines future directions for research.

2. Related Works

2.1. Cycle-Consistent Generative Adversarial Networks

CycleGAN is a deep learning framework designed for unpaired image-to-image translation, addressing tasks where paired datasets are unavailable. Unlike traditional GANs, which rely on paired datasets for training, CycleGAN employs two generators and two discriminators to learn mappings between two domains [15,16], as shown in Figure 1.

The model includes two generators ($G$ and $F$) and two discriminators ($D_X$ and $D_Y$). The generator $G$ maps data from domain $X$ (visible images) to domain $Y$ (near-infrared images), while the generator $F$ maps data in the reverse direction, from domain $Y$ to domain $X$.

The adversarial loss helps the generator learn to deceive the discriminator into recognizing the generated images as authentic, whereas the cycle-consistency loss (${=L}_{cycle})$ ensures that the translations between domains $X$ and $Y$ remain consistent in both directions. The framework enforces a cycle consistency constraint, ensuring that an image translated from one domain to another can be reverted to its original form. This feature makes CycleGAN particularly valuable for tasks where paired data are unavailable, such as style transfer, domain adaptation, and cross-modal image generation.

$$L_{GAN}(G,D_Y,X,Y)=E_{y\sim p_{data\left(y\right)}}[log{D}_y(y)]+E_{x~p_{data\left(x\right)}}[log\left(1-D_Y\left(G\left(x\right)\right)\right],$$

(1)

where $G$ and $D_Y$ represent the generator and discriminator for domain $Y$, respectively. $X$ and $Y$ denote the source and target domains, while E represents the expectation operator, which calculates the average value of the data distribution. The terms $x~p_{data\left(x\right)}$ and $y\sim p_{data\left(y\right)}$ refer to samples drawn from the data distributions of $X$ and $Y$, respectively. The goal is for $G$ to minimize the difference between $G\left(x\right)$ and real images from $Y$, while $D_Y$ maximizes this difference. This creates a continuous adversarial process where $G$ and $D_Y$ iteratively improve.

The cycle consistency loss ensures that an image translated to the target domain and then back to the source domain remains consistent with the original input. Mathematically, this is expressed as follows:

$$L_{cycle}\left(G,F\right)=E_{x~p_{data\left(x\right)}}[\parallel F\left(G\left(x\right)\right)-x{\parallel }_1]+E_{y\sim p_{data\left(y\right)}}\left[\parallel G\left(F\left(y\right)\right)-y{\parallel }_1\right],$$

(2)

where $G$ and $F$ are the generators for the transformations between domains, and $x$ and $y$ represent data points from the source and target domains, respectively. The notation E represents the expectation operator, which calculates the average value over the data distribution. Specifically, $x~p_{data\left(x\right)}$ and $y\sim p_{data\left(y\right)}$ indicate that $x$ and $y$ are samples drawn from the data distributions of domains $X$ and $Y$, respectively. The term $\parallel {\parallel }_1$ denotes the normalization, which computes the absolute difference between the elements. The first term, $E_{x~p_{data\left(x\right)}}[\parallel F\left(G\left(x\right)\right)-x{\parallel }_1]$, ensures that applying $G$ followed by $F$ on $x$ results in a reconstruction of $x$ itself. Similarly, the second term, $E_{y\sim p_{data\left(y\right)}}\left[\parallel G\left(F\left(y\right)\right)-y{\parallel }_1\right]$, ensures that applying $F$ followed by $G$ on $y$ reconstructs $y$. This is the cycle-consistency loss $L_{cycle}\left(G,F\right),$ which ensures that both transformations $G$ and $F$ are inverses of each other, promoting consistency between the source and target domains. This bidirectional cycle consistency preserves structural integrity and features during translation, ensuring minimal information loss.

$$L_{total}(G,F,D_X,D_y)=L_{GAN}(G,D_Y,X,Y)+L_{GAN}(F,D_X,Y,X)+\lambda L_{cycle}(G,F),$$

(3)

where $L_{total}(G,F,D_X,D_y)$ is a combination of the adversarial losses from both domains and the cycle-consistency loss, with a weighting factor $\lambda$ applied to the cycle-consistency term. The weight $\lambda$ controls the importance of the cycle-consistency loss relative to the adversarial losses. This total loss is minimized during training to ensure that both the generators $G$ and $F$ perform effective domain transformations while maintaining cycle consistency.

Additionally, an adversarial loss is introduced for the mapping function $F$: $Y$ → $X$ and its discriminator $D_X$. The adversarial loss is defined as $\underset{F}{\min }\underset{D_X}{\max }{L}_{GAN}(F,D_X,Y,X)$ where $F$ represents the generator function that transforms data between $Y$ to domain $X$, and $D_X$ is the discriminator for domain $X$. The notation $\underset{F}{\min }\underset{D_X}{\max }$ indicates a minimax optimization process, where $F$ aims to minimize the adversarial loss $L_{GAN}\left(F,D_X,Y,X\right)$ and $D_X$ simultaneously tries to maximize it. The terms $Y$ and $X$ denote the target and source domains, respectively. The adversarial loss, $L_{GAN}$, measures the difference between the generated distribution and the real data distribution in domain $X$. This creates a symmetric adversarial loss for the reverse translation from domain $Y$ to domain $X$, ensuring that both mapping functions are optimized through their respective discriminators.

The network architecture comprises a sophisticated combination of generators and discriminators that work together to achieve bi-directional domain translation. The generator architecture includes three critical components: an encoder, a transformer, and a decoder. The encoder compresses input images into a low-dimensional feature space through convolution layers, instance normalization, and ReLU activation functions.

The transformer, the core component of the generator, incorporates nine residual blocks inspired by the ResNet architecture [17]. These blocks preserve input image features during transformation by leveraging residual connections, which help address optimization challenges. The decoder then generates the transformed image using a combination of transposed convolutions, instance normalization, and ReLU activation functions.

For the discriminator, we employ a PatchGAN architecture that operates on 70 × 70 patches. This design enables efficient parameter usage and localized feature discrimination, allowing the network to process various image sizes with fewer parameters while maintaining high accuracy. The discriminator architecture includes convolution layers, instance normalization, and leaky ReLU components, which work together to effectively distinguish between real and generated images.

The loss calculation methodology utilizes the least squares generative adversarial networks (LSGAN) loss, which offers several advantages over conventional GAN loss functions. LSGAN improves stability during training, produces higher-quality images, and minimizes the risk of mode collapse. This loss function reduces the discrepancy between real and generated images, resulting in realistic and accurate transformations.

This comprehensive framework effectively generates synthetic NIR images from visible inputs, overcoming traditional challenges posed by the scarcity of paired training data by handling unpaired datasets, thereby eliminating the need for time-consuming and costly data labeling.

2.2. Image Fusion

Multi-image fusion techniques combine information from multiple images of the same scene into a single composite image, preserving the most critical features from each input. This process is especially useful when images are captured under varying conditions, such as differences in lighting, focus, or wavelength. By merging these diverse inputs, the fused image enhances overall quality and retains essential details that might otherwise be lost when relying on a single exposure or setting. One method used to preserve critical features is a latent low-rank representation (LatLRR).

LatLRR was proposed to address the limitations of traditional low-rank representation (LRR), which struggles with insufficient or noisy observations [18]. To improve robustness, LatLRR incorporates both observed ($X_o$) and unobserved ($X_Z$) data into its dictionary formulation. This is achieved by minimizing the nuclear norm of the representation matrix (Z):

$$X_o=[X_o, X_Z] Z,$$

(4)

where $X_o$ represents the observed data matrix, $X_Z$ denotes the unobserved (hidden) data, and Z is the representation matrix that captures relationships among data points. By incorporating $X_Z$, LatLRR effectively accounts for hidden data effects, even when the hidden data are unavailable. To recover these hidden data effects, LatLRR formulates a convex optimization problem. For noiseless data, it minimizes the nuclear norms of both the representation matrix (Z) and a latent low-rank matrix (L):

$$X=XZ+LX,$$

(5)

where $X$ is the input data matrix, $XZ$ represents the principal features that describe the core data structure, and $LX$ corresponds to the latent low-rank features that capture additional details. When data are corrupted, an additional sparse noise term (E) is introduced [19], leading to the extended problem:

$$X=XZ+LX+E,$$

(6)

where $E$ accounts for sparse noise or outliers in the data, ensuring the model remains robust under corrupted conditions. This formulation enables LatLRR to effectively reconstruct hidden effects and provide reliable representations, even in challenging environments.

The integration of NIR and visible imaging offers a promising solution to further enhance image quality. By combining spectral translation with advanced blending techniques, this fusion approach surpasses traditional multi-exposure fusion methods, addressing the limitations of individual modalities and advancing the field of image enhancement.

3. Proposed Method

As illustrated in Figure 2, the algorithm progresses through the following stages: CycleGAN training, image blending for detail enhancement, image blending with gamma correction for global toning, CLAHE, and color compensation. The primary objective is to enhance object details in dark regions by leveraging NIR data. Initially, the algorithm addresses the challenge of lacking paired NIR and visible datasets. While paired training was considered, the absence of matching datasets led to the adoption of unpaired learning (as described in Section 2.1) [20]. This unpaired approach facilitates the creation of a synthetic matching dataset, which is then used to train the model and generate synthetic NIR data [21,22]. Both the visible and NIR images are transformed from the RGB channel to the Lab channel, with blending performed exclusively on the L channel. The first step in image blending for detail enhancement involves incorporating details from the NIR image while preserving the natural features of the visible (VIS) image. To mitigate noise introduced by the NIR image, a bilateral filter is applied, reducing noise while retaining edge details.

Next, image blending for detail enhancement applies global gamma correction to enhance the image’s overall brightness. This step helps reduce computational costs while producing a slightly brighter output. CLAHE is then applied to improve visibility and enhance finer details. This final step involves color compensation, as shown in the fourth step (Section 3.3). Here, the synthesized luminance channel is combined with the color-restored a’ and b’ channels. These components are then converted into RGB channels to produce the final output image:

▶

Extraction of hidden information: Sequential unpaired-to-paired transformations are conducted to generate a synthetic NIR image, revealing details hidden in dark environments;

▶

Enhanced visibility: Dual blending is utilized to synthesize the NIR and VIS images, improving overall visibility;

▶

Detail refinement: Gamma correction and CLAHE are applied to refine image details, enhance contrast, and improve brightness.

▶

Color compensation: The final step involves applying color compensation to produce a visually balanced and color-accurate image.

3.1. Unpaired and Paired Dataset Training Using CycleGAN

This paper presents a novel two-stage training strategy designed to address the limited availability of paired visible–NIR training data––a common bottleneck in low-light enhancement systems. As illustrated in Figure 3, the first stage involves unpaired training to generate fake visible images from NIR inputs [23]. The real visible image passes through a resnet generator (vis->NIR) to create a fake NIR image, which then passes through another generator (NIR->vis) to create a fake visible image. The loss is calculated after this process, ensuring that the generated images are as close to real images as possible. In this paper, the focus is on testing real NIR images to generate fake visible images. This necessity is evident in Figure 4. In Figure 4b, the generated fake NIR image demonstrates significant limitations, failing to preserve essential information characteristic of fake NIR domain images. Therefore, this necessitates improvements in the training process. To address these limitations, we proceed with pair training to enhance the results.

The generated fake visible images and real NIR images are then used for pair learning. This process also undergoes the same steps, leveraging the previously mentioned resnet generator. Initially, the fake visible image passes through a resnet generator (vis->NIR) to create a fake NIR image, which is then processed through another generator (NIR->vis) to create a fake visible image. Loss is calculated to ensure that the generated images closely resemble real ones. This process of loss calculation improves the image quality, resulting in refined fake visible images and real NIR images used in pair learning. As a result, the fake NIR image, illustrated in Figure 4c, demonstrates significant improvements, successfully preserving essential information characteristic of fake NIR domain images. This improvement underscores the effectiveness of the proposed domain translation approach. Combining CycleGAN’s ability to perform domain transformations without paired data with the accuracy of paired training, this method provides a robust, effective solution to this challenging task [24].

3.2. Visible–NIR Fusion

The luminance information is stored exclusively in the L channel, allowing it to be processed independently without directly affecting the original color information in the chromatic channels. As a result, the blending process is carried out solely on the L channel. Image blending for detail enhancement calculates the luminance difference between the NIR and visible images [25,26]. The initial blending operation is defined as follows:

$$L_{diff}={\displaystyle \frac{{(L}_{NIR}-L_{VIS})-min(L_{NIR}-L_{VIS})}{\mathit{max}\left(L_{NIR}-L_{VIS}\right)-min(L_{NIR}-L_{VIS})}},$$

(7)

$$L_{Blend1}=L_{NIR}\times \left(1-L_{diff}\right)+L_{VIS}\times L_{diff},$$

(8)

where $L_{VIS}$ and $L_{NIR}$ represent the luminance components of the visible and NIR images, respectively. The term $L_{diff}$ denotes as a difference metric, typically used to measure the relative contribution of the two luminance components. It ranges from 0 to 1, where higher values emphasize $L_{VIS}$, and lower values prioritize $L_{NIR}$. This combined image retains the details of the visible image while simultaneously benefiting from the noise reduction and contrast enhancement effects of the NIR image. Figure 5 illustrates how $L_{Blend1}$ adapts the contributions of $L_{VIS}$ and $L_{NIR}$ based on the difference metric $L_{diff}$. This ensures a smooth and adaptive blending of the luminance components. This equation combines information from both images to create a clearer final image.

However, this direct blending approach is susceptible to quality degradation due to inherent white noise in NIR images. Image blending (with gamma) for global toning applies a bilateral filter to the difference map to mitigate this limitation, producing $L_{switching}$, which effectively suppresses noise while preserving edge information [27]. The second blending operation is expressed as:

$$L_{Blend2}=L_{VIS}^{\gamma }\times \left(1-L_{switching}\right)+L_{Blend1}^{\gamma }\times L_{switching},$$

(9)

where $L_{VIS}$ represents the luminance component of the visible image. $L_{Blend1}$ is the blended luminance obtained from a prior step, and $L_{switching}$ is a weighting factor derived from a bilateral filter. $L_{switching}$ ranges from 0 to 1 and adjusts the relative contributions of $L_{VIS}$ and $L_{Blend1}$. $\gamma$ (=0.4) is a parameter used in gamma correction, which is a non-linear operation used to encode and decode luminance or tristimulus values in image processing. This approach ensures a smooth and adaptive transition between $L_{VIS}$ and $L_{Blend1}$ based on the bilateral filter’s weighting. $L_{Blend2}$ integrates spatial information to refine the final luminance blend. The blending equation $L_{Blend2}=L_{VIS}^{\gamma }\times \left(1-L_{switching}\right)+L_{Blend1}^{\gamma }\times L_{switching}$ highlights why this method is effective. The term $L_{switching}$ acts as a weighting factor that dynamically balances the contributions of $L_{VIS}$ and $L_{Blend1}$, ensuring that more weight is given to the less noisy and more detailed image. When $L_{switching}$ is close to 0, the contribution from $L_{VIS}$ is dominant, preserving the details from the visible image. Conversely, $L_{switching}$ is close to 1, and the contribution from $L_{Blend1}$ becomes dominant, leveraging the benefits of the noise-reduced NIR image. This refined blending strategy emphasizes details in regions with substantial differences and maintains the natural structure of the visible image in areas with smaller differences. This dual approach ensures that the final blended image retains high-quality details while minimizing noise, leading to a sharper and more visually appealing result. In addition to gamma correction, CLAHE (Contrast Limited Adaptive Histogram Equalization) is applied to improve local contrast and improve the visual quality of the image. The tile size used for CLAHE was (8, 8) and the clip limit was set to 3.0 [28]. This method helps to prevent excessive noise amplification while effectively improving the contrast in small, contextually defined regions of the image.

Figure 6a–d illustrate the process. Figure 6a shows the L-channel during twilight, representing the original input. Figure 6b illustrates the unpaired and paired training modules proposed in the method. Figure 6d depicts the enhanced image after applying the proposed fusion technique. Figure 6e shows the result of processing with CLAHE alone under the same conditions. Compared to Figure 6e, the image in Figure 6d displays the windows of the left building and railroad tracks more brightly and clearly. The combined application of global gamma correction and CLAHE forms a robust post-processing framework that substantially enhances the perceptual quality of the fused image. By strategically manipulating luminance distribution and local contrast, this method extracts and highlights critical visual information that might otherwise be difficult to discern in the original spectral domains.

3.3. Color Space Transformation and Compensation

After converting the image to the Lab color space, the color information from the original image is separated into the a and b channels [29], while the luminance information is contained on the $L$ channel. This separation enables independent processing of the luminance channel without directly affecting the original color information. However, during the fusion process of NIR and visible images, imbalances can arise between the modified luminance channel and the color channels ($a$ and $b$), potentially leading to unnatural color representations.

When $L^{\prime }$, $a$, and $b$ are converted back to the RGB color space, excessive color distortion, such as oversaturation, can occur. This is because chroma—representing color saturation—is defined as the distance of the ($a$, $b$) point in the color space, as shown in Equation (15). The non-linear relationship between chroma ($c$) and luminance ($L$) exacerbates this issue, as the rate of change in chroma is not proportional to changes in luminance. The color correction function, $C{C}_{gain}$, is determined experimentally through constants $\mathsf{\alpha }$ and $\mathsf{\beta }$, with values of $\alpha$ = 1.009 and $\beta$ = 0.705. Conventional color correction methods, which rely solely on the luminance variation ratio, often result in excessive color representation and significant distortion, thus compromising the natural appearance of the image. To address this issue and preserve color accuracy, we propose a color correction method that retains the original color characteristics while incorporating the enhanced luminance information from the fusion process. The color correction process is mathematically expressed as follows:

$$L_{gain}={\displaystyle \frac{L^{\prime }}{0.5\ast \left(L_{VIS}+L_{NIR}\right)}},$$

(10)

$$C{C}_{gain}=\alpha \times {L_{gain} }^{\beta },$$

(11)

$$c=\sqrt{a^2+b^2},$$

(12)

where $L^{\prime }$ is the refined luminance obtained from a previous blending step, and $L_{VIS}$ and $L_{NIR}$ are the luminance components of the RGB and NIR images, respectively. The term $L_{gain}$ calculates a gain factor that adjusts the blended luminance $L^{\prime }$ relative to the average of $L_{VIS}$ and $L_{NIR}$. The denominator ($0.5\ast \left(L_{VIS}+L_{NIR}\right))$ represents the average luminance value, which serves as a baseline for normalization. By dividing $L^{\prime }$ by this average, the gain factor highlights the relative enhancement or suppression of luminance in the final result.

Next, the $C{C}_{gain}$ term applies a color correction gain to $L_{gain}$, where $\alpha$ (=1.009) is a scaling factor and $\beta$ (=0.705) is an exponent that introduces a non-linear adjustment. This non-linear transformation ensures that the gain adapts perceptually, providing a more balanced color correction. The formula $C{C}_{gain}=\alpha \ast {L_{gain} }^{\beta }$ allows for flexible control over the color correction process.

$$a^{\prime }=C{C}_{gain}\times a,$$

(13)

$$b^{\prime }=C{C}_{gain}\times b,$$

(14)

where $a$ and $b$ correspond to the chromatic components of the image, and $a^{\prime }$ and $b^{\prime }$ are the adjusted chromatic components after the color correction process. The multiplication by $C{C}_{gain}$ ensures that the chroma is properly scaled in a way that corresponds to the luminance adjustments, preserving a balance between brightness and color saturation.

Finally, the luminance channel $L^{\prime }$ and color channels $a^{\prime }$ and $b^{\prime }$, processed through the color correction method, are combined to produce the final synthesized image. As shown in Figure 7b, the region marked by the red rectangle exhibits distortion, appearing lighter compared to the original image in Figure 7a. In contrast, Figure 7c, which includes the proposed correction, eliminates this distortion, maintaining the original color fidelity and ensuring a more natural representation.

4. Simulations

In this study, our model was trained using an unpaired dataset comprising 3300 pairs from the low-light datasets by Wei at al. [9], Jiang et al. [30], and publicly available data from the internet. Additionally, a paired dataset also comprising 3300 pairs, was used from the NIR datasets by Sa et al. [31]. The evaluation conducted a comprehensive performance evaluation of the proposed model using a range of image datasets, including established benchmark collections such as Loh et al. [32], and Meylan et al. [33,34]. To rigorously assess the effectiveness of this approach, we compared the results with several existing tone-mapping algorithms under different imaging conditions. The experimental setup involved testing various image types to address specific challenges: low-light scenarios to evaluate tone compression in both bright and dark regions, outdoor scenes to assess detail preservation, and color-rich environments to examine both color reproduction and tone compression simultaneously. Image processing was performed on images with dimensions of 1312 × 2000 pixels, resulting in a processing time of 2.29 s per image. The evaluation was carried out on a Windows 10 Education 64-bit system with an Intel(R) Core™ i7-9700K CPU @ 3.6 GHz, a TITAN RTX (NVIDIA, Santa Clara, CA, USA), and 96 GB of DDR4 memory, utilizing Python 3.8 and MATLAB R2020b for computational analysis. The training module was configured with a batch size of 1000 epochs and a learning rate of $2\times {10}^{-4}$. Through comparative analysis, we found that existing methods such as Reinhard and L1L0 tended to apply a uniform tone increase across image regions, resulting in a lack of dynamic detail. While methods like iCAM06, Kwon et al., Retinexnet, and Kim et al. maintained visibility in dark scenes, they introduced noticeable color distortions. These included unnatural saturation levels and an overall reddish color cast, especially in Retinexnet. In contrast, the proposed method outperformed these alternatives by effectively compressing tones in detailed regions. Specific elements, such as buses, transmission towers, cars, windows, and road surfaces, exhibited nuanced and accurate contrast representations. This allowed our approach to preserve and emphasize critical structural and textural information with remarkable fidelity, producing a balanced and natural visual output.

4.1. Comparative Experiments

To assess the performance of the proposed multi-exposure fusion method, we conducted comparative experiments against several widely used fusion techniques, including iCAM06, L1L0, Kwon, Reinhard, Retinexnet, and Kim et al. Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14 display the results produced by each method using images with varying brightness levels, demonstrating the effectiveness of the proposed method in different scenarios.

When the input image depicted a night scene, all methods except the proposed one exhibited noticeable noise in the sky. As shown in Figure 8 and Figure 9, the proposed method effectively produced a clear output for the night scene, reducing sky noise. However, despite this improvement, the proposed method introduced some white noise, which is a limitation of the synthetic NIR generation process. This effect is linked to the combination of unpaired and paired training in the proposed modules, which may cause inconsistencies during image synthesis.

Figure 10 and Figure 11 present an image captured during twilight. It is clear that iCAM06, L1L0, Kwon, Reinhard, and Lee et al. methods suffer from over-saturation, particularly in areas near the sun. Additionally, Retinexnet, shown in Figure 10 and Figure 11, exhibits a noticeable reddish bias, causing significant color distortion compared to the original input. In contrast, the proposed method avoids color distortion while maintaining high contrast. This allows for clear visibility of details, even in the darker areas of the image, revealing faint objects typically obscured in low-light conditions. The proposed method thus demonstrates superior performance in challenging low-light scenarios.

Figure 12 illustrates an indoor scene with an outdoor view, highlighting key differences among the evaluated methods. Traditional approaches such as iCAM06, L1L0, Kwon et al., and Reinhard show subtle noise artifacts in their outputs. While Retinexnet achieves high visibility through enhanced sharpness, it suffers from substantial global color distortion. Kim et al.’s method encounters over-saturation issues, resulting in an unnatural background. In contrast, the proposed method maintains natural saturation levels while preserving object clarity and color accuracy. This balanced approach allows for clear visualization of both indoor and outdoor features, such as trees and curtains, demonstrating the method’s robustness in handling scenes with complex lighting conditions.

Figure 13 and Figure 14 depict a bright outdoor scene, emphasizing several notable differences among the methods. Traditional approaches like iCAM06, L1L0, Kwon et al., and Reinhard exhibit over-saturation, which diminishes the natural appearance of the images. Although Retinexnet produces sharp images, it introduces a pronounced reddish bias, particularly affecting the natural tones of faces, as seen in Figure 13. Kim et al.’s method maintains good color fidelity but struggles with accurate color reproduction of objects like trucks, as seen in Figure 13. In contrast, the proposed method outperforms these techniques in daylight conditions by preserving natural color fidelity and avoiding color distortion, as demonstrated in Figure 13 and Figure 14.

4.2. Quantitative Evaluations

The performance of the proposed method was quantitatively evaluated using advanced image quality metrics. The blind/referenceless image spatial quality evaluator (BRISQUE) assesses image quality by modeling statistical features of locally normalized luminance coefficients, capturing natural scene statistics to quantify visual distortions [35]. The spatial and spectral entropy-based quality (SSEQ) metric evaluates image quality by analyzing spatial and spectral entropy features, effectively quantifying both spatial distortions and spectral inconsistencies, making it well suited for assessing images under diverse conditions [36]. The spatial-spectral sharpness measure (S3) evaluates image sharpness by combining spatial and spectral characteristics, utilizing local spectral slope and total variation to produce a comprehensive sharpness index [37].

The localized perceptual contrast-sharpness index (LPC-SI) analyzes how well contrast and structural information are preserved, demonstrating effectiveness in maintaining fine image details and structural integrity [38]. The natural image quality evaluator (NIQE) provides a no-reference perceptual quality assessment by comparing image characteristics against a statistical model of natural scene statistics, offering an objective measure of image quality without requiring a reference image [39].

The multimedia assessment and inference of quality with attention (MANIQA) leverages advanced attention mechanisms to assess image quality, incorporating transformer-based architectures to capture complex perceptual features and provide nuanced quality evaluation [40]. The convolutional neural network-based image quality assessment (CNNIQA) employs deep learning techniques to extract hierarchical features, utilizing convolutional neural networks to generate comprehensive quality scores that reflect perceptual image characteristics [41].

Figure 15 presents the results of the evaluation based on the scores for 25 comparison images and their averages. Figure 16 illustrates the images used for score metrics, showcasing the diversity of inputs evaluated and highlighting the consistent performance of the method across various scenarios.

The proposed method consistently ranks highly across a variety of quality assessment metrics, demonstrating exceptional stability and minimal performance variation. In contrast to existing synthesis methods, which often exhibit considerable discrepancies in evaluation results, our approach provides robust and uniform performance across different quality assessment frameworks.

Notably, the method excels in the BRISQUE assessment, which quantifies visual distortions by modeling natural scene statistics. As shown in

Table 1. Comparison of metric scores. (↑) Higher scores are preferable, and (↓) lower scores are preferable. Bold font highlights the best results in each corresponding metric. Superscript numbers indicate the ranking among comparison models (superscript ^1,2 represent score ranks). Bold font highlights the best result.

	iCAM06	L1L0	Kwon et al. [6]	Reinhard (2012) [8]	Retinexnet	Kim et al. [10]	Proposed
BRISQUE (↓)	25.272	26.424	22.602	25.946	20.895	22.098	20.473 1
SSEQ (↓)	21.316	23.052	16.311	23.842	21.995	18.165	16.475 2
S3 (↑)	0.160	0.139	0.232	0.173	0.178	0.225	0.245 1
LPC_SI (↑)	0.902	0.888	0.925	0.917	0.919	0.921	0.948 1
NIQE (↓)	3.427	3.325	4.530	3.289	4.309	3.319	3.001 1
MANIQA (↑)	0.2246	0.2361	0.2392	0.2353	0.2851	0.2288	0.3043 1
CNNIQA (↓)	20.311	20.920	30.122	24.135	19.254	22.006	18.802 1

, the proposed method achieved a BRISQUE score of 20.473, ranking first, and outperforming the second-best score of 20.895 by approximately 2.02%. This highlights the method’s ability to produce visually appealing results with minimal distortion.

The performance on the SSEQ metric, where lower values indicate better quality, is also remarkable. The proposed method scored 16.475, trailing the leading score of 16.311 by just 1.01%, reflecting its strong competitive performance in preserving structural quality.

In the S3 metric, the proposed method scored 0.245, showing an improvement of approximately 5.60% over the second-best score of 0.232. This demonstrates the method’s effectiveness in enhancing overall structural similarity and maintaining high-quality synthesis.

For LPC_SI, the proposed method achieved a score of 0.948, surpassing the second-best score of 0.925 by approximately 2.49%. This indicates its exceptional performance in detailed feature representation and perceptual quality.

In the NIQE metric, the proposed method scored 3.001, outperforming the second-best score of 3.289 by approximately 8.75%. This validates the method’s ability to maintain high perceptual quality without relying on reference images.

In the MANIQA assessment, the proposed method achieved a superior score of 0.3043, compared to 0.2851 for the second-best method, demonstrating an improvement of approximately 6.74%. This showcases the method’s effectiveness in attention-based image quality evaluation.

Finally, in the CNNIQA metric, which employs deep learning for evaluation, the proposed method scored 18.802, surpassing the second-best score of 19.254 by approximately 2.35%. This reinforces the method’s superior image synthesis capabilities.

Overall, the proposed method stands out in image quality performance, particularly in key areas such as detail representation, noise reduction, and tone-compression effects. The consistently high rankings across various sophisticated image quality assessment metrics highlight the method’s robustness and effectiveness in advanced image synthesis. The results in Table 1 further underscore the method’s ability to achieve competitive or superior performance across diverse evaluation frameworks, showcasing its versatility and reliability in different contexts.

Finally, the processing time was calculated for the transformation into clean images based on the 1312 × 2000 resolution criterion. Comparing results with other existing methods, as shown in

Table 2. Runtime performance: lower times are preferable. Superscript numbers indicate the ranking among comparison models (superscript ^1,2 represent score ranks). Bold font highlights the best result.

	iCAM06	L1L0	Kwon et al. [6]	Reinhard (2012) [8]	Retinexnet	Kim et al. [10]	Proposed
Process time	5.39 s	5.57 s	22.54 s	1.10 s 1	6.40 s	26.78 s	2.29 s 2

, the proposed method exhibited the fastest processing speed, outperforming all other techniques except for Reinhard (2012). This demonstrates the efficiency of the proposed method in terms of computational performance, while maintaining image quality.

The proposed method shows effective image representation performance for high contrast and dim surrounding conditions. However, it faces limitations in real-time image transformation due to multiple computational steps with the deep learning module and method-based blending processing. Therefore, embedding the proposed algorithm into boards such as specialized ASIC boards is necessary for practical application. Future research should also focus on real-time operations integrated into video surveillance systems.

5. Conclusions

This study presents a novel approach to enhancing image quality by combining image-to-image translation between the visible and NIR domains with advanced post-processing techniques. To address the challenge of limited paired visible–NIR training data, we employed an unpaired CycleGAN architecture to generate synthetic visible images. These synthetic images were then integrated into a paired training process with real NIR images, allowing for a robust and effective fusion of the two domains.

Our method consists of several key stages. It begins with initial image synthesis using CycleGAN, followed by the extraction of luminance channel information. Next, detail preservation techniques such as gamma correction and CLAHE are applied to enhance local contrast and structural details. Finally, a color compensation step is performed to ensure accurate RGB reconstruction, preserving the vibrancy and fidelity of the original color information.

The experimental results highlight the advantages of our approach over traditional tone-mapping algorithms. First, the proposed method achieves superior tone compression while enhancing local contrast, resulting in images with exceptional clarity. Second, unlike conventional methods that often produce muted or distorted colors, the color compensation technique ensures vibrant and accurate color reproduction. By combining the strengths of deep learning-based image translation with traditional image processing techniques, this hybrid approach effectively addresses the limitations of visible–NIR fusion. The proposed method successfully preserves structural detail and color fidelity, offering a promising solution for advanced image enhancement applications.