An Unsupervised Fundus Image Enhancement Method with Multi-Scale Transformer and Unreferenced Loss

Abstract

Color fundus images are now widely used in computer-aided analysis systems for ophthalmic diseases. However, fundus imaging can be affected by human, environmental, and equipment factors, which may result in low-quality images. Such quality fundus images will interfere with computer-aided diagnosis. Existing methods for enhancing low-quality fundus images focus more on the overall visualization of the image rather than capturing pathological and structural features at the finer scales of the fundus image sufficiently. In this paper, we design an unsupervised method that integrates a multi-scale feature fusion transformer and an unreferenced loss function. Due to the loss of microscale features caused by unpaired training, we construct the Global Feature Extraction Module (GFEM), a combination of convolution blocks and residual Swin Transformer modules, to achieve the extraction of feature information at different levels while reducing computational costs. To improve the blurring of image details caused by deep unsupervised networks, we define unreferenced loss functions that improve the model’s ability to suppress edge sharpness degradation. In addition, uneven light distribution can also affect image quality, so we use an a priori luminance-based attention mechanism to improve low-quality image illumination unevenness. On the public dataset, we achieve an improvement of 0.88 dB in PSNR and 0.024 in SSIM compared to the state-of-the-art methods. Experiment results show that our method outperforms other deep learning methods in terms of vascular continuity and preservation of fine pathological features. Such a framework may have potential medical applications.

1. Introduction

With the development of computer-aided diagnosis, color fundus images are used to diagnose and monitor early ophthalmic disease. However, the quality of images is interfered with by a variety of categories and varying degrees of degradation. Some major contributing factors, such as uneven illumination, low contrast, artifacts, and blurred fundus images, limit the analysis and diagnosis of fundus images. These low-quality images can also reduce the accuracy of downstream image processing tasks such as disease classification [1,2,3,4,5] and retinal vessel segmentation [6,7,8,9]. Therefore, the influence of degradation factors on the image must be eliminated before computer-aided analysis can be conducted for diagnosis.

Traditional image enhancement methods include histogram equalization, spatial domain filtering, and greyscale transformations. However, these methods have a limitation in generalizing and even sometimes produce non-existent artifacts when processing low-quality color fundus images, affecting the diagnostic accuracy and preventing a direct application to fundus images. With the development of deep learning, many learning-based methods have achieved good results in image enhancement.

In the field of natural images, for example, Wei et al. [10] present the method Retinex-Net, based on the Retinex theory, to enhance low-illumination images. Wang et al. [11] advocate for GLAD-Net, which uses an encoder–decoder network to extract prior global illumination information from low-illumination images, thereby guiding the process of low-illumination enhancement. In the realm of medical images, Li et al. [12] introduce NuI-Go, an enhancement network designed to remove light unevenness from color fundus images by utilizing a non-local semantic module. Lee et al. [13] propose a deep learning-centric model engineered to automatically augment low-quality retinal fundus images, with the aim of amplifying image quality and neutralizing multifaceted image degradation. However, large numbers of medical fundus images are hard to obtain, and it is impractical to obtain pairs of high- and low-quality fundus images of the same patient. Thus, these supervised learning-based methods do not apply to fundus image enhancement.

To avoid the dependence on image pairs, Guo et al. [14] propose Zero-DCE, an unsupervised learning-based low-illumination enhancement method with a series of unreferenced loss functions. Cheng et al. [15] introduce EPC-GAN, a method that includes a contrast loss function and a fundus prior loss function based on a pre-training task for classifying diabetic retinopathy classes. This approach strives to prevent alteration of important information in the images and over-enhancement problems. To ensure the preservation of structural features in the fundus while enhancing the image, Ma et al. [16] present the Still-GAN method. This approach incorporates a luminance constraint loss and a structure-preserving loss function into CycleGAN, in order to maintain the structural information in the image. Both of these methods prove effective in enhancing the dark areas of the fundus. However, they can cause some blurring or semantic disruption of pathological areas due to the lack of treatment of pathological features.

For the field of color fundus images, where high- and low-quality image pairs cannot be acquired, generative adversarial networks are more suitable for achieving color fundus image enhancement tasks. Inspired by EnlightenGAN [17], we propose an unsupervised learning-based enhancement method for color fundus images. Existing GAN networks face difficulties in capturing structural and pathological features such as retinal blood vessels and lesion characteristics. Furthermore, fine-scale features are susceptible to being blurred or erased by deeper network models. Consequently, we incorporate the Swin Transformer into the generator structure, aiming to amplify the network’s capacity for learning structured features. The main contributions of this paper are as follows:

• Based on the Swin Transformer, we propose a global feature extraction module. The module first extracts shallow features using convolution layers and then extracts deep features using multiple RSTBs (Residual Swin Transformer Blocks). We embed the module in a bottleneck layer to learn the correlation between local and global information in the deep feature space. We also enhance the ability of the model to learn global features at the spatial level.
• We devise a luminance attention mechanism, based on a priori knowledge. This mechanism, implemented through a luminance map, is integrated within the original same-layer skip connections of the U-Net and merged with the feature map. As a result, the attention mechanism imposes constraints that contribute to a more balanced brightness in the enhanced image.
• We improve the discriminator used for the natural image enhancement task by designing a non-reference color fundus image quality loss function based on this network to make it more suitable for color fundus image enhancement. We introduce an illumination loss function and a structure-preserving loss function. These functions allow the network to preserve vascular and pathological features during enhancement, reducing fictitious features and feature loss.

2. Related Work

2.1. Traditional Approaches

Image enhancement has been a long-standing research topic in computer vision, and there have been some classical approaches, such as Retinex [18] and multi-scale Retinex models [19]. Ref. [20] improves the ability to enhance low-light-level images based on the Retinex model by adding noise mapping to the model. Guo et al. [21] propose a method that estimates the illuminance of each pixel by finding the maximum value of each pixel RGB channel species and, in turn, constructs a structure before constructing the illuminance map. These methods are equally applicable to color fundus image enhancement. For example, ref. [22] presents a method using contrast-limited adaptive histogram equalization to recover images. P. Dail et al. [23] introduce a method to enhance fundus images by fusing background information with the original retinal image. While these methods are effective, they rely on global image statistics and mapping functions that may produce non-existent artifacts and distortion.

2.2. Deep Learning Approaches

Deep learning techniques, while well-established and effective in many areas of computer vision, face unique challenges when applied to the enhancement of fundus images. For instance, Eilertsen et al. [24] apply a pixel-level loss function constraint model to solve high-dynamic-range tasks, while Dong et al. [25] employ convolutional networks for super-resolution reconstruction. Despite these efforts, the subtle pathological features and unique structure of the retina make it difficult for general convolution-based neural networks to enhance fundus images effectively. Nul-Go Net, a deep learning-based method for retinal image enhancement, strives to address these challenges by progressively removing inhomogeneous illumination through recursive residual learning. However, this method may generate artificial features not present in the original image, potentially misleading clinical diagnosis.

2.3. Generating Adversarial Networks

The main objective of a generative adversarial network remains to create images that closely resemble the original ones. Goodfellow [26] first introduces GAN, a network composed of two parts: a generator and a discriminator. It optimizes a specific loss function through adversarial learning. Mirza et al. [27] put forward a conditional GAN that can produce new image classes by adding extra label constraints to the input. In contrast, Zhu et al. [28] present a cycle consistency loss function and its network model CycleGAN. This innovation breaks the limitation of needing paired datasets for training, facilitating the transformation of images between two domains. You et al. [29] propose the Cycle-CBAM method for color fundus image enhancement, which applies Cycle-Consistency Loss to regulate the conversion between high/low-quality images. Although this approach enhances the subjective quality of the image to some extent, it may lead to the generation of non-existent vascular structures.

3. Method

As shown in Figure 1, we take U-Net as the generator and use global discriminators and local discriminators to guide the network model. In this section, we first introduce two important modules, the global feature extraction module and the luminance attention module. Then for details, we introduce the discriminator and the loss function.

3.1. Global Feature Extraction Module

Raj et al. [30] argue that the bottleneck layer at the bottom of U-Net assists in extracting non-local image feature information. Based on this, they add a global feature extraction module consisting of a Residual Dense Block (RDB) to the bottleneck layer of U-Net. Drawing inspiration from SwinIR, we introduce the Global Feature Extraction Module (GFEM), as shown in Figure 1. GFEM extracts deep-level feature data and learns the correlation between local and global information, thereby further enhancing the vascular continuity of the enhanced images. The procedure is as follows:

$$Fea=M_{GFEM}\left(x\right)$$

(1)

where $Fea$ denotes the deep feature data extracted, $M_{GFEM}\left(\right)$ denotes the global feature extraction module, and x denotes the input information. GFEM contains convolution layers and residual Swin Transformer blocks (RSTBs). The process of extracting intermediate feature information by RSTB can be expressed as follows:

$$M_i=M_{RSTB}\left(M_{i-1}\right),i=1,2,\dots ,N_{RSTB}$$

(2)

$$D=Conv\left(M_{N_{RSTB}}\right)$$

(3)

where $M_{RSTB}\left(\right)$ denotes the RSTB module, and Mt denotes the i-th intermediate feature data. D denotes the final extracted deep feature data, and $Conv\left(\right)$ denotes the convolution layer. The feature information obtained through this module provides the basis for the subsequent fusion of feature information.

3.2. Swin Transformer Layer

Swin Transformer Layer (STL) [31] finds its genesis in the standard multi-head self-attention of the original Transformer layer [32], distinguished by the incorporation of localized attention and shifted window schemes. The STL is composed of Multi-Head Attention (MSA) and a Multilayer Perceptron (MLP). For a given input $x=H\times W\times C$, STL uses a sliding window mechanism to convert the input to $\frac{HW}{M^2}\times M^2\times C$ and divides $M\times M$ non-overlapping local windows. The mathematical formula for MSA is as follows:

$$Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d_k}}+B)V$$

(4)

where B is the position code and $d_k$ is the size of the last dimension of K. Q, K, and V are calculated as follows:

$$Q=X{W}_Q,K=X{W}_K,V=X{W}_V$$

(5)

The multilayer perceptron has two fully connected layers with feature transformations between the layers using GELU. Both the MSA and the MLP use a residual connection whose mathematical expression is

$$X=MSA\left(LN\right(X\left)\right)+X$$

(6)

$$X=MLP\left(LN\right(X\left)\right)+X$$

(7)

where $LN\left(\right)$ denotes the LayerNorm layer.

3.3. Luminance Attention

First, we use image conversion to obtain luminance images of low-quality fundus pictures. The formula is as follows:

$$M_{GrayScale}=0.299\times x_R+0.587\times x_G+0.144\times x_B$$

(8)

where $x_R$, $x_G$, $x_B$ denote the regularized values of the three low-quality color fundus image channels.

Unlike natural images, according to the principles of color fundus photography, useful information is only retained in the circular area. The black part of the background contains no useful information and introduces some invisible noise. Inspired by the Dark Channel Prior [33], we omit the information in the black area of the background. The formula for its add-mask operation is as follows:

$$M_{IA}=(1-M_{GrayScale})\otimes M_{mask}$$

(9)

where $M_{IA}$ denotes the luminance map, $M_{mask}$ is the background mask map for x in the map, and ⊗ denotes element multiplication. The brighter the original image, the less the value of $M_{IA}$ corresponds to its pixels.

Next, we stitch the original image x with $M_{IA}$ and feed the result into the enhancement network. The generators of the augmented network are symmetric U-Net structures. The left-hand side of the generator consists of a convolutional kernel size of $3\times 3$ and a maximum pooling layer with a kernel size of $2\times 2$. The ith convolution layer from left to right is denoted as $Con{v}_{left}^i$, and the pooling layer is denoted as $Dow{n}_{left}^i$. The right-hand side of the generator consists of a kernel size of $2\times 2$ upsampling operators and a convolution layer with a convolution kernel size of $3\times 3$. The ith upsampling operator from right to left is denoted as $U{p}_{right}^i$, and the convolution layer is denoted as $Con{v}_{right}^i$. The output of its upsampling operator is noted as $Ou{t}_{up}^i$. The result obtained by $M_{IA}$ after the ith maximum pooling layer is noted as $M_{IA}^i$. Thus, the formula for luminance attention can be expressed as follows:

$$Ou{t}_{Attn}^i=Ou{t}_{down}^i\otimes M_{IA}$$

(10)

where $Ou{t}_{Attn}^i$ denotes the output of luminance attention, and ⊗ denotes element-by-element multiplication. Finally, we splice $Ou{t}_{Attn}^i$ and $Ou{t}_{up}^i$ and use them as input to $Con{v}_{right}^i$.

3.4. Dual Discriminator and Generator

Natural image enhancement algorithms, while enhancing the image, may alter some fine details but generally do not affect the overall image quality. However, fundus images, unlike natural images, contain significant structural features and fine-scale vascular and pathological details. For enhanced color fundus images, we aim to retain these original detailed features without compromising the fine-scale attributes. To accomplish this, we employ global and local discriminators to constrain the network [17]. The global discriminator assesses the realism of the enhanced color fundus image relative to the true fundus image. The local discriminator, on the other hand, randomly crops five image blocks from the fundus image as input. This strategy helps to avoid over- or under-enhancement in local areas. The expression for our global discriminator’s loss function is

$$L_D^{Global}=E_{y\sim Y}\left[{(D_{Ra}(y,y{}^{\prime })-1)}^2\right]+E_{y{}^{\prime }\sim Y{}^{\prime }}\left[{\left(D_{Ra}(y{}^{\prime },y)\right)}^2\right]$$

(11)

RealisticGAN [34] proposes the use of a discriminator to estimate the probability of real data being more authentic than randomly generated fake data. Inspired by RealisticGAN, we improve its loss function to make it more suitable for color fundus image enhancement tasks. The expression is

$$D_{Ra}(y,y{}^{\prime })=C\left(y\right)-E_{y{}^{\prime }\sim Y{}^{\prime }}\left[C\left(y{}^{\prime }\right)\right]D_{Ra}(y{}^{\prime },y)=C\left(y{}^{\prime }\right))-E_{y\sim Y}\left[C\left(y\right)\right]$$

(12)

where y denotes the true color fundus image and $y^{\prime }$ denotes the enhanced image. C denotes the global discriminator. We adopt LSGAN [35] as the loss function of the local discriminator. Its expression is as follows:

$$L_D^{Local}=E_{y_p\sim Y_{patches}}\left[{(D\left(y_p\right)-1)}^2\right]+E_{y_p{}^{\prime }\sim Y_{patches}}\left[{(D\left(y_p{}^{\prime }\right)-0)}^2\right]$$

(13)

where $y_p$ and $y_p{}^{\prime }$ denote local blocks of size $32\times 32$ cut at random from y and $y{}^{\prime }$.

Our goal is to enhance the low-quality color fundus image domain X into a high-quality color fundus image domain Y′. The generator constraint function expression is

$$L_D^{Global}=E_{y{}^{\prime }\sim Y{}^{\prime }}\left[{(D_{Ra}(y{}^{\prime },y)-1)}^2\right]+E_{y\sim Y}\left[{\left(D_{Ra}(y,y{}^{\prime })\right)}^2\right]$$

(14)

$$L_D^{Local}=E_{y_p{}^{\prime }\sim Y_{patches}{}^{\prime }}\left[{(D\left(y\right)-1)}^2\right]$$

(15)

3.5. Loss Function

3.5.1. Self Feature Preserving Loss

Ma et al. [16] show that non-uniformity of brightness affects the quality of fundus images. Inspired by EnlightenGAN [17], we use the Self Feature Preserving Loss. The function ensures that the image features before and after enhancement are preserved to themselves. The formulation of the function is as follows:

$$L_{SFP}\left(x\right)=\frac{1}{W_{i,j}{H}_{i,j}}\sum _{x=1}^{W_{i,j}}\sum _{x=1}^{H_{i,j}}{({\varphi }_{i,j}\left(x\right)-{\varphi }_{i,j}\left(G\left(x\right)\right))}^2$$

(16)

where x is the input low-quality fundus image and $G\left(x\right)$ denotes the generator. ${\varphi }_{i,j}\left(x\right)$ denotes the feature map extracted by pre-training, and i, j denotes j-th convolution layer after the i-th maximum pooling layer from the training model. $W_{i,j}$ $H_{i,j}$ represent the width and height of the feature map, respectively. This study defines the value of i as 5 and j as 1.

3.5.2. Illumination Loss

Even when using the same device, variations in color fundus images can occur due to differences among operators, potentially leading to light leakage or insufficient brightness and ultimately reducing the overall quality of the fundus image. To mitigate this issue, we employ an illumination constraint function, drawing inspiration from StillGAN [16]. This function initiates by calculating the global average luminance of the image I, followed by partitioning the image into non-overlapping patches of equal size. We then compute the average luminance of each patch to derive the illumination matrix D. Ultimately, the difference in average brightness between I and D is calculated, and the function is minimized to constrain the overall brightness uniformity of the image. The formulation is as follows:

$$L_{ill}\left(x\right)=E_{x\in X}[E_{global}\left[\right|upsampling{E}_{local}^{p\times p}[G\left(x\right)-E_{global}\left[G\left(x\right)\right]]\left|\right]]$$

(17)

where $x\in X$ denotes the input low-quality fundus image, $G\left(x\right)$ is the generator, and p denotes the non-overlapping region size. $E_{local}$ denotes the calculation of the local brightness average of the image, and $E_{global}$ denotes the global brightness average of the image. $upsampling$ is represented as bilinear interpolation upsampling.

3.5.3. Structure Loss

While the illumination loss function can improve the image brightness problem, it can also result in the loss of feature information in the enhanced image. We use a structural loss function to ensure that the enhanced image retains the structural features in the original image. Using similarity between original and enhanced images, the mean $\overline{\tau }$ is calculated for each identical region of the image before and after enhancement. This function is finally obtained by minimizing $1-\overline{\tau }$, which is defined as follows:

$$L_{ST}(x,y{}^{\prime })=E_{x\in X,y{}^{\prime }\in Y{}^{\prime }}[1-\frac{1}{M}\sum _{i=1}^M\frac{{\delta }_{x_i,y_i{}^{\prime }+c}}{{\mu }_{x_i}{\mu }_{y{}^{\prime }}+c}]$$

(18)

where $x\in X$ represents the input low-quality fundus image and $y{}^{\prime }\in Y{}^{\prime }$ is the high-quality fundus image after enhancement by generator G. ${\delta }_{x_i,y_i{}^{\prime }}$ denotes the covariance between $x_i$ and $y_i{}^{\prime }$. ${\mu }_{x_i}$ and ${\mu }_{Y_i}$ are the standard deviations of $x_i$ and $y_i{}^{\prime }$, respectively. c is a constant used to avoid fluctuations in values.

Thus, the overall function of the generator of the proposed method for color medical fundus image enhancement in this paper can be defined as

$$L=L_{SFP}^{Global}+L_{SFP}^{Local}+L_G^{Global}+L_G^{Global}+{\lambda }_{ill}{L}_{ill}+{\lambda }_{ST}{L}_{ST}$$

(19)

where ${\lambda }_{ill}$ and ${\lambda }_{ST}$ are the weight coefficients of $L_{ill}$ and $L_{ST}$, respectively, which are set to ${\lambda }_{ill}=5$ and ${\lambda }_{ST}=50$ in this paper.

4. Experiment and Results

4.1. Datasets

In order to better train our model, we chose the dataset EyeQ [36], a subset obtained by re-labeling from EyePACS. It is a robust dataset that provides ample scope for testing and validation, contributing to a more accurate and reliable analysis. EyeQ has a total of 28,792 color fundus images with three quality levels: ‘Good’, ‘Usable’, and ‘Reject’ The pathological features of the images containing the “Good” label are visible; images containing the “Usable” label have a slight degradation factor, but the subject structure and pathological features in the image can be identified; images labeled "Reject" are considered to have a significant degradation factor and cannot be used for diagnostic tasks. In addition, images of subject structures such as the optic disc and the retina’s macular region that are not visible are also labeled as ‘Reject’. EyeQ contains 12,543 images from the training set and 12,649 images from the test set. All images are pre-processed using cofe-Net [37]. The images marked as “Good” in the test set are degraded to the corresponding low-quality images containing three degradation factors: illumination, artifacts, and blur. In particular, we use images labeled as ‘good’ and ‘usable’ to train and test the network model, as images labeled as ‘rejected’ lack a significant amount of subject structure and pathological features.

4.2. Implementation

The selection of model parameters during the training phase is predominantly determined by experimentation and optimization. The choice of input image size balances computational efficiency and the level of detail captured. As such, we set the input image size to 384 × 384 and the batch size per iteration to 8, as shown by [17], for 100 epochs. The input image block size for the local discriminator is set to 64 × 64. We opt for Adam as the optimizer with a learning rate set at 0.0001. For better visual results, we use a 512 × 512 images as input during the testing phase. The model is built on PyTorch and trained using NVIDIA GeForce RTX3090, each card with 24 GB of memory. Processing time for each image of size 512 × 512 × 3 is approximately 0.061 s.

4.3. Ablation Study

In this paper, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure) [38] are used as quantitative analysis evaluation indicators. The definition of PSNR is as follows:

$$PSNR=10\times lo{g}_{10}\left(\frac{{(2^b-1)}^2}{MSE}\right)$$

(20)

$$MSE=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{i=0}^{n-1}{[I(i,j)-K(i,j)]}^2$$

(21)

where b represents the bit depth of the image, and m and n represent the width and height of the image, respectively. $I(i,j)$ and $K(i,j)$ denote the pixel values at the coordinates of the reference image and the enhanced image. The mathematical definition of SSIM is as follows:

$$SSIM=\frac{\left(2{\mu }_x{\mu }_y\right)(2{\delta }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\delta }_x^2+{\delta }_y^2+c_2)}$$

(22)

where x and y represent the reference image and the enhanced image, and ${\mu }_x$ and ${\mu }_y$ represent the mean values. ${\delta }_x^2$ and ${\delta }_Y^2$ represent the variance, and Oxy represents the covariance of x and y. We use 8 bit RBG images in this study, so $c_1=(0.01\times {(2^8-1)}^2)$.

For the ablation experiments, we train models with different modules on EyeQ. A high SSIM score indicates a more detailed image. As shown in

Table 1. Analysis of the impact of different components on the model.

Module	No GFEM	No L-Attention	GFEM + L-Attention
PSNR	23.47	23.58	24.32
SSIM	0.8870	0.8929	0.8932

, the GFEM + L-attention module achieves a PSNR value of 24.32, representing an improvement of 0.85 over the No GFEM module and 0.74 over the No L-attention module. Furthermore, the GFEM + L-attention module exhibits an SSIM value of 0.8932, which is 0.0062 higher than the No GFEM module and 0.0003 higher than the No L-attention module. Although the margin with the No L-attention module is minimal, it nonetheless underscores the superior performance of the GFEM + L-attention module in retaining structural similarity of the image. Figure 2 shows the impact of the different components on the model. In Sam 1, the absence of luminance attention causes the image to be partially over-lit. Sam 2 shows the impact of GFEM. When there is no GFEM, noise in the dark areas is incorrectly identified and enhanced, resulting in false features. It results in the appearance of details of blood vessels that are not present in the original image.

4.4. Comparison with State of the Art

In this section, we compare the performance of the proposed model with the current state-of-the-art methods.

4.4.1. Uneven Illumination Enhancement Comparison

We compare with other GAN-based image enhancement methods, and the comparison results are shown in Figure 3. The first column is the original color fundus image. The second column is pix2pix [39] trained on the dataset based on the synthetic high- and low-quality images. The third to fifth columns are CycleGAN, CutGAN [40], and StillGAN, trained on the dataset using real unpaired high- and low-quality color fundus images. The last column shows the results produced by our method.

As shown in Figure 3, from each column, the images enhanced by the pix2pix method show significant uneven enhancement problems, with the worst performance in terms of brightness uniformity. It may be due to a domain gap problem when the results of its training on a synthetic image dataset are migrated to a real low-quality fundus image enhancement task. CycleGAN achieves good results in terms of overall image quality but performs poorly in terms of color stability. CutGAN performs best in overall image perception, almost eliminating overexposure or underexposure. However, it tends to generate false features in areas of severe degradation that are overexposed or underexposed, which may interfere with diagnosis. StillGAN has some enhancement, but the contrast between the vessels and the background could be more significantly enhanced. The images enhanced by this method have better color consistency and overall brightness uniformity both in single images and in longitudinal contrast and provide significant contrast enhancement of the fundus vessels.

As shown in

Table 2. Average PSNR and SSIM results on test set.

Method	PSNR (dB)	SSIM
pix2pix [39]	23.2	0.8946
CycleGAN [28]	22.84	0.8430
CutGAN [40]	21.89	0.8534
StillGAN [16]	23.44	0.8694
Ours	24.32	0.8932

, our method achieves superior results, reaching a PSNR index of 24.32 dB. This is an improvement of 0.88 dB compared to the second-best method, StillGAN, and a significant 2.43 dB better than the lowest-scoring method, CutGAN. For the SSIM index, our method scores 0.8932, slightly lower than the highest-scoring method, Pix2pix (by 0.0014), but surpasses the lowest-scoring method, CycleGAN (by 0.0502). However, Pix2pix shows the worst performance in terms of brightness uniformity. This might be a domain gap issue when its training results on synthesized image datasets are transferred to the task of enhancing real low-quality color fundus images. Compared to Pix2pix, which uses synthesized image training datasets, our method might be more suited for real clinical images.

4.4.2. Structural Analysis

Color fundus images are diagnosed based on anatomical and pathological features in the images. Therefore, fundus image enhancement needs to maintain image viewing, structural features, and pathological features. To verify the retention of structural features after enhancement, we use Iter-Net [41] as the retinal vessel segmentation method. We segment the retinal vascular structure after enhancement by different methods. The degree of visibility and continuity of blood vessels in the segmentation results are analyzed to determine the effect of image enhancement. The comparison results are shown in Figure 4. The enhanced fundus image from our method can segment more vascular detail than the original image. CycleGAN and CutGAN have poorer vascular continuity and show vascular details that were not present in the original image. In the dark areas of the image, the low contrast of the background color of the blood vessels results in poor image visibility. The application of our method manifests a notable enhancement in the depiction of vascular details. However, the vascular structures enhanced by Pix2pix, CycleGAN, and CutGAN appear relatively disordered, while StillGAN and our proposed method demonstrate superior performance in terms of vascular detail and continuity.

4.4.3. Pathological Feature Analysis

Figure 5 shows the effect of image enhancement on the retention of pathological features. After enhancement by our method, the edges of pathological features in the dark areas remain clear and are distinguished from the background color. The pathological features enhanced by CycleGAN and CutGAN show blurring and a reduction in contrast. The pix2pix enhancement is not apparent. $Sam2$ is showcasing an area where delicate vessels interweave with hard exudates. When enhanced through our approach, both the vascular and pathological features surpass those observed in the original image. In darker regions of the original image, StillGAN’s enhancement unveils substantial minor hemorrhages or potential symptoms of microaneurysms, which do not present in the original image. We infer that the method may be mistakenly identifying and excessively amplifying noise within these darker regions as minor pathological features. The vascular structures enhanced by Pix2pix, CycleGAN, and CutGAN appear disordered, introducing vascular branches that should not originally exist. Comparatively, while StillGAN achieves superior edge sharpness of hard exudates compared to other methods, it also brings about an increase in image noise. The overall enhancement effect of the Pix2pix method is not substantial, and it might even introduce issues of uneven brightness. In terms of the overall perception of the image, CycleGAN and CutGAN demonstrate significant performance. However, they face challenges in preserving fine-scale features, which can lead to the loss of intricate details and the presence of blurriness. StillGAN performs well in terms of overall image perception and fine-scale feature retention. However, it suffers from a deficiency in vascular continuity and excessive enhancement. In summary, our method effectively enhances the overall quality of the image and achieves high fidelity in the fundus image’s fine-scale structure and pathological features.

5. Conclusions

In this study, we present a novel unsupervised framework for enhancing low-quality color retinal images. Moreover, we design a Global Feature Extraction Module and incorporated a non-reference function to retain delicate feature information within the images. To tackle the issue of uneven illumination in the images, we propose a brightness attention mechanism that is based on prior knowledge. Despite the absence of paired training data, the proposed framework is readily adaptable to real-world, low-quality color retinal images. Experimental results on publicly available color retinal image datasets show that our approach surpasses existing methods in terms of PSNR and SSIM, with improvements of 0.88 dB and 0.024, respectively. Additionally, it demonstrates that our method is better than other deep learning approaches in preserving vascular continuity and subtle pathological characteristics. However, we have observed that in some cases, the degradation factors are so severe that the valuable information in the retinal images is obscured, making restoration challenging. Consequently, the framework we have proposed might not perform well for such images. This may constitute the object of future studies.