FE-GAN: Fast and Efficient Underwater Image Enhancement Model Based on Conditional GAN

Abstract

The processing of underwater images can vastly ease the difficulty of underwater robots’ tasks and promote ocean exploration development. This paper proposes a fast and efficient underwater image enhancement model based on conditional GAN with good generalization ability using aggregation strategies and concatenate operations to take full advantage of the limited hierarchical features. A sequential network can avoid frequently visiting additional nodes, which is beneficial for speeding up inference and reducing memory consumption. Through the structural re-parameterization approach, we design a dual residual block (DRB) and accordingly construct a hierarchical attention encoder (HAE), which can extract sufficient feature and texture information from different levels of an image, and with 11.52% promotion in GFLOPs. Extensive experiments were carried out on real and artificially synthesized benchmark underwater image datasets, and qualitative and quantitative comparisons with state-of-the-art methods were implemented. The results show that our model produces better images, and has good generalization ability and real-time performance, which is more conducive to the practical application of underwater robot tasks.

1. Introduction

With the research and application of AUVs (autonomous underwater vehicles) and ROVs (remote operated vehicles), ocean exploration has achieved many breakthrough results. Over the last several years, many AUVs and ROVs have been applied to ship hull inspection, underwater target detection and tracking [1,2], pipeline leak detection [3], underwater cable inspection [4], and mineral prospecting. The preeminent underwater image contains a wealth of semantic and construction information, which can help the AUV to complete tasks, such as classification and recognition, efficiently and accurately [5]. However, due to the nonlinear attenuation of light caused by underwater particles, it is difficult to obtain high-resolution underwater images despite the use of high-pixel cameras. Therefore, image enhancement algorithms that can effectively enhance image perception and statistical quality have become an urgent need. Its application can vastly promote the development of ocean exploration.

Natural light is absorbed and scattered when propagating in seawater. Usually, red light with the longest wavelength is absorbed the fastest, and the propagation distance is the shortest. The green and blue light with a shorter wavelength will travel farther [6] but will be affected by the scattering and refraction of particles in the sea during the diffusion process. Based on this situation, images taken in shallow waters with natural light usually show a blue-green color. In the deeper sea (over 60 m), there is no natural light at all [7]. Limited by the characteristics of natural light spreading, underwater images have problems, such as color distortion, deviation, and uneven brightness [8]. We use image enhancement technology to meet these challenges. Specifically, the raw image needs to be defogged and deblurred to make the color hue consistent with the ground truth, thereby improving the image quality and highlighting useful information.

Many scholars have carried out in-depth research on the scattering phenomenon of light propagating in the medium. Through the inversion of this process, the distorted images (fogging, blurring, color unevenness, etc.) captured are operated to obtain the clear images as the desired output [9]. Some research groups have applied part of these models to underwater image-processing tasks and have made some progress. However, the attenuation parameter changes nonlinearly during the propagation process, so it is tricky to estimate [10]. Specific parameters set in the model limit it only to be used in given scenarios, resulting in insufficient robustness, so they do not satisfy the expected performance. The methods based on deep learning, especially the rapid development of CNN and GAN, provide scholars with another research idea. Some underwater image enhancement models based on CNN and GAN train paired and unpaired images, extract local and global image features, and optimize transmission maps. These models complete the image enhancement process by generating the target image output by the generator of the network, which improves the image contrast and color. Significant results have been achieved in enhancement. Nevertheless, since real underwater image datasets are difficult to obtain directly, many models are trained on artificially synthesized datasets, which usually cannot cope with different types and have insufficient generalization ability. Considering the requirements of underwater equipment for computing power and real-time processing, there is still room for a model with high efficiency and strong generalization performance.

This article proposed an underwater image enhancement model FE-GAN (fast and efficient generative adversarial network) to solve these problems. A hierarchical attention encoder (HAE) can extract deeper features and texture information, while preserving the overall structure of the image. Different loss functions based on texture and content are combined with weights to constrain the generator and discriminator. Through the learning of paired images, FE-GAN achieved end-to-end underwater image enhancement, which effectively improved the image quality. Results on different datasets prove that the model also has good generalization ability. The main contributions of this paper are as follows:

• We present a hierarchical attention encoder (HAE) to fully extract texture detail information, and a dual residual block (DRB) can more efficiently utilize residual learning to accelerate network inference.
• Through structural re-parameterization, we equate complex modules to simple convolutional layers, which accelerates the model during inference while maintaining a good enhancement effect.
• Based on HAE and DRB, we construct a fast and efficient underwater image enhancement network. Experiments on different datasets show that the enhanced image can achieve higher PSNR and SSIM values, and the mAP value also achieved significant results in the object detection task.

2. Related Work

2.1. Physical-Based Methods

As a crucial processing technology in the field of computer vision, image enhancement can purposefully emphasize the holistic or partial characteristics of an image. It can also expand the difference between the features of different objects in the image, improve the image quality, enrich the amount of information, and strengthen the recognition effect. The early underwater imaging model was presented by Ref. [11], where it is supposed that the scattering coefficient of each color channel remains constant within the camera’s sensitivity range. Later, the initial model was deeply simplified [12,13], thinking that the attenuation coefficients in all color channels are consistent, and applying the simplified model to underwater image restoration tasks. Inspired by these models, some researchers assigned a value to the attenuation coefficient of each color channel, respectively, in the model. Berman et al. [14] proposed a method to set different spectral profiles for relevant water types. Spier et al. [15] adopted an approach for estimating the medium properties using only images of backscattered light from the system’s light sources. Their experimental results show that the refined model can achieve a better effect.

Derya Akkaynak et al. [9] proposed the latest underwater imaging model; they attributed the scattering of light to the direct transmission attenuation and the backscatter attenuation. They are guided by scene reflectance and the spectrum of ambient light, which is advantageous to color reconstruction and image enhancement. Combined with this new model, the author of [16] proposed an underwater scene depth estimation method based on image blur and light absorption, which can be used in image formation models to restore and enhance underwater images. The refined imaging model will cause a vast increase in the parameters that need to be estimated, and the computational complexity will also increase sharply. Moreover, the physical-based method model is too singular to adapt to a variety of complex underwater scenes. Thus, the generalization and real-time ability of the application are greatly restricted.

2.2. Learning-Based Methods

In recent years, deep learning gradually occupied a leading position in the field of computer vision with its high plasticity and universality. Inspired by this trend, some scholars proposed to use the computing power of convolutional neural networks to calculate the parameters that need to be estimated in the physical imaging model [16]. Then the deep learning methods and physical models were combined for underwater image enhancement research.Liu et al. [17] proposed an integration method based on the Akkaynak–Treibit physical model [9]. Specifically, the physical model guides network learning and the network model design for components and coefficients estimation. Liu et al. [18] introduced VDSR (very-deep super-resolution reconstruction) into underwater resolution applications and proposed an underwater ResNet model for enhancement tasks. This model made good progress in automatic color enhancement, dehazing, and contrast adjustment. Qi et al. [19] proposed UICoE-Net, which introduced correlation feature matching units to provide rich complementary information for a mutual enhancement between images in the same scene.

The emergence of the GAN (generative adversarial network) opened up another path for image enhancement issues. However, the training process of GAN is usually unstable. Refs. [20,21,22] proposed different loss functions to constrain the discriminator, which effectively alleviates this situation. The conditional GAN proposed by the author of [23] is suitable for image-to-image translation tasks and achieves the mission of using images as conditions and generating corresponding output images.

For the existing synthetic and real underwater image datasets, many GAN-based methods have been proven to have achieved good results in underwater image enhancement. Cycle-GAN [24] and Dual-GAN [25] learn the mutual mapping between the two domains through the “cycle consistency loss” from unpaired data, ensuring that the network can complete the image generation task through unpaired image datasets. IPMGAN [17] proposed a new physical model integration network framework for underwater image enhancement. Through network training, the parameters and coefficients of the image degradation model are learned to reconstruct clear underwater images. FUnIE-GAN [26] proposed a real-time underwater image enhancement model based on fully convolutional conditional GAN, and formulated a multi-modal objective function. Experiments proved that the model achieved good results in both image enhancement and underwater human posture estimation. Zhou et al. [27] proposed a domain-adaptive learning framework, embedding a domain adaptive mechanism to eliminate the domain gap. UW-GAN [28] processes the single input image with a coarse-level network, then concatenates the result with the input image, and sends it to a fine-level network for a generation.

The application scenarios of most existing models are still very restricted, and it is rare to achieve good results in both real and synthetic underwater image datasets. Considering that image enhancement can be applied to the actual scene of underwater robots in the future, real-time performance is an indispensable part of model testing. We will explain the results of our model in terms of generalization ability and real-time testing in the following section.

3. Proposed Model

3.1. Overall Architecture

In image-related tasks, the generator of GAN receives a random noise z, then generates an image $G\left(z\right)$; the discriminator accepts $G\left(z\right)$ to determine if the image is authentic ($D\left(x\right)=1$) or not ($D\left(x\right)=0$) [29]. Therefore, in the training process of GAN, the goal of generator G is to generate images as real as possible to deceive discriminator D, and the target of D is to distinguish whether the images generated by G are real or not. In the optimal state, G generates a picture $G\left(z\right)$ that is sufficient to “make it fake”, that is, $D\left(G\right(z\left)\right)=0.5$ [30]. When the model converges, the images, both generated and real, have the same distribution.

As shown in Figure 1, we proposed a generative network G, which bypasses the output of each encoder $e_i$ to the input of its mirror decoder $d_i$. This aggregated connection can fuse limited hierarchical feature information, making the decoder use fewer parameters to ensure the efficiency of the network. The encoder is preceded by an initial module which consists of a convolutional layer with a kernel size of $4\times 4$, and a spatial maximum pooling in a $3\times 3$ area with a stride of 2. This initial module can effectively remove some redundant information by reducing the dimension, maintaining the invariance of features, and also ensuring the integrity of the generated image. The decoder is followed by a final module, which performs zero padding and upsampling operation on the $64\times 64$ images twice. After that, the final output image is transferred to the discriminator along with the ground truth image. The Markov discriminator [23] is adopted in our model.

3.1.1. Generator

The generator adopts the information multi-distillation module method to fuse the information of the encoder and its mirror decoder, improve the feature representation via the attention mechanism, and aggregate the hierarchical features. The details of the hierarchical attention encoder (HAE) are shown in Figure 2. Each encoder contains two basic modules composed of the ERB and residual block. To construct the relationship between local signals, we stack the ERB and ReLU operator. Batch normalization (BN) can capture the global context during training while incorporating it into convolutions during inference [31]. Although the dense residual block will help the generator to capture global information in the spatial dimension and adaptively focus on discriminative information, the skip connection introduces additional memory consumption and slows down inference. The enhanced residual block (ERB) [31,32] based on structural re-parameterization can effectively solve this problem. As shown in Figure 3, ERB is composed of two dual residual blocks (DRB) in series, which are applied for deep feature learning. The residual module can provide a more stable gradient during the backpropagation of the training and avoid gradient disappearance. During inference, we equate DRB to a $3\times 3$ convolution layer to reduce memory consumption and accelerate the model. Ultimately, the encoder learns 512 feature maps of size $8\times 8$. The decoder ($d_1$, $d_2$, $d_3$, and $d_4$) uses a convolution and deconvolution structure in series, as shown in Figure 4. This Conv-Deconv module has been proven to be effective in super-resolution tasks [33], which can minimize the potential artifacts generated during the upsampling process. Meanwhile, to reduce the computational complexity and increase the weight of discriminant information, the number of channels in B is reduced to $1/4$ of the original. This multi-stage convolutional connection facilitates gradient propagation and assists in training deeper networks. Features from multiple levels can be integrated using cascading mechanisms at local and global levels. The hierarchical pyramid structure can reconstruct residuals of different resolutions, and the local and global cascading mechanism also contributes to boosting expressive ability.

3.1.2. Discriminator

For the discriminator, we use a Markov discriminator [23], which is similar to that in the literature [26]. This discriminator can distinguish the true and false of the $N\times N$ block in the image, and the N here can be much smaller than the real size of the image so that the network has fewer parameters, runs faster, and can be applied to images of any size. Moreover, the three consecutive discriminator blocks maintain a certain high resolution and high detail for ultra-high resolution and image clarity in style transfer. The details of the discriminator are shown in Figure 5, and the input is set to $256\times 256\times 6$. There are a total of four modules, each of which performs convolution operations with a kernel size of $4\times 4$ and a stride of 2 on the image, followed by the BN and LeakyReLU layers. The final output is $16\times 16\times 1$, representing the average effective response of the discriminator.

3.2. Objective Function Formulation

The conditional generative adversarial network introduces additional auxiliary information and can learn the mapping $G:\{X,Z\}\to Y$. The objective function expresses as

$$\begin{array}{cc}\hfill L_{cGAN}=& \hfill E_{X,Y}\left[logD\left(Y\right)\right]+E_{X,Y}\left[log(1-D(X,G(X,Z)\left)\right)\right]\hfill \end{array}$$

(1)

where the generator G tries to minimize $L_{cGAN}$, and the discriminator D tries to maximize it [34]. Therefore, a game state is formed, which is

$$\begin{array}{c}\hfill GS=argmi{n}_{G}ma{x}_D{L}_{cGAN}\left(G,D\right)\end{array}$$

(2)

To further improve the quality of the generated image, we introduce the pixel-level and image-level loss functions into the objective function formulation.

Pixel-level: Existing research shows that the $L_1$ loss can efficiently capture high-frequency information in images. On the one hand, measure the pixel value between the real and generated image, ignoring the difference between the adjacent pixel values; on the other hand, the introduction of blur can be avoided, making G focus more on the color distortion of the image, and produce more realistic results. The water medium absorbs different wavelengths of light disparity, and the suspended particles have the effect of absorbing and scattering light, so most of the underwater images appear blue-green. Therefore, to make the enhanced image have a similar color style to the natural image, we add the $L_1$ loss to the objective function:

$$L_1=\frac{1}{N}\sum _{i=1}^N|Y_i-G\left(X_i\right)|$$

(3)

where Y represents the real image, and $G\left(X\right)$ represents the image generated by G.

Image level: Inspired by [35,36], according to the activation function generated by the ReLU layer of the pre-trained VGG-19 network, we define the content loss at the image level. This loss ignores the differences between per-pixels, encourages them to have similar feature representations in terms of content and perceived quality, and ensures that the image generated by the G and the real image have similar content. The formula for this loss function is as follows:

$$L_{con}\left(G\right)=E_{X,Y,Z}\left[{∥\mathsf{\Phi }\left(Y\right)-\mathsf{\Phi }\left(G\right(X,Z\left)\right)∥}_2\right]$$

(4)

The model we proposed uses paired image training, and an objective function is constructed for this purpose to guide G to generate an enhanced image that is highly similar to the real image in color and overall content. D filters out and discards images with different local textures and styles. Specifically, our model uses the following objective function for paired image training:

$$\begin{array}{c}\hfill L_{total}=argmi{n}_{G}ma{x}_D{L}_{cGAN}+{\lambda }_1{L}_1+{\lambda }_2{L}_{con}\end{array}$$

(5)

where ${\lambda }_1=0.7$, ${\lambda }_2=0.3$ respectively represent pixel-level and content-level loss function factors obtained based on experience.

4. Experiment

4.1. Dataset

In this section, we chose a relatively complete set of real and artificial synthetic underwater images to test the enhancement effect of the proposed model.

4.1.1. EUVP Dataset

In this dataset, part of the images are collected by seven different camera equipment; the other part comes from images captured in YouTube videos. It was proposed by Ref. [26], which contains three different categories: underwater_dark (5550 pairs), underwater_imagenet (3700 pairs), underwater_scenes (2185 pairs). The ground truth used for reference is generated by the trained CycleGAN [24]. Here, we use 3 different classifications of image pairs for training, another 515 images, and their corresponding reference images for testing.

4.1.2. ImageNet Dataset

This dataset uses the images with good brightness and visibility collected from Imagenet as ground truth. Similar to the EUVP dataset, using the trained CycleGAN [24] to degrade the ground truth images to generate distorted images, a total of 6128 pairs of images makes this dataset. In this experiment, we randomly selected 5500 pairs of images for training and the remaining 628 pairs for testing.

4.1.3. Mixed Dataset

Due to the lack of real underwater images, Silberman et al. in [37] proposed a synthetic underwater image dataset NYU-v2. The distorted images are generated based on the underwater image model and image synthesis algorithm. This dataset has generated the ocean (I, IA, IB, II, III, where Type I is the clearest, Type III is the most turbid) and coastal (1, 3, 5, 7, 9, where Type 1 is the clearest and Type 9 is the most turbid) types of images. In addition, UIEBD is a dataset composed of 890 real underwater images with their corresponding reference images [38]. In this experiment, we randomly selected 1250 pairs of images from the NYU-v2 dataset, and 800 pairs from the UIEBD dataset, then mixed them for training. The remaining 90 pairs of images from UIEBD were used as the test set Test-R90. There are 60 real underwater images without reference in UIEBD, which are severely distorted and difficult to restore. We also named them Test-C60 for this experiment.

4.2. Implement Details

We used Pytorch 1.8.0 to implement the FE-GAN model. Due to the limitation of memory, all pictures were resized to $256\times 256$ before training, which can also preserve global information better. For each dataset, 200 epochs with a batch_size of 8 were trained on an NVIDIA GeForce RTX 2060 graphics card. In the training process, we used Adam as the optimizer. Now, we will analyze the experimental results in both qualitative and quantitative aspects.

4.3. Qualitative Evaluation

We apply the FE-GAN model to real and artificially synthesized underwater image datasets, process paired and unpaired distorted images, and compare them with the corresponding ground truth images. Figure 6 shows some examples of the FE-GAN model in the EUVP dataset. It can be seen that our model effectively improves the contrast of the distorted image and makes the overall color more vivid. Moreover, it also improves the sharpness, so the image appears clearer. Here, we selected several state-of-the-art models. Among them, pix2pix [23], ResGAN [39], UGAN [40], FUnIE-GAN [26] are models based on the learning method, Mband-En [41] and Uw-HL [42] are physical-based models. As can be seen from Figure 7, the images generated by the physical-based models (Mband-En and Uw-HL) have over-saturated brightness and over-exposure, simultaneously, and the whole image presents red and yellow hues. The background of the image generated by ResGAN is too dark, and the contrast is not distinct; UGAN and FUnIE-GAN are not thorough enough to deal with the blue-green hues of the distorted image, so the background is still light green.

For the ImageNet dataset, we randomly selected 628 pairs of real underwater images for testing. Figure 8 shows a part of the test results. Through the processing of our FE-GAN model, the generated image effectively repairs the overall blue-green hue, and the entire brightness of the image is improved. The enhanced image effectively restores the red light, which is absorbed fastest, better distinguishes the background and foreground, and the overall image is clearer. We also selected several state-of-the-art learning-based models, UWCNN [43], CycleGAN [24], UGAN [40], IPMGAN [17], and compared them with our experimental results. As can be seen from Figure 9, the image generated by UWCNN is dark overall, but some areas are too bright to highlight the pivotal areas in the image. CycleGAN and UGAN have poor repair effects on green hues, and obvious traces of the original image can still be seen in the background. The sharpening effect of the image generated by IPMGAN is not good, and the holistic image is pale yellow and not clear enough.

To further verify the generalization ability of FE-GAN, we selected 990 images from the artificially synthesized dataset for testing and compared them with the corresponding ground truth images. Figure 10 shows the effect of partially generated images. The overall hue of the image generated by FE-GAN is close to the ground truth. At the same time, the repair effect of some overexposed areas in the distorted image is also better. We can see more texture information from the restored image. Considering the complex layout of the images in the dataset, the holistic effect of the enhanced image can meet the practical applications. However, it has to be mentioned that whether it is a real or artificial dataset, our model does not perform well for images with insufficient light. It cannot restore the brightness and details of the image very well, and the enhanced image is still relatively dark.

4.4. Quantitative Evaluation

To quantitatively analyze the enhancement effect of the FE-GAN model on the paired underwater image, we choose PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) as reference indicators. PSNR is an index used in the image field to measure the quality of reconstructed images, which is defined by taking the logarithm of MSE (mean squared error). Given a generated image I of size $m\times n$ and its corresponding ground truth image K, their MSE is expressed as

$$MSE=\frac{1}{mn}\sum _{i=1}^{m-1}\sum _{j=1}^{n-1}{\left[I(i,j)-K(i,j)\right]}^2$$

(6)

Since we resized the image before the experiment, the values of m and n are both 256. The definition of PSNR is as follows:

$$PSNR=10lo{g}_{10}\frac{MA{X}_I^2}{MSE}$$

(7)

where $MA{X}_I$ represents the maximum possible pixel value of the image, where each pixel is represented by 8 in binary, so $MA{X}_I=2^8-1=255$.

SSIM is a metric used to measure the similarity of images, and it can also be used to judge the quality of images after compression. It is mainly composed of three parts: luminance, contrast, and structure contrast. These are expressed as follows:

$$l(x,y)=\frac{2{\mu }_x{\mu }_y+c_1}{{\mu }_x^2+{\mu }_y^2+c_1}$$

(8)

$$c(x,y)=\frac{2{\sigma }_x{\sigma }_y+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}$$

(9)

$$s(x,y)=\frac{{\sigma }_{xy}+c_3}{{\sigma }_x{\sigma }_y+c_3}$$

(10)

where, the ${\mu }_x$, ${\mu }_y$ represent the mean value of x, y; ${\sigma }_x^2$, ${\sigma }_y^2$ represent the variance of x, y; and ${\sigma }_{xy}$ represent the covariance of x, y. $c_1={\left(k_{1}L\right)}^2$, $c_2={\left(k_{2}L\right)}^2$ are two constants, and L represents the range of pixel values [0, 255]. The definition of SSIM is as follows:

$$SSIM=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}$$

(11)

UIQM is a non-referenced underwater image quality evaluation metric based on the human visual system excitation, mainly for the degradation mechanism and imaging characteristics of underwater images. Using UICM (color measurement index), UISM (sharpness measurement index), UIConM (contrast measurement index) as the evaluation basis. UIQM is expressed as a linear combination of these three indexes. The larger the value, the better the color balance, clarity, and color of the image. UIQM expresses as follows:

$$UIQM=c_{1}UICM+c_{2}UISM+c_{3}UIConM$$

(12)

where $c_1=0.0282$, $c_2=0.2953$, $c_3=3.5753$ [44].

Table 1. Quantitative comparison results with the state-of-the-art methods in the EUVP dataset.

Model	PSNR	SSIM	UIQM
Mband-EN [41]	12.11	0.4565	2.28
Uw-HL [42]	18.85	0.7722	2.62
Pix2Pix [23]	20.27	0.7081	2.65
ResGAN [39]	14.75	0.4685	2.62
UGAN [40]	19.59	0.6685	2.72
FUnIE-GAN [26]	21.92	0.8876	2.78
MFFN [45]	24.73	0.8456	2.32
FE-GAN	26.83	0.8779	2.87

shows the quantitative comparison results of our model and state-of-the-art models on the EUVP dataset. We chose 1k paired images and tested PSNR, SSIM, and UIQM, respectively. As can be seen from the table, our model achieved better results in the two measurement indicators of PSNR and UIQM. SSIM achieves second place after FUnIE-GAN among the learning-based methods. In general, after comparing with ground truth images, the generated images of our model are better.

In the ImageNet dataset, we randomly selected 5500 pairs of images for training and the remaining 628 pairs for testing.

Table 2. Quantitative comparison results with the state-of-the-art methods in the ImageNet dataset.

Model	PSNR	SSIM
UWCNN [43]	15.4190	0.6127
CyCleGAN [24]	22.3160	0.7464
UGAN [40]	18.6562	0.5702
FUnIE-GAN [26]	24.3204	0.8193
IPMGAN [17]	23.5439	0.8142
FE-GAN	24.5396	0.8106

shows the test results of our model with UWCNN [43], CycleGAN [24], UGAN [40], FUnIE-GAN [26], IPMGAN [17] on the ImageNet dataset test set. Similarly, the PSNR value of the image generated by our model reached the highest results.

For the Mixed dataset, we selected Test-R90 (90 paired images) and Test-C60 (60 unpaired images) as the test sets of paired and unpaired images respectively and compared them with the same methods in qualitative evaluation. Both of these test sets are from the UIEBD dataset, which is more challenging. These images were taken in a poor light environment, and the overall number of this dataset is small, which brings a certain degree of difficulty to training. Here, we selected UCycleGAN [46], WaterNet [38], UWCNN [43], Unet-U [47], Ucolor [48], and FUnIE-GAN [26] models for comparison.

Table 3. PSNR test results of Test-R90. FE-GAN achieved the highest value.

Model	PSNR
UCyCleGAN [46]	16.61
Water-Net [38]	19.81
UWCNN [43]	16.69
Unet-U [47]	18.14
Ucolor [48]	20.63
FUnIE-GAN [26]	20.27
FE-GAN	20.68

shows the PSNR test results of Test-R90. Our model achieved the best value. Figure 11a shows some of the test results. The image generated by our model can improve the clarity and brightness of the image while retaining the overall layout, effectively improving the yellow-green hue problem.

Table 4. UIQM test results of Test-C60. FE-GAN achieved the highest value.

Model	UIQM
UCyCleGAN [46]	0.91
Water-Net [38]	0.97
UWCNN [43]	0.84
Unet-U [47]	0.94
Ucolor [48]	0.88
FUnIE-GAN [26]	0.98
Framework [27]	0.99
MDP Framework [49]	0.97
FE-GAN	1.01

shows the test results of Test-C60. Since there is no reference image in this dataset, we only chose UIQM as a comparison indicator. Similarly, our model achieved better results. Figure 11b shows part of the generated image of Test-C60. The overall brightness of the image is significantly improved, which will be helpful for the recognition of underwater images and resource exploration.

The application of underwater image enhancement technology to underwater detection equipment is an important research direction. Several aspects should be taken into consideration, such as FLOPs, number of parameters, and inference time during deploying on resource-limited devices. We chose fps as a metric to measure inference time, which expresses as

$$fps=\frac{framNum}{elapsedTime}$$

(13)

Table 5. Real-time comparison results with UGAN and FUnIE-GAN.

Model	FLOPs (G)	Paramaters (M)	fps (Hz)
UGAN [40]	18.143	54.4	93.1
FUnIE-GAN [26]	7.834	7.02	217.9
FE-GAN	2.84	11.476	204.1

shows the comparison results among FE-GAN, UGAN, and the more lightweight network FUnIE-GAN. The amount of parameters of the FE-GAN model is much fewer than that of UGAN. However, since the FE-GAN encoder is deeper than the FUnIE-GAN network, the amount of parameters is slightly larger than FUnIE-GAN. Furthermore, due to the structural re-parameterization module’s ability to speed up network inference and reduce memory consumption, our model has the lowest FLOPs, which means that the real-time performance is better. The fps indicators of FE-GAN and FUnIE-GAN both exceed 200. In summary, the FE-GAN model can meet the current real-time requirements of underwater detection.

For AUVs and ROVs, during underwater exploration activities, the purpose of improving the image quality is to improve the accuracy of tasks such as object detection and classification. We chose the pre-trained YOLOv5 as the object detection model and tested the images before and after enhancement on the EUVP dataset. Part of the test results is shown in Figure 12.

The first line is the unprocessed original distorted images, and the second line is the FE-GAN processed images. Compared with the original distorted image, the processed image has a more natural tone and increased brightness, so the target in the image is clearer and easier to identify. The results in the second, fifth, and last columns show that the fuzzy target can be detected in the processed image. Other examples show that the recognition error of the processed image is alleviated.

In addition, we downloaded the Aquarium Combined dataset, then trained and tested this dataset on the same hardware environment as the FE-GAN enhancement experiment. The object detection test was performed before and after the FE-GAN processing. Here, we chose YOLOv5 as the object detector. Figure 13 shows part of the test results. The overall blue tone of the original image is obvious, and the processed image effectively repairs this problem, making the image closer to the ground truth situation. Figure 14, Figure 15 and Figure 16 are the precision curve and confusion matrix trained on YOLOv5 for this dataset.

In the Aquarium Combined dataset, there are seven types of targets to be detected: fish, jellyfish, penguin, puffin, shark, starfish, and stingray. Here we used mAP (mean average precision) as a reference metric.

Table 6. The mAP comparison results of object detection with the state-of-the-art methods in the Aquarium Combined dataset.

Model	Fish	Jellyfish	Penguin	Puffin	Shark	Starfish	Stingray	All Classes
FUnIE-GAN [26]	0.587	0.631	0.664	0.406	0.657	0.799	0.674	0.631
UGAN [40]	0.599	0.588	0.683	0.411	0.709	0.738	0.707	0.634
Pix2Pix [23]	0.580	0.587	0.662	0.416	0.697	0.775	0.652	0.624
FE-GAN	0.622	0.601	0.720	0.498	0.740	0.780	0.747	0.672

shows FUnIE-GAN, UGAN, Pix2Pix, and FE-GAN in the above different types of mAP, and the average mAP of all classes. FE-GAN obtained the highest mAP value in the five categories of detection, and the average mAP of all classes also reached 0.672, which is the highest value among the above models.

In the case of insufficient natural light, the image obtained with the artificial light source itself is extremely distorted. Although the brightness and details of the image enhanced by FE-GAN were restored partially, there is still a large gap from the image style under natural light, which is also the focus of future research.

5. Ablation Study

We conducted feature fusion experiments between the encoder and decoder utilizing concatenate and aggregation, respectively. Isola et al. [23] used concatenate in the connection of encoder and decoder in U-Net, achieving encouraging results in the task of generating images of conditional GAN. In [26,50], concatenate is further proven to have better results in underwater image enhancement tasks. The aggregation method increases the amount of information describing the characteristics of the image, which means only the amount of information in one dimension is increasing [51]. In this case, Equations (14) and (15) ($X_i$, $Y_i$ represent the two input channels respectively) show that the aggregation method can greatly reduce the amount of computation, which is very beneficial for FE-GAN. We used the concatenate and aggregation methods to carry out information fusion experiments between the encoder and decoder.

Table 7. Experimental results of information fusion on model inference speed using concatenate and aggregation.

Fusion Method	GFLOPs	Parameters	fps
add	2.84	11.476	204.1
concatenate	5.1	11.854	171.0

shows the impact of the two methods on the inference speed with 44.31% promotion in GFLOPs and 3.19%reduction in number of parameters, and fps also increased 19.3%. To further verify the effectiveness of this choice, we tested the comparison network on the test set of the ImageNet dataset (628 paired images) and Test-R90 (90 paired images) of the Mixed dataset. The experimental results are shown in

Table 8. Test results of the ImageNet dataset. Using aggregation and concatenate connection between encoder and decoder.

ImageNet	Add	Concatenate
PSNR	24.5396	22.8863
SSIM	0.8106	0.7178

and

Table 9. Test results of the Mixed dataset. Using aggregation and concatenate connection between encoder and decoder.

Test-R90	Add	Concatenate
PSNR	20.6837	20.2470
SSIM	0.8074	0.6804

below.

$$Z_{concat}=\sum _{i=1}^c\left(X_i✶K_i\right)+\sum _{i=1}^c\left(Y_i✶K_i\right)$$

(14)

$$Z_{add}=\sum _{i=1}^c\left[\left(X_i+Y_i\right)✶K_i\right]$$

(15)

Here, we also chose PSNR and SSIM as the evaluation indicators that regard aggregation and concatenate as the connection mode between the encoder and the decoder. Under the same experimental conditions, the test results using the aggregation operation method perform better in both PSNR and SSIM values. It should be pointed out that because the training set and test set of the Mixed dataset are relatively small, the experimental gap here is not very large.

In recent years, many learning-based methods used $L_1$, $L_2$, and $smooth{L}_1$ loss to design the objective function. All of these loss functions have been proven to achieve significant results in different tasks. We used these three different loss functions to design the objective function of the entire model from the pixel level. Similarly, we conducted experiments on two different datasets, and the results are shown in

Table 10. Results of different loss functions under the ImageNet dataset.

ImageNet	${\mathit{L}}_1\mathit{loss}$	${\mathit{L}}_2\mathit{loss}$	$\mathit{smooth}{\mathit{L}}_1\mathit{loss}$
PSNR	24.5396	24.1444	24.4511
SSIM	0.8106	0.7249	0.7095

and

Table 11. Results of different loss functions under the Mixed dataset.

Test-R90	${\mathit{L}}_1\mathit{loss}$	${\mathit{L}}_2\mathit{loss}$	$\mathit{smooth}{\mathit{L}}_1\mathbf{loss}$
PSNR	20.6837	19.6374	20.3014
SSIM	0.8074	0.6411	0.7859

. $Smooth{L}_1$ loss solves the problem that the derivative is not unique at 0 and optimizes the convergence ability of the model, and it avoids the shortcomings of $L_2$ loss increasing with the square error. For low-level tasks, such as image enhancement, the stable gradient of $L_1$ loss is more conducive to the convergence of the model and the optimization of training. $SmoothL1$ loss is more suitable for tasks such as object detection.

Structural reparameterization is used in our encoder to speed up inference. As shown in Figure 3, we equate DRB to a $3\times 3$ convolutional layer during inference, for which we test the performance of the model on different datasets, respectively.

Table 12. FE-GAN (without-re) represents the model used in the inference stage. The metrics under the dataset represent PSNR and SSIM, respectively.

-	FE-GAN (Training)	FE-GAN (Inference)
EUVP	25.78/0.7568	26.83/0.8779
ImageNet	24.67/0.7538	24.54/0.8106
Test-R90	20.31/0.7196	20.68/0.8074
GFLOPs	3.21	2.84
parameters(M)	11.566	11.476

lists the results of the comparative experiments. The method using structural reparameterization performs better on two datasets, and its inference speed does increase, with 11.52% promotion in GFLOPs and 0.78% reduction in number of parameters.

6. Conclusions

In this paper, we proposed an underwater image enhancement model based on a conditional generative adversarial network. Structural reparameterization methods improved the ability of the model to extract features while also speeding up inference. The color, brightness, and contrast of the generated image were distinctly improved. Simultaneously, our model conducted qualitative and quantitative analysis experiments on real underwater images and artificial synthetic image datasets respectively, which effectively demonstrates the generalization ability of the model. Compared with the state-of-the-art methods, our model achieved better results. Nevertheless, our model does not perform well in enhancing darker images, especially in recovering details and textures, which means that it is still challenging in deeper waters, where artificial light sources are needed. Next, we will try to optimize more network modules with structural reparameterization to improve the enhancement effect of the model on images with insufficient brightness, and focus on the practical application in underwater object detection and scene analysis.