Underwater Image Enhancement Algorithm Based on Adversarial Training

Abstract

Ocean observation is the first step in the development of the ocean, whose abundant resources and strategic significance are attracting increasing attention. Observation methods based on visual sensor networks have received great attention from researchers due to their visualization capability and high information capacity. However, below the sea surface, objective factors such as blurriness, turbulence, and underwater color casting can cause image distortion and affect the acquisition of images. In this paper, the enhancement of underwater images is tackled using an adversarial learning-based approach. First, pre-processing is applied to address the significant color casting in the dataset, thus enhancing feature learning for subsequent style transfer. Then, corresponding improvements are made to a generative adversarial network’s structure and loss functions to better restore the features of the network output. Finally, evaluations and comparisons are performed using underwater image quality assessment metrics and several public datasets. Through multidimensional experiments, the proposed algorithm is shown to exhibit excellent performance in both subjective and objective evaluation metrics compared to state-of-the-art algorithms, as well as in practical visual applications.

1. Introduction

With the accelerated progress of ocean exploration, the underwater environments observed are becoming increasingly complex, making imaging processes more challenging. In such environments, the influence of the environmental conditions and the limitations of imaging devices make high-quality imaging of underwater targets difficult and ineffective [1]. The underwater images acquired not only suffer from severe color distortion and significant blurring, but also exhibit uneven brightness, artifacts, and other issues. Traditional image enhancement algorithms have limited effectiveness in improving image quality in such adverse environments, and are thus unsuitable for underwater vision applications, where preserving the integrity and accuracy of image information is crucial. This necessitates the development of image enhancement techniques specifically designed for underwater scenarios [2].

A degraded underwater image is presented in Figure 1, where it is clearly evident that the main challenges that underwater image enhancement (UIE) typically needs to address are color casting and blurriness. Color casting refers to the distortion of colors in images, which is caused by light refraction, absorption, and scattering in the aquatic environment. This phenomenon results in a deviation of hues, saturation levels, and brightness of specific colors from their true representation. On the other hand, image blurring in underwater photography is primarily induced by light diffusion, water turbidity, and the movement of marine organisms. These factors contribute to the loss of sharpness and detail in underwater images, leading to a lack of crispness and clarity in the visual output. However, in more adverse imaging conditions, considering these aspects alone cannot lead to image enhancement effects of superior quality. Under such adverse underwater conditions, algorithms need to demonstrate superior generalization and feature restoration capabilities. To address these challenges, based on the previous analysis, in this paper, enhancement and restoration solutions for underwater imaging under adverse conditions are investigated using adversarial learning. On the one hand, a dataset with prominent color casting characteristics is utilized to enhance feature learning for style transfer, which is a technique that aims to apply the style characteristics of one image onto another image, resulting in a new synthesized image. On the other hand, corresponding improvements are made to the network structure and loss function to improve the network’s output feature restoration ability.

Based on the above, the work of this paper is summarized as follows:

• To achieve better style transfer effects for UIE, in this study the training dataset is pre-processed based on color balance and fusion theories. The color balance weights assigned to images with more severe background color casting (i.e., images with higher intensities near the channel histogram centers) are relatively larger to compensate for the increased color degradation.
• A high-dimensional semantic cyclic loss function is proposed based on the theoretical analysis of style transfer and content loss. Through the generation of two images twice using the generator in adversarial learning, the extraction of features from the generated images and their comparison with the initial ones using a VGG-19 [3] feature extraction network module, a high-dimensional semantic cyclic loss function is formed. The training process demonstrates good convergence.
• Improvements are made to the generator network structure in a generative adversarial network (GAN), including the activation functions and upsampling layers. This improves loss reduction and the quality of the images generated during training.

The remainder of this paper is organized as follows: Section 2 provides an overview of the evolution of UIE technology and represents the latest research findings in the field. In Section 3, the intricacies of the algorithmic and model design are delineated. In Section 4, simulation and experimental validation are undertaken, followed by a thorough analysis of the outcomes. Finally, in Section 5 the the entirety of the document is encapsulated in a comprehensive summary.

2. Related Work

In recent years, the field of underwater vision has stimulated extensive attention towards the issue of UIE, owing to its wide range of applications. However, due to the inherent limitations of imaging conditions and devices in the underwater environment, UIE remains a persistently challenging task [2]. Enhancement methods can be broadly categorized into two principal categories: non-deep learning (DL)-based methods and DL-based methods. While the former possess advantages in terms of algorithmic complexity, they tend to exhibit poor visual effects and limited generalization capabilities. In contrast, the latter leverage the power of DL to learn and adapt to the features of reference datasets for the effective restoration of degraded underwater images. While DL-based methods are heavily dependent on hardware conditions and may possess high algorithmic complexity, they tend to exhibit strong migration properties and better generalization capabilities for multi-class scenes.

With the rapid improvement in graphics hardware computing power, DL methods have made great achievements in various image research and application fields, and the field of UIE is no exception. As one of the most classical DL frameworks, GANs [4] were first proposed in 2014 based on game theory. The generator and discriminator created according to the characteristics of the game were very simple and elegant, and the method of adversarial training was understandable and clear. GANs have been widely used in image style transfer, super-resolution, image enhancement, and other applications [5,6]. Among them, the CycleGAN [7] algorithm, which evolved from GANs, was a great success at the 2017 International Conference on Computer Vision, and achieved significant improvement in the field of style transfer. Fabbri [8] et al. applied image transfer to UIE based on the idea of CycleGAN, improved the loss function according to the characteristics of underwater imaging, and used paired datasets for enhancement training to achieve better performance. Islam [9] et al. extended CycleGAN and previous work and proposed FUGAN, which had a loss function based on the global content, color, local texture, and style information of the image, and was applied on the network structure of a conditional GAN; they also created the EUVP UIE dataset. In [10], an end-to-end underwater GAN for UIE and depth estimation was proposed. First, a shallow generative network was used to roughly estimate the depth map and, then, a more detailed underwater feature layer network was used to splice the estimated shallow and coarse depth maps with the input image and calculate the final depth map. Liu et al. [11] proposed a UIE method based on object-guided twin adversarial contrastive learning, where a bilateral-constraint closed-loop adversarial enhancement module was used to alleviate the requirement of paired data in unsupervised mode through the coupling of a twin inverse mapping, thus retaining more information features. However, the algorithms mentioned above have not demonstrated notable improvements in terms of network structure design, and exhibit high computational cost and complexity during the training process. This intricacy poses a potential hindrance to their practicality, particularly in real-time applications or environments with limited computational resources.

In addition, a task-aware feedback module was embedded in the enhancement process to guide the algorithm to update towards the target image. Wu et al. [12] proposed a two-level underwater image convolutional neural network UIE algorithm based on structural decomposition. By decomposing the original underwater image into high- and low-frequency components, a two-stage underwater enhancement network was proposed, which included a preliminary enhancement network and a refinement network. Qi [13] et al. proposed a UIE network with semantic attention mechanism guidance and multi-scale perception, in which semantic information was introduced as high-level feedback through regional enhancement feature learning. This multi-scale perception resulted in a better learning of local enhancement features of semantic regions. The fused features were consistent in semantics and contributed to effective visual enhancement. Panetta et al. [14] used an enhanced GAN model to improve underwater target tracking performance, and further introduced a cascaded residual network model for UIE, which improved the accuracy and success rate of the tracker. Zhou et al. [15] proposed a domain-adaptive learning UIE algorithm based on physics model feedback. A domain-adaptive mechanism was embedded in the learning framework to eliminate the gap between domains, and a physical constraint was used as a feedback controller to perform UIE. Zhang et al. [16] attempted to combine traditional methods with DL and proposed a multi-input dense connection generator network (MDNet) for underwater image enhancement. Additionally, they devised a multi-component loss function to improve the visual quality of the generated images. Guan et al. [17] changed the learning of the self-attention mechanism by introducing a trainable weight to balance the effect of the mechanism, improving the self-adaptive capability of the model. Lin et al. [18] proposed the dilated GAN (DGAN) method, which added an additional loss function using structural similarity. Yang [19] proposed a triple-branch dense block-based generative adversarial network (TDGAN) for the quality enhancement of underwater images, which improved performance and feature extraction efficiency, and retained more image details. However, almost all of the image enhancement methods based on DL have a common trade-off; due to the feature influence of the reference dataset, recovering more details usually means that more noise will be generated during the enhancement process.

In conclusion, methods based on DL mainly learn and fit a data feature distribution of the reference dataset [20,21]. In general, most enhancement algorithms are based on the training and learning strategy of fitting [22], which mainly consists of two parts. The first part is to improve the loss function of the training process, design an efficient learning task objective function, and quantify the loss function for the task; the final loss function should ultimately cause the network to converge. The second part is to optimize the structure of the network to improve its feature downsampling ability during training, and feature retention and reduction during upsampling. The first key point pertains to the direction of learning and fitting, while the second focuses on feature-level implementation. However, another important factor affecting DL model performance is the characteristics of the dataset, which directly affect the network capability of learning to fit the corresponding features. To enhance the generalization ability of the neural network, the characteristics of the reference dataset are also extremely important for the training process.

3. Design of Algorithms and Models

In this section, a UIE algorithm based on image pre-processing and adversarial learning is proposed. Its main idea is as follows: for adverse underwater environment image enhancement, the training data are pre-processed using color balance and fusion methods to optimize the features. The pre-processing follows a two-step approach, where in the first step a physics-based model approach targeting image feature mechanisms is adopted, while in the second step the performance of the UIE algorithm is further optimized using adversarial learning methods.

3.1. Color Balance and Fusion Pre-Processing

UIE mainly deals with two problems, namely color casting and blurring, both of which are particularly prevalent in underwater environments. Other issues that cause the loss of image features also need to be considered, such as uneven lighting and artifacts. In DL, the features of the training dataset have a significant impact on the mapping results of the generated images. In this paper, DL is combined with a physics-based approach to enhance underwater images. Before performing UIE, color balancing and fusion mappings are applied to the images using the white balance theory. Feature pre-processing based on mapping coefficients helps to restore the subtle changes in the images before and after enhancement. Figure 2 illustrates the steps involved in the pre-processing of the training set images.

In Figure 2, the image is input into a network trained using the training dataset, and the white balancing process is applied first to obtain an image whose colors are closer to those of the real scene. Next, sharpening is employed to enhance the edges and details of the image, making the image clearer. By applying the gamma function for correction, the perceived brightness and contrast of the image can be adjusted to output the image brightness value. Then, the global contrast coefficient is estimated through Laplacian filtering. Finally, multi-scale feature fusion is realized using a Laplacian and a Gaussian pyramid to obtain the final output image.

The pre-processing method for the underwater image dataset in this paper comprises a two-step strategy that combines white balance-based methods with image multi-scale fusion. The inconsistent imaging depth in underwater environments results in variations in color wavelength. The purpose of white balancing is to correct the color projection deviation encountered in such situations. This approach does not require explicit inversion of the optical model and can be used to correct images with relatively consistent background tones in the dataset. Following white balancing, image fusion is applied for the enhancement of the edges and details of the images, and to alleviate the contrast loss caused by backward scattering. The “gray world” method can also be used to remove the blue background tone from underwater images. However, this method can cause disruption in the red channel features, which mainly occurs because in underwater images the red channel values are too low, leading to excessive channel value compensation (since the gray world theory divides each channel based on its average value). To address this issue, it is necessary to calculate a correction for the red channel:

$$I_{rc}\left(x\right)=I_r\left(x\right)+\alpha ·({\overline{I}}_g-{\overline{I}}_r)·(1-I_r\left(x\right))·I_g\left(x\right)$$

(1)

where $I_r$ and $I_g$ represent the values of two channels in the image, r represents red, g represents green. The application of this equation means that channel values are not processed based on their original values but are rather normalized dynamically. Similarly, ${\overline{I}}_r$ and ${\overline{I}}_g$ represent the average values of the corresponding channels.

Underwater light is strongly absorbed and attenuated, especially in turbid waters or areas with high concentrations of plankton, resulting in significant attenuation of the blue channel. To address this issue, the compensation for the attenuation of the blue channel is calculated as follows:

$$I_{bc}\left(x\right)=I_b\left(x\right)+\alpha ·({\overline{I}}_g-{\overline{I}}_b)·(1-I_b\left(x\right))·I_g\left(x\right)$$

(2)

In a similar fashion to the previous equation, $I_b$ and $I_g$ are the blue and red channel values of the image, and $\alpha$ in Equations (1) and (2) takes the value 1. Once the channel corrections have been applied, the image is sharpened using a sharpening factor:

$$S=\frac{I+N(I-G\ast I)}{2}$$

(3)

where I represents the original image, G represents the image after smoothing using a filter (usually a Gaussian filter), * represents the convolution operation, and $N\{.\}$ represents a linear normalization operator whose effect is typically histogram stretching. This operator scales the intensity of all color pixels in the image through the application of a unique scaling factor, thus improving the intensity of all color pixels to cover the entire dynamic range.

After sharpening, the next step is fusion. The Laplacian contrast weight $W_L$ is used to estimate the global contrast and is obtained through Laplacian filtering. The calculation of the Laplacian sampling layer $L_i$ corresponding to the output image $I\left(x\right)$ is as follows:

$$\begin{array}{cc}\hfill I\left(x\right)& =I\left(x\right)-G_1\left\{I\left(x\right)\right\}+G_1\left\{I\left(x\right)\right\}\hfill \\ \hfill & \triangleq L_1\left\{I\left(x\right)\right\}+G_1\left\{I\left(x\right)\right\}\hfill \\ \hfill & =L_1\left\{I\left(x\right)\right\}+G_1\left\{I\left(x\right)\right\}-G_2\left\{I\left(x\right)\right\}+G_2\left\{I\left(x\right)\right\}\hfill \\ \hfill & =L_1\left\{I\left(x\right)\right\}+L_2\left\{I\left(x\right)\right\}+G_2\left\{I\left(x\right)\right\}\hfill \\ \hfill & =\dots \hfill \\ \hfill & =\sum _{l=1}^N{L}_1\left\{I\left(x\right)\right\}\hfill \end{array}$$

(4)

where L represents the Laplacian pyramid sampling and $G_i$ represents the Gaussian pyramid sampling, with i representing the i-th layer. Both the Gaussian and the Laplacian pyramids have the capability of multi-scale representation and image reconstruction, yet they emphasize different aspects of feature information. The Gaussian pyramid primarily focuses on the overall structure and texture information of the image, whereas the Laplacian pyramid is more concerned with the local detailed information of the image. For underwater image deblurring, the Laplacian weight alone is insufficient for color variation discrimination and contrast restoration. To address this problem, an additional contrast evaluation metric is introduced.

The Laplacian contrast weight $W_L$ is a combination of $L_i$. The saturation weight ($W_S$) aims to enhance textural features in underwater scenes through contrast improvement. To measure the level of image saliency, a saliency estimation method is used. In regions with high brightness, the effect of the saturation weight is more pronounced:

$$W_S(x,y)=\sum _{(m,n)\in N}d[\mathit{p}(x,y),\mathit{q}(m,n)]$$

(5)

Larger convolution kernels can have a better image noise smooth effect, and so a $10\times 10$ convolution kernel is used here. In Equation (5), N is the small neighborhood of the pixel block $(x,y)$ obtained by filtering and downsampling the image, and d is the Euclidean distance between pixel vectors $\mathit{p}$ and $\mathit{q}$.

The saturation weight $\left(W_{Sat}\right)$ is used to retain color information even when the intensity values of the image enter the saturated range, and is defined as follows:

$$W_{Sat}=\sqrt{\frac{1}{3\Delta }}$$

(6)

$W_{Sat}$ combines the channel values of $R_k$, $G_k$, and $B_k$ with the k-th image’s background light intensity $L_k$, while $\Delta$ is defined as follows:

$$\Delta ={\left(R_k-L_k\right)}^2+{\left(G_k-L_k\right)}^2+{\left(B_k-L_k\right)}^2$$

(7)

In practical applications, each individual weight coefficient has its advantages, so a combination of the three coefficients is used. For each input k, the three weights $W_L$, $W_S$, and $W_{Sat}$ discussed earlier are summed to obtain the aggregated weight $W_k$. Then, the aggregated weights are normalized on a per-pixel basis by dividing each pixel’s weight with the sum of the weights for the corresponding pixel across all images, resulting in the following equation:

$${\overline{W}}_k=\frac{\beta \left(W_k+\sigma \right)}{{\displaystyle \sum _{k=1}^K}{W}_k+K\sigma }$$

(8)

where $\sigma$ is a small regularization term to ensure that the input corresponds to the output, with a value of 0.1, and $\beta$ is a correction factor with a value of 0.85.

The relationship between the weight parameters obtained during the normalization process can be used to reconstruct the image $R\left(x\right)$ through weighted fusion between the image and pixel positions x:

$$R\left(x\right)=\sum _{k=1}^K{\overline{W}}_k\left(x\right)I_k\left(x\right)$$

(9)

where $I_k$ represents the input. In order to achieve a good image fusion effect, all the aforementioned operations need unified image dimensions. Therefore, in the implementation process, image pyramids are used to sample the images based on their resolutions in order to obtain the same dimensions. Following a traditional multi-scale fusion strategy [23], each input $I_k$’s features are decomposed into a Laplacian pyramid [24], while the corresponding weights are decomposed into a Gaussian pyramid. To ensure the uniformity of the feature dimensions, these two pyramid structures should have the same numbers of levels. To obtain the final weight information, the Laplacian inputs and Gaussian pyramid weights are fused, as calculated below:

$${\mathcal{R}}_l\left(x\right)=\sum _{k=1}^K{G}_l\left\{{\overline{W}}_k\left(x\right)\right\}{L}_l\left\{I_k\left(x\right)\right\}$$

(10)

where l represents the pyramid sampling level, and k represents the number of input images.

3.2. Generative Adversarial Learning Principles

GANs [2] are based on the concept of a “game”, and utilize an adversarial process between a generator and a discriminator to estimate a generative model [25]. The process is similar to a scenario where a police officer tries to distinguish counterfeit money from genuine currency; the generator can be considered as the team producing counterfeit money and attempting to use it without detection, while the discriminator acts as the police officer, aiming to verify the authenticity of the currency. The training process of adversarial learning is akin to that of a competitive game where both sides improve their technique until counterfeit and genuine items become indistinguishable. Figure 3 illustrates the structure of a GAN.

In adversarial learning, the data generated by the generator and the real data are input into the discriminator, which judges considering the features of the two data inputs. The loss is then fed back to both models during label verification. Adversarial learning requires training both models simultaneously: the generator G, which aims to fit the training data distribution, and the discriminator D, which estimates the probability that a sample comes from the training data or the generator. In the function space of G and D, the ideal scenario is when the discriminator D outputs each of the two values with a probability of 0.5, thus achieving a game equilibrium. The generator and discriminator are composed of multilayer perceptrons, enabling the training of the entire system model using gradient-based back-propagation. During sample training or generation, there is no need for approximate inference networks or Markov chains; instead, the quality of network training is evaluated through a qualitative and quantitative assessment of the generated samples.

Figure 4 shows the process of data distribution changes during the training of a GAN. Training involves updating the discriminator distribution (represented by the blue dashed line) to differentiate between samples from the real data distribution (black dashed line) $p_x$ and samples from the generated distribution $p_g$ (green solid line). The lower line z represents random noise initially distributed uniformly, while the upper line represents real data. The upward arrows represent the mapping of uniformly distributed random noise z through the generator to the real data x. During the adversarial process, the real and the generated data can be easily distinguished by the discriminator D through an activation function (blue dashed line). After multiple iterations, the distribution change from (a) to (d) occurs, and the two distributions become almost identical. At this point, the discriminator cannot differentiate between the two distributions, and the output becomes a straight line with a value of 0.5, achieving an adversarial balance. Neither the generator nor the discriminator can further improve their accuracy.

3.3. Design of Adversarial Learning Loss Function

The use of DL for UIE can be considered as a style transfer process from the original domain to the target domain. Style transfer between images is a type of visual and graphic reconstruction problem, with the objective of learning a feature mapping relationship from the training set such that the input and target images have the same content style. In recent years, GANs have excelled in the field of image style transfer, with numerous representative style transfer algorithms emerging. In this paper, improvements are made based on style transfer algorithms, including the loss function and network structure.

The overall process diagram of this paper’s algorithm is shown in Figure 5. Recent algorithms used for this purpose, such as CycleGAN [7], UGAN [8], or FUGAN [9], consider only pixel-level cycle loss or unidirectional high-dimensional semantic loss. CycleGAN and VGG-19 provide good inspiration, as they take into account high-dimensional image feature information for the cycle loss function. The main difference in the proposed model is the definition of a cycle loss function based on high-dimensional semantic information, which includes both content- and pixel-level cycle losses. In the proposed function, high-dimensional semantic information is extracted from the original image using VGG-19’s feature extraction module. However, in our opinion, L2 loss is too susceptible to outliers and may result in feature blurring during training with high-dimensional semantic loss. Therefore, in this paper, an L1 loss is employed for high-dimensional semantic information, as follows:

$$\begin{array}{cc}\hfill L_{\mathrm{con}}\left(G\right)& =E_{x\sim \mathrm{Pdata}\left(x\right)}\left[{\parallel \mathrm{\Phi }\left(x\right)-\mathrm{\Phi }\left(F\left(G\left(x\right)\right)\right)\parallel }_1\right]\hfill \\ \hfill & +E_{y\sim \mathrm{Pdata}\left(y\right)}\left[{\parallel \mathrm{\Phi }\left(y\right)-\mathrm{\Phi }\left(G\left(F\left(y\right)\right)\right)\parallel }_1\right]\hfill \end{array}$$

(11)

where $\mathrm{\Phi }$ represents the feature extraction operation performed by the VGG-19 network, and G and F are the image generation functions that distinguish the generation from the original domain to the target domain and vice versa. The form of the adversarial loss in this paper is as follows:

$$\begin{array}{cc}\hfill L_{\mathrm{adv}}\left(G,D,X,Y\right)& =E_{y\sim \mathrm{Pdata}\left(y\right)}\left[\log D\left(y\right)\right]\hfill \\ \hfill & + E_{x\sim P\mathrm{Pdata}\left(x\right)}\left[\log \left(1-D\left(G\left(x\right)\right)\right)\right]\hfill \end{array}$$

(12)

In addition to the cycle loss with high-dimensional semantics, in order to better calculate the network weights and generate multidimensional feature information in images, pixel-level loss functions are also taken into account in this paper. These include pixel-level cycle loss and content loss, which are formulated as follows:

$$\begin{array}{cc}\hfill L_{\mathrm{cyc}}(G,F)& =E_{x\sim \mathrm{Pdata}\left(x\right)}\left[{\parallel F(G\left(x\right)-x)\parallel }_1\right]\hfill \\ \hfill & + E_{y\sim \mathrm{Pdata}\left(y\right)}\left[{\parallel G(F\left(y\right)-y)\parallel }_1\right]\hfill \end{array}$$

(13)

$$L_{\mathrm{pixel}}=E_{x\sim \mathrm{Pdata}\left(x\right)}\left[{\parallel Y-G\left(x\right)\parallel }_1\right]$$

(14)

Finally, the loss function for the adversarial learning generator is calculated as the weighted sum of the above loss functions:

$$L_G=L_{\mathrm{adv}}+{\lambda }_1{L}_{\mathrm{cyc}}+{\lambda }_2{L}_{\mathrm{pixel}}+{\lambda }_3{L}_{\mathrm{con}}$$

(15)

where the weight coefficients ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are normalized, with ${\lambda }_1=0.25$, ${\lambda }_2=0.5$, and ${\lambda }_3=0.1$.

The discriminator loss mainly comes from two parts, namely the real and the generated data. The overall loss function is the weighted average of the two, calculated as follows:

$$\begin{array}{cc}\hfill L_D& =E_{x\sim \mathrm{Pdata}\left(x\right)}\left({\parallel D\left(x\right)-V\left(x\right)\parallel }_1\right)+E_{y\sim \mathrm{Pdata}\left(y\right)}\left(\parallel D\left({G\left(y\right)-F\left(y\right)\parallel }_1\right)\right.\hfill \end{array}$$

(16)

where $V\left(x\right)$ represents the labels of the real data, defined as a tensor of the same size as the input image, with all values equal to 1. Similarly, $F\left(x\right)$ represents the labels of the fake data, defined as a tensor of the same size as the input image, with all values equal to 0. When the data pass through the discriminator, the latter outputs a value in the range of $[0,1]$, which indicates the probability of the data being true values.

3.4. Adversarial Learning Network Architecture

The generator network, shown in Figure 6, plays a crucial role in ensuring the quality of the generated images. In this study, some improvements are made to the original generator network structure. On one hand, to ensure that the training feature accuracy encompasses comprehensive texture details, the ResNet network structure is utilized for training, where the upsampling layer is replaced with PixelShuffle [26]. On the other hand, considering the negative samples’ influence on gradient updates, LeakyReLU is used as the activation function, which improves the training performance in negative gradients. The generator network structure includes an initialization layer, a ResNet network, upsampling and downsampling layers, and a final output layer.

The discriminator network structure is shown in Figure 7; it is the original architecture proposed [7], which includes four discriminator modules and an output layer. In adversarial learning, the discriminator is mainly responsible for distinguishing between real and generated images. During training, especially in the early stages when the generator’s capability is low, the discriminator’s loss is sufficiently small to handle the discrimination of real and generated images. The training process is shown in Algorithm 1.

Algorithm 1: Adversarial learning models train algorithms

4. Overall Experimental Results and Comparative Analysis

As mentioned earlier, the comprehensive evaluation of underwater image quality should consider factors such as colorfulness, saturation, and contrast, which reflect the image’s quality. Common evaluation metrics include the underwater color image quality evaluation (UCIQE) metric [27], the underwater image quality measure (UIQM) [28], and entropy. To comprehensively evaluate the performance of the trained DL models and demonstrate the superiority of the proposed algorithm, the UIEB [2] and EUVP [9] datasets were used; the latter contains two sub-datasets corresponding to different scenes. In the UIEB dataset, which contained reference images, the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) were used as evaluation metrics to assess the quality of the generated images. The compared algorithms include four DL algorithms, namely CycleGAN [7], FUGAN [9], Shallow-Uwnet [29], and MDNet [16], as well as three non-DL algorithms, which were BLOT [30], CB [31], and RGHS [32].

4.1. Design and Testing of Adversarial Learning Model Training

For the experiments, the adversarial learning network structure was first designed in Spyder software and the logic of the code implementation was initially verified. To improve efficiency, all code models were trained on the GPU. The specific experimental platform parameters were as follows: the processor was an Intel Xeon E5-2640v4 with four GTX1080Ti graphics cards. The operating system was Ubuntu 16.04 and the system RAM was 32 GB at 2400 MHz. To achieve sufficient training accuracy, the number of training iterations (epochs) was set to 200. A large batch size would cause significant fluctuations in the training process loss, making it difficult to observe the parameter updates during training. Additionally, a large batch size may lead to weak feature learning from the batch. Therefore, we set the batch size to 1. The ADAM network optimizer was also used, with a learning rate of 0.0002, and the coefficient range for calculating running averages and squared gradients was $(0.5,0.999)$. Learning rate decay occurred after 100 iterations. The image transformation size was set to $256\times 256$ with three channels. The input data were normalized before training.

To verify that the proposed loss function logic met the training requirements, the overall loss function and content loss of the high-dimensional semantic information ($L_{con}$) were observed during the training process. Figure 8 shows the convergence of the loss functions during the training of the adversarial learning model.

From Figure 8, it can be seen that the loss of high-dimensional semantic information during the training process initially fluctuated significantly but stabilized and decreased to a stable value as the number of iterations increased. This indicates that the designed loss function demonstrates good convergence during model training. In addition, the overall generation loss followed a similar trend. The process transitions from oscillation to stability and eventually convergence, and is therefore suitable for model training.

4.2. Evaluation Metrics

The clarity and color richness of underwater images are highly correlated. The UCIQE is a commonly used underwater image quality metric that quantifies the color, saturation, and contrast aspects of underwater optical images. The UCIQE avoids excessive reliance on saturation deviation because an excessive emphasis on saturation tends to highlight dark areas, which are often present in underwater images due to low lighting conditions. Overcompensating in these areas can lead to evaluations where a high index value does not reflect the perceived image quality.

Compared to other color image evaluation metrics, UCIQE encompasses quantified degradation metrics of underwater images through three quality quantization indicators: chromaticity, contrast, and saturation. UCIQE has higher correlation with the prediction and perceived quality of underwater images, and it can be calculated using an efficient real-time algorithm with low complexity.

UCIQE is used to evaluate the quality of underwater images in terms of color chromaticity, saturation, and contrast. However, a single evaluation metric cannot comprehensively reflect the quality of underwater images. To objectively evaluate whether the output image is in line with human perception, the UIQM evaluation metric was also used to evaluate the performance of the proposed model [28]. The UIQM is a quantification metric based on the human visual system, which is highly correlated with the perception of human visual perception and the perceived quality of underwater images. The UIQM takes into account three dimensions of underwater image characteristics measurement, namely the Underwater Image Colorfulness Measure (UICM), the Underwater Image Sharpness Measure (UISM), and the Underwater Image Contrast Measure (UIConM).

The UIQM is a linear combination of UICM, UISM, and UIConM:

$$\begin{array}{cc}\hfill UIQM& =c_1\ast UICM+c_2\ast UISM+c_3\ast UIConM\hfill \end{array}$$

(17)

It is worth noting that the UIQM in Equation (17) has three parameters, $c_1$, $c_2$, and $c_3$. The selections of these parameters are application-dependent. For general applications, the combination coefficients are obtained using multiple linear regression. For the results shown in this paper, a generic coefficient set $c_1=0.0282$, $c_2=-0.2953$, and $c_3=3.5753$ was used for the calculation of UIQM.

Information entropy represents the amount of information in an image, and its expression is as follows:

$$H=-\sum _{i=0}^{255}{P}_i\log P_i$$

(18)

where $P_i$ represents the proportion of pixels with a grayscale value of i in the total pixels.

In order to comprehensively evaluate the quality of images generated through DL, it is necessary to evaluate their color feature composition in an unreferenced manner, and also to compare the texture features of the generated images to those of the reference images. Therefore, two additional evaluation metrics were adopted: the SSIM and the PSNR. The SSIM is used to measure the structural similarity between two images and is calculated as follows:

$$SSIM(x,y)=\left(\frac{2{\mu }_x{\mu }_y+c_1}{{\mu }_x^2+{\mu }_y^2+c_1}\right)\left(\frac{2{\sigma }_{xy}+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}\right)$$

(19)

where $\mu$ represents the mean, $\sigma$ represents the standard deviation, ${\sigma }_{xy}$ represents the covariance of x and y, and c is a constant to ensure stability. SSIM evaluates the similarity of two images from the perspectives of brightness, contrast, and structure.

The PSNR is commonly used to measure the reconstruction quality of an image. It is based on the Mean Squared Error (MSE) between the generated image x and the ground truth image y, and its equation is as follows:

$$PSNR=10{\log }_{10}\left(\frac{MA{X}_I^2}{MSE}\right)$$

(20)

4.3. Generalization Comparison Experiment

In the first experiment, the generalization ability of the trained adversarial learning model was tested and its enhancement performance was evaluated across multiple classes of scenes. Experiments were conducted on the validation set of the IMAGENET sub-dataset in EUVP, which contains various underwater image features, including color distortion and different object characteristics. Randomly selected enhanced images from the experiments are shown in Figure 9, while

Table 1. Average index value of each algorithm on the IMAGENET dataset.

Evaluation Index	Original Image	CB	BLOT	Cycle-GAN	RGHS	FU-GAN	The Proposed Method
UCIQE (↑)	0.569	0.575	0.631	0.579	0.623	0.583	0.571
UIQM (↑)	2.592	3.010	2.192	2.814	2.200	2.763	3.093
Entropy (↑)	7.438	7.612	7.340	7.542	7.322	7.580	7.655

shows the average metric values for each of the compared algorithms on the respective sub-datasets.

Next, subjective and objective experiments were conducted on the SCENES sub-dataset of EUVP. The comparison of enhancement results and metrics is presented in

Table 2. Average index value of each algorithm on the SCENES dataset.

Evaluation Index	Original Image	CB	BLOT	Cycle-GAN	RGHS	FU-GAN	The Proposed Method
UCIQE (↑)	0.566	0.589	0.657	0.589	0.624	0.588	0.594
UIQM (↑)	2.466	3.067	2.246	2.785	2.155	2.767	3.066
Entropy (↑)	7.531	7.579	7.446	7.579	7.430	7.579	7.606

and Figure 10, while

Table 3. Average index value of each algorithm on the UIEB RAW dataset (1).

Evaluation Index	Original Image	CB	BLOT	Cycle-GAN	RGHS	FU-GAN	The Proposed Method
UCIQE (↑)	0.548	0.589	0.626	0.580	0.628	0.576	0.586
UIQM (↑)	2.364	3.003	2.242	2.949	2.391	3.075	3.301
Entropy (↑)	7.273	7.481	6.971	7.489	7.509	7.521	7.614
SSIM (↑)	0.795	0.853	0.768	0.752	0.827	0.707	0.790
PSNR (↑)	20.823	21.245	18.739	2.947	23.484	20.457	22.467

and

Table 4. Average index value of each algorithm on the UIEB RAW dataset (2).

Evaluation Index	Original Image	Shallow- Uwnet	MDNet	The Proposed Method
UIQM (↑)	2.364	3.090	2.078	3.301
SSIM (↑)	0.795	0.810	0.820	0.790
PSNR (↑)	20.823	20.780	22.270	22.467

, and Figure 11 are the corresponding results from the UIEB RAW dataset.

Observing the enhancement results from the three datasets, it is evident that, compared to the CB algorithm, the algorithm in this paper exhibits better contrast and color saturation, resulting in an overall brighter image. Compared to the BLOT algorithm, the proposed algorithm performed better in color bias reduction, overexposure avoidance, and blurriness minimization. Compared to the CycleGAN algorithm, the proposed algorithm also shows superior results in residual color bias and overall enhancement; while the FUGAN enhancement is effective, it tends to overcorrect colors, leading to other color bias issues in the image. Compared to RGHS, which has the best performance among the various control groups, the proposed algorithm exhibited lesser residual color bias. In summary, the images enhanced by the proposed algorithm generally demonstrated optimal color bias correction, color saturation, brightness, and contrast, and relatively low noise levels.

From Table 1, Table 2, Table 3 and Table 4, it can be observed that, on the IMAGENET dataset, the proposed algorithm achieved the highest average values for the UIQM and entropy metrics. In terms of subjective and objective evaluations, the proposed algorithm performed best on this sub-dataset. In the SCENES dataset, the proposed algorithm also yielded the highest average values for the UIQM and entropy metrics, indicating its superior overall enhancement performance. In the UIEB RAW dataset, since reference images were available, the two additional metrics SSIM and PSNR were included. The RGHS algorithm achieved the highest average values for these two metrics, while the proposed algorithm also obtained the highest average values for both metrics. Therefore, on this dataset, the proposed algorithm surpassed the other algorithms in their respective specialties, with each algorithm having its own strengths.

Considering the three datasets collectively, the proposed algorithm performed optimally on two sub-datasets and demonstrated its advantages in the other sub-dataset compared to the other algorithms. Thus, the proposed algorithm exhibits good generalization ability across both synthetic and real-world datasets.

4.4. Harsh Underwater Environment Comparison Experiment

The generalization ability of the proposed algorithm shows its versatility in enhancing underwater images in general scenes. To evaluate the algorithm’s superior performance in imaging under harsh underwater environments, a comparative experiment on the CHALLENGING (CHAL) sub-dataset from the UIEB dataset was conducted.

From

Table 5. Average index value of each algorithm on the UIEB CHAL dataset.

Evaluation Index	Original Image	CB	BLOT	Cycle-GAN	RGHS	FU-GAN	The Proposed Method
UCIQE (↑)	0.518	0.550	0.586	0.555	0.612	0.546	0.556
UIQM (↑)	1.985	2.381	1.836	2.645	1.916	2.876	3.099
Entropy (↑)	6.934	7.195	6.854	7.219	7.325	7.215	7.447

, it can be seen that, compared to the other algorithms, the proposed algorithm exhibited significant improvement in the UCIQE metric, surpassing them by a large margin in terms of UIQM and entropy values. However, relying solely on objective evaluation metrics has certain limitations. As shown in Figure 12, although RGHS achieves a high UCIQE score, its enhancement effect is not as good as those of the proposed algorithm in certain challenging underwater environments. The RGHS exhibited issues such as excessive color correction, residual color distortion, and poor exposure in dark areas. In contrast, the proposed algorithm performed better in terms of contrast, color saturation, and clarity in the majority of image enhancements, aligning more with human perception and producing images of better quality. Thus, it can be concluded that the proposed algorithm outperforms other algorithms in terms of enhancement effectiveness in harsh underwater environments, both subjectively and objectively.

4.5. Ablation Experiment

In this paper, the white balance and the multi-scale fusion algorithms were adopted for image pre-processing. In order to verify the roles of the pre-processing module in the entire network model, it is necessary to conduct ablation experiments.

The ablation experiments were designed to exclude one or both of the pre-processing modules from the network model, and then evaluate the performance of the modified network model on a benchmark dataset. By comparing the evaluation results of the modified network model with those of the original network model, the effectiveness of the pre-processing module and the DL module can be quantitatively assessed.

The experiments were conducted using the UIEB RAW dataset. The experimental environment and parameter settings were consistent with those described in the previous sections. The experimental results are shown in Figure 13 and

Table 6. Ablation experiment result on the UIEB RAW dataset.

Pre-Processing Module	DL Module	UCIQE (↑)	UIQM (↑)	Entropy (↑)	SSIM (↑)	PSNR (↑)
✓		0.574	3.111	7.384	0.847	19.935
	✓	0.579	2.984	7.476	0.754	21.119
✓	✓	0.586	3.301	7.614	0.790	22.467

.

According to the results of the ablation experiments, it can be concluded that incorporating pre-processing modules leads to better overall performance. First, the addition of the pre-processing modules has a significant effect on improving image quality. By applying white balance and multi-scale fusion algorithms, color balance and detail richness are improved. Compared to the absence of pre-processing, the images with pre-processing are clearer, more natural, and reproduce colors more accurately. Second, the introduction of pre-processing modules also has a positive impact on the model’s prediction accuracy. By optimizing the images, the model is less disturbed during feature extraction and prediction, thereby improving prediction accuracy. Compared to the absence of pre-processing, the model with pre-processing achieved significant improvements in all evaluation metrics. Finally, comparing the results of the ablation experiments, it can be inferred that pre-processing modules have a greater impact on the performance of the entire system. In this experiment, white balance and multi-scale fusion algorithms were proven to be two important pre-processing modules, and their presence greatly improved the performance of the model in image processing tasks. In addition, the incorporation of DL modules also made a positive contribution to the model’s performance, demonstrating the importance of DL in image feature extraction and prediction tasks.

To this end, the results of the ablation experiments indicate that the incorporation of pre-processing modules is crucial for improving the performance of image processing systems. The application of white balance and multi-scale fusion algorithms can significantly improve image quality and prediction accuracy, while the incorporation of DL modules further enhances the model’s ability. These experimental results provide valuable guidance for the selection and optimization of pre-processing algorithms for similar image processing tasks.

4.6. Multidimensional Comprehensive Evaluation

To comprehensively evaluate the algorithm’s performance, after verifying the algorithm’s superiority across multiple datasets using subjective and objective evaluations, comparative experiments were conducted on feature detection in practical scenarios. First, feature detection was performed using the SIFT [33] algorithm, which detects image features that are positively correlated with image metrics such as contrast and color saturation. In this experiment, the detection of features is facilitated when the image textures are clearer, as this results in correspondingly clear transformation results. Therefore, the number of SIFT key-points is a quantitative indicator reflecting the richness of the features in an image. Seven underwater images named as Scene 1 to Scene 7 and obtained from harsh environments, as presented in Figure 14 and described in

Table 7. Comparison of feature point detection numbers in selected underwater images.

Evaluation Index	Original Image	CB	BLOT	Cycle-GAN	RGHS	FU-GAN	The Proposed Method
Scene 1	232	549	630	650	725	600	791
Scene 2	16	189	98	70	181	25	391
Scene 3	20	129	65	83	98	60	133
Scene 4	174	313	299	315	449	287	428
Scene 5	64	241	234	235	314	73	516
Scene 6	19	29	45	86	73	88	94
Scene 7	9	8	14	37	29	22	30
Summary	534	1458	1385	1476	1869	1155	2383

, were used to demonstrate the key-points detected after the application of the different algorithms.

From Table 7, it can be observed that the proposed algorithm ranked second in terms of the number of detections in the two scenes and first in five scenes and overall. Moreover, the total number of detected key-points exceeded that of the other compared algorithms by an order of magnitude. In harsh underwater imaging environments, the proposed algorithm significantly enhances the image quality, resulting in approximately a 4.46 times increase in the number of detected key-points compared to the original images. SIFT is commonly used for feature matching scenarios.

Next, the RANSAC algorithm was used to filter feature points for image pairs with the same features and then SIFT feature matching was performed. Figure 15 shows the SIFT feature matching results for a pair of original underwater images and a pair of images enhanced using the proposed algorithm. From the feature matching comparison, it can be seen that the original images had 102 matches, while the enhanced images had 153 matches. Therefore, the proposed algorithm achieved a significant enhancement of the features in the underwater images, which is a key step for underwater imaging applications.

5. Conclusions

This paper focused on the enhancement of images obtained in harsh underwater environments. Due to the limited color correction information contained in the reference images in different datasets, the feature learning ability of style transfer algorithms is inhibited in UIE applications. To address this, traditional image processing methods were introduced to pre-process to reduce the significant color distortion of features in the underwater image dataset, thus enhancing the algorithm’s ability of scene transfer learning. Additionally, an adversarial learning algorithm was proposed with high-dimensional semantic cycle loss to tackle issues such as blurriness, color distortion, and brightness artifacts in harsh underwater environments. Based on this algorithm, corresponding DL network structures were designed, which improved the algorithm’s generalization ability and feature restoration accuracy across multiple scene categories during training. As evidenced by the multidimensional experiments presented, the proposed algorithm exhibits excellent performance in both subjective and objective evaluation tests, as well as in practical visual applications.