AMSMC-UGAN: Adaptive Multi-Scale Multi-Color Space Underwater Image Enhancement with GAN-Physics Fusion

Abstract

Underwater vision technology is crucial for marine exploration, aquaculture, and environmental monitoring. However, the challenging underwater conditions, including light attenuation, color distortion, reduced contrast, and blurring, pose difficulties. Current deep learning models and traditional image enhancement techniques are limited in addressing these challenges, making it challenging to acquire high-quality underwater image signals. To overcome these limitations, this study proposes an approach called adaptive multi-scale multi-color space underwater image enhancement with GAN-physics fusion (AMSMC-UGAN). AMSMC-UGAN leverages multiple color spaces (RGB, HSV, and Lab) for feature extraction, compensating for RGB’s limitations in underwater environments and enhancing the use of image information. By integrating a membership degree function to guide deep learning based on physical models, the model’s performance is improved across different underwater scenes. In addition, the introduction of a multi-scale feature extraction module deepens the granularity of image information, learns the degradation distribution of different image information of the same image content more comprehensively, and provides useful guidance for more comprehensive data for image enhancement. AMSMC-UGAN achieved maximum scores of 26.04 dB, 0.87, and 3.2004 for PSNR, SSIM, and UIQM metrics, respectively, on real and synthetic underwater image datasets. Additionally, it obtained gains of at least 6.5%, 6%, and 1% for these metrics. Empirical evaluations on real and artificially distorted underwater image datasets demonstrate that AMSMC-GAN outperforms existing techniques, showcasing superior performance with enhanced quantitative metrics and strong generalization capabilities.

1. Introduction

In the 21st century, underwater image enhancement technology has found extensive applications in marine exploration, research, and monitoring, providing crucial support for resource development, ecological conservation, scientific inquiry, and security defense. However, due to adverse underwater conditions, factors such as light absorption, scattering, color attenuation, flow, and turbulence result in uneven brightness, low contrast, blurring, and distortion in the images, hindering the acquisition of high-quality underwater imagery. In recent years, conventional and deep learning-based techniques for enhancing underwater imagery have seen significant advancements, demonstrating a degree of maturity in addressing the inherent low-contrast issues in underwater images. However, these methods continue to face constraints imposed by the intricate underwater environment, thereby exhibiting limited adaptability. Moreover, they demand substantial resources in terms of data and computational power, contributing to information degradation concerns. Addressing these perceptual limitations is of paramount importance, necessitating the advancement of top-tier underwater image enhancement methodologies aimed at optimizing the acquisition of superior image signals in submerged settings. To enhance underwater image quality, this study put forwards an adaptive multi-scale multi-color space underwater image enhancement method, referred to as AMSMC-UGAN. This approach integrates generative adversarial networks (GANs) with physical models to discern the mapping between distorted underwater images and their high-quality counterparts. The primary contributions of this study encompass the following.

• This study employs RGB, HSV, and Lab color spaces for cross-color-space feature extraction, specifically addressing the limitations of using RGB alone in underwater settings. By transforming RGB images into additional color space types and integrating all three color spaces as input to the network, these compensatory measures effectively liberate the color limitation that exists in a single color space.
• This study introduces a novel approach involving the adaptive selection of diverse physical models enabled by a membership degree function based on evaluation metrics. These models leverage prior image information to guide deep learning and integrate information from multiple color spaces at the input data level. This innovative approach significantly enhances image quality, thereby improving the generalization performance of underwater image enhancement models across diverse scenarios.
• This study presents a novel multi-scale feature module aimed at refining the granularity of image information to enhance image details. Operating at three scales, this module extracts features tailored to each scale. By integrating diverse image details within each scale, it facilitates a more comprehensive understanding of degradation patterns and effectively separates degradation from image content. This approach enables a thorough analysis of degradation distribution related to various image details while preserving the integrity of the original image content.

The study’s contributions include utilizing RGB, HSV, and Lab color spaces for cross-color-space feature extraction and introducing adaptive selection of different physical models and a multi-scale feature module to enhance underwater images. These advancements have significantly improved the performance of underwater image enhancement models. However, there are still future research and improvement directions. Future research can explore the following application areas: applying this method to underwater robotics technology, enhancing visualization in marine biology research, preprocessing for image retrieval [1], and improving image quality in underwater monitoring systems. These application studies are expected to further drive the development of the underwater image enhancement field, providing better solutions for practical applications.

2. Related Work

In underwater image enhancement, there are two main approaches: conventional methods and deep learning-driven methods. Conventional methods can be further divided into non-physics-based and physics-based enhancements.

2.1. Non-Physics Module-Based Methods

Methods for enhancing underwater images that rely on non-physical models often overlook the unique degradation mechanisms specific to underwater imagery. These methods primarily involve image processing in either the spatial or frequency domain to improve contrast or correct colors. Common approaches include enhanced algorithms using histogram equalization. Ghani et al. [2] introduced an adaptive histogram enhancement method using Rayleigh stretch for contrast improvement, enhancing contrast, refining details, and reducing issues like over-enhancement, oversaturation, and noise. Li et al. [3] proposed an integrated approach for underwater image enhancement by combining an improved underwater white balance algorithm with histogram stretching, demonstrating superior outcomes in color correction, haze reduction, and detail enhancement. However, these methods have not accounted for image degradation factors. Directly processing underwater images may lead to problems such as color distortion and incomplete dehazing.

2.2. Physics Module-Based Methods

The image restoration method based on a physics-based model utilizes the underwater scattering model as its foundation, transforming image enhancement into an estimation of model parameters. One of the most classic underwater scattering models [4,5] can be described as follows:

$$E_c=J_c\left(x\right)+T_c\left(x,+A_4-T_c\left(x\right)\right),T_c\left(x\right)={\mathrm{e}}^{a\mathrm{d}\left(x\right)}$$

(1)

In the equation,$C\in \left\{R,G,B\right\}$represents the three channels of a standard RGB image,$E_c\left(x\right)$ is the underwater image to be restored, $J_c\left(x\right)$ is a clear aerial image,$A$ is ambient light,$T_c\left(x\right)$ represents the transmittance corresponding to each pixel, and $\alpha$ is the scattering coefficient.

He et al. [6] introduced an image restoration method based on the underwater imaging model known as the dark channel prior (DCP) algorithm. Li et al. [7] proposed a technique employing red channel correction and blue–green channel dehazing, utilizing a gray-world algorithm for red channel color correction and adaptive exposure for handling overexposed and underexposed regions, thus enhancing visibility and contrast. Han et al. [8] accounted for backscattering effects during imaging, altering the light source to mitigate scattering and acquiring orthogonal polarized images. They introduced a polarization-based point spread estimation method. Meng et al. [9] integrated color balance and volume methods into underwater image enhancement, recovering images using color balance when the red channel value approximates the blue channel. The maximum a posteriori (MAP)-based restoration using DCP and a sharpening-based approach reduces blurriness, enhancing visibility and foreground texture preservation. While this category of methods has achieved some efficacy, accurate restoration of clear underwater images proves challenging in the absence of valid prior assumptions.

2.3. Deep Learning-Based Methods

The underwater image enhancement technique based on deep learning involves training convolutional neural networks (CNNs) and generative adversarial networks (GANs) on extensive datasets to predict clear underwater images. Perez et al. [10] initiated a CNN-based model for enhancing underwater images, creating a dataset of degraded and clear image pairs to establish the mapping using deep learning. Yang et al. [11] proposed an end-to-end neural model with color correction and adaptive refinement, resulting in high-quality enhancements in details, contrast, and color. However, it struggles with style and texture preservation. Another approach by Yang et al. [12] utilized GANs to generate clear underwater images using a multi-scale structure and dual discriminators to capture local and global semantic information. Wu et al. [13] introduced fusion water-GAN (FW-GAN), refining and encoding original images using multiple inputs and achieving satisfying enhancement results through multi-scale fusion features and an attention mechanism. These end-to-end methods can learn mapping without relying on physical models, but face challenges like limited receptive fields and loss of structural information. Enhancing the network’s ability to aggregate features from various scales is crucial.

2.4. Literature Analysis

In conclusion, within this section, a comprehensive literature analysis will be conducted, with a particular focus on reviewing the recent contributions and limitations of advancements in the field of underwater image enhancement in recent years, as depicted in

Table 1. Contributions and limitations of past research in the field of underwater image enhancement.

Methods	Author	Algorithm	Limitations
Non-physics module-based	Ghani et al.	Adaptive histogram equalization based on Rayleigh stretching	The Rayleigh distribution is not universally suitable for all underwater imaging conditions.
	Li et al.	Hybrid approach for enhanced underwater image enhancement	In challenging underwater conditions, excessive amplification of noise and artifacts.
Physics module-based	He et al.	Restoring images using the underwater imaging model—dark channel prior (DCP) algorithm	In challenging underwater conditions, recovery may yield unsatisfactory results with potential for artifacts.
	Li et al.	A method using red channel correction and blue–green channel dehazing	This approach may not fully adapt to unstable underwater conditions regarding light intensity and color distribution.
	Han et al.	A polarization-based point spread estimation method	A polarization-based point spread estimation method. Light direction and surface properties affect accuracy in complex conditions.
	Meng et al.	Hybrid underwater image restoration through color correction and sharpening	Possible noise and artifact overemphasis during enhancement.
Deep learning-based	Perez et al.	A CNN-based underwater image enhancement model	Restricted adaptability to different underwater conditions and varied image quality.
	Yang et al.	A trainable end-to-end neural model	It struggles with overall style and texture preservation.
	Yang et al.	A GAN-based underwater image enhancement method	Higher complexity, hinders real-time efficiency.
	Wu et al.	A multi-scale fusion generative adversarial network	Prior knowledge may limit enhancement in challenging underwater scenarios.

.

Through an in-depth literature analysis, it is evident that both traditional underwater image methods and pure deep learning approaches currently face three significant challenges.

• The limitation of a single color space in underwater environments, failing to fully capture complex color features and affecting the effectiveness of image processing.
• Image quality variations and feature uncertainties across diverse underwater scenes, making it difficult for a single model to adapt to this diversity effectively.
• The diverse degradation phenomena in underwater images, posing challenges in accurately modeling the degradation distribution for different image information.

A thorough literature analysis reveals significant challenges faced by both traditional underwater image methods and pure deep learning approaches, including limitations of a single color space, variations in image quality and feature uncertainty across different underwater scenes, and various degradation phenomena present in underwater images. Multi-color spaces possess color characteristics that single-color spaces lack. Extracting features from multiple color spaces effectively overcomes the constraints of single-color spaces, enabling models to address color deviation issues more effectively. Adaptive physical model integration with deep learning model decisions simultaneously offers good scene adaptability and automatic learning features of image content. By integrating physical models and deep learning models, the model achieves better generalization performance on images obtained from different underwater scenes. Multi-scale feature extraction, characterized by extracting features with different distributions and characteristics at different scales, effectively models random degradation factors, thereby aiding in comprehensively learning degradation distributions of different image information under the same content. Therefore, the proposed AMSMC-UGAN method, through the incorporation of three modules—multi-color space, adaptive physical model integration with deep learning model decisions, and multi-scale feature extraction—provides outstanding solutions to the challenges identified in the literature analysis above.

3. Materials and Methods

Inspired by the underwater image enhancement method in recent years, this study addresses some problems of the findings of Section 2.4, which will be elaborated on in Section 3.

3.1. Network Architecture

AMSMC-UGAN, illustrated in Figure 1, employs an encoder–decoder generator. The input image undergoes restoration based on a physical model and a conversion to Lab and HSV color spaces, generating respective outputs. The network selectively applies physical model-based restoration algorithms, including total variation [14], dark channel prior [6],contrast improvement based on inverse filtering [15], and resolution reconstruction via sparse representation [16], then the underwater depth estimation parameters of the image were estimated by the underwater physical model and transmitted to the multi-scale module of the image as a fusion factor of the deep learning model, as shown in line 23 of Algorithm 1, to guide and constrain the deep model’s output. Next, the multi-scale module processes various color spaces and employs attention mechanisms to enhance critical features, significantly improving the model’s performance. The output undergoes up-sampling and reconstruction using deconvolution layers.

The AMSMC-UGAN discriminator is modeled as PatchGAN [17]. The input of the discriminator block is a pair consisting of the reconstructed image generated by the generator and the original image. The output of the discriminator block is a discrimination result, which measures the degree of difference between the generated reconstructed image and the original image. The definition of the AMSMC-UGAN is elaborated in Algorithm 1.

Algorithm 1: Definition process of AMSMC-UGAN

1: function underwater transmission estimator (x):

2:   Initialize a parameter controlling the effect of the atmosphere. $\omega$

3:   $TransmissionEstimator(observed\_image,recovered\_image)=1-\omega \left(\frac{recovered\_image}{max(observed\_image)}\right)$

4: function multi-scale-module (x):

5:   $scale1\left(x\right)=ReLU\left(BN\right(Conv2D(x,3\to \mathrm{64,3}\times \mathrm{3,1},1))\times 3$

6:   $scale2\left(x\right)=ReLU\left(BN\right(Conv2D(x,3\to \mathrm{64,5}\times \mathrm{5,1},2))\times 3$

7:   $scale3\left(x\right)=ReLU\left(BN\right(Conv2D(x,3\to \mathrm{64,7}\times \mathrm{7,1},3))\times 3$

8: function RGB to HSV (x):

9:   Initialize the maximum value and minimum value among r,g,b. $maxc$ $\mathrm{m}\mathrm{i}\mathrm{n}c$

10: Initialize the difference between $maxc$ and $minc$. $∆c$

11: $RGBtoHSV(rgb)=\left[\begin{array}{c}H\\ S\\ V\end{array}\right],H=\left\{\begin{array}{c}60°\times \left(\frac{g-b}{∆c}\right),ifmaxc=r\\ 60°\times \left(2+\frac{b-r}{∆c}\right),ifmaxc=g\\ 60°\times \left(4+\frac{r-g}{∆c}\right),ifmaxc=b\end{array}\right.,S=\frac{∆c}{maxc},V=maxc$

12: function RGBtoLab (x):

13: $normalized\_rgb=x/255.0$,

14: $linear\_rgb=\left\{\begin{array}{c}{\left(\frac{{rgb}_{normalized}+0.55}{1.055}\right)}^{2.4},if{rgb}_{normalized}>0.04045\\ \frac{{rgb}_{normalized}}{12.92},otherwise\end{array}\right.$

15: switch to XYZ and normalize XYZ, obtain $f_x,{\mathrm{f}}_{\mathrm{y}}{\mathrm{f}}_z$

16: switch to Lab

$\mathrm{L}=\left\{\begin{array}{c}116\times {\mathrm{f}}_{\mathrm{y}}-16,\mathrm{if}\mathrm{Y}>0.008856451679035631\\ 0.9032962962962963\times \mathrm{Y},\mathrm{otherwise}\end{array}\right.,\mathrm{a}=500\times \left(f_x-{\mathrm{f}}_{\mathrm{y}}\right),\mathrm{b}=200\times \left(f_y-{\mathrm{f}}_z\right)$

17: $RGBtoLab\left(rgb\right)=stack(L,a,b)$

18: function AMSMC-UGAN (x):

19: Input raw picture data $X$ and ground-truth picture $Y$

20: Calculate PSNR:

$PSNR(X,Y)=10\times {log}_{10}\left(\frac{{255}^2}{MSE}\right)$

21: Defining image quality grades

$Grade=\left\{\begin{array}{c}“poor,”if0<PSNR<15\\ “fair,”if10<PSNR<20\\ “average,”if15<PSNR<30\\ “good,”if25<PSNR<40\end{array}\right.$

22: Selecting an underwater restore model to obtain recovered image

$recovered\_image=\left\{\begin{array}{c}TV\left(X\right),ifGrade=“poor”\\ CED\left(X\right),ifGrade=“fair”\\ DCP\left(X\right),ifGrade=“average”\\ SR\left(X\right),ifGrade=“good”\end{array}\right.$

23: obtain underwater physical estimation:

$Pe=TransmissionEstimator(X,recovered\_image)$

24: Initialize $Contr5$ and $CBAM$

$Generatordecoder\left(x\right)=Contr5\left(Contr4\right(Contr3\left(Contr2\right(Contr1\left(x\right)\left)\right))$

$Generatorencoder(x)=CBAM\left[cat\left(\genfrac{}{}{0pt}{}{scale1\left(rgb\right)}{\begin{array}{c}scale1\left(hsv\right)\\ scale1\left(lab\right)\\ scale1\left(Pe\right)\end{array}}\right),cat\left(\genfrac{}{}{0pt}{}{scale2\left(rgb\right)}{\begin{array}{c}scale2\left(hsv\right)\\ scale2\left(lab\right)\\ scale2\left(Pe\right)\end{array}}\right),cat\left(\genfrac{}{}{0pt}{}{scale3\left(rgb\right)}{\begin{array}{c}scale3\left(hsv\right)\\ scale3\left(lab\right)\\ scale3\left(Pe\right)\end{array}}\right)\right]$

$Generator\left(x\right)=Generatordecoder\left(Generatorencoder\right(x\left)\right)$

Initialize different discriminator blocks. $DB1,DB2,DB3,DB4$

$Discriminator\left(x\right)=divide\_into\_patches\left(DB4\right(DB3\left(DB2\right(DB1\left(Generator\right(x)+Y)\left)\right)),16\times 16)$

3.2. Generator

The generator comprises an encoder–decoder structure. Next, the components of the encoder and decoder will be detailed.

3.2.1. Encoder

The encoder is composed of an adaptive decision enhancement module, a multi-color-space module, and a multi-scale feature extraction-fusion module.

Adapting process Figure 2 shows the strategy of the physical model of an adaptive deep learning network for image enhancement.

To mitigate excessive enhancement and ensure the appropriate selection of physical models, this study use membership degree functions to adaptively choose these physical models to fuse with GAN.

In this study, image quality is defined into four levels.

Poor image quality typically indicates severe distortion between the processed image and the original image, with PSNR values in the range of (0, 15).

Fair image quality is often attributed to low brightness and contrast in the image, resulting in a certain degree of distortion between the processed image and the original image, indicating relatively poor image quality, with PSNR values in the range of (10, 25).

When image quality is average, it indicates relatively good image quality, approaching the level of imperceptible distortion, yet factors affecting the PSNR value still exist, such as haze, with PSNR values in the range of (15, 30).

When image quality is good, it indicates very high image quality, approaching the quality of the original image, with PSNR values in the range of (25, 40).

As expressed in Algorithm 1 lines 21 and 22, the image quality levels were determined by PSNR value calculation. Four different physical models were selected based on the quality assessment:

• “Poor”-quality images employ total variation (TV) reconstruction, ideal for denoising and reconstruction purposes.
• “Fair”-quality images utilize CED (inverse filtering-based image contrast restoration physical model) to address contrast issues.
• “Average”-quality images make use of dark channel prior (DCP) for haze removal and detail enhancement.
• “Good”-quality images utilize sparse representation reconstruction, enhancing resolution and detail.

The fuzzy logic toolbox defines membership functions for fuzzy sets representing quality levels (“poor”, “fair”, “average”, “good”, “excellent”). Adjusting the physical model and implementing deep learning techniques based on these quality ratings prevents excessive enhancement.

Multi-color space module The approach in this study employs a multi-color space encoder. It densely connects features extracted from the RGB channel with those from the HSV and Lab channels, thereby providing specific image characteristics for each color space, enriching various aspects of image information.

Figure 3a–c showcases visual representations of the channels extracted from different color spaces. These visuals offer an intuitive display of how different color spaces impact image processing in underwater settings, highlighting the limitations of the RGB channel in terms of color distortion and reduced contrast.

Multi-scale feature extraction fusion module This study inputs images processed through three color spaces and a physical model into a multi-scale feature extraction module for feature extraction. Figure 4 illustrates the structure of the multi-scale feature extraction module, where scale1, scale2, and scale3 correspond to feature extraction at large, medium, and small scales respectively. Each scale extraction block consists of three convolutional layers (Conv2d), differing in kernel size, stride, and padding parameters. Specifically, all Conv2d layers in scale1 have a kernel size of 3 × 3, stride of 1, and padding of 1; all Conv2d layers in scale2 have a kernel size of 5 × 5, stride of 1, and padding of 2; and all Conv2d layers in scale3 have a kernel size of 7 × 7, stride of 1, and padding of 3. Additionally, the number of channels for each scale changes progressively from an initial 3 channels to 16 channels, then to 32 channels, and finally to 64 channels. This multi-scale feature extraction method enables the extraction of different hierarchical information from different scale features, providing a richer feature representation for subsequent processing and analysis. The variation in kernel size and channel number at different scales helps capture features of different scales and levels, enhancing the model’s capability and diversity in expressing image features. Unlike other methods, the multi-scale feature extraction module used in this study preserves information integrity and does not employ pooling layers for down-sampling. This design decision ensures that the size of output feature maps at different scales is 256 × 256 × 64, avoiding the need for up-sampling during the fusion stage. This approach prevents the loss of feature information and effectively utilizes image information for multi-scale feature extraction in image enhancement tasks. By not using pooling layers for down-sampling, the multi-scale feature extraction module retains more detailed information, enhancing feature precision and integrity. The output feature maps’ size of 256 × 256 × 64 indicates that throughout the feature extraction process, both spatial dimensions and channel numbers of the images are adequately preserved and expressed. Therefore, in subsequent image enhancement tasks, these feature maps of different scales can be comprehensively analyzed and processed to improve image quality and detail expression. Moreover, by eliminating the need for up-sampling steps, computational burden is reduced and operational efficiency is increased, making the multi-scale feature extraction module more feasible and effective in practical applications. To determine the number of layers corresponding to each scale in the multi-scale module, this study conducted validation experiments. As shown in the line chart in Figure 4, it can be observed that when the number of layers corresponding to each scale in the multi-scale module is set to 3, the image generation achieves optimal results. This holds true for both the UIEB and EUVP datasets, as the PSNR evaluation scores are highest when the number of layers is set to 3.

RGB color space is processed using a physical model. The output, along with data from other color spaces, is fed into a multi-scale feature extraction module. Each scale extraction block captures diverse image features, enhancing the model’s expressiveness. Distinct image features at various scales are integrated to obtain fused feature representation. (See Figure 4 and Figure 5 for visual representation).

3.2.2. Decoder

The attention mechanism’s output is up-sampled via deconvolution layers, integrating multi-scale features for enriched representation. This deepens the information granularity of the image, improving spatial awareness by emphasizing both local details and global context. The final result is obtained through five layers of deconvolution, ensuring spatial consistency.

Figure 6 illustrates the fusion of features at different scales, which are then entered into the channel spatial attention module (CBAM) [18] before undergoing the decoding process. Firstly, at the stage of fusion between the physical model and deep learning features, the output of the physical model is concatenated with multi-scale features extracted from different color spaces (HSV, RGB, Lab) at various scales. At each scale, the concatenated features result in feature maps of size 256 × H × W, where H and W represent the height and width of the feature maps, respectively. Secondly, at the input attention mechanism (CBAM) stage, CBAM operations are performed on the concatenated features at each scale to enhance feature representation. Following CBAM, the size of the feature maps at each scale remains 256 × H × W. Lastly, at the up-sampling stage, spatial dimensions of the feature maps are progressively increased using transposed convolutional layers. After the first transposed convolution (convtr1), the size of the feature maps becomes 128 × 2 H × 2 W, due to the doubling of height and width. In the second transposed convolution (convtr2), the size of the feature maps is 64 × 4 H × 4 W, achieved by concatenating with fused features from the preceding scale. Similarly, after the third transposed convolution (convtr3), the size of the feature maps becomes 32 × 8 H × 8 W, and after the final transposed convolution (convtr4), the size is 16 × 16 H × 16 W. Ultimately, the fifth transposed convolution (convtr5) yields the final output with the desired number of output channels.

3.3. Discriminator

The AMSMC-UGAN discriminator scores input image blocks to determine authenticity. Each element in the final output matrix actually represents a relatively large receptive field in the original image, which corresponds to a patch in the original image. The patch (as illustrated in Figure 7) structure preserves high-frequency info and is computationally efficient. The discrimination block (Figure 7) contains a 4 × 4 convolution layer, batch normalization (BN layer), spectral normalization (SN layer) [19], and leaky ReLU activation. SN ensures k-Lipschitz condition, stabilizing GAN training, and balancing of generator and discriminator. It controls gradient norm, improving stability.

3.4. Objective Function Formulation

In order to improve the visual, structural, and perceptual resemblance of generated images to real images within AMSMC-UGAN, four types of loss functions have been employed for training. This approach enables the evaluation of the quality of enhanced images in this study The objective function is denoted as follows:

$$L_{total}=L_D+L_G+{\lambda }_e{L}_{edge}+{\lambda }_1{L}_1+{\lambda }_s{L}_{SSIM}$$

(2)

In the equation, $L_D$ is the loss of the discriminator D, $L_G$ is the loss of the generator G, $L_{edge}$ is the edge loss [20], $L_1$is the L1 distance loss, $L_{SSIM}$is the SSIM loss [21], ${\lambda }_e$, ${\lambda }_1$, and ${\lambda }_s$ correspond to the weights of the edge loss function, L1 distance loss function, and similarity loss function, respectively. These functions are defined in detail below.

Discriminator D loss: The overall loss of the discriminator network$L_{D }$is composed of two components:$loss\_real$ and $loss\_fake$. The$loss\_real$ represents the adversarial loss for real samples. On the other hand, the $loss\_fake$ represents the adversarial loss for generated samples.

$$L_D=0.5\left(los{s}_{real}+los{s}_{f}ake\right)\times 10$$

(3)

The processing approach aims to balance real and generated samples to train the discriminator to accurately distinguish between them, providing useful feedback for the generator.

Generator G loss: The total loss of the generator network $L_G$ is obtained by summing three loss terms: $loss\_GAN$, 7 times of the $loss\_1$, and 3 times of the $loss\_con$. The $loss\_GAN$ is computed by calculating the mean squared error between the discriminator’s output result $D\left(G\right(X\left)\right)$ and the labels marked as real samples—$pred\_real$. The $loss\_1$ is computed by calculating the L1 distance loss between the generator’s output $G\left(X\right)$ and the real reference data $Y$. The $loss\_con$ is computed by applying the VGG19 pretrained loss function to the generator’s output $G\left(X\right)$ and the real reference data $Y$. The loss function $L_G$ is represented as:

$$L_G=los{s}_{GAN}+7\times los{s}_1+3\times los{s}_{c}on$$

(4)

The varied loss terms guide generator training to meet adversarial objectives, minimize L1 differences with the target, and capture content features.

Edge loss: Underwater images frequently experience high-frequency information loss. The edge loss is employed to constrain the high-frequency components between the real and generated enhanced images, aiming to retain the high-frequency information.

$$L_{edge}=\sqrt{{\left(\nabla Y_x-\nabla G{\left(x\right)}_x\right)}^2+{\left(\nabla Y_y-\nabla G{\left(x\right)}_y\right)}^2}$$

(5)

In this equation, let $Y$ represent the target image, $G\left(X\right)$ denote the output image from the generator, $\nabla Y\_x$ and $\nabla Y\_y$ represent the horizontal and vertical gradients of the target image, respectively, and $\nabla G\left(x\right)\_x$ and $\nabla G\left(x\right)\_y$ represent the horizontal and vertical gradients of the generated output image. The edge loss is computed using the squared differences of target and generated images’ gradients.

L1 loss: This study employs the $L_1$ distance to supervise the model in generating results that are globally similar to the reference image. The $L_1$ loss function is represented as:

$$L_1=E\left[{\left|Y-G\left(x\right)\right|}_1\right]$$

(6)

SSIM loss: this study utilizes the SSIM loss to constrain the brightness, contrast, and structure between the generated images and the ground-truth real images:

$$SSIM\left(x,y\right)=\left(\frac{{2\mu }_x{\mu }_y+c_1}{{\mu }_x^2+{\mu }_y^2+c_1}\right)\left(\frac{{2\sigma }_{xy}+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}\right)$$

(7)

where ${\mu }_x\left({\mu }_y\right)$ represents the mean value, ${\sigma }_x^2\left({\sigma }_y^2\right)$ represents the variance of $x\left(y\right)$, and ${\sigma }_{xy}$ refers to the covariance of $x\left(y\right)$, In addition, $c_1={(255\times 0.01)}^2$ and $c_2={(255\times 0.03)}^2$ are set as two constants to avoid division by zero. The SSIM loss is computed using the SSIM values. The SSIM loss function is represented as:

$$L_{SSIM}=E\left[1-SSIM\left(Y-G\left(x\right)\right)\right]$$

(8)

Here, let $Y$ represent the target image and $G\left(X\right)$ denote the output image from the generator. Finally, this study sets the weight for each loss function based on the literature: ${\lambda }_e=0.5$, ${\lambda }_1=0.1$, ${\lambda }_s=0.1$.

4. Experiments

4.1. Implementation Details

This study implemented AMSMC-UGAN using the PyTorch library in Python on an NVIDIA GeForce RTX 4090. The GPU has a memory capacity of 24 GB. The software versions used were Python 3.8 and Torch 1.13.1. Training used 13,920 EUVP [22] and 890 UIEB [23] paired samples. Data were split for validation/testing. Images are resized to 256 × 256. Layer filter weights were initialized from a normal distribution; Adam optimizer with LR 0.003, batch size 16; model trained for 200 epochs to obtain the final version.

4.2. Experiment Settings

4.2.1. Dataset Splitting

This study employed two datasets: enhancing underwater visual perception (EUVP) with 13,920 samples and the underwater image evaluation benchmark (UIEB) with 890 images. Due to the larger quantity and wider variability of the synthetic dataset, more samples were allocated to the training set to enhance the model’s understanding of underwater environment characteristics. Validation and test sets received smaller allocations to evaluate performance under synthetic underwater conditions. The real underwater dataset, though smaller, was more representative and received fewer samples in the training set. Larger allocations for validation and test sets facilitated performance evaluation under real underwater conditions, improving the model’s generalization to real-world scenarios. The dataset is divided as in

Table 2. Dataset setting.

Dataset	Total Images	Training Set	Validation Set	Test Set
EUVP	13,920	9600 (70%)	1920 (14%)	2400 (16%)
UIEB	890	600 (67%)	120 (13%)	170 (20%)
Test Subset	Description
Test-EP1000	Paired synthetic underwater images with ground-truth real images (2400)
Test-UP100	Paired real underwater images with ground-truth real images (170)

.

4.2.2. Comparison Method

In this study, AMSMC-UGAN is compared with five image enhancement methods. This study compared five UIE methods: UGAN [24], FUnie-GAN [22], Ucolor [25], U-shape [26], and SyrealNet [27].

4.2.3. Evaluation Metrics

The primary image quality evaluation metrics used were PSNR and SSIM, which were quantitatively compared with other methods at both pixel and structural levels. Higher PSNR and SSIM values indicate better image quality. Additionally, UIQM was employed as an evaluation metric, considering brightness, contrast, and structure. Higher UIQM values indicate good perceptual effects in different image scenarios.

4.3. Visual Comparison

Table 3. Quantitative comparison of UIEB dataset and EUVP dataset. ↑ indicates higher values are preferable.

Methods	UIEB			EUVP
Name	PSNR ↑	SSIM ↑	UIQM ↑	PSNR ↑	SSIM ↑
UGAN	19.85	0.72	3.1686	24.80	0.76
FUnie-GAN	19.18	0.66	3.1162	25.87	0.77
Ucolor	20.81	0.82	2.9426	22.86	0.75
U-shape	17.18	0.63	2.9004	24.91	0.77
SyrealNet	18.60	0.78	3.1439	20.71	0.75
AMSMC-UGAN	23.66	0.87	3.2004	26.04	0.77

presents the quantitative results of comparing different UIE algorithms on the UIEB dataset and the EUVP dataset, including UGAN, FUnie-GAN, Ucolor, U-shape, and SyrealNet. This study primarily employed PSNR, SSIM, and UIQM as evaluation metrics for the UIEB dataset, and PSNR and SSIM for the EUVP dataset.

The results in Table 3 indicate that the AMSMC-UGAN algorithm achieves state-of-the-art performance on three full-reference image quality assessment metrics for UIE tasks, significantly outperforming several other UIE methods, thereby validating the robustness of the proposed AMSMC-UGAN in underwater image enhancement. Additionally, visual comparisons with other UIE methods are demonstrated separately on the UIEB dataset and the EUVP dataset in Figure 8 and Figure 9.

This study has reproduced all the comparison methods mentioned, maintaining their original hyperparameter settings to ensure optimal training effectiveness. The optimizers used for SyrealNet, UGAN, and FUnie-GAN are all Adam, with a learning rate set to 0.001. Ucolor employs the AdamW optimizer with a learning rate of 1 × 10⁻⁶ (10⁻⁶), while U-shape uses SGD with a learning rate of 0.1. AMSMC-UGAN utilizes the Adam optimizer with a learning rate of 0.0001. This setting aids in accelerating convergence, enhancing training stability, and mitigating issues such as gradient explosion and vanishing gradients. Additionally, the lower learning rate facilitates fine-tuning of the optimization process, preventing the oversight of optimal solutions.

Figure 8 depicts five randomly selected cases from the UIEB dataset, while Figure 9 showcases five randomly selected cases from the EUVP dataset. In Figure 8, the proposed method (AMSMC-UGAN) was visually compared with other underwater image enhancement (UIE) methods using real underwater image data. The method proposed in this study demonstrates certain advantages in visual presentation. For instance, as observed in Figure 8a–e, the visual results of UGAN, FUnie-GAN, U-color, U-shape, and Syeanet still exhibit issues such as haze, color deviation, and low contrast. In contrast, the proposed method (AMSMC-UGAN) effectively addresses these issues related to haze, color deviation, and low contrast. Similarly, in Figure 9, the proposed method was visually compared with other UIE methods using synthetic real image data. The method proposed in this study continues to demonstrate certain advantages in visual presentation. For example, Figure 9b,e reveal that the mentioned comparative methods fail to handle the deviations in green and yellow colors effectively in synthetic images. Conversely, the proposed method (AMSMC-UGAN) addresses the issues of green and yellow color deviations, ensuring that the images better represent the conditions in the air and achieve the effect of eliminating color deviations. Furthermore, in Figure 9a,d,c, it is evident that AMSMC-UGAN preserves more details of the images and approaches closer to the paired reference images. The results obtained from AMSMC-UGAN consistently demonstrate visual superiority and a more natural appearance in test images. This suggests strong indications of the robust generalization capabilities of AMSMC-UGAN for real-world applications.

In order to better verify the effectiveness of AMSMC-UGAN method in detail and color restoration, this study conducted RGB distribution tests and feature point matching tests, as shown in Figure 10. As shown in Figure 10a, after underwater image enhancement using AMSMC-UGAN, the RGB distribution of the output is close to that of the ground-truth reference image, which indicates that the AMSMC-UGAN method is effective in color correction for underwater images during underwater enhancement tasks. As shown in Figure 10b, comparing the feature point matching with the methods listed in Table 1, it can be seen that AMSMC-UGAN has the best feature matching performance, with the most feature points matched the ground-truth reference image. This also indicates that the AMSMC-UGAN method is effective in detail restoration for underwater images during underwater enhancement tasks.

4.4. Adaptive Control Verification Experiment

This experiment validates the adaptive control strategy for underwater image enhancement using the fuzzy membership function on various test datasets. By integrating different physical models with deep learning models for enhancement processing of the same underwater image, this experiment aimed to validate whether the integration of four selected physical models into deep learning models can improve image quality for various underwater conditions. These conditions include low contrast, haze, color deviation, blurred detail information, and low resolution. The validation is performed on data sampled from real underwater images (Test-UP100), as shown in Figure 11a–e.

In Figure 11a,b, (a) shows the original input image with low contrast, haze, and color deviation. The proposed adaptive image enhancement control strategy (Section 3) addresses these issues: (b) reconstructs (a), recovering its information; (c) improves contrast; (d) removes haze; and (e) enhances resolution. The experiments in Figure 11 demonstrate that by adaptively incorporating physical models into deep models, the image quality gradually improves when dealing with underwater images suffering from low contrast, haze, color deviation, blurred details, and low resolution, as evidenced by the results in

Table 4. Validation using data sampled from real underwater images (Test-UP100).

Methods	PSNR (dB) (Increase%)	SSIM (Increase%)
Raw	18.51	0.77
TV + Generator	26.22 (41.6%)	0.89 (15%)
CED + Generator	26.30 (42%)	0.89 (15%)
DCP + Generator	28.20 (52%)	0.87 (12%)
SR + Generator	31.55 (70%)	0.89 (15%)

. Compared to the initial images, the adaptive integration of physical models into deep models through AMSMC-UGAN leads to progressively increasing gains in the quality evaluation metric (PSNR), with improvements of 41.6%, 42%, 52%, and 70%, respectively. Real underwater image experiments (Test-UP100) validate this strategy’s effectiveness in improving contrast, haze, and color deviation.

The adaptive image enhancement control strategy effectively improves underwater image quality and avoids excessive or subtle enhancements. It also enhances the network’s generalization performance. Table 4 supports these conclusions.

4.5. Ablation Experiment

In order to evaluate the effectiveness of various key components within the network, this study conducted ablation experiments on both UIEB and EUVP, comparing the performance of each network in enhancing underwater images, as demonstrated in Figure 12.

The strategy involves removing key components from the original model to obtain five models lacking these critical components. As the key components gradually decrease, the image enhancement results of the network output are far inferior to those of the original network. Based on the performance on the UIEB and EUVP data shown in Figure 11, reducing the involvement of key components leads to color deviation issues (either biased towards blue or green) in underwater images, as well as problems such as blurred details and low contrast. The extent of image enhancement is far inferior to networks where all components are involved. This demonstrates the effectiveness of each component in underwater image enhancement. Additionally, this study continued to use PSNR and SSIM as evaluation metrics to score each network in the ablation experiment.

Table 5. Ablation experiments on the UIEB dataset and EUVP dataset. ↑ indicates higher values are better. Here, APM represents the adaptive physics encoder, and MC represents the multi-color space encoder. √ represents the existence of this component, × represents the absence of this component.

Methods					UIEB
Module	PM	MS	MCS	CBAM	PSNR ↑	SSIM ↑
1	√	√	√	√	23.66	0.87
2	×	√	√	√	22.01	0.81
3	×	×	√	√	21.64	0.80
4	×	×	×	√	21.15	0.80
5	×	×	×	×	20.61	0.77

and

Table 6. Ablation experiments on the UIEB dataset and EUVP dataset. ↑ denotes higher values are better. Here, APM represents the adaptive physics encoder, and MC represents the multi-color space encoder. √ represents the existence of this component, × represents the absence of this component.

Methods					EUVP
Module	PM	MS	MCS	CBAM	PSNR ↑	SSIM ↑
1	√	√	√	√	26.04	0.77
2	×	√	√	√	24.47	0.76
3	×	×	√	√	23.94	0.77
4	×	×	×	√	23.57	0.75
5	×	×	×	×	23.13	0.74

indicate that the key components this study designed are effective for underwater image enhancement tasks.

5. Conclusions

This paper introduces a novel underwater image enhancement model, leveraging multi-color space features and physically-based restoration. The model combines multi-scale feature extraction from various color spaces and integrates physical models to guide the enhancement process, aligning outputs with real-world characteristics. An adaptive decision control mechanism prevents over- or under-enhancement. Experimental results showcase the effectiveness and advantages of this approach, validating the importance of key components through ablation studies.