Enhancement of Underwater Images through Parallel Fusion of Transformer and CNN

Abstract

Ocean exploration is crucial for utilizing its extensive resources. Images captured by underwater robots suffer from issues such as color distortion and reduced contrast. To address the issue, an innovative enhancement algorithm is proposed, which integrates Transformer and Convolutional Neural Network (CNN) in a parallel fusion manner. Firstly, a novel transformer model is introduced to capture local features, employing peak-signal-to-noise ratio (PSNR) attention and linear operations. Subsequently, to extract global features, both temporal and frequency domain features are incorporated to construct the convolutional neural network. Finally, the image’s high and low frequency information are utilized to fuse different features. To demonstrate the algorithm’s effectiveness, underwater images with various levels of color distortion are selected for both qualitative and quantitative analyses. The experimental results demonstrate that our approach outperforms other mainstream methods, achieving superior PSNR and structural similarity index measure (SSIM) metrics and yielding a detection performance improvement of over ten percent.

1. Introduction

Exploration of the ocean is vital for harnessing its abundant resources [1]. Underwater robots are crucial instruments to explore the ocean, which enables image-based target detection tasks. Due to light attenuation and scattering in seawater, the quality of underwater images is often compromised (Figure 1). Consequently, underwater image processing faces significant challenges, and underwater image enhancement is a significant research area within the fields of computer vision and underwater robotics [2,3].

Underwater image enhancement aims to improve the quality of distorted images by restoring its color distortion [4]. Some scholars have explored non-deep learning approaches and have made some progress. Non-deep learning methods rely on statistical assumptions and models to enhance underwater images, such as Underwater Dark Channel Prior (UDCP) [5], Image Blur Recovery [6], and Underwater Light Attenuation Prior (ULAP) [7]. Cheng et al. [8] pointed out that the dissolved substances in water can weaken the imaging process, and influence the attenuation parameters of light propagation in water. Drews et al. proposed an underwater prior method by utilizing the red channel information [9]. Li et al. proposed an underwater light attenuation prior (ULAP) model to restore image quality [10]. Ma et al. designed a wavelet transform network that decomposes input images into frequency maps to enrich image details [11]. However, the complexity of underwater environments often leads to inaccuracies in parameter estimation for these methods.

Currently, neural networks have been widely employed in various visual tasks [12]. In contrast, extensive datasets and specialized loss functions have been utilized by deep learning techniques to train deep neural networks for image quality enhancement, including models like Underwater Residual Network (UResNet), Shallow Underwater Network (UWNet), and Underwater Convolutional Neural Network (UWCNN) [13]. Mean square error loss and edge difference loss are used to optimize convolutional neural networks for image enhancement [14]. By employing conventional convolutions, Naik et al. developed a network specifically for underwater image enhancement, which demonstrates effective enhancement capabilities on public datasets [15]. Li et al. introduced a residual network-based underwater image enhancement algorithm [16]. Chen et al. introduced an end-to-end neural network enhancement model that integrates residual structures and attention mechanisms [17]. Current enhancement algorithms predominantly rely on convolutional neural networks, but they often utilize a single feature-extraction backbone. However, the features extracted from these models are often insufficiently detailed.

Wang et al. utilized Generative Adversarial Networks (GANs) to design a feature enhancement network [18,19]. Moreover, the underwater generative adversarial networks (UGANs) [20] scheme has been developed for UIE tasks, employing an encoder–decoder structure [21,22] that effectively preserves rich semantic information. Junjun Wu et al. have developed a multi-scale fusion generative adversarial network named Fusion Water-GAN (FW-GAN), which aims to improve underwater image quality while effectively preserving rich semantic information. This network integrates four convolutional branches to achieve this goal [23]. Kei et al. created a dataset that includes both image and sonar data specifically designed for low-light underwater environments. They utilized a Generative Adversarial Network (GAN) to improve the image quality. Experimental results show that this method achieves better detection performance [24]. Zhang et al. collected images from different angles and then calculated the camera poses for each angle. They fed the collected image sequences and their corresponding poses into a Neural Radiance Field (NeRF), synthesizing new viewpoints and improving the effect of 3D image reconstruction [25]. Adversarial learning methods are mostly based on object detection with similar quality or visibility, and acquiring clear sample data for these models remains a formidable task.

Deep learning encompasses various backbone architectures, including the widely utilized Convolutional Neural Networks (CNNs) and transformers [26,27], which has gained popularity in computer vision tasks. The Transformer architecture incorporates features like multi-head mechanisms and multi-layer perceptions, making it versatile for a range of visual tasks [28]. Zamir et al. [29] employed an encoder_decoder structure to obtain features at different scales, achieving image enhancement in rainy and foggy weather conditions. Song et al. modified attention modules within the network layers, constructing a parameter-adjustable dehazing network [30]. Although the transformer architecture shows great potentiality for computer vision tasks, its high computational complexity often results in increased computational load and longer processing times.

Despite advancements in current methods for solving underwater image distortions, challenges persist in achieving high-quality restoration. To produce high-quality images, our research motivation is to overcome the uncertainty in generated images by utilizing the complementary features of different neural network frameworks. This paper introduces a new model for enhancing the quality of underwater images, aiming to tackle issues such as color shifts, unrealistic colors, and reduced contrast [31]. Furthermore, a matrix linear computation approach has been designed to minimize the computational delays caused by the stacked network. To this point, an innovative approach is proposed to extract both local and global image features. This network integrates visual transformer models with CNN networks to enhance the overall restoration process. Additionally, information fusion weights are calculated from the Fourier transform features of original images. The main advantages of our work are as followed:

(1)

A novel transformer model that extracts local features has been proposed. It incorporates both PSNR attention and linear operations to effectively reduce computational load and alleviate color artifacts.

(2)

Furthermore, a novel global feature extraction network is devised, which leverages both temporal and frequency domain characters to enrich image features.

(3)

Additionally, a feature fusion method leveraging Fourier transform has been proposed to optimize global and local feature weights. High-frequency Fourier components enhance the global features, while low-frequency components refine local features. This approach effectively integrates various features without leaving noticeable fusion trace at the boundaries.

2. Materials and Methods

Figure 2 depicts our detection framework. This network incorporates both CNN and transformer backbones, which is designed to extract both global and local features, respectively. These extracted features are fused at the smallest down-sampling size. Additionally, the low-frequency and high-frequency information of the original image is obtained via Fourier transform, serving as fusion weights for the extracted CNN and transformer features.

2.1. Two Branches’ Feature Extraction Network

Conventional low-light image enhancement networks usually employ convolutional structures within the feature layers, predominantly extracting image information from the image’s bright regions. These regions are rich in visible content or signals. However, in some non-prominent regions, fine-grained features may be lost, leading to reduced detection accuracy.

Based on this, two backbones for image enhancement are employed. Information from both global and local regions complements each other, and this differentiation can be determined based on the distribution of Fourier transforms on the image. On one hand, for image regions characterized by high-frequency Fourier transforms, their CNN features predominantly manifest in the globally salient information. On the other hand, in image regions corresponding to low-frequency Fourier transforms, the transformer predominantly captures local detailed information. This methodology facilitates the extraction of diverse image features.

As to the encoder framework, two different backbones for feature extraction were developed. One employs transformers, while the other utilizes convolutional structures. Each backbone incorporates a three-level and top-down feature pyramid extraction network. The input dimension of the first layer features is 6 dimensions, yielding 64 dimensions as output. Both the second and third levels’ feature extraction operations utilize 64 channels.

CNNs excel at extracting edges, textures, and simple shapes from images. On the other hand, transformers excel at identifying long-range dependencies and inferring local information. By integrating the local features extracted by transformers with the global textures identified by CNNs, richer and more diverse representations can be produced. This hybrid approach is more effective at handling noise and variations in data. Moreover, hybrid models can better adapt to different types of data, which can deal with spatial and sequential information simultaneously. Therefore, this method can enhance the model’s recognition and classification capabilities.

2.2. Implementation of the CNN Branch

Convolutional Neural Networks (CNNs) are types of deep learning models specifically designed to process image data. CNNs employ convolutional layers for extracting local features, pooling layers to diminish data dimensionality and computational complexity and fully connected layers for classification. The core advantages of CNNs lie in their local connections and shared weight, which make them particularly effective for image recognition and classification tasks.

CNNs extract features by sliding convolutional kernels over the pixels’ matrix of the input image, computing the weighted sum of local regions to generate feature maps (Figure 3b). “*” denote the multiplication of two parameters. The kernels capture local features like edges and textures. Subsequently, activation functions perform nonlinear mapping operations on these features, enhancing the model’s ability to filter key characteristics. The parameters of the network are optimized through the error back propagation and iterative learning processes. This learning process automatically constrains the input and maximizes the activation of the output.

Residual Network (ResNet) is an improved version of CNNs, introducing skip connections or residual connections (Figure 4a). These connections allow gradients to pass directly from later layers to earlier layers, addressing the gradients’ vanishing and exploding problems in the networks’ training process. By stacking more layers, ResNet achieves deeper network architectures than the traditional CNNs and has demonstrated significant performance in various visual tasks [33].

Different from the ResNet network, some researchers have noted that the brightness degradation on images primarily resides in the magnitude components of Fourier transformation, while the rest exists in the phase components [34]. Inspired by previous research on Fourier transformation, this backbone further introduces the correlation properties between magnitude component and brightness to enhance feature extraction effectiveness. In this backbone, two_stage feature architecture is designed (Figure 4b). In stage one, the brightness of low-light image features is enhanced by optimizing the amplitude in Fourier space. In stage two, features from convolutional neural networks are further integrated.

In stage one, as to the input image x with dimensions of H × W, its transformation of the frequency space can be represented as Equation (1):

$$F\left(x\right)\left(u,v\right)=X\left(u,v\right)={\displaystyle \frac{1}{\sqrt{HW}}}{\displaystyle \sum _{h=0}^{H-1}{\displaystyle \sum _{w=0}^{W-1}x\left(h,w\right)}}{e}^{-j2\pi \left({\displaystyle \frac{h}{H}}u+{\displaystyle \frac{w}{W}}v\right)}$$

(1)

where h and w denote coordinates in the temporal domain, while u and v represent coordinates in the frequency domain. To acquire frequency features, the Fourier processing (FFT) block is utilized to extract the amplitude and phase components (Figure 4). Subsequently, two 1 × 1 convolutional layers with Leaky activation are applied to each branch. Finally, an inverse Fourier transform (iFFT) is applied to convert these two components back to the spatial domain.

The Fourier transformation primarily relies on convolutional changes in the frequency domain to enhance brightness, while lacking convolution operations in the temporal domain for extracting details. Therefore, in the second stage, convolution operations in the temporal domain are employed to enrich features. Finally, to optimize feature fusion, dimension addition and reduction operations are applied to the features of the temporal-frequency domain, respectively.

Compared with the commonly used residual network (Figure 4a), our method with the Fourier Transform (Figure 4b) can convert convolution operations into multiplication operations in the frequency domain, thereby enhancing the efficiency of convolution calculations. By enhancing specific frequency components, specific edge features can be accentuated, which is especially useful for image enhancement. However, during the inverse Fourier Transform process, some important information may be lost. Therefore, integrating features from the temporal domain is crucial to preserve specific characteristics.

2.3. Implementation of the Transformer Branch

Unlike CNNs, which extract features through global attentions, transformers capture local features from token regions. Local feature extraction is achieved through the matrixes’ multiplication, which has been validated in various high-level and low-level vision tasks. Assuming the feature dimensions are h*w*C and the token size is p*p, the total number of feature tokens can be calculated as m = (h/p)*(w/p)*C. Moreover, multi-head self-attention (MSA) modules and multi-layer perceptions (MLP) are employed in the transformer. Assuming the input features to the transformer have the same dimensions, tokens are merged into a sequence of features with multiple heads (Figure 5a). Q*K^T = (R^d*n)*(R^d*n)^T = R^d*d, and its computation load = d²*n. Then, (R^d*d)*(R^d*n) = R^d*n, and its computation load = d²*n. In summary, the total computation load = 2d²*n. The calculation process of feature transformation and computational complexity are depicted in Equation (2) and Equation (3), respectively.

$$\left\{\begin{array}{l}{q}_i=k_i=v_i=LN\left(F_1,F_2,\dots ,F_m\right)\hfill \\ {\stackrel{⌢}{y}}_i=MSA\left(q_i,k_i,v_i\right)+y_{i-1}\hfill \\ y_i=MLP\left(LN\left({\stackrel{⌢}{y}}_i\right)\right)+{\stackrel{⌢}{y}}_i\hfill \end{array}\right.$$

(2)

$$\widehat{x}=\mathrm{softmax}\left({\displaystyle \frac{Q\times K^T}{\sqrt{d}}}\right)\ast V$$

(3)

where Q is the query matrix, K is the key matrix, and V is the value matrix.

To alleviate calculation complexity, the method in Figure 5b replaces traditional dot-product multiplication with element-wise multiplication. Different colors denote different dimensions. Typically, Q, K, and V have dimensions of R^d×n. To compute the attention weights for extracting features from underwater images, the query matrix is initially multiplied by a trainable parameter vector (w_n ∈ Rⁿ). This process results in the generation of a global attention vector of size η_d in Equation (4):

$${\eta }_d={\displaystyle \frac{\exp \left(Q\ast w_n/\sqrt{n}\right)}{{\displaystyle \sum _{j=1}^d\exp \left(Q\ast w_n/\sqrt{n}\right)}}}$$

(4)

Next, the K matrix undergoes element-wise multiplication with the global attention vector η_d to yield the global query vector q. As shown in Figure 5b, q ∈ R^1*n. Subsequently, this global vector q is element-wise multiplied with the V matrix to generate global features that merge information from both the Q-matrix and K-matrix. Unlike the previous dot-product computations, the computational load of element-wise multiplication is linearly related to the parameters (d*n), alleviating the overall computational load. Following this, we perform another transformation to activate the final information in Equation (5):

$$\left\{\begin{array}{c}q={\displaystyle \sum _{i=1}^d{\eta }_i\ast K_i}\\ x=T\left(q\ast V\right)\end{array}\right.$$

(5)

where T denotes the activation operation. To mitigate the influence of extremely dark regions on inference, an SNR map is utilized to guide the learning attention of the transformer. For an input image I ∈ R^{H x W x 3}, with its corresponding SNR map S ∈ R^{H x W}, S is adjusted into S′ ∈ R^{h x w} to align with the dimensions of the feature map F. Then, S’ is partitioned into m patches, and the average value for each patch is calculated. S_i ∈ [0, 1], where I = {1, ..., m}. This masking mechanism effectively prevents the influence of features with very low signal-to-noise ratio (SNR), as illustrated in Figure 5b. The i-th mask value of S′ is defined in Equation (6):

$$S_i=\left\{\begin{array}{c}0,\hspace{1em}{S}_i<s\\ 1,\hspace{1em}{S}_i>s\end{array}\right., i=\left\{1,\dots ,m\right\}$$

(6)

The masking calculation process for the x parameter is expressed in Equation (7):

$$\widehat{x}=x\ast S$$

(7)

As shown in

Table 1. Computation load for different parameters.

Parameters	Computation Load	Computation Load Summation
${\eta }_d$	d*n	3d*n
q	d*n
$\widehat{x}$	d*n

, the total calculation load is 3d*n, which is far less than the previous calculation load 2*d²*n. According to the commonly used dot-product multiplication in Figure 5a, the computational complexity of the self-attention mechanism scales quadratically with the sequence length (d²), causing a significant increase in resource consumption when the sequence length is large. Due to the large number of parameters in each layer, transformer models are typically much larger than CNNs, requiring more memory and computational power for training and inference. By adopting the proposed hybridized block modular approach, the computational load can be reduced from 2d²n to 3dn, offering a substantial advantage. The integration of CNN and Transformer models increases complexity and computation time. Therefore, the algorithm’s complexity is reduced through the matrixes’ element-wise multiplication.

It is worth noting that similar block or token attention has been developed. Zhou et al. divided large images into smaller blocks, utilizing a trained rCNN as a block descriptor for image forgery detection [35]. To develop a well-suited denoising model, Bei et al. introduced a block matching and grouping method, applying a convolutional neural network (CNN) within each block for 3D filtering [36]. To generate high-resolution landslide susceptibility maps, Abbas et al. created an innovative hybrid block-based neural network model, integrating expert modular structures and divide-and-conquer strategies with a genetic algorithm (GA) [37]. In this method, each sub-network module employs input blocks, layers of hidden blocks, and an additional decision block (Figure 6a). Different from the independent block’s method, the element-wise multiplication operation is developed in the research to extract the cross-attention (Figure 6b).

To evaluate the performance of different block-based methods, the two structures in Figure 6 were employed to enhance images across three distinct scenarios. Despite the significant development and approved capability of image processing systems through the advanced block-based or modular structures, our presented model in this study offers three significant advantages. Firstly, it can reduce learning losses and accelerate model convergence (Figure 7a). Secondly, it captures global forward-backward attention more effectively and extracts the continuous features, thereby minimizing information loss from independent blocks (Figure 7b,c). Thirdly, it produces high-quality enhanced images with higher PSNR and SSIM indexes (Figure 7d).

2.4. Fusion Attention Based on High-Pass and Low-Pass Filters

The torch.add method is commonly used to perform element-wise addition of tensors [35]. This method checks the shapes of the input tensors. When the shapes are aligned, the addition operation is performed element by element. This means that corresponding elements from each tensor are added together, producing a new tensor as the result. The result of the addition operation can be stored into a new tensor. While this method can combine different features, it cannot differentiate or utilize the advantages of different features.

Different from the traditional torch.add method, the significant difference of low-frequency and high-frequency features is valuable and can be utilized. The display of an image relies on trigonometric frequency components. High-frequency signals cause rapid changes, leading to sharp edges within the image. Conversely, low-frequency signals result in more gradual changes, contributing to a smoother image appearance. The role of filters is to pass or suppress certain frequency components of an image.

The Fourier transform acts as a bridge between the temporal domain and the frequency domain (Figure 8). Ideal low-pass filtering, a method for image smoothing, retains low-frequency components. The transfer function of an ideal low-pass filter is represented in Equation (8):

$$\left\{\begin{array}{l}D\left(u,v\right)={\displaystyle \sum _{x=0}^{M-1}{\displaystyle \sum _{y=0}^{N-1}f\left(x,y\right)e^{-j2\pi \left(ux/M+vy/N\right)}}}\hfill \\ L\left(u,v\right)=e^{-D^2\left(u,v\right)/2D_0^2}\hfill \\ L\left(x,y\right)={\displaystyle \sum _{x=0}^{M-1}{\displaystyle \sum _{y=0}^{N-1}L\left(u,v\right)e^{j2\pi \left(ux/M+vy/N\right)}}}\hfill \end{array}\right.$$

(8)

where M and N denote the length and height of the image, respectively. D(u, v) denotes the frequency domain of the image, and f(x, y) represents the temporal domain of the image. The range of u is [0, M − 1], and the range of v is [0, N − 1]. D(u, v) denotes the distance from the point (u, v) in the frequency domain to the center, while D₀ denotes the cutoff frequency. L(u, v) denotes the low-pass filters in the frequency domain. L(x, y) denotes the low-pass results in the temporal domain. Figure 9 illustrates the corresponding low-pass filter functions and their corresponding filtering outcomes.

Different from the low-pass filters, high-pass filters enhance the details and edges of the image by eliminating low-frequency components. The basic principle is to set the low-frequency components in the frequency domain to zero and only retain the high-frequency components. H(u, v) denotes the high-pass filters in the frequency domain. H(x, y) denotes the high-pass results in the temporal domain. Figure 10 illustrates the associated high-pass filter functions and their filtering results. The transfer function of a high-pass filter is designed by Equation (9):

$$\left\{\begin{array}{l}D\left(u,v\right)={\displaystyle \sum _{x=0}^{M-1}{\displaystyle \sum _{y=0}^{N-1}f\left(x,y\right)e^{-j2\pi \left(ux/M+vy/N\right)}}}\hfill \\ H\left(u,v\right)=1-e^{-D^2\left(u,v\right)/2D_0^2}\hfill \\ H\left(x,y\right)={\displaystyle \sum _{x=0}^{M-1}{\displaystyle \sum _{y=0}^{N-1}H\left(u,v\right)e^{j2\pi \left(ux/M+vy/N\right)}}}\hfill \end{array}\right.$$

(9)

Fourier transform is utilized to calculate fusion weights for integrating different backbone features (Figure 11b). Low-frequency features are extracted by the transformer to align with locally smooth features. Sharp edge details are captured by convolutional neural networks to match the high-frequency features. Both high- and low-frequency features are normalized to the [0, 1] range. The different fusion calculation is illustrated in Equation (10):

$$\left\{\begin{array}{l}{F}^{\prime }=F_{CNN}+F_{Transformer}\hfill \\ F^{″}=F_{CNN}\ast H\left(x,y\right)+F_{Transformer}\ast L\left(x,y\right)\hfill \end{array}\right.$$

(10)

Low-frequency Fourier variations correspond to transformer features, while high-frequency Fourier variations correspond to CNN features. Compared with the commonly used torch.add operation in Figure 11a, the combination of CNNs and transformers can fully leverage different advantages. This method not only enhances feature representation capabilities and robustness, but also optimizes the use of computational resources.

3. Experimental Validation

3.1. Dataset and Experimental Designation

We evaluate the algorithm’s performance by using two publicly accessible datasets: LSUI [36] (Large-Scale Underwater Image dataset) and UIEB dataset [37]. LSUI comprises 5000 underwater images with varying exposure levels. The UIEB dataset includes pairs of low-exposure and high-exposure images, with 800 pairs designated for training, 150 pairs for validation, and 90 pairs for testing. The LSUI and UIEB datasets play crucial roles in underwater image enhancement research. LSUI, with its large and diverse data volume, offers ample material for training and testing deep learning models. Due to its high-quality annotated image pairs, UIEB is a key resource for evaluating and optimizing algorithms. By utilizing the two datasets, it provides us with powerful and robust underwater image evaluation benchmarks.

Our framework was implemented in PyTorch [38], and the training and testing processes are conducted on a computer equipped with a 2080Ti GPU. Gaussian distribution was used to randomly initialize the network training parameters. And standard data augmentation techniques, such as vertical and horizontal flipping, are applied. Our encoder frame includes three layers, which are followed by a feature fusion module. Similarly, the decoder comprises three layers, utilizing ChannelShuffle for up-sampling operations. Adam optimizer [39] with an initial learning rate of 1 × 10⁻³ was used to minimize loss. The learning rate was decreased by 0.1 after every 100 iterations.

During the training process, the model’s performance was evaluated through loss functions (such as MSE, PSNR, etc.), which measures the discrepancy between the output and ground truth images. The model’s weights are saved during each training epoch. The .ckpt files are used to save training weights. The loss function is expressed in Equation (11):

Total Loss = α*MSE + β*(1 − SSIM) + γ*PSNR

(11)

Here, α, β, and γ are the weighting coefficients used to optimize different components’ influence in the loss functions. Through the defined loss function, the performance of the underwater image enhancement model can be effectively evaluated and optimized, improving the quality of enhanced images. When needed, the optimal weight of the model can be loaded from the storage files for inference and further training.

3.2. Ablation Study

For the underwater images, evaluation metrics include the Peak Signal-to-Noise Ratio (PSNR) [40], Structural Similarity Index (SSIM) [41], and the Mean Squared Error (MSE). MSE represents the mean squared error between two approximate images I and K, as defined in Equation (12):

$$MSE={\displaystyle \frac{1}{mn}}{\displaystyle \sum _{i=0}^{m-1}{\displaystyle \sum _{j=0}^{n-1}[I\left(i,j\right)-K\left(i,j\right)}}{]}^2$$

(12)

The PSNR metric represents the signal ratio of maximum value to mean squared error. It is represented by the logarithmic decibel units, as indicated in Equation (13):

$$PSNR=10\ast {\log }_{10}\left({\displaystyle \frac{MA{X}_I^2}{MSE}}\right)=20\ast {\log }_{10}\left({\displaystyle \frac{MA{X}_I}{\sqrt{MSE}}}\right)$$

(13)

where MAX_I denotes the maximum value of image color. Higher PSNR values indicate a clearer image. SSIM requires two input images to assess their similarity. One of the images is an uncompressed and undistorted image, and the other is the restored image. So, SSIM can serve as a metric for quality assessment. Assuming x and y are the two input images, the SSIM(x, y) is defined in Equation (14):

$$SSIM\left(x,y\right)={\left[l\left(x,y\right)\right]}^{\alpha }{\left[c\left(x,y\right)\right]}^{\beta }{\left[s\left(x,y\right)\right]}^{\gamma }$$

(14)

Here, α > 0, β > 0 and γ > 0. l(x, y), c(x, y), and s(x, y) are defined in Equations (15) and (16):

$$\left\{\begin{array}{l}l\left(x,y\right)={\displaystyle \frac{2u_x{u}_y+c_1}{u_x^2+u_y^2+c_1}}\hfill \\ c\left(x,y\right)={\displaystyle \frac{2{\sigma }_{xy}+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}}\hfill \\ s\left(x,y\right)={\displaystyle \frac{{\sigma }_{xy}+c_3}{{\sigma }_x{\sigma }_y+c_3}}\hfill \end{array}\right.$$

(15)

$$\left\{\begin{array}{l}{u}_x={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{x}_i}\hfill \\ {\delta }_x={\left({\displaystyle \frac{1}{N-1}}{\displaystyle \sum _{i=1}^N{\left(x_i-{\mu }_x\right)}^2}\right)}^{1/2}\hfill \\ Cov\left(X,Y\right)=E\left(X-E\left(X\right)\right)\left(Y-E\left(Y\right)\right)\hfill \end{array}\right.$$

(16)

Among them, c1, c2, and c3 are constants, respectively. To prevent system errors caused by the zero denominator, smaller values are adopted. In the actual calculation, it is common to assign α = β = γ = 1. c3 = c2/2. σ_xy represents the covariance of x and y. SSIM is simplified in Equation (17):

$$SSIM\left(x,y\right)={\displaystyle \frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left({\sigma }_{xy}+c_2\right)}{\left(u_x^2+u_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}}$$

(17)

The ablation study is a commonly used method in machine learning to evaluate the importance and contribution of various components in a model. By systematically removing certain parts of the model and observing the changes in performance, it is possible to identify which parts are important. A similar approach has been adopted by other scholars, as indicated in reference [42].

Rigorous ablation experiments were conducted to evaluate the proposed techniques. These experiments were conducted on the LSUI and UIEB datasets, evaluating three key factors: CNN features enhanced by Fourier transform, transformer features based on PSNR attention and linear operations, and feature fusion with Fourier weights. Figure 12 illustrates the enhancement effects in each ablation experiment. In Figure 12, (b–d) all use the same input from (a). Additionally,

Table 2. Comparative test of ablation experiments (“✓” denote the adopted scheme).

Structures				Fusion		LSUI		UIEB
CNN	Fourier	Transformer	SNR Attention	Additive Fusion	Fourier Fusion	PSNR	SSIM	PSNR	SSIM
✓						15.22	0.47	13.03	0.42
✓	✓					18.82	0.64	16.77	0.60
✓	✓	✓			✓	24.83	0.79	21.70	0.70
✓	✓	✓	✓	✓		24.42	0.75	21.56	0.69
✓	✓	✓	✓		✓	26.53	0.83	23.85	0.78

presents the comparison metrics of PSNR and SSIM for the ablation study. “√” denote the adopted scheme.

Experimental results indicate that image quality can be enhanced through the utilization of CNN and transformer architectures, respectively. Additionally, the integration of CNN and transformer features yields a notable improvement on the image enhancement.

By utilizing the appropriate PyTorch libraries, the best trained model was loaded for the test verification. Through normalization and resizing operations, the input images are standardized to align with preprocessing steps. Time tools are used to record the start and end times during the model inference. And the inference time for a single image is obtained by calculating the difference between the end and start times. In the experiment (

Table 3. Comparison of different matrix-multiplication attention and computation latency.

Backbones	Attention	Latency (ms)
Transformer	Dot-product Transformer	2.5 ms
	Element-wise Transformer	2.1 ms
Transformer + CNN	Dot-product Transformer	3.0 ms
	Element-wise Transformer	2.6 ms

), two types of backbone feature extraction networks are employed and the times are recorded, respectively. The experiment demonstrates that element-wise transformer attention can significantly reduce the time consumption. Additionally, while the dual-channel approach increases detection time, our method achieves the satisfied detection with the similar time consumption of the single-transformer approach.

3.3. Feature Visualization Process

To validate the robustness of our feature extraction method, the feature visualization process was conducted in Figure 13. These visualized features include two types of network features. Transformer features, extracted within the Token range, improve the local perception accuracy. The transformer network captures long-range features through the self-attention mechanism. The self-attention mechanism allows the Transformer to integrate features from any position within the image. This is crucial for tasks such as image restoration and color correction. According to the visualized results, it is easy to find that the transformer can effectively restore the local color and structure of images, overcoming the defects of CNNs in the local feature-extraction process.

Conversely, CNN features provide a global perspective, contributing to improve the global perception accuracy. By visualization, it is easy to find that CNNs excel at capturing the obvious features of images. Through convolution operations, CNNs can efficiently extract image details such as edges and textures, thereby effectively suppressing noise and enhancing boundary detail. Deeper convolutional layers enables CNNs to progressively extract high-quality features from images, which is significantly effective for removing random noise of underwater images.

Furthermore, we obtained high-pass filter and low-pass filter features by the Fourier transform, which are subsequently employed as fusion weights for the two backbones’ features. High-pass filters extract edge details of images, whereas low-pass filters capture smooth information. This complementary information is multiplied with the transformer and CNN features, respectively. This matching process enhances the accuracy of feature extraction and fusion.

Visual results show that the integration of CNNs with transformers yields superior image enhancement effects. In summary, CNNs can remove most noise and enhance the overall color and structure, while transformers can restore local details of the image. This combination effectively reduces noise and significantly improves the overall image quality. The effect is particularly notable when processing complicated underwater images.

Our proposed methods are compared with other methods. Figure 14b,c show the visualized global features, including the improved CNN network and the traditional ResNet network. The results indicate that the proposed method appears to have more prominent edge features, while the traditional ResNet method extracts relatively blurred features. The experiments demonstrate the superiority of the proposed method that integrates both the time-domain and frequency-domain features.

Different feature-fusion methods are also compared in Figure 14d,e. The results show that our method can optimize fusion weights for different objects, which enhances the feature diversity. In contrast, the torch.add method reduces the diversity and prominence of features.

3.4. Comparison with Current Methods

Our approach was qualitatively compared with other state-of-the-art (SOTA) image enhancement methods, including MIR-Net [40], U-Net [41], WaterNet [43], and Ucolor [44]. Additionally, the proposed backbone is compared quantitatively with the traditional CNN and Transformer architectures.

3.4.1. Qualitative Analysis

Visual samples of LSUI are displayed in Figure 15, which are also compared with other commonly used methods. The proposed approach demonstrates outstanding clearance, showcasing finer details, consistent colors, and higher visibility. Additionally, the method’s outputs display fewer visual artifacts, especially in zones with complicated textures.

A visual comparison of the UIEB dataset is presented in Figure 16, highlighting our method in dealing with noisy and low-light images. The results indicate that our approach can significantly enhance image brightness, enriches details, and suppresses noise.

3.4.2. Quantitative Analysis

To compare with other image restoration networks, PSNR and SSIM are used to evaluate performance. Generally, higher SSIM implies the image presence with more details and structure. The codes for comparison were obtained from the corresponding publications, and all detection experiments were conducted on the same original input dataset, without using optimized images from the intermediate process.

Table 4. Comparative evaluation with different image enhancement networks.

Methods	LSUI		UIEB
	PSNR	SSIM	PSNR	SSIM
CNN [16]	15.28	0.50	13.68	0.48
MIR-Net [40]	18.80	0.66	16.78	0.63
U-net [41]	19.45	0.78	17.46	0.76
WaterNet [43]	19.62	0.80	19.27	0.83
Ucolor [44]	21.62	0.84	20.67	0.81
Transformer [26]	22.83	0.79	21.70	0.70
Ours	24.49	0.85	22.79	0.81

provides a comparative analysis of various methods, indicating that our algorithm outperforms others in achieving the highest PSNR and SSIM scores.

Compared to the Transformer method [26], it is worth noting that our linear multiplication backbone only utilizes 60% of the parameters. Additionally, in comparison with the Ucolor-based approach [44], our method exhibits superior performance. Furthermore, our method outperforms MIR Net [40], U-net [41], and WaterNet [43], yielding improvements of 1-3 in PSNR and 0.1-0.3 in SSIM.

3.5. Comparison on Detection Tasks

To evaluate the effect of underwater image enhancement on detection tasks, the enhanced images were integrated into a series of detection algorithms, including the single-stage methods SSD, RetinaNet, and GIoU [45,46]. These enhanced images were utilized as inputs for various detection tasks. The obtained detection results demonstrate that the proposed method exceeds other competing methods in detection accuracy [47,48]. The visualized detection results in Figure 17 correspond with the objective outcomes, demonstrating our approach’s superiority.

By utilizing precision-recall and recall-confidence curves as evaluation metrics [49], Figure 18 presents a quantitative comparison of visual detection. Due to the improved color and brightness, our method demonstrates a notable enhancement in precision and recall indexes [50]. The enhanced images by this method show superior detection outcomes, marking a significant enhancement over other competing techniques.

4. Conclusions

The influence of light absorption and scattering by the surrounding water leads to the loss of certain details and color information in underwater images. To address issues, such as low illumination, reduced contrast, and color shift in underwater imagery, an underwater image enhancement algorithm is proposed based on the parallel fusion of transformer and CNN. Experiments indicate that this approach can effectively combine the local context capture ability of transformers with the global feature extraction capability of CNNs, thereby improving the richness and accuracy of extracted features. To effectively reduce computational load and alleviate color artifacts, a novel transformer model integrates the PSNR attention and linear operations. Through a mathematical method, this method can reduce computational complexity from 2d²n to 3dn while simultaneously extracting constrained features. Additionally, by leveraging both temporal and frequency domain characters, a novel global feature extraction network is devised to enrich image features. The high-frequency and low-frequency information from the input image’s Fourier transform are extracted, which are used to fuse different backbone’s features. Experiments show that this method optimizes the fusion weights for the Transformer and CNN features, enriching the diversity of representation features. Compared with the current mainstream algorithms, this method achieves optimal values in objective evaluation metrics and also produces superior subjective perceptual quality in the generated images.