Underwater Image Enhancement Using Improved CNN Based Defogging

Abstract

Due to refraction, absorption, and scattering of light by suspended particles in water, underwater images are characterized by low contrast, blurred details, and color distortion. In this paper, a fusion algorithm to restore and enhance underwater images is proposed. It consists of a color restoration module, an end-to-end defogging module and a brightness equalization module. In the color restoration module, a color balance algorithm based on CIE Lab color model is proposed to alleviate the effect of color deviation in underwater images. In the end-to-end defogging module, one end is the input image and the other end is the output image. A CNN network is proposed to connect these two ends and to improve the contrast of the underwater images. In the CNN network, a sub-network is used to reduce the depth of the network that needs to be designed to obtain the same features. Several depth separable convolutions are used to reduce the amount of calculation parameters required during network training. The basic attention module is introduced to highlight some important areas in the image. In order to improve the defogging network’s ability to extract overall information, a cross-layer connection and pooling pyramid module are added. In the brightness equalization module, a contrast limited adaptive histogram equalization method is used to coordinate the overall brightness. The proposed fusion algorithm for underwater image restoration and enhancement is verified by experiments and comparison with previous deep learning models and traditional methods. Comparison results show that the color correction and detail enhancement by the proposed method are superior.

1. Introduction

The ocean contains various abundant resources. However, it has not been effectively explored and exploited by humans, especially for the underwater world. Underwater image processing plays an indispensable role in the underwater operations by human or underwater robots, e.g., underwater archeology, environment monitoring, device maintenance, object recognition, and search and salvage. Affected by the suspending particles and the attenuation of light in water, the underwater optical images are of low quality, with flaws such as color distortion, low contrast, and even blurred details. Therefore, the degraded underwater images require enhancement and restoration to obtain more underwater information. In enhancing underwater images, classic techniques include histogram equalization, wavelet transform and Retinex algorithm [1,2,3,4]. However, there are some shortcomings for each of these algorithms. For example, loss of details of image and excessive enhancement might happen to histogram equalization. The effectiveness of wavelet transform is limited in processing the images obtained in shallow waters. For Retinex algorithm, the halo effect needs to be solved in the case of large brightness difference. To avoid those shortcomings, two or three out of these enhancement techniques are combined for examples in [5,6,7]. Results show that a fusion algorithm can provide a more comprehensive image enhancement effect.

To restore underwater images, an imaging model is required. A representative model is proposed by Chiang and Chen [8]. The key to using this model is to estimate the transmission map and the global ambient light. In past years, many researchers have engaged in the research on the estimation of these parameters. Generally, the studies can be viewed as two types: priori driven methods and data driven methods.

Prior driven approaches extract the transmission map and the atmospheric light environment through various priors or assumptions, so as to achieve image restoration. A typical method is the Dark Channel Prior (DCP) method [9]. However, the rules of DCP tends to be violated since the red light is absorbed extremely quickly in underwater environment, resulting an imbalance between the three channels. To modify the DCP method, Drews et al. [10] proposed an Underwater Dark Channel Prior (UDCP), which only uses the blue and green channels to calculate the dark channel. Galdran et al. [11] came up with the Red Channel Prior (RCP), which obtains a reliable transmission diagram based on the relationship between the attenuation of the red channel and the DCP. However, due to the decrease of reliability, such modifications only achieve minor improvement in the accuracy of the estimation of transmission map. Based on the fact that the red channel attenuates much faster than green and blue channels, Carlevaris-Bianco et al. [12] proposed a new prior which estimates the depth of the scene with the aid of attenuation difference. Wang et al. [13] developed a new method, called maximum attenuation identification (MAI), to derive the depth map from degraded underwater images. Li et al. [14] estimated the medium transmission by reducing the information loss in a local block for red channel. However, the reliance on color information in these priors results in the underestimation of the transmission of objects with green or blue color. Berman et al. [15] suggested estimating transmission based on the haze line assumption and estimating attenuation coefficient ratios based on the gray-world assumption. However, this method may fail when the ambient light is significantly brighter than the scene because most pixels will point at the same direction, and it is difficult to detect the haze lines.

Theoretically, when the depth of the scene increases, the transmission will decrease, and the ambient light will have a greater impact on the image quality. Thus, the estimation of ambient light often selects the pixel area with the largest depth in the image as the reference pixels. Usually, the method based on color information and the method based on edge information are used to select these reference pixels more accurately. Chiang et al. [8] inferred the ambient light from the pixel with the highest brightness value in an image. Selecting pixels in the red channel as a reference is an alternative method because the intensity in the red channel is much lower than the intensities of blue and green [11]. However, since the reference pixels are selected based on color information, objects with the same color may interfere with the selection. Berman et al. [15] proposed an estimation of ambient light according to the edge map associated with smooth non-texture areas. However, this method is not suitable if the image contains a large object with smooth surface. Moreover, sometimes there is no ideal reference pixel in an image. For example, an image photographed from a downward angle may not contain pixels with deep depth. Overall, the prior-based methods tend to make large estimation errors when the adopted priors are not valid. Lack of reliable priors for generic underwater images has become a major hurdle that impedes the prior-based approaches to achieve further progress.

In recent years, a data driven method, deep learning, has become more and more popular in image processing. Deep learning-based methods rely on learning the relationship between images, which makes it possible to avoid estimation errors due to invalid priors in the method. In deep learning methods, Convolutional Neural Network (CNN) is a representative approach and has been widely applied to the underwater image processing. Initial applications are mainly focused on the estimation of the transmission in images, for example in [16,17,18,19]. These CNN based models are trained with synthetic data set to regress transmission and obtain more refined restored images than conventional methods. Further applications concern both the transmission and the ambient light, for example in [20,21,22]. However, due to the assumption that three channels have the same transmission, these methods only partly solve the effect of scattering. Although the image contrast is improved, color deviation cannot be well corrected. Wang et al. [23] firstly used T-network and A-network to estimate the blue channel, and then estimated the red and green channels through the relationship between the channels to restore underwater images. In order to learn the differences between different channels of underwater images, Li et al. [24], Sun et al. [25], and Uplavikar et al. [26] directly use a data-driven end-to-end network to learn the mapping relationship between underwater degraded images and clear images. It is known that the performance of data driven based methods largely depends on the quality of training data. The underwater images used for training are usually obtained by synthesis without color deviation (e.g., [16,17,18]) or with color deviation (e.g., [20]). However, due to the complex underwater environment, the color deviation of underwater images only contains various tones of blue or green. To improve the quality of the synthetic data set, Li et al. [27] used Generative Adversarial Networks (GAN) to produce underwater degraded images. However, the restored underwater images are still not realistic enough compared with real underwater images.

As discussed above, each enhancement or restoration method has inherent disadvantage in processing underwater images. One way to gain a high-quality underwater image is a fusion algorithm in which each module can perform well while its disadvantage can be compensated by other module. In this paper, a fusion algorithm is proposed. It includes a color balance algorithm based on the CIE Lab color model, an end-to-end CNN defogging algorithm based on foggy training set and a brightness equalization module. In this way, the mapping relationship between the underwater blurred image and the clear image can be obtained without synthesizing the data set of the underwater image. The inaccurate mapping due to the insufficient reality of the synthesized images can be also avoided. The color balance algorithm uses the channel value redistribution in the color channel to achieve color balance. The defogging algorithm uses deep separable convolution instead of standard convolution to reduce the amount of parameters in the calculation process. Moreover, sub-network is incorporated into the defogging module to reduce the depth of the main network. In the CNN structure, the Basic Attention module is used, which mainly includes Channel Attention block (CA) and Pixel Attention block (PA). CA is used to trace the difference in the distribution of fog between channels while PA is used to record the difference in haze weight between pixels. By aggregating different contextual information of images, a pooling pyramid module is added to improve the capability of the network model that contains global information. Moreover, cross-layer connections are used to preserve edge information. The brightness equalization module uses the Contrast Limited Adaptive Histogram Equalization method in the L channel (Luminance channel in CIE Lab color model) to coordinate the overall brightness.

The main contributions of the study include the construction of CNN to defog the underwater images, and the combination of three techniques to achieve more comprehensive quality of underwater images as well. To verify the fusion algorithm proposed, some degraded underwater images are processed and the results are compared between the proposed method and other algorithms including classic and advanced ones such as CNN-based algorithm. The underwater images used in CNN are derived from UIEB Dataset [28].

The rest of the paper is organized as: Section 2 gives the description of underwater imaging model and the color model used in the study; Section 3 illustrates the methods employed in the proposed fusion algorithm and the training dataset; Section 4 presents the image processing results and corresponding explanation; and the conclusion is given in the final section.

2. Models in Underwater Imaging Process

2.1. Underwater Optical Imaging Model

Figure 1 shows an underwater imaging model. In an underwater environment, the light captured by the camera is mainly composed of scene radiation and ambient light reflection. This model can be expressed as:

$$I_c(x)=J_c(x)t_c(x)+A_c(1-t_c(x)),c\in \left\{r,g,b\right\}$$

(1)

where x represents a pixel in the original underwater image and c represents three color channels; $I_c(x)$ is the image captured by the camera; $J_c(x)$ represents a clear image or the so-called restoration image; $A_c$ represents the global ambient light and $t_c(x)$ represents the transmission map.

According to (1), the restoration image $J_c(x)$ can be expressed as:

$$J_c(x)=\frac{1}{t_c(x)}{I}_c(x)-A_c\frac{1}{t_c(x)}+A_c.$$

(2)

It can be seen from the underwater imaging model that the key to restoring an underwater image is to estimate the transmittance of the image map and the corresponding ambient light value. To apply NN to the estimation of transmittance and ambient light, Li et al. [29] introduced a variable by deforming the atmospheric scattering model. This method can also be applied to underwater imaging models. Model (2) can be modified as:

$$J_c(x)=K_c(x)I_c(x)-K_c(x)+b,$$

(3)

where

$$K_c(x)=\frac{(1/t_c(x))(I_c(x)-A_c)+(A_c-b)}{I_c(x)-1},$$

(4)

and b is the constant deviation value (usually set to 1).

As can be seen from definition of K_c(x), the transmittance and ambient light value are jointly estimated, which can avoid the cumulative error caused by the training for transmittance and the training for ambient light.

2.2. CIE Lab Color Model

The CIE Lab color model is a device-independent color system [30]. It uses a digital method to describe human visual perception and makes up for the deficiencies of the RGB and CMYK color modes. The Lab color model is composed of three elements: L, a and b. L is the luminance channel, with a range of [0, 100], which means a variation from pure black to pure white. The a channel covers colors from dark green to gray, to bright pink, with the value range of [−128, 127]. The b channel includes colors from bright blue to gray, to yellow, with the value range of [−128, 127].

3. Underwater Image Enhancement Methods

Based on the models in Section 2, in this section some measures to enhance the underwater images are proposed. In order to improve the quality of underwater images, a fusion algorithm is constructed which involves a color balance algorithm, a CNN-based defogging algorithm and contrast limited adaptive histogram equalization (CLAHE). The color balance algorithm is used to eliminate the color deviation caused by the attenuation of light in the water while the CNN defogging algorithm and CLAHE are used to improve the contrast of the image. The process of the fusion algorithm is shown as Figure 2.

3.1. Color Balance Algorithm

As can be recognized from the proposed scheme in Figure 2, the first step is to achieve the color balance. It is known that in underwater environments, the light attenuation varies with its wavelength. As the depth increases, an underwater image changes from greenish to bluish. Figure 3 presents four underwater images and their gray histograms in the CIE Lab color model. It can be seen from the second and third images that when the image is greenish, a large number of values in the a channel are on the left side of the histogram. The fourth image shows that when the image is bluish, a large number of values in the b channel are on the left side of the histogram. As can be recognized from the first image that when image color shift is weak, the channel values of the a and b channels are concentrated in the middle of the histogram.

In order to reduce the influence of color shift, a color balance method based on CIE Lab color model is proposed. By moving the channel value to the middle position of the channel histogram, color balance is achieved. At the first step, the RGB image is converted into the Lab image. The second step is to obtain the channel values Ma and Mb of the median of channels a and b, respectively. Then, one can obtain the correction value offset1 of the a channel and the correction value offset2 of the b channel by Equations (5) and (6), respectively. At the third step, the a channel and the b channel are updated by adding their respective correction values. For more intuitive results, the range of channel values is converted from [−128, 127] to [0, 255] by satisfying the boundary conditions that if the calculated new channel value is greater than 255, it is set to 255; if it is less than 0, let it be 0. Finally, by converting the processed Lab image to an RGB image, a color-balanced image is obtained.

$$offset1=128-Ma$$

(5)

$$offset2=128-Mb$$

(6)

To verify the effectiveness of the proposed color balance algorithm, it is compared with three conventional color balance algorithms including gray world [31], color shift check correction [32], and automatic white balance [33]. Figure 4 presents the visual comparison of the color balance results of six underwater images. It can be seen that in the case of weak color degradation (images No.1, No.2 and No.3), all methods have good processed results. In the case of serious color degradation (images No.4, No.5 and No.6), the proposed method gains a better color balance effect than the other three methods. For the sixth bluish image, even over-exposure happens to the other three methods.

3.2. CNN based Defogging Algorithm

It can be seen from Figure 4 that although the color deviation is alleviated by the proposed color balance method, it seems that the processed image is covered with a thick layer of fog. In order to improve the contrast of the image, an end-to-end neural network defogging algorithm based on the underwater optical imaging model is proposed, shown as the second step in Figure 2.

3.2.1. Network Architecture

In the study, a modified underwater imaging model as (3) is employed to restore underwater images. In this model, a comprehensive index K(x) is introduced. The motivation is that calculating the ambient light and the transmission map separately will result in the accumulated errors derived from each individual estimation step. Instead, direct estimation of K(x) will reduce such errors. For this purpose, an end-to-end CNN based defogging network is established, as shown in Figure 5. The index K(x) is estimated by a UD-net. Afterwards, the restoration image can be obtained by using (3). To increase the speed of the network, depthwise separable convolution is employed instead of standard convolution. Depthwise separable convolution is a form of decomposed convolution [34]. It separates standard convolution integrals into depthwise convolution and pointwise convolution. In this way, the calculation burden and model size can be significantly reduced. Moreover, it can be seen that in the network construction for estimation of K(x), a sub-network is formed to shorten the depth of the main network [17]. As can be recognized from Figure 4, in the process of sub-network, three different depthwise separable convolutions (11*11, 9*9, 7*7) combined with BMU module are firstly designed to obtain a larger receptive field and learn better global information. Then, the sub-network performs 3*3 depthwise separable convolution and uses a sigmoid activation function to linearize the upsampling result. Finally, a 3-channel feature map will be obtained and output to the main network.

As can be seen from the construction of the sub-network, it involves five operations: depthwise separable convolution, basic attention module, max-pooling, upsampling and linear combination. It is noted that most conventional image defogging networks such as multi-scale CNN (MSCNN) [17] and all-in-one network [29] equally treat the distribution of fog on different channels and pixels. As a result, the fog density cannot be well estimated especially in complex scenes. In the study, a feature attention block [35] is introduced to solve such a problem. As shown in Figure 6, the feature attention (FA) block consists of Channel Attention (CA) and Pixel Attention (PA), which can provide additional flexibility in dealing with different types of information. In the CA part, firstly the channel-wise global spatial information is transformed into a channel descriptor by using global average pooling. The shape of the feature map changes from C*H*W to C*1*1. Then, to get the weights of different channels, features pass through two convolution layers with ReLu and sigmoid activation functions. Finally, to obtain the output, element-wise product is performed by using the input and the obtained weights of the channel. In the PA part, it directly feeds the input (i.e., the output of the CA part) into two convolution layers with ReLu and sigmoid activation functions. The shape of the feature map changes from C*H*W to 1*H*W. Finally, element-wise multiplication is performed by using the input and the weights of the pixel. It is noted that besides feature attention module, a basic attention block (BAB) structure adopts the way of local residual learning. Local residual learning allows the less important information such as thin haze or low-frequency region to be bypassed through multiple local residual connection so that the main network can focus on effective information. Different from the Convolutional Block Attention Module (CBAM) proposed in [36], the BAB module in the study uses only average pooling instead of the combination of average pooling and max pooling in CBAM, in the meanwhile residual learning is added to the BAB module.

One of the shortcomings of conventional underwater image enhancement methods is the loss of details, for example the classic histogram equalization. In order to make the processed results contain more details, a pyramid pooling block is incorporated into defogging network shown as Figure 5. The pyramid pooling block that locates before the last convolution layer is an enhancing module (EM) to expand the representational ability of network [37]. The use of pyramid pooling can learn more context information based on different receptive fields. As a result, the image processing results can be refined. Figure 7 presents the structure of pyramid pooling block. It is designed to integrate the details of features from multi-scale layers and to obtain global context information by learning on different receptive field. As can be seen, a four-scale pyramid is built by downsampling the outputs of the conventional layers with factors 4*, 8*, 16*, and 32*. Each scale layer contains a 1*1 convolution. After upsampling, feature maps before and after the pyramid pooling are concatenated. Subsequently, a 3*3 convolution is added to align feature maps.

3.2.2. Implementation Details

In this section, we specify the implementation details of our proposed UD-Net. UD-Net framework is employed in the study. The training environment of the UD-Net is: Windows10; NVIDIA GeForce RTX 3080; Inter^® Core™ i5-8500.

The dataset used is the fogging dataset synthesized by Li et al. [38], which is based on the Outdoor Training Set (OTS) depth database by setting different atmospheric light intensity A_c and transmittance t_c(x). In total, 32,760 images out of the dataset are taken as the training set, while 3640 images are taken as the validation set. The dimension of images from used dataset is 320*320. In the training process, Gaussian random variables are used to initialize the network weights. The activation function ReLU is used to add non-linearity to the network after each convolution. The loss function is mean squared error (MSE). Learning the mapping relationship between hazy images and corresponding clean images is achieved by minimizing the loss between the training result and the corresponding ground truth image. The optimizer is the Adaptive Moment Estimation (Adam). Detailed parametric setting is as follows. The learning rate of the network is set to 0.0001; the number of batch samples (batch_size) is set to 8; the epoch is set to 10; the decay parameter of weight is set to 0.001. Detailed parameter settings of our proposed UD-Net (as shown in Figure 5) are summarized in

Table 1. The parameter settings of the UD-Net model.

Type	Input Size (C*H*W)	Kernel Size	Activation Function
DS-Conv1	3*320*320	11*11(Depthwise Conv) 1∗1(Pointwise Conv)	ReLu
BMU	6*320*320	-	-
DS-Conv2	6*320*320	9*9(Depthwise Conv) 1*1(Pointwise Conv)	ReLu
BMU	6*320*320	-	-
DS-Conv3	6*320*320	7*7(Depthwise Conv) 1*1(Pointwise Conv)	ReLu
BMU	6*320*320	-	-
DS-Conv4	6*320*320	3*3 (Depthwise Conv) 1*1 (Pointwise Conv)	Sigmoid
DS-Conv5	3*320*320	7*7 (Depthwise Conv) 1*1 (Pointwise Conv)	ReLu
BAB	3*320*320	-	-
Concat1	3*320*320 3*320*320	-	-
DS-Conv6	6*320*320	5*5 (Depthwise Conv) 1*1 (Pointwise Conv)	ReLu
BAB	6*320*320	-	-
Concat2	3*320*320 6*320*320	-	-
DS-Conv7	6*320*320	5*5 (Depthwise Conv) 1*1 (Pointwise Conv)	ReLu
BAB	9*320*320	-	-
Concat3	9*320*320 9*320*320	-	-
CA	18*320*320	-	-
PA	18*320*320	-	-
DS-Conv8	18*320*320	3*3 (Depthwise Conv) 1*1 (Pointwise Conv)	Sigmoid
Concat4	3*320*320 13*320*320	-	-
Pyramid Pooling	16*320*320	-	-
Equation3	3*320*320	-	-

.

3.3. Contrast Limited Adaptive Histogram Equalization (CLAHE)

As aforementioned, a restoration model as (3) applies to processing hazy images in the air and underwater images as well. However, in most cases, the underwater background is darker than the air background. It implies that a further treatment is required for underwater images. In the study, Contrast Limited Adaptive Histogram Equalization (CLAHE) is used to process the Luminance channel (L channel in CIE Lab color space) of underwater images after defogging. In this way, the processed images can have better brightness and contrast. Figure 8 shows the result after CLAHE processing. The parameter setting is that the contrast limit is 1.1 in the area of 3*3. As can be seen, after defogging and subsequent CLAHE process, the quality of the underwater images is significantly improved.

4. Validation

To validate the proposed method, underwater images are processed and evaluated from both subjective and objective aspects by comparing with several classic and advanced methods used in underwater image processing, including RCP [11], integrated color model (ICM) [39], relative global histogram stretching (RGHS) [40], multi-scale Retinex with color restoration (MSRCR) [41], FUnIE-GAN [42], and UIE-CNN [43]. The images are taken from the UIEB Dataset [28], in which different degradation conditions are considered.

4.1. Subjective Visual Evaluation

The subjective visual evaluation consists of three parts, i.e., the comparison with other algorithms, ablation experiment and edge information detection.

4.1.1. Comparison with Other Algorithms

Figure 9 presents the results of twelve underwater images processed by six methods mentioned and the proposed method in the paper. It can be seen from the comparison that the proposed method generally outperforms the other algorithms, in terms of color balance, image details and contrast. For ICM and RCP, although they can improve the contrast of images, the color deviation cannot be eliminated well. For RGHS, it only shows good results in processing bluish images. Moreover, the robustness of RGHS is poor. For the MSRCR algorithm, it can effectively alleviate the color distortion. However, the processed image is too bright and the overall contrast is low. FUnIE-GAN has demonstrated the ability to restore underwater images to a certain extent, but the effect is mediocre. UIE-CNN performs well in general, but it does not work well when faced with more severely distorted scenes, as can be recognized from the results w.r.t. the images No.1 and No.12.

4.1.2. Ablation Experiment

In order to verify the effectiveness of sub-modules in the proposed fusion algorithm in this paper, six sets of images were subjected to ablation experiments in which each module is gradually removed in the combined algorithm. The results are shown in Figure 10. It can be seen from the ablation experiment that each algorithm module has achieved the corresponding processing effect, which helps to enhance the quality of the final output image.

In order to further verify the dehazing module, it is used to process some fogging images. The proposed method is compared with the several widely used defogging methods including DCP [9], AOD [29], DehazeNet [16], and MSCNN [17]. From the synthetic objective testing set (SOTS) [38], images of different haze concentrations are selected. The qualitative comparison of the visual effect is presented in Figure 11, in which eight images are investigated. As can be recognized, color distortion happens to DCP method (severe in No.6) due to the underlying prior assumptions. In this case, the depth details in image might be lost. For AOD method, it cannot remove the haze well when the fog density is large. DehazeNet method tends to output low-brightness images. MSCNN recovers images with excessive brightness. By contrast, the proposed defogging network produces better images when dealing with hazy images. This can be further confirmed by using two typical objective evaluation metrics including PSNR (Peak Signal to Noise Ratio) and SSIM (Structural SIMilarity) [44]. The quantitative evaluation results and comparison are listed in

Table 2. Evaluation of processed images by metric PSNR.

Image	Original	DCP	AOD	DehazeNet	MSCNN	Proposed
No.1	12.638	18.654	22.217	27.049	18.990	29.611
No.2	12.997	15.875	19.347	20.950	19.085	21.273
No.3	13.459	18.352	20.307	24.699	16.823	25.507
No.4	16.682	20.165	20.163	19.749	20.419	22.580
No.5	19.280	22.441	18.615	19.265	20.105	29.165
No.6	14.409	8.884	21.446	23.034	18.961	26.693
No.7	16.278	18.663	17.311	20.327	18.079	19.585
No.8	12.989	23.429	21.984	19.421	19.005	26.688
No.9	11.933	16.027	20.168	24.3000	18.059	24.872
No.10	18.369	14.753	20.134	18.185	18.647	26.373

and

Table 3. Evaluation of processed images by SSIM.

Image	Original	DCP	AOD	DehazeNet	MSCNN	Proposed
No.1	0.8172	0.8791	0.9177	0.9358	0.8848	0.9376
No.2	0.7794	0.8284	0.8511	0.8615	0.8272	0.8792
No.3	0.7085	0.8629	0.8371	0.8763	0.7199	0.8811
No.4	0.8627	0.8690	0.9065	0.8256	0.8878	0.8921
No.5	0.8947	0.8922	0.8588	0.7070	0.8715	0.9255
No.6	0.7714	0.5671	0.8775	0.8415	0.7918	0.9096
No.7	0.8661	0.9115	0.9218	0.9275	0.8952	0.9312
No.8	0.7430	0.8546	0.8612	0.8546	0.8027	0.8999
No.9	0.7024	0.8403	0.8340	0.8805	0.8024	0.8918
No.10	0.8669	0.8191	0.9112	0.7406	0.8686	0.9218

. The PSNR and SSIM values of the proposed method are greater than those of others in almost all cases, which indicates that the proposed CNN based defogging network has better applicability and robustness. It is noted that all the best performance indices are emphasized by bold font.

4.1.3. Edge Information Detection

In the field of image classification, target detection and feature recognition, the acquisition and application of image edge information play an important role. The higher the quality of the image is, the more obvious the edge information is. On the contrary, the low-quality images have less obvious edge information. To further demonstrate the practicability of proposed method, three representative images, No.2, No.8 and No.12, are selected to study the performance of edge detection. These three images are obtained from different environments and different backgrounds. Canny edge detection is performed. Figure 12 shows the results of edge detection by different methods. It can be seen that the edge information obtained by the proposed method is more abundant than the other methods.

4.2. Objective Quality Evaluation

Subjective visual evaluation is easily affected by human feelings. Different observers may draw different conclusions on the same group of pictures. Therefore, in the study, an objective quality evaluation of the underwater image processing results is also carried out. The intrinsic characteristic indices of the image are expressed in digital forms to objectively measure the image quality. In this paper, average gradient, RMS (Root Mean Square) contrast, UCIQE (underwater color image quality evaluation) and information entropy are employed to objectively evaluate the quality of the enhanced underwater images.

The average gradient metric is related to the texture transformation of the image and the difference in image details. To a certain extent, this metric can well reflect the contrast and sharpness of the image. In general, the larger the value of the average gradient is, the higher the clarity of an image is. The average gradient is defined as:

$$\mathrm{G}=\frac{1}{3}{\displaystyle \sum _{\lambda }\frac{1}{M\times N}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N\sqrt{\frac{{\left(\frac{\partial f}{\partial x}\right)}^2+{\left(\frac{\partial f}{\partial y}\right)}^2}{2}}},\lambda \in \left\{r,g,b\right\}}},$$

(7)

where M and N are the width and height of an image; $f=f(x,y)$ represents the gray value at the image point (x, y); $\frac{\partial f}{\partial x}$ represents the horizontal gradient; $\frac{\partial f}{\partial y}$ represents the vertical gradient.

The value of image contrast indicates the degree of grayscale difference. Generally, the higher the contrast of the image is, the higher the quality of image is. RMS contrast [45], Weber contrast [46], and Michelson contrast [47] are commonly used indices. In this paper, the RMS contrast is taken, which can be expressed as:

$${\sigma }_{{\mathrm{I}}_{w\times h}}=\sqrt{\frac{1}{M\times N}{\displaystyle \sum _{{\mathrm{I}}_{w\times h}}{(I(x,y)-\frac{1}{M\times N}{\displaystyle \sum _{{\mathrm{I}}_{w\times h}}I(x,y)})}^2}},$$

(8)

where I(x, y) represents the pixel gray value of a certain point (x, y) in an image. Obviously, $\frac{1}{M\times N}{\displaystyle \sum _{{\mathrm{I}}_{w\times h}}I(x,y)}$ is the average pixel value of the image.

Compared with the conventional metrics such as average gradient and RMS, UCIQE provides a comprehensive evaluation way. Chroma, saturation, and luminance contrast of an image can be evaluated by this metric [48]. In general, the higher UCIQE value is, the better quality an image has. UCIQE metric is defined as:

$$\mathrm{UCIQE}=C_1*{\sigma }_{\mathrm{c}}+C_2*co{n}_1+C_3*{\mu }_{\mathrm{s}},$$

(9)

where ${\sigma }_c$ is the standard deviation of chromatically, $co{n}_1$ is the contrast value of brightness, ${\mu }_s$ is the mean value of saturation; and $C_i$ stands for the weight coefficient. In the paper, $C_1$ = 0.4680, $C_2$ = 0.2745, and $C_3$ = 0.2576 are chosen by reference to [49].

Besides UCIQE, another comprehensive metric, information entropy, is selected to evaluate the proposed image enhancement method. Information entropy refers to the average amount of information contained in an image [49]. In general, the larger the value of entropy information is, the more information the image includes. Information entropy can be described as:

$$\mathrm{Entorpy}=-{\displaystyle \sum _{i=0}^{\mathrm{n}}p(i){\log }_{2}p(i)}$$

(10)

where n is the maximum gray level of an image; i is the gray level of the pixel point; p(i) represents the probability when the pixel value of the image is equal to i.

For the twelve underwater images that have been evaluated from subjective visual aspect, the results of the above four objective evaluation metrics by seven different image processing methods are listed in

Table 4. Evaluation of processed images by the metric of average gradient.

Image	Original	ICM	RCP	RGHS	MSRCR	Funie-Gan	UIE-CNN	Proposed
No.1	24.331	24.364	20.656	27.648	10.137	26.819	25.403	31.201
No.3	6.275	8.182	7.966	10.830	9.420	9.989	12.546	12.793
No.2	18.656	21.751	22.939	20.698	18.730	25.734	28.551	34.538
No.4	9.529	10.874	9.784	13.082	8.074	13.524	13.975	17.885
No.5	21.251	23.584	24.144	21.861	18.109	24.615	27.504	33.193
No.6	11.282	16.151	14.872	22.276	16.546	16.093	22.616	23.838
No.7	5.816	9.545	11.132	12.674	11.001	11.156	13.169	13.545
No.8	12.310	16.316	15.453	19.139	16.041	18.450	22.144	24.061
No.9	8.056	10.491	9.459	8.902	7.799	14.788	11.588	18.859
No.10	14.023	15.418	15.224	16.292	14.701	17.182	18.910	23.877
No.11	4.346	4.448	5.988	4.670	2.260	4.955	6.234	8.356
No.12	6.568	9.536	7.189	12.708	7.547	7.525	9.786	14.672

,

Table 5. Evaluation of processed images by the metric RMS contrast.

Image	Original	ICM	RCP	RGHS	MSRCR	Funie-Gan	UIE-CNN	Proposed
No.1	27.902	27.763	22.399	31.601	12.282	29.589	28.991	35.248
No.3	6.849	8.569	8.105	11.174	9.347	9.722	12.438	12.933
No.2	17.271	20.068	19.187	23.768	17.298	20.239	26.006	32.057
No.4	10.772	11.990	10.946	14.624	9.931	14.278	15.480	19.069
No.5	20.476	22.605	20.969	24.332	17.122	22.348	26.325	31.810
No.6	10.864	15.153	13.481	21.010	15.203	14.858	21.018	22.291
No.7	6.500	10.401	11.489	13.783	11.593	11.163	13.935	14.232
No.8	6.771	8.207	8.622	12.665	7.527	9.760	14.758	15.863
No.9	7.870	10.135	8.666	14.299	8.002	9.133	11.477	17.819
No.10	13.028	14.243	14.117	15.126	14.128	16.270	17.572	21.821
No.11	4.823	4.843	5.026	5.928	2.472	6.407	6.448	8.217
No.12	7.609	10.620	8.090	14.104	8.413	7.492	10.755	16.526

,

Table 6. Evaluation of processed images by UCIQE.

Image	Original	ICM	RCP	RGHS	MSRCR	Funie-Gan	UIE-CNN	Proposed
No.1	0.4788	0.4811	0.4042	0.4948	0.2205	0.4905	0.5121	0.5101
No.3	0.3162	0.4216	0.3746	0.4661	0.3296	0.3980	0.4591	0.4801
No.2	0.3646	0.4316	0.4307	0.4571	0.3198	0.4143	0.4277	0.4803
No.4	0.4022	0.4084	0.3825	0.4408	0.2441	0.3086	0.3701	0.4130
No.5	0.4070	0.4391	0.4230	0.4831	0.2687	0.4202	0.4469	0.4971
No.6	0.3552	0.3650	0.3451	0.4421	0.2534	0.3285	0.3636	0.4504
No.7	0.2589	0.3868	0.3867	0.4453	0.3042	0.3762	0.4045	0.4558
No.8	0.2781	0.3375	0.3324	0.3820	0.2475	0.4148	0.3780	0.4351
No.9	0.2645	0.3127	0.2715	0.4002	0.1964	0.2948	0.3550	0.4306
No.10	0.3641	0.4055	0.3833	0.4346	0.2895	0.4295	0.3930	0.4457
No.11	0.3843	0.3867	0.3362	0.5021	0.1061	0.3986	0.4090	0.4910
No.12	0.3407	0.3880	0.2637	0.4398	0.3040	0.3536	0.3820	0.4742

and

Table 7. Evaluation of processed images by information entropy.

Image	Original	ICM	RCP	RGHS	MSRCR	Funie-Gan	UIE-CNN	Proposed
No.1	7.1083	7.1127	7.0121	7.2949	5.6834	7.0808	7.1897	7.5803
No.3	6.6447	7.0609	7.1197	7.4704	6.9240	7.0686	7.4953	7.5347
No.2	7.1348	7.3507	7.3037	7.5808	6.9350	7.2779	7.6315	7.7812
No.4	7.0564	7.2178	7.1287	7.4650	6.2549	7.3907	7.5082	7.5268
No.5	7.4798	7.6411	7.5685	7.7038	7.0556	7.6303	7.6892	7.9086
No.6	6.8364	7.3210	7.4770	7.7686	6.9720	7.2046	7.7281	7.6687
No.7	6.4284	7.1044	7.3634	7.2833	7.0669	6.9440	7.3246	7.4753
No.8	6.3631	6.6527	6.8150	7.3212	6.2720	6.8051	7.3697	7.4480
No.9	6.6158	6.9918	6.7102	7.4969	6.0511	6.8190	7.1184	7.5610
No.10	7.5602	7.6870	7.7123	7.7445	6.9438	7.6212	7.6772	7.8194
No.11	7.3333	7.3553	7.4231	7.6748	7.3566	7.3653	7.3805	7.7256
No.12	6.8462	7.3881	7.0322	7.4631	6.4893	6.9610	7.3525	7.7023

. In detail, Table 4 gives the results evaluated by the metric average gradient; Table 5 is with the RMS metric; Table 6 is with the UCIQE; Table 7 is with the entropy information.

It can be clearly seen from Table 4 and Table 5 that the proposed method is significantly better than other methods in terms of average gradient and contrast. In terms of the comprehensive metrics including UCIQE and entropy information shown in Table 6 and Table 7, the performance of the proposed outperforms the other methods in almost all cases. Two exceptions for UCIQE metric include the results obtained by UIE-CNN for image No.1 and by RGHS for image No.4. Nevertheless, it is noted that in these two cases the results obtained by the proposed method gains the second best evaluation. For the metric entropy information, only one exception happens to the image No.6. In this case, the result obtained by RGHS is the best while the proposed method gains the third best evaluation.

In general, it can be summarized that the proposed underwater image enhancement strategy is better than other methods, in terms of contrast, color coordination, edge information and information. It performs well in dealing with the problems of color degradation, image blur and low contrast of underwater images.

4.3. Complexity

Table 8. The time it takes for the algorithm to process underwater images.

Algorithm	Cost (Seconds)	Module	Cost (Seconds)
ICM	118.83
RCP	77.03
RGHS	155.89
MSRCR	56.76
Funie-Gan	5.70
UIE-CNN	6.65
Proposed	165.05	Color Restoration Module	153.27
		End-to-end Defogging Module	10.67
		Brightness Equalization Module	1.12

shows the comparison of the time spent in processing 100 underwater images by the proposed algorithm and other algorithms including ICM, RCP, RGHS, MSRCR, Funie-Gan and UIE-CNN. It can be seen that the most of the time spent in the proposed algorithm is with a color restoration module that is based on a traditional method.

5. Conclusions

A hybrid underwater image enhancement and restoration algorithm is proposed that is composed of a color balance algorithm, an end-to-end defogging network algorithm, and a CLAHE algorithm. By studying the color imaging characteristics of underwater images and the histogram distribution characteristics of the image in the CIE Lab color space, a color balance algorithm is proposed to eliminate the color deviation of underwater images. Then, according to the similarity of underwater image imaging characteristics and foggy imaging characteristics, a CNN-based dehazing network is proposed to remove the blur of underwater images and to improve the contrast. Due to the difference between the foggy air environment and the underwater environment, a CLAHE algorithm is used to process the luminance channel in the CIE Lab color space to improve contrast and brightness of underwater images.

The proposed underwater image processing strategy is verified by using foggy air images and validated by using representative underwater images. Processed results show that the proposed method can effectively eliminate the color deviation and ambiguity in underwater images and meanwhile improve the image contrast. Comparison is conducted between the proposed method and other classic and advanced methods. In addition to the visual effect evaluation, four commonly used image quality evaluation metrics (average gradient, RMS contrast, UCIQE, and information entropy) are used to evaluate the effectiveness and advantages of the proposed algorithm. The comparison results indicate that the proposed algorithm is superior to other algorithms in terms of performance and robustness.

Notably, Table 8 shows that the proposed algorithm spends more time in processing images than other algorithms, which affects its real-time performance especially when a large number of images are to be processed. How to improve the real-time performance of the fusion algorithm will be studied in future work.