Robust Chromatic Adaptation Based Color Correction Technology for Underwater Images

Featured Application

This work aims for recovering realistic colors of underwater images, which is very important for the detection and classification of underwater targets.

Abstract

Recovering correct or at least realistic colors of underwater scenes is a challenging issue for image processing due to the unknown imaging conditions including the optical water type, scene location, illumination, and camera settings. With the assumption that the illumination of the scene is uniform, a chromatic adaptation-based color correction technology is proposed in this paper to remove the color cast using a single underwater image without any other information. First, the underwater RGB image is first linearized to make its pixel values proportional to the light intensities arrived at the pixels. Second, the illumination is estimated in a uniform chromatic space based on the white-patch hypothesis. Third, the chromatic adaptation transform is implemented in the device-independent XYZ color space. Qualitative and quantitative evaluations both show that the proposed method outperforms the other test methods in terms of color restoration, especially for the images with severe color cast. The proposed method is simple yet effective and robust, which is helpful in obtaining the in-air images of underwater scenes.

1. Introduction

Underwater imaging is increasingly used in many important applications such as marine biology and archaeology, underwater surveying and mapping [1]. However, captured underwater images are generally degraded by scattering and absorption. The color cast and the contrast loss are the main consequences of underwater imaging degradation processes, and the goal of underwater image processing is to rectify the color cast and enhance visibility [2,3]. Many image formation model based (IFM-based) image restoration methods and IFM-free image enhancement methods have been raised to achieve this goal [1,4]. The restoration methods try to reverse the underwater imaging process according to some priors, such as dark channel prior (DCP) [5,6] and scene depth map [3,7]. The enhancement methods use qualitative subjective criteria to produce more visually pleasing images [4]. Recently, many deep-learning-based methods have been developed for underwater image restoration and enhancement [3,8,9,10,11,12]. However, the deep-learning-based methods are hindered by the lack of large training datasets [12].

Most of the underwater image restoration and enhancement methods try to correct the color cast and enhance the contrast simultaneously. The color correction methods are used casually and there is no specialized comprehensive comparison between different color correction methods on underwater images. This paper focuses on the color correction of underwater images with the assumption that sunlight is the only light source. Color cast is generated due to the variation of light attenuation, which is caused by absorption and scattering, in different wavelengths [9]. To be exact, during the light propagation in the water medium, color cast is formed in two processes. One is the light travels downward before reaching the scene and the other is the light travels to the camera after reflected by the scene [7,13]. Since the object of interest called foreground is usually closer to the camera than the background, the color cast of the foreground formed in the second process is generally little. The color cast formed in the first process is dominant, especially for the images captured in deep water. Without considering the color cast formed in the second process, it looks like that an underwater image is captured with the scene illuminated by an unknown light source. Using a sunlight-like light source instead will make the underwater images look natural.

Chromatic adaptation is a study of change in the photoreceptive sensitivity of the human visual system under various viewing conditions, such as illumination [14]. The chromatic adaptation mechanism tries to eliminate the effect of the illuminant and has been widely applied in many fields [15,16,17,18]. In this paper, a chromatic adaptation based color correction method is proposed for underwater images and a comparison between this method and several conventional methods [18,19,20,21,22] is drawn. Example results obtained by the proposed method can be seen in Figure 1.

The contributions of this paper are as follows. First, a simple yet effective and robust color correction method for underwater images is proposed. Second, a comprehensive comparison between several typical white balance and color correction methods on underwater images is drawn. Third, several key issues to be concerned in the color correction of underwater images are discussed.

2. Related Work

Color correction is also known as white balance in image processing. Many popular existing automatic white balance algorithms are based on low level image features due to their simple concept and surprising performance [23], such as the gray-world (GW) and white-patch (WP) methods. The GW method works under the assumption that, given an image with sufficient color variations, the average of reflectance of a scene is achromatic [19,20]. The WP method, also known as the perfect reflector method, is based on the hypothesis that the brightest pixel in an image corresponds to an object point on a glossy or specular surface [20,24] which reflects the full range of light that it captures [18]. Consequently, the color of the brightest pixel is considered as the color of the light source and the brightest pixel will be assigned as the reference white point for white balance. The dynamic-threshold (DT) method [20] is another simple white balance method. It finds reference white points around the mean chromaticity coordinate and takes the ratios of the maximum luminance value to the mean RGB values of the white points as the channel gains.

Conventional white balance methods and histogram equalization methods are often directly used to correct the color cast for underwater images [2,7,25,26]. Considering the characteristics of underwater imaging, many specialized methods have been raised. To weaken the reddish cast produced by GW, an adaptive GW is developed by merging the global and local averages [27]. Two variants of shades of grey (SG) method [28,29] are also adopted to efficiently implement the color correction. Henke et al. [30] uses a binary depth map obtained by applying DCP [5] to GB channels to estimate multiple channel gains for different regions in underwater image. Ancuti et al. [31,32] blends the raw image with its color transfer image according to a weight map reflecting the desired level of correction. Li et al. [33] developed a learning-based weakly supervised color transfer model. Liu et al. [34] corrects the regional or full-extent color deviation via frequency-based color-tone estimation. Liu et al. [35] apply local surface reflectance statistical prior to the Retinex image formation model. A statistical-based (SB) method [21,36] is applied to compress the outliners as well as expand the mid-range, which has the effect of image enhancement due to the expanded dynamic range.

Most of these methods process RGB color channels separately, which is prone to introduce artifacts especially for underwater images due to the uneven attenuations in the three channels. Implementing color correction in color space other than RGB space is a good solution to this problem. Bianco et al. [37] performs the GW hypothesis in Ruderman opponent color space of Lαβ to correct the color of underwater images. Emberton et al. [38] proposed a chromatic adaptation based water-type dependent white balance (WTDWB) method which divides underwater images into three categories including blue-dominated, turquoise and green-dominated images. The illuminations of the turquoise and the green-dominated images are respectively estimated by applying WP and GW on the nonlinear RGB images. For blue-dominated images, white balance is not applied to inhibit the introduction of artifacts [38].

3. Method

The proposed method is based on the chromatic adaptation theory and the WP hypothesis. The brightest p percent of pixels are determined based on the luminance values. The average chromaticity coordinate of these brightest pixels is assigned as the color of the reference white point. To get an accurate average chromaticity, the chromaticity coordinates are calculated in a uniform chromatic space of CIE 1960 UCS space. Von Kries theory [39] is then applied to implement the color correction in CIE 1931 XYZ space. The flow of the proposed method is shown in Figure 2 and the details are as follows.

3.1. Chromatic Adaptation Theory

Based on the Von Kries theory [39], the tristimulus values [X_s Y_s Z_s]^T under the source illumination could be mapped to the tristimulus values [X_d Y_d Z_d]^T under the destination illumination as follows:

$$\left[\begin{array}{c}{X}_d\\ Y_d\\ Z_d\end{array}\right]=M^{-1}\left[\begin{array}{ccc}\alpha & 0& 0\\ 0& \beta & 0\\ 0& 0& \gamma \end{array}\right]M\left[\begin{array}{c}{X}_s\\ Y_s\\ Z_s\end{array}\right]$$

(1)

where M is a 3 × 3 matrix, which transforms the tristimulus values [X Y Z]^T in the CIE 1931 XYZ space to the LMS cone-like space. Several transforms have been derived from different data sets and different criteria [16]. The Bradford transform [40,41] is adopted in this paper and the matrix M is:

$$M=\left[\begin{array}{ccc}0.8951& 0.2664& -0.1614\\ -0.7502& 1.7135& 0.0367\\ 0.0389& -0.0685& 1.0296\end{array}\right]$$

(2)

In Equation (1), α, β, and γ are defined as:

$$\alpha =\frac{L_{dw}}{L_{sw}}, \beta =\frac{M_{dw}}{M_{sw}}, \gamma =\frac{S_{dw}}{S_{sw}}$$

(3)

where [L_dw M_dw S_dw] and [L_sw M_sw S_sw] are the two cone signals corresponding to the destination white points [X_dw Y_dw Z_dw] and the source white point [X_sw Y_sw Z_sw], satisfying:

$$\left[\begin{array}{c}{L}_{dw}\\ M_{dw}\\ S_{dw}\end{array}\right]=M\left[\begin{array}{c}{X}_{dw}\\ Y_{dw}\\ Z_{dw}\end{array}\right], \left[\begin{array}{c}{L}_{sw}\\ M_{sw}\\ S_{sw}\end{array}\right]={\rm M}\left[\begin{array}{c}{X}_{sw}\\ Y_{sw}\\ Z_{sw}\end{array}\right]$$

(4)

3.2. Transform from sRGB to XYZ

As described above, if the XYZ values of each pixel in an image and the XYZ values of the white point under the scene illumination are known, the XYZ values of this image under a known canonical illuminant could be obtained. For different RGB color spaces, such as sRGB, Adobe RGB and Prophoto RGB, the transforms from the RGB space to the XYZ space are different due to different reference primaries and illuminations [42]. Given an RGB image, it is difficult to judge the RGB color space in which the image is encoded since the color space information recorded in the exchangeable image file format header can be changed or lost after post-processing by using image editing software [43]. A learning-based method can be used to identify the color space for an arbitrary RGB image in the subsequent research. Here, we simply assume that the images are encoded in sRGB color space since sRGB is representative of the majority of devices on which color is and will be viewed [15]. For an image encoded in the sRGB color space, the CIE 1931 XYZ values can be computed from the RGB values in three steps [44]. First, the digital code values [R_nbit G_nbit B_nbit] are converted to the nominal range [0, 1] following Equation (5). Since the same operation will be performed on the three channels, the notation V is used to represent any of the three channels.

$$V{\prime }_{sRGB}=\frac{V_{n\mathrm{bit}}}{2^n-1}$$

(5)

where n represents the number of bits per channel, for example, n = 8 for 24-bit RGB images. Second, V′_sRGB is linearized, known as de-gamma correction, as follows:

$$\{\begin{array}{ll}{V}_{sRGB}=\frac{V{\prime }_{sRGB}}{12.92}\hfill & \mathrm{if} V{\prime }_{sRGB}\le 0.03928\hfill \\ V_{sRGB}={(\frac{V{\prime }_{sRGB}+0.055}{1.055})}^{2.4}\hfill & \mathrm{else}\hfill \end{array}$$

(6)

Third, the linear RGB values [R_sRGB G_sRGB B_sRGB] are converted to XYZ values using the following derived relationship:

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\left[\begin{array}{ccc}0.4127& 0.3586& 0.1808\\ 0.2132& 0.7172& 0.0724\\ 0.0195& 0.1197& 0.9517\end{array}\right]\left[\begin{array}{c}{R}_{sRGB}\\ G_{sRGB}\\ B_{sRGB}\end{array}\right]$$

(7)

3.3. Color Correction

To estimate the white point under the scene illumination, the XYZ values of the image are first converted to the uniform chromatic space CIE 1960 UCS by Equation (8) [45]. The average chromaticity coordinate of the brightest p percent of pixels, evaluated by the Y values representing the brightness, is computed and denoted by ($\overline{u}$, $\overline{v}$). It is taken as the chromaticity coordinate of the white point under the scene illumination in this paper and converted back to the XYZ space by Equation (9) getting [X_sw Y_sw Z_sw].

The CIE standard illuminant of D65 is taken as the destination illumination to keep consistent with the sRGB color space. That means [X_dw Y_dw Z_dw] = [0.9505 1.0 1.0888]. Thus, the XYZ values of the restored image, i.e., [X_d Y_d Z_d], can be computed by Equation (1), and the RGB values can be obtained based on the reverse relationship of Equations (5)–(7).

$$\{\begin{array}{c}u=\frac{4X}{X+15Y+3Z}\\ v=\frac{6Y}{X+15Y+3Z}\end{array}$$

(8)

$$\{\begin{array}{l}x=\frac{3\overline{u}}{2\overline{u}-8\overline{v}+4}\hfill \\ y=\frac{\overline{v}}{\overline{u}-4\overline{v}+2}\hfill \end{array}, \{\begin{array}{l}{X}_{sw}=x\cdot \frac{Y_{sw}}{y}\hfill \\ \begin{array}{l}{Y}_{sw}=1\\ Z_{sw}=(1-x-y)\cdot \frac{Y_{sw}}{y}\end{array}\hfill \end{array}$$

(9)

4. Results

Each of the compared method will be described briefly in turn. In the GW method, the ratios of half the RGB values of the white point under the canonical illumination to the mean RGB values of the whole image are taken as the channel gains. The WP method takes the ratios of the RGB values of the white point under the canonical illumination to the mean RGB values of the brightest 10% pixels evaluated by the sum of the RGB values as the channel gains. The SB method was coded exactly following the paper [21] with μ fixed as 2.3. We reproduced the color correction method proposed by Bianco et al. [37] without implementing the contrast improvement. In our method, p is set to 10.

Evaluating a restored image by comparing it with the ground-truth is hard to achieve since it is difficult if not impossible to obtain medium-free in situ images [46]. The restored images are usually evaluated subjectively or objectively based on color cards or objectively based on non-reference image quality metrics. However, the reliability of the existing non-reference image quality metrics is in doubt since they always produce results that are inconsistent with visual observation [1,8,46]. Therefore, only subjective assessment and color-card-based objective assessment, for images from a Haze-line dataset, are adopted in this paper.

4.1. Evaluation on UIEB Dataset

The UIEB dataset [8] is used to test the proposed method first. In the UIEB dataset, the reference images are manually selected by 50 volunteers from the potential reference images generated by 12 image enhancement methods [8]. Following [8], the proposed method is also test on the five categories of images: greenish and bluish images, downward looking images, forward looking images, low backscatter scenes, and high backscatter scenes. The results of different methods and the corresponding references are shown in Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7. Zoom in the images to see more details.

It can be seen from Figure 4 and Figure 5 that the proposed method does not show obvious advantages on the images with little color cast, which is consistent with expectations. From Figure 6, we can see that the proposed method could correct the color cast well though the restored image still does not look very good due to the unimproved contrast. The restored images of the total 890 images with reference images in the UIEB dataset are compared between the test methods. We found the GW, WP and SB [21] methods often cause artifacts, especially in the dark regions of the images with severe color cast, as shown in Figure 8. Bianco’s method [37] often over-grays the images making the color compressed or lost, as shown in Figure 4 and Figure 5, and Figure 9. The proposed method has consistently outperformed the other methods on the images with severe color cast. Many restored images obtained by the proposed method look even more natural than the reference images, as shown in Figure 10.

4.2. Evaluation on Haze-line Dataset

The results on Haze-line dataset [13] are shown in Figure 11. The result of haze-line method [13] was generated using code released by the authors. For evaluation, the mean angular error between the six grayscale patches of each chart and a pure gray color was computed as done in Reference [13]:

$$\overline{\psi }=\frac{1}{6}{\displaystyle \sum _{x_i}{\cos }^{-1}\left[\frac{I(x)\cdot (1,1,1)}{\Vert I(x)\Vert \cdot \sqrt{3}}\right]}.$$

(10)

Lower angular error indicates a more accurate color restoration. Many images in the Haze-line dataset contain multiple color charts at different distances from the camera. The charts are ordered by its distance to the camera and chart #1 is closest to the camera. The angular errors for each chart, image, and method are shown in

Table 1. Angular error (unit: degrees) in RGB space between the gray-scale patches and a pure gray color, for each chart in each image, and all methods. Lower is better. From left to right are raw underwater images, the results of gray-world (GW) method, white-patch (WP) method, statistical-based (SB) method [21], Bianco’s (GWLαβ) method [37], proposed method and haze-lines method [13].

Image No.	Raw	GW	WP	SB	GWLαβ	Ours	Haze-Lines
3008#1	31.41	26.87	27.51	9.85	7.42	4.67	2.55
3008#2	35.27	35.26	35.30	10.93	8.04	5.10	3.43
3008#3	35.27	35.26	35.31	11.41	8.04	5.14	3.07
3008#4	35.28	35.27	35.33	10.53	8.53	5.60	0.79
3008#5	35.28	35.27	35.34	11.74	8.36	5.56	0.71
3204#1	30.41	27.20	26.67	9.10	5.43	5.65	8.44
3204#2	28.61	21.09	23.25	9.24	5.04	5.30	12.46
3204#3	35.29	35.27	35.36	9.74	6.51	7.24	4.22
3204#4	35.30	35.27	35.37	11.58	6.25	7.30	7.65
3204#5	35.31	35.28	35.40	9.48	6.70	7.55	3.73
4485#1	33.49	31.15	29.44	8.67	6.56	2.54	5.84
4485#2	35.58	35.27	35.45	3.66	8.50	4.01	1.90
4485#3	35.59	35.27	35.43	9.75	7.42	4.06	2.12
4491#1	35.19	29.79	30.98	9.54	7.58	3.27	7.67
4491#2	35.49	35.28	35.39	5.68	8.12	3.24	5.06
4491#3	35.60	35.32	35.47	9.74	8.23	4.25	5.19
5450#1	29.68	27.30	23.99	13.58	5.90	4.27	6.12
5450#2	28.72	21.90	22.20	7.69	4.72	3.52	7.30
5450#3	32.65	31.28	29.44	10.16	5.16	3.90	5.46
5450#4	34.24	31.45	31.34	10.76	5.45	4.60	4.49
5450#5	35.58	35.37	35.48	7.42	6.89	6.23	4.78
5469#1	32.41	30.50	29.47	9.56	4.74	3.37	6.95
5469#2	35.56	35.33	35.48	8.03	5.49	4.81	3.32
5469#3	35.43	35.28	35.37	11.16	4.68	3.67	7.50
5469#4	35.59	35.33	35.52	6.97	5.51	5.23	3.06
5478#1	33.25	30.30	29.43	9.54	3.90	2.40	8.35
5478#2	33.27	30.30	29.43	9.58	3.89	2.41	8.33
5478#3	35.55	35.29	35.40	9.26	4.40	3.77	4.68
5478#4	35.74	35.34	35.57	7.00	4.97	5.23	4.95
Average	34.00	32.38	32.25	9.36	6.29	4.62	5.18

. The lowest average angular error is got by our method, as evident from Table 1.

4.3. Evaluation on Sea-Thru Dataset

Since Sea-thru dataset [7] only provides unprocessed sensor data stored in RAW photo formats, the software of Imaging Edge Desktop and Adobe Photoshop Lightroom was used to convert the ARW files to TIF and JPG files without any post-processing. No reference images are provided in the Sea-thru dataset [7], so only the results obtained by the test methods are given in Figure 12. It can be seen that the proposed method could remove the color cast well and it generates more colorful and realistic images than Bianco’s method [37].

5. Discussion

5.1. Artifacts Caused by Over-Enhancement

Conventional color correction methods which calculate a gain for each channel in RGB space, such as GW, WP, SB [21], and DT [20] methods, prone to over-enhance the severely attenuated channel resulting in obvious artifacts. The pixels with nonzero values in the severely attenuated channel tend to be over-enlarged, as shown in Figure 13 and Figure 14. The methods which implement the color correction in other color spaces, such as Bianco’s method [37], Emberton’s method [38], and the proposed method do not produce obvious artifacts. Therefore, it is a good solution to avoiding artifacts.

5.2. Linearization of RGB Images

The linear RGB images are generally non-linearized, known as gamma correction, in the image signal processing pipeline embedded in a camera’s hardware [47] to adapt the output characteristics of monitors. It is very important and necessary to correct the non-linearity of RGB images, known as de-gamma correction, to ensure the color correction is implemented in a linear RGB coordinate system [37]. For verification, the linearization process is removed from the flow of the proposed method, and several typical results are compared in Figure 15. We can see that artifacts are also produced in some images without linearization and the performance of color correction becomes slightly worse. However, the artifacts are slight compared with the conventional color correction methods, such as GW, WP, and SB [21].

5.3. Underwater Image Enhancement

To get in-air images of underwater scenes by image enhancement, color correction methods should be combined with contrast improvement methods. Figure 16 shows several enhanced images obtained by applying a simple contrast improvement method named contrast limited adaptive histogram equalization (CLAHE) [48] after the proposed color correction method. It can be seen that the color cast and contrast are both significantly improved. However, most of the existing contrast improvement methods are designed to deal with a special problem and some of them will also change the image color. It is still important to look for a more effective and robust underwater image enhancement method.

5.4. Evaluation of Restored Images

Since the inconsistency between the common non-reference image quality metrics and the visual observation has been observed [1,8,46], color-card-based evaluation methods are more reliable. However, it is not accurate enough only using the grayscale patches [7,13]. Taking advantage of the color patches on color charts could evaluate the color accuracy better [11].

6. Conclusions

A simple chromatic adaptation-based color correction method for underwater images is proposed in this paper. The underwater RGB image is first linearized to make its pixel values proportional to the light intensities arrived at the pixels, ensuring the color correction operation is performed in a linear space. The illumination is estimated in the uniform chromatic space of CIE 1960 UCS using the brightest 10% pixels based on the WP hypothesis. The chromatic adaptation transform is implemented in the device-independent color space of CIE 1931 XYZ. Experiments show that the proposed method is quite robust and could produce visually pleasing results while the other methods often introduce artifacts or over-enhancement. How to combine the color correction method with contrast improvement methods or model-based restoration methods to get the in-air images of underwater scenes will be the focus of follow-up work.