1. Introduction
As one of the most direct approaches to perceive the world below the surface, optical images can provide plenty of useful information for various underwater applications, such as marine geology surveys, underwater mining, fishery and marine archaeology [
1,
2,
3,
4,
5]. However, compared with images from terrestrial environment, underwater images often suffer from severe degradation problems, which can impact their reliability and utility in underwater applications. The reason of underwater image degradation is in many aspects. The first one is the degradation caused by the water body. As shown in
Figure 1, light with different wavelengths is attenuated in different ratios in the water body. This uneven attenuation leads to ubiquitous color bias in underwater images. Suspended particles in water can also cause degradation in underwater images. As depicted in
Figure 1, particles near the transmission path from photographed scene to camera cause small-angle scattering (forward scattering) of incident light, and particles in surrounding environment induce ambient light into the camera lens by large-angle scattering (backscattering). These redirections of light lead to blurriness and hazy looks in underwater images. As the going deep of underwater explorations, artificial lighting devices are added to provide necessary illumination for dark deep-sea environment, as depicted in
Figure 1. Due to the limited range and inhomogeneity of artificial illumination, problems like dark background and bright spot are often seen in deep-sea images. Besides, if the light source is chromatic, the color balance of underwater images will also be affected. To facilitate the usage of underwater images, these degradation problems should be addressed.
To date, there have been many attempts to improve the quality of underwater images. According to recent literature reviews [
6,
7], they are usually classified into two main categories: underwater image enhancement and underwater image restoration. Methods of underwater image enhancement require no prior knowledge about the environment and mainly aim at improving the visual quality of underwater images. For example, in [
8], Iqbal et al. used histogram stretching in the RGB color space to restore the color balance, and the stretching of saturation and intensity in the HSI color space to improve the colorfulness and contrast of underwater images. In [
9], Ancuti et al. proposed a multi-scale fusion method that combined the result images of color correction and contrast enhancement by four weight maps about image luminance, contrast, chromatic and saliency. In [
10], Ghani and Isa reduced the color deviation in underwater images by using the characteristics of Rayleigh distribution, and improved the image saturation and contrast by stretching corresponding components in the HSV color space. In [
11], Li et al. proposed a weakly supervised color transfer method to correct color distortion in underwater images.
Underwater image restoration methods, on the other hand, attempt to recover the true scene radiances from degraded underwater images. These methods use models to analyze the mechanism of underwater image degradation, and restore the images by reversing the degradation process and using model parameters deduced via prior knowledges. In these methods, the simplified image formation model (IFM) is frequently used for its effectiveness and simplicity [
6], and due to its similarity to the model of outdoor hazy image, the Dark Channel Prior (DCP) from outdoor image dehazing [
12] is also widely introduced in methods based on this model. In [
13], Galdran et al. proposed the Red Channel Prior based on the DCP to recover the lost contrast in underwater images. This new prior reversed the red channel to deal with the strong attenuation of red light in the water body. In [
14], Drews Jr et al. proposed the Underwater DCP (UDCP) from the traditional DCP by excluding the red channel in producing the prior. Apart from the DCP-related priors, there are also other priors proposed for underwater image restoration. In [
15], Carlevaris-Bianco et al. proposed a prior by comparing the maximum intensity of the red channel to the maximum intensity in the green and blue channels over a small image patch. In [
16] and [
17], Peng et al. defined a new prior from image blurriness and used it to improve the quality of images from various underwater environments. There are also some other methods that combined the features of former two categories. For example, in the work of Hou et al. [
18], the UDCP is used together with quad-tree subdivision and Gamma correction to improve the contrast and saturation of underwater images. And in [
19], Qing et al. proposed a comprehensive method with adaptive dehazing and adaptive histogram equalization to remove the scattering and restore the color balance of underwater images. Usually for these methods, no strict classification is made.
From the study of the relevant literature, we have also noticed that most of the published works are designed for solving water-caused problems, i.e., color deviation and low contrast caused by the attenuation and scattering of light in the water body, while only a few of them have considered the degradation caused by artificial lighting. Moreover, in latter works, the study of lighting-caused degradation is also seemed to be limited in the range of local problems like bright spots [
13] or vignetting [
20], but more general problems like the influence to the distribution of color and brightness in the whole image range are rarely studied.
In this paper, we propose an underwater image enhancement method for different illumination conditions based on a new model of underwater image degradation. In the new model, illumination is included in the modeling of the single-pixel intensity, so its influences to local regions as well as the whole image range are covered in this model. The proposed method is composed of two components: color-tone correction and fusion-based descattering. The first component is based on a frequency-based color-tone estimation strategy. By changing its application range and using necessary modification filters, it can be used to correct the global color cast in uniformly-illuminated images and regional color cast in non-uniformly-illuminated images. The second component is used to solve the residual degradation problems that are related to the scene-camera distance. This component adopts a fusion strategy to enhance images under different states. Experiments on laboratory and open-water images of different depths and lighting conditions prove the effectiveness of the proposed method. According to qualitative and quantitative evaluation results, the proposed method can improve the color balance and contrast of underwater images, and restore the color accuracy and visibility of badly-illuminated regions in non-uniformly illuminated images.
The rest of this paper is organized as follows: In
Section 2, the enhancement of underwater images with different illumination conditions is studied theoretically based on the model study of underwater images with arbitrary illumination conditions. In
Section 3, the overall framework and individual components of the proposed method are introduced. In
Section 4, the proposed method is evaluated on images from shallow water, laboratory and deep sea images, with comparison to three state-of-the-art underwater image enhancement and restoration methods. And in the last section, conclusions of this work are presented.
2. Problem Formulation and Model Improvement
To study the solution of underwater image degradation problems caused by the water body and the light source, the Jaffe–McGlamery model [
21,
22], a general model of underwater image formation, is reviewed first. In the Jaffe–McGlamery model, the irradiance of a monochromatic underwater image is formulated as the linear combination of following three components: the direct component
, the forward scattering component
and the backscattering component
, i.e.,
where
represents an image point, and
is the wavelength of incident light. By weighing with the spectral response [
23] of the detector, the monochromatic irradiances of the whole spectrum are integrated and transformed into the pixel values of an image in the RGB color space, i.e.,
, where
is the spectral response of channel
,
, and
is a factor about the imaging system and the unit conversion from light irradiance to pixel intensity.
In the model of monochromatic image irradiance (i.e., Equation (1)), the direct component
corresponds to the irradiance that has been exponentially decayed after being reflected by the photographed scene, i.e.,
where
is the irradiance of incident light on the scene point,
is the reflectance of the scene point,
is the volume attenuation coefficient,
is the distance between the scene point and the camera, and
represents other parameters of the imaging system. According to [
22],
is a function of the light source irradiance and the attenuation along the transmission path from the light source to the scene. Using
to represent the beam pattern of the light source,
to represent the angle from the light source to the scene point, and
to represent the distance between the scene point and the light source, the incident irradiance
is calculated as
. The formula of
is given by
, where
is the angle from the camera to the scene point,
is the transmittance of the lens, and
is the F number of the camera of focal length
.
The forward scattering component
corresponds to the reflected light with small-angle scattering. It is calculated by convoluting the direct component with a point spread function
:
Here, “
” denotes convolution,
, where
and
are empirical constants,
represents the inverse Fourier transform, and
is radial frequency. According to [
22], a more accurate representation of
in Equation (2) should also include the forward scattering process, i.e.,
where
represents the more accurate representation of
.
The final component, backscattering
, corresponds to the light that enters the camera without being reflected by the scene. In its original form, it is calculated as the superposition of the illuminated volume elements weighted by the value of the volume-scattering function [
22]:
where
is the volume-scattering function,
is the irradiance received by the small volume element in layer
,
is the distance from the layer
to the camera, and
represents parameters of the camera system. According to [
22],
, like
of Equation (2), is a function of the source light and the attenuation along the source-to-scene path; mathematically,
, where
is the distance from the element in layer
to the light source. Forward scattering should also be included for a more accurate calculation of
.
is similar to
in Equation (2) and is calculated by
, where
is the angle from the camera to the element in layer
, and
is the sickness of the layer.
In [
24], a simplified formula for calculating backscattering in uniformly illuminated conditions is provided, which is
where
represents the backscattering of infinite distance. According to [
24], it is a function of incident irradiance, the reciprocal of volume attenuation coefficient and the full-angle integral of volume scattering function, i.e.,
.
Clearly, in the Jaffe–McGlamery model, water-caused degradation and the influence of light source are well covered, which makes it a good simulation tool for general underwater images. But for the task of underwater image restoration, its complexity hinders its usage. Instead, a simplified version, the IFM model, is more commonly used to restore underwater images, as mentioned in the former section. The formula of the IFM model is given as follows:
where
is the underwater image,
is the target image that represents scene radiance,
is the transmission map (TM) that equals to
, and
is the background light (BL). For IFM-based methods, the restoration of an underwater image is achieved by estimating and eliminating TM and BL via proper priors about the input underwater image.
By comparing the Jaffe–McGlamery model and the IFM model, it’s very clear that the IFM model is an approximation of the Jaffe–McGlamery model in the case of uniform illumination. In the IFM model, the product of
and
is replaced by
, and the original form of the backscattering component is replaced by the simplified formula in Equation (5). The forward scattering component and the other parameters are omitted because of their relatively low influence on the value of pixel intensity, and the integral of irradiances of the whole spectrum is simplified by using corresponding parameters of each color channel (i.e.,
and
) to keep the solvability of the model. Apparently, the IFM model can be invalid in the condition of inhomogeneous illumination, because the condition of using Equation (5) is not satisfied. Moreover, the estimated scene radiance can also be inaccurate due to the discrepancy of
from normal daylight. To improve the model accuracy while remaining its simplicity, we make a small change to the IFM model by adding a parameter to represent incident light. The modified model is given as follows:
Here
stands for the incident light of the scene that corresponds to
for
.
corresponds to the
in Equation (4). In the case of uniform illumination, it is simplified as
, as in the IFM model.
For the case of inhomogeneous illumination, the simplification of backscattering cannot be directly applied, but based on the assumption that light field changes gradually in space, the inhomogeneous illumination can be approximated as homogeneous in small regions. By applying the simplification of backscattering to these small regions, the above model is transformed into:
where
represents a small neighborhood of
that receives homogeneous illumination. Apparently, the case of homogenous illumination that is assumed in the IFM model is a special case of this new model.
Due to the uncertainty and spatial variance of and , priors for solving from the new model are hard to derive. However, since and mainly cause color deviation in the regional or full-image scale, their influence can be suppressed by the idea of white balance under the Gray World assumption. For an image with uniform illumination and invariant scene-camera distance, i.e., , white balance can be easily applied with a linear transformation whose general form is , because all the terms that result in color deviation in (i.e., , and ) are global constants in this model. In the general formula of linear transformation, and are abstract parameters that respectively represent the general effects of offsetting and scaling processes. These abstract parameters need to be concretized according to the color-deviation characters of to ensure an unbiased transformation result. For example, in the color-tone correction method presented in the next section, the offsetting ( and scaling ( processes are performed with a subtraction of the biased color tone and a classic histogram stretching, and the subtracted color-tone image is estimated on the basis of the frequency character of to approximate the true color-deviation condition of .
For an image with uniform illumination and variant scene-camera distance, i.e., , a similar linear transformation can remove the color deviation resulted from and , and partly from , because former two terms are spatial constants and the last term is not. For the case of non-uniform illumination, i.e., , the color cast caused by and can still be suppressed by linear transformation in region , based on the assumption that the illumination is uniform in . But in practice, the criterion of uniform illumination cannot be applied very strictly, because the Gray World assumption can be invalid in extremely small regions with colorful objects. So instead of suppressing the regional constants and , it is more practical to apply linear transformation by pixels, with corresponding parameters estimated from loosely-defined uniform regions and refined by smoothing filters to compensate the inter-regional differences.
After linear transformation, the global or regional color-tone deviation related to incident light is suppressed. To solve the residual problems about
, the result image is further transformed into the following form:
Apparently, this image is in a similar form as the IFM model. Assuming a good linear transformation, both
and
of this image are supposed to be free from color deviations caused by the incident light, i.e.,
is close to
and
is a constant. So the restoration of
can be applied by estimating and eliminating
(TM) and
(BL) from image
. But to ensure a good estimation of
, priors should be choosen carefully to fit the character of
. Moreover, a bad linear transformation is always possible, no matter how robust the transformation method is, so the situation of
being deviated from
should also be considered in the restoration process.