With the proposed model, a simulated underwater image can be easily generated by assigning values to corresponding parameters. However, considering the diversity of underwater image degradation problems in the real world, it is very hard to choose the right parameters in the simulation process. Therefore, we propose to adopt another strategy that first expands the value range of model parameters as much as possible, then selects simulated images according to their quality and reliability.
3.2.1. Underwater Image Simulation with Broadened Ranges of Parameter Values
In this subsection, we introduce our strategies for broadening the value ranges of model parameters in the simulation process. Since spatially varying illumination is not included in previous simulation works [
2,
19,
20], we especially proposed a method to effectively simulate variant illumination conditions in underwater scenes.
- (1)
Simulation of lighting parameters and
In real-world conditions, illumination of underwater images can be extremely complicated. Due to the absorption and scattering effects in the water body, incident light is often degraded before reaching the object, and thus brings initial color deviations to the underwater images. Furthermore, in the deep sea, artificial illumination is the only light source, and its non-uniformity often results in uneven brightness and limited visibility in the underwater images. To simplify the simulation of underwater lighting parameters, we use the combination of the following three basic patterns to approximate real-world underwater illumination: (a) uniform pattern with constant intensities for simulating shallow-water illumination, (b) parallel pattern whose intensities degrade with depth, and (c) artificial pattern that provides spatially uneven illumination. Diagrams and real-world examples of these patterns are presented in
Figure 2. For simplification, we also assume
and
are identical and use parameter κ to adjust the relative intensity of backscattering light.
The proposed formula for simulating underwater illumination is shown in Equation (9). In the right half of Equation (9), the first part is the intensity of uniform lighting (i.e., pattern (a)), the second part is the intensity of parallel lighting (i.e., pattern (b)), and the last part is the intensity of artificial lighting (i.e., pattern (c)). Here we use
and
to represent the initial intensities and weights of these patterns, and their values are randomly selected from [0.9, 1.0] and [0, 1], respectively, in the simulation process. The sum of all weight parameters is equal to 1.
The distribution of parallel lighting follows the Beer–Lambert’s Law. Therefore, its simulation requires the initial-light-intensity parameter, light-attenuation parameter, and light-traveling-distance parameter. We randomly picked a light-camera distance from [0, 2 m] and added it to the transmission distance to approximate the light-traveling-distance parameter. The simulation method of light-attenuation parameter will be covered in the following part. The simulation method of transmission distance is given afterwards.
The simulation of artificial lighting is the most complicated part. In real-world underwater conditions, the distribution of artificial lighting can be influenced by the number, position, orientation, and beam pattern of light sources, which makes it difficult to simulate quickly. For simplification, we fixed the orientation of the light sources to orthographic projection, and used 2D Gaussian distribution to approximate real beam patterns. The difference of arbitrary and fixed light modes is depicted in
Figure 3. Clearly, by using these simplifications, the simulation of imbalanced artificial lighting can be done in a much easier way.
In the parameter setting of artificial lighting simulation, the total number of light sources is selected from [0, 2] according to the character of the targeted illumination pattern. The beam pattern is approximated by a Gaussian distribution with peak value at and standard deviation proportional to the width of image by a random rate in [0.2, 1.1]. The distance from scene point to light source is calculated by , where is the height of the -th light source, and is randomly selected from [−1 m, 1 m], is the projection of the -th light source and is randomly picked from the pixel range, and is used to convert pixel distance to meter distance and is proportional to the ratio between the largest transmission distance and the image width, the scaling parameter of which is randomly selected from [0.1, 1]. To ensure the validity of light intensity, we use a scalar to adjust the intensity level, so that the highest value is equal to 1.
In
Figure 4, five sets of simulated underwater lighting parameters are presented, which shows the ability of proposed method in simulating diverse illumination conditions, including uneven illumination with artificial light source in use.
- (2)
Parameter setting of channel-wise attenuation coefficients
To cover a wide range of water types in the simulation, we use attenuation coefficients of Jerlov water types [
41] to generate values of channel-wise attenuation coefficients
, as in [
11]. The Jerlov water types include two sets, where types I–III correspond to open ocean water and types 1–9 correspond to coastal water. From types I–III to 1–9, the turbidity of the water body grows gradually, and the color deviation of underwater images varies from blue to green to yellow. To generate
from attenuation coefficient curves, a straightforward method is to pick one set of values at predefined response wavelengths as done in [
40], but considering the diversity of camera sensors, we expand the predefined wavelengths to nearby intervals, and randomly select wavelength values in each simulation. Then, we linearly combine the attenuation coefficients of adjacent Jerlov water types and multiply it with random scalars, so as to introduce additional variation and simulate in-between water types of these Jerlov water types. Mathematically,
where
represents the attenuation coefficient of Jerlov water type
(
corresponds to types I–III to 1–7, note that we do not include type IA and IB because they are very similar to type I.) at wavelength
(randomly selected within the
intervals of response wavelengths in [
40], i.e.,
= 620 nm,
= 540 nm, and
= 450 nm),
is the random weighting parameter within [0, 1], and
is the random scalar parameter from [0.9, 1.1]. Apparently, the proposed method can cover a wider range of attenuation coefficients than the Jerlov water type [
41] itself.
- (3)
Parameter setting of transmission distance
Instead of using RGB-D data to get paired
and
as in previous methods [
11,
28,
29], we use megaDepth [
42], a CNN model for single-view depth prediction, to estimate the initial transmission distance
from
, and randomly scale it to get
for the simulation process, mathematically,
where
,
, and
are random scalars that are picked from [0, 5 m], [2 m, 10 m], and [0.5, 2], respectively. In this way, the diversity of scenes and distance ranges of the simulated images will not be limited by the finite RGB-D datasets, and thus increases the number and diversity of simulated underwater images.
- (4)
Parameter setting of κ
As previously mentioned, is a scalar about the camera system and backscattering. In the simulation, it is set as with randomly selected from , so as to prevent overexposure while controlling the proportion of backscattering.
- (5)
Generating underwater image
After the simulation of all aforementioned parameters, underwater images can be generated by substituting corresponding parameters in Equation (8). In
Figure 5, we present some samples of simulated underwater images generated by the proposed method. Obviously, the proposed method is able to generate various degradation appearances for a given image, including images with deviated color, low contrast, and insufficient lighting.
Note that the diversity shown in
Figure 5 is actually lower than the diversity the proposed method can reach, because the proposed method uses continuous values in most parameters. However, higher variance also indicates higher possibility to produce unrealistic simulated images, which should be excluded from the training process for higher efficiency. In the next subsection, we propose a simple but efficient method for selecting images into the training sets.