Seeing through Wavy Water–Air Interface: A Restoration Model for Instantaneous Images Distorted by Surface Waves

Abstract

Imaging through a wavy water–air interface is challenging since light rays are bent by unknown amounts, leading to complex geometric distortions. Considering the restoration of instantaneous distorted images, this paper proposes an image recovery model via structured light projection. The algorithm is composed of two separate parts. In the first part, an algorithm for the determination of the instantaneous shape of the water surface via structured light projection is developed. Then, we synchronously recover the distorted airborne scene image through reverse ray tracing in the second part. The experimental results show that, compared with the state-of-the-art methods, the proposed method not only can overcome the influence of changes in natural illumination conditions for WAI reconstruction, but also can significantly reduce the distortion and achieve better performance.

1. Introduction

Viewing an airborne scene through a wavy water–air interface using a submerged camera creates a virtual periscope, which is of great significance in both the military application field and marine biology research [1,2,3,4]. Unlike other underwater imaging systems, the difficulties in such imaging scenarios mainly come from the water–air interface (WAI). Water surface fluctuations are complex and random movements, which will lead to irregular geometric distortion and motion blur in the image, causing the distortion of airborne scenes. Removing such distortions from an instantaneous distorted image is challenging since the shape of the interface is not known a priori and must be estimated simultaneously with the real scene image.

In previous works, recovering images distorted by a wavy water surface mainly relied on high-resolution video streams [5,6,7,8,9,10,11,12]. These methods always require high complexity and large data sets, so they are difficult to apply to real-time observation scenarios. However, in application scenarios such as path planning and obstacle avoidance for underwater vehicles, aerial target detection, recognition and tracking, etc., recovery methods for instant images are particularly important.

Previous studies show that the distorted scene image can be effectively recovered by reconstructing the shape of the water–air interface [13,14,15,16,17,18,19,20,21,22,23,24,25]. Milder et al. [15,16] emphasized first the estimation of the shape of the distorted water surface and then reconstructing the scenes. They proposed a method to estimate the water surface by analyzing the sky brightness; in their image system, an upward-looking submerged camera was used to capture the panoramic above-surface scene. The incident light from above will gradually disappear as the normal surface deviates from the viewing line of sight. Assuming that it was completely dark underwater and the sky was uniform, the brightness of the sky thus determined the surface radial slope, and they used a harmonic wave model to estimate the water surface and inverse ray tracing to reconstruct a distortion-free image. Turalev et al. [17] studied the recovery method of water surface distortion, and then designed an experimental setup. They first utilized multiple illumination sources (red to illuminate the water surface and blue for the underwater object) to capture an image of the object and a glitter pattern of the water surface simultaneously. Then, the slope of the water surface was estimated using the glitter pattern [18]. Finally, the geometrical distortion effects of the underwater object image were eliminated by multiple accumulations of short-exposure images [19,20,21,22]. Alterman et al. [23] added an additional imaging sensor to measure the wavy water surface in real time. It works as an adaptive optics system but utilizing the sun as a guide star. The wavefront sensor consists of a pinhole array imaging the sun onto a diffuser plane directly behind it and a camera capturing the distribution of images of the sun on the diffuser plane. These pinholes have an extremely narrow field of view, so the water surface on their own line of sight is small enough to be assumed isoplanatic. Then, sampled normal vectors of the water surface were deduced from the position of each sun image through pinholes and used to estimate the water surface. Yoav Y. Schechner et al. [24] considered the problem of multi-view stereo through a dynamic refractive interface. They used multiple cameras along a wide baseline to observe a scene under uncorrelated distortions and recover sparse point clouds. R. H. Gardashov et al. [25] developed a method for recovering the single instantaneous images of underwater objects distorted by surface waves. They first determined an instantaneous shape of the wavy sea surface using the characteristics of the sun glints, and then corrected an image distorted by the wavy surface through reverse projection. However, it was only applicable to scenarios of monitoring from the sky, since there were no characteristics of sun glints when viewing the airborne scenes from underwater [15,23]. The above methods all attempted to first estimate the slope distribution of the water surface and then recovered the distorted images. However, several current methods of recovery of the water surface are not suitable for harsh application conditions, requiring special illumination settings or relying on special ambient illumination conditions (requiring the sun or uniform brightness in the sky). Furthermore, the accuracy of the estimation is also unable to meet the requirements.

In order to overcome the dependence of previous methods on natural illumination conditions and realize the distortion correction of instantaneously distorted images, an image restoration model based on structured light projection is proposed in this paper. Compared to previous approaches, our method does not require special natural illumination [15,16,17,23,25], multiple viewpoints [24] or a complex experimental setup [17,23]. It only requires a simple projection setup and an image of the distorted scene. The main contributions of this paper are as follows: (1) we propose a new image restoration model for instant images via structured light projection, (2) we introduce a WAI reconstruction algorithm based on structured light, (3) we analyze some of its limitations.

2. Optical Analysis of Imaging through Refractive Media

2.1. Snell’s Window

By imaging through the water–air interface using a submerged camera, one can observe the whole sky. However, it does not stretch ${180}^{\circ }$ from horizon to horizon, as it does above water. Instead, it is compressed into a circular area spanning approximately ${97.2}^{\circ }$, regardless of the observer’s depth. According to Snell’s Law [26,27], this occurs because light rays are bent when entering or exiting water. The shrunken sky (celestial hemisphere) seen by submerged observers is called Snell’s window (SW). SW is surrounded by a dark field that represents light that is entirely internally reflected from the sea and back to the observer from the underside of the water’s surface.

As shown in Figure 1, a submerged camera observes from underwater at 3D location $C$. $\mathrm{CM}$ and $\mathrm{CN}$ are the total reflection boundaries, $\angle \mathrm{MCN}=97{.2}^{\circ }$. The conical area formed by the rotation of $\mathrm{CM}$ and $\mathrm{CN}$ around the optical axis is the Snell cone, and the circular area on the water surface between them is Snell’s window, the boundary of which is the extinction boundary, and the outside is the dark field.

Assuming that the brightness value of the hemisphere space above the water surface is 1, the brightness value of the underwater scene is 0, and the energy loss of absorption and scattering by the water body is ignored. Figure 2 shows the normalized illuminance as a function of the height angle of incident light rays (blue) and the normalized illuminance as a function of the elevation angle of the field of view of the camera (red) when the water surface is still. According to Figure 2, it does not receive the irradiance from the sky at all when the elevation angle of the camera ${\theta }_{\mathrm{water}}\in (0^{\circ },{41.4}^{\circ })\cup ({138.6}^{\circ },{180}^{\circ })$. This occurs because, within this field of view, light rays are totally internally reflected from the sea and back to the observer from the underside of the water’s surface. In deep water, there is very little light coming from below and so this part is dark and shows no apparent color or structure. Therefore, the field of view of the underwater camera is constricted within the Snell window within ${\theta }_{\mathrm{water}}\in ({41.4}^{\circ },{138.6}^{\circ })$.

2.2. Optical Properties of Sea Water

The energy attenuation of light in water is mainly caused by the absorption attenuation of the water body and the scattering effect of suspended particles in the water. Studies have shown [28,29,30] that the transmission of light in water is caused by two independent physical processes, namely absorption and scattering, and the energy decays exponentially. According to the Beer–Lambert law, the radiance attenuation of monochromatic light can be expressed as

$$I_d=I_0\hspace{0.33em}{e}^{-c({\lambda }_i)\hspace{0.33em}·d},$$

(1)

where $I_d$ is the intensity of a radiation of wavelength ${\lambda }_i$ with the starting intensity $I_0$ after traveling the distance $d$ through a material with the attenuation coefficient $c({\lambda }_{\mathrm{i}})$. The attenuation coefficient $c({\lambda }_{\mathrm{i}})$ consists of the energy losses due to the absorption coefficient $a({\lambda }_i)$ and the scattering coefficient $b({\lambda }_i)$:

$$c ({\lambda }_i)=a({\lambda }_i)+b({\lambda }_i).$$

(2)

C. Smith et al. [31] performed a detailed measurement of the attenuation coefficient of light in clear seawater and gave the measurement data of the absorption attenuation coefficient and scattering coefficient of seawater as a function of wavelength. The absorption and scattering attenuation coefficients of seawater as a function of wavelength are shown in Figure 3. The results show that seawater displays the selective absorption of light of different wavelengths, in addition to the existence of scattering properties. As shown in Figure 3, it is not difficult to find that the transmittance of seawater in the blue–green band of the spectrum is relatively large, and the light energy attenuation is the smallest, which is called the “blue–green window”. Therefore, in structured light projection, a light source in this band can be selected for wavefront sampling to reduce the influence of light absorption and scattering on the WAI reconstruction.

3. Materials and Methods

3.1. Model Descriptions

The system model is shown in Figure 4, in which the projector first projects an adaptive and adjustable structured light pattern onto the water surface, and then the camera s acquires the distorted structured light image from the diffuser plane, while the camera v captures the airborne scene through the same WAI. The algorithm is composed of two separate parts. In the first part, an algorithm for the determination of the instantaneous shape of the water surface from structured light is developed. Then, we synchronously recover the distorted airborne scene image through reverse ray tracing in the second part.

3.2. WAI Reconstruction Algorithm Based on Finite Difference

In this section, we propose an algorithm for the determination of the instantaneous shape of the water surface from structured light. According to the law of reflection, we first calculate the WAI normals of sampled points using the location information of the feature points between the reference structured light image and the distorted structured light image. Then, the WAI shape is estimated utilizing the finite difference method.

3.2.1. Sampling of WAI Normals

The algorithm first takes advantage of the position information of the feature points of the structured light pattern to perform the quasi-periodic sampling of the wave surface to be measured. Figure 5 shows a simulation example of WAI sampling via structured light projection. Figure 5a is the preset structured light pattern, and Figure 5b is the projection on the water–air interface when the water surface is flat (note: in the daytime or moonlit night, no projection is formed on the water–air interface; a virtual image is introduced for the convenience of analysis). Given the system parameters, the location distribution of the WAI sampling points can be obtained using perspective projection transformation [32,33,34], as shown in Figure 5c.

The structured light pattern of blue–green stripes projected by the projector is reflected by the water surface, forming a distorted structured light image on the diffuser plane. This section solves the sampled normals of the WAI according to the relationship between the incident ray, the reflected ray and the normal vector.

As shown in Figure 6, the global coordinate system is established with the projected center $o^{\mathrm{pro}}$ as the origin, and the $z$ axis denotes the height above the projector. $p_k$ is the 3D location of the arbitrary feature point on the structured light pattern to be projected, and the corresponding projected ray is reflected by the WAI, at 3D location $q_k$, and then the reflected ray irradiates a spot on the diffuser ${\prod }^{\mathrm{difusser}}$, at 3D location. ${\widehat{N}}_k$ is the corresponding WAI normal, and ${s^{\prime }}_k$ is the corresponding spot when the WAI is flat.

The unit vector of projected ray ${\widehat{v}}_k^{\mathrm{p}}$ is always known, given the preset structured light pattern. As shown in Figure 5c, the set $\left\{q_k\right\}$ represented as $\left\{{q^{\prime }}_k\right\}$ is periodic when the WAI is still. Meanwhile, $\left\{q_k\right\}$ is quasi-periodic, having a perturbation to periodicity, when the WAI is wavy. Considering that the variations in the height of the water surface are small compared to the work depth of the system ($\mathsf{\Delta }h\hspace{0.33em}\hspace{0.33em}\ll \hspace{0.33em}\hspace{0.33em}{h}_0$), we have

$$q_k\approx (q_x,q_y,h_{ 0})=p_k+(h_{ 0}/c_k^{\mathrm{p}})\hspace{0.33em}·\hspace{0.33em}{\widehat{v}}_k^{\mathrm{p}},$$

(3)

where $h_0$ is the system height, which is the average underwater depth of the projector. The value can be determined in the field, using a pressure-based depth gauge. $c_k^{\mathrm{p}}$ is the propagation length of ${\widehat{v}}_k^{\mathrm{p}}$ at the z-axis. Moreover, the 3D location of the spot $s_k$ can be extracted from the distorted structured light image. We adopt the corner detection algorithm of Reference [35] for feature extraction and matching. Therefore, the vector of reflected ray ${\widehat{v}}_k$ is ${\widehat{v}}_k=s_k-q_k/‖s_k-q_k‖$. Using the vector form of the law of reflection at the water interface,

$${\widehat{v}}_k\times {\widehat{N}}_k={\widehat{v}}_k^{\mathrm{p}}\times {\widehat{N}}_k\Rightarrow ({\widehat{v}}_k^{\mathrm{p}}-{\widehat{v}}_k)\times {\widehat{N}}_k=0.$$

(4)

Here, $\times$ is the cross product. Using the axial components of the vectors ${\widehat{v}}_k^{\mathrm{p}}$, ${\widehat{v}}_k$ and ${\widehat{N}}_k$, Equation (4) can be converted into the dot product form,

$$\left[\begin{array}{ccc}0& -c_k^{\mathrm{p}}+c_k& b_k^{\mathrm{p}}-b_k\\ c_k^{\mathrm{p}}-c_k& 0& -a_k^{\mathrm{p}}+a_k\\ -b_k^{\mathrm{p}}+b_k& a_k^{\mathrm{p}}-a_k& 0\end{array}\right]·\left[\begin{array}{c}{N}_x\\ N_y\\ N_z\end{array}\right]=0,$$

(5)

where $a_k^{\mathrm{p}}$, $b_k^{\mathrm{p}}$, $c_k^{\mathrm{p}}$ are the $x$, $y$ and $z$ components of the vector ${\widehat{v}}_k^{\mathrm{p}}$, respectively. $a_k$, $b_k$, $c_k$ are the axial components of the vector ${\widehat{v}}_k$. $N_x$, $N_y$, $N_z$ are the axial components of the vector ${\widehat{N}}_k$. Equation (5) also can be phrased in matrix form

$$A{\widehat{N}}_k=0,$$

(6)

where

$$A=\left[\begin{array}{ccc}0& -c_k^{\mathrm{p}}+c_k& b_k^{\mathrm{p}}-b_k\\ c_k^{\mathrm{p}}-c_k& 0& -a_k^{\mathrm{p}}+a_k\\ -b_k^{\mathrm{p}}+b_k& a_k^{\mathrm{p}}-a_k& 0\end{array}\right].$$

(7)

The WAI normal ${\widehat{N}}_k$ is estimated by solving Equation (6): it is the null subspace of $A$. This process is repeated for each sampled point located at $q_k$. It yields a set of sampled vectors $\left\{{\widehat{N}}_k\right\}$ corresponding to the set $\left\{q_k\right\}$.

3.2.2. Reconstruction of the WAI

Assume that $h (x,y)$ is the height of an arbitrary sampled point $(x,y)$ of the water surface, and $Z$ is a 2D height field. The gradient of $Z$ can be given by

$$\nabla Z=\frac{\partial \hspace{0.33em}h\hspace{0.33em}(x,y)}{\partial \hspace{0.33em}x}\hspace{0.33em}\widehat{i}+\frac{\partial \hspace{0.33em}h\hspace{0.33em}(x,y)}{\partial \hspace{0.33em}y}\hspace{0.33em}\widehat{j}=Z_x(x,y)\hspace{0.33em}\widehat{i}+Z_y(x,y)\hspace{0.33em}\widehat{j},$$

(8)

where $Z_x(x,y)$, $Z_y(x,y)$ are the $x$ and $y$ components of the two-dimensional numerical gradient of the arbitrary sampled point $(x,y)$, respectively. $\widehat{i}$ and $\widehat{j}$ are the unit vectors in the $x$, $y$ axis directions, respectively. Therefore, we can obtain the first-order partial differential equation as follows:

$$\left\{\begin{array}{c}\frac{\partial \hspace{0.33em}h\hspace{0.33em}(x,y)}{\partial x}=Z_x(x,y)\\ \frac{\partial \hspace{0.33em}h\hspace{0.33em}(x,y)}{\partial y}=Z_y(x,y)\end{array}\right.,$$

(9)

The sampled normals $\left\{{\widehat{N}}_k\right\}$ corresponding to the sampled points $\left\{q_k\right\}$, estimated in Section 3.2.1, are known. Moreover, the normal vector of an arbitrary sampled point on the wave surface can be expressed as $\left[Z_x(x,y)\hspace{0.33em},\hspace{0.33em}{Z}_y(x,y)\hspace{0.33em},\hspace{0.33em}-1\right]$. For the gradient operator of the 2D discrete function $h\hspace{0.33em}(x,y)$, according to the finite difference theory [36], we use the central difference formula to approximate the first derivative. Therefore, Equation (9) becomes

$$\begin{array}{cc}\frac{\partial \hspace{0.33em}h\hspace{0.33em}(x,y)}{\partial x}=\frac{h\hspace{0.33em}(x_{i+1},y_j)\hspace{0.33em}-\hspace{0.33em}h\hspace{0.33em}(x_{i-1},y_j)}{\mathsf{\Delta }{x}_i+\mathsf{\Delta }{x}_{i-1}}& i=1,2\dots n;\hspace{0.33em}\mathsf{\Delta }{x}_i=x_{i+1}-x_i;\hspace{0.33em}\mathsf{\Delta }{x}_{i-1}=x_i-x_{i-1}\\ \frac{\partial \hspace{0.33em}h\hspace{0.33em}(x,y)}{\partial y}=\frac{h\hspace{0.33em}(x_i,y_{j+1})\hspace{0.33em}-\hspace{0.33em}h\hspace{0.33em}(x_i,y_{j-1})}{\mathsf{\Delta }{y}_j+\mathsf{\Delta }{y}_{j-1}}& \hspace{0.33em}\hspace{0.33em}j=1,2\dots m;\hspace{0.33em}\mathsf{\Delta }{y}_i=y_{i+1}-y_i;\hspace{0.33em}\mathsf{\Delta }{y}_{i-1}=y_i-y_{i-1}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\end{array},$$

(10)

where $m$, $n$ represent the dimensions of the grid of sampled points $\left\{{\widehat{N}}_k\right\}$. Noting that $H$ is the vector of length $\mathrm{mn}$ representing the height field $h\hspace{0.33em}(x,y)$ sampled on a $m\times n$ grid, the two vectors of length $\mathrm{mn}$ representing each component of the gradient field can be written as

$$\begin{array}{l}{G}_x·\hspace{0.33em}H=Z_x\\ G_y·\hspace{0.33em}H=Z_y\end{array}$$

(11)

where $G_x$ and $G_y$ are two sparse matrices, of size $\mathrm{mn}\times \mathrm{mn}$, defining the linear combinations of the elements of $H$ to produce each gradient. Equation (11) can be merged into a single linear system,

$$G\hspace{0.33em}·\hspace{0.33em}H={[Z_x,Z_y]}^{\mathrm{T}}=\Xi ,$$

(12)

where $G={[G_x,G_y]}^T$ is a rectangular sparse matrix of size $2\mathrm{mn}\times \mathrm{mn}$, and $\Xi$ is a vector of length $2\mathrm{mn}$. This system thus gives $2\mathrm{mn}$ equations with $\mathrm{mn}$ unknowns. It is over-determined, so a direct inversion is not possible. However, an estimate of $H$ may be obtained, by minimizing the residual [37],

$${‖G\hspace{0.33em}·\hspace{0.33em}H-\Xi ‖}^2\to \min ,$$

(13)

where $‖\hspace{0.33em}·\hspace{0.33em}‖$ represents the Euclidean norm. The WAI shape estimated by the numerical integration of the inverse gradient operator is sparse and discontinuous, whereas the WAI is typically smooth and integrable. Thus, we further perform the bicubic interpolation algorithm [38] for $H$ to estimate the WAI shape.

3.3. Image Restoration Algorithm through Ray Tracing

Component $V$ of the imaging sensor views the airborne scene through the wavy WAI, as shown in Figure 4. As the shape of the WAI is known, according to the principle of 3D camera imaging, this section proposes an image restoration algorithm based on inverse ray tracing. The principle of the algorithm is shown in Figure 7.

An internal coordinate system of $V$ consists of its optical axis and the lateral pixel coordinates of the image plane. The origin of the coordinate system of camera v is at the optical center $o^{\mathrm{lab}}$. The optical axis intersects the image plane at location $c$. The focal length of camera v is $f$. In this coordinate frame, the 3D location of the pixel $u$ is

$$u^{\mathrm{cam}}={[u^{\mathrm{T}},-f]}^{\mathrm{T}},$$

(14)

where $u$ is the 2D location of the pixel on the image plane, and $\mathrm{T}$ represents transposition. Relative to the global coordinate system, the pose of camera v is defined by a rotation matrix $R$ and a translation vector $t$. In the global coordinate system, the 3D location of the pixel $u$ can be expressed as

$$u^{\mathrm{lab}}=R^{\mathrm{T}}(u^{\mathrm{cam}}-t).$$

(15)

Let $u^{\mathrm{cam}}=0$ in Equation (15), and the origin of camera v in the global coordinate system is

$$o^{\mathrm{lab}}=-R^{\mathrm{T}}t.$$

(16)

In the global system, according to inverse ray tracing [31], the back-projected ray from $u^{\mathrm{lab}}$ through $o^{\mathrm{lab}}$ can be given by

$$I^{\mathrm{w}}(u)\equiv {\left\{o^{\mathrm{lab}}+{\widehat{v}}_{\mathrm{w}}l\right\}}_{\forall l\hspace{0.33em}>\hspace{0.33em}0},$$

(17)

where the ray direction vector is

$${\widehat{v}}_{\mathrm{w}}=(u^{\mathrm{lab}}-o^{\mathrm{lab}})/‖u^{\mathrm{lab}}-o^{\mathrm{lab}}‖,$$

(18)

where $l$ denotes the propagation length along the ray. In a perspective system, the inverse projected ray intersects the water–air interface at

$$q(u)=\mathrm{WAI}\cap I^{\mathrm{w}}(u),$$

(19)

where the WAI estimated in Section 3.2 is known. The WAI normal vector is $\widehat{N}$. According to the vector form of Snell’s law, the direction vector of the airborne observing ray is given by

$${\widehat{v}}_{\mathrm{a}}=n{\widehat{v}}_{\mathrm{w}}+\widehat{N}\hspace{0.33em}[\sqrt{1-n^2+n^2{({\widehat{v}}_{\mathrm{w}}·\widehat{N})}^2}-n{\widehat{v}}_{\mathrm{w}}·\widehat{N}],$$

(20)

where $n$ is the refractive index of water, $n=4/3$. ${\widehat{v}}_a$ is the undeflected direction vector pointing to the object. Supposing that the airborne rays $\left\{{\widehat{v}}_{\mathrm{a}}\right\}$ of all of the pixels in the distorted image intersect with a plane in the air ${\prod }^{\mathrm{object}}$, we can easily recover the distorted image through perspective projection.

4. Limitations

The limitations of system component $V$ are described in detail in Ref. [23]. Hence, we focus here on the limitations, sensitivities and resolution trade-off concerning only the structured light projection system $S$. Primarily, $S$ is not limited by natural illumination conditions. It is suitable for daytime as well as moonlit nights. Other sensitivities and limitations of $S$ are geometric, as analyzed next.

4.1. Sensitivity to Variations in $\widehat{N}$ for Structured Light

The WAI normal is perturbed around the $z$ axis. The direction vector of the reflected ray $\widehat{v}$ is obtained by the law of reflection,

$$\widehat{v}={\widehat{v}}^{\mathrm{p}}-2\widehat{N}\hspace{0.33em}(\widehat{N}\hspace{0.33em}·\hspace{0.33em}{\widehat{v}}^{\mathrm{p}}),$$

(21)

where ${\widehat{v}}^{\mathrm{p}}$ is the direction vector of the projected ray from the projected center through an arbitrary feature point. Substituting $\widehat{N}=-\widehat{z}$ into Equation (21) yields the direction vector of the reflected ray, when the WAI is flat, ${\widehat{v}}^{\prime }$. When the water is wavy, the reflected angle changes by

$$\psi =arccos(\widehat{\mathbf{v}}\cdot {\widehat{\mathbf{v}}}^{\prime })$$

(22)

The perturbation has two principal components. The first principal component is the meridional plane component, which is in the $xoz$ plane. In this component, the WAI normal $\widehat{N}$ rotates around the $y$ axis. Here, we analyze the sensitivity of structured light to changes in $\widehat{N}$ by taking the projected ray along the optical axis of the system as an example. Assuming that ${\theta }_{\mathrm{pro}}$ is the elevation angle of the projector, and the unit vector ${\widehat{v}}^{\mathrm{p}}={\widehat{v}}_{{\mathrm{o}}^{\mathrm{pro}}}^{\mathrm{p}}={[\cos {\theta }_{\mathrm{pro}},0,\sin {\theta }_{\mathrm{pro}}]}^{\mathrm{T}}$, the corresponding normal is

$$\widehat{N}={\widehat{N}}_{\mathrm{xoz}}={(\hspace{0.33em}\sin \theta \hspace{0.33em},\hspace{0.33em}0\hspace{0.33em},\hspace{0.33em}-\cos \theta \hspace{0.33em})}^{\mathrm{T}},$$

(23)

where $\theta$ is the inclination angle of the WAI. Another perturbation component is the sagittal plane, which resides in the $yoz$ plane. In this component, $\widehat{N}$ rotates around the $x$ axis; then,

$$\widehat{N}={\widehat{N}}_{\mathrm{yoz}}={( 0,\sin \theta ,-\cos \theta )}^T,$$

(24)

Substituting Equations (23) or (24) into Equations (21) and (22), we derive the angular perturbation of the reflected ray. Sensitivity to perturbations is assessed by $d\psi /d\theta$. Figure 8 shows the angle of deflection of the reflected ray as a function of the inclination of the WAI when ${\theta }_{\mathrm{pro}}={30}^{\circ },\hspace{0.33em}{45}^{\circ }$, respectively. The results reveal that the sensitivity of the structured light to the variations in the WAI normal is a constant value, namely $d\psi /d\theta =2$.

4.2. Resolution Analysis

In this paper, we perform quasi-periodic sampling for the WAI, utilizing the position information of the feature points of the structured light pattern, while the neighboring sampling interval is approximately equal to $D$ (Figure 9). Reducing $D$ enables the recovery of shorter WAI wavelengths. However, as we describe below, a shorter D decreases the angular resolution of $\widehat{N}$. The angular resolution of $\widehat{N}$ increases with ${\mathrm{z}}_h$. Here, we further analyze the relationship between the angular resolution of $\widehat{N}$ with the interval $D$ and ${\mathrm{z}}_h$ in the meridional plane, based on geometric optics.

Assume that the $x$ coordinate of the sampled point $q_k$ is $h_{ k}$, while $h_{ k+1}-h_{ k}\approx D$. The projected ray ${\widehat{v}}_k$ reflects by the WAI, and then irradiates a spot on the diffuser plane, at location ${}^{{S}}k$. The reflected angle can be given by

$${\theta }_k^{\mathrm{w}}=\arccos \hspace{0.33em}({\widehat{v}}_k·\hspace{0.33em}-\widehat{z})={\theta }_k^{\mathrm{flat}}+{\psi }_k=\arccos \hspace{0.33em}({{\widehat{v}}^{\prime }}_k·\hspace{0.33em}-\widehat{z})+{\psi }_k,$$

(25)

where ${\psi }_k$ is the angular deviation of the reflected ray (relative to the angle in flat water, ${\theta }_k^{\mathrm{flat}}$). ${}^{{S}}k$ can be calculated as

$$s_k=h_k+(h_0-z_h)\hspace{0.33em}\tan {\theta }_k^{\mathrm{w}},$$

(26)

where $h_{\hspace{0.33em}0}$ is the average underwater depth of the projector. As the projector follows a central projection, the reflected angles of the adjacent sampling points $h_{\hspace{0.33em}k}$ and $h_{\hspace{0.33em}k+1}$ can be expressed as

$${\theta }_k^{\mathrm{flat}}=\arctan \frac{h_{\hspace{0.33em}k}}{h_{\hspace{0.33em}0}},$$

(27)

$${\theta }_{\hspace{0.33em}k+1}^{\hspace{0.33em}\mathrm{flat}}=\arctan \frac{h_{\hspace{0.33em}k}+D}{h_{\hspace{0.33em}0}}.$$

(28)

Suppose that the projected ray ${\widehat{v}}_{k+1}$ corresponding to $h_{\hspace{0.33em}k+1}$ is parallel to the projected ray ${\widehat{v}}_k$ corresponding to $h_{\hspace{0.33em}k}$, and the corresponding location of the spot $s_{k+1}^0$ is

$$s_{k+1}^0=h_{\hspace{0.33em}k+1}+\frac{h_{\hspace{0.33em}0}-z_h}{h_{\hspace{0.33em}0}}\hspace{0.33em}{h}_{\hspace{0.33em}k}.$$

(29)

Actually, the location of the spot $s_{k+1}$ on the diffuser plane corresponding to $h_{k+1}$, according to the central projection, is given by

$$s_{k+1}=h_{\hspace{0.33em}k+1}+\frac{h_{\hspace{0.33em}0}-z_h}{h_{\hspace{0.33em}0}}{h}_{\hspace{0.33em}k+1}.$$

(30)

From Equations (29) and (30), we obtain

$$‖\overrightarrow{s_{{}_{k+1}}^0{s}_{k+1}}‖=\frac{h_{\hspace{0.33em}0}-z_h}{h_{\hspace{0.33em}0}}D.$$

(31)

Let the sensor $S$ determine $s_k$ with spatial uncertainty $\mathsf{\Delta }p$. This uncertainty may be due to the camera pixel size, diffuser characteristics and so on. This spatial uncertainty converts to uncertainties in the measured angular deviation $\mathsf{\Delta }\psi$ and the measured angular deviation of the inclination angle of the WAI normal $\mathsf{\Delta }\theta$,

$$\mathsf{\Delta }p=\frac{h_{\hspace{0.33em}0}-z_h}{{\cos }^2{\theta }_k^{\mathrm{w}}}\mathsf{\Delta }\psi \approx \frac{h_{\hspace{0.33em}0}-z_h}{{\cos }^2{\theta }_k^{\mathrm{flat}}}\frac{d\psi }{d\theta }\mathsf{\Delta }\theta ,$$

(32)

Given $\mathsf{\Delta }p$, ${\mathrm{z}}_h$, $h_{\hspace{0.33em}0}$ and $\mathsf{\Delta }\theta$, when the WAI is flat, it is easy to establish the correspondence of $h_k$ with $s_k$, since $s_k<s_{k+1}$. To maintain correct correspondence, this relation has to be satisfied also when the WAI is wavy, ${\forall }_k$:

$$s_k+\mathsf{\Delta }p/2<s_{k+1},$$

(33)

Equation (33) includes the small margin $\mathsf{\Delta }p/2$, which guarantees that the adjacent projected spots $s_k$, $s_{k+1}$ do not merge into a single spot. From Equations (25), (26), (32) and (33),

$$\begin{array}{rr}D+\frac{h_{\hspace{0.33em}0}-z_h}{h_0}D-\frac{\mathsf{\Delta }p}{2}& >(h_{\hspace{0.33em}0}-z_h)(\tan {\theta }_k^{\mathrm{w}}-\tan {\theta }_{k+1}^{\mathrm{w}})\hfill \\ \hspace{0.33em}\hfill & \approx \frac{h_{\hspace{0.33em}0}-z_h}{{\cos }^2{\theta }_k^{\mathrm{flat}}}({\psi }_k-{\psi }_{k+1})\hfill \\ \hfill \hspace{0.33em}& \approx \frac{h_{\hspace{0.33em}0}-z_h}{{\cos }^2{\theta }_k^{\mathrm{flat}}}\frac{d\psi }{d\theta }({\theta }_k-{\theta }_{k+1})\hfill \end{array}.$$

(34)

Therefore, when $h_{\hspace{0.33em}0}-z_h$ is too large or too small, $D$ limits the dynamic range of the WAI slope changes $\left[{\theta }_k-{\theta }_{k+1}\right]$; beyond this range, it will cause a spot detection error.

Assume that the WAI has spatial period $\lambda$, in which the WAI inclination varies as $\theta (x)=\mathsf{\Phi }\cos (2\pi x/\lambda )$, and $\mathsf{\Phi }>0$. In the worst case, at which $s_{k+1}-s_k$ is minimal, the WAI angle changes maximally between samples: $\left|{\theta }_k-{\theta }_{k+1}\right|=2\mathsf{\Phi }$. To avoid correspondence errors, the sampled interval D must satisfy

$$D>\frac{h_{\hspace{0.33em}0}}{2h_{\hspace{0.33em}0}-z_h}(\frac{\mathsf{\Delta }p}{2}+\frac{h_{\hspace{0.33em}0}-z_h}{{\cos }^2{\theta }_k^{\mathrm{flat}}}\frac{d\psi }{d\theta }2\mathsf{\Phi })\approx \frac{h_{\hspace{0.33em}0}}{2h_{\hspace{0.33em}0}-z_h}(\frac{\mathsf{\Delta }p}{2}+2\mathsf{\Phi }\frac{\mathsf{\Delta }p}{\mathsf{\Delta }\theta }).$$

(35)

As described in Section 3.2, the WAI sampling is quasi-periodic when the WAI is wavy, while the sampling tends to be periodic whose period is D around when the perturbation amplitude of the water surface becomes smaller. According to the Nyquist sampling criterion $(\lambda /2)>D$, combining it with Equation (35) yields

$$\frac{2h_{\hspace{0.33em}0}-z_{\mathrm{h}}}{h_{\hspace{0.33em}0}}\lambda >\frac{\mathsf{\Delta }p}{2}\frac{\mathsf{\Phi }}{\mathsf{\Delta }\theta }(8+\frac{2\mathsf{\Delta }\theta }{\mathsf{\Phi }}),$$

(36)

where $\mathsf{\Delta }\theta /\mathsf{\Phi }$ represents the relative angular resolution in which the WAI can be recovered: the ratio of uncertainty to dynamic range. Since $2\mathsf{\Delta }\theta /\mathsf{\Phi }\ll 8$, we obtain an uncertainty principle

$$\frac{2h_{\hspace{0.33em}0}-z_{\mathrm{h}}}{h_{\hspace{0.33em}0}}\lambda >\frac{\mathsf{\Delta }p}{2}\frac{\mathsf{\Phi }}{\mathsf{\Delta }\theta }(8+\frac{2\mathsf{\Delta }\theta }{\mathsf{\Phi }}).$$

(37)

As Equation (37) shows, the relative WAI slope angular resolution can be traded off for the spatial resolution $\left(\lambda \right)$ of the WAI, before errors stemming from aliasing and correspondence take effect.

5. Results

In the experiment, the proposed method was implemented in the MATLAB environment (MathWorks Co., Natick, MA, USA). This section first demonstrates the process of the WAI reconstruction and the image restoration. To verify the performance of the proposed method, we make a comparison with the state-of-the-art method, namely Alterman’s method [23], whose application scenarios and scopes are similar to those of our method. We tested the two methods with the same data set. The source codes and test data are available online in [39].

5.1. System Parameters

In this paper, we coded our image recovery scheme described above, which can simulate the process of WAI reconstruction and image restoration according to the system parameters. The system parameters including the projector, camera v for observing airborne scenes and camera s for capturing the structured light image are shown in

Table 1. System parameters of the projector, camera v and camera s.

System Parameters	Projector	Camera v	Camera s
CCD/LCD size	$4.1\times 4.1$ mm	$5.0\times 4.0$ mm	$5.0\times 4.0$ mm
Image resolution	$1000\times 1000$	$800\times 600$	$800\times 600$
$f$	4.2 mm	3.0 mm	2.0 mm
Rotation matrix	$\left[\begin{array}{ccc}0.6428& 0& -0.7660\\ 0& 1& 0\\ 0.7660& 0& 0.6428\end{array}\right]$	$\left[\begin{array}{ccc}0.8988& 0& -0.4384\\ 0& 1& 0\\ 0.4384& 0& 0.8988\end{array}\right]$	$\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$
Translation vector	$\left[\begin{array}{ccc}0& 0& 0\end{array}\right]$	$\left[\begin{array}{ccc}-96.17& 0& -46.90\end{array}\right]$	$\left[\begin{array}{ccc}-286.40& 0& 190\end{array}\right]$

. Moreover, the following parameters are included: $h_{ 0}=150\mathrm{mm}$, ${\theta }_{\mathrm{pro}}={40}^{\circ }$, $z_h=60\mathrm{mm}$.

5.2. WAI Reconstruction

5.2.1. WAI Simulation

The motion of ocean waves is a complex random process. Using a spectrum to describe ocean waves is one of the most effective means to study ocean waves, since the modeling process of the spectrum is based on a large number of actual observation data [40,41,42,43]. According to the Longuet–Higgins model [40], the height distribution of ocean waves can be expressed as

$$\eta (x,y,t)={\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{a}_{ij}\cos [{\omega }_{i}t-k_i(x\cos {\theta }_j+y\sin {\theta }_j)+{\epsilon }_{ij}]}},$$

(38)

where $\eta (x,y,t)$ represents the height distribution of a point $(x,y)$ on the WAI at time $t$, $a_{ij}$ is the amplitude of each harmonic, ${\omega }_i$ is the harmonic frequency, $k_i$ is the wave number of the harmonic, ${\theta }_j$ is the azimuth of the harmonic, ${\epsilon }_{ij}$ is the initial phase of the harmonic, and $M$, $N$ represent the sampling number of the frequency range of the wave spectrum and the sampling number of the azimuth angle, respectively.

The amplitude of the harmonics $a_{ij}$ can be expressed in the wave spectrum as

$$a_{ij}=\sqrt{2S({\omega }_i,{\theta }_j)d\omega d\theta }=\sqrt{2S({\omega }_i)\phi ({\theta }_j)\mathsf{\Delta }\omega \mathsf{\Delta }\theta },$$

(39)

where $S(\omega ,\theta )$ is the direction spectrum. $S(\omega )$ and $\phi (\theta )$ denote the spectrum and directional distribution functions of ocean waves, respectively. $\mathsf{\Delta }\omega$ and $\mathsf{\Delta }\theta$ are the sampling interval for frequency and direction angle, respectively. According to the linear wave theory [44], $k_i$ and ${\omega }_i$ meet the dispersion equation

$$k_i=\frac{{\omega }_i{}^2}{\mathrm{g}},$$

(40)

where $g$ is the acceleration of gravity.

There are various versions of the directional spectrum. We use the P-M spectrum and the directional distribution function suggested by ITTC (International Towing Tank Conference), whose expression is as follows [45]:

$$S(\omega ,\theta )=S(\omega )\phi (\theta ),$$

(41)

$$S(\omega )=\frac{8.1\times {10}^{-3}\times g^2}{{\omega }^5}\exp \left[-0.74\times {\left(\frac{g}{U\omega }\right)}^4\right],$$

(42)

$$\phi (\theta )=\frac{2}{\pi }{\cos }^2(\theta ),$$

(43)

where $U$ is the average wind speed at a height of 19.5 m above the sea surface. In Figure 10, we show the simulated ocean wave when $U=1.0 \mathrm{m}/\mathrm{s}$, $t=10 \mathrm{s}$.

5.2.2. Reconstruction of the WAI

An example of a full-system computer simulation is shown in Figure 11. A distorted structured light image was captured by camera s from the diffuser plane. The structured pattern projected by the projector is reflected by the WAI (Figure 10) and then forms the distorted structured light on the diffuser plane. Using the locations of feature points of the distorted structured light image and reference image, the WAI shape is estimated based on the finite difference method.

Intuitively, the reconstructed WAI is similar to the ground truth, but it exhibits a bias, as shown in Figure 11f. We use the root mean square error (RMSE) as the objective evaluation index. RMSE is given by

$$\mathrm{RMSE}=\sqrt{\frac{1}{RC}({\displaystyle \sum _{i=1}^R{\displaystyle \sum _{j=1}^C\eta (x_i,y_j)-\overline{\eta }}}}),$$

(44)

From Equation (44), the value of RMSE is 1.7845 mm. The absolute error distribution between the reconstructed WAI and the ground truth is shown in Figure 12. The unrecoverable bias is explained in Section 3.2.1. The research results show that the algorithm can reconstruct the wave surface when the error is allowed.

5.2.3. Comparative Analysis with Alterman’s Method

Alterman’s method [23] has strict requirements for natural illumination, is not suitable for cloudy weather and requires sunlight. Moreover, the accuracy of WAI reconstruction varies with the position of the sun. Here, we adopt Alterman’s method and our algorithm to reconstruct the same WAI (Figure 10), respectively. The relationship between the absolute error distribution of the reconstructed WAI and the ground truth with the incident angle of the sunray ${\theta }_{\mathrm{sun}}$ using Alterman’s method is shown in Figure 13. Figure 11 shows the RMSE and the maximum absolute error ($A{e}_{\max }$) as a function of the zenith angle of the sunray using Alterman’s method. Combining Figure 13 and Figure 14, it is obvious that the restoration accuracy is limited by the illumination conditions. As the zenith angle of the sunray increases, the maximum absolute error and the stability of the system decrease. Furthermore, as shown in Figure 14, the minimum mean square error of Alterman’s method is 3.4603 mm, while the RMSE of our algorithm is 1.7845 mm.

The research results show that, compared with the model included in Alterman’s method, the proposed algorithm can overcome the influence of changes in natural illumination conditions for WAI reconstruction and improve the accuracy of WAI restoration.

5.3. Image Restoration

The submerged camera images a checkerboard located at height $z_a=h_0+1000$ through the wavy WAI. The results are shown in Figure 15; the size of the image is $300\times 250 \mathrm{pixel}$. The scatter plot shows the coordinates of the corner points of the color-coded square checkerboards in the following three images, namely the ground truth (red), the distorted image (purple) and the recovery image (blue). In addition, the standard deviation (STD) of the corner positions in the distorted image is 26.2783 pixels, and the STD of the image restored by the estimated WAI is reduced to 1.2247 pixels. The results show that the image restoration method via structured light projection can significantly reduce the distortion.

5.3.1. Image Quality Metrics

We use three standard image quality/similarity metrics for quantitative evaluation: (1) mean square error (MSE) [2], (2) peak signal-to-noise ratio (PSNR) [46], (3) structural similarity index (SSIM) [47], in which the expressions of MSE, PSNR and SSIM are, respectively,

$$\mathrm{MSE}=\frac{{\displaystyle \sum _{i=1}^{\mathrm{w}}{\displaystyle \sum _{j=1}^h{\left[F(i,j)-I(i,j)\right]}^2}}}{w\times h},$$

(45)

$${\mathrm{PSNR}=10\log }_{10}\left(\frac{\max {(I)}^2}{\mathrm{MSE}}\right),$$

(46)

$$\mathrm{SSIM}=\frac{(2u_F{u}_I+c_1)(2{\sigma }_{FI}+c_2)}{(u_F^2+u_I^2+c_1)({\sigma }_F^2+{\sigma }_I^2+c_2)}.$$

(47)

where $I$ is the ground-truth image. $\max (·)$ denotes the maximum possible value. $u_F$ and $u_I$ represent the means of the image $F$ and the image $I$, respectively. ${\sigma }_F^2$ and ${\sigma }_I^2$ represent the variance of $F$ and $I$, respectively. ${\sigma }_{FI}$ is the covariance of $F$ and $I$. $c_1=(0.01\times L)$ and $c_2=(0.03\times L)$ are constants, in which L represents the dynamic range of pixel values.

5.3.2. Results of Quantitative Analysis

Figure 16 shows the results of the two methods to recover the distorted image, and the comparison results are presented in

Table 2. Comparison of image quality metrics among the proposed method, the distorted image and Alterman’s method [23].

	SSIM (H)			MSE (L)			PSNR (H)
	Data 1	Data 2	Data 3	Data 1	Data 2	Data 3	Data 1	Data 2	Data 3
Distortion	0.5584	0.6558	0.6140	0.1367	0.0653	0.0914	8.6414	11.8538	10.3925
Alterman [23]	0.6814	0.6784	0.6270	0.0520	0.0441	0.0514	12.8389	13.5571	12.8878
Proposed method	0.7630	0.7877	0.7434	0.0317	0.0461	0.0297	15.00	13.4651	15.2656

. The results show that compared with Alterman’s method, the proposed method has a significant improvement in performance indicators such as PSNR, MSE and SSIM, which proves the effectiveness of the algorithm.

6. Conclusions

The image restoration model via structured light projection is a novel approach for virtual periscopes. Different from previous methods, we do not require special natural illumination [15,16,23,25], multiple viewpoints [24] or image accumulation processes [17,18,19,20,21,22]; we only require a simple projection setup and an image of the distorted scene. This means that our method can be applicable to more scenarios—for instance, monitoring the habits of seabirds, path planning and obstacle avoidance for underwater vehicles, airborne target detection, recognition and tracking and seafloor mapping, etc.

In Section 5, we coded our image recovering scheme described above, which can simulate the process of WAI reconstruction and image restoration according to the system parameters. Compared with the state-of-the-art method, our method can overcome the influence of changes in natural illumination conditions for WAI reconstruction and improve the accuracy of WAI restoration. Furthermore, the results show that our method can significantly reduce the distortion and performs better in the recovery of distortions. In the future, we plan to conduct further tests in the laboratory and at sea to verify the effectiveness of our approach.

Similar to other instantaneous distorted image restoration algorithms [11,15,17,23,25], the proposed method still cannot eliminate the problem of loss of image details caused by random refraction of the WAI (marked in red in Figure 16). However, this problem can later be handled by image fusion algorithms [48,49,50,51] after images are corrected by our process. Rather than addressing the full-blown distortions in raw images, such video post-processing methods may handle more easily images whose distortions are residual.

In Section 4, the limitation of the algorithm is analyzed, and it is found that errors stemming from aliasing and correspondence will occur in structured light when the spatial frequency of the WAI is large. In the future, it is necessary to further explore a solution to this problem, such as structured light encoding [52], optical flow method [53], etc.