Research on a Measurement Method for the Ocean Wave Field Based on Stereo Vision

Featured Application

This technology can realize near-field wave load monitoring on ships and it can also provide support for short-term wave prediction.

Abstract

The wave parameter is an important environmental input condition. Traditional contact wave measurement methods are unable to meet the requirements of high precision, non-contact, and ship wave field assessment. Alternatively, stereo vision technology can realize a non-contact and mobile form of measurement. However, this technology suffers from poor timeliness and adaptability. This paper proposes a comprehensive wave measurement method that is based on stereo vision, wherein the gridding of siftGPU is used to achieve the fast matching of large images. The whole algorithm can be run within 6 s and it guarantees more than 20,000 feature-matching logarithms. Furthermore, by utilizing the least squares method and sea surface wave surface theory, the sea surface base level can be calculated without control points, along with the inversion of the sea wave parameters (wave height, period, and wave direction) and error point fitting. The rationality and superiority of the algorithm were verified through multiple comparison experiments. Compared with the Richard Brancker Research (RBR) wave height meter, the measurement error of the wave height is less than 10%, the period error is less than 0.5 s, and the wave direction error is less than 10° with the proposed method.

1. Introduction

Ocean waves indicate a classic and important research direction in the field of physical oceanography [1]. The wave parameter is also an important environmental input condition [2]. The directional spreading of ocean waves plays an important role in various aspects of ocean engineering, such as wave-induced loads, nonlinear wave evolution, and wave breaking [3]. Moreover, consistent sea state time series are essential for building coastal protection or offshore structures [4]. Current wave observation methods can be divided into two categories [5], the first of which is the “point” measurement method that is represented by wave buoys, which requires a fixed mooring line and is troublesome to deploy and maintain [6]. Another one is the “surface” measurement method that is represented by a synthetic aperture radar that obtains wave data depending on the selection of inversion parameters. It has a low measurement accuracy. Owing to such problems in the existing wave observation methods, research on non-contact, mobile and high-precision wave field measurement technology has become one of the important tasks for many countries [7].

Stereoscopic wave measurement technology is a wave measurement technology that uses two photographic cameras to synchronically collect images of the sea’s surface. It can directly extract the three-dimensional spatiotemporal distribution of the sea surface fluctuations through the principle of image matching and lens imaging and it obtains the direction spectrum [8]. This technology can effectively make up for the deficiencies of buoys and the remote sensing observation method. Therefore, the development of stereo vision wave measurement technology has important scientific significance and a practical application value for the theoretical development of ocean waves and their application in ocean remote sensing and ocean engineering [9].

The direct linear transformation (DLT) theory was proposed by Abdel-Aziz in 1971; it takes several or even ten days for each pair of images to be completely analyzed, thereby affecting the post-processing time [10]. In 2009, Vies et al. developed a stereo-photographic ocean wave measurement system and performed an experiment on the coast of Scheveningen [11]. They successfully measured the evolution of waves and they found that the measured sea surface area reached 1800 m² [12]. Furthermore, in 2011, Wanek developed a fully automated trinocular photography ocean wave observation system (ATSIS) [13]. Different from the other photography systems, ATSIS uses three synchronized cameras to form an image acquisition system in the shape of a “product” and it simulates the measurement scene through the previously recorded camera frame’s orientation in the laboratory in order to calibrate the external orientation elements. The ATSIS calibration process is cumbersome and the measurement area is limited by the camera’s frame, indeed the ATSIS measurement area is only about 10 m² [14]. In 2010, Cai Zhenghan used three synchronous cameras to form an image acquisition system in the shape of a “pin” [15] and then obtained the coordinates of the control point through the means of assessing the underwater makeup. It was necessary to set the calibration points with the current pool and the overall operation was complex [16]. In 2011, Yu Heng of Henan University captured a target image of the Yellow River model water flow vertically through a binocular system. By combining the improved Canny edge detection algorithm with the morphological connected domain segmentation method, he finally extracted and identified the river’s potential width and the flow velocity of the Yellow River model [17]. In 2012, Jiang Wenzheng, through his digital photogrammetry wave measurement system, reported a good study on the feature matching of left and right images [18]. Moravec’s corner detector is a corner detection operator that is based on gray variance that extracts the points with the maximum and minimum gray variance in four main directions as the feature points [19]. First, the Moravec corner points were extracted from the top layer of a pyramidal image. Then, the left and right images were extracted. Next, the corner points that were extracted from the top-level image of the pyramid were matched based on the principle of a maximum cross-correlation measure, thus obtaining the top-level matching pair set. In the matching pair sets, each matching pair was in the corresponding lower-level area of the pyramid and then matching that was based on a maximum cross-correlation measure was performed. After that, the matching sets of the left and right images were obtained. This feature matching method provides matching pairs as coordinates for the three-dimensional reconstruction of waves and, in this way, realizes sea surface wave measurement. However, the matched results that are obtained via this method have many mismatches. Accordingly, in the future, complex algorithms need to be introduced in order to eliminate the mismatches, alternatively algorithmically accurate wave heights can be introduced in the three-dimensional reconstruction [20]. In 2014, Cui Hao et al. achieved the acquisition of wave crest coordinates based on contour extraction technology, which reduced the influence of the acquisition accuracy on the two-dimensional coordinates of the wave crest in non-contact wave photogrammetry technology, but only the wave’s crest was studied [21].

This paper presents a comprehensive process algorithm of wave measurement that is based on stereo vision.

(1)

The matching part utilizes the gridding siftGPU method, wherein first the common area of the left and right cameras is determined and, then, grid zoning for the common area is carried out. The grid coordinates of the left camera are used to preliminarily locate the corresponding grid coordinates of the right camera and GPU multithreading processing is used in order to realize multi feature points and fast matching. Compared with the conventional siftGPU, the matching time with the proposed approach can be controlled below 6 s while the number of matching points can be maintained above 20,000, an outcome that is evidently much better than that of the traditional siftGPU algorithm.

(2)

In this paper, the least squares method and wave theory are used to normalize the sea surface-based plane calibration, plane error point fitting, and wave parameter inversion. The design’s method is simple and feasible. Compared with the traditional wave height meter, the wave height range is 0.2 m–1 m, the measurement error is less than 10%, and the periodic measurement error is less than 0.5 s.

(3)

The Hough transform and least squares method have been used herein to calculate the wave’s direction value. It was observed that, for a given wave image, the detection results and the original direction are basically the same.

The rest of this paper is organized as follows. Section 2 introduces the whole process of utilizing the algorithm of wave measurement that is based on stereo vision. Section 3 presents the experimental tests and results. Finally, the conclusions and directions for future work are given in Section 4 and Section 5.

2. Materials and Methods

2.1. Stereo Calibration

The purpose of calibration is to obtain a mapping model between the target that is in three-dimensional coordinate space and the two-dimensional image [22]. Essentially, the internal parameters, external parameters, and distortion coefficients that are obtained through this calibration are the basis of the image correction, parameter inversion, and three-dimensional reconstruction. This paper refers to the method that was proposed by Zhang Zhengyou [23] in order to achieve dual-objective targeting. The specific steps are listed as follows:

(1)

Construct a checkerboard calibration board with black and white grids, wherein the length and size of the black and white grids are 10 cm and 9 cm × 7 cm, respectively.

(2)

Use the binocular camera to take 25 images of the calibration board from different angles.

(3)

Use the algorithm to identify the pixel coordinates of the corner points (i.e., the black and white grid intersections) in the image for matching.

(4)

Calculate the inversion in order to obtain the internal parameters, external parameters, and lens distortion coefficients of the binocular camera.

2.2. Stereo Matching

2.2.1. Scale-Invariant Feature Transform (SIFT) Feature Extraction and Matching

The first part of the SIFT process is the production of a Gaussian scale space [24], in which each octave is computed by means of the convolution between a down-sampled image and different Gaussian kernels, such as $\mathrm{L}(\mathrm{x},\mathrm{y},\mathsf{\sigma })$ in Equation (1), where $\mathrm{G}(\mathrm{x},\mathrm{y},\mathsf{\sigma })$ is a variable-scale Gaussian kernel and $\mathrm{I}(\mathrm{x},\mathrm{y})$ is an input image.

Next, the second part includes a difference of Gaussian (DoG) scale space [25], this DoG is a close approximation of the Laplacian of Gaussian (LoG), as expressed in Equation (2). Compared with LoG, DoG has lower computational costs and uses approximately the same function to extract stable features.

$$\mathrm{L}(\mathrm{x},\mathrm{y},\mathsf{\sigma })=\mathrm{G}(\mathrm{x},\mathrm{y},\mathsf{\sigma })\times \mathrm{I}(\mathrm{x},\mathrm{y})$$

(1)

$$\mathrm{D}(\mathrm{x},\mathrm{y},\mathsf{\sigma })=\mathrm{L}(\mathrm{x},\mathrm{y},\mathrm{k}\mathsf{\sigma })-\mathrm{L}(\mathrm{x},\mathrm{y},\mathsf{\sigma })$$

(2)

The third part of SIFT contains the pixel and sub-pixel localizations of a keypoint [26]. The extreme pixel is compared with 26 neighboring pixels in a DoG scale space, which is chosen as the candidate pixel keypoint. Then, the localization of the sub-pixel keypoint is implemented using a quadratic approximation, which is computed using the second-order Taylor expansion.

The fourth part of the SIFT algorithm uses 128 dimension descriptors [27] to conduct the feature matching process. The SIFT feature descriptors represent the image gradient magnitudes and orientations in a 4 × 4 sub-region around the candidate feature.

Unfortunately, this strategy needs high computational power and memory consumption, which limit the use of SIFT in real-time engineering applications [28]. Notably, the above steps of the primal SIFT can run on a GPU in parallel fashion. It is shown that the performance of SiftGPU is nearly real-time for the images and the quality of the features that are extracted by SiftGPU is approximately the same as those that are extracted by the primal SIFT. However, since the GPU’s memory is limited, large images need to be down-sampled for size reduction before applying SIFT [29].

Due to the limitations of GPU and SIFT, a gridding siftGPU method is proposed in this work in order to extract and match the features from large images more efficiently.

2.2.2. Gridding SiftGPU Feature Extraction and Matching

First, we used several pairs of wave images to perform the image matching through the use of traditional SIFT, determine the common area of the left and right images, and determine the left image edge points $({\mathrm{u}}_{\mathrm{l}\mathrm{m}\mathrm{i}\mathrm{n}},{\mathrm{v}}_{\mathrm{l}\mathrm{m}\mathrm{i}\mathrm{n}})$ and right image edge points $({\mathrm{u}}_{\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{x}},{\mathrm{v}}_{\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{x}})$. Here, the common area of the left image is (${\mathrm{u}}_{\mathrm{l}\mathrm{m}\mathrm{i}\mathrm{n}}$ − ${\mathrm{u}}_{\mathrm{l}\mathrm{m}\mathrm{a}\mathrm{x}}$, ${\mathrm{v}}_{\mathrm{l}\mathrm{m}\mathrm{i}\mathrm{n}}$ − ${\mathrm{v}}_{\mathrm{l}\mathrm{m}\mathrm{a}\mathrm{x}}$), while that of right image is (0 − ${\mathrm{u}}_{\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{x}}$, 0 − ${\mathrm{v}}_{\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{x}}$), and the non-overlapping area is cropped.

Subsequently, we divided the common area of the left and right images into matching blocks. At this stage, a specific matching block size was selected according to the actual resolution of the image. It was assumed that the pixel value of a block in the left image was the same as the pixel value in the matching block in the right image. Meanwhile, the size of the areas corresponding to the same matching block are not equal and the pixel coordinates do not have a simple one-to-one correspondence. The calculation method is given in Figure 1:

Considering the camera distortion [30], there are:

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{i}\mathrm{l}}={\mathrm{x}}_{0\mathrm{l}}+{\mathrm{u}}_{\mathrm{i}\mathrm{l}}+{\mathrm{k}}_1({\mathrm{x}}_{0\mathrm{l}}+{\mathrm{u}}_{\mathrm{i}\mathrm{l}}){\mathrm{r}}_{\mathrm{i}\mathrm{l}}^2\\ {\mathrm{y}}_{\mathrm{i}\mathrm{l}}={\mathrm{y}}_{0\mathrm{l}}+{\mathrm{v}}_{\mathrm{i}\mathrm{l}}+{\mathrm{k}}_1({\mathrm{y}}_{0\mathrm{l}}+{\mathrm{v}}_{\mathrm{i}\mathrm{l}}){\mathrm{r}}_{\mathrm{i}\mathrm{l}}^2\end{array}\right.$$

(3)

In above formula, $({\mathrm{u}}_{\mathrm{i}\mathrm{l}},{\mathrm{v}}_{\mathrm{i}\mathrm{l}})$ are the pixel coordinates of the left image, $({\mathrm{x}}_{0\mathrm{l}},{\mathrm{y}}_{0\mathrm{l}})$ are the center coordinates of the left image, k is the distortion parameter of the left and right cameras, and r is the distance between the pixel coordinates and center coordinates, ${\mathrm{r}}_{\mathrm{i}\mathrm{l}}=\sqrt{{(\mathrm{x}+\mathrm{u})}^2+{(\mathrm{y}+\mathrm{v})}^2}$.

As shown in Figure 2, when it is assumed that the vector coordinates of OA are $\stackrel{\rightharpoonup }{\mathrm{r}}=({\mathrm{x}}_{\mathrm{i}\mathrm{l}},{\mathrm{y}}_{\mathrm{i}\mathrm{l}},-\mathrm{f})$ and that $\stackrel{\rightharpoonup }{\mathrm{o}}$ is the vector coordinates of OO’, then the normal vector of the plane is expressed as follows:

$$\stackrel{\rightharpoonup }{\mathrm{n}}=\stackrel{\rightharpoonup }{\mathrm{r}}\times \stackrel{\rightharpoonup }{\mathrm{o}}$$

(4)

Then the normal vector n in the right camera coordinate system is expressed as follows:

$${\mathrm{N}}_{\mathrm{r}}={\mathrm{R}}^{-1}\mathrm{N}$$

(5)

In above formula, $\mathrm{R}$ is the rotation matrix of the left camera and N represents the matrix corresponding to normal n. Therefore, the right camera pixel coordinates are:

$${\stackrel{\rightharpoonup }{\mathrm{n}}}_{\mathrm{r}}({\mathrm{x}}_{\mathrm{i}\mathrm{r}}\stackrel{\rightharpoonup }{\mathrm{i}}+{\mathrm{y}}_{\mathrm{i}\mathrm{r}}\stackrel{\rightharpoonup }{\mathrm{j}}-\mathrm{f}\stackrel{\rightharpoonup }{\mathrm{k}})=0$$

(6)

where f is the focus of the camera, N is the vector corresponding to matrix n, and $({\mathrm{x}}_{\mathrm{i}\mathrm{r}},{\mathrm{y}}_{\mathrm{i}\mathrm{r}},-\mathrm{f})$ are the coordinates of the right camera.

Considering the camera distortion, there are:

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{i}\mathrm{r}}={\mathrm{x}}_{0\mathrm{r}}+{\mathrm{u}}_{\mathrm{i}\mathrm{r}}+{\mathrm{k}}_1({\mathrm{x}}_{0\mathrm{r}}+{\mathrm{u}}_{\mathrm{i}\mathrm{r}}){\mathrm{r}}_{\mathrm{i}\mathrm{r}}^2\\ {\mathrm{y}}_{\mathrm{i}\mathrm{r}}={\mathrm{y}}_{0\mathrm{r}}+{\mathrm{v}}_{\mathrm{i}\mathrm{r}}+{\mathrm{k}}_1({\mathrm{y}}_{0\mathrm{r}}+{\mathrm{v}}_{\mathrm{i}\mathrm{r}}){\mathrm{r}}_{\mathrm{i}\mathrm{r}}^2\end{array}\right.$$

(7)

where k is the distortion parameter of the right camera, r is the distance between the pixel coordinates and the center coordinates, and $({\mathrm{x}}_{0\mathrm{r}},{\mathrm{y}}_{0\mathrm{r}})$ are the center coordinates of the right image.

In this way, the pixel coordinates of the left and right matching grids were obtained and the position of the left and right corresponding windows was confirmed.

After determining the corresponding matching blocks in the left and right images, we then performed multi-threaded siftGPU accelerated matching on the matching grids in order to achieve fast and accurate matching.

2.3. Sea Surface Base Plane Calibration

We then converted the image coordinates to camera coordinates and the specific method for this conversion is described as follows:

Assume that the coordinates of the wave target point in the left camera coordinate system ${\mathrm{O}}_{\mathrm{w}}-{\mathrm{X}}_{\mathrm{w}}{\mathrm{Y}}_{\mathrm{w}}{\mathrm{Z}}_{\mathrm{w}}$ are $({\mathrm{x}}_{\mathrm{w}},{\mathrm{y}}_{\mathrm{w}},{\mathrm{z}}_{\mathrm{w}})$ and that the coordinates of the corresponding image points on the left and right images are $({\mathrm{u}}_{\mathrm{l}},{\mathrm{v}}_{\mathrm{l}}),({\mathrm{u}}_{\mathrm{r}},{\mathrm{v}}_{\mathrm{r}})$. According to the conversion relationship, the formula for converting the left and right image points to the left camera coordinate system can be given as:

$${\mathrm{z}}_{\mathrm{c}\mathrm{l}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{l}}\\ {\mathrm{v}}_{\mathrm{l}}\\ 1\end{array}\right]={\mathrm{H}}_{\mathrm{l}}\times \left[\begin{array}{c}{\mathrm{x}}_{\mathrm{w}}\\ {\mathrm{y}}_{\mathrm{w}}\\ {\mathrm{z}}_{\mathrm{w}}\\ 1\end{array}\right],{\mathrm{z}}_{\mathrm{c}\mathrm{r}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{r}}\\ {\mathrm{v}}_{\mathrm{r}}\\ 1\end{array}\right]={\mathrm{H}}_{\mathrm{r}}\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{w}}\\ {\mathrm{y}}_{\mathrm{w}}\\ {\mathrm{z}}_{\mathrm{w}}\\ 1\end{array}\right]$$

(8)

$${\mathrm{H}}_{\mathrm{l}}={\mathrm{A}}_{\mathrm{l}}\times \left[\begin{array}{cc}\mathrm{E}& 0\end{array}\right]$$

(9)

$${\mathrm{H}}_{\mathrm{r}}={\mathrm{A}}_{\mathrm{r}}\times \left[\begin{array}{cc}\mathrm{R}& \mathrm{T}\end{array}\right]$$

(10)

In above formula, ${\mathrm{A}}_{\mathrm{l}},{\mathrm{A}}_{\mathrm{r}}$ are the internal parameter matrices of the left and right cameras, respectively; $\mathrm{R},\mathrm{T}$ are the rotation and translation matrices of the dual target, which can be obtained by calibration, and, then, ${\mathrm{H}}_{\mathrm{l}}$ and ${\mathrm{H}}_{\mathrm{r}}$ can be obtained. Expanding the above formula can provide the following formulas:

$$\mathrm{M}=\left[\begin{array}{c}\begin{array}{ccc}{\mathrm{u}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}31}-{\mathrm{h}}_{\mathrm{l}11}& {\mathrm{u}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}32}-{\mathrm{h}}_{\mathrm{l}12}& {\mathrm{u}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}33}-{\mathrm{h}}_{\mathrm{l}13}\end{array}\\ \begin{array}{ccc}{\mathrm{v}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}31}-{\mathrm{h}}_{\mathrm{l}21}& {\mathrm{v}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}32}-{\mathrm{h}}_{\mathrm{l}22}& {\mathrm{v}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}33}-{\mathrm{h}}_{\mathrm{l}23}\end{array}\\ \begin{array}{ccc}{\mathrm{u}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}31}-{\mathrm{h}}_{\mathrm{r}11}& {\mathrm{u}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}32}-{\mathrm{h}}_{\mathrm{r}12}& {\mathrm{u}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}33}-{\mathrm{h}}_{\mathrm{r}13}\end{array}\\ \begin{array}{ccc}{\mathrm{v}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}31}-{\mathrm{h}}_{\mathrm{r}21}& {\mathrm{v}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}32}-{\mathrm{h}}_{\mathrm{r}22}& {\mathrm{v}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}33}-{\mathrm{h}}_{\mathrm{r}23}\end{array}\end{array}\right]$$

(11)

$$\mathrm{U}=\left[\begin{array}{c}{\mathrm{h}}_{\mathrm{l}14}-{\mathrm{u}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}34}\\ {\mathrm{h}}_{\mathrm{l}24}-{\mathrm{v}}_{\mathrm{l}}{\mathrm{h}}_{\mathrm{l}34}\\ {\mathrm{h}}_{\mathrm{r}14}-{\mathrm{u}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}34}\\ {\mathrm{h}}_{\mathrm{r}24}-{\mathrm{v}}_{\mathrm{r}}{\mathrm{h}}_{\mathrm{r}34}\end{array}\right]$$

(12)

where $\mathrm{M},\mathrm{U}$ can be obtained from ${\mathrm{H}}_{\mathrm{l}}$, ${\mathrm{H}}_{\mathrm{r}}$, and the pixel coordinates $({\mathrm{u}}_{\mathrm{l}},{\mathrm{v}}_{\mathrm{l}}),({\mathrm{u}}_{\mathrm{r}},{\mathrm{v}}_{\mathrm{r}})$. Then the camera coordinates of the wave space point $({\mathrm{x}}_{\mathrm{w}},{\mathrm{y}}_{\mathrm{w}},{\mathrm{z}}_{\mathrm{w}})$ can be obtained through the following formula:

$$\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{w}}\\ {\mathrm{y}}_{\mathrm{w}}\\ {\mathrm{z}}_{\mathrm{w}}\end{array}\right]={({\mathrm{M}}^{\mathrm{T}}\times \mathrm{M})}^{-1}{\mathrm{M}}^{\mathrm{T}}\mathrm{U}$$

(13)

Hereafter, we converted the camera coordinates to the object coordinates; the specific method is as follows:

Select the calm sea images that match. The sea surface base plane equation is given as:

$$\mathrm{a}{\mathrm{x}}_{\mathrm{w}}+\mathrm{b}{\mathrm{y}}_{\mathrm{w}}+{\mathrm{z}}_{\mathrm{w}}+\mathrm{c}=0$$

(14)

In the formula, a and b are the normal vectors in the X and Y directions and c is the average distance from the camera to the sea surface base plane.

Through the transformation of the coordinate system, the coordinates of the feature points in the camera coordinate system are obtained as:

$$\mathrm{B}=\left[\begin{array}{ccc}{\mathrm{x}}_{\mathrm{w}1}& {\mathrm{y}}_{\mathrm{w}1}& 1\\ {\mathrm{x}}_{\mathrm{w}2}& {\mathrm{y}}_{\mathrm{w}2}& 1\\ \dots & \dots & \dots \\ {\mathrm{x}}_{\mathrm{w}\mathrm{n}}& {\mathrm{y}}_{\mathrm{w}\mathrm{n}}& 1\end{array}\right]$$

(15)

$$\mathrm{L}={\left[\begin{array}{cccc}{\mathrm{z}}_{\mathrm{w}1}& {\mathrm{z}}_{\mathrm{w}2}& \dots & {\mathrm{z}}_{\mathrm{w}\mathrm{n}}\end{array}\right]}^{\mathrm{T}}$$

(16)

The parameters of the sea surface base plane can be calculated using the least squares method as:

$$[\mathrm{a},\mathrm{b},\mathrm{c}]={({\mathrm{B}}^{\mathrm{T}}\times \mathrm{B})}^{-1}\times {\mathrm{B}}^{\mathrm{T}}\times \mathrm{L}$$

(17)

Normalizing the plane normal vector that was obtained from each image results in:

$$\stackrel{\rightharpoonup }{\mathrm{n}}=\frac{\mathrm{a}\mathrm{i}+\mathrm{b}\mathrm{j}-\mathrm{k}}{\sqrt{{\mathrm{a}}^2+{\mathrm{b}}^2+1^2}}$$

(18)

In the formula, $\stackrel{\rightharpoonup }{\mathrm{n}}$ is the normal vector of the sea surface base plane.

The spherical coordinate transformation is performed on the normal vector, which is convenient for calculating the changing relationship between the image coordinate system and the object coordinate system.

$$\mathsf{\phi }=\arccos (\mathrm{c}/\sqrt{{\mathrm{a}}^2+{\mathrm{b}}^2+1^2})$$

(19)

$$\mathsf{\theta }=\arctan (\mathrm{b}/\mathrm{a})$$

(20)

In these formulas, $\mathsf{\phi }$ and $\mathsf{\theta }$ correspond to the spherical coordinates of the sea surface base plane.

Then, the conversion of the rotation and translation matrices from the camera coordinate system to that object coordinate system is expressed as:

$${\mathrm{R}}_{\mathrm{w}}=\left[\begin{array}{ccc}\sin \mathsf{\phi }& -\cos \mathsf{\phi }& 0\\ \cos \mathsf{\theta }\cos \mathsf{\phi }& \cos \mathsf{\theta }\sin \mathsf{\phi }& -\sin \mathsf{\theta }\\ \sin \mathsf{\theta }\cos \mathsf{\phi }& \sin \mathsf{\theta }\sin \mathsf{\phi }& \cos \mathsf{\theta }\end{array}\right]$$

(21)

$${\mathrm{T}}_{\mathrm{w}}={\mathrm{R}}_{\mathrm{w}}{\left(\begin{array}{ccc}0& 0& -\mathrm{c}\end{array}\right)}^{\mathrm{T}}$$

(22)

Subsequently, the camera coordinates are obtained based on the sea surface base plane equation for the conversion from the camera coordinate system $\left({\mathrm{x}}_{\mathrm{w}},{\mathrm{y}}_{\mathrm{w}},{\mathrm{z}}_{\mathrm{w}}\right)$ to the object coordinate system $\left({\mathrm{x}}_{\mathrm{c}},{\mathrm{y}}_{\mathrm{c}},{\mathrm{z}}_{\mathrm{c}}\right)$. This conversion equation is:

$${({\mathrm{x}}_{\mathrm{c}},{\mathrm{y}}_{\mathrm{c}},{\mathrm{z}}_{\mathrm{c}})}^{\mathrm{T}}={\mathrm{R}}_{\mathrm{w}}{({\mathrm{x}}_{\mathrm{w}},{\mathrm{y}}_{\mathrm{w}},{\mathrm{z}}_{\mathrm{w}})}^{\mathrm{T}}+{\mathrm{T}}_{\mathrm{w}}$$

(23)

2.4. Wave Parameter Inversion

1.

Wave Height and Period

The center point of the matching block was selected as the wave inversion point. In order to ensure the accuracy of the calculation, the center point of the four matching blocks was selected in order to calculate the wave’s height and the changes in the wave data throughout 10 min was recorded. Next, the spectrum was obtained by the use of the Fourier transform and the main frequency range was selected. Following this, we selected the frequency of 0–0.5 Hz, filtered out the high-frequency part, and performed an inverse Fourier transform in order to obtain the time-domain curve corresponding to the main frequency. Then, we used the zero-crossing method to calculate the wave height and period.

2.

Wave Direction

The wave direction calculation method that was employed in this paper is as follows:

Essentially, the Hough transform is a feature extraction method. In this work, the Hough transform was performed on the left image and the specific steps are as follows:

• • Use a two-dimensional Gaussian function to the detect the extreme points. Obtain the coordinates of those extreme points $({\mathrm{u}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}})$.
  • Convert the planar coordinates $({\mathrm{u}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}})$ to polar coordinates ($\mathsf{\rho },\mathsf{\theta }$) by using:

    $$\mathsf{\rho }=\mathrm{u}\mathrm{c}\mathrm{o}\mathrm{s}\mathsf{\theta }+\mathrm{v}\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\theta }$$

    (24)
  • Calculate the maximum value with a two-dimensional accumulator $\mathrm{A}(\mathrm{a},\mathrm{b})$ as:

$$\mathrm{A}(\mathrm{a},\mathrm{b})=\mathrm{A}(\mathrm{a},\mathrm{b})+1$$

(25)

The camera resolution is ${\mathrm{M}}_1\times {\mathrm{M}}_2$ and, therefore, the parameter coordinate range is:

$$\mathsf{\rho }\in \left[-2\sqrt{{\mathrm{M}}_1}/2,2\sqrt{{\mathrm{M}}_2}/2\right]$$

(26)

$$\mathsf{\theta }\in \left[-\pi /2,\pi /2\right]$$

(27)

The maximum value corresponding to the accumulator in the statistical parameter coordinate system is $\left[\mathsf{\rho },\mathsf{\theta }\right]$. Record all of the pixel coordinates on the line.

Next, calculate the corresponding $\left({\mathrm{u}}_{\mathrm{i}\mathrm{h}},{\mathrm{v}}_{\mathrm{i}\mathrm{h}}\right)$ and $\left({\mathrm{x}}_{\mathrm{i}\mathrm{h}\mathrm{c}},{\mathrm{y}}_{\mathrm{i}\mathrm{h}\mathrm{c}},{\mathrm{z}}_{\mathrm{i}\mathrm{h}\mathrm{c}}\right)$. If ${\mathrm{z}}_{\mathrm{i}\mathrm{h}\mathrm{c}}\ge 0.95\times {\mathrm{z}}_{\max }$, calculate the normal vector of the straight line $\mathsf{\rho }=\mathrm{x}\cos \mathsf{\theta }+\mathrm{y}\sin \mathsf{\theta }$. On the other hand, if ${\mathrm{z}}_{\mathrm{i}\mathrm{h}\mathrm{c}}\le 0.95\times {\mathrm{z}}_{\max }$, select the corresponding coordinates to carry out the least squares fitting, so as to calculate the normal vector of the main peak line;

The normal vector of each straight line corresponds to two angles; thus it is necessary to determine the direction of the wave motion. In this paper, true north and counterclockwise rotation are considered as the positive direction when judging the change in the intercept of the wave crest line that was calculated in the frame images afterwards, so as to determine the wave movement direction.

3. Results

3.1. Laboratory Verification

The relevant experiment was carried out in the China Shipbuilding Research Center. The experiment used two cameras with a resolution of 6280 × 3158 in order to obtain the impact pair (shown in the Figure 3). The image shooting time was 17 January 2022. Moreover, a calibration box was used to check the system algorithm in order to measure its accuracy and the specific parameters are as follows:

• Calibration Results

The calibration results of the internal and external parameters of the camera are in

Table 1. System calibration parameters.

Code No.	Title 2
Left camera internal reference (pixel)	$\left[\begin{array}{ccc}\text{13,868.3}& 0& 2973.9\\ 0& \text{13,860.9}& 1614.1\\ 0& 0& 1\end{array}\right]$
Right camera internal reference (pixel)	$\left[\begin{array}{ccc}\text{13,911.8}& 0& 3047.1\\ 0& \text{13,903.6}& 1691.9\\ 0& 0& 1\end{array}\right]$
Distortion coefficient of left camera (pixel)	$\left[\begin{array}{cc}-0.11& -0.58\end{array}\right]$
Distortion coefficient of right camera (pixel)	$\left[\begin{array}{cc}-0.15& 1.18\end{array}\right]$
Common area boundary parameters (pixel)	$\left[\begin{array}{cc}580& 158\\ 5700& 3000\end{array}\right]$
Rotation matrix (pixel)	$\left[\begin{array}{ccc}0.9997& -0.0254& -0.0009\\ 0.0254& 0.9996& 0.0110\\ -0.0011& -0.0110& 0.9999\end{array}\right]$
Translation matrix (pixel)	$\left[\begin{array}{c}-267.56134\\ -1.75769\\ 15.63543\end{array}\right]$

:

2.

Gridding siftGPU Parameter Calculation

For the left picture, ${\mathrm{u}}_{\mathrm{l}\mathrm{m}\mathrm{i}\mathrm{n}}$ = 580 and ${\mathrm{v}}_{\mathrm{l}\mathrm{m}\mathrm{i}\mathrm{n}}$ = 158; whereas, for the right picture, ${\mathrm{u}}_{\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{x}}$ = 5700 and ${\mathrm{v}}_{\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{x}}$ = 3000. Meanwhile, the gridding matching block size was 300 × 300 and the number of matching blocks was 190. Here, we chose the left picture (580, 158) and the right image (0, 0) for verification and the matching results are provided in the following Figure 3.

Following this, siftGPU and gridding siftGPU were used to match image 1 and image 2 and the matching result is shown in the following Figure 4, Figure 5, Figure 6 and Figure 7.

It can be seen from Figure 4, Figure 5, Figure 6 and Figure 7 that the matching number of feature points that are based on the gridding siftGPU algorithm is obviously better than that of the points that are based on the siftGPU algorithm. The calculation results are shown in the

Table 2. Matching time and feature points numbers.

Code No.	siftGPU	Gridding siftGPU
Threshold	0.5	0.5
Image 1 matching time (s)	31.7	6
Image 1 matching no.	3003	24,424
Image 2 matching time (s)	29.9	5.8
Image 2 matching no.	2996	24,321

.

By comparison, the matching time of the gridding siftGPU was 1/6 of that of the siftGPU, whereas the match quantity that was based on the gridding siftGPU was 8 times that which was based on siftGPU.

3.

Calibration Results for the Base Plane

Twenty feature points were selected in image 2 in order to calculate the base plane. The selected area is the ground feature in image 2, which is shown in the Figure 8:

The code numbers and coordinates of the selected feature points are shown in

Table 3. Sea surface base plane calibration feature point coordinates.

Code No.	ul (mm)	vl (mm)	ur (mm)	vr (mm)
561	462.869	1400.559	117.233	1557.912
523	602.897	1604.144	244.895	1763.765
757	569.487	2124.842	181.769	2283.181
697	758.91	1509.639	404.889	1673.864
605	676.614	2716.894	254.76	2878.093
826	956.753	2342.16	553.65	2510.346
687	1266.115	1209.11	926.654	1386.103
377	1207.998	1668.579	843.952	1843.557
638	1228.33	2294.944	828.212	2470.034
892	1526.094	886.002	3618.396	2288.871
664	1388.562	1534.327	1030.845	1714.759
593	1378.192	1638.199	1014.254	1818.597
780	1326.869	2057.044	940.375	2235.672
806	1513.996	1317.925	1168.24	1501.776
815	1471.925	2253.082	1072.941	2435.356
867	4628.848	346.125	4336.211	608.107
872	4956.792	1102.291	4623.232	1372.855
553	4666.975	1550.444	4307.868	1814.572
339	4113.929	1368.649	3762.998	1618.408
927	4781.349	1701.514	4414.901	1968.805

.

The calculation parameters of the sea surface base plane are as follows:

a = 2.3715, b = −1.5122, c = −8428.2437

The calculation parameters of ${\mathrm{R}}_{\mathrm{w}}$ and ${\mathrm{T}}_{\mathrm{w}}$ are:

$${\mathrm{R}}_{\mathrm{w}}=\left[\begin{array}{ccc}-0.9983& -0.059& 0\\ -0.042& 0.717& -0.696\\ 0.041& -0.695& 0.718\end{array}\right]$$

$${\mathrm{T}}_{\mathrm{w}}=\left[\begin{array}{c}0\\ 5868.045\\ 6049.904\end{array}\right]$$

Additionally, the scatter diagram is shown in Figure 8.

As shown in Figure 9, the original 3D point images of image 1 and image 2 are consistent with the original image. The blue points represent the ground features and the yellow features represent the calibration box features. The white area is the unrecognized area, due to the strong light.

Next, 14 points were selected in order to verify the measurement accuracy of the system. The obtained measurement results were compared with the tape (the minimum scale was 1 mm) and the error was basically controlled within 1%. The results are shown in

Table 4. Measurement accuracy verification of the calibration box image 1.

Code No.	x (mm)	y (mm)	z (mm)	Calibration Box Width (mm)	Error
7	63.67009	483.8519	690.5644	690	0.08%
8	102.5964	349.7061	692.4865	690	0.36%
9	157.9489	−134.73	689.9507	690	0.01%
10	−127.546	750.4248	696.5345	690	0.95%
11	−127.597	745.3457	691.4332	690	0.21%
12	56.47901	491.2691	685.9605	690	0.59%
13	102.3608	353.218	695.7244	690	0.83%
14	−133.579	377.7314	691.3577	690	0.20%
15	92.31968	365.5422	690.5268	690	0.08%
16	232.3613	−295.85	688.9657	690	0.15%
17	114.8581	−181.829	690.5955	690	0.09%
20	42.98887	480.5747	692.0436	690	0.30%
21	−109.301	716.635	694.6232	690	0.67%
22	119.2851	316.8094	689.2604	690	0.11%

and

Table 5. Measurement accuracy verification of calibration box image 2.

Code No.	x (mm)	y (mm)	z (mm)	Calibration Box Height (mm)	Error
1	−164.961	424.689	1202.88	1204	0.09%
2	−101.668	237.8386	1202.787	1204	0.10%
3	−206.176	433.3536	1208.533	1204	0.38%
4	−155.174	375.2888	1202.448	1204	0.13%
5	−102.034	231.6729	1197.777	1204	0.52%
6	−250.352	385.6709	1207.963	1204	0.33%
7	−155.551	376.6541	1203.015	1204	0.08%
8	−133.514	398.3701	1200.575	1204	0.28%
9	−183.244	358.0353	1205.239	1204	0.10%
10	−229.913	410.1861	1208.479	1204	0.37%
11	221.4955	801.7397	1185.754	1204	1.52%
12	−194.046	434.6676	1207.884	1204	0.32%
13	−233.126	391.6223	1211.303	1204	0.61%
14	−171.595	490.7635	1206.244	1204	0.19%

.

3.2. Offshore Test Verification

The sea test was performed on the Qingdao Blue Valley No. 1 fixed platform on 19 January 2022. The image was captured simultaneously by two QHY26m industrial cameras and the lens that was used was a Nikon 50 mm/1.4D fixed-focus. The pixels of the QHY26M industrial camera were arranged to a value of 6280 × 3158, the pixel size was 3.76 um, and the height of the camera from the sea surface was about 10 m (relative to the tide level). The exposure time was set to 2 ms, the sampling frequency was 2 Hz, and the test position was the hatch of the platform. The test interval was between 14:00–16:00 and, in each observation, the data were collected for 10 min. The results of this work are compared with an RBR wave height meter’s data for verification. Furthermore, the test parameters are consistent with the laboratory verification parameters and the test pictures are shown in Figure 10:

The size of the field was ${\mathrm{X}}_{\min }=-1500 \mathrm{mm}$, ${\mathrm{X}}_{\max }=1000 \mathrm{mm}$, ${\mathrm{Y}}_{\min }=-1000 \mathrm{mm}$, and ${\mathrm{Y}}_{\max }=1000 \mathrm{mm}$. The coordinates of the center point (0, 0) were selected in order to verify the accuracy of the measured wave height and period. The measurement results are shown in the Figure 11:

As shown in the Figure 11, in the same time interval, the time–domain variation curve of wave height as measured at grid point (0, 0) with the proposed stereo vision wave measuring system is consistent with the time–domain variation trend of the wave height as measured by an RBR wave height meter. Moreover, the measurement results at grid point (−320, 0) are consistent with those at grid point (0, 0), in the same time interval.

The calculation results are shown in

Table 6. Measurement accuracy verification.

Time	Parameter	RBR	Stereo Vision	Error
14:00	1/3 wave height (m)	0.870	0.857	1.5%
	1/10 wave height (m)	1.160	1.107	4.6%
	1/100 wave height (m)	1.498	1.447	3.4%
	Average wave height (m)	0.556	0.552	0.7%
	Wave period (s)	6.167	6.615	0.45
15:00	1/3 wave height (m)	0.945	0.942	0.3%
	1/10 wave height (m)	1.203	1.220	1.4%
	1/100 wave height (m)	1.476	1.594	8.0%
	Average wave height (m)	0.615	0.622	0.1%
	Wave period (s)	6.903	7.054	0.35
16:00	1/3 wave height (m)	1.033	1.029	0.3%
	1/10 wave height (m)	1.309	1.300	1.4%
	1/100 wave height (m)	1.704	1.639	8.0%
	Average wave height (m)	0.689	0.716	0.1%
	Wave period (s)	6.407	6.728	0.32

:

As shown in the

Table 7. Data comparison for different measured coordinates.

Time	Parameter	Coordinate (−320, 0)	Coordinate (0, 0)	Error
14:00	1/3 wave height (m)	0.942	0.939	0.3%
	1/10 wave height (m)	1.220	1.218	1.4%
	1/100 wave height (m)	1.594	1.602	8.0%
	Average wave height (m)	0.622	0.625	0.1%
	Wave period (s)	7.054	7.253	0.35

, the wave height error that was based on binocular stereo vision is less than 10% and the period is less than 0.5 s, relative to the results of the RBR wave height meter.

The wave direction diagram is shown in the Figure 12:

As shown in Figure 12, the wave direction of the normal vector was basically consistent with the actual movement direction of the wave. The wave direction measurement error is ±10 degrees and the resolution is 1 degree. These findings imply that the measurement results are accurate. The calculation results are shown in

Table 8. Measurement accuracy verification of wave direction.

Time	Parameter	RBR	Stereo Vision	Error
14:00	Wave direction (degrees)	170	175	5
15:00	Wave direction (degrees)	171	177	6
16:00	Wave direction (degrees)	170	172	2

:

A 3D reconstruction of the wave field is shown in Figure 13:

As is evident from Figure 12, the proposed method can effectively realize the three-dimensional reconstruction of an ocean wave field.

The wave height value of the 3D inversion is basically the same as the measurement result of the RBR wave height meter.

4. Discussion

Based on the results that were achieved by utilizing stereo vision in wave-height measurement, the following conclusions have been drawn:

(1)

Relying on the actual sea trial measurement results, it can be confirmed that the stereo vision wave measurement method can enable the accurate measurement of wave parameters (wave height, period, and wave direction) [30]. This is because the stereo vision technology uses the parallax formula to calculate the distance to the measured object, while the height of the object can be measured through corresponding coordinate conversion. Through the calculation of the elevation value of a fixed area, the wave height and the period of the sea wave can be analyzed. Furthermore, the wave direction can be detected using the normal vector of the wave crest line in the image, therefore it is necessary to recognize and fit the line features.

(2)

In order to realize a real-time operation of the whole process algorithm, the richness of the wave field’s features, and the accuracy of the measurement results, it is necessary to use a high-resolution camera for the measurement. However, this results in a too long matching time and very high computer memory costs. In this regard, the gridding siftGPU algorithm that is proposed in this paper considers that the common area of the left and right images can be gridded without down-sampling and the number of feature pairs can be matched using GPU multithreading processing, hence the matching is faster.

At present, radar is used for onboard wave measurements, however radar measurement is expensive and it has a low level of accuracy [31]. The fixed-point measurement methods mainly include ADCP and the wave heigh meter, both of which have high requirements for the water’s depth and they are also troublesome to place and recover. ADCP needs to be installed on the ground, while the wave heigh meter requires mooring line fixation [32]. In addition, both of these methods can only realize single point measurement and the measurement results will degrade due to the process of wave breaking under strong wave conditions. In contrast, stereo vision wave measurement technology can realize a non-contact and high-precision measurement. The main limitation of the approach that is presented in this paper is the high requirements of computer memory and GPU. Besides this, the internal and external parameters of the camera need to be calibrated manually before the measurement is undertaken. The calibration parameters also need to be kept unchanged throughout the actual application.

Binocular stereo wave measurement technology is a promising field, nevertheless there are still some problems that need to be solved in order to fully enable this technology for future engineering application [33]. These include the system’s motion compensation and applicability to complex lighting conditions (rainy and foggy days). The pre-calibration part can form a standard library for the internal and external parameters of the camera, which can be called directly in the practical application, so as to adapt to different measurement distances. Under complex lighting conditions, the illumination adaptability of the algorithm needs to be further studied. The relationship between the illumination intensity and the best aperture and focal lengths can be trained by deep learning in order to realize all-weather automatic focusing [34]. The system needs to consider the motion compensation algorithm and hardware for the motion of the platform. Taking a ship as an example, rolling and surge will not affect the measurement accuracy of the system and the heave direction needs to be compensated for by the integration of the results of the acceleration measurement, such as rolling and pitch. For yaw, a three degree of freedom motion compensation platform needs to be developed for the hardware’s compensation. Furthermore, with the development of ship and marine intelligent chemical engineering, marine measurement technology that is based on vision will become more and more important.

5. Conclusions

This paper presents a comprehensive processing algorithm for wave measurement that is based on stereo vision.

(1)

Compared with the conventional siftGPU, based on 6280 × 4210 image processing, the matching time can be controlled below 6 s and the number of matching points can be kept above 20,000, indicating much better performance than that of the traditional siftGPU algorithm.

(2)

In this paper, the least squares method and wave theory have been used to normalize the sea surface base plane calibration, plane error point fitting and wave parameter inversion. This design method is simple and feasible. Compared with RBR wave height measurement, the wave height range is 0.2 m–1 m, the measurement error is less than 10%, and the periodic measurement error is less than 0.5 s.

(3)

In this paper, the Hough transform and least squares method were used to calculate the wave direction value. Because the RBR wave height instrument does not have the function of wave direction measurement, compared with the original wave image, the detection results and original direction are basically the same.