Long-Distance Sea Wave Sparse Matching Algorithm for Sea Level Monitoring System

Abstract

Benefiting from rich agricultural land, easy transport and fishing, etc., more and more people are moving to live in coastal areas, with more than 200 million people now living in coastal areas that are vulnerable to extreme sea level events. Sea level information is useful for coastal societies. Image measurement is rapidly developing as a new type of measurement tool. A multicamera-based sea level monitoring system along Japan’s coast near the Pacific is proposed in this paper, and a long-distance sea wave matching method for this system is described. The whole system employs multiple binocular vision systems to take sea surface images and obtain the sea level height based on the disparity between the field of views of the left and right cameras, forming a local measurement and overall analysis monitoring system. Sea level monitoring requires a high processing accuracy and speed to realize a timely response to extreme events. Thus, the paper extracts sea waves and integrates a sea wave’s appearance features as feature points and descriptors and pioneers the idea of searching deterministic features for fast image processing. The average stereo matching precision of the proposed method is up to 89.9% with a running time smaller than 40 ms for most pairs of images. Experiments on various real sea surface image pairs are conducted to validate the effectiveness of the proposed method.

1. Introduction

In recent decades, the pace of coastal development has accelerated, with investments worth billions of dollars [1,2,3]. Due to rising population levels and coastward migration, up to 800 million people will live in flood-prone areas by the 2080s [4]. Sea level change can result in great impacts on coastal populations and infrastructure [5,6]. Tidal knowledge is needed for port operation, efficient dispersal of pollutants, safe navigation, and for the efficient exploitation of the power of the tide for energy generation. Precise measurement of nontidal sea level change can guard against the risk of flooding due to tsunamis or storm surges, which are destructive natural disasters [7]. The sea level changes take place on timescales of hours to centuries, due to tides, air pressure, wind, local surface heating, and density effects [8]. The changes are different between low and high latitudes, shallow waters and the deep ocean, and some other local factors [9]. Thus, to understand such a complex natural phenomenon and make the most of it to serve us, it is clear that more measurement systems and models of it are necessary [10].

The sea level changes caused by tidal or climate change exhibit long-term and stable characteristics, whose measurement requirements are accuracy and long-term stability, with the integration of sea level every 6 or 15 min, or even hourly [8]. The measurement methods can be divided into two categories: coastal measurement and offshore measurement [11]. Tide poles [12], stilling-well gauges [13], and acoustic reflection gauges are traditional coastal measurement devices for tide level measurement. However, coastal currents and waves interacting with coastal topography can produce local gradients on the sea surface, making the measurement result higher or lower by several centimeters than the actual value, and the measurement region is also limited to several coastal sites [8].

For tsunami detection, ocean hydrodynamic behavior research, flooding warning, and some other applications, the sea level changes away from coast, even in the deep ocean, are required. Bottom pressure gauges [14] and buoys are the main instruments. In contrast to tide level measurement, tsunami detection or flooding warning system requires the rapid transmission of data to a center and a high-rate sampling of the sea level at typically once per minute or faster [15]. Some measurement and forecast systems such as the DARTII system [16] in the United States and the submarine cable system [17] in Japan are in practical use. These systems use a limited number of buoys or submarine cables to measure information of sea level, detect the occurrence of tsunamis, and forecast the arrival time and scale of each area. Since these systems do not measure tsunamis in each area in real time, there is a possibility that large prediction errors will occur [18], as in the case of the 3.11 Great Earthquake in Japan, where the earliest tsunami warning predicted that in Iwate Prefecture, the sea level height on shore would reach 3 m, but post-tsunami surveys showed that the average wave height was up to 11.8 m. This fact forced agencies in many parts of the world to consider whether the oceanographic infrastructure and information transmission were adequate for the purpose of tsunami detection and warning. The conclusion was that the infrastructure was incomplete, requiring investment in new equipment [8]. As tsunamis or floods are rare events, investment in their measurement equipment alone is not cost-effective; thus, a tsunami measurement system with long-term and stable sea level monitoring capability is targeted, which can be used for other applications such as tide monitoring, typhoon detection, and so on, when there is no tsunami.

To date, video monitoring technique represents a supplemental solution to the traditional approaches, with low installation and management costs, noncontact measurement and a high sampling of monitoring targets. Valentini et al., [19] described a new system for video monitoring of the Apulian coast, which could automatically perform image acquisition, processing, and dissemination of results on a web port. The system verified the efficacy and accuracy of video in coastal area monitoring compared with the traditional method. Wave research based on the stereo system started to become more common after a partially supervised 3D stereo system called Wave Acquisition Stereo System (WASS) was proposed [20]. A novel video observational system relying on various stereo techniques to reconstruct the 3D wave surface was developed [21,22]. Other seminal reconstruction methods adopted local methods to compute the disparity map [20,23,24]. In 2017, Bergamasco et al. proposed an open-source pipeline for the 3D stereo reconstruction of ocean waves [25]. They specifically described all the steps required to estimate dense point clouds from stereo images. The system was mounted 12 m above the mean sea level, covering an area of $85\times 65$ m${}^2$. Currently, most stereo systems for wave acquisition are used to compensate for the lost details of sea waves on a small scale, and the 3D reconstruction scope is limited to the sea area near the system.

The stereo vision system only uses two cameras as measurement instruments; it is easy to install and maintain, and noncontact measurement makes it robust enough for long-term measurement and survive an energetic event. Automatic image acquisition and processing allow it to calculate sea level rapidly enough to resolve the occurrence of extreme events. For these reasons, combining multiple stereo systems to develop a new, efficient, and automatic sea level monitoring system is feasible, and our laboratory started this research in 2013. Since the system acquires sea surface images directly, it can also be used for other applications such as typhoon monitoring and port transport management, etc.

It is well-known that at the origin of a tsunami, its wavelength is long and causing little change in sea level height [26]. Only when it is close to the coast will it cause a large sea level change. Thus, a sea level monitoring system with a tsunami warning function must have the ability to measure with high accuracy over long distance. Access to real-time sea level data is essential: for different subsea topography, a tsunami 20 km away takes about 15–30 min to reach the coast, if the sea-level measurement interval is smaller than 1 s, it can provide coastal people with 15–30 min of escape time before the tsunami comes ashore. Difficulties with existing sea level measurement networks concern whether their hardware and data transmission methods are suitable for tsunami monitoring, and whether they are complete enough [8]. In this paper, a much more complete and flexible sea level measurement system based on stereo vision is proposed, which scans the sea surface to expand monitoring coverage and carry a telescope to monitor the distant sea areas (up to $2\times {10}^4$ m away [27]). It aims to obtain long-term, stable, accurate, and real-time sea level information.

2. System Description

Figure 1 is the schematic of the proposed system. Each stereo vision part is connected to its local detection center to send back sea surface data by a fixed line; one stereo vision part takes 30 pairs of sea surface images within 1 s. A local detection center processes the received sea surface data from its connected stereo systems, calculates most of the sea waves’ heights to infer the sea level, then output the local sea level height to its local base station. The base station gathers sea level information from its connected detection center to analyze if there is a tsunami or other extreme event happening. In this monitoring system, the stereo system acts as the eyes to collect sea surface data, the local detection center is the computing tool to process data, and the local base station acts as the brain of the system to record sea level information and judge if an alert needs be released. Note that the system only releases the alert to the Japan Meteorological Agency (JMA) and utilizes its alert-releasing telecommunications system to warn people.

The system was deployed along the Japan coast near the Pacific. Figure 2 shows the deployment: 60 stereo systems, 17 detection centers, and 10 base stations were established. The distance between two adjacent stereo systems ranged from 25 to 40 km to ensure a coverage of the whole coast. The monitoring distance of the deployed stereo system was 20 km. To extend the monitoring coverage of the system, the cameras were set on a spin platform so that they could rotate left and right.

Figure 3 shows the proposed stereo system configuration. To meet the needs of long-distance monitoring, the baseline length of the system was up to about 30 m. As the figure shows, the left and right cameras were set on a spin platform to scan the sea surface and take photos through a control panel, then transmit these photos to a processing server to conduct the sea level height calculation. The control panel was used to rotate the cameras.

3. Method

In this section, we introduce the internal algorithms involved in the proposed system, including controlling the camera to acquire images, calibrating the camera to figure out its internal and external parameters, and stereo matching to achieve disparity.

3.1. Image Acquisition

The image acquisition system was implemented by a master–slave following mechanism, with a master camera scanning the sea surface and a slave camera following it to maximize the overlap of their field of views. It used a double closed-loop feedback mechanism to implement follow-up control in the slave camera, angle feedback for rough control, and image feedback for fine control. Figure 4 shows the control flow of the image acquisition system.

The captured images were copied and transmitted to the detection center for further processing, disjointing the image processing from the acquisition and guaranteeing a decentralized framework. This way, the acquisition phase could be preserved during components maintenance.

3.2. Camera Calibration

Existing calibration methods usually use a known standard plate to calibrate cameras, which is effective for close-range 3D measurement but when the monitoring distance is very long, it is difficult to make an appropriate standard plate. Thus, a calibration method [28] was employed to calibrate the intrinsic parameters and extrinsic parameters of the cameras over long distances, as well as the position relationship between the PTZ head center and the camera center. The calibration model is shown as in Figure 5. By the leveling device of a fixed-focal-length camera, we simplified the calibration process. Only the focal length of the camera and the rotation angle around the Y-axis were considered. $\alpha$, $\beta$, and $\gamma$ represent the angle between $OC$ and the X-axis, Y-axis, and Z-axis, respectively. We assumed that $\beta$ and $\gamma$ were equal to zero and the world coordinate original point was the original point of the left camera’s coordinates, the Z-axis of the world coordinate system was vertical to the baseline, the X-axis of the world coordinate system was parallel to the baseline, and the Y-axis of the world coordinate system was parallel to the Y-axis of the camera coordinate system, thus simplifying the perspective projection matrix as Equation (1).

$$Z_c·\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]·\left[\begin{array}{cccc}cos\alpha & 0& -sin\alpha & t_{x}cos\alpha -t_{z}sin\alpha \\ 0& 1& 0& t_y\\ sin\alpha & 0& cos\alpha & t_{x}sin\alpha +t_{z}cos\alpha \\ 0& 0& 0& 1\end{array}\right]·\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]$$

(1)

where $Z_c$ is the distance from the world point to the image center, and $t_x$, $t_z$ are the elements of the translation vector. For our supposed model, $t_x=0$, $t_z=0$ for the left camera, $t_x=b$, $t_z=0$ for the right camera, and b is the base distance between the left and right cameras. f is the camera’s focal length. ${\left[\begin{array}{ccc}x& y& 1\end{array}\right]}^T$ is the target coordinate of camera coordinate system and ${\left[\begin{array}{cccc}{X}_W& Y_W& Z_W& 1\end{array}\right]}^T$ is the target coordinate of the world coordinate system. $\alpha$ is the angle between the X-axis and the baseline. Combining the perspective projection matrices of the left and right cameras, we can achieve (2):

$$\left\{\begin{array}{c}{Z}_W=b/\{tan[arctan(x_2/f)-{\alpha }_2]-tan[arctan(x_1/f)-{\alpha }_1]\}\hfill \\ X_{W1}=Z_W\times tan(arctan(x_1/f)-{\alpha }_1)\hfill \\ Y_{W1}=y_1\times (Z_{W}cos{\alpha }_1-X_{W1}sin{\alpha }_1)/f\hfill \end{array}\right.$$

(2)

where b is the length of the baseline, $x_1$ and $x_2$ are the camera coordinates, and ${\alpha }_1$ and ${\alpha }_2$ are the rotation angles around the Y-axis. For two points in a captured image, the length between them can be calculated by the following Equation (3):

$$Le{n}_{i{i}^{\prime }}=\sqrt{{(Z_{Wi}-Z_{W{i}^{\prime }})}^2+{(X_{Wi}-X_{W{i}^{\prime }})}^2+{(Y_{Wi}-Y_{W{i}^{\prime }})}^2}$$

(3)

where $Le{n}_{i{i}^{\prime }}$ is the length between two points with world coordinates ($X_{Wi}$, $Y_{Wi}$, $Z_{Wi}$) and ($X_{W{i}^{\prime }}$, $Y_{W{i}^{\prime }}$, $Z_{W{i}^{\prime }}$). For a standard calibration board, we need to minimize the difference between the true value and the calculated value to solve unknown parameters, as in the following Equation (4).

$$min=\sum _{i=1}^n(Le{n}_i-Len)$$

(4)

Equation (4) has four unknown parameters, b, ${\alpha }_1$, ${\alpha }_2$, and f. To solve the problem of increasing deviation and difficult-to-detect corner points on a standard calibration board caused by extending the monitoring distance, a hollow plastic bar with 0.5 m red and white stripes was used to replace the standard board as Figure 6 shows; at the same time, a sequence of images was taken to calculate these parameters and the Levenberg–Marquardt method was used to optimize the solution.

3.3. Stereo Matching Algorithm

Tsunami/flooding forecasts need to locate the accurate site within a very short time, which requires the image processing of the system to be high-precision and fast, and stereo matching is its key step. However, for long-distance sea surface images, stereo matching suffers from three difficulties:

• Dissimilarity: the continuous nonrigid motion of sea water and long baseline length make the feature points in the right and left camera views not exactly alike.
• Indistinguishability: repetitive similar sea waves make it difficult to establish distinguishable descriptors.
• Computation load: to realize real-time monitoring of the sea level height, the computation load of the stereo matching must be decreased.

The frequently used techniques of stereo matching mainly include point-to-point matching [29,30,31,32], graph partition matching [33,34], matching by voting [35,36], and matching by deep learning [37,38,39,40].

For long-distance sea surface images, point-to-point matching is sensitive to noise. Compared with single-point matching, region detectors such as EBR [41], IBR [42], MSER [43], silent region [44], Harris affine, Hessian-affine [45] are much more stable. Thus, for the proposed system, we detected sea waves to perform matching.

Recently, stereo matching utilizing deep neural networks has achieved significant advances; these algorithms can be classified into two main categories: (1) supervised learning algorithms [46,47,48,49] and (2) unsupervised learning algorithms [38,50,51]. In [52], a Siamese network was built to generate sea wave feature descriptors and realize sparse stereo matching. Although the matching accuracy was sufficient, the time consumption was tremendous. We have also been working on stereo matching of sea surface images using a deep learning network. However, there are still some difficulties remaining unsolved, such as the lack of ground truth for supervised networks, the low accuracy of unsupervised network results, time and memory consumption, etc. Therefore, the paper still focused on stereo matching using traditional descriptors.

In this section, we propose a long-distance sea surface image sparse matching method based on deterministic features. Firstly, a sea wave’s appearance features were integrated to perform sparse matching and then the deterministic features were used to eliminate incorrect matching and realize fast matching and were searched by our defined condition. A whole flowchart is drawn to describe this process. The feature region extraction and appearance feature generation were based on [27]. Next, we introduce the appearance features, then give the selection criteria for deterministic features and introduce the method of searching deterministic features. It is a training process based on a manually matched training set.

3.3.1. Appearance Features

The epipolar constraint is usually used to reduce the searching region of a matching process. Thus, the sea wave location can be used as one of the discriminative features. A sea wave is a feature region, and we took its barycenter to represent its location. This was determined by the nature of the extraction algorithm: the same sea wave in one image pair was extracted by different thresholds, causing the extracted sizes, shapes, and edges to be different within one image pair.

For a specific sea wave, size, height, and width did not change a lot simultaneously within one image pair, though the sea wave was extracted with different thresholds. Furthermore, with a relatively accurate barycentric coordinates, the diagonal lengths (${45}^{\circ }$ and ${135}^{\circ }$) could be used to reflect the sea wave shape. The circularity could be used to measure the similarity between a sea wave and a circle. The brightness distribution could also be used as a discriminative feature. Figure 7 shows the description of these features.

3.3.2. Selection Criteria of Deterministic Features

We extracted sea waves within multiple image pairs, obtained two sea wave feature vector sets $L=\{x_1^L,x_2^L,...,x_n^L\}$ (superscript L indicates that the feature vector comes from left images) and $R=\{x_1^R,x_2^R,\dots ,x_m^R\}$ (superscript R indicates that the feature vector comes from right images). After randomly combining the elements in sets L and R, we obtained $N=m\ast n$ pairs of combinations $\{(x_1^L,x_1^R),(x_1^L,x_2^R),\dots ,(x_n^L,x_m^R)\}$. The ground truth of their matching results were stored in set $P=\{p_1,p_2,\dots ,p_N\}$, where $p_i$ equaled 0 or 1, representing incorrect and correct matching labels, respectively.

Table 1. Extracted sea wave features (part of train data).

No.	${\mathit{f}}_1$	${\mathit{f}}_2$	${\mathit{f}}_3$	${\mathit{f}}_4$	${\mathit{f}}_5$	${\mathit{f}}_6$	${\mathit{f}}_7$	${\mathit{f}}_8$	Match?
(1)	0.961	0.994	0.954	1.000	0.984	0.936	0.966	0.831	1
(2)	0.261	0.831	0.958	1.000	0.899	0.862	0.964	0.271	0
(3)	0.914	0.811	0.810	0.936	0.971	0.793	0.963	0.032	1
(4)	0.131	0.580	0.882	0.628	0.848	0.880	0.958	0.990	0
(5)	0.955	0.843	0.931	0.741	0.848	1.000	0.952	0.034	1
(6)	0.862	0.974	0.839	0.848	0.899	0.758	0.951	0.282	0
(7)	0.694	0.939	0.900	0.853	0.903	0.914	0.950	0.250	0
(8)	0.990	0.692	1.000	0.711	0.902	0.886	0.947	0.164	1
(9)	0.987	0.578	0.964	0.578	0.886	0.956	0.946	0.065	1
(10)	0.056	0.944	0.526	0.829	0.961	0.750	0.944	0.715	0
(11)	0.961	0.797	0.867	0.689	0.849	0.727	0.941	0.151	1
(12)	0.651	0.483	0.547	0.253	0.882	0.765	0.939	0.352	0

shows 12 pairs of extracted sea waves.

$f_i$ reflects the similarity of $(x_n^L,x_m^R)$ in each component, where a large value reflects a high similarity. Observing the data in Table 1, it is easy to find that some appearance features played a decisive role in the matching process, while some appearance features only worked occasionally. Figure 8 shows the matching result when we simply summed the similarities of each feature.

The left image shows the matching result, the black solid lines represent true positive matches (correct matches and detected by the matching method), red solid lines represent false positive matches (incorrect matches but detected by the matching method), and red dotted lines represent false negative matches (correct matches but not detected by the matching method). The right 8 images are the enlarged images of sea wave pairs which belong to false positive and false negative matches.

By observing these enlarged images, we found that when a false negative match happens (sea wave pairs $nos.$ 1, 2, 3, 4, 5, 6), at least two feature vectors had one component changing significantly. Taking the $no.$ 1 enlarged image pair as an example, in the left image, the sea wave can be taken as a whole sea wave, but in the right image, it splits into two waves, which may have been caused by different extraction thresholds, shooting angles, and times. Obviously, this would make the two feature vectors different in the height component ($f_4$) and cause a false negative match. When a false positive match happens ($nos.$ 7 and 8), the two matched feature vectors were much more similar with each other than other sea waves’ feature vectors. Taking the $no.$ 7 enlarged image pair as an example, in the left image, it is a small sea wave, but in the right image, it is connected with its right-side sea wave and forms a large sea wave, where there is no other sea wave around the $no.$ 7 sea wave. Thus, according to the Euclidean distance between two feature vectors, the two sea waves were matched and led to a false positive match. To avoid these two types of incorrect matching, we need a matching strategy that allows one or more than one component to be matched feature vectors despite having large differences, simultaneously avoiding false positive matching caused by the low similarity.

To realize fast matching based on a global analysis of these features, selectively robust to some nondeterministic features, and sensitive to some deterministic features, we propose a weighted selection algorithm. It is a training process based on a manually labeled training set. We propose that the deterministic features should possess the following characteristics:

• Entropy: a deterministic feature should significantly reduce the entropy of the training data set after matching by its split property;
• Stability: a deterministic feature should be stable in both correct and incorrect matching sets;
• Distinguishability: a deterministic feature must differ significantly in the correct and incorrect match sets.

We propose a two-step matching method. Figure 9 is the whole flow chart of the method. The information gain ratio was used to find the deterministic features and their split properties, so as to select the primary matching candidates. Then, we calculated the weight of the selected deterministic features to score each selected feature, eliminating false positives based on their scores. A “false positive” means an incorrect match that was selected as a candidate in the first step. Now, we can give a definition of a deterministic feature: it is a feature selected by the information gain ratio, and it can possess a large weight in the matching process. For sea wave matching, the deterministic features were not the only ones.

3.3.3. Searching Deterministic Features

We created a training set of 853 pairs of sea wave data (336 pairs of true positive matches; 517 pairs of false positive and true negative matches). The deterministic features had to have the capability for most of the training samples to be correctly categorized when used for matching. To measure this capability numerically, we introduced the information gain ratio (Equation (5)) to roughly choose the deterministic features.

$$InfoGainRation(P,f_k)=\frac{InfoGain(P,f_k)}{SplitInf{o}_{\left(f_k\right)}\left(P\right)}$$

(5)

where P represents the label set of the input training samples, 1 for correctly matched samples and 0 for others, and $f_k$ is the feature, $k\in \{1,2,\dots ,8\}$. For our feature vectors with a real physical meaning, we added some constraints to speed up the judgment threshold searching process:

(a) $f_1$ represented the location correlation degree. For a true positive match, it tended to 0. Thus, we did not need to find the split threshold within the range $(\delta ,+\infty ),\delta \gg 0$. In this paper, we chose $\delta =\frac{1}{2}(ma{x}_j\left|f_1^{x_j^L}F{f}_1^{x_j^R}\right|+mi{n}_j\left|f_1^{x_j^L}F{f}_1^{x_j^R}\right|)$.

(b) $f_2$, $f_3$,$f_4$, $f_6$, and $f_7$ ranged within (0,1]. They reflected the similarity of sea waves’ sizes and shapes. When the similarity was smaller than 0.1, obviously, it was incorrectly matched. Thus, we only searched the judgment threshold within [0.1,1]. Through these steps, we could initially filter out some combinations of deterministic features and their judgment thresholds. A decision tree used for initializing the primary matching results was established, as Figure 10 shows. The variable $f_{kB}^A$ in the circle indicates that the deterministic feature used to classify is $f_k$, the training sea wave number to be classified is A, and among all the training sea waves to be classified, the true positive matching number is B. The number under the circle is the judgment threshold. When the sea waves’ feature $f_k$ is greater than this number, it is classified into the decision node’s left subtree or subnode, otherwise into the right subtree or subnode. The square nodes are leaf nodes. One leaf node represents a matching result. The variables $N{o}_B^A$ and $Ye{s}_B^A$ in the leaf node indicate that the sea wave to be matched is classified as incorrectly matched (false positive and true negative) or correctly matched (true positive), respectively, and A and B have the same meanings as those of the decision node.

The red arrow path represents one of the judgment paths. It means that if the integrated feature vector $[f_1,f_2,f_3,f_4,f_5,f_6,f_7,f_8]$ of a pair of sea waves satisfies $f_1<0.09435,f_2>0.1268,f_5<0.4403,f_8>0.3676$, it is a true positive match.

Until now, we have selected some deterministic features and their judgment thresholds. With these, we can get preliminary matching results. These features meet the requirement of entropy, but there are still some problems, such as generous judgment criteria leading to increased mismatch rates. To eliminate mismatch, we added a weight for each selected feature and scored each preliminary matching results to remove incorrect matches with low scores. We defined the weights as follows:

$$w_i=\kappa {\mu }_{iP}exp\left|{\mu }_{iP}-{\mu }_{iN}\right|(\sum _i{\sigma }_{iP}-{\sigma }_{iP})$$

(6)

where $\kappa$ is the normalization constant, ${\mu }_{iP}$ is the average of all correct matched waves’ $f_i$, ${\mu }_{iN}$ is the average of all incorrect matched waves’ feature $f_i$; if the distance $exp\left|{\mu }_{iP}-{\mu }_{iN}\right|$ between ${\mu }_{iP}$ and ${\mu }_{iN}$ is greater, it means that the feature $f_i$ is more differentiated between correct and incorrect match sets. Thus, when a feature is more distinguishable, its corresponding weight is higher.

${\sigma }_{iP}$ is the standard deviation of all correct matched waves’ feature $f_i$. A smaller standard deviation means that the feature is less volatile and more stable. There are three main parts in the weight definition. The first is ${\mu }_{iP}$; it can measure the similarity of feature $f_i$ of the left and right camera views, the higher the value of this part, the higher the similarity and the higher the weight of this feature $f_i$. The second part is $exp\left|{\mu }_{iP}-{\mu }_{iN}\right|$, which means a large distance between two averages causes the weight of this feature to increase. The final part is ${\sum }_i{\sigma }_{iP}-{\sigma }_{iP}$; the more stable the feature is, the smaller the standard deviation is, and the weight of this feature is larger. This met our requirements on stability and distinguishability.

Now, we give a specific example to show how the proposed method matched the sea waves which were not matched by the simple similarity measurement described in former chapter. Taking the former false negative matched sea wave $no.$ 1 (in Figure 8, marked by green circle) as an example, we extracted its feature vector. At the same time, we also extracted its correctly matched sea wave’s (in Figure 8, marked by yellow circle) feature vector, as well as those of the sea wave in the red circle, and recorded them in

Table 2. Feature vectors of waves in green, yellow, red circles.

Color	${\mathit{f}}_1$	${\mathit{f}}_2$	${\mathit{f}}_3$	${\mathit{f}}_4$	${\mathit{f}}_5$	${\mathit{f}}_6$	${\mathit{f}}_7$	${\mathit{f}}_8$
Green	(293,462)	1808	49	60	0.49	38	46	3.8
Yellow	(455,463)	1209	52	33	0.71	39	42	2.4
Red	(837,672)	864	41	27	0.70	32	34	1.6

:

As the left and right images of sea wave $no.$ 1 were different (see Figure 8), from Table 2, we find that the extracted circularity ($f_5$) and heights ($f_4$) (Table 2, recorded in the gray block) of truly matched sea waves (Figure 8, sea waves in green and yellow circles) were much more different than falsely matched sea waves (Figure 8, sea wave in red and yellow circles). Matching by similarity took the false match as the final matching candidate (in Figure 8, as the sea wave in the green circle was connected to another sea wave, sea wave $no.$ 1 was left without a match to any other sea wave). By the proposed deterministic feature, the truly matched sea waves passed through the red arrow path (see Figure 10) and became one of the final matching candidates. Figure 11 shows the matching result.

Comparing Figure 11 with Figure 8, we find that the false negative matches ( as seen in $nos.$ 1, 2, 3, 4, 5, and 6), which had corresponding waves in the right image but were not detected by the matching method described in Section 3.3.1 were eliminated and correctly matched by our proposed method. Sea waves $nos.$ 7 and 8 were false positive matches by the matching method described in Section 3.3.1. Sea wave $no.$ 7 did not have a corresponding sea wave in the right image, as it was removed from the final matching result by our proposed method; sea wave $no.$ 8 had a corresponding sea wave in right image, and it was correctly matched. From the matching result, we can conclude that most false negative matches and false positive matches could be removed by our proposed method.

3.4. Sea Level Height Measurement

Many factors cause sea level changes; ocean tide and climate change cause the sea level to have a long period and regular change, while tsunami, storm surges, air pressure and wind cause the sea level to exhibit short-term and irregular changes [8]. Comparing the measurement sea level height with the predicted height according to the tide law, we could infer the scale of other influences. We extracted many points on the sea surface, some were located on the top of sea wave, some were located on the bottom, and some were located in the middle, as Figure 12 shows. Assuming that the points followed a normal distribution, we calculated the average of the height of all the points as the sea-level height according to the following Equation (7):

$$h=\frac{{\sum }_{i=1}^N{Y}_i}{N}$$

(7)

where $Y_i$ is the height value of one point, h is the calculated sea-level height, and N is the number of all matched sea waves.

4. Experiments and Results

In this section, we apply the proposed stereo matching method to sea surface images taken in 2016, 2017, and 2022 from two sites: Fukuoka Institute of Technology (FIT, ${33}^{\circ }{41}^{\prime }$ N, ${130}^{\circ }{26}^{\prime }$ E) and Fukuoka Kenritsu Suisan High School (FKSH, ${33}^{\circ }{47}^{\prime }$ N, ${130}^{\circ }{27}^{\prime }$ E). The monitoring distances of these images were 1.4–2.0 $\times {10}^4$ m, 0.4–1.0 $\times {10}^4$ m, and 0.8–$1.4\times {10}^4$ m, respectively. In order to evaluate the performance, we compared the final matching results with those of the Euclidean distance method (a simple similarity measurement of the appearance features) and a RANSAC+SURF method. Precision, recall and runtime were the three main evaluation terms. We also measured the sea level height for 6 days near the Shingu coast by the proposed system, and it was compared with the data provided by the Japan government.

4.1. Matching Results on Different Sea Surface Images

In this section, we firstly test the performance of our method by comparing the matching results with Euclidean distance matching and the RANSAC+SURF method. Figure 13 shows the comparison of the final matching results, (a), (b), (c), (d), (e), and (f) are six groups of comparison results, the left column is the RANSAC+SURF matching result, the middle column is the matching result by the Euclidean distance, and the right column is the result from our proposed method. (a) and (b) were taken in Sep 2022 from FIT with the same monitoring distance of 0.8–1.4 $\times {10}^4$ m. (c) and (d) were taken in Mar 2017 from FIT with a monitoring distance of 0.4–$1.0\times {10}^4$ m. (e) and (f) were images taken in Feb 2016 from FKSH with a monitoring distance of 1.4–$2.0\times {10}^4$ m. From the comparison results, we find that the proposed stereo matching method could obtain stable and correct results on these six images under different light condition.

We also conducted a quantitative comparison on these image pairs. The performance was characterized by the precision and recall, as shown in

Table 3. Sparse matching results.

No.	RANSAC+SURF		Euclidean Method		Our Method
	Precision	Recall	Precision	Recall	Precision	Recall
(a)	77.8	66.7	88.2	85.2	87.5	88.9
(b)	75.0	40.0	84.6	86.7	88.9	90.0
(c)	90.0	10.3	94.0	54.0	97.6	91.9
(d)	76.9	24.0	89.2	60.9	89.2	80.5
(e)	52.9	41.2	80.0	44.1	86.2	85.3
(f)	50.0	34.8	90.9	47.8	90.0	86.9
Average	70.4	36.2	87.8	63.1	89.9	87.3

.

4.2. Computational Complexity

To judge if two sea waves could be matched, we needed to find the deterministic features and calculate their weighted average. It was faster than the Euclidean distance and normalized cross-correlation (NCC) used in SURF or SIFT: $O\left(n\right)$, where n is the dimension of the feature descriptor. We also conducted a quantitative comparison on the runtimes of these three methods. Figure 14 shows the comparison results calculated from 1260 pairs of images captured by the proposed system over 6 days.

4.3. Sea Level Measurement Results

A stereo system was established to monitor the Shingu coast of Japan. Two telephoto lens cameras were set on the building A of the Fukuoka Institute of Technology to take photos of the sea. Thirty images were captured per second, each shooting time was 5 s, which means 150 images were captured during one shooting. The data from the cameras were saved automatically. Figure 15 shows the measured relative height change and its comparison with government tidal data during 6 days. The data collection locations are shown in Figure 16, the yellow triangle is our measurement location. Since there were no standard sea level height data provided by the government at our measurement location, we chose the nearest location to gather government data and the red star represents the location the government data came from. Since the two locations did not overlap, there was a discrepancy between the two comparison data. However, the change trends of the two types of data were consistent.

5. Discussion

From Table 3, in pairs (a) and (b) of sea surface images, the RANSAC+SURF method performed well in precision, because there was a mountain in the images which was rigid and characteristic, as Figure 13 shows. However, the recall was very low. The average recall was smaller than 55%, meaning that most sea waves were not matched. For the (c)–(f) pairs of sea surface images, the RANSAC+SURF method performed poorly both in precision and recall, because the sea surface images were monotonous without any rigid or characteristic object in them. Compared with the RANSAC+SURF method, our method did not suffer from these problems; it improved the precision and recall by 7.0% and 54.2% on average.

To illustrate that the poor matching results of the RANSAC+SURF algorithm were not caused by the absence of feature points, we marked all the feature points extracted by the RANSAC+SURF method and our method on a pair of images. We show the results in Figure 17.

The left column shows the feature points of the RANSAC+SURF algorithm in black circles, and the right column shows the feature points of our method and the Euclidean distance matching method in red circles. It is obvious that the RANSAC+SURF algorithm extracted many more feature points than our method. It means the descriptor and the matching strategy of our method improved the final matching results. To verify the rapidity of the algorithm, the experiment was conducted on 1260 sea surface image pairs taken by our system. Observing the left three images of Figure 14, we find the runtime of our proposed matching method and the Euclidean distance matching method were very close. Observing the left comparison results, we find that our method was surprisingly more effective than the RANSAC+SURF algorithm, especially when the number of extracted sea waves was large.

We used the proposed system to perform actual measurements within 4–10 km to verify its feasibility and from Figure 15, we find that all changes of the sea level were consistent with the tidal data provided by JMA; at the same time, different from government tidal data, the proposed method was sensitive to small changes of sea surface caused by other environmental changes such as wind and air pressure, as Figure 15 shows. To further verify the correctness of the sea level height measurements, we observed the height of the islands on the sea surface at different times within one day to determine the change trends in sea level height, as Figure 18 shows. In this figure, we find that on 14 September, the height of the island above the sea surface increased from morning to evening, meaning that the sea level height decreased within that day. It was consistent with results shown in Figure 15. It is obvious to find that the sea level height change trends observed from images for these 6 days were exactly the same as those measured by the system and published by the government seen in Figure 15.

6. Conclusions

Nowadays, with more and more people choosing to live in low-lying coastal zones, the monitoring of sea level has become more and more important both in coastal zone planning and coastal disaster warning systems. Existing sea level measurement systems are insufficient in terms of sea level sampling rate, high device and maintenance costs, and low survival rate during energy events. In this paper, we proposed a sea level measurement system, which used images to monitor sea level height and we focused on the stereo matching of this system. Automatic image acquisition and processing allow a system to update sea level information fast. Long-distance shooting improves the existing rate during energy events and decreases the equipment maintenance costs. However, long-distance sea surface image pairs lack feature points and the feature points from the left and right camera views are not exactly alike, which is one of the main difficulties for a stereo system to extend its monitoring range. Thus, in this paper, we proposed to use a sea wave’s appearance features and search the deterministic features to solve the problem of the lack of feature points and complete matching, while allowing the existence of differences in left and right camera views as well as speeding up the matching computation process. Tsunami or flooding warning requires the monitoring system to sample sea level height within one minute or faster. The proposed stereo matching method decreased the final feature vector dimension into 8D, and found the deterministic features for matching and false positive elimination, thus rapidly reducing the computation load; for most sea surface images, the stereo matching took shorter than ${24}^{-1}$ s. Images have been used in many marine monitoring systems, such as shoreline detection, sea wave reconstruction, and so on. However, all such systems focus on short-distance measurements due to the low precision of long-distance calibration and the lack of feature points. In this paper, we pioneered the idea of extending the monitoring range of stereo systems to a distant sea area of 20 km by improving the calibration and stereo matching method. Stereo matching experiments performed on images under different light conditions ensured that the system could run stably during long-term monitoring, and the measured sea level being consistent with the government data verified the effectiveness of the proposed system. In the future, we will focus on other research projects of the system, including but not limited to 24 h image capture, image measurement in bad weather, stereo matching by deep learning, etc.