Wave Height and Period Estimation from X-Band Marine Radar Images Using Convolutional Neural Network

Abstract

In this study, a deep learning network for extracting spatial-temporal features is proposed to estimate significant wave height ($H_s$) and wave period ($T_s$) from X-band marine radar images. Since the shore-based radar image in this study is interfered with by other radar radial noise lines and solid target objects, to ensure that the proposed convolutional neural network (CNN) extracts the image features accurately, it is necessary to pre-process the radar image to eliminate interference. Firstly, a pre-trained GoogLeNet is used to extract multi-scale depth space features from the radar images to estimate $H_s$ and $T_s$. Since CNN-based models cannot analyze the temporal behavior of wave features in radar image sequences, self-attention is connected after the deep convolutional layer of the CNN to construct a convolutional self-attention (CNNSA)-based model that generates spatial-temporal features for $H_s$ and $T_s$ estimation. Simultaneously, $H_s$ and $T_s$ measured by nearby buoys are used for model training and reference. The experimental results show that the proposed CNNSA model reduces the RMSD by 0.24 m and 0.11 m, respectively, in $H_s$ estimation compared to the traditional SNR-based and CNN-based methods. In $T_s$ estimation, the RMSD is reduced by 0.3 s and 0.08 s, respectively.

1. Introduction

X-band radar has become a valuable tool for oceanographic studies due to its high spatial and temporal resolution [1]. The Bragg resonance interaction between X-band electromagnetic waves and the centimeter-scale ripple waves induced by local winds generate radar backscatter from the sea surface. The sea surface changes are imaged by the backscattering of electromagnetic waves [2]. In addition, marine radar images are used to effectively estimate sea surface features, such as wind [3,4], wave parameters [2,5,6,7], and current [8,9,10]. Different sea surface parameters, such as significant wave height ($H_s$), wave period ($T_s$), and wave direction, are essential for the safety of various marine activities and the development of coastal areas [11]. In situ sensors, such as wave buoys, have traditionally been used for wave measurements. In contrast, X-band marine radars can detect and measure wave parameters over a broader range and have relatively low maintenance costs.

The echo signal produced by the X-band radar, presented in a light and dark striped image, is called sea clutter, formed by the backscattering of electromagnetic waves emitted by radar across the sea. The base signal (short waves) is modulated by longer waves with several mechanisms, such as hydrodynamic modulation, tilt modulation, and shadow modulation, resulting in longer waves visible in radar images [12]. The conventional spectral analysis method obtains the image spectrum via 3-D Fast Fourier transformation (FFT) of the radar image sequence [2], using the dispersion relation as a filter to extract the image spectrum from the background noise. Then, the wave period, wave direction, and wavelength can be deduced from the filtered spectrum [2,6]. Based on the estimation relation of the synthetic aperture radar [13], the signal-to-noise ratio (SNR) of the wave is calculated from the wave spectrum, and the estimation is completed by the linear relationship between the $H_s$ and the square root of the SNR. The coefficients can be determined using in situ measurements. Later studies have found that $H_s$ is not exactly linearly proportional to the square root of the SNR due to variations in sea states, different methods of calculating SNR, and differences in radar systems. In addition to SNR-based methods, there are some alternative methods that have been proposed to estimate wave parameters, such as empirical orthogonal function-based methods [14], iterative least squares-based methods [1], 2D continuous wavelet transform-based methods [15], array-beamforming-based methods [16], shadowing mitigation-based methods [17], and synchrosqueezed wavelet transform-based methods [18]. Additionally, there are some other methods for estimating $H_s$ that have been proposed, such as shadowing-based methods [19,20], ensemble empirical mode decomposition-based methods [21], correlation analysis-based methods [22], and variational mode decomposition-based methods [23]. It is worth noting that machine learning algorithms have been applied to $H_s$ estimation, which can simplify the cumbersome steps of previous algorithms and improve computational efficiency. It is also possible to estimate more accurate results using methods including a support vector regression (SVR)-based method [24], artificial neural network (ANN)-based methods [25,26], a convolutional neural network (CNN)-based method [27], a convolutional gated recurrent unit network (CGRU)-based method [14], or a temporal convolutional network (TCN)-based method [28]. In addition, random forest (RF)-based machine learning methods have been used to estimate wave directions and periods [29].

CNNs can extract spatial features of image sequences but are incapable of temporal analysis. In this study, a novel $H_s$ and $T_s$ estimation model combining CNN with self-attention (abbreviated as CNNSA) is proposed. This method can extract the spatial features of radar image sequences via CNN, and introduces a self-attention layer in the feature vector of the CNN to capture dependencies in the time series and achieve spatial-temporal feature extraction of radar images. Section 2 introduces the data pre-processing methods used, including median filtering based on the two-layer decision and the adaptive region growing repair method. The structure and components of the proposed CNN and convolutional self-attention (CNNSA)-based models are illustrated in Section 3. The training and testing results obtained from shore-based marine radar data using SNR, CNN, and CNNSA-based models are presented and compared with each other in Section 4. The results are discussed in Section 5. Finally, the conclusion and outlook of this work appear in Section 6.

2. Data Pre-Processing

Interference from other marine radars produces radial noise lines in radar images, as shown by the dense radial pixel lines in Figure 1. In this study, the radar images collected are from shore-based X-band radar, which also is affected by interference from targets like ships, which appear as bright spots in the radar images. The pre-processing process of the radar images is shown in Figure 2.

2.1. Median Filtering Based on the Two-Layer Decision

According to the characteristics of the wave texture in the radar image, an improved median filtering method is used to screen the pixel points for noise. The region involved in $H_s$ and $T_s$ estimation is intercepted in a radar polar coordinates image. In this study, a sub-image with azimuth ±30° and range in 600–2500 m is selected and converted into a grey image of 0–255, denoted as $I(x,y)$, which is of size $n\times n$, n = 256. First, the size of the sliding window $L_1$ is set to $m\times 1$, and in this study, m = 3. The $I(x,y)$ is expanded according to the sliding window size; the pixel image size after the expansion is $(n+m-1)$, and the starting point is the first position of the expanded image in order to find the noise point to be processed. Then, the center point $L_1$ is set to determine whether the median of the grey value of each pixel point is the $L_1$. If it is the median, the grey value of the center point will not be processed. If it is not the median, $L_1$ is set to determine whether it is a noise point. Next, a threshold $C_1$ is set to determine whether the pixel point is a noise point. $C_1=\frac{1}{5}{A}_{avg}$ calculates the difference between the grayscale value of the maximum pixel point in each sliding window ($B_{max}(i,j)$) and the grayscale value of the minimum pixel point ($B_{min}(i,j)$), expressed as B. If B > $C_1$, the noise point is determined. Finally, the noise point is replaced by the median value calculated in $L_1$. $A_{avg}$ is represented as follows:

$$A_{avg}=\frac{1}{x}\frac{1}{y}\sum _{j=1}^y\sum _{i=1}^{x}I(i,j)$$

(1)

2.2. Adaptive Region Growing Repair Method

Targets on the sea surface can cause a lack of wave texture, leading to errors when estimating $H_s$ and $T_s$ from radar images using deep learning-based methods. To avoid this problem, this study uses an adaptive region-growing repair method to recover the image region disturbed by targets. The process is divided into four parts. The first part uses the adaptive threshold to determine whether the target is produced. The initial parameter $C_2$ is set with threshold $D_1$, where $A_{max}$ is the maximum value in the grayscale value of all pixel points in $I(x,y)$. It is then determined whether $A_{max}$ is greater than $D_1$, and if the output is based on the original grayscale value, the following calculation is required.

$$D_1=A_{avg}+C_2$$

(2)

$$C_2=\frac{A_{max}+1}{2}$$

(3)

The second part uses gradient descent to find the initial growth point of the target and identifies the target area in $I(x,y)$. The grayscale values of all the pixel points in $I(x,y)$ are sorted from smallest to largest, and the location of a specific grayscale value is selected as the growth point of the target. The sliding window is set to $L_2=m\times m$, so that $A_{max}$ is used as the center point and initial growth point of the sliding window $L_2$. This point is used as the starting point to search for pixel points in the sliding window that have similar features to the centre point of the sliding window. It is also used as the noise starting point until the traversal stops when there are no pixel points in the sliding window that have similar features, in order to determine the area where the target is located. The pixel point screening method for similar features is $C_{(i,j)}-C_{cen}<D_2$, where $C_{(i,j)}$ denotes the grayscale value of the pixel point inside $L_2$, $C_{cen}$ denotes the grayscale value of the center point of $L_2$, and $D_2$ is the filtering threshold. If the equation is valid, pixel points with similar features are searched for using $C_{(i,j)}$ as a new starting point until the equation is not valid inside the sliding window. These steps are repeated to find the remaining targets in the image. $D_2$ is defined as follows:

$$D_2=\frac{1}{3}{A}_{avg}$$

(4)

The third part determines whether the target is a real target at the time. The number of pixel locations occupied by each target object $N_i$ is calculated, and it is determined whether the total number of pixel locations occupied by each target object is less than the maximum upper limit of single target object recognition where $area=m\times m (m=N_i/6)$. The integer is retained, and if it is valid, it is considered to be a target object. The average grey value $B_{avg}$ of the pixel points in the target area and the target threshold $D_3$ are calculated. If $B_{avg}>D_3$, then it is considered a real target. $D_3$ is defined as follows:

$$D_3=\frac{5}{4}{A}_{avg}$$

(5)

The fourth part is based on a mean-filling transition algorithm to restore the target and real wave recovery. The grayscale value of the pixel point where the target is located is set to 0. According to the size of the sliding window $L_2$, $I(x,y)$ is expanded to $(n+2m)\times (n+2m)$, and the mean value of the four-pixel points that are m points away from the centre of the noise point in the distance direction and orientation direction is selected to fill the image, instead of the noise point. Figure 3 shows the radar sub-image, processed radial noise line, and the image of the target object.

3. The CNNSA-Based Estimation Model

3.1. CNN

CNNs are a deep learning model designed to process and analyze data with a grid structure, such as images and videos. They are unique in their convolutional layers, which are locally aware of the input data through the utilization of sliding windows, thus effectively capturing the spatial relationships of image or video features [30]. The basic building blocks of CNNs include convolutional operations, pooling operations, and fully connected layers. Convolutional operations detect various features in images, such as edges, textures, and shapes. In contrast, pooling operations are used to reduce the size of the feature map, reduce computational complexity, and extract higher-level features. In this study, GoogLeNet [31] is selected as the pre-training model among many convolutional network models. Compared with other classical CNNs, such as AlexNet [32] and VGGNet [33], GoogLeNet has a lighter weight and more efficient structure, adopting the “Inception” module structure to reduce the number of parameters by using multi-scale convolutional kernels. There are also deeper networks that can learn more complex feature representations.

GoogLeNet consists of several “Inception” modules, each containing a series of convolutional kernels of different scales to capture image features in parallel. Its overall structure consists of the following key components:

(1)

Convolutional and pooling layers. These are used to extract the basic features of the image, such as edges and texture;

(2)

Inception modules. Each “Inception” contains multiple parallel convolutional kernels and pooling operations to capture features at different scales and levels. The results of these parallel operations are cascaded together to form the module outputs, with the primary goal of improving the feature representation of the model without adding too many parameters;

(3)

Global average pooling layer. This layer averages the values of each channel of the feature map to generate a fixed-size feature vector. This reduces the fully connected layer’s dimensionality and helps reduce overfitting.

(4)

Fully connected layer. This layer integrates information from different features to capture complex relationships in the data. In this study, the regression task of the fully connected layer is utilized to estimate the H_S and T_S of the radar image.

The structure of the GoogLeNet-based estimation model is shown in Figure 4.

3.2. Self-Attention

The CNN-based model performs well in capturing the depth features of the radar image, but there are no features to capture the temporal correlation between the entire image sequence. Self-attention allows the model to dynamically capture long-distance dependencies throughout the input sequence, which can relate to the context of the image sequence. The computation process of self-attention, which includes the computation of Query, Key, Value, Attention Score, and the final weighted summation, is shown in Figure 5. Firstly, the mapping map x obtained by deep convolution is multiplied by the weight matrices $W_Q$, $W_K$, and $W_K$ to get the feature spaces $f\left(x\right)$, $g\left(x\right)$, and $h\left(x\right)$, respectively:

$$\begin{gathered} f\left(x\right)=W_Q·x \\ f\left(x\right)=W_K·x \\ f\left(x\right)=W_V·x \end{gathered}$$

(6)

Then, after multiplying the transpose of the feature space $f\left(x_i\right)$ with the feature space $g\left(x_j\right)$, the product is normalized with a soft classifier to obtain the attention map ${\beta }_{j,i}$:

$${\beta }_{j,i}=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(s_{ij}\right)}{\sum _{i=1}^N\mathrm{e}\mathrm{x}\mathrm{p}\left(s_{ij}\right)}, where s_{ij}=f{\left(x_i\right)}^T·g(x_j)$$

(7)

Finally, the feature space $h\left(x\right)$ is multiplied by the attention map and then with a 1 × 1 convolution to obtain the self-attention feature map, o:

$$o_j=v\left(\sum _{j=1}^N{\beta }_{j,i}·h\left(x_i\right)\right), where h\left(x_i\right)=W_V·x_i$$

(8)

3.3. CNN-SA Model

In this study, a novel deep learning model with temporal and spatial feature extraction is proposed by embedding self-attention as a feature extraction module in the CNN model. The model captures the spatial information of radar images through the convolutional layer, pooling layer, and Inception structure of GoogLeNet. Adding self-attention after the “Inception” structure can dynamically capture long-range dependencies in image sequences. This makes capturing spatial and temporal relationships in radar image sequences more conducive to estimating $H_s$ and $T_s$ from the model. The flowchart of the CNNSA-based estimation model for $H_s$ and $T_s$ is shown in Figure 6, and detailed parameter information is shown in

Table 1. Detailed parameter information for CNNSA-based estimation models.

Type	Input Size	Output Size	Patch Size/Stride	Filters
Input	256 × 256	256 × 256 × 1
Convolution	256 × 256 × 1	128 × 128 × 64	7 × 7/2	64
Max pool	128 × 128 × 64	64 × 64 × 64	3 × 3/2
Convolution	64 × 64 × 64	64 × 64 × 64	1 × 1/1	64
Convolution	64 × 64 × 64	64 × 64 × 192	3 × 3/1	192
9 × Inception	64 × 64 × 192	8 × 8 × 1024
Self-attention	8 × 8 × 1024	8 × 8 × 1024
Average pool	8 × 8 × 1024	1 × 1 × 1024	7 × 7/1
linear		1 × 1 × 2

.

4. Results

4.1. Data Overview

The X-band marine radar data used in this study were collected in Pingtan, Fujian Province, from 16 December 2010 to 10 January 2011. The detailed parameter information of the radar is shown in

Table 2. Radar Information.

Parameters	Value
Transmit frequency	9.41 GHz
Polarization	Horizontal
Antenna rotation speed	22 r/min
Range resolution	7.5 m
Antenna height	45 m
Horizontal beam width	2°
Azimuth coverage	360°

. The rotation period of the radar antenna was about 2.5 s, and a radar image sequence contained 32 images, that is, 80 s of each image sequence interval. The wave buoy that measured the reference data was deployed approximately 0.85 km from the radar. Since the references $H_s$ and $T_s$ were measured every 20 min, temporal interpolation is required to provide synchronized references for each radar image sequence. Figure 1 shows the position relationship between a radar image under polar coordinates and wave buoy placement. During data collection, $H_s$ was between 1.3 and 3.5 m and $T_s$ was between 6.5 and 10 s. At the same time, the synchronized local wind speeds ranged from 5.8 to 18.5 m/s, as shown in Figure 7, making the proposed method estimated under typical wind and wave conditions. The synchronized buoy references and radar images used in the study were not present at every moment, and the corresponding data are shown in Figure 7.

4.2. Model Train

In this study, the first image of each sequence was input into the pre-processing process in Figure 2, which selected the radar image sub-area and filtered out the radial noise lines and targets so that high-quality sub-images could be provided for the estimation of $H_s$ and $T_s$, based on deep learning for better learning of feature information in the images. The significant wave heights used in this study were 1.3–3.5 m, and the training sample contained images from all ranges. Since the reference $H_s$ and $T_s$ were measured around every 20 min before training, all the image sequences (excluding the rainy day) were first sorted according to the simultaneous $H_s$ and $T_s$ obtained from interpolated buoy measurements. Then, at each 0.5 m interval between 1.3–3.5 m, 70% and 30% of the samples were taken for training and testing of the model. The proposed CNNSA-based model was trained using the Pytorch framework, installed on a Windows 10 PC with two 2.10 GHz Xeon (R) Gold 6230R CPUs. The initial learning rate was 0.0003 using the Adam optimizer and the MSE loss function. Batch size was set to 36, and the number of epochs was 50.

4.3. Result Analysis

The SNR-based method [7] and the CNN-based method were also applied to the same testing set for comparison, to demonstrate the effectiveness of the CNNSA-based $H_s$ and $T_s$ estimation model. In this study, the $H_s$ and $T_s$ obtained in two continuous buoy measurement intervals were averaged using a time-moving average with the same moving time as the buoy measurement interval. The root-mean-square differences (RMSDs), correlation coefficients (CCs), and biases between the reference value and the estimated value using the SNR-based, CNN-based, and CNNSA-based methods are summarized in

Table 3. Comparisons of results using different methods for H_S estimation.

Method	Testing (Without Averaging)			Testing (With Averaging)
	RMSD (m)	CC	Bias	RMSD (m)	CC	Bias
SNR	0.56	0.64	−0.20	0.54	0.65	−0.20
CNN	0.45	0.76	−0.04	0.41	0.77	−0.04
CNNSA	0.35	0.85	−0.03	0.30	0.86	0.04

and

Table 4. Comparisons of results using different methods for T_S estimation.

Method	Testing (Without Averaging)			Testing (With Averaging)
	RMSD (m)	CC	Bias	RMSD (m)	CC	Bias
SNR	0.60	0.65	0.12	0.57	0.65	0.11
CNN	0.46	0.74	−0.10	0.35	0.77	−0.1
CNNSA	0.37	0.89	0.03	0.27	0.91	0.03

, respectively. The estimation results of the three methods for the test sample before and after the moving average are shown in Figure 8. In addition, Figure 9 shows the estimation results in the time series.

From Table 3, it can be seen that the RMSD of the proposed method is reduced by 0.21 m and 0.1 m, and the CC is improved to 0.85 when compared with the SNR-based and CNN-based methods, which is the $H_s$ estimation without averaging. Further, when the $H_s$ estimation result is processed by sliding for an average of 30 min, it can be seen from Table 3 that the $H_s$ estimation result is further improved, with an RMSD and CC of 0.3 m and 0.86, respectively, based on the CNNSA method. Similarly, Table 4 shows the $T_s$ estimation. The RMSD of the proposed method is reduced by 0.23 s and 0.09 s, and the CC is improved to 0.89 compared to the SNR-based and CNN-based methods, which is the $T_s$ estimation without averaging. The estimated results of $T_s$ were processed by sliding for an average of 30 min. As can be seen from Table 4, the estimated $T_s$ is also improved, and the RMSD and CC based on the proposed method are 0.27 s and 0.91. To visualize the estimation results of the three methods regarding $H_s$ and $T_s$, Figure 8 shows the estimation results of $H_s$ and $T_s$ under average sliding and without average sliding of the three methods.

5. Discussion

As can be seen from Figure 8a, when Hs is 2–3 m, the estimated result of $H_s$ is small, while when $H_s$ is greater than 3 m, the estimated result of $H_s$ is larger. In Figure 8b, the SNR-based $T_s$ estimation results show a large dispersion as a whole, which is due to the sensitivity of SNR-based methods to radar image noise, resulting in a large deviation in the estimation results. It can be seen from Figure 8c,e that the CNNSA-based method has a better linear correlation in $H_s$ estimation than the CNN-based method. Similarly, in Figure 8d,f, the CNNSA-based method also produces a better linear correlation in the estimation of $T_s$. It can be seen from Table 3 and Table 4 that the correlation coefficients of $H_s$ and $T_s$ estimation based on the CNNSA method are 0.86 and 0.91, respectively, after the 30-min moving average, both of which are higher than those of the CNN-based method and SNR-based method. Further, Figure 9a and b, respectively, show the variation trend of $H_s$ and $T_s$ estimated by the three methods over time, and it can be seen that the results estimated by the CNNSA-based method are closer to the variation trend of buoy references.

In summary, the proposed CNNSA-based regression estimation model has different degrees of improvement in the estimation of $H_s$ and $T_s$ compared to the other two methods. Figure 8 shows the degree of fit of the three methods; the proposed CNNSA-based model estimates of $H_s$ and $T_s$ are closer to the black line after applying the time-moving average. Further, the closeness of the three methods to the buoy reference is demonstrated in Figure 9, with $H_s$ and $T_s$ estimation of the proposed method being closer to the buoy references. All these results show that the proposed method improves the accuracy of $H_s$ and $T_s$ estimation using X-band marine radar images and simultaneously validates the reasonableness and effectiveness of the proposed CNNSA-based estimation model.

6. Conclusions

In this study, a deep neural network was used to estimate $H_s$ and $T_s$ in an X-band marine radar backscatter image sequence. Since the shore-based radar images in this study were interfered with by other radar radial noise lines and solid target objects, to make the convolutional neural network (CNN) extract the image features more accurately, it was necessary to pre-process the radar images to eliminate the interference. Firstly, a pre-trained GoogLeNet was used to extract multi-scale depth space features from the radar images to estimate $H_s$ and $T_s$. Since CNN-based models cannot analyse the temporal features of wave features in radar image sequences, self-attention was connected after the deep convolutional layer of the CNN in order to construct a convolutional self-attention (CNNSA)-based model that generated spatial-temporal features for $H_s$ and $T_s$ estimation. Both models were trained and tested using collected shore-based radar data. The buoy measurements were also interpolated and used as baseline values for performance evaluation. The experimental results showed that both CNN-based and CNNSA-based models can improve the estimation results of $H_s$ and $T_s$. The RSMD of $H_s$ estimation is reduced by 0.13 m and 0.24 m, respectively, and the RSMD of $T_s$ estimation is reduced by 0.22 s and 0.3 s, respectively, compared with the traditional SNR-based method. Overall, the CNNSA-based model showed better results for estimating $H_s$ and $T_s$ simultaneously. Therefore, the estimation accuracy is further improved by using temporal information while extracting multi-scale depth space features.

Due to the data limitations of this experiment, in future work, a collection of sea state data and radar data containing a larger number of images and a wider range for model training and validation is required, and radar data and reference data should also be collected at different locations to validate the proposed model further.