Ship Berthing Information Extraction System Using Three-Dimensional Light Detection and Ranging Data

Abstract

Safe and efficient berthing is essential to ensure maritime transportation and the safety of ships and ports. Three-dimensional (3D) light detection and ranging (LiDAR) can monitor and support ship berthing because it provides abundant target information and offers excellent advantages in measuring accuracy. Hence, a berthing information extraction system has been developed based on 3D LiDAR. Principal component analysis is used to calculate a ship’s heading and the normal vector, and the feature points of the bow and stern are determined. The segments passing through the points are obtained via region growing. The bow and stern are recognized by the similarity of the normal vector of the segments and ship’s heading according to the positions of the ship relative to the berth through visibility analysis. Qualitative and quantitative calculated analyses of the distance, velocity, and approach angle of the dynamic ship’s bow and stern relative to the dock are performed based on the feature points of 3D LiDAR data. A laser scanner, used as the detection unit, efficiently monitored the Ro–Ro ship Ocean Island berthing at Lushun Port in field experiments. On-site applications demonstrated the feasibility and effectiveness of the proposed method for the recognition of dynamic ship target and ensuring safe ship berthing.

1. Introduction

The ship berthing maneuver is a complex procedure, given the development of large ships, especially VLCC, bulk carriers and large container ships [1]. When the ship meets the terminal facility, berthing energy is generated, and berthing velocity is the most important factor of berthing energy [2]. If the berthing velocity is out of control, there is a risk of damage to the terminal facility and hull. Meanwhile, even minor yawing or list can cause a collision with STS cranes, as shown in Figure 1. The ports must close facilities near the incident, and the repair or replacement may require a year [3]. Moreover, for the safety of the ship and the dock, berthing is a process during which the distance and velocity change very slowly to zero, while the required unberthing velocity is lower after the ship leaves the fender. Thus, ensuring safe berthing has become a pressing concern for port authorities, pilots, and captains [4]. During berthing, the surrounding environment of the ship, in relation to berthing velocity and distance of approach, must be precisely determined [5,6]. Therefore, studying ship positioning and monitoring near the shore is essential to help ensure the safety of the ship and the port.

Ship berthing originally mainly relied on the experience of the pilot and captain. External factors such as hydrometeorology can easily induce large subjective deviations, and the development of large-scale ships means that such deviations can cause enormous harm. Owing to the development of green and smart ports, some advanced ports and ships have begun to introduce various kinds of speed and range measurement equipment to assist safe and automated ship berthing and mooring [7,8]. Berthing aid equipment can be installed on board the ship or at the berth [9]. Onboard devices measure the offshore distance of the ship based on its position and require at least two specific points on the ship for measurement. Early onboard berthing systems were based on sonar and used the principle of the Doppler Effect to measure the speed; however, the accuracy was not high. Currently, many berthing systems use satellite positioning and visual systems to obtain information. Hiroyuki and Peng et al. used GPS to assist in ship berthing [10,11,12]. Oceanstar used GNSS data to help ships berth on the shore [13]. However, in this approach, the combined positioning machine must be placed in a specific position depending on the shape of the ship. Since the GNSS signal must be prevented from being obscured, the locator heavily relies on the communication network transmitting the positioning information. Mizuchi and Yuen et al. proposed a markerless vision-based measurement system based on stereo cameras [14,15]. It showed sufficient improvements in accuracy, and was tested at a distance of 25 m, but it still has problems of installation and maintenance.

Dock-based devices are installed on the berth. Previously, Shi introduced the use of sonar devices to monitor berthing [16]. In 1990, a docking system that integrated sonar was designed for measuring the speed of approach with environmental and meteorological sensors [17]. However, the sonar in their system easily produced scattering, refraction, and interference, consequently reducing the measurement accuracy. Infrared, radar, and ultrasonic waves have also been used to dock ships and monitor berthing, but infrared and ultrasonic wave detectors are significantly affected by the weather, and the distance and accuracy of detection are limited. Radar cannot be used to its full potential because of the range factor and high costs [7,18].

Light detection and ranging (LiDAR) uses laser wavelengths in the visible and near-infrared domains, and has been used for several decades [19]. One-dimensional (1D)/two-dimensional (2D) lasers installed on the docks are currently widely used. The two laser probes installed at both ends of a berth can be used to measure the distances from the bow and stern to the dock. Yu measured the bow and stern distance based on two laser ranging sensors and calculated berthing parameters [20]. Perkovic et al. used laser ranging sensors at container terminals and integrated them with metocean data [21]. However, finding targets and determining the location of installation are difficult for 1D/2D lasers due to the limitation of visual range, and this will cause inaccurate measurements. In response, three-dimensional (3D) LiDAR lasers are gradually being used. The large-scale and high-precision reconstruction of scanned objects is possible using 3D laser scanning [22]. Compared to traditional single-point measurements, 3D laser scanning is characterized by high precision and the ability to approximate the original data [23]. Perkovic et al. used two laser scanners to monitor a berthing ship, and estimated the ship’s bow and stern by minimizing the sum of squares of distances between the points and berth. The berthing system was applied on the dock [3]. Yan et al. extracted the mold line of a berthing ship from slices of 3D point cloud data [24]. Their method applied 3D LiDAR to berthing monitoring and obtained valuable results, which give meaningful and forward-looking information for docking systems. During the ship berthing process, it is first necessary to determine the information of the bow and stern to guide the safe berthing operation. However, the existing methods pay more attention to 2D plane information, and have not developed a dedicated method for the recognition of the bow and stern automatically. To the best of our knowledge, an efficient and effective method to recognize the bow and stern and monitor the berthing ship based on 3D LiDAR data is yet to be developed. Moreover, to apply three-dimensional feature extraction to monitor berthing information, it is necessary to overcome the difficulties associated with accurately identifying a dynamic ship’s motion parameters and the interference of port environments.

Herein, we study a berthing information extraction system based on 3D LiDAR for ship berthing. To this end, we establish an algorithm for the automatic capturing of the berthing information. More specifically, in data preprocessing, the methods of statistical outlier removal and pass-through are used to filter the random noise and fixed facility data. In berthing information extraction, principal component analysis (PCA) is used to determine the ship reference frame (SRF) and the feature points of bow and stern. In addition, according to the visibility analysis based on the position of the laser and the ship, six ship berthing attitudes can be obtained. Region growing and the comparison of normal vectors are used to recognize the bow and stern points of the ship. Key parameters such as the distance and velocity of the ship’s bow and stern relative to the dock are calculated through the bow and stern points. Based on experiments and applications, the method is demonstrated to offer accurate berthing information.

2. Methodology

Figure 2 shows the operating mode of a 3D LiDAR for monitoring a berthing ship. A 3D LiDAR is generally placed at a central position on the dock and the view of the scanner is adjusted to capture the ship’s hull. The vertical height H and horizontal distance R from the scanner’s viewpoint to the edge of the dock are obtained via measurement. Through coordinate transformation, the positional information of the ship relative to the dock can be indirectly obtained.

2.1. Data Preprocessing

2.1.1. Data Conversion

Since different types of laser scanners may produce different data formats, the data are first converted into an easy-to-use format, P (X′, Y′, Z′, D, R, Tr) ASCII, to facilitate the unified processing. X′, Y′, Z′ represents the three-dimensional position of target point P along the X′, Y′, and Z′-axes of a right-hand system, D represents the distance from point P to the scanner, R represents the signal reflectivity, and Tr represents the rotational matrix of the scanner. The origin O′ of the scanner coordinate system is the scanner’s viewpoint. The X′ and Y′-axes are on the horizontal plane of the scanner coordinate system, where the Y′-axis is generally in the scanning direction. The Z′-axis represents the vertical direction. Then, the scanner coordinate system is converted to the berth coordinate system as shown in Figure 3. For the berth coordinate system, the X-axis is along the berth line, the Y-axis is perpendicular to berth line outward, and the Z-axis is perpendicular to the plane of the berth. The origin O is set to the midpoint of the berth. ∆X, ∆Y and ∆Z are the spatial translations from O′ to O in the X, Y and Z directions, respectively. α, β, and γ are the rotation parameters of the three axes, respectively. The relationship between P (X′, Y′, Z′) of the scanner coordinate system O′-X′Y′Z′ and P (X, Y, Z) of the berth coordinate system O–XYZ is given by

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=R^{\prime }\left[\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\\ Z^{\prime }\end{array}\right]+\left[\begin{array}{c}\Delta X\\ \Delta Y\\ \Delta Z\end{array}\right]$$

(1)

where

$$R^{\prime }=R^{\prime }\left(\alpha ,\beta ,\gamma \right)=\left[\begin{array}{ccc}cos\gamma & sin\gamma & 0\\ -sin\gamma & cos\gamma & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}cos\beta & 0& -sin\beta \\ 0& 1& 0\\ sin\beta & 0& cos\beta \end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\alpha & sin\alpha \\ 0& -sin\alpha & cos\alpha \end{array}\right]$$

(2)

2.1.2. Data Filtering

Random noise points can be mainly attributed to surface factors of the measured object and errors inherent to the measurement equipment. The random noises may be mistaken as the ship points and they need to be filtered. This type of noise is characterized by distinct outliers. In this study, the idea of statistical outlier removal is used to remove the noise of outliers [25]. In statistical outlier removal, the k-d tree is built from the point cloud and the distribution of the distance from each point to the other points in the k neighborhood is calculated to obtain the corresponding local average distance. However, the calculation of distance costs time, and the distance of the k neighborhood is replaced with the number of the points in the k radius. Then, the entire point cloud and its average number u_k and standard deviation σ_k are estimated. Points with an average number outside the standard range are considered noise points. The remaining cloud points p* that meet the requirements are estimated as follows:

$$p^{\ast }=\left\{p_i{}^{\ast }\in p|(u_k-{\sigma }_k\cdot \rho )\le \overline{n}\le ({\mu }_k+{\sigma }_k\cdot \rho )\right\}$$

(3)

where $\overline{n}$ is the average number per point in the k radius, u_k and σ_k are the average number and standard deviation of the k radius for the entire point cloud, and ρ is the threshold. If $\overline{n}$ is outside the standard range, it is defined as an outlier and removed from the data.

The fixed facility data include the cranes and other port infrastructures. The static background takes up a large amount of the LiDAR data and needs to be filtered to improve the subsequent extraction accuracy and efficiency of the ship’s feature points. Because of the fixed position, pass-through is used to filter the static background via the coordinate range.

2.2. Berthing Information Extraction

Previous studies on ship berthing monitoring have showed that the recognition of bow and stern is limited by visual range, the installation location and the choice of 1D/2D sensors. In this study, a berthing information extraction method is designed to improve the accuracy of monitoring ship berthing (Figure 4). The feature points of the main hull can be used to obtain the dynamic parameters of the ship during the berthing process. Figure 4 presents the flowchart of berthing information extraction. In bow and stern recognition, first, we determine the ship’s heading, establish SRF, and calculate the feature points B and S of bow and stern; second, we use region growing to derive the segments passing through points B and S; last, we analyze six positions of the ship relative to the berth, compare the normal vectors of the segments and the ship’s heading to recognize bow and stern, and figure out the bow and stern point and the true ship’s heading. In the berthing parameters calculation, based on the bow and stern points obtained, the distance, velocity, and approach angle are calculated.

2.2.1. Bow and Stern Recognition

PCA is used to estimate the normal vector of the neighboring surface and determine the ship’s heading, $\overrightarrow{h}$ [26]. The covariance matrix C and the standard eigenvalue equation are given by

$$\left\{\begin{array}{c}C=\frac{1}{K}{\displaystyle \sum _{i=1}^k(P_i-\overline{P}){(P_i-\overline{P})}^T}\\ C{V}_j={\lambda }_j{V}_j\hspace{0.33em}(j\in (0,1,2))\end{array}\right.$$

(4)

where K is the number of nearest neighbors to point P, and $\overline{P}$ is the 3D centroid of the nearest neighbors. Solving the eigenvalue equation via singular value decomposition (SVD), λ_j is the jth feature value of the covariance matrix, and V_j is the vector corresponding to the jth eigenvalue. If the eigenvalues satisfy λ₀ ≤ λ₁ ≤ λ₂, the normal vector of the point P_i is the eigenvector corresponding to λ₀. The curvature of the point P_i is $\frac{{\lambda }_0}{{\lambda }_0+{\lambda }_1+{\lambda }_2}$. The normal, the curvature, the scale, and the set of k-neighbors of point P_i are defined as nP_i, λP_i, lP_i and kP_i, respectively. The orientation of nP_i is ambiguous, so nP_i is oriented by

$$n{P}_i\cdot (C_o-C_i)>0$$

(5)

where C_o and C_i are the coordinates of viewpoint o and point i. In the calculation of the ship’s heading, the lower half of the data is projected onto the x–y plane due to the shape characteristics of the ship. K is the number of all the points, $\overline{P}$ is the centroid of the points, the orientation of the ship’s heading, $\overrightarrow{h}$, is defined in the right-hand system, and SRF is established.

Further, we determine the feature points of the bow and stern by calculating the minimum and maximum of the u values in SRF, which are points B′(u, v, z) and S′(u, v, z), where v and z are the averages of the v and z values, and convert these to the berth coordinates B’(x, y, z) and S’(x, y, z). As shown in Figure 5, the ship coordinates are converted to the berth coordinates:

$$\left(\begin{array}{l}x\\ y\end{array}\right)=\left(\begin{array}{cc}\cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{array}\right)\left(\begin{array}{l}u\\ v\end{array}\right)$$

(6)

where x and y are the berth coordinates, and u and v are the ship coordinates. θ is the included angle of $\overrightarrow{h}$ and $\overrightarrow{ox}$:

$$\theta =\mathrm{arc}\cos (\frac{\overrightarrow{h}\cdot \overrightarrow{ox}}{\overrightarrow{\left|h\right|}\cdot \overrightarrow{\left|ox\right|}})$$

(7)

Given that {nPo,λPo,lPo,kPo} and {nPo’,λPo’,lPo’,k Po′ } are defined as mentioned above for seed points Po and Po′, the region growing is applied to extract the segments of bow and stern, which is performed as follows: (1) Points B′ and S′ may be at the corner of the bow, which will affect the calculation of the normal vector, so we determine points Pb and Ps with the smallest curvature in the neighborhood kB′ and kS′ of B′ and S′ as the first seed points, respectively. (2) Then, we create list T₁ and T₂ to store all the points on the surface where Po and Po′ are, and traverse each of the unprocessed points P_i, P_j in their neighborhood kPo, kPo′. We add P_i, P_j to lists T₁ and T₂ if the conditions are satisfied. This study proposes χ and chooses three parameters, ð, th_o, and th_p, in [27]:

$$\begin{array}{l}\arccos (L\cdot n\stackrel{T_1}{Po})>\chi ,\\ \arccos (n{P}_i\cdot n\stackrel{T_1}{Po})<\partial ,\\ |n{P}_i\cdot \overrightarrow{P_i\stackrel{T_1}{Po}}|<t{h}_o,\\ ||\overrightarrow{P_i\stackrel{T_1}{Po}}||<t{h}_p\end{array}$$

(8)

$$\begin{array}{l}\arccos (L\cdot n\stackrel{T_2}{Po\prime })>\chi ,\\ \arccos (n{P}_j\cdot n\stackrel{T_2}{Po\prime })<\partial ,\\ |n{P}_j\cdot \overrightarrow{P_j\stackrel{T_2}{Po\prime }}|<t{h}_o,\\ ||\overrightarrow{P_j\stackrel{T_2}{Po\prime }}||<t{h}_p\end{array}$$

(9)

where ð, χ = 5° are constant angle thresholds, L is the normal vector of the ship’s heading, th_o, th_p = 20lPo / 20lPo′ are the orthogonal and parallel distance thresholds, and lPo, lPo′ are defined as the distances between Po, Po′ and their closest neighboring points. Th_o, th_p are self-adapted by the scale of Po, Po′. The first condition ensures that the bow/stern and hull are segmented: if point Pb/Ps satisfies $\arccos (L\cdot nPb/Ps)>\chi$, Pb/Ps is considered the point on bow/stern, and Po/Po′ need to satisfy the first condition, or they only need to satisfy the last three conditions. The second condition ensures that P_i and P_j are on the same surface with Po and Po′. The last two conditions make the points compact in case the facilities on the ship are counted. Once all the points in kPo and kPo′ are traversed, Po and Po′ are marked as processed. The region is growing point by point until all the points in lists T₁ and T₂ are processed, the results of which are the regions of bow and stern with all the points stored in lists T₁ and T₂.

The change in ship attitude will change the position and angle of the bow and stern relative to the dock, and the distance from the bow and stern to the dock cannot be accurately calculated. Regarding this problem, as shown in Figure 6, six ship berthing attitudes are presented through visibility analysis. The main hull consists of ①bow, ②hull, and ③stern, which can be used to obtain the distances from the ship’s bow and stern to the dock. In view of these visibility analysis results, it was necessary to find a method to enhance the differences between the bow and stern, so as to effectively identify the bow and stern. To this end, the bow and stern points are defined as the points with the minimum y coordinate values of the bow and stern part, taking into account the needs of berthing. (a), (b) are held very briefly and rarely seen in berthing, thus, these positions are considered neglected. The normal vectors L₁, L₂ and L of the two segments and the ship’s heading are compared to recognize the bow and stern:

$$\begin{array}{l}\cos ({\eta }_1)=\frac{\overrightarrow{L_1}\cdot \overrightarrow{L}}{\overrightarrow{|L_1}|\cdot |\overrightarrow{L}|},\\ \cos ({\eta }_2)=\frac{\overrightarrow{L_2}\cdot \overrightarrow{L}}{\overrightarrow{|L_2}|\cdot |\overrightarrow{L}|}\end{array}$$

(10)

$\cos ({\eta }_1)$ and $\cos ({\eta }_2)$ range from [−1,1], and threshold value γ is set to distinguish the ship attitudes in Figure 6c–f. If $\cos ({\eta }_1)$ > $\cos ({\eta }_2)$ > γ, the ship attitude corresponds to Figure 6c,d, the segment corresponding to vector L₂ is at an acute angle to the ship’s hull, the point with the minimum y value is determined as the bow point B, and point B′(S′) of the segment corresponding to vector L₁ is the stern point S. If $\cos ({\eta }_1)$ > γ > $\cos ({\eta }_2)$, the ship attitude corresponds to Figure 6e,f, the segment corresponding to vector L₂ is at a substantially perpendicular angle to the ship’s hull, and the point with the minimum y value is determined as the stern point S. In ship attitude (e)(f), it is difficult to recognize the bow because the cosine values in Equation (10) of the bow and hull are close. The distance of the two segments is calculated to compare the bow and hull. Thus, if the distance is less than 1m, the ship attitude corresponds to Figure 6f, the segment is considered the hull, and the first seed point B′(S′) of the segment is the bow point B, or, the ship attitude corresponds to Figure 6e, the segment is considered the bow and the bow point B is defined as the point with the maximum y value of the hull. The plane of the hull is given by

$$a(x-X_S)+b(y-Y_S)+c(z-Z_S)+D=0$$

(11)

where D = −(aX_S + bY_S + cZ_S), (X_S, Y_S, Z_S) is the coordinate value of stern point S, and L (a, b, c) is the normal vector of the ship. The distance of a point P₀ (x₀, y₀, y₀) to the plane is

$$d=\frac{|a{x}_0+b{y}_0+c{z}_0+D|}{\sqrt{a^2+b^2+c^2}}$$

(12)

d is compared with the set threshold value d_max; if d < d_max, P₀ (x₀, y₀, y₀) is kept and the bow point with maximum y value is calculated. When the ship and the dock are close enough, the two segments are the same hull with $\cos ({\eta }_1)$ = $\cos ({\eta }_2)$, and the bow/stern point is point B′(S′), determined by the distance between point B′(S′) and the bow/stern point of the previous frame.

2.2.2. Berthing Parameters Calculation

The distance from the ship’s bow to the dock is given by

$$D_1=\left|Y_B\right|$$

(13)

The distance from the stern to the dock is given by

$$D_2=\left|Y_S\right|$$

(14)

where Y_B and Y_S are the Y coordinates of bow and stern point B and S according to Section 2.2.1.

The true ship $\overrightarrow{H}$ heading is defined:

$$\overrightarrow{H}=\left\{\begin{array}{c}\overrightarrow{h},\overrightarrow{h}\cdot \overrightarrow{SB}>0\\ -\overrightarrow{h},\overrightarrow{h}\cdot \overrightarrow{SB}<0\end{array}\right.$$

(15)

The approach angle $\theta$ of the ship is defined:

$$\theta =\left\{\begin{array}{c}\mathrm{arc}\cos (|\frac{\overrightarrow{H}\cdot \overrightarrow{ox}}{|\overrightarrow{H}|\cdot \overrightarrow{|ox|}}|),\overrightarrow{H}\cdot \overrightarrow{oy}>0\\ -\mathrm{arc}\cos (|\frac{\overrightarrow{H}\cdot \overrightarrow{ox}}{|\overrightarrow{H}|\cdot \overrightarrow{|ox|}}|),\overrightarrow{H}\cdot \overrightarrow{oy}<0\end{array}\right.$$

(16)

where $\overrightarrow{ox}$ and $\overrightarrow{oy}$ are the unit vectors of x-axis and y-axis.

3. Experiment

3.1. Location and the Ship

The experiment of monitoring berthing was performed in Lushun Port, China, as shown in Figure 7. The bulk carriers and Ro–Ro ships are the main types of ships berthing in the port. The parameters of the monitored ship are presented in

Table 1. Parameters of the ship.

Name	Total Length/m	Width/m	Total Tons	The Type of Ship
Ocean Island	134.8	23.4	15,560	Ro–Ro Ship

.

3.2. Measurement and Apparatus

Table 2. Specifications of the X300 laser.

Item	Parameter
Visual range	Horizontal 180°
	Vertical 180° (90° × 2)
Ranging distance	0–200 m~300 m (80% reflectivity)
Scanning rate	>40,000 dots per second
Angle resolution	1.35′ (Horizontal and vertical planes; the distance between points at 100 m is 37 mm)
Precision	<6 mm (at a distance of 50 m)
	<40 mm (at a distance of 300 m)

presents the specifications of the laser scanner applied in the experiment. The position and view of the scanner are adjusted to capture the ship’s range.

Table 3. Experimental details.

Item	Parameter
The vertical height from the scanner’s viewpoint to the dock H	1.21 m
The horizontal distance from the scanner viewpoint to the dock R	1.25 m
The rotation parameters of the three axes of the laser scanner	0

shows the position and rotation parameters of the scanner relative to the dock. In Section 2.1.1, the coordinate conversion is performed based on the parameters.

4. Results and Discussion

4.1. Converted Data

Before exploring the method based on the 3D LiDAR data, analyzing the ship’s berthing process is necessary. The berthing process of a ship can be divided into two stages: arriving (from when the ship starts braking to when it reaches the frontier waters of the berth, as shown in Figure 8a) and berthing (when the ship moves from the frontier waters to when it is secured in its berth, as shown in Figure 8b). A ship first adjusts its attitude in the arrival area to be suitable for berthing. When the ship is close to the dock, to make the ship parallel near the dock, sometimes the blind use of external forces may cause one end of the ship to deflect or bounce owing to excessive speed, damaging the ship or docking facilities [28]. The operators must carefully control the closing angle and speed [29,30]. The above process requires the most attention during the berthing of a ship. Therefore, the data we obtained through 3D LiDAR data focus on the berthing process in Figure 8b. Figure 9 presents the 3D LiDAR data of the ship berthing process after data conversion (Section 2.1.1). A ship usually closes parallel to the berth as much as possible until it contacts the dock to ensure safe berthing. However, parallel berthing is an ideal state because of improper operation or random changes in the operational environment. Therefore, in the actual berthing process of large ships, the angle berthing method is often used, in which the closing angle should be minimized. As shown in Figure 9, the ship used the angle berthing method: one end of the ship approached the berth first, and the other end gradually moved closer at a small angle. Given these primary results, finding a filter method to remove noise points and a recognition method to determine the ship’s bow and stern in order to monitor the berthing process effectively is necessary.

4.2. Filtered Data

The results of Section 4.1 indicate that the 3D LiDAR data of ship berthing are complex and noisy. Therefore, analyzing and monitoring the ship berthing process is challenging. According to its distribution characteristics, the noise data can be divided into fixed facility and random noise data, as shown in the converted data of Figure 10. Therefore, filtering the noise data is necessary to analyze the ship berthing accurately, and the filtering methods are shown in Section 2.1.2. The data filtering results are presented in Figure 11, where the pass-through and statistical outlier removal techniques filter the fixed facility and random noise data, respectively. Additionally, the mean distances to nearest neighbors of points are calculated to verify the same effectiveness of the proposed filtering method as the usual statistical outlier removal. As shown in Figure 12, the mean distances to the nearest neighbors of points are all less than 12 m after filtering, which is significantly smaller than that of the raw point cloud.

4.3. Performance of the Bow and Stern Recognition Method Proposed

Noise data are removed after filtering, but the crucial issue in berthing is recognizing the bow and stern to provide accurate guidance for ship maneuvering. In the Section 2.2.1, the ship’s heading $\overrightarrow{h}$ is defined via PCA, and SRF is established, as shown in Figure 13. Then, the maximum and minimum coordinate points are calculated in the direction of $\overrightarrow{h}$, and the bow and stern feature points are determined.

Subsequently, region growing is used to obtain the segments passing through the bow and stern feature points. The bow and stern are recognized based on normal vector comparison of the segments and the ship’s heading, $\overrightarrow{h}$. The bow, stern and hull can be clearly distinguished, as shown in

Table 4. Normal vector comparison of bow, stern, and hull with ship’s heading.

Cos(η)	Bow	Stern	Hull
Average	0.862	0.044	1.000
Max	0.877	0.060	1.000
Min	0.829	0.032	0.998

.

Figure 14 depicts the bow and stern recognition result and reveals three ship positions relative to the scanner displayed in Figure 6. Figure 14a recognizes the stern, and it corresponds to Figure 6e and Figure 14a’); the stern of the ship first approaches the berth at a small angle. Figure 14b recognizes the bow and stern, and it corresponds to Figure 6f and Figure 14b’; the bow of the ship also gradually moves closer to the berth. Figure 14c recognizes the bow, and it corresponds to Figure 6d and Figure 14c’; the ship is almost parallel to the berth at this time. Finally, the bow and stern points are obtained according to the recognized parts of ship. Because the LiDAR data are dense at a close distance, a voxel of 0.2 m × 0.2 m × 0.2 m downsampling is used to reduce the amount of data. However, the raw data are used to calculate the bow and stern points at the last step, so there is no loss of accuracy. The algorithm is performed on a PC with Intel Core i7-1065G7 1.30 GHz CPU and 4 GB RAM, and the computing times are all less than 450 ms. The time of the recognition algorithm is only costly when the ship attitude changes in Figure 6.

In order to verify the robustness of the algorithm, the LiDAR data are simulated through visibility analysis based on the ship’s 3D model. The changes in input data and ship size are shown in Figure 15a–c. Because the ship attitude shown in Figure 6c did not occur in the real berthing experiment, it is added in Figure 15b. The recognitions are presented in Figure 15a’,b’. Because the other ships that have been berthed will interfere with the extraction of the target ship, a scenario wherein there are other ships already berthed is set in Figure 15c. The berthed ships are considered to exist if segments are shown in Figure 15c’ through Euclidean distance. The segments are filtered when the coordinate changes of the centroids are less than the threshold in consecutive frames before recognition.

In terms of the stability and accuracy of the measurement, the existing ship fitting method RANSAC is fast, well-engineered, and its accuracy is high. Figure 16 presents the top view of the fitting in this case, with max distance set to 0.05 m, and it is divided into several planes causing confusion. Given the measurement and terminal condition of this experiment, the distance of the point is calculated directly in this paper.

4.4. Analysis of Dynamic Berthing Parameters

After recognition of the ship’s bow and stern, berthing parameters are calculated as in the Section 2.2.1. Figure 17, Figure 18 and Figure 19 present the fitted distances, velocity, and approach angle of the stern and bow relative to the dock when the ship was being berthed, respectively. The bow and stern distance relative to the dock kept decreasing throughout the berthing process. The bow velocity increased and then decreased, while the stern velocity did the opposite, and the approach angle rapidly decreased within approximately 150~325 s. The results indicate that the ship was berthed under the force of lateral thrust, which is the most critical operation in the berthing process. There was an increase in approach angle within approximately 325~485 s. Furthermore, the lateral thrust might have been adjusted during this time. Then, the angle was gradually decreasing, and the ship gradually became parallel to the berth.

During the experiment, the scanning range included the whole ship’s berthing process; hence, the berthing ship’s 3D laser point cloud data were obtained. The scanner verification was based on comparing the position measured using the scanner and a 2D laser rangefinder. A camera was used as well to estimate the ship’s dynamics visually. The experimental results indicate the following: the speed and distance of the ship were stable, and no large-scale deviations were present in the monitored values. The approach angle during berthing was small, and it decreased as the ship approached the dock. A parallel approach was used in the final moment of contact, whereat the approach angle α was close to zero. The experimental data confirm that the results are in line with the actual berthing situation. Moreover, continuous experiments and applications were performed with Ship Yukun at Dalian port (Figure 20). The monitoring data were recorded and displayed using our developed web page, enabling us to use the method to monitor ship berthing effectively. Moreover, the ship berthing information extraction system can also be integrated into the berth allocation and monitoring system [31,32] of the terminal operating systems so as to provide aid information.

5. Conclusions

In this study, a method of berthing information extraction using 3D LiDAR data is proposed to monitor a ship’s berthing process. In the data preprocessing, data are converted into a unified format; the scanner coordinate system is converted into the berth coordinate system; according to the characteristics of the random noise and fixed facilities data, the statistical outlier removal and pass-through technique can effectively filter them. A bow and stern recognition method automatically defines the bow and stern segments and points. Then, the distance, the velocity of the bow and stern, and the approach angle to the dock are calculated. The field experiment was conducted based on the Ro–Ro ship Ocean Island at Lushun port, China, to evaluate the performance of the proposed method. The results show the fast and accurate recognition of the bow and stern and the berthing process. Therefore, this is an effective and accurate method that can be used to extract ship berthing information that overcomes the limitations of 1D/2D laser-based monitoring technologies. In future work, berthing processes in different terminal environments and with different ship sizes need to be considered in order to reflect actual situations, and the dock-based LiDAR should be compared with the onboard LiDAR to form a system. The ship berthing information extraction system can easily be integrated into other terminal operating systems to provide extended effective aid information.