Experimental Research on Overwater and Underwater Visual Image Stitching and Fusion Technology of Offshore Operation and Maintenance of Unmanned Ship

Abstract

The new way of offshore operation and maintenance based on unmanned ships has outstanding advantages. Aiming at the problem of lack of overall understanding of the complex environment above and under the water surface during the operation and maintenance of unmanned ships, a stitching and fusion technology of overwater and underwater visual images for unmanned ships is proposed. The software and hardware framework of the overwater and underwater visual image fusion system is constructed, the image processing methods in different environments are defined, and the accurate acquisition of obstacle information is realized. In the two experimental scenarios, the stitching accuracy of the obstacle model based on an extended neighborhood method can reach more than 85% within the obstacle distance of 8 m and more than 80% within the obstacle distance of 14 m. An image-driven Frustum–PointNets detection algorithm is proposed to obtain comprehensive obstacle avoidance information. In addition, the average accuracy of the three-dimensional detection of the algorithm is up to 91.40%. These results are significant and have a good reference value, as it demonstrates that the stitching and fusion method can effectively obtain the comprehensive information of overwater and underwater objects for unmanned ship.

1. Introduction

The ocean is rich in oil and gas, energy, and biological resources, and ocean development has become the strategy of various countries [1]. With the development of the ocean, more and more offshore structures have been put into use, such as offshore wind power platforms and oil production platforms [2,3]. The safety maintenance of offshore platforms is very important. However, the operation and maintenance of conventional ships require a lot of manpower and financial resources, and there are still great risks [4]. At present, the new way of maritime operation and maintenance based on unmanned ships is a development trend.

With the continuous development and improvement of surface unmanned ship technology, the potential business model of unmanned ships has been further explored [5]. The application scenarios of unmanned ships are very broad, involving fields such as surveying and mapping, environmental monitoring, search and rescue, and security patrol [6]. Autonomous collision avoidance in various scenarios is a current hotspot of unmanned ship technology [7,8]. Accurate perception of various obstacles in water in complex environments and obtaining the full picture of collision avoidance objects are the key to autonomous collision avoidance for unmanned ships [9,10].

According to the position of obstacles relative to the water surface, they can be divided into obstacles above the water, obstacles under the water, and obstacles across the water surface. Among them, the shape of obstacles across the water surface extends from underwater to above the water surface, such as icebergs, reefs, and most marine structures. The overall appearance of such obstacles is usually difficult to be obtained by a single sensor. During the navigation of unmanned ships, marine radar and underwater sonar can be used to quickly perceive long-distance obstacles, make collision avoidance decisions, and stay away from the area where the obstacles are located [11]. However, when unmanned ships perform offshore operation and maintenance tasks, they inevitably need to get close to offshore platforms and structures. These offshore structures usually belong to obstacles across the water surface. The irregular shape and different degrees of surface unevenness of their overwater and underwater parts will also pose a threat to the safety of unmanned ships [12]. Therefore, it is of great significance to use visual perception technology and to study image stitching and fusion technology of overwater and underwater images of obstacles which are across the water surface, so as to obtain the full picture of the obstacles and provide comprehensive environmental information for unmanned ships.

In terms of obstacle perception technology for unmanned ships, Szpak Z.L. et al. [13] proposed a method of background subtraction to establish a single Gaussian background model in order to identify and segment the target and the water surface background in the image obtained by a camera, so as to obtain the contour of the target. In order to realize the omnidirectional perception of obstacles, Braginsky B. et al. [14] installed a collision avoidance sonar in the front and on the side of the underwater robot, respectively. Combined with the obstacle perception information of the two sonars, an OAA double-layer obstacle avoidance method was used to realize effective avoidance of the obstacles. On the basis of visual image perception of obstacles, Fang M. et al. [15] proposed research on the control technology of the navigation direction of the vehicle by using the position co-ordinates of the target and the vehicle. This control can select the optimal obstacle avoidance decision path to achieve obstacle avoidance. Gu J. et al. [16] used forward-looking sonar to image the underwater obstacle target. The characteristics of the target were obtained through the analysis and preprocessing of the image, and the detection of the obstacle target was realized according to the main feature matching. After acquiring the sonar image of the underwater target, Williams D.P. [17] used an unsupervised target detection method to achieve accurate measurement of the target’s distance, orientation, and other information. Although there are corresponding recognition technologies for obstacles above the water and obstacles under the water at present, there is a lack of effective overall perception methods for obstacles across the water surface. In particular, there is little research on reconstruction and image stitching and fusion technologies of the above water and underwater parts of the obstacles.

In terms of underwater image detection methods, Raihan A J. et al. [18] considered the distortion of underwater images due to absorption, scattering, polarization, diffraction, and low natural light penetration, and proposed an effective method to restore underwater images. This method can detect more edges and more SURF points by considering the wavelength, attenuation, and backscattering coefficients of red, blue, and green light.

In terms of visual image stitching technology, Berus L. et al. [19] proposed a new edge detection and image stitching method. The results show that, to a certain extent, only the Canny edge detection filter can adequately distinguish the rough boundaries of low-resolution images. Xiu C. et al. [20] proposed a stitching method based on adaptive weighted fusion, which uses an adaptive weighting method to fuse the brightness difference of overlapping areas, realizes smooth transition of stitched images, and eliminates stitching seams. Patel K. et al. [21] proposed a key frame extraction method based on block histogram difference and edge matching rate. The histogram difference value of each frame is calculated, and edge matching is performed on the adjacent images of the acquired video image through the edge detection method of the Prewitt operator. If more than half of them can be successfully matched, the system considers that the selected frame does not meet the key frame requirements. Lu J. et al. [22] proposed an image stitching method based on fuzzy inference, using contrast-limited adaptive histogram equalization to increase the matching points of target surfaces in low-contrast images, reducing ghosting and improving stitching quality. Pollard S. et al. [23] proposed a view synthesis method based on edge matching and transmission of a trinocular vision system. This method does not rely on dense correspondence and has high computational efficiency. Through edge matching, interpolation, occlusion detection, and correction, the automatic view synthesis of three groups of images is finally carried out. In terms of image fusion technology, Li J. et al. [24] proposed a new method for accurate stitching of images acquired by multiple pushbroom imagers, established a relative orientation model based on homophonic points, and then used a strict geometric imaging model to generate seamless stitched images, which improved the precision. Selvaraj A. et al. [25] obtained the edge curve features with better effect through multi-scale decomposition of the visible light image and infrared image obtained in the experiment, and fused the high-frequency coefficient features by using the consistency verification method, which not only suppressed the noise of the image, but also retained the detailed information of the image. Ch M.M.I. et al. [26] proposed a multifocal image fusion method based on cross bilateral filtering and non-subsampling contourlet transform. In this method, the obtained original image and approximate components are filtered bilaterally to obtain the approximate weight graph, the details in the image components are combined by the weighted average method to obtain the detail weight graph, and the detail weight graph is combined with the original image to realize the image fusion processing. Hayat N. et al. [27] proposed a multi-exposure image fusion (MEF) technology that does not require manual operation. This technology generates high-quality output images by acquiring multi-exposure input images. This method obtains the weight graph of the original image by analyzing the information of image contrast, saturation, and exposure, and uses the multi-resolution pyramid decomposition method to mix and generate the final fused image. Paramanandham N. et al. [28] proposed a new image fusion technology based on the similarity of continuous pixel intensity. The noise-free and noisy parts of the source image are detected, and the optimized particle swarm algorithm is used for image fusion. Experiments were carried out on the images affected by Gaussian noise, Poisson noise, and speckle noise, and high-quality fused images were obtained. At present, most edge stitching technologies are applied in the fields of file restoration, broken coins stitching, aerial images, etc. There is little research on overwater and underwater image stitching. Most of the visual image fusion techniques are researched on two or more images with overlapping areas, while there is less research on segmented images without overlapping areas. In addition, the research on the fusion technology of overwater images and underwater images of navigation obstacles of unmanned ships is even less.

Therefore, in view of the difficulty in obtaining the overall information of obstacles across the water surface and the overall picture of the environment during the operation and maintenance of the unmanned ship at sea, in this paper, the overwater and underwater visual perception technology is used to obtain the obstacle images. By expanding the neighborhood of the edge of the obstacle image, combining the characteristic information of the neighborhood with the outer edge contour, and stitching and fusing the overwater and underwater images of the obstacle, the overall panoramic information of the obstacle is finally obtained, so as to provide useful information for unmanned ship operation.

2. Features and Framework of Unmanned Ship Visual Image Fusion System

The fusion characteristics of the overwater and underwater images of unmanned ship obstacles are analyzed, and then the hardware and software frameworks of the unmanned ship visual image stitching and fusion system are established.

2.1. Analysis of Visual Image Fusion Characteristics

For the overwater image of obstacles, due to the mirror effect of the water surface, the image of obstacles will produce reflection. In low sea conditions, the characteristics of the obstacle reflection are similar to the actual image of the obstacle, so the symmetry can be used to remove the reflection to obtain the actual image of the obstacle. In high sea conditions, the reflection of the obstacle is quite different from the actual image. At this time, the reflection can be regarded as the image background. Therefore, the actual image of the obstacle can be obtained by removing the background. After obtaining the overwater image of the obstacle, in order to reduce the huge amount of data processed by the system and reduce the complexity of feature extraction, it is necessary to perform edge detection on the overwater obstacle image to reduce the redundant information of the pixels and to obtain the fusion features of the main edges and their extended neighborhoods.

For the underwater images of obstacles, there will be a certain fogging phenomenon due to the refraction, reflection, and absorption of light waves. In addition, impurities in water will also produce noise interference to the images. In low sea conditions, the underwater part of the obstacle will form a mirror image with the water surface as the symmetry axis. In this case, the symmetry can be used to remove the mirror image and segment the underwater part image of the obstacle. In high sea conditions, the mirror image of the underwater part of the obstacle begins to be distorted and blurred, so it can be removed as the background without affecting the segmentation and acquisition of the underwater part of the obstacle.

In the process of unmanned ship operation, the image, contour, distance, orientation, and other information of the overwater part of the obstacle are obtained through the overwater camera and lidar, and the same information of the underwater part of the obstacle are obtained through the underwater camera and sonar ranging system. Then, the overwater image and the underwater image are fused based on the edge features of the two images. In order to reduce the amount of calculation of image processing and improve the robustness of image fusion, this paper carries out image edge detection on both the overwater image and the underwater image, and completes the image stitching of the obstacle through the characteristics of the pixel points of the image edge and its extended neighborhood.

2.2. Hardware Framework of Visual Image Fusion System

Obtaining more accurate information of the overwater and underwater parts of the obstacle is very important for the final visual image fusion. After determining the basic structure of the hull and the power system of the unmanned ship, this paper designs a reliable and efficient visual image fusion system for the unmanned ship obstacles.

The hardware framework of the visual image fusion system is shown in Figure 1. It consists of two parts: information display and parameter control system, and visual image fusion and collision avoidance decision system.

The information display and parameter control system is realized by the host computer on shore, and its function module mainly includes two parts: ➀ the obstacle information display module, which can display the size, distance, and orientation of the obstacles, and ➁ the control parameters display and input module, of which the control parameters mainly include emergency stop, remote control, path planning, and other instructions. Through the information transmission system, the host computer on shore can receive the environmental information obtained by each environmental sensing device in real time and display the data processed by the system on the interface of the host computer.

The visual image fusion and collision avoidance decision system is divided into the following five modules: ➀ overwater obstacle perception system, which obtains overwater information of the obstacles, ➁ underwater obstacle perception system, which obtains underwater information of the obstacles, ➂ navigation and positioning system, which obtains real-time position and attitude information of the unmanned ship, ➃ energy power system, which provides power supply of unmanned ship, and ⑤ information transmission system, which performs remote wireless transmission of information of the unmanned ship.

2.3. Software Framework of Visual Image Fusion System

The software framework designed in this paper is shown in Figure 2, which is mainly divided into two modules: ➀ human computer interaction module, which is mainly for the convenience of shore-based staff to manually intervene and control the unmanned ship when necessary and to timely grasp the current heading and hull condition of the unmanned ship, and ➁ image stitching and fusion module, which obtains the information of the overwater part of the obstacle through the underwater camera and lidar, obtains the information of the underwater part of the obstacle through the underwater camera and sonar ranging system, and, finally, carries out image fusion to obtain the full picture of the obstacle.

3. Fusion Technology of Overwater and Underwater Images

After the overwater and underwater images of the obstacles of the unmanned ship are obtained, the images are first preprocessed, then obstacle segmentation and edge detection are performed, and then the overwater and underwater images are spliced, and, finally, the obstacle information is fused.

3.1. Image Preprocessing

Technical analysis and method realization are carried out, respectively, for overwater image defogging, underwater image clarity, and reflection detection.

3.1.1. Defogging Processing of Overwater Image

In view of the problems of insufficient transmittance estimation, slow processing speed, and high complexity in the traditional defogging technology in complex environments, and in order to meet the requirements of dynamic adaptability, processing speed, and image detail enhancement of image defogging during the navigation of the unmanned ship, transmittance optimization and fast estimation of atmospheric light values are conducted in this paper on the basis of the dark channel prior principle [29].

3.1.2. Clarity Processing of Underwater Image

The factors causing overwater image fogging and underwater image fogging are different. The fogging of the overwater image is mainly due to the influence of water vapor or fog, while that of the underwater image is mainly caused by the absorption and scattering of light by the water body. In this paper, an improved relative global histogram stretching method is used to restore and enhance the underwater image of the obstacle.

3.1.3. Obstacle Reflection Detection

Under low sea conditions, especially in the case of sufficient lighting, the actual image of the obstacle and its reflection have little difference in color and brightness, forming a strong reflection. In this paper, the underlying descriptor differential feature extraction method is used to extract the distance feature and intensity feature of the pixel point. Then, the symmetry axis detector based on multi-instance learning is used to detect the symmetry axis between the actual image and the reflection of the obstacle.

3.2. Image Segmentation and Edge Detection

After image preprocessing, the obstacles in the overwater image and underwater image need to be segmented and edge detected. In this paper, an obstacle image segmentation method based on an automatic multi-seed region growing algorithm is adopted, and then the edge detection method based on Canny algorithm is used to detect the contour of the obstacle.

3.2.1. Image Segmentation

The image to be detected includes obstacle region M and background region N. The ideal image segmentation is to accurately distinguish the obstacle region M from the background region N. However, in the actual image, there is a blurred transition region between M and N, which makes it difficult to segment them accurately. In order to achieve effective segmentation and better clarify the segmentation algorithm, the obstacle region M is divided into strong obstacle region P₁ and weak obstacle region P₂, and the background region N is divided into weak background region P₃ and strong background region P₄. The distribution of the four regions is shown in Figure 3.

According to the above division method, the following relationship exists:

$$\{\begin{array}{c}M=P_1+P_2\\ N=P_3+P_4\end{array}$$

(1)

Among them, the strong obstacle region P₁ and the strong background region P₄ are considered to have obvious image features, which are easy to be segmented and rarely produce segmentation errors, while the weak obstacle region P₂ and the weak background region P₃ are in the blurred transition region, which is difficult to accurately segment, and it is considered that there will be segmentation errors in general.

The segmentation idea of this paper is to find the strong background region P₄, then set seed points on P₄, and execute the region growth algorithm to realize the final background segmentation of the image.

The strong background region P₄ can be determined by the following formula.

$$P_4=N_1\cap N_2\cap \cdots \cap N_n (n \ge 2)$$

(2)

where N₁, N₂, … and N_n are the background area of the image obtained by segmentation method 1, segmentation method 2, … and segmentation method n, respectively.

The specific method of this paper is to obtain the background region N₁ through brightness threshold segmentation, and then obtain the background region N₂ through saturation threshold segmentation, and then use the common region of N₁ and N₂ as the strong background region P₄. It should be noted here that, in the strong background area P4, due to the fluctuation of water surface, P4 may be one connected area or several unconnected areas. Selecting seed points from each area can avoid false detection and missing detection. The flow chart is shown in Figure 4.

3.2.2. Image Edge Detection

After image preprocessing, the edge detection method based on Canny algorithm can be directly used for obstacle contour detection. However, for the application scenario of unmanned ships, the image information obtained may be different in resolution and noise pollution. The gamma transform is used to enhance the detail information of the dark part of the image, and then the median filter is used to denoise the image. The traditional Canny algorithm edge detection method cannot adapt to the edge detection of obstacle images under all sea conditions. This paper improves the above problem by introducing an adaptive threshold. The flowchart of the improved algorithm is shown in Figure 5.

In this paper, the original 2 × 2 Roberts operator of the Canny algorithm is replaced by a 3 × 3 Sobel operator. On this basis, two gradient directions, 45° and 135°, respectively, are added to obtain the gradient size more accurately. The calculated gradient size and angle after increasing the gradient direction are:

$$G\left(x,y\right)=\sqrt{G_x^2\left(x,y\right)+G_y^2\left(x,y\right)+G_{{45}^0}^2\left(x,y\right)+G_{{135}^0}^2\left(x,y\right)}$$

(3)

$$\theta \left(x,y\right)={\tan }^{-1}\left(G_y\left(x,y\right)/G_x\left(x,y\right)\right)$$

(4)

The gradient size calculation method by increasing the gradient direction of 45° and 135° can obtain more accurate edge detection results and reduce the interference of redundant and complex information.

In this paper, by comparing and analyzing the statistical information of pixel intensity in the obstacle image, the optimal threshold size is automatically determined, and an adaptive threshold method based on the calculation of the gradient intensity statistical data is realized. The specific implementation steps of this method are to count the gradient size of all pixels in the obstacle image, obtain the maximum V_max, minimum V_min, and median values V_med of single channel pixel intensity, and calculate its average value according to the following formula.

V = (V_max + V_med + V_min)/3

(5)

Then, the high threshold and low threshold of the adaptive threshold are adjusted through the value of a parameter $\sigma$. The algorithm is as follows:

$$lower=\max \left(0,\left(1-\sigma \right)\times V\right)$$

(6)

$$upper=\min \left(255,\left(1+\sigma \right)\times V\right)$$

(7)

After many experiments, it is found that, when $\sigma$ = 0.32, the processing effect of the algorithm is the best. The method can quickly obtain the high threshold and low threshold of the double-threshold detection and reduce the time consumption of the algorithm.

3.3. Image Stitching and Fusion

This part realizes the stitching of the overwater image and underwater image of the obstacle through the matching and calculation of image coding points. In addition, the 3D fusion detection of the obstacle is carried out by using the overall contour of the stitched obstacle as the image driver.

3.3.1. Image Edge Stitching

The representation of edge contour can be described by chain codes, which describe the edge characteristics of the image by obtaining the coding of the image boundary. The eight-way code direction symbol used in this paper is shown in Figure 6. The eight numbers 0–7 in the eight-way chain code represent eight directions, respectively. In the eight-way chain code rule, the boundary of a pixel in the image is shown in Figure 7.

The principle of fast stitching and matching of the overwater and underwater edge detection binarized images of unmanned ship obstacles by chain code is as follows: first, obtain the characteristic original code of the region of interest (ROI) of edge detection of the overwater image and underwater image, respectively. Then, judge the original codes of the two images. If the original codes are the same, they will be matched. Otherwise, they will not be matched. Finally, the ROI of the edge detection binarized image of the overwater and underwater obstacle is stitched. According to the original code of the matched image obtained in the above steps, the corresponding subcode space is established, and the subcode is matched through the complementary relationship between the subcodes, which is established by the original code to be matched. If the subcode can be matched, it is considered that the pixels of the edge detection binary image of the overwater obstacle and the underwater obstacle corresponding to the subcode can be stitched; otherwise, it will not be stitched.

The complementary relationship of the subcodes of the eight-way chain code is shown in

Table 1. Complementary relationship of eight-way chain code values.

	Complementary Relationship
code value	0	1	2	3	4	5	6	7	0
Complementary code value	4	5	6	7	0	1	2	3	4

. In order to reduce the calculation of unnecessary pixels, only the underwater images of obstacles and the pixels of the edges to be stitched and their neighborhoods of the underwater images are coded. For example, if the chain code is shown in Figure 8, its original code is 10276534 and its complement is 54632170.

In order to minimize the excessive coding of the image and the possible noise influence, this paper takes the image to be stitched as the benchmark, uses the disk model to define the characteristics of each pixel, and determines the starting point of coding and the coding principle of the image edge. There are two types of points that need to be encoded in the edge detection binarized image of the obstacle water image and the underwater image: one is the feature point less than the radius of the disc, and the other is the longest end point of the encoded line segment.

The detection criteria for feature points smaller than the radius are:

$$R_1=\underset{i=1}{\overset{n}{\prod {\lambda }_i}}-k{\left({\displaystyle \sum _{i=1}^n{\lambda }_i}\right)}^2$$

(8)

$$\left|{\lambda }_{i}E-M\right|=0$$

(9)

$$M={\displaystyle \sum w\left(r_i\sin {\theta }_i,r_i\cos {\theta }_i\right)}\times r^2\left(\begin{array}{cc}{\sin }^2{\theta }_i& \sin {\theta }_i\cos {\theta }_i\\ \sin {\theta }_i\cos {\theta }_i& {\cos }^2{\theta }_i\end{array}\right)$$

(10)

$$\omega \left(r\sin \theta ,r\cos \theta \right)=R_{\omega }\left(n\right)$$

(11)

$$R_{\omega }\left(n\right)=\epsilon \left(n-\omega +1\right)-\epsilon \left(n\right)$$

(12)

$$\frac{f\left(r_i,{\theta }_i\right)-f\left(r_j,{\theta }_j\right)}{r_i-r_j}=k$$

(13)

where $R_1$ represents the code point detection function, $M$ represents the function matrix of corner scanning, $\omega$ and $R_{\omega }$ both refer to the rectangular window function, and $f\left(r,\theta \right)$ is the curve-fitting function for the contour.

The detection standard for the longest end point of the encoded line segment is:

$$I\left(r,\theta \right)=f(r,\theta )+\delta \times g\left(r,\theta \right)$$

(14)

$$\hspace{0.17em}\left|I\left(r,\theta \right)-f\left(r_i,{\theta }_i\right)\hspace{0.17em}\right|\ne 0$$

(15)

where $\hspace{0.17em}\forall {\theta }_i,{\theta }_j\in \left[0,2\pi \right]$, $g\left(r,\theta \right)$ function represents the direction, and $\delta$ represents the function that can be fine-tuned.

The specific process is: firstly, the disc model is established for the edge detection images of the overwater obstacle and the underwater obstacle. Then, set the edge of the images to be stitched and their extended neighborhood as the coded ROI area. Take the disc point O determined by the stitching model as the starting point, and take the disc radius R as the maximum coding length to determine the first coding point. Then, take this point as a new starting point and set the coding direction within $\pm 90°$ and the coding length to be less than or equal to R to select the next coding point. According to the characteristics of the obstacle images, when selecting the next coding point, first determine the direction, and then determine the coding length. According to the coding rules, select the next coding point again until all the coding points suitable for coding in the disc model are selected. The flowchart is shown in Figure 9.

It is assumed that all coding points have a total of n decision directions, and the ordering and coding length are determined according to the coding points. Then, the coding direction of each coding point is $\alpha =2\pi /n$, and the coding length is $L_i=\left(R-r_i\right)\times \left({\theta }_i-{\theta }_{i-1}\right)$. The Euclidean distances of R/4, R/2, and R for the coding length $L_i$ are calculated, and then the coding length corresponding to the shortest Euclidean distance calculated is selected as the final decision coding length, so as to obtain the original code of the obstacle images. The formulas for solving the n-direction normalized differential code $s_i$ and the n-direction complement code $t_i$ are as follows:

$$\{\begin{array}{l}{S}_i={\left(a_{i+1}\right)}_n-{\left(a_i\right)}_n\\ S_n={\left(a_0\right)}_n-{\left(a_n\right)}_n\\ t_i={\left(S_i\right)}_o+(n+1)/2\end{array}$$

(16)

where ${\left(a_i\right)}_n$ refers to the original code value of the i-th point of the n-direction, and ${\left(S_i\right)}_o$ refers to the code value of the i-th point after the normalized difference.

Next, the subcode space is established to obtain the subcode. First, three space sets are defined. $E^n$, $S^n$, and $T^n$, respectively, are used to represent the space sets of the original code, the normalized differential code, and the complement code. The corresponding relationship between the three is as follows:

$$\{\begin{array}{c}{E}^n\stackrel{f\left(s\right)}{\to }{S}^n\\ S^n\stackrel{f\left(t\right)}{\to }{T}^n\end{array}$$

(17)

where the subcode space $S_i$ and $T_i$ are subsets of sets $S^n$ and $T^n$, respectively.

The test takes the underwater image of an iceberg model as an example to calculate its normalized differential code. Figure 10 is the coding model of the underwater image of the iceberg model, where 1 refers to the relative position, 2 refers to the stage radius, and 3 refers to the concentric angle. Because the stitching of the overwater image and the underwater image of the obstacle is based on the edge feature, this paper selects the edge and its extended neighborhood as the coding point. The final selection of the coding point for the underwater image of the obstacle is shown in Figure 11. From this, it can be concluded that the original code of the center pole code of the edge of the underwater image of the obstacle is 4541474345435414541460, and the corresponding complement code is 0105030701071050105024.

Normalized chain code is conducive to the unique representation of a code point, but it does not have rotation invariance. In the process of image stitching, it is required that the chain code belonging to a certain edge is unique and does not change after moving and rotating. Normalized differential code has this characteristic. Therefore, in order to obtain the chain code that meets the stitching requirements, the original code needs to be differentiated and then normalized to obtain the normalized differential code. Next, taking the original code of the underwater image of the obstacle as an example, the solution of the normalized differential code is carried out, as shown in Figure 12.

The normalized differential code of the underwater image of the obstacle can be calculated as 1177275317532241753357, while that of the overwater image of the obstacle can be calculated as 1177275317462241753357 by using the same calculation method.

It can be seen that, because the acquisition and processing of the image are affected by the water environment, perception equipment, and processing algorithm, the normalized differential codes of the edge of the overwater image and the underwater image are not exactly the same. However, the matching ratio of the two codes has reached more than 90%, which meets the conditions of edge matching within the allowable range of error.

After obtaining the normalized differential codes of the extended neighborhood edge of the overwater image and the underwater image of the obstacle, the fast matching and stitching of the images can be carried out. Feature extraction and matching calculation must meet the following conditions:

$$\exists m\hspace{0.17em},\hspace{0.17em}{S}_m\in S_i\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}{S}_m\in S_j$$

(18)

while edge stitching must meet the following conditions:

$$\exists a,b\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}{S}_a\in S_i\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}{t}_b\in t_i\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}{S}_a=t_b$$

(19)

where $S_i$ and $S_j$, respectively, refer to the set of edge normalized differential codes to be matched between the overwater image and the underwater image of the obstacle, $S_m$ and $S_a$ are their subcodes, $t_i$ is the set of complementary codes to be stitched, and $t_b$ is the subcode of $t_i$. Due to the existence of errors, it is impossible to obtain a completely consistent normalized differential code of the edge to be stitched of the overwater image and the underwater image of the obstacle. Therefore, in the stitching process, the subcode space is used for matching and stitching. The following takes the 5-bit normalized differential code 01234 as an example to illustrate the subcode space establishment, which is shown in

Table 2. Establishment of subcode space.

Original Code	5-Bit Code Long Subspace	4-Bit Code Long Subspace	3-Bit Code Long Subspace	2-Bit Code Long Subspace	1-Bit Code Long Subspace
01234	01234	0123, 1234, 2340, 3401, 4012	012, 123, 234, 340, 401	01, 12, 23, 34, 40	0, 1, 2, 3, 4

.

It can be seen from the table that the establishment rule of subcode space is to move cyclically according to the sequence of coding points, so that the continuity and accuracy of each subcode during stitching can be ensured. In the matching process, the complexity of the calculation method of the extracted features also directly affects the final stitching effect. The more complex the calculation method is, the more accurate the feature representation will be, which can further reduce the stitching error rate, but it will also reduce the speed of matching and stitching.

3.3.2. Fusion Algorithm

The perception of obstacles based on visible images only obtains the obstacles’ contour and texture, and lacks the depth information of the obstacles, while the point cloud information obtained by radar and sonar contains the depth information of the obstacles. Image information can help radar and sonar to extract and segment targets faster, and images incorporating depth information are given information such as obstacle size and position. Therefore, more comprehensive information of obstacles can be obtained by fusing the image information with the point cloud information obtained by radar and sonar.

PointNet [30] is a classic architecture that uses neural networks to process the target point clouds. The architecture applies the maximum pooling function to extract the features from the target point clouds. In order to effectively utilize the spatial relationship between point clouds, PointNet also connects the point-by-point features and global features of the target, and uses the obtained aggregated information to classify and segment the target. It directly processes the target point cloud, which greatly reduces the loss of information. However, since PointNet uses a multi-layer perception method to extract the target point cloud point by point, the relationship between points is not fully considered, which can easily cause misclassification and mis-segmentation. Therefore, according to the needs of this paper, an image-driven Frustum–PointNet algorithm is proposed.

In the obtained point cloud data, the redundant water surface, background, and other information are nonimportant information. In order to make the information processing of the point cloud network more inclined to the target area, this paper also introduces an attention module, which can make the network learn the location of the target and enhance the expression of the area. The point cloud target type of obstacles is n × 3, which has two dimensions: point and channel. It is necessary to add the attention mechanism to these two dimensions. Figure 13 is the schematic diagram of the channel attention module, and Figure 14 is the schematic diagram of the point attention module.

For the channel attention module, first the obtained point cloud features are input, in which the channel features of the points are solved by maximum pooling and average pooling, and then multi-layer perceptron learning is carried out to obtain the distribution attention of the channel.

The processing process of the point attention module has something in common with the channel. The acquired attention channel features are sent to the multi-layer perceptron for activation to obtain the attention weight of the point attention feature, and then multiplied with the input feature to obtain the point attention features. In this paper, the feature of the target area is obtained first through the channel attention module, and the important points of the target area are obtained through the point attention module, so as to suppress invalid information and enhance the role of effective information.

Compared with the view projection or voxelization method, PointNet point cloud network can minimize the loss of target information, so that the learned 3D information of the target is more complete, especially for the small obstacles; it also has better detection performance.

The Frustum–PointNet algorithm can utilize both 2D image and point cloud information for obstacle detection. Firstly, the 2D image of the obstacle is upgraded to the 3D space through the ascending dimension projection, and then the view cone point cloud is extracted. The 3D segmentation of the input view cone point cloud is performed through the Frustum–PointNet network to obtain the target area of the view cone point cloud, the co-ordinate transformation of the obtained target point cloud is carried out through t-net, and, finally, the 3D frame regression is carried out through PointNet to obtain the specific parameters of the obstacle. The stitched image of obstacles is used to drive the obstacle detection of the 3D point cloud, which can realize accurate positioning of the point cloud. The specific algorithm flowchart is shown in Figure 15.

4. Experiments of the Overwater and Underwater Image Fusion System

The experimental platform is set up, the fusion technology experiment of the overwater and underwater visual images of the unmanned ship is carried out, and, finally, the results are analyzed.

4.1. Layout of the Experiments

The unmanned ship of the experimental platform is independently developed by the research group, which is equipped with various sensors, industrial computers, and upper computers required for the experiment. It can realize the basic functions of target detection, path planning, autonomous berthing, and static and dynamic collision avoidance under specific conditions. The equipment models and characteristics used in this paper are shown in

Table 3. The model and characteristics of the equipment.

	Model	Physical Map	Characteristics
Camera	DS-2CD1211-I3		With infrared sensing function, the lens aperture is 4mm, the horizontal field of view is 81°, the resolution can reach 1920 × 1080@25fps, the current mainstream video transmission protocols can be supported, and the adaptive network port can be switched between 10 M or 100 M.
Lidar	Pandar-40p		It has 40 sets of laser transceiver pairs, which can be rotated at all angles to obtain 3D point cloud information of the target in real time, and its static ranging accuracy is less than ±2 cm.
Underwater camera	UC5032		The shell is made of 316 L stainless steel, which is corrosion-resistant and can adapt to complex underwater environments. The maximum pressure-resistant water depth is 60 m. The 2.8–12 zoom lens can capture fast-moving objects at a frame rate of 60. It supports current mainstream protocols.
Sonar	FarSounder-500		Its horizontal viewing angle can reach 90°, vertical angle can reach 60°, and the operating frequency is 61 kHz. It enables single ping 3D imaging.
GPS	BD970 GNSS		Obtains the exact latitude and longitude of the drone.
3D Electronic compass	SEC345		Obtains the heading angle, roll angle, and pitch angle of the unmanned ship.

. On this basis, this paper studies the fusion method of the overwater and underwater visual images of obstacles for the unmanned ship in specific scenes, so as to realize the effective collision avoidance of the unmanned ship against the obstacles with both overwater and underwater bodies. In this paper, the iceberg model and pontoon model are selected as obstacles to carry out experimental research in a wave pool, as shown in Figure 16.

The running environment of the experiment is shown in

Table 4. Experiment running environment information.

Category	Name	Model	Performance
hardware	CPU	Intel I7-6700k	4 GHz × 4
	RAM	DDR4	2400 MHZ 32 GB
	GPU	NVIDIA GTX 1060(6G)	4.4TFLOPs of single-precision floating-point operations
software	system	Windows 10
	environment	Python3.6
	DL framework	Pytorch1.2.0

. The experimental environment is configured on the laboratory host computer, and the model is built and trained on the basis of the Pytorch framework. The collection of the data set is carried out under the conditions of the laboratory. A total of 500 training samples and 100 test samples are collected, and the test samples include obstacle images and their corresponding point clouds. The data set contains the calibration files between each sensor, and the correspondence between the obstacle images and the radar point cloud space and sonar point cloud space can be obtained through the calibration files.

The system interface of information display and parameter control of the experimental platform is shown in Figure 17, which can display the real-time navigation data of the unmanned ship and the obstacle avoidance information of obstacles in real time, and conduct human–vessel interaction with the unmanned ship in real time.

4.2. Image Segmentation and Obstacle Edge Detection

Figure 18 shows the segmentation results of an obstacle. It can be seen that the method in this paper can eliminate the reflection and background, and retain the obstacle information to the greatest extent. The experimental results show that the image segmentation method based on the automatic multi-seed region growing algorithm in this paper can not only keep the segmentation accuracy of the region growing image, but also avoid the problem that the traditional region growing algorithm has great dependence on the seed point and seed growth order. The selection of seed points does not need artificial participation, which improves the automation performance of the image processing.

For the edge detection of the overwater and underwater images of obstacles, the results of the adaptive threshold algorithm proposed in this paper are compared with the results of the traditional Canny algorithm where the threshold is set manually, as shown in Figure 19. It can be seen that the false edges of the obstacles detected by the algorithm in this paper are greatly reduced, the loss of the true edges is relatively less, and the obtained edges are smoother. The algorithm retains the important edges of the images to the greatest extent, which lays a good foundation for the subsequent image stitching and fusion processing.

In order to further verify that the adaptive threshold algorithms in this paper have a good edge detection effect while maintaining high anti-noise characteristics, this paper makes a comparative study on the detection effect of the common threshold algorithms. The comparison index is the average peak signal-to-noise ratio (PSNR) [31], for which the mathematical expression is:

$$\{\begin{array}{l}PSNR=10\times {\log }_{10}(\frac{{255}^2}{MSE})\\ MSE=\frac{1}{mn}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{(I(i,j)-K(i,j))}^2}}\end{array}$$

(20)

where MSE is the overall mean-square error of the two images before and after denoising, m and n are the length and width of the image, i and j represent the (i, j) pixel of the image, and I(i, j) and K(i, j) represent the pixel value of the point (i, j) before and after denoising respectively.

The comparison results are shown in Figure 20. In addition, the larger the PSNR, the better the image quality. Among the five algorithms, the peak signal-to-noise ratio of the algorithm in this paper is higher, and the average signal-to-noise ratio can reach 7.05. Combining with Figure 21, it can be seen that the algorithm in this paper has a better enhancement effect on the obstacle image and can retain the useful information of the image.

4.3. Quick Stitching of Images

Figure 21 shows the comparison between the stitching results of the overwater image and the underwater image of the obstacles and the real object. Due to the influence of objective environmental conditions, the detection errors of the system and shipborne sensors, and the light of the overwater and underwater environment, there are some errors in the stitching results. However, the errors are within the allowable range and the stitching results have achieved the desired effect.

The quality of the experimental results is measured by the edge description integrity error of the normalized differential code and the stitching error in this paper. The edge description integrity error is calculated by the following formula:

$${\eta }_1=\left|1-\frac{S}{C}\right|$$

(21)

$$S={\displaystyle \sum _{i=1}^n{a}_i}\hspace{0.17em},\hspace{0.17em}{a}_i\in \left[0,R\right]$$

(22)

where $S$ is the total length of the normalized differential codes of the coding points, $C$ is the fitting boundary length of the edge to be stitched in the obstacle image, and $a_i$ is the code length of the normalized differential code. The edge description integrity error in this paper is within 10%, which is an acceptable error range.

The calculation formula of the stitching error is:

$${\eta }_2=\frac{E_a+E_b}{S_a+S_b}$$

(23)

where E_a and E_b are, respectively, the lengths of the normalized differential codes of the code points that are not stitched in the two images to be stitched. S_a and S_b are, respectively, the total length of the normalized differential codes of the coding points of the two images.

Four wave conditions are set in the experiment, which are no wave, calm wave, smooth wave, and slight wave. For each wave condition, seven groups of experiments were carried out when the obstacle was 2 m, 4 m, 6 m, 8 m, 10 m, 12 m, and 14 m away from the bow of the unmanned ship. The results are shown in Figure 22.

Due to the systematic errors and the complexity of the environment, some points with large errors appear in the experiment, so it is necessary to eliminate these abnormal points. It can be seen from the experimental results that the better the water environment and the closer the distance between the obstacle and the unmanned ship, the higher the image acquisition quality, the clearer the texture, and the higher the image stitching accuracy. For the experimental scene of the obstacle 1, the image stitching accuracy is close to 90% when the obstacle distance is less than 8 m, and the accuracy is more than 80% when the obstacle distance is less than 14 m. For the experimental scene of the obstacle 2, the image stitching accuracy is close to 85% when the obstacle distance is less than 8 m, and the accuracy is also more than 80% when the obstacle distance is less than 14 m. The image stitching accuracy can meet the accuracy requirement of obstacle avoidance of the unmanned ship in the test. In the obstacle 1 and the clam wave environment, the method in this paper is compared with the methods of SITF [32], Harris [33], and Shape-Optimizing [34], and the results are shown in Figure 23. It can be seen that, with the increase in distance, due to the weakening of some edge features of obstacles in the image, the stitching accuracy of all methods gradually decreases, but the accuracy of this paper is always the best.

The stitching efficiency of the overwater and underwater obstacle images is mainly reflected in two aspects: one is the chain code acquisition time, and the other is the matching completion time. In this paper, the stitching efficiency of the normalized differential code and the traditional Freeman chain code are compared experimentally. The comparisons of their chain code acquisition time and matching completion time are shown in

Table 5. Chain code acquisition time comparison.

Environmental Conditions	Freeman Chain Code	Normalized Differential Code
No wave	326 ms	351 ms
Calm wave	341 ms	380 ms
Smooth wave	369 ms	404 ms
Slight wave	390 ms	426 ms

and

Table 6. Comparison of the average time to complete a stitching matching.

Operation	Freeman Chain Code	Normalized Differential Code
Matching search	336 ms	121 ms
Stitching completed	379 ms	145 ms
Similarity search	235 ms	101 ms

, respectively.

According to the data in the above two tables, it can be seen that the chain code acquisition time of the normalized differential code is longer than that of the Freeman chain code, but the time to complete a stitching matching of the normalized differential code is far less than that of the Freeman chain code. From the overall analysis of the stitching process, the processing speed of the normalized difference code method has great advantages. It can reduce some false matching and reduce the matching time, which is more suitable for the image stitching of the unmanned ship in this paper.

4.4. Experiment Results

Figure 24 shows the fusion detection results of the iceberg model obstacle using the image-driven Frustum–PointNet detection algorithm. The stitched image of the obstacle is used as the image driver and the point channel attention module and the channel attention module are added to ensure the overwater and underwater point cloud images can be quickly and accurately identified. In addition, it is fused with the obstacle edge contour image to obtain the 3D regression border of the obstacle, so as to realize the overall accurate positioning of the obstacle. Then, the obtained information, such as the direction, position, and size of the obstacle, is sent to the information display and parameter control system of the unmanned ship to realize the obstacle perception and obstacle avoidance control of the unmanned ship.

In order to verify the performance of the image-driven Frustum–PointNet algorithm proposed in this paper, the algorithm is compared with several other classical detection algorithms, including the MV3D algorithm [35], the AVOD algorithm [36], and the Frustum–PointNet algorithm [37] in the same experimental environment.

Table 7. Comparison of 3D detection accuracy of different fusion algorithms.

Fusion Algorithm	3D Detection Accuracy When Obstacle Distance Is 4 m	3D Detection Accuracy When Obstacle Distance Is 8 m	3D Detection Accuracy When Obstacle Distance Is 12 m
MV3D	75.98%	74.58%	72.98%
AVOD	85.21%	84.71%	82.56%
Frustum–PointNet	84.39%	83.01%	81.59%
Image-driven Frustum–PointNet	92.12%	91.90%	90.17%

shows the comparison of their 3D detection accuracy. Figure 25 shows the relationship between accuracy and distance. It can be seen that the accuracy of each method decreases as the distance increases, while our method is still at a high level.

It can be seen from the data in the above table that, except the MV3D algorithm, the 3D detection accuracy of the remaining algorithms is greater than 80%. In addition, the detection accuracy of the image-driven Frustum–PointNet algorithms is superior to other algorithms. In addition, when the obstacle distance increases, the detection accuracy of each algorithm decreases. The reason is that, the farther the distance is, the systematic errors and random errors of each sensor in acquiring information will increase due to the influence of the environment, resulting in the decline in the performance of the algorithm. However, the algorithm in this paper can still maintain its superiority, and the performance index can also meet the experimental requirements.

The running speed of each fusion algorithm is also compared in this experiment, and the results are shown in

Table 8. Speed comparison of different fusion algorithms.

Fusion Algorithm	Inference Time (ms)	Running Speed (Frames Per Second)
MV3D	320	2.6
AVOD	130	9.4
Frustum–PointNet	150	5.5
Image-driven Frustum–PointNet	75	12.5

.

As can be seen from the above table, the average inference time of the image-driven Frustum–PointNet algorithm proposed in this paper is 75 ms, and the running speed is 12.5 frames per second, which is better than other algorithms. The image-driven Frustum–PointNet algorithm not only improves the performance of obstacle detection, but also reduces the time consumption and improves the robustness of the algorithm by adding the attention module.

5. Conclusions

Aiming at the problem of lack of overall understanding of the complex environment above and under the water surface during the operation and maintenance of unmanned ships, a fusion technology of overwater and underwater visual images for unmanned ships is proposed in this paper. The software and hardware framework of the overwater and underwater visual image fusion system is constructed, the image processing methods in different environments are defined, and experimental research is carried out. The following conclusions are obtained in the experiment:

(1)

For the edge detection of the overwater and underwater images of obstacles, the false edges of the obstacles detected by the adaptive threshold algorithm proposed in this paper are greatly reduced, the loss of the true edges is relatively less, and the obtained edges are smoother. The algorithm retains the important edges of the images to the greatest extent, which lays a good foundation for the subsequent image stitching and fusion processing.

(2)

The image stitching accuracy of the obstacle model based on the extended neighborhood method can reach more than 85% within the obstacle distance of 8 m, and more than 80% within the obstacle distance of 14 m, which can meet the accuracy requirements of obstacle avoidance of unmanned ships.

(3)

The accuracy of the three-dimensional detection performance and three-dimensional positioning performance of the image-driven Frustum–PointNets algorithm proposed in this paper can reach more than 90%. The algorithm not only improves the performance of obstacle detection, but also reduces the time consumption and improves the robustness of the algorithm by adding the attention module.

(4)

In this paper, an overwater and underwater visual image stitching and fusion scheme is proposed, and the experimental verification research is carried out, which has achieved good results. The results are significant and have good reference value as they demonstrate that the stitching and fusion method can effectively obtain the comprehensive information of overwater and underwater objects for unmanned ships. However, because all research is based on laboratory conditions, it is often easy to ignore some influencing factors of actual sea conditions. Therefore, this algorithm may need to consider more actual scene conditions and carry out more detailed and in-depth research.