Object Extraction Algorithm for the First-Frame Image of Unmanned Surface Vehicles Based on a Radar-Photoelectric System

Abstract

The radar-photoelectric system is a perception system to detect the surrounding environment based on marine radar and a photoelectric device. Mast obscuration, green water, and multi-object scenes are special scenes that appear in the first-frame image during the navigation of unmanned surface vehicles. The perception system cannot accurately obtain the object information in mast obscuration and green water scenes. The radar-guided object cannot be stably extracted from the first-frame image in multi-object scenes. Therefore, this paper proposes an object extraction algorithm for the first-frame image of unmanned surface vehicles based on a radar-photoelectric system. The algorithm realizes the field-of-view adaptation to solve the problem that the features of the radar-guided object are incomplete in the first-frame image and improve the detection accuracy of the local features by 16.8%. The algorithm realizes the scene recognition of the first-frame image to improve the robustness of object tracking. In addition, the algorithm achieves the stable extraction of the radar-guided object in multi-object scenes.

1. Introduction

An unmanned surface vehicle (USV) [1,2,3,4,5,6] is an intelligent vehicle with autonomous navigation, autonomous obstacle avoidance [7], and dynamic object tracking capabilities. It is suitable for performing dangerous and repetitive missions. USVs often encounter harsh operating conditions, such as heavy fog, huge waves, and heavy rain. Military and civilian applications of USVs are widespread. USVs mainly perform dynamic object tracking [8,9,10,11], mine detection [12], and offshore defense missions in the military. In civilian, USVs mainly perform hydrological monitoring, ocean resource exploration, maritime search and rescue, and seabed exploration [13]. As shown in Figure 1, “Tianxing-1” is the USV performing various sea trials in this paper.

Object detection and tracking are the basis for achieving the autonomous navigation of USVs. The diversity of the object’s information contained in the image determines the result of object detection. The research of USV field-of-view adaption has significance in improving the diversity of object information. The USV will encounter scenes where the object in the image is obscured during navigation. The recognition of the occluded scenes significantly improves the robustness of object tracking. When there are multiple objects in the image, matching the radar-guided object with the object in the image is of interest to improve the stability of object tracking.

1.1. USV Perception System

The USV perception system obtains information about surrounding objects for USVs and is a crucial technology for USVs’ development. The surrounding situational map can be constructed, enabling USVs to understand the current environment. In addition, the perception system is necessary for the safe navigation of USVs. As shown in Figure 2, the USV perception system consists of a perception computer, a marine radar [14,15,16], and a photoelectric device. The CPU model of the perception computer is i7-7700t, and the GPU model is 2080ti. The resolution of the photoelectric visible light sensor is $1920\times 1080$, and the field of view is 1.97 to 40 degrees. The resolution of the infrared sensor is $640\times 512$, and the field of view is 2 and 10 degrees. The range of the laser rangefinder is 20 m to 10 km with the error of less than 5 m, the tracking accuracy of the photoelectric servo is 1 mrad, and it can generally navigate in the sea condition of grade 4. The azimuth accuracy of the navigation radar is 0.2 degrees, and the effective detection range is 40 m to 18 km.

Marine radar can obtain object information on distance, direction, speed, and heading. The perception computer consists of a communication module, a radar module, a defogging module, a detection-tracking module, and a decision module. The communication module is used for information transfer between the sensors and the perception computer. The radar module analyzes the object information and removes false objects caused by waves. The defogging module is used to defog the foggy images to improve their quality. The detection-tracking module is used for object detection and object tracking. The decision module determines the process to complete the mission based on the current mission requirements. The photoelectric device consists of a camera sensor and a laser sensor. The image obtained by the camera is essential information for object detection [17,18,19,20]. The laser sensor is used for the precise positioning of the object. The positioning information from laser sensors is more accurate than that from marine radar. As shown in Figure 3, the information transfer diagram of the USV perception system is shown.

First, the marine radar detects the objects around the USV to obtain the object information. Then, the marine radar sends the object information to the perception computer. The perception computer selects the specified object according to the current demand in the mission and transmits this information to the photoelectric device through control instruction. When the photoelectric device obtains the optical image, the image and state information are sent back to the perception computer. Then, the detection-tracking module detects the object in the image and obtains the pixel coordinates and category of the object. When the photoelectric device has obtained the object information, it controls the motor to track the radar-guided object. Through this process, the real-time dynamic object tracking by the photoelectric device is realized. When the decision module determines that it is in a stable tracking state, the perception computer will send commands to the photoelectric device so that the laser sensor can obtain the precise position information of the object. In addition, the monitor center at the shore can send specific task instructions to the perception computer through the radio station, and the display interface of the perception computer can also be transmitted to the monitor center through the radio station to realize real-time monitoring of the USV.

1.2. Image Edge Detection Algorithm

Edge detection [21,22,23,24] is a fundamental topic in image processing and computer vision. It is one of the most critical aspects of image processing. In a digital image, the area where the image’s grayscale changes drastically is called the edge. Due to the vital position and broad application of edge detection technology in image processing, the research on edge detection has been highly valued by people for many years.

The edge is the boundary of the area where the grayscale changes sharply in the image. There is always an edge between two adjacent areas with different grayscale values, resulting from discontinuous grayscale values. L.G. Roberts proposed a first-order edge detection differential operator that uses the gradient of the image grayscale distribution to characterize the image grayscale change called the Roberts operator [25]. Since then, new theories and methods for edge detection have emerged, such as the Sobel operator [26,27], Kirsh operator, and Prewitt operator [28]. These first-order differential edge detection operators are widely used today because of their slight computational complexity and simple operation. Subsequent refinement processing is generally required when detecting the edge of an image, which will affect the accuracy of edge localization.

It can be known from mathematical theory that the local maximum point of the first-order differential of the function corresponds to the zero-crossing point of the second-order differential, which means that the peak value of the first-order differential can be obtained at the edge point of the image. The cross-section of the second-order differential can be obtained. Therefore, the Laplacian second-order differential edge detection operator was subsequently generated. Compared with the first-order differential edge detection operator, the Laplacian operator [29] is more sensitive to noise and enhances the influence of noise on the detection effect. Based on the above ideas, Hildreth and Marr proposed a second-order edge detection operator that is smoothed and then derived. This operator first smooths the image with a Gaussian function and then uses the isotropic Laplacian operator to derive the image, the zero-crossing point of the second derivative is determined as the edge point, and this operator is also called the Laplacian of Gaussian operator (LOG operator).

Canny proposed three primary criteria for evaluating the performance indicators of edge detection methods and expressed the above three criteria in precise mathematical form for the first time. According to the requirements, the optimal edge detection filter corresponding to a given edge model is obtained by using the optimal numerical analysis method, and the famous Canny operator [30,31] is proposed based on the above theory. Under the influence of Canny’s thought, many similar optimization filter operators were developed later, such as the Deriche operator, Bournnane operator, etc.

1.3. The Content of This Paper and Highlights

When the photoelectric device points to the radar-guided object, there is a case that the features of the radar-guided object are incomplete in the first-frame image. During the navigation of the USV, it may encounter the scene of mast obscuration or green water, which causes the problem of missing objects. When the mast or water blocks the object, the perception system cannot detect the object. In addition, when the photoelectric device points to the object, there may be multiple objects of the same type in the field of view. At this time, the object detection algorithm cannot accurately return the radar-guided object information, resulting in the problem of tracking the wrong object. Therefore, this paper proposes an object extraction algorithm for the first-frame image of unmanned surface vehicles based on a radar-photoelectric system. First, the object’s distance information is obtained by the marine radar. Then, the size of the photoelectric device’s field of view is calculated so that the object can appear in the field of view with a suitable size. According to the pixel size, the field of view is adjusted again. When the scene in the first frame image is identified as the mast obscuration or green water scene, the perception system will stop the subsequent calculations. When multiple objects of the same type appear in the first frame image, the algorithm can stably extract radar-guided objects. The main contributions and organization of this paper can be summarized as the following points:

1.

A field-of-view adaption algorithm is proposed

The field-of-view adaption algorithm adjusts the photoelectric device’s field of view according to the object’s distance and size. When the marine radar obtains the object’s distance information, the perception system calculates the field of view size so that the object can appear in the field of view with a suitable size. Then, according to the object’s pixel size, the secondary field-of-view adjustment is performed so that the object can appear in the field of view with the best size to obtain more detailed information about the object improving the detection accuracy of the local features by 16.8%.

2.

A scene recognition algorithm is proposed

Scene recognition is an algorithm that recognizes the particular scene in the first-frame image. First, the scene recognition algorithm extracts the edge information by the edge detection algorithm and then statistically analyzes the edge information to determine whether the scene is a mast obscuration or green water scene. The perception system will stop the subsequent calculations when the image is detected as these scenes. The algorithm realizes the scene recognition of the first-frame image to improve the robustness of object tracking.

3.

A radar-guided object extraction algorithm is proposed

When the photoelectric device points to the object, there is a case that multiple objects of the same type are in the field of view. At this time, the radar-guided object extraction algorithm analyzes each object in the image and then filters the non-guided objects in the field of view to stably return the radar-guided object information of the first-frame image. The algorithm solves the problem of the unstable extraction of the radar-guided object in multi-object scenes.

In the second section of the paper, we analyze the imaging principle of the object and propose a field-of-view adaptive algorithm. The algorithm is realized by means of two field-of-view adaptations. In the third section of the paper, we analyze the image characteristics of the mast obscuration and green-water scene. For this feature, we propose the scene recognition algorithm. The algorithm realizes the scene recognition of the first-frame image to improve the robustness of object tracking. In the fourth section of the paper, we analyze the case of multiple objects within the same field of view and propose a radar-guided object extraction algorithm. The algorithm solves the problem of the unstable extraction of the radar-guided object in multi-object scenes.

2. Field-of-View Adaption Algorithm

The field-of-view adaption algorithm is an algorithm that adaptively adjusts the field of view of the photoelectric device according to the object’s distance and size. After the photoelectric device obtains the information of the radar-guided object, the photoelectric device calculates the longitude and latitude information of the object to obtain the azimuth information. The photoelectric device rotates, and the radar-guided object appears in the field of view. After the marine radar obtains the distance information of the object, the perception system calculates the field of view so that the object can appear in the field of view with a suitable size. Then, according to the pixel size of the object, the secondary field-of-view adjustment is performed so that the object can appear in the field of view with the best size to obtain more detailed information about the object. The algorithm can ensure the stability of USVs in various missions. As shown in Figure 4, it is a schematic diagram of the relative position of the USV and the object. The marine radar detects the surrounding environment and can obtain the distance information $d$ of the target and the azimuth $\omega$ of the target.

As shown in Figure 5, it is a schematic diagram of the USV’s field of view. The field of view is $\beta$, the field of view occupied by the object is $\gamma$, the actual width of the image on the plane where the object is located is $l$, the pixel width of the image is $l_{image}$, the actual width of the object in the image is $b$, and the pixel width of the object is $b_{image}$. In addition, the field-of-view adaptation coefficient is $\lambda$. The adaptation coefficient $\lambda$ is the basis for adjusting the size of the field of view, which can be expressed as:

$$\lambda =\frac{b}{l}$$

(1)

The object distance information obtained by the marine radar is $d_{radar}$, the estimated object size information is $b_{radar}$, and the adaptation coefficient is ${\lambda }_{radar}$.

Based on the above information, the expected field of view ${\beta }_{radar}$ is calculated as follows:

$${\lambda }_{radar}=\frac{b_{radar}}{l}$$

(2)

$${\gamma }_{radar}=2\arctan (\frac{b_{radar}}{2d_{radar}})$$

(3)

$${\beta }_{radar}=2\arctan (\frac{l}{2d_{radar}})=2\arctan (\frac{b_{radar}}{2{\lambda }_{radar}{d}_{radar}})$$

(4)

By the above calculation, the expected field of view ${\beta }_{radar}$ can be obtained so that the object can appear in the field of view at a suitable size after the photoelectric device points to the object, and the pixel width $b_{image1}$ of the object can be obtained. At this time, the precise distance $d_{laser}$ of the object can be obtained by the laser sensor. The size of the actual width $b_{object}$ of the object can be obtained by calculation, and the calculation process is as follows:

$$\frac{l}{2}=d_{laser}\cdot \tan ({\beta }_{radar})$$

(5)

$$\frac{b_{object}}{l}=\frac{b_{image1}}{l_{image}}$$

(6)

$$b_{object}=\frac{2b_{image1}{d}_{laser}\tan ({\beta }_{radar})}{l_{image}}$$

(7)

Since there are significant errors in the object distance information and object size information obtained by the maritime radar, a secondary field-of-view adaption adjustment is required to make the object appear in the field of view with the best size to ensure that the perception system obtains the best visual information of the object. The first adaption field-of-view adjustment is based on the object’s information obtained from the marine radar, while the second adaption field-of-view adjustment is based on the object’s pixel size in the image.

In the secondary field-of-view adaption algorithm, the expected field of view of the photoelectric device is ${\beta }_{pixel}$, the field of view occupied by the object is ${\gamma }_{pixel}$, the actual width of the image on the plane where the object is located is $l$, and the pixel width of the image is $l_{image}$. The actual width of the object is $b_{object}$, the pixel width of the object is $b_{image2}$, and the object distance measured by the laser sensor is $d_{laser}$. In addition, the adaption coefficient in the secondary field-of-view adaption algorithm is ${\lambda }_{pixel}$, which can be expressed as:

$${\lambda }_{pixel}=\frac{b_{object}}{l}$$

(8)

Based on the above information, the expected field of view ${\beta }_{pixel}$ is calculated as follows:

$$\frac{b_{object}}{l}=\frac{b_{image2}}{l_{image}}$$

(9)

$$\frac{{\gamma }_{pixel}}{2}=\arctan (\frac{b_{object}}{2d_{laser}})$$

(10)

$$\frac{{\beta }_{pixel}}{2}=2\arctan (\frac{l}{2d_{laser}})$$

(11)

$${\beta }_{pixel}=2\arctan \left(\frac{b_{image1}\tan ({\beta }_{radar})}{{\lambda }_{pixel}{l}_{image}}\right)$$

(12)

The expected field of view can be obtained through the above calculation so that the object can appear in the field of view with the best size. As shown in Figure 6, it is a schematic diagram of the secondary field-of-view adaption algorithm of the USV.

Ten different maritime test environments are selected to test the field-of-view adaptive algorithm. Comparing the first frame without field-of-view adaptive processing with the image with field-of-view adaptive processing shows that the object’s local feature detection accuracy is improved by 16.8%.

3. Scene Recognition Algorithm

The scene recognition is an algorithm for the perception system to recognize the special scenes in the first frame image. After the photoelectric device has pointed to the object, the scene can be classified as follows, according to the characteristics of the image. The first is the single-object scene, where only the radar-guided object is in the image. The second is the different kind of multi-object scene, in which there are multiple objects but in different categories. The third is the multi-object scene of the same category. In this scene, there are multiple objects in the same category. The fourth is the green-water scene, where the water blocks the object. The fifth is the mast-obscuration scene, where the mast occludes the object. The perception system cannot identify and detect the object if the scene is the green-water scene or the mast-obscuration scene.

3.1. Recognition of the Mast-Obscuration Scene

The radar-guided object, the mast of the USV, and the photoelectric device are in a straight line when the scene of the first frame image is the mast-obscuration scene. As shown in Figure 7, it is the images of the mast-obscuration scene. The perception system cannot identify and detect the object when the scene is the mast-obscuration scene.

There is an area where the sky area, the sea–sky line, and the sea area are all occluded, analyzing the mast-obscuration scene. In addition, the image gradient of the particular area is slight. The edge of the image can be detected according to the image gradient characteristics of the region. We determine whether it is a mast-obscuration scene, according to the edge detection result.

The Prewitt operator is an edge detection algorithm based on the first-order differential operator. The algorithm uses the grayscale difference between the upper and lower pixels and the grayscale difference between the left and right adjacent points to extract the edge information in the image. The principle is to use two directional templates to perform neighborhood convolution with the image; one template is used to detect horizontal edges, and another is used to detect vertical edges. Since the purpose of edge detection is not to obtain accurate object’s edge information, the Prewitt operator edge detection algorithm can ensure the edge extraction effect and meet the perception system’s real-time requirements.

Image $A$ is the grayscale image of the first frame image, and $f(x,y)$ is the grayscale value at $(x,y)$ in image $A$. $G_x$ and $G_y$ are the approximate values of the horizontal and vertical grayscale partial derivatives, respectively. The mathematical expressions are as follows:

$$\begin{array}{l}{G}_x=[f(x+1,y-1)+f(x+1,y)+f(x+1,y+1)]\\ -[f(x-1,y-1)+f(x-1,y)+f(x-1,y+1)]\end{array}$$

(13)

$$\begin{array}{l}{G}_y=[f(x-1,y-1)+f(x,y-1)+f(x+1,y-1)]\\ -[f(x-1,y+1)+f(x,y+1)+f(x+1,y+1)]\end{array}$$

(14)

After the above calculation, the approximate value $G_x$ of the horizontal gray partial derivative and the approximate value $G_y$ of each pixel’s vertical gray partial derivative can be obtained. Gradients in two directions can be obtained for each pixel. The following formula calculates an approximation of the gradient:

$$G=\sqrt{G_x{}^2+G_y{}^2}$$

(15)

When the gradient $G$ is greater than the gradient threshold $G_{\max }$, the pixel is determined to be a bright spot, and the original gray value of the pixel is retained. When the gradient $G$ is less than or equal to the gradient threshold $G_{\max }$, the pixel is determined to be a dark point, and the gray value at this pixel is zero. Through the above determination, an edge-detected image $B$ is obtained. As shown in Figure 8, it is the edge detection result of the mast-obscuration scene.

The analysis of the edge detection result shows that the sea–sky line appears to be interrupted due to mast occlusion. In addition, the sea region under the mast occlusion has no sea noise information. According to the above image features, this paper segments the image horizontally and counts the number of bright spots within each region. As shown in Figure 9, a schematic diagram of the image segmentation is shown.

The image above shows that the number of bright spots in the segmented area occluded by the mast is significantly smaller than that in the segmented area not obscured by the mast. $B$ is the edge detection image, the resolution of image $B$ is $p_x\times p_y$, and $k$ is the serial number of the segmented area in image $B$. $n_k$ is the number of bright spots in the $k$ segmented area, $x$ is the horizontal coordinate of the pixel in the image $B$, $y$ is the vertical coordinate of the pixel in the image $B$, and $a_k$ is the left critical value of the $k$ segmented area in the $x$ direction. The $bright(x,y)$ function is the bright spot determination function, $b$ is the gray level of the pixel at $(x,y)$ in image $B$, and $b_0$ is the determination threshold in the bright spot determination function. The calculation process is as follows:

$$bright(x,y)=\{\begin{array}{l}0,b<b_0\\ 1,b\ge b_0\end{array}$$

(16)

$$n_k={\displaystyle \sum _{i=a_k}^{a_{k+1}}{\displaystyle \sum _{j=0}^{p_y}bright(x_i,y_j)}}$$

(17)

By the above calculation, the number of bright spots in each segmentation region in the image can be obtained. As shown in Figure 10, the statistical curves of the bright spots in the segmented regions are shown.

The above curves show that there is no segmented area with less than 100 bright spots in the scene not blocked by the mast and multiple divided areas with less than 100 bright spots in the statistical result of the scene blocked by the mast. $mast(k)$ is the decision function of the mast-obscuration scene, $n_k$ is the number of bright spots in the $k$ segmented area in the statistics, and $n_m$ is the judgment threshold in the decision function of the mast-obscuration scene. $n_{mast}$ is the number of the mast-obscuration areas, and $n_{total}$ is the number of the segmented areas. The calculation process is as follows:

$$mast(k)=\{\begin{array}{l}0,n_k>n_m\\ 1,n_k\le n_m\end{array}$$

(18)

$$n_{mast}={\displaystyle \sum _{k=1}^{n_{total}}mast(k)}$$

(19)

$r_m$ is the ratio of the number of mast occlusion areas to the total number of divided areas, $c_{mast}$ is the judgment result of the mast-obscuration scene, and $r_{mast}$ is the judgment threshold of the mast-obscuration scene. If the ratio $r_m$ is less than the determination threshold $r_{mast}$, the determination result $c_{mast}$ is equal to 0, and the scene is determined to be a non-mast-obscuration scene; if the ratio $r_m$ is greater than or equal to the determination threshold $r_{mast}$, the determination result $c_{mast}$ is equal to 1, and the scene is determined to be a mast-obscuration scene. The expression is as follows:

$$r_m=\frac{n_{mast}}{n_{total}}$$

(20)

$$c_{mast}=\{\begin{array}{l}0,r_m<r_{mast}\\ 1,r_m\ge r_{mast}\end{array}$$

(21)

3.2. Recognition of the Green-Water Scene

The radar-guided object is obscured by water when the scene of the first frame image is the green-water scene. As shown in Figure 11, it is the image of the green-water scene. The perception system cannot identify and detect the object when the scene is the green-water scene.

The camera of the photoelectric device is covered by water, analyzing the green-water scene. For the non-green-water wave scene, the edge information of the sky part above the sea–sky line in the image is little, while for the green-water scene, the edge information of that increases significantly. So, it is determined whether it is a green-water scene by the edge detection result. Laplacian edge detection is an edge detection algorithm that uses a second derivative to extract image edge information. The Laplacian operator is an isotropic operator. Isotropic operator means that the use of one operator can sharpen borders and lines in any direction without directionality. Like the Sobel and Prewitt operators, they use different operators to extract edges in different directions. Isotropy is the advantage of the Laplacian operator that distinguishes it from other first-order differential operators. The Laplacian operator for the grayscale image $f(x,y)$ in the first frame can be defined as:

$${\nabla }^{2}f(x,y)=\frac{{\partial }^{2}f}{\partial x^2}+\frac{{\partial }^{2}f}{\partial y^2}$$

(22)

For a two-dimensional discrete image, the following calculation can be used as an approximation to the second-order partial differential:

$$\frac{{\partial }^{2}f}{\partial x^2}=f(x+1,y)+f(x-1,y)-2f(x,y)$$

(23)

$$\frac{{\partial }^{2}f}{\partial y^2}=f(x,y+1)+f(x,y-1)-2f(x,y)$$

(24)

Therefore, the expression of the Laplacian operator is:

$${\nabla }^{2}f(x,y)=f(x,y+1)+f(x,y-1)+f(x+1,y)+f(x-1,y)-4f(x,y)$$

(25)

According to the above expression, the following filter template can be obtained:

$$G=\left[\begin{array}{ccc}0& 1& 0\\ 1& -4& 1\\ 0& 1& 0\end{array}\right]$$

(26)

When convolving the image with the Laplacian operator, if the absolute value of the response is greater than the threshold $T$, the pixel is determined to be a bright spot, and the original gray value of the pixel is retained. If the absolute value of the response is less than or equal to the threshold value $T$, it is determined that the pixel is a dark point, and the gray value of the pixel is zero. Through the above determination, an edge-detected image is obtained. As shown in Figure 12, it is the edge detection result of the green-water scene.

For the non-green-water wave scene, the edge information of the sky part above the sea–sky line in the image is little, while for the green-water scene, the edge information of that increases significantly. Therefore, this paper divides the image vertically and counts the number of bright spots in each area according to the above image characteristics. As shown in Figure 13, it is a schematic diagram of the vertical region segmentation of the image.

$D$ is the image obtained after edge detection, and the resolution of image $D$ is $p_x\times p_y$. $h$ is the serial number of the segmented area in image $D$, and $n_h$ is the number of bright spots in the $h$ segmented area. $x$ is the horizontal coordinate of the pixel in image $D$, $y$ is the vertical coordinate of the pixel in image $D$, $a_h$ is the left critical value of the $h$ segmented area in the $x$ direction, and the $bright(x,y)$ function is the bright spot determination function. $b$ is the pixel’s gray level at $(x,y)$ in image $D$, and $b_1$ is the decision threshold of the bright spot determination function. The calculation process is as follows:

$$bright(x,y)=\{\begin{array}{l}0,b<b_1\\ 1,b\ge b_1\end{array}$$

(27)

$$n_h={\displaystyle \sum _{i=a_h}^{a_{h+1}}{\displaystyle \sum _{j=0}^{p_y}bright(x_i,y_j)}}$$

(28)

By the above calculation, we can obtain the number of bright spots in each segmented area in the image. As shown in Figure 14, it is the statistical curves of bright spots in the segmented regions.

The above curves show that there is no segmented area with less than 100 bright spots in the green-water scene and multiple divided areas with less than 100 bright spots in the non-green-water scene. $water(h)$ is the decision function of the green-water scene, $n_h$ is the number of bright spots in the $k$ segmented area in the statistics, $n_w$ is the judgment threshold in the decision function of the mast-obscuration scene, $n_{water}$ is the number of the mast-obscuration areas, and $n_{total}$ is the number of the segmented areas. The calculation process is as follows:

$$water(h)=\{\begin{array}{l}0,n_h<n_w\\ 1,n_h\ge n_w\end{array}$$

(29)

$$n_{water}={\displaystyle \sum _{h=1}^{n_{total}}water(h)}$$

(30)

$r_w$ is the ratio of the number of mast occlusion areas to the total number of divided areas, $c_{water}$ is the judgment result of the mast-obscuration scene, and $r_{water}$ is the judgment threshold of the mast-obscuration scene. If the ratio $r_w$ is less than the determination threshold $r_{water}$, the determination result $c_{water}$ is equal to 0, and the scene is determined to be a non-green-water scene. If the ratio $r_w$ is greater than or equal to the determination threshold $r_{water}$, the determination result $c_{water}$ is equal to 1, and the scene is determined to be a green-water scene. The expression is as follows:

$$r_w=\frac{n_{water}}{n_{total}}$$

(31)

$$c_{water}=\{\begin{array}{l}0,r_w<r_{water}\\ 1,r_w\ge r_{water}\end{array}$$

(32)

4. Radar-Guided Object Extraction Algorithm

Real-time object tracking by the perception system is an essential part of dynamic object tracking of the USV. The above process includes three parts: First, the marine radar detects the object and guides the photoelectric device to complete the object pointing. Second, the perception computer performs object detection on the sequence images returned by the photoelectric device. Third, the photoelectric device realizes object tracking according to the object’s position and performs laser positioning on the object.

Depending on the type and number of objects in the field of view, scenes can be classified into single-object scenes, different kinds of multi-object scenes, and the same kind of multi-object scenes. If there is only a single object in the field of view, the perception computer can obtain the pixel coordinates $(x,y)$ of the object through the object detection algorithm and obtain the off-target distance of the object. The resolution of the first frame image is $p_x\times p_y$, and the bullseye coordinate of the image is $(x_0,y_0)$; then, the off-target amount $(x_b,y_b)$ of the object can be expressed as follows:

$$x_0=\frac{p_x}{2}$$

(33)

$$y_0=\frac{p_y}{2}$$

(34)

$$x_b=x-x_0=x-\frac{p_x}{2}$$

(35)

$$x_y=y-y_0=y-\frac{p_y}{2}$$

(36)

After the photoelectric device has pointed to the object, if there are multiple objects of the same type in the field of view, the object detection algorithm cannot accurately obtain the position of the radar-guided object. The paper proposes a radar-guided object extraction algorithm, which can filter non-guided objects. The proposed algorithm improves the robustness of object tracking and enables the perception system to achieve a stable return of the radar-guided object distance in the first frame image even in a complex environment. Although the object distance error obtained by the marine radar is relatively large, the object azimuth information is relatively accurate. Therefore, the radar-guided objects are located in the middle region of the image. Based on the above characteristics, the objects outside the region of interest (ROI) can be filtered, and the interference of other objects with the returned results can be reduced.

$\alpha$ is the maximum deviation angle of the object detected by the marine radar. $\beta$ is the field-of-view angle of the photoelectric device, and the resolution of the first frame image is $p_x\times p_y$. $A_0$ is the maximum deviation of the actual pixel position of the object in the horizontal direction. $\epsilon$ is the fault tolerance parameter of the ROI. $A_{\min }$ is the minimum critical value of the ROI. $A_{\max }$ is the maximum critical value of the ROI. The expression is as follows:

$$A_0=\frac{\alpha }{\beta }·P_x$$

(37)

$$A_{\min }=\frac{P_x}{2}-(A_0+\epsilon )$$

(38)

$$A_{\max }=\frac{P_x}{2}+(A_0+\epsilon )$$

(39)

In this case, several objects of the same category are in the ROI. According to the position of each object in the horizontal direction, the following calculation is performed.

$i$ is the serial number of each object in the ROI of the first frame image. $x_i$ is the horizontal pixel coordinate of object $i$, and $d_i$ is the distance between the object and the center of the image in the horizontal direction. $n$ is the number of objects in the ROI of the image, and $p_i$ is a parameter for sorting. $c_i$ is the new serial number of the object $i$. The calculation process is as follows:

$$d_i=\left|x_i-\frac{P_x}{2}\right|$$

(40)

$$p_i={\displaystyle \sum _{j=1}^{n}sign(d_i-d_j)}$$

(41)

$$c_i=\{\begin{array}{l}\frac{n+1}{2}+sign(p_i)·\left|p_i-p_{\frac{n+1}{2}}\right|, n \mathrm{is} \mathrm{odd}\\ \frac{n}{2}+sign(p_i)·\frac{\left|p_i-p_{\frac{n}{2}}\right|}{2}, n \mathrm{is} \mathrm{even}\end{array}$$

(42)

The above calculation shows each object’s new serial number $c_i$ in the ROI of the first frame image. However, when two objects of the same type are located on both sides of the image centerline, and the absolute values of the two objects are close, the radar-guided object cannot be stably extracted. Therefore, this paper performs correlation calculations on the objects’ position in adjacent frames.

$x_b^k$ is the off-target distance in the horizontal direction and $y_b^k$ is the off-target distance in the vertical direction. $S_k$ is the associated flag of two adjacent frames. The calculation process is as follows:

$$x_b^k=x-\frac{p_x}{2}$$

(43)

$$y_b^k=y-\frac{p_y}{2}$$

(44)

$$S_k=sign(x_b^k\cdot x_b^{k-1})$$

(45)

If $S_k$ is 0 or 1, it means that the object correlates with the object of the previous frame; if $S_k$ is −1, it means that the object has no correlation with the object of the previous frame, and the object selection result of the previous frame is retained. Estimate the object’s pixel size based on the distance and category. According to the estimated pixel size, the objects in the ROI are screened again to obtain the final radar-guided object.

$sort(c_i)$ is the function of object screening. $j$ is the object serial number retained after screening, $w_j$ is the object pixel width, and $h_j$ is the object pixel height. $s_j$ is the object pixel area, and $m$ is the number of objects reserved for screening. The calculation process is as follows:

$$j=sort(c_i)$$

(46)

$$s_j=w_j·h_j$$

(47)

$${\Vert s\Vert }_{\infty }=\underset{1\le j\le m}{\max }\left|s_j\right|$$

(48)

This paper tests the radar-guided object extraction algorithm in the case of multiple buoys in a simple background, the case of multiple buoys in a complex background, and the case of multiple ships in a simple and the case of multiple ships in the complex background.

As shown in Figure 15a, it is the test of the radar-guided object extraction algorithm in the case of multiple buoys in simple background. In the same field of view, there are two objects of the same type. After the marine radar detection, according to the mission requirements, the decision module autonomously selects the object closest to the USV. In this image, the green buoy is the object guided by the radar, and the red buoy is successfully filtered out. As shown in Figure 15b, it is the test of the radar-guided object extraction algorithm in the case of multiple buoys in a complex background. The background behind the buoy is complicated. During the test, the red buoys was successfully filtered out.

As shown in Figure 16a, it is the test of the radar-guided object extraction algorithm in the case of multiple ships in the simple background. In the same field of view, there are two objects of the same type. After the marine radar detection, according to the mission requirements, the decision module autonomously selects the object closest to the USV. In this image, the ship in the middle is the object guided by the radar, and another ship is successfully filtered out. As shown in Figure 16b, it is the test of the radar-guided object extraction algorithm in the case of multiple ships in a complex background. The background behind the ships is complicated. During the test, another ship is successfully filtered out.

As shown in Figure 17, it is the flow chart of the USV object tracking. The part marked with an asterisk is the main research content of this paper.

5. Conclusions

When the photoelectric device points to the radar-guided object, there is a case that the object’s pixel size in the first-frame image is not suitable, resulting in missing object information. During the navigation of the USV, it may encounter the scene of mast obscuration or green water. When the mast or water blocks the object, the perception system cannot detect the object. In addition, when the photoelectric device points to the object, there may be multiple objects of the same type in the field of view. At this time, the object detection algorithm cannot accurately return the radar-guided object information, resulting in the problem of tracking the wrong object. Therefore, this paper proposes an object extraction algorithm for the first-frame image of unmanned surface vehicles based on a radar-photoelectric system.

The first is the field-of-view adaption algorithm. The algorithm adjusts the photoelectric device’s field of view according to the object’s distance and size. When the marine radar obtains the object’s distance information, the perception system calculates the field of view size so that the object can appear in the field of view with a suitable size. Then, according to the object’s pixel size, the secondary field-of-view adjustment is performed so that the object can appear in the field of view with the best size to obtain more detailed information about the object. The second is the scene recognition algorithm. The algorithm recognizes the particular scene in the first-frame image. First, the scene recognition algorithm extracts the edge information by the edge detection algorithm and then statistically analyzes the edge information to determine whether the scene is a mast obscuration or green water scene. The perception system will stop the subsequent calculations when the image is detected as these scenes. The third is the radar-guided object extraction algorithm. When the photoelectric device points to the object, there is a case that multiple objects of the same type are in the field of view. At this time, the radar-guided object extraction algorithm analyzes each object in the image and then filters the non-guided objects in the field of view to stably return the radar-guided object information of the first-frame image.