1. Introduction
Marine operation equipment is gradually developing in the direction of unmanned operation, and unmanned surface vehicles (USVs) have become one of the research hot spots in marine operation equipment [
1]. The obstacle avoidance performance of the unmanned ship is one of the most direct and comprehensive performances of its autonomous navigation. Establishing a test system for the obstacle avoidance performance of the unmanned ship based on experimental data, and formulating corresponding technical indicators to quantitatively evaluate the navigation of the unmanned ship, the performance of control, path planning, path tracking, and autonomous obstacle avoidance are the most intuitive basis for evaluating the autonomous navigation performance of unmanned ships [
2].
Hyo-Gon Kim et al. proposed a collision avoidance algorithm for unmanned ships based on the International Regulations for Preventing Collisions at Sea (COLREGS) and verified the degree of compliance between the algorithm and COLREGS [
3]. Yingjun Hu et al. established a multi-ship real-time collision risk analysis system based on the overall requirements of COLREGS and based on five factors affecting collision avoidance risk [
4]. Yaseen Adnan Ahmed et al. proposed a new intelligent collision detection and resolution algorithm based on fuzzy logic. The fuzzy membership functions of input and output are deduced, and a decision-making method of minimum speed and maximum heading change is proposed, which can effectively solve the problem of complex multi-ship encounters [
5]. The above studies have focused on solving the obstacle avoidance problem but failed to conduct research from the perspective of performance testing.
Phanthong et al. proposed the path re-planning technology and underwater obstacle avoidance technology performance of the unmanned surface vehicle [
6]. Simulation tests and sea surface tests were designed, respectively. In the simulation, it was tested whether the unmanned surface vehicle could avoid a single obstacle or multiple stationary obstacles. For obstacles and moving obstacles in the sea surface test, the actual unmanned surface vehicle is automatically controlled to test its real-time obstacle avoidance performance against static obstacles in the field of view of the multi-beam forward-looking sonar, but quantitative evaluation of obstacle avoidance performance has not been made. In order to test the effectiveness of the extended Kalman filter and simple PID control law on the automatic heading, automatic speed, and straight travel control tasks of the unmanned surface vhicle, Massimo Caccia et al. used the prototype catamaran Charlie to conduct sea trials [
7]. Yuanchang Liu et al. adopted a simulation method to test the unmanned surface vehicle formation path planning algorithm proposed by him. The formation path planning performance of the unmanned surface craft was tested in the environment of a single moving obstacle and multiple moving obstacles [
8]. In order to test the obstacle avoidance performance of an unmanned surface vehicle based on radar under port conditions, Jose Villa and others conducted simulation and physical tests, respectively [
9]. In the simulation, a series of GPS waypoints constituted a predefined path. There is a fixed obstacle in the middle of the path to test whether the unmanned surface vehicle can navigate without obstacles. The physical test was carried out on Lake Pyhjrvi in Tampere, Finland. It mainly tested the path tracking and static obstacle avoidance performance of the real unmanned surface vehicle.
In order to verify the effectiveness of the target tracking and motion prediction algorithm of the unmanned surface vehicle in the chaotic environment, Petr Švec et al. developed a set of experimental devices to follow the target unmanned craft [
10]. Based on the altitude and AIS tracking data from the Trondheim Fjord in Norway, Eivind Meyere et al. tested whether the unmanned ship complies with the COLREGS and the degree of the path tracking of the unmanned ship to evaluate the performance of the trained unmanned craft in challenging, dynamic, real scenes [
11]. However, they did not propose specific evaluation technical indicators to quantitatively evaluate its performance.
China has also carried out relevant research in the field of unmanned ship performance testing. In November 2018, Asia’s first unmanned ship offshore test site was officially opened in Wanshan, Zhuhai [
12]. The test site covers an area of more than 770 km
2. It is not only be used for unmanned ship testing requirements, but also meets the testing needs of other types of offshore operating platforms. In 2019, the China Classification Society (CCS) awarded Shanghai Jiao Tong University’s Key Laboratory of Marine Intelligent Equipment and Systems of the Ministry of Education a service provider accreditation certificate for the unmanned ship test site [
13]. In recent years, China has launched many unmanned ship races. On 8 August 2017, the Chinese Institute of Electronics hosted the “First (China) Unmanned Ship Open” in Dishui Lake, Shanghai. On 9 July of the following year, the Chinese Institute of Electronics and Shanghai Gangcheng Development Co., Ltd. co-hosted the “Second (China) Unmanned Ship Open”. On 30 October 2020, the first “National Intelligent Unmanned Boat Search and Rescue Competition” opened in Dalian. The competition set up four events for contestants to compete [
14]. The above competitions are usually judged based on the timeliness and accuracy of unmanned ships to complete tasks, but there is no systematic test equipment and quantitative evaluation model.
Haifang Mu conducted demand demonstrations for the test sea area of the test site, the shore-based integrated command and control center, the communication system, the shore-based health management and comprehensive support system, and the evaluation system [
15]. The demonstration initially clarified the key links, related functions and performance requirements of the test site. In terms of testing methods, Hongdong Wang and others designed technical indicators and corresponding test tasks based on user goals and constructed a task-based comprehensive evaluation system [
16]. In order to explore the influence of wind and wave coupling, a berthing computational fluid dynamics (CFD) model was established, and the characteristics of speed field, pressure field, and vortex were obtained under different speeds, wind directions, and quay wall distances [
17]. The above results can provide control strategy for unmanned ship berthing safety, and provide a theoretical basis for unmanned ship route planning and obstacle avoidance, safety design, etc.
In order to verify the feasibility of the proposed dynamic obstacle avoidance algorithm for unmanned ships based on elliptical collision cones, Huayan Pu performed simulation tests and actual ship tests, respectively. The simulation test is based on MATLAB. The initial position of the designed dynamic obstacle is outside the safe distance of the unmanned ship. As the distance between the dynamic obstacle and the unmanned ship continues to approach, it is tested whether the unmanned ship can avoid obstacles in accordance with the international maritime collision avoidance rules. In the actual ship test, the “Jinghai” unmanned ship was used as the unmanned ship under test. First, a route was planned on the host computer, and then the “Jinghai” was allowed to sail along the route. The working ship, acting as a dynamic obstacle, sailed to the “Jinghai” as planned, and tested whether the “Jinghai” could avoid obstacles in accordance with the international maritime collision avoidance rules [
18].
In summary, the current performance testing of unmanned ships lacks mature measurement and control systems and unified testing standards. The test instrument is directly placed on the unmanned ship under test, and the performance evaluation is mostly based on qualitative evaluations, such as the time taken, the number of obstacles avoided, and the number of tasks completed. It does not involve real-time navigation data, calculation of technical indicators, etc., nor does it consider the quality of the unmanned ship’s mission. Although it can judge whether the autonomous navigation function of the unmanned ship is good or poor, it does not specify which indicators are insufficient and the direction to be improved, which has little effect on the performance improvement of the unmanned ship.
Aiming at the evaluation needs of the unmanned ship’s obstacle avoidance performance, this paper constructs the unmanned ship’s obstacle avoidance performance evaluation model by collecting the navigation data of the unmanned ship. The research on the evaluation index of the unmanned ship’s obstacle avoidance performance is carried out, and the feasibility and effectiveness of the test system is verified.
3. Obstacle Avoidance Performance Test Results and Evaluation of Unmanned Ships
Based on the mobile buoy measurement and control system, the unmanned ship navigation monitoring experiment and obstacle avoidance performance test experiment are carried out to verify the effectiveness of the test system. The experiments were all carried out in the coastal waters of Longxue Island, Guangzhou. The test site was 900 m long, 800 m wide, and 8 m deep. A mobile buoy was arranged in the test site. Through the unmanned ship navigation monitoring experiment, the effectiveness of the buoy measurement and control system and the unmanned ship navigation monitoring method was verified. We used the test system to test the obstacle avoidance performance of the unmanned ship in the test sea area and quantitatively evaluated the obstacle avoidance performance of the unmanned ship from the three aspects of obstacle perception, static obstacle avoidance and dynamic obstacle avoidance.
The obstacle perception test experiments include unmanned ships to identify buoy obstacles in the range of 0–100 m, and unmanned ships to identify small islands 100 m away. The former places the buoy as an obstacle in the test sea area, and measures the size of the unmanned ship on the sea surface in the test sea area in advance, which is about 1.51 m × 1.51 m × 2.10 m. The diameter of the buoy base is 1.51 m, and the specified error is within 0.2 diameter length as a reasonable error range [
23], that is, the allowable error is 0.3 m. The experiment was carried out 3 times, and in the three sets of experiments, the tested unmanned craft could accurately identify the buoy when the distance was less than 90 m, 70 m, and 70 m. In the second and third experiments, at distances greater than 70 m, the recognition distance began to be greater than 0.3 m, but the actual difference was different: the second and third differences were 0.51 m and 2.91 m, respectively.
The basic measurement data for this test were 90 m, 70 m, and 70 m, respectively. Using the cost function method, we can calculate the distance cost value of the measured unmanned ships’ identification buoy according to Formula (9), as shown in Formula (18):
In the island identification test, the unmanned ship first sails to a designated area about 100 m away from the island, and the unmanned ship sends the measured distance and azimuth of the island to the shore base for comparison with the actual distance and azimuth. The specified distance error is 0.2 island width as a reasonable range [
24], that is, the distance error is not more than 1.5 m. The experiment measured the distances for the unmanned ship to identify the islands as being 110 m, 105 m, and 100 m respectively, that is, the basic measurement data for this test were 110 m, 105 m, and 100 m, respectively. Using the cost function method, according to Formula (10), we can calculate the cost value of the unmanned ship to identify the longest distance of the island as shown in Formula (19):
The static obstacle test is divided into the single obstacle avoidance test, continuous obstacle avoidance test and inflection point obstacle test [
25]. During the test, the buoy appeared on the path of the unmanned ship, and the unmanned ship did not know the latitude and longitude of the position of the buoy in advance. The experiment was carried out 3 times to calculate the cost function values of single obstacle static obstacle avoidance, continuous obstacle static obstacle avoidance, and inflection point obstacle static obstacle avoidance as shown in Formulas (20)–(22).
The dynamic obstacle test mainly includes three experimental scenarios: chasing, confrontation, and cross encounter. From these three aspects, the obstacle avoidance performance of the unmanned ship was tested, according to the test plan. After the unmanned ship starts to sail, each buoy records the unmanned ship’s navigation data and sends it to the shore. After shore-based fusion, the navigation trajectory of the unmanned ship and the navigation trajectory of the dynamic obstacle buoy are displayed on the satellite map, and the response distance, minimum distance, and obstacle avoidance time are calculated. The experimental results are shown in
Table 7,
Table 8 and
Table 9.
According to the evaluation model in Formula (13), calculate the corresponding result cost function value of the dynamic obstacle avoidance test experiment of chase, confrontation, and cross encounter
, as shown in Formulas (23)–(25):
The unmanned ship obstacle avoidance performance test experiment is completed. Through the cost function method combined with the basic measurement data calculated in the experimental process, each evaluation factor of the evaluation factor layer is evaluated. Furthermore, the fuzzy comprehensive evaluation method can be used to quantitatively evaluate the obstacle avoidance performance of the unmanned ship in the three evaluation aspects of obstacle perception, static obstacle avoidance and dynamic obstacle avoidance.
(1) Quantitative evaluation of obstacle perception performance.
According to Equations (9) and (10), the two evaluation factors of obstacle perception in the next layer are the obstacle perception of 50–100 m buoy and the perception of small islands beyond 100 m. The performance in this experiment is {“good”, “fair”, “fair”}, {“good”, “good”, “fair”}, and the quantitative evaluation of obstacle perception performance in this experiment can be obtained from Equation (15) for the following:
For the obstacle perception performance of the tested unmanned ship, the membership degree of “good” is 0.4, and the membership degree of “fair” is 0.6. According to Equation (8), the obstacle perception performance score can be obtained as shown in Equation (27):
(2) Quantitative evaluation of static obstacle avoidance performance.
According to Formula (12), the three evaluation factors of the next layer of static obstacle avoidance are single obstacle static obstacle avoidance, continuous obstacle static obstacle avoidance and inflection point obstacle static obstacle avoidance. The performance in this experiment is {“fair”, “fair”, “good”}, {“fair”, “good”, “fair”}, {“good”, “good”, “fair”}. From Equation (16), the quantitative evaluation of static obstacle avoidance performance in this experiment is shown in Equation (28):
In the evaluation of the static obstacle avoidance performance of the tested unmanned ship, the membership degree that is good is 0.44, and the membership degree that is “fair” is 0.56. According to Equation (8), the static obstacle avoidance performance score can be obtained as shown in Equation (29):
(3) Quantitative evaluation of dynamic obstacle avoidance performance.
According to Formula (14), the three evaluation factors of the next layer of dynamic obstacle avoidance are the performance of chasing dynamic obstacle avoidance, facing dynamic obstacle avoidance and cross encounter dynamic obstacle avoidance performance. The performances in this experiment are {“poor”, “fair”, “fair”}, {“fair”, “fair”, “fair”}, {“excellent”, “fair”, “fair”}. From Equation (17), the quantitative evaluation of dynamic obstacle avoidance performance in this experiment is shown in Equation (30):
In the dynamic obstacle avoidance performance evaluation of the tested unmanned ship, the membership degree of “excellent” is 0.158, the membership degree of “fair” is 0.769, and the membership degree of “poor” is 0.073. According to Formula (8), the dynamic obstacle avoidance performance score can be obtained as shown in Formula (31):
In summary, in this test, the unmanned ship’s obstacle perception performance score is 68, the static obstacle avoidance performance score is 68.8, and the dynamic obstacle avoidance performance score is 64.13.
(4) Analysis of test results.
From the test results, the obstacle perception, static obstacle avoidance and dynamic obstacle avoidance performance scores of the tested unmanned ships are 68, 68.8, and 64.13, respectively. According to the evaluation set in Chapter 5, the score can be specified with (90,100) as “excellent”, (80,90) as “good”, (60,80) as “fair”, and 60 points or less as “poor”. Therefore, the three evaluation aspects of the unmanned ship’s obstacle avoidance performance are rated as “fair”. In the obstacle perception test, the longest distances at which the unmanned ship can identify buoys are 90 m, 70 m, and 70 m respectively. Reach the ideal level of 0.9, 0.7, and 0.7. The corresponding evaluations of the scores are “good”, “fair”, and “fair”, which is more reasonable. In the small island identification test, the unmanned ship can accurately identify small islands at 110 m, 105 m, and 100 m, indicating that the ability to perceive large objects is better. Since the proportion of the sensing buoy is 4 times the proportion of the sensing island, it is finally corresponding to “fair”, according to the test score, and the performance of sensing small objects at sea needs to be further improved.
In the static obstacle avoidance test, the return distance of the unmanned ships in the previous two single-obstacle static obstacle avoidance tests was too large and the obstacle avoidance time was too long, which resulted in the evaluation as “fair”. This was improved at the third time, and the score improved. In the continuous static obstacle avoidance, the score was also affected by the regression distance problem for the first time, and the score improved after the last two improvements. The inflection point obstacle test is relatively simple. The unmanned ship performed well in the first two tests, but the return distance increased in the third time, and the response distance with a larger proportion was also reduced, which ultimately lowered the score by one level. According to the proportions of the three groups of test items [0.165, 0.515, 0.320], the final score calculated corresponds to “fair”. The response distance of the unmanned ship in static obstacle avoidance can be appropriately increased, and the return distance should be optimized and reduced.
In the dynamic obstacle avoidance test, the minimum distance of the unmanned ship from the buoy in the first overtake scene test was 3.18 m, which was less than the width of the unmanned ship, which resulted in the score of this test being “poor”. This was adjusted for the last two times, and the corresponding performance evaluation was changed to “fair”. In the face-to-face test, the basic data in the three tests of the unmanned ship were relatively stable, and all the basic data were in the range of 0.3 to 0.5 in the ideal situation; the corresponding evaluations were all at the “fair” level. In the cross-encounter scenario test, the unmanned ship adopted three different strategies to avoid obstacles. For the first time, the method of sailing around behind the buoy was adopted. The reaction distance was the farthest, and the obstacle avoidance time was earlier. This not only avoided collisions, but also saved obstacle avoidance time under safe conditions. The corresponding evaluation was “excellent”. In the second test, the unmanned ship used a method of slowing down until it stopped sailing to avoid the buoy. This method was conservative and ensured safety, but the response distance was the smallest among the three sets of tests, and the final evaluation was “fair”. In the third test, the unmanned ship used the acceleration method to accelerate at a far distance, and finally completed the dynamic obstacle avoidance. The response distance of the unmanned craft in this test was far, which made up for the shortcoming of the small minimum distance, and the corresponding evaluation of the score was “fair”. According to the proportions of these three groups of test items [0.220, 0.306, 0.474], the calculated score finally corresponds to “fair”. According to the evaluation results, the obstacle avoidance strategy of the unmanned ship can be improved; for example, the response distance of the unmanned ship’s dynamic obstacle avoidance can be appropriately increased.