An Anti-Occlusion Approach for Enhanced Unmanned Surface Vehicle Target Detection and Tracking with Multimodal Sensor Data
Abstract
:1. Introduction
- A neural network for matching point cloud and image targets is constructed, addressing the problem of the incorrect matching of multi-modal information for multiple occluded targets. Simultaneously, the training process of this network replaces the tedious joint calibration work between the camera and LiDAR;
- A method which integrates data from LiDAR and a camera and considers the occlusion relationships between targets for WSTDT is proposed.
2. Related Work
3. Problem Statement
4. Method
4.1. Target Detection and Multi-Target Tracking
4.1.1. Water Surface Target Detection
4.1.2. Multi-Target Tracking
Algorithm 1 Multi-target Tracking |
Input: Output: # target tracking list (including location, category, speed, and tracking id) 1: # create an empty tracking list 2: 3: # assign a Kalman filter to each detected target in the initial detection frame 4: # and add them to the tracking list 5: 6: 7: # maintain the tracking loop while there are still new detection frames 8: # predict the state of each tracked target in the current frame 9: 10: 11: # match the tracked targets and detected targets using the Hungarian algorithm 12: # for the successfully matched targets, update their tracking status 13: 14: 15: # reset the count of undetected times 16: 17: # for the unmatched tracking targets 18: # increase its consecutive undetected times 19: 20: # for the unmatched detection targets 21: # assign a Kalman filter for it 22: # and add it to the tracking list 23: 24: # remove tracking targets that have not been detected consecutively for 10 times from the tracking list 25: 26: 27: 28: 29: 30: |
4.2. Target Matching Network and Improved YOLOv9
4.2.1. Target Matching Network
4.2.2. Improved YOLOv9
5. Experiments and Results
5.1. Comparison of Target Matching Methods
5.1.1. Data Collection
5.1.2. Comparative Experiment
5.2. Experiment of Target Detection and Multi-Target Tracking
5.2.1. Input Data Processing
5.2.2. Results of Target Detection and Multi-Target Tracking
5.3. Stability Evaluation
5.3.1. Evaluation under Foggy and Low-Light Conditions
5.3.2. Impact of Varying Ship Densities
5.3.3. Long-Term Stability Testing
5.3.4. Effect of External Disturbances
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Naus, K.; Wąż, M.; Szymak, P.; Gucma, L.; Gucma, M. Assessment of Ship Position Estimation Accuracy Based on Radar Navigation Mark Echoes Identified in an Electronic Navigational Chart. Measurement 2021, 169, 108630. [Google Scholar] [CrossRef]
- Yan, Z.; Cheng, L.; He, R.; Yang, H. Extracting Ship Stopping Information from AIS Data. Ocean. Eng. 2022, 250, 111004. [Google Scholar] [CrossRef]
- Almeida, C.; Franco, T.; Ferreira, H.; Martins, A.; Santos, R.; Almeida, J.M.; Carvalho, J.; Silva, E. Radar Based Collision Detection Developments on USV ROAZ II. In Proceedings of the OCEANS 2009-EUROPE, Bremen, Germany, 11–14 May 2009. [Google Scholar]
- Thombre, S.; Zhao, Z.; Ramm-Schmidt, H.; Vallet Garcia, J.M.; Malkamaki, T.; Nikolskiy, S.; Hammarberg, T.; Nuortie, H.; Bhuiyan, M.Z.H.; Sarkka, S.; et al. Sensors and AI Techniques for Situational Awareness in Autonomous Ships: A Review. IEEE Trans. Intell. Transport. Syst. 2022, 23, 64–83. [Google Scholar] [CrossRef]
- Farahnakian, F.; Heikkonen, J. Deep Learning Based Multi-Modal Fusion Architectures for Maritime Vessel Detection. Remote Sens. 2020, 12, 2509. [Google Scholar] [CrossRef]
- Boonchoo, T.; Ao, X.; Liu, Y.; Zhao, W.; Zhuang, F.; He, Q. Grid-Based DBSCAN: Indexing and Inference. Pattern Recognit. 2019, 90, 271–284. [Google Scholar] [CrossRef]
- Liu, Z.; Zhang, Y.; Yu, X.; Yuan, C. Unmanned Surface Vehicles: An Overview of Developments and Challenges. Annu. Rev. Control 2016, 41, 71–93. [Google Scholar] [CrossRef]
- Kufoalor, D.K.M.; Johansen, T.A.; Brekke, E.F.; Hepsø, A.; Trnka, K. Autonomous Maritime Collision Avoidance: Field Verification of Autonomous Surface Vehicle Behavior in Challenging Scenarios. J. Field Robot. 2020, 37, 387–403. [Google Scholar] [CrossRef]
- Thompson, D.J. Maritime Object Detection, Tracking, and Classification Using Lidar and Vision-Based Sensor Fusion. Master’s Thesis, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA, 2017. [Google Scholar]
- Woo, J.; Lee, J.; Kim, N. Obstacle Avoidance and Target Search of an Autonomous Surface Vehicle for 2016 Maritime RobotX Challenge. In Proceedings of the 2017 IEEE Underwater Technology (UT), Busan, Republic of Korea, 21–24 February 2017. [Google Scholar]
- Kamsvåg, V. Fusion between Camera and Lidar for Autonomous Surface Vehicles. Master’s Thesis, Norwegian University of Science and Technology, Trondheim, Norway, 2018. [Google Scholar]
- Wu, Y.; Qin, H.; Liu, T.; Liu, H.; Wei, Z. A 3D Object Detection Based on Multi-Modality Sensors of USV. Appl. Sci. 2019, 9, 535. [Google Scholar] [CrossRef]
- Clunie, T.; DeFilippo, M.; Sacarny, M.; Robinette, P. Development of a Perception System for an Autonomous Surface Vehicle Using Monocular Camera, LIDAR, and Marine RADAR. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 30 May–5 June 2021. [Google Scholar]
- Lu, Z.; Li, B.; Yan, J. Research on Unmanned Surface Vessel Perception Algorithm Based on Multi-Sensor Fusion. In Proceedings of the 2022 4th International Conference on Frontiers Technology of Information and Computer (ICFTIC), Qingdao, China, 2–4 December 2022. [Google Scholar]
- Wang, L.; Xiao, Y.; Zhang, B.; Liu, R.; Zhao, B. Water Surface Targets Detection Based on the Fusion of Vision and LiDAR. Sensors 2023, 23, 1768. [Google Scholar] [CrossRef]
- Liu, D.; Zhang, J.; Jin, J.; Dai, Y.; Li, L. A New Approach of Obstacle Fusion Detection for Unmanned Surface Vehicle Using Dempster-Shafer Evidence Theory. Appl. Ocean. Res. 2022, 119, 103016. [Google Scholar] [CrossRef]
- Chen, J.; Wang, H. An Obstacle Detection Method for USV by Fusing of Radar and Motion Stereo. In Proceedings of the 2020 IEEE 16th International Conference on Control & Automation (ICCA), Singapore, 9–11 October 2020. [Google Scholar]
- Haghbayan, M.-H.; Farahnakian, F.; Poikonen, J.; Laurinen, M.; Nevalainen, P.; Plosila, J.; Heikkonen, J. An Efficient Multi-Sensor Fusion Approach for Object Detection in Maritime Environments. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention Is All You Need. arXiv 2017, arXiv:1706.03762. [Google Scholar]
- Kim, A.; Osep, A.; Leal-Taixe, L. EagerMOT: 3D Multi-Object Tracking via Sensor Fusion. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 30 May–5 June 2021. [Google Scholar]
- Wang, X.; Fu, C.; He, J.; Wang, S.; Wang, J. StrongFusionMOT: A Multi-Object Tracking Method Based on LiDAR-Camera Fusion. IEEE Sens. J. 2023, 23, 11241–11252. [Google Scholar] [CrossRef]
- Wang, C.-Y.; Yeh, I.-H.; Liao, H.-Y.M. YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information. arXiv 2024, arXiv:2402.13616. [Google Scholar]
- Breitinger, A.; Clua, E.; Fernandes, L.A.F. An Augmented Reality Periscope for Submarines with Extended Visual Classification. Sensors 2021, 21, 7624. [Google Scholar] [CrossRef] [PubMed]
- Bingham, B.; Aguero, C.; McCarrin, M.; Klamo, J.; Malia, J.; Allen, K.; Lum, T.; Rawson, M.; Waqar, R. Toward Maritime Robotic Simulation in Gazebo. In Proceedings of the OCEANS 2019 MTS/IEEE SEATTLE, Seattle, WA, USA, 27–31 October 2019. [Google Scholar]
- Amosa, T.I.; Sebastian, P.; Izhar, L.I.; Ibrahim, O.; Ayinla, L.S.; Bahashwan, A.A.; Bala, A.; Samaila, Y.A. Multi-Camera Multi-Object Tracking: A Review of Current Trends and Future Advances. Neurocomputing 2023, 552, 126558. [Google Scholar] [CrossRef]
Year of Research Publication | Sensors | Need for Joint Calibration | Feature Matching Capability * |
---|---|---|---|
2017 [9] | GPS/INS | Yes | Bad |
2D-LiDAR | |||
Monocular camera | |||
2017 [10] | 2D-LiDAR | Yes | Bad |
Monocular camera | |||
2018 [12] | 3D-LiDAR | Yes | Good |
Monocular camera | |||
2018 [11] | 3D-LiDAR | Yes | Bad |
Monocular camera | |||
2018 [18] | 3D-LiDAR | Yes | Bad |
Monocular camera | |||
2020 [17] | GPS | Yes | Normal |
Radar | |||
Binocular camera | |||
2021 [13] | IMU | Yes | Bad |
3D-LiDAR | |||
Radar | |||
Monocular camera | |||
2022 [14] | 3D-LiDAR | Yes | Bad |
Monocular camera | |||
2022 [16] | 3D-LiDAR | Yes | Normal |
Millimeter-wave Radar | |||
Binocular camera | |||
3D-LiDAR | |||
2023 [15] | Monocular camera | Yes | / |
2D-LiDAR |
Component | Specification |
CPU | i7-10700 |
GPU | RTX-3060 |
Memory | 24 GB RAM |
Storage | 256 GB SSD |
Operating System | Linux Ubuntu 20.04 LTS (noetic) |
Method | Ordinary | Complex | ||||
---|---|---|---|---|---|---|
Acc (%) | TP + TN | GT | Acc (%) | TP + TN | GT | |
IOU | 83.00 | 874 | 1053 | 76.76 | 1199 | 1562 |
SDIOU [21] | 73.22 | 771 | 1053 | 67.54 | 1055 | 1562 |
MLP (Ours) | 89.36 | 941 | 1053 | 84.83 | 1325 | 1562 |
Attention + MLP (Ours) | 92.02 | 969 | 1053 | 89.44 | 1397 | 1562 |
Occ + Attention + MLP (Ours) | 92.88 | 978 | 1053 | 90.59 | 1415 | 1562 |
Method | FP | FN | IDSW | MOTA (%) | FPS |
---|---|---|---|---|---|
IoU-based | 5802 | 1511 | 354 | 71.9 | 23.73 |
Ours | 1437 | 1511 | 354 | 87.9 | 21.98 |
Environment | Recall (%) | Precision (%) |
---|---|---|
Normal | 92.14 | 98.84 |
Light Fog | 96.43 | 98.15 |
Heavy Fog | 52.50 | 85.03 |
50% Lighting | 94.64 | 98.49 |
20% Lighting | 60.36 | 92.31 |
Environment | Accuracy (%) | TP + TN | GT |
---|---|---|---|
Normal | 92.93 | 880 | 947 |
Light Fog | 91.34 | 865 | 947 |
Heavy Fog | 94.83 | 898 | 947 |
50% Lighting | 92.82 | 879 | 947 |
20% Lighting | 93.56 | 886 | 947 |
Environment | FP | FN | IDSW | Fixed MOTA (%) | FPS |
---|---|---|---|---|---|
Normal | 1437 | 5204 | 354 | 74.36 | 21.98 |
Light Fog | 1492 | 4965 | 532 | 74.38 | 21.65 |
Heavy Fog | 2541 | 6716.5 | 354 | 64.77 | 22.13 |
50% Lighting | 1465 | 5198 | 354 | 74.28 | 21.72 |
20% Lighting | 1959 | 6440 | 534 | 67.26 | 22.04 |
Number of Ships | Number of Other Obstacles | FP | FN | IDSW | MOTA (%) |
---|---|---|---|---|---|
3 | 16 | 911 | 360 | 5 | 90.19 |
6 | 16 | 1115 | 207 | 2 | 91.71 |
9 | 16 | 1270 | 302 | 6 | 91.31 |
12 | 16 | 1643 | 322 | 247 | 0.8771 |
Standard Deviation | FP | FN | IDSW | MOTA (%) |
---|---|---|---|---|
0 | 1437 | 1511 | 354 | 87.89 |
0.5 | 1593 | 1633 | 349 | 86.90 |
1 | 2712 | 1349 | 446 | 83.48 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zheng, M.; Li, D.; Chen, G.; Wang, W.; Yang, S. An Anti-Occlusion Approach for Enhanced Unmanned Surface Vehicle Target Detection and Tracking with Multimodal Sensor Data. J. Mar. Sci. Eng. 2024, 12, 1558. https://doi.org/10.3390/jmse12091558
Zheng M, Li D, Chen G, Wang W, Yang S. An Anti-Occlusion Approach for Enhanced Unmanned Surface Vehicle Target Detection and Tracking with Multimodal Sensor Data. Journal of Marine Science and Engineering. 2024; 12(9):1558. https://doi.org/10.3390/jmse12091558
Chicago/Turabian StyleZheng, Minjie, Dingyuan Li, Guoquan Chen, Weijun Wang, and Shenhua Yang. 2024. "An Anti-Occlusion Approach for Enhanced Unmanned Surface Vehicle Target Detection and Tracking with Multimodal Sensor Data" Journal of Marine Science and Engineering 12, no. 9: 1558. https://doi.org/10.3390/jmse12091558