This is an early access version, the complete PDF, HTML, and XML versions will be available soon.
Open AccessArticle
RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition
by
Minwei Chen
Minwei Chen 1,
Yajun Liu
Yajun Liu 1,
Zenghui Zhang
Zenghui Zhang 1,*
and
Weiwei Guo
Weiwei Guo 2
1
Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, Shanghai 200240, China
2
Center of Digital Innovation, Tongji University, Shanghai 200092, China
*
Author to whom correspondence should be addressed.
Sensors 2024, 24(15), 4803; https://doi.org/10.3390/s24154803 (registering DOI)
Submission received: 24 June 2024
/
Revised: 18 July 2024
/
Accepted: 22 July 2024
/
Published: 24 July 2024
Abstract
Robust object detection in complex environments, poor visual conditions, and open scenarios presents significant technical challenges in autonomous driving. These challenges necessitate the development of advanced fusion methods for millimeter-wave (mmWave) radar point cloud data and visual images. To address these issues, this paper proposes a radar–camera robust fusion network (RCRFNet), which leverages self-supervised learning and open-set recognition to effectively utilise the complementary information from both sensors. Specifically, the network uses matched radar–camera data through a frustum association approach to generate self-supervised signals, enhancing network training. The integration of global and local depth consistencies between radar point clouds and visual images, along with image features, helps construct object class confidence levels for detecting unknown targets. Additionally, these techniques are combined with a multi-layer feature extraction backbone and a multimodal feature detection head to achieve robust object detection. Experiments on the nuScenes public dataset demonstrate that RCRFNet outperforms state-of-the-art (SOTA) methods, particularly in conditions of low visual visibility and when detecting unknown class objects.
Share and Cite
MDPI and ACS Style
Chen, M.; Liu, Y.; Zhang, Z.; Guo, W.
RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition. Sensors 2024, 24, 4803.
https://doi.org/10.3390/s24154803
AMA Style
Chen M, Liu Y, Zhang Z, Guo W.
RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition. Sensors. 2024; 24(15):4803.
https://doi.org/10.3390/s24154803
Chicago/Turabian Style
Chen, Minwei, Yajun Liu, Zenghui Zhang, and Weiwei Guo.
2024. "RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition" Sensors 24, no. 15: 4803.
https://doi.org/10.3390/s24154803
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details
here.
Article Metrics
Article Access Statistics
For more information on the journal statistics, click
here.
Multiple requests from the same IP address are counted as one view.