A Target Detection Algorithm Based on Fusing Radar with a Camera in the Presence of a Fluctuating Signal Intensity

Abstract

Radar point clouds will experience variations in density, which may cause incorrect alerts during clustering. In turn, it will diminish the precision of the decision-level fusion method. To address this problem, a target detection algorithm based on fusing radar with a camera in the presence of a fluctuating signal intensity is proposed in this paper. It introduces a snow ablation optimizer (SAO) for solving the optimal parameters of the density-based spatial clustering of applications with noise (DBSCAN). Subsequently, the enhanced DBSCAN clusters radar point clouds, and the valid clusters are fused with monocular camera targets. The experimental results indicate that the suggested fusion method can attain a Balance-score ranging from 0.97 to 0.99, performing outstandingly in preventing missed detections and false alarms. Additionally, the fluctuation range of the Balance-score is within 0.02, indicating the algorithm has an excellent robustness.

1. Introduction

With vehicle ownership increasing, traffic safety and congestion problems have become increasingly severe. According to the statistical analysis, traffic accidents caused by humans, such as distraction and fatigue driving, are as high as 94% [1,2]. In this context, developing intelligent driving vehicles that aim to reduce traffic accidents has become a research hotspot [3,4]. The intelligent driving system mainly uses sensors such as LiDAR, cameras, and radar to sense information about targets [5,6]. Where LiDAR offers benefits such as high detection precision and an extensive detection range, but it is susceptible to weather and difficult to obtain the texture features of targets [7]. Although cameras are sensitive to target textures [8,9,10,11], they usually suffer from drawbacks such as large-ranging errors and susceptibility to weather [12,13]. On the contrary, radar can not only provide precise information about distance and speed, but can also work well around the clock [14,15,16,17]. However, it is difficult for radar to acquire the attributes of the targets [18].

In summary, each type of sensor has drawbacks, making it challenging to meet the requirements of complex environments. Therefore, using multi-sensor fusion methods to sense targets has become a trend in intelligent driving. Information fusion techniques can fully utilize the benefits of various sensors, thus enhancing redundancy and stability in the recognition system [19]. Currently, the radar-vision (RV) fusion method has become mainstream due to its ability to achieve a high detection accuracy with minimal computational and cost sacrifices [20,21].

The RV fusion methods are divided into data level, feature level, and decision level [22]. Where the decision-level fusion strategy first processes the data detected by different sensors individually, and then compares, evaluates, and correlates the processing results [23]. Because the structure and idea of this strategy are simple, it has become mainstream. However, its performance is usually limited by specific combinations of sensors and algorithms. For example, if radar false alarms are frequent, the accuracy of the fusion algorithm will dramatically decrease.

In [24], the authors utilize the constant false-alarm rate (CFAR) to process the radar data so as to reduce noise and false alarms. However, the method’s performance will drop drastically when the strength of the noise or interfering signal is strong. To solve this problem, ref. [25] plotted targets detected by radar on the image, and applied a Gaussian Mixture Model (GMM) to segment the dynamic targets from the static background. However, the method will be limited by the coordinate transformation error. In [26], the authors filtered out the background objects, such as road signs and bins, by setting a speed threshold. Nevertheless, because this method compares the velocity of all objects detected by radar to the same threshold, it is prone to situations where the valid targets will be misclassified as disturbances. For solving this problem, ref. [27] processed the radar point clouds using the density-based spatial clustering of applications with noise (DBSCAN), which can separate targets with a different reflectivity. Then, the valid objects from the radar and camera were fused. Nonetheless, DBSCAN requires pre-setting parameters, which will limit its generalizability and accuracy [28].

To summarize, the results of existing decision-level fusion methods usually depend on the specific combination of algorithms for sensors. In particular, when the echo signal strengths of the targets to be detected are diverse, the radar’s false alarms will increase dramatically. Accompanying, the effectiveness of fusion will decrease. To tackle this issue, a target detection algorithm based on fusing radar with a camera in the presence of a fluctuating signal intensity is proposed in this paper. Firstly, it introduces the snow ablation optimizer (SAO) [29] for solving the optimal parameters of DBSCAN. And the enhanced DBSCAN, which we name SAO-DBSCAN, is applied for clustering radar point clouds. Then, the targets in the image are extracted using the YOLOv5s model [30]. Finally, targets identified by radar and camera are integrated through a decision-level fusion strategy. Experiments indicate that the suggested fusion method can achieve excellent accuracy and be applied to scenarios where the intensities of the echo signal vary. This research makes several contributions, which are outlined below:

(1) We introduce the SAO to optimize the parameters of DBSCAN. This method does not require manual parameter adjustment and has excellent generality.

(2) We use the proposed SAO-DBSCAN to cluster the radar point clouds, which can automatically determine the optimal parameters based on the distribution of the point clouds, thereby significantly decreasing the radar false alarms.

(3) We fuse the valid clusters, which are obtained by the proposed SAO-DBSCAN, with the objects detected by a camera. It can enhance the precision of the decision-level fusion method by reducing false alarms from radar.

The remainder of the paper is structured as follows: Section 2 exhibits the fusion model. Section 3 describes some critical steps and algorithms of fusion. In Section 4, we showcase and analyze the experimental findings. Finally, Section 5 summarizes the conclusions.

2. Information Fusion Model

For the fusion model suggested in this study, radar primarily delivers the targets’ positions and velocities, while the camera supplies their classifications. In addition, the camera is also responsible for acquiring the lane information, which helps fuse video data with radar from a same lane line. The schematic is illustrated in Figure 1.

For the decision-level fusion strategy, it is crucial to process the data detected by each sensor separately before fusion. In the vision-based target detection module, we first use a second-order polynomial to extract the lane information in the picture [31]. Then, the YOLOv5s model is employed to identify the objects on the image. In the radar-based target detection module, data from radar are first pre-processed. Then, the radar point clouds reflected from each target are clustered by the proposed SAO-DBSCAN. Subsequently, we utilize the lane lines to eliminate interferences that lie outside the lanes, and mark the lane information of the valid objects that remain [32].

In the fusion-based target detection model, data detected by the two sensors are temporally aligned through software. And the spatial alignment is achieved based on the positional relationship between the two sensors. Next, we calculate the intersection over union (IOU) based on the detection boxes between objects identified by different sensors in the same lane. Finally, the value of IOU is used to achieve target-matching between sensors for information fusion.

3. Realization of Fusion

3.1. Radar Data Processing

3.1.1. Signal Model of the FMCW Radar

Here, the frequency-modulated continuous wave (FMCW) radar has been utilized to gather information about targets. Its basic principle is constructing an intermediate frequency (IF) signal using the transmitted and echo signals [33]. Assuming that the sampling period is ${\mathrm{T}}_{\mathrm{s}}$, the expression for the IF signal is as follows:

$${\mathrm{S}}_{\mathrm{IF}}(\mathrm{m},\mathrm{l},\mathrm{n})={\mathrm{e}}^{\mathrm{j}4\mathsf{\pi }{\mathrm{f}}_0\mathrm{R}}{\mathrm{e}}^{\mathrm{j}2\mathsf{\pi }{\displaystyle \frac{\mathrm{dsin}\theta }{\lambda }}\mathrm{m}}{\mathrm{e}}^{\mathrm{j}2\mathsf{\pi }{\displaystyle \frac{2{\mathrm{f}}_0\mathrm{v}}{\mathrm{c}}}\mathrm{l}}{e}^{\mathrm{j}2\mathsf{\pi }\left({\mathrm{f}}_0{\displaystyle \frac{2\mathrm{v}}{\mathrm{c}}}+{\displaystyle \frac{\mathrm{B}2\mathrm{R}}{{\mathrm{T}}_{\mathrm{c}}}}\right){\mathrm{nT}}_{\mathrm{s}}}$$

(1)

where ${\mathrm{S}}_{\mathrm{IF}}(\mathrm{m},\mathrm{l},\mathrm{n})$ denotes the sample at the m−th antenna, the l−th frequency-modulation (FM) cycle, and the n−th sampling moment. ${\mathrm{T}}_{\mathrm{c}}$ is the duration of an FM cycle. ${\mathrm{f}}_0$ represents the starting frequency of the transmitted signal. $\mathrm{c}$ indicates the velocity of light. $\mathrm{d}$ indicates the separation of neighboring array elements whose value is $\mathrm{d}=2\mathsf{\pi }{\mathrm{f}}_0/\mathrm{c}$. Additionally, $\theta$, $\mathrm{R}$, and $\mathrm{v}$ represent the azimuth, distance, and velocity, respectively, between the target and radar. After pre-processing the IF signal, we can generate the radar point clouds for clustering.

3.1.2. The Improved DBSCAN Algorithm

DBSCAN [34] is a clustering algorithm based on density that focuses on grouping regions with varying densities into separate clusters. It is composed of two concepts: $\epsilon$ and $\mathrm{MinPts}$. Where $\epsilon$ represents the domain radius of the sample points, and $\mathrm{MinPts}$ signifies the minimum quantity of point clouds within each cluster. Based on the above concepts, the samples can be divided into core, border, and noise points by DBSCAN, as shown in Figure 2.

The core point indicates that the quantity of points contained in its $\epsilon -\mathrm{neighborhood}$ is at least MinPts. The border points are characterized by having fewer samples than MinPts in their $\epsilon -\mathrm{neighborhood}$, but they are located within the core point’s $\epsilon -\mathrm{neighborhood}$. The noise points are those except the core and border points. The basic modeling steps for DBSCAN are shown below:

(1) Choose any sample ${\mathrm{X}}_{\mathrm{i}}$ in the dataset as a point $\mathrm{P}$ in the space.

(2) Search for samples in the $\epsilon -\mathrm{neighborhood}$ of point $\mathrm{P}$. If the number of samples exceeds or is equal to MinPts, all points within the $\epsilon -\mathrm{neighborhood}$ of $\mathrm{P}$ belong to the same cluster.

(3) Repeat Step (2) starting from the neighboring samples of point $\mathrm{P}$ until all samples in the dataset have been traversed. Ultimately, clusters that represent targets and noise are obtained.

The advantages of DBSCAN are that it can recognize noise in radar point clouds, as well as handle clusters with different shapes and sizes. However, its generalizability is poor. This is because the parameters of DBSCAN have to be pre-set, which makes it challenging to utilize in situations with significant fluctuations in point cloud density. To adaptively solve the optimal parameters according to the density of the point clouds, we introduce the SAO algorithm to improve DBSCAN. The methodology mainly consists of three phases: initialization, exploration, and exploitation. Additionally, a dual population scheme is utilized to balance the exploration and exploitation. Its basic principles are as follows:

Fistly, we define the optimization function $\left(\mathrm{fitness}\right)$ of SAO as follows:

$$\mathrm{fitness}=1-\mathrm{S}$$

(2)

where $\mathrm{S}$, taking the value in the range of $[-1,1]$, is called the average silhouette coefficient [35] of clustering. And, as the value of $\mathrm{S}$ increases, the clustering performance will improve. Assuming that the point clouds are divided into C clusters by DBSCAN, the formula for $\mathrm{S}$ is as follows:

$$\mathrm{S}={\displaystyle \frac{1}{\mathrm{C}}}\sum _{\mathrm{i}=1}^{\mathrm{C}}\mathrm{S}\left({\mathrm{p}}_{\mathrm{i}}\right)$$

(3)

$$\mathrm{S}\left({\mathrm{p}}_{\mathrm{i}}\right)={\displaystyle \frac{\mathrm{b}\left({\mathrm{p}}_{\mathrm{i}}\right)-\mathrm{a}\left({\mathrm{p}}_{\mathrm{i}}\right)}{max\{\mathrm{b}\left({\mathrm{p}}_{\mathrm{i}}\right),\mathrm{a}\left({\mathrm{p}}_{\mathrm{i}}\right)\}}}$$

(4)

where $\mathrm{S}\left({\mathrm{p}}_{\mathrm{i}}\right)$ denotes the silhouette coefficient of each cluster. The role of $\mathrm{a}\left({\mathrm{p}}_{\mathrm{i}}\right)$ is to quantify the cohesion within the cluster. Its value is derived from the average of the distances between ${\mathrm{p}}_{\mathrm{i}}$ and other samples in its $\epsilon -\mathrm{neighborhood}$. $\mathrm{b}\left({\mathrm{p}}_{\mathrm{i}}\right)$ denotes the smallest value of the mean distances from ${\mathrm{p}}_{\mathrm{i}}$ to all points in the neighboring clusters, which is used to quantify the separation among clusters.

During the initialization phase, the upper (U) and lower (L) limits of the parameters to be optimized are pre-set. Then, the snow cluster $\mathbf{J}$ is generated:

$$\mathbf{J}=\mathrm{L}+\beta \times (\mathrm{U}-\mathrm{L})$$

(5)

where $\beta$ is a random number with a value in the range of [0,1]. As the parameters of DBSCAN are $\epsilon$ and MinPts, $\mathbf{J}$ can be written as follows:

$$\mathbf{J}=\left[\begin{array}{cc}{\epsilon }_1& {\mathrm{MinPts}}_1\\ {\epsilon }_2& {\mathrm{MinPts}}_2\\ ⋮& ⋮\\ {\epsilon }_{\mathrm{K}}& {\mathrm{MinPts}}_{\mathrm{K}}\end{array}\right]$$

(6)

where $\mathrm{K}$ denotes the size of the snow cluster.

During the exploration phase, a probability density function of the standard normal distribution is used to model the highly dispersive Brownian Motion phenomenon [36], which occurs when the snow-melted water is transformed into steam. Its mathematical expression is as follows:

$${\mathrm{f}}_{\mathrm{BM}}\left(\mathrm{x}\right)={\displaystyle \frac{1}{\sqrt{2\mathsf{\pi }}}}{\mathrm{e}}^{-{\mathrm{x}}^2/2}.$$

(7)

Therefore, the positional update formula for this phase is as follows:

$$\mathrm{J}(\mathrm{j},\mathrm{t}+1)={\mathrm{J}}_{\mathrm{elite}}\left(\mathrm{t}\right)+{\mathrm{p}}_{\mathrm{BM}}(\mathrm{j},\mathrm{t})\otimes ({\beta }_1\times ({\mathrm{J}}_{1\mathrm{st}}\left(\mathrm{t}\right)-\mathrm{J}(\mathrm{j},\mathrm{t}))+(1-{\beta }_1)\times (\overline{\mathrm{J}}\left(\mathrm{t}\right)-\mathrm{J}(\mathrm{j},\mathrm{t})))$$

(8)

where ⊗ denotes entry-wise multiplications. ${\beta }_1$ denotes a random number from 0 to 1. ${\mathrm{p}}_{\mathrm{BM}}(\mathrm{j},\mathrm{t})$ is a random vector generated with a Gaussian distribution, which is used to represent the phenomenon of Brownian Motion. $\mathrm{J}(\mathrm{j},\mathrm{t})$ denotes the particle j in the snow cluster during iteration t. ${\mathrm{J}}_{1\mathrm{st}}\left(\mathrm{t}\right)$ represents the current optimal particle. $\overline{\mathrm{J}}\left(\mathrm{t}\right)$ denotes the mean value of the whole snow cluster. And ${\mathrm{J}}_{\mathrm{elite}}\left(\mathrm{t}\right)$ is an elite set. The mathematical expressions for $\overline{\mathrm{J}}\left(\mathrm{t}\right)$ and ${\mathrm{J}}_{\mathrm{elite}}\left(\mathrm{t}\right)$ are as follows:

$$\overline{\mathrm{J}}\left(\mathrm{t}\right)={\displaystyle \frac{1}{\mathrm{K}}}\sum _{\mathrm{j}=1}^{\mathrm{K}}\mathrm{J}(\mathrm{j},\mathrm{t})$$

(9)

$${\mathrm{J}}_{\mathrm{elite}}\left(\mathrm{t}\right)=[{\mathrm{J}}_{1\mathrm{st}}\left(\mathrm{t}\right),{\mathrm{J}}_{2\mathrm{nd}}\left(\mathrm{t}\right),{\mathrm{J}}_{3\mathrm{rd}}\left(\mathrm{t}\right),{\mathrm{J}}_{1/2}\left(\mathrm{t}\right)]$$

(10)

where ${\mathrm{J}}_{2\mathrm{nd}}\left(\mathrm{t}\right)$ is the second-best particle and ${\mathrm{J}}_{3\mathrm{rd}}\left(\mathrm{t}\right)$ is the third-best. ${\mathrm{J}}_{1/2}\left(\mathrm{t}\right)$ is the mean value of the snow cluster ranked in the top 50 percent of fitness values:

$${\mathrm{J}}_{1/2}\left(\mathrm{t}\right)={\displaystyle \frac{1}{{\mathrm{K}}_1}}\sum _{\mathrm{j}=1}^{{\mathrm{K}}_1}\mathrm{J}(\mathrm{j},\mathrm{t}).$$

(11)

Here, the value of ${\mathrm{K}}_1$ is taken to be half of $\mathrm{K}$.

During the exploitation phase, the snow-melt model mainly develops around the current optimal solution rather than further extending its high dispersion. The positional update formula for this phase is as follows:

$$\mathrm{J}(\mathrm{j},\mathrm{t}+1)={\mathrm{S}}_{\mathrm{M}}\left(\mathrm{t}\right)\times {\mathrm{J}}_{1\mathrm{st}}\left(\mathrm{t}\right)+{\mathrm{p}}_{\mathrm{BM}}(\mathrm{j},\mathrm{t})\otimes ({\beta }_2\times ({\mathrm{J}}_{1\mathrm{st}}\left(\mathrm{t}\right)-\mathrm{J}(\mathrm{j},\mathrm{t}))+(1-{\beta }_2)\times (\overline{\mathrm{J}}\left(\mathrm{t}\right)-\mathrm{J}(\mathrm{j},\mathrm{t})))$$

(12)

where ${\beta }_2$ denotes a random number that takes a value in the range of $[-1,1]$. And ${\mathrm{S}}_{\mathrm{M}}\left(\mathrm{t}\right)$ is the snow-melt rate, which is expressed as follows:

$${\mathrm{S}}_{\mathrm{M}}\left(\mathrm{t}\right)=(0.35+0.25\times {\displaystyle \frac{{\mathrm{e}}^{\frac{\mathrm{t}}{{\mathrm{t}}_{\max }}-1}}{\mathrm{e}-1}})\times {\mathrm{e}}^{{\displaystyle \frac{-\mathrm{t}}{{\mathrm{t}}_{\max }}}}$$

(13)

where ${\mathrm{t}}_{\max }$ denotes the maximum number of iterations.

Finally, the SAO algorithm designs a dual population scheme. It aims at balancing the exploration and exploitation phases. At the early stage of the optimization search, this scheme randomly divides the entire snow cluster into two equal-sized sub-populations, which are responsible for exploration and exploitation. As the iteration progresses, the membership of sub-population 1 dedicated to exploration gradually decreases. At the same time, the membership of sub-population 2 dedicated to exploitation gradually increases. Figure 3 illustrates the block diagram of the suggested SAO-DBSCAN.

3.2. Video Data Processing

In this study, the YOLOv5s model is used to extract the targets in the image. Its structure is shown in Figure 4. This model mainly consists of the feature extraction network (Backbone), the feature fusion network (Neck), and the detection head (Head) [37,38]. Where the Backbone network is used to extract the targets’ features. The Neck network is used to fuse the features of different scales. The Head network is used to perform classification, regression, and prediction. It contains three detection heads, which are used to detect large, medium, and small targets.

3.3. Spatio-Temporal Calibration

3.3.1. Temporal Alignment

A popular method for aligning the sampling times of multi-sensors is based on timestamps [39]. Because the sampling periods for radar and video are 10 and 30 FPS, we chose that of radar as the baseline. Assume that the starting alignment time of radar and camera is $\alpha$ seconds, the corresponding frame is as follows:

$$\mathrm{F}=\alpha \times 30$$

(14)

where the value of $\alpha$ is recorded by the software. Because video has three times the FPS of radar, the frame $\mathrm{i}$ of radar data is aligned with the frame $\mathrm{F}+\mathrm{i}\times 3$ of the image. Figure 5 illustrates the principle of temporal alignment.

The horizontal axis indicates time, which is measured in FPS. The red boxes represent the radar data, the yellow boxes represent the video data, and the purple boxes represent the data after time synchronization.

3.3.2. Spatial Calibration

Spatial synchronization involves projecting radar-identified objects onto the image and quantifying their positions in pixels [40]. Figure 6 depicts the basic principle of coordinate transformation.

In this study, we take the world coordinate system as a reference, which is aligned with the camera. Then, the radar coordinate system can be obtained by rotating and translating that of camera. The transformation formula can be expressed as follows:

$$\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{ca}}\\ {\mathrm{y}}_{\mathrm{ca}}\\ {\mathrm{z}}_{\mathrm{ca}}\\ 1\end{array}\right]=\left[\begin{array}{cc}{\mathbf{R}}^{\mathrm{rc}}& {\mathrm{T}}^{\mathrm{rc}}\\ 0^{\mathrm{T}}& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{ra}}\\ {\mathrm{y}}_{\mathrm{ra}}\\ {\mathrm{z}}_{\mathrm{ra}}\\ 1\end{array}\right]$$

(15)

where $({\mathrm{x}}_{\mathrm{ca}},{\mathrm{y}}_{\mathrm{ca}},{\mathrm{z}}_{\mathrm{ca}})$ denotes the coordinates with an origin at the optical center of camera, and $({\mathrm{x}}_{\mathrm{ra}},{\mathrm{y}}_{\mathrm{ra}},{\mathrm{z}}_{\mathrm{ra}})$ denotes the coordinates with origin at the center of radar. ${\mathbf{R}}^{\mathrm{rc}}$ denotes a rotational relationship between the camera and radar, and ${\mathrm{T}}^{\mathrm{rc}}$ denotes a translational relationship. As shown in Figure 6a, the camera is fitted in the rectangular box to ensure no angular deflection between the two sensors. Thus, ${\mathbf{R}}^{\mathrm{rc}}$ is a unit matrix of dimension $3\times 3$. Then, by measuring, we can obtain the elements of the matrix ${\mathrm{T}}^{\mathrm{rc}}$ are as follows:

$${\mathrm{T}}^{\mathrm{rc}}=\left[\begin{array}{c}\Delta \mathrm{x}\\ \Delta \mathrm{y}\\ \Delta \mathrm{z}\end{array}\right]=\left[\begin{array}{c}0\\ -180\\ -17\end{array}\right]$$

(16)

where $\Delta \mathrm{x}$, $\Delta \mathrm{y}$ and $\Delta \mathrm{z}$ denote the offset of the corresponding coordinate axes between the centers of the radar and the camera, respectively, which are measured in millimeters.

As shown in Figure 6b, a two-dimensional imaging plane is usually used to represent the position of a target in an image. Then, the transformation relationship between the camera coordinate and that of image can be expressed as follows:

$${\mathrm{z}}_{\mathrm{ca}}\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{I}}\\ {\mathrm{y}}_{\mathrm{I}}\\ 1\end{array}\right]=\left[\begin{array}{cccc}\mathrm{f}& 0& 0& 0\\ 0& \mathrm{f}& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{ca}}\\ {\mathrm{y}}_{\mathrm{ca}}\\ {\mathrm{z}}_{\mathrm{ca}}\\ 1\end{array}\right]$$

(17)

where $\mathrm{f}$ represents the camera’s focal length. However, the above coordinate systems are all in units of length, which are not conducive to subsequent operations. Therefore, it is necessary to convert the image coordinate system ${\mathrm{o}}_{\mathrm{I}}-{\mathrm{x}}_{\mathrm{I}}{\mathrm{y}}_{\mathrm{I}}$ into that of pixel ${\mathrm{o}}_{\mathrm{p}}-{\mathrm{u}}_{\mathrm{p}}{\mathrm{v}}_{\mathrm{p}}$. These two coordinate systems are in the same plane, and their differences are that they have different origins and units of measurement. Here, the origin ${\mathrm{o}}_{\mathrm{p}}$ of the pixel coordinate system is located at the top left corner of the imaging plane, as shown in the shaded portion of Figure 6b. Assuming that the corresponding physical lengths of each pixel on the ${\mathrm{x}}_{\mathrm{I}}$ and ${\mathrm{y}}_{\mathrm{I}}$ axes are $\mathrm{dx}$ and $\mathrm{dy}$, respectively, the coordinate transformation relationship between the image and pixel is as follows:

$$\left\{\begin{array}{c}{\mathrm{u}}_{\mathrm{p}}={\displaystyle \frac{{\mathrm{x}}_{\mathrm{I}}}{\mathrm{dx}}}+{\mathrm{u}}_{\mathrm{p}0}\hfill \\ {\mathrm{v}}_{\mathrm{p}}={\displaystyle \frac{{\mathrm{y}}_{\mathrm{I}}}{\mathrm{dy}}}+{\mathrm{v}}_{\mathrm{p}0}\hfill \end{array}\right.$$

(18)

$$\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}}\\ {\mathrm{v}}_{\mathrm{p}}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{\displaystyle \frac{1}{\mathrm{dx}}}& 0& {\mathrm{u}}_{\mathrm{p}0}\\ 0& {\displaystyle \frac{1}{\mathrm{dy}}}& {\mathrm{v}}_{\mathrm{p}0}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{I}}\\ {\mathrm{y}}_{\mathrm{I}}\\ 1\end{array}\right]$$

(19)

where $({\mathrm{u}}_{\mathrm{p}0},{\mathrm{v}}_{\mathrm{p}0})$ denotes the position of point ${\mathrm{o}}_{\mathrm{I}}$ under the pixel coordinate system. To summarize, the transformation formula from the radar coordinate system to that of the pixel is as follows:

$${\mathrm{z}}_{\mathrm{ca}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}}\\ {\mathrm{v}}_{\mathrm{p}}\\ 1\end{array}\right]=\left[\begin{array}{cccc}{\displaystyle \frac{\mathrm{f}}{\mathrm{dx}}}& 0& {\mathrm{u}}_{\mathrm{p}0}& 0\\ \\ 0& {\displaystyle \frac{\mathrm{f}}{\mathrm{dy}}}& {\mathrm{v}}_{\mathrm{p}0}& 0\\ \\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}{\mathbf{R}}^{\mathrm{rc}}& {\mathrm{T}}^{\mathrm{rc}}\\ 0^{\mathrm{T}}& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{ra}}\\ {\mathrm{y}}_{\mathrm{ra}}\\ {\mathrm{z}}_{\mathrm{ra}}\\ 1\end{array}\right]$$

(20)

It can be further simplified as:

$$\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{p}}\\ {\mathrm{v}}_{\mathrm{p}}\\ 1\end{array}\right]={\displaystyle \frac{1}{{\mathrm{z}}_{\mathrm{ca}}}}{\mathbf{K}}_{\mathrm{int}}{\mathbf{K}}_{\mathrm{ext}}\left[\begin{array}{c}{\mathrm{x}}_{\mathrm{ra}}\\ {\mathrm{y}}_{\mathrm{ra}}\\ {\mathrm{z}}_{\mathrm{ra}}\\ 1\end{array}\right]$$

(21)

$${\mathbf{K}}_{\mathrm{int}}=\left[\begin{array}{cccc}{\displaystyle \frac{\mathrm{f}}{\mathrm{dx}}}& 0& {\mathrm{u}}_{\mathrm{p}0}& 0\\ \\ 0& {\displaystyle \frac{\mathrm{f}}{\mathrm{dy}}}& {\mathrm{v}}_{\mathrm{p}0}& 0\\ \\ 0& 0& 1& 0\end{array}\right]=\left[\begin{array}{cccc}{\alpha }_{\mathrm{x}}& 0& {\mathrm{u}}_{\mathrm{p}0}& 0\\ \\ 0& {\alpha }_{\mathrm{y}}& {\mathrm{v}}_{\mathrm{p}0}& 0\\ \\ 0& 0& 1& 0\end{array}\right]$$

(22)

$${\mathbf{K}}_{\mathrm{ext}}=\left[\begin{array}{cc}{\mathbf{R}}^{\mathrm{rc}}& {\mathrm{T}}^{\mathrm{rc}}\\ 0^{\mathrm{T}}& 1\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ \\ 0& 1& 0& -180\\ \\ 0& 0& 1& -17\\ \\ 0& 0& 0& 1\end{array}\right]$$

(23)

where ${\mathbf{K}}_{\mathrm{int}}$ is the internal parameters matrix of the camera. ${\mathbf{K}}_{\mathrm{ext}}$ is the external parameters matrix, which can be measured using the Zhengyou Zhang calibration method [41]. The specific calibration process is as follows:

Firstly, a checkerboard with $10\times 8$ squares is selected as the calibration board, and the side length of each square is $0.02\mathrm{m}$. To obtain calibration results with a high accuracy, we need to take at least 20 images. The calibration images with different angles and attitudes are shown in Figure 7a.

Next, the internal corner points of the checkerboard grid in each image are extracted using the calibration toolbox. Finally, the internal parameters of the camera are derived, as shown in

Table 1. Calibration results of the internal camera parameters.

Internal Parameters	Horizontal Axis/Pixel	Vertical Axis/Pixel
Equivalent focal length	${\alpha }_{\mathrm{x}}=1545.9$	${\alpha }_{\mathrm{y}}=1550.4$
Principal point	${\mathrm{u}}_{\mathrm{p}0}=1001.1$	${\mathrm{v}}_{\mathrm{p}0}=529.5$

.

The error of the internal parameters calibration is shown in Figure 7b. Because the average error is within 0.19 pixels, it meets the requirements for practical applications.

3.4. Information Fusion Strategy

At the beginning, we plot the region of interest (ROI) of the radar-detected object on the image. According to the dimensional limits of the outer contour of the vehicle [42], we set the vehicle height to $2.4\mathrm{m}$ and width to $2.0\mathrm{m}$. Thus, the width $\left({\mathrm{W}}_{\mathrm{ra}}\right)$ and height $\left({\mathrm{H}}_{\mathrm{ra}}\right)$ of the ROI occupied by the radar point in the imaging plane are as follows:

$$\left\{\begin{array}{cc}{\mathrm{W}}_{\mathrm{ra}}=2.4\times {\displaystyle \frac{{\alpha }_{\mathrm{x}}}{{\mathrm{y}}_{\mathrm{ra}}}}\hfill & \\ & \\ {\mathrm{H}}_{\mathrm{ra}}=2.0\times {\displaystyle \frac{{\alpha }_{\mathrm{y}}}{{\mathrm{y}}_{\mathrm{ra}}}}\hfill \end{array}\right.$$

(24)

where both $\left({\mathrm{W}}_{\mathrm{ra}}\right)$ and $\left({\mathrm{H}}_{\mathrm{ra}}\right)$ are measured in pixels. Assume that the area of the radar ROI is ${\mathrm{S}}_{\mathrm{ra}}$ as well as ${\mathrm{S}}_{\mathrm{ca}}$ for the camera, as shown in Figure 8. Therefore, the equation for IOU is given below:

$$\mathrm{IOU}={\displaystyle \frac{{\mathrm{S}}_{\mathrm{ra}}\cap {\mathrm{S}}_{\mathrm{ca}}}{{\mathrm{S}}_{\mathrm{ra}}\cup {\mathrm{S}}_{\mathrm{ca}}}}$$

(25)

Then, IOU is used to determine the degree of correlation between objects from the two sensors. The fusion strategy is established as illustrated in

Table 2. Targets detection accuracy in different environments.

Value of IOU	Information on the Bounding Box	Sensor Detection Results	Output
$\mathrm{IOU}\ne 0$	${\mathrm{S}}_{\mathrm{ra}}\ne 0,{\mathrm{S}}_{\mathrm{ca}}\ne 0$	Both radar and camera recognize the same object.	Output the valid information recognized by radar and camera.
$\mathrm{IOU}=0$	${\mathrm{S}}_{\mathrm{ra}}=0,{\mathrm{S}}_{\mathrm{ca}}\ne 0$	Radar misses the target.	Output the valid information detected by camera.
	${\mathrm{S}}_{\mathrm{ra}}\ne 0,{\mathrm{S}}_{\mathrm{ca}}=0$	camera misses the target.	Output the valid information detected by radar.
	${\mathrm{S}}_{\mathrm{ra}}=0,{\mathrm{S}}_{\mathrm{ca}}=0$	There is no target.	There is no information to output.

. Where, radar mainly provides the target’s distance, speed, and angle, while the camera mainly provides the category. In addition, if radar misses while the camera detects the target, the distance is provided by the camera.

3.5. Evaluation Indicators

In this study, we choose Precision, Recall, and Balanced-score as performance evaluation indicators [43]. The formulas are as follows:

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(26)

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(27)

$$\mathrm{Balance}-\mathrm{score}={\displaystyle \frac{(\mathrm{Precision}\times \mathrm{Recall})}{\mathrm{Precision}+\mathrm{Recall}}}\times 2$$

(28)

where $\mathrm{Precision}$ is the correct detection probability. $\mathrm{Recall}$ denotes the detection probability. $\mathrm{Balance}-\mathrm{score}$ is the reconciled average of $\mathrm{Precision}$ and $\mathrm{Recall}$. $\mathrm{TP}$ indicates the quantity of valid objects that are accurately identified. $\mathrm{FP}$ indicates the quantity of invalid objects that are alarmed. $\mathrm{FN}$ indicates the quantity of valid objects that have been missed.

4. Results and Discussion

In this section, we analyze the algorithms’ performance in detail, using roads on campus as experimental scenarios. The camera and radar are mounted on the same tripod, and the height of the camera’s optical center from the ground is $1.28\mathrm{m}$. The positional relationship between the camera and radar are aligned with those outlined in Section 3.3.2. And, the details of tools and environments are provided in

Table 3. Details of tools and environments.

Name	Version	Function
Radar	Texas Instruments (TI) AWR2243	- -
Camera	Hewlett-Packard (HP) 1080p	- -
GPU	NVIDIA GeForce RTX 3060	- -
CPU	i5-11400	- -
Operating system	Windows11	- -
Python	3.8	- -
Pytorch	11.3	- -
CUDA	12.3	- -
Pycharm	2023	Running YOLOs for detecting targets in images
MATLAB	R2022b	Running radar and fusion algorithms

.

4.1. Validation of YOLOv5s Algorithm

To verify that it is reasonable to adopt YOLOv5s as a visual target detection algorithm, we compared it with several mainstream algorithms, as shown in Figure 9. Where all of the models use pre-trained weights, which are provided by the official [44] and trained on the COCO dataset. Compared with YOLOv8s and YOLOv9s, YOLOv5s performs better in parameters (Params), floating-point operations per second (FLOPs), and FPS. Furthermore, while YOLOv5s has slightly fewer Params and FLOPs than YOLOv7-tiny, its AP value improves by 5.5% relative to YOLOv7-tiny. Therefore, we can conclude that YOLOv5s strikes a balance between spatio-temporal complexity and detection accuracy. The experimental outcomes justify the choice of YOLOv5s to recognize objects captured by the camera.

To validate the performance of YOLOv5s model in practical applications, we use it to process images we have captured ourselves. As illustrated in Figure 10, YOLOv5s performs well despite the targets being in harsh environments, such as exposure, low light, long distance, and partial occlusion.

4.2. Validation of SAO-DBSCAN Algorithm

4.2.1. Comparison with Kmeans Algorithm

A tripod mounted with a radar and camera was fixed on the road during experiments. And, three stationary targets to be detected were placed at $(-1.10\mathrm{m},17.40\mathrm{m})$, $(-1.18\mathrm{m},34.40\mathrm{m})$, and $(0.50\mathrm{m},9.40\mathrm{m})$. Then, the information of the objects is pre-processed after collecting them using the radar. Subsequently, different clustering algorithms are used to cluster the radar point clouds, and the results are displayed in Figure 11.

Experiments show that the Kmeans, DBSCAN, and SAO-DBSCAN provide different clustering effects. In Figure 11a, the Kmeans demands the actual quantity of objects to be determined before clustering, but it is impossible for a vehicle-mounted radar. In contrast, the density-based clustering algorithms in Figure 11b,c do not need this information. In the density-based clustering algorithms, clusters are considered to be regions in space that are densely populated with points. On the contrary, low-density points among the clusters are considered as noise. Furthermore, the clusters can take any shape, and the points in the clusters can be distributed arbitrarily. As the density-based clustering algorithms have these features, which are remarkably close to those of targets detected by vehicle-mounted radar, they are preferable for processing data from the vehicle-mounted radar.

4.2.2. Comparison with DBSCAN Algorithm

To demonstrate the robustness of SAO-DBSCAN, we use it to process point clouds with fluctuating density. During the experiments, the targets to be detected are constantly kept stationary, and three frames of data are repeatedly captured by radar. The actual positions of the targets are shown in Figure 12.

Figure 13a, Figure 14a and Figure 15a display the original radar point clouds after pre-processing. Experiments show that the density of the point clouds will change in different frames, even if they come from a same target. Subsequently, we obtain different clusters using the traditional DBSCAN and the proposed SAO-DBSCAN. Where the parameters of DBSCAN are $\epsilon =1$ and $\mathrm{MinPts}=3$, and that of SAO-DBSCAN are $\mathrm{L}=[1,3]$ and $\mathrm{U}=[2,5]$. In Figure 13b, DBSCAN can accurately distinguish the boundaries among different targets with the pre-setting parameters. However, in Figure 14b and Figure 15b, the clustering effects of DBSCAN show apparent defects. On the contrary, in Figure 13c, Figure 14c and Figure 15c, the proposed SAO-DBSCAN can automatically solve the optimal values of $\epsilon$ and $\mathrm{MinPts}$ so as to correctly distinguish each class of targets.

In the following, we place targets with different reflective properties on the road, aiming to verify the ability of SAO-DBSCAN to distinguish among them. The actual locations of targets are shown in Figure 16.

Figure 17a shows the point clouds after pre-processing and filtering. The cars have multiple reflective surfaces, which will cause the point clouds obtained from the same reflective surface to be dense while those among different reflective surfaces will be discrete. Therefore, the same large target is misreported as multiple targets by DBSCAN, as shown by the arrows in Figure 17b. On the contrary, the proposed SAO-DBSCAN can solve this problem perfectly. It can calculate the optimal values of $\epsilon$ and $\mathrm{MinPts}$ according to the reflective properties of the objects so as to classify them correctly. The results can be found in Figure 17c.

In summary, DBSCAN requires pre-setting the specific values of $\epsilon$ and $\mathrm{MinPts}$. When the density of the point clouds fluctuates heavily, its performance will decrease drastically. In contrast, the proposed SAO-DBSCAN only requires a reasonable range of values for the parameters, which are related to the signal reflection performance of the radar. Consequently, it can work in changing scenarios and it exhibits an excellent clustering performance.

4.3. Validation of Information Fusion

4.3.1. Visualization of Information Fusion

Keep the positions of the targets consistent with those depicted in Figure 16 in Section 4.2.2, and collect their information by radar and camera. Then, the targets in the image are extracted using the YOLOv5s model, and the interferences located beyond the lanes are removed. The effective clusters detected by radar are extracted using DBSCAN and SAO-DBSCAN, respectively. Where the clustering results are consistent with those depicted in Figure 17b,c in Section 4.2.2. Subsequently, we calculate the average of the clusters’ coordinates to obtain their exact locations. Finally, the effective targets detected by radar and camera are fused, as illustrated in Figure 18.

Here, the purple bounding boxes indicate that both the two sensors recognize the same objects, and the red boxes indicate that only the radar detects targets. In Figure 18a, there are false severe positives when using DBSCAN to cluster radar point clouds, which will degrade the performance of the decision-level fusion method. On the contrary, the proposed SAO-DBSCAN in Figure 18b can effectively reduce the false alarms from the radar, thus enhancing the accuracy of fusion.

4.3.2. Performance Analysis of Algorithms

For proving the superiority of the fusion model that we suggest, we compare it with other related methods in comparison experiments. Where 199 frames of radar and video data were collected simultaneously for each scenario. Figure 19 demonstrates the details of target movement in various scenarios.

Table 4. Performance of various algorithms.

Detection Scene	Algorithm	Sensors Combination Solutions	Recall	Precision	Balance-Score
Scene 1	DBSCAN	radar	95.24%	90.70%	0.93
	SAO-DBSCAN	radar	99.26%	90.93%	0.95
	Fusion [29]	Camera-radar	99.31%	97.96%	0.98
	Fusion (ours)	Camera-radar	99.08%	97.96%	0.99
Scene 2	DBSCAN	radar	94.87%	66.82%	0.78
	SAO-DBSCAN	radar	93.53%	95.44%	0.94
	Fusion [29]	Camera-radar	99.11%	68.10%	0.81
	Fusion (ours)	Camera-radar	99.11%	95.59%	0.97

displays the performance of various algorithms. Where the parameters of DBSCAN are set to $\epsilon =1$ and $\mathrm{MinPts}=3$, and that of SAO-DBSCAN are set to $\mathrm{L}=[1,3]$ and $\mathrm{U}=[2,5]$.

In Table 4, both of these two fusion methods adopt the decision-level strategy. Additionally, ref. [29] uses DBSCAN to cluster the radar point clouds, whereas the proposed fusion model uses SAO-DBSCAN. In Scene 1, the experimental results indicate that the Balance-score for the suggested SAO-DBSCAN shows a 0.02 improvement over DBSCAN. Simultaneously, the fusion method that we suggest demonstrates a 0.01 enhancement compared to [29]. In Scene 2, the Balance-score of SAO-DBSCAN is improved by 0.16 over DBSCAN, and the fusion method is improved by 0.18 over [29]. Therefore, the proposed fusion method is more accurate than other methods in different scenarios.

In addition, comparing the results of Scene 2 with Scene 1, we notice that the generalizability of DBSCAN is poor. With the same parameters, the Balance-score of DBSCAN changes from 0.93 to 0.78, which has a decrease of 0.15. This is due to the fact that the parameters in DBSCAN have to be pre-set, making it difficult to apply in changing situations. Furthermore, because the accuracy of radar decreases dramatically, the Balance-score of the fusion algorithm [29] changes from 0.98 to 0.81, decreasing by 0.16. Conversely, the fluctuation of the Balance-score for the suggested SAO-DBSCAN is only 0.01, as well as that of the proposed fusion algorithm is only 0.02. Consequently, the experiments demonstrate that the fusion algorithm suggested here is robust.

Finally, we provide a detailed analysis of the time and space complexity for the fusion algorithm, illustrated in

Table 5. Spatio-temporal complexity analysis.

Algorithm	Time Overhead/s	Space Overhead/MB
Fusion [29]	0.94	1291
Fusion (ours)	0.90	1301

. Where 199 frame samples from Scene 1 shown in Figure 19a are used for the evaluation. The experiments indicate that, in comparison with [29], the inferring time for the suggested fusion method rises by just 0.04 s, while the space increases by 10 MB. Therefore, we can conclude that the fusion method suggested here could attain a greater detection accuracy while incurring a slightly larger time and space cost compared to [29].

5. Conclusions

Radar point clouds will experience variations in density, which may cause incorrect alerts during clustering. In turn, it will diminish the precision of the decision-level information fusion method. To address this problem, we suggest a target detection algorithm based on a fusing radar with a camera in the presence of a fluctuating signal intensity. Firstly, we introduce SAO to solve the optimal parameters of DBSCAN, and use the proposed SAO-DBSCAN to cluster the radar point clouds. Then, the YOLOv5s model is employed to identify the objects on the image. Finally, the information of objects detected by radar and vision is fused. The experimental results indicate that the suggested fusion method can attain a Balance-score ranging from 0.97 to 0.99, performing outstandingly in preventing missed detections and false alarms. Additionally, the Balance-score fluctuates within 0.02, indicating that the fusion method is robust. However, it still suffers from the disadvantages of a high spatio-temporal complexity. Therefore, in subsequent endeavors, we will try our best to reduce the time and space overhead while achieving a higher detection accuracy.