A Robust Target Detection Algorithm Based on the Fusion of Frequency-Modulated Continuous Wave Radar and a Monocular Camera

Abstract

Decision-level information fusion methods using radar and vision usually suffer from low target matching success rates and imprecise multi-target detection accuracy. Therefore, a robust target detection algorithm based on the fusion of frequency-modulated continuous wave (FMCW) radar and a monocular camera is proposed to address these issues in this paper. Firstly, a lane detection algorithm is used to process the image to obtain lane information. Then, two-dimensional fast Fourier transform (2D-FFT), constant false alarm rate (CFAR), and density-based spatial clustering of applications with noise (DBSCAN) are used to process the radar data. Furthermore, the YOLOv5 algorithm is used to process the image. In addition, the lane lines are utilized to filter out the interference targets from outside lanes. Finally, multi-sensor information fusion is performed for targets in the same lane. Experiments show that the balanced score of the proposed algorithm can reach 0.98, which indicates that it has low false and missed detections. Additionally, the balanced score is almost unchanged in different environments, proving that the algorithm is robust.

1. Introduction

According to surveys, around 1.35 million fatalities and 50 million injuries occur globally each year due to car accidents [1,2]. The quality of vehicle hardware is undoubtedly the foundation of safe driving. However, drivers’ inexperience and inattention are the leading causes of car accidents compared to the current mature and reliable technology for producing vehicle hardware [3]. On the contrary, there is no such thing as inattention, fatigued driving, drunk driving, or operating errors with automated driving. Therefore, the development of autonomous driving to reduce collisions has become a focus of discussion in various fields. As one of the core hardware for realizing autonomous driving, vehicle sensors can provide rich environment sensing information like the human sensory system [4]. So, they can be used to reduce traffic accidents caused by drivers. Radar, lidar, and cameras, as several sensors widely used in self-driving cars, are often combined to provide more comprehensive target information and more accurate detection results than a single sensor [5]. The more mainstream approaches to combining sensors include fusing information from radar and cameras, lidar and cameras, or all three [6]. Lidar has two significant drawbacks, namely that it costs more than other sensors and has low reliability in complex environments [7]. Even though the current price of lidar has shown a downward trend, it is still restricted by factors such as lighting conditions, extreme weather, and electromagnetic disturbance [8,9,10]. Cameras enable object detection and semantic understanding, which are impossible with lidar. For instance, the data obtained through a camera can be utilized to identify traffic signs and gesticulations [11,12]. Radar can accurately obtain information on the range, direction, and velocity of an object [13,14], which are challenging to obtain with a camera. Therefore, approaches that integrate radar and camera data have become prevalent in both research and commercial applications [15].

In decision-level information fusion, the ranges of interest (ROIs) of different objects usually overlay each other, which may lead to false alarms. In [16], when the radar ROIs of the same target overlap, the authors use the most extensive box as the preferred one and then calculate the average ROI from two ROIs with similar sizes and bottoms. However, this method can only address cases where the ROIs of targets detected by the same sensor overlap. Additionally, radar often makes false detections so that irrelevant targets are considered valid [17]. In order to eliminate spurious detections, the density-based spatial clustering of applications with noise (DBSCAN) method is employed for grouping radar point clouds [18]. Experiments have shown that many radar misdetections can be eliminated by using this additional clustering method.

The visual target detection parts of the above fusion algorithms use a traditional hand-designed feature approach. This approach usually requires a priori knowledge to recognize targets on the road [19]. Nevertheless, the complexity and variability of scenarios will make target detection based on a priori appearance challenging. Unlike traditional manual feature design methods, deep learning methods can automatically learn the features of targets. Meanwhile, deep learning methods unify feature extraction and classifier learning in a single framework and are capable of end-to-end learning. Therefore, they have received extensive attention in recent years [20,21,22].

The visual detection component in [23] uses Faster R-CNN [24] to detect targets. Although this method can achieve high accuracy without prior knowledge, the two-stage detection algorithm is too slow to use in practical engineering projects. In [25], a technique for detecting objects that combines millimeter wave radar and optical sensors is introduced. In this publication, the YOLOv4 algorithm is utilized for real-time object detection. This algorithm boasts the ability to detect targets promptly. However, the architecture of YOLOv4 is intricate, and it has a substantial model footprint. So, there are still high requirements for computing capability and device storage resources.

Compared to using a single sensor for target detection, all of these fusion methods mentioned above are able to obtain better detection performance. However, these methods still suffer from the disadvantages of low detection accuracy, insufficient system robustness, and high computational cost. In particular, when the ROIs between different targets detected by different sensors overlap, the decision-level information fusion methods will suffer from the problems of a low target matching success rate and imprecise multi-target detection accuracy. In order to address these problems, a robust target detection algorithm based on the fusion of frequency-modulated continuous wave (FMCW) radar and a monocular camera is proposed in this paper.

Firstly, the method employs a reliable lane marker-based detection algorithm [26] to process the image and extract lane information. Then, the two-dimensional fast Fourier transform (2D-FFT), constant false alarm rate (CFAR), Angle-FFT, and DBSCAN algorithms are applied to pre-process the radar measurements. Furthermore, through the exclusion of null targets and the establishment of lane boundary thresholds, the valid targets are selected from the radar detections. Next, YOLOv5 [27] is used to extract the targets from the visual data. Immediately, the targets identified as valid by both vision and radar are sorted into their respective lanes. Ultimately, data from both the radar and camera are integrated within the same lane by aligning their spatial and temporal dimensions. This is achieved through the calibration of the camera, transformation of coordinate systems, and synchronization of sensor sampling instances. The experimental results indicate that this technique not only successfully alleviates the constraints associated with relying on a single sensor for sensing, but also bolsters the precision of information fusion. The primary contributions of the proposed methodology can be encapsulated in the subsequent points:

(1) We introduce a lane detection algorithm to filter out interfering targets outside the lanes, reducing the amount of data processing in the subsequent fusion algorithm;

(2) The positions of the detected lane lines are used as the effective detection ranges of targets instead of the traditional manual setting method, which improves the universality of the algorithm;

(3) We use the YOLOv5s model to detect targets in images and fuse the information of targets with the same lane markings from different sensors. This method can reduce the mutual interference between different targets in neighboring lanes, improving the fusion algorithm’s accuracy.

The other parts of this paper are summarized below. Section 2 examines the information fusion framework. We present some critical steps and algorithms for decision-level information fusion in Section 3. Section 4 collects data from realistic environments to validate the algorithm proposed in this paper. Section 5 gives conclusions based on the experimental results.

2. Information Fusion Framework

The framework for information fusion at the decision level [28,29] used in this paper is shown in Figure 1.

The fusion framework in this paper consists of three parts: visual target detection, radar target detection, and decision-level information fusion. In the visual target detection part, the images are first processed using a lane detection algorithm to obtain lane information. Then, target detection is performed using YOLOv5 and a ranging algorithm based on data fitting [30] to obtain the type and distance of the target. In the radar target detection part, algorithms such as 2D-FFT [31], CFAR [32], and Angle-FFT are first executed to obtain the original point clouds detected by radar. In accordance, the point clouds are clustered by the DBSCAN algorithm [33]. Then, the lanes are utilized to filter out the interfering targets located outside the lanes and group the remaining valid targets. In the information fusion part, targets from the two sensors in the same lane are first spatiotemporally calibrated. Then, the intersection over union (IOU) method is used to match the targets from different sensors. Finally, corresponding fusion strategies are developed based on the matching results.

3. Realization of Fusion

3.1. FMCW Radar Equation

Figure 2 presents the principle of the frequency-modulated continuous wave (FMCW) radar Equation [34]. Its fundamental ideology is to construct periodic frequency slopes, which are proportional to the modulation time. Supposing that the frequency modulation range is $\mathrm{B}$ and the duration is ${\mathrm{T}}_{\mathrm{chirp}}$, the expression for slope ${\mathrm{f}}_{\mathrm{T}}\left(\mathrm{t}\right)$ is derived:

$${\mathrm{f}}_{\mathrm{T}}\left(\mathrm{t}\right)={\mathrm{f}}_0+\frac{\mathrm{B}}{{\mathrm{T}}_{\mathrm{chirp}}}\mathrm{t}$$

(1)

Here, ${\mathrm{f}}_0$ denotes the initial frequency of the signal, while ${\mathrm{T}}_{\mathrm{chirp}}$ takes a value within the range of $[-{\mathrm{T}}_{\mathrm{c}}/2,{\mathrm{T}}_{\mathrm{c}}/2]$. Assuming that the transmitted signal is ${\mathrm{S}}_{\mathrm{T}}\left(\mathrm{t}\right)=cos\left({\psi }_{\mathrm{T}}\left(\mathrm{t}\right)\right)$, the phase ${\psi }_{\mathrm{T}}\left(\mathrm{t}\right)$ can be calculated by integrating Equation (1):

$$\begin{array}{cc}\hfill {\psi }_T\left(\mathrm{t}\right)& =2\pi {\int }_{-{\mathrm{T}}_{\mathrm{c}}/2}^{\mathrm{t}}{\mathrm{f}}_{\mathrm{T}}\left(\mathrm{t}\right)\mathrm{dt}\hfill \\ & =2\pi {\int }_{-{\mathrm{T}}_{\mathrm{c}}/2}^{\mathrm{t}}({\mathrm{f}}_0+\frac{\mathrm{B}}{{\mathrm{T}}_{\mathrm{chrip}}}\mathrm{t})\mathrm{dt}\hfill \\ & =2\pi \left({\mathrm{f}}_0\mathrm{t}+\frac{1}{2}\frac{\mathrm{B}}{{\mathrm{T}}_{\mathrm{chirp}}}{\mathrm{t}}^2\right)-{\psi }_{\mathrm{T}0}\hfill \end{array}$$

(2)

where the value of ${\psi }_{\mathrm{T}0}$ is ${\psi }_{\mathrm{T}0}=\pi {(\mathrm{f}}_0{\mathrm{T}}_{\mathrm{c}}-\frac{1}{4}\frac{{\mathrm{T}}_{\mathrm{c}}^2}{{\mathrm{T}}_{\mathrm{chirp}}}\mathrm{B})$. Phase $\mathsf{\Delta }\psi \left(\mathrm{t}\right)={\psi }_{\mathrm{T}}\left(\mathrm{t}\right)-{\psi }_{\mathrm{T}}(\mathrm{t}-\tau )$ of the down-converted signal is

$$\mathsf{\Delta }\psi \left(\mathrm{t}\right)=2\pi \left({\mathrm{f}}_0\tau +\frac{\mathrm{B}}{{\mathrm{T}}_{\mathrm{chirp}}}\mathrm{t}\tau -\frac{1}{2}\frac{\mathrm{B}}{{\mathrm{T}}_{\mathrm{chirp}}}{\tau }^2\right)$$

(3)

Here, $\tau$ represents the duration required for the emitted signal to contact the target and reflect. The final entry in Equation (3) is negligible since $\tau /{\mathrm{T}}_{\mathrm{chirp}}$ takes on a value much less than 1. Suppose the target’s distance and speed are $\mathrm{Y}$ and $\mathrm{V}$, respectively. From this, we can deduce that $\tau$ is given by $\tau =2\left(\mathrm{Y}+\mathrm{Vt}\right)/\mathrm{c}$. Bringing $\tau$ into Equation (3) would lead to

$$\mathsf{\Delta }\psi \left(t\right)=2\pi \left(\frac{2{\mathrm{f}}_0\mathrm{Y}}{\mathrm{c}}+\left(\frac{2{\mathrm{f}}_0\mathrm{V}}{\mathrm{c}}+\frac{2\mathrm{BY}}{{\mathrm{cT}}_{\mathrm{chirp}}}\right)\mathrm{t}+\frac{2\mathrm{BV}}{{\mathrm{cT}}_{\mathrm{chirp}}}{\mathrm{t}}^2\right)$$

(4)

In the equation above, $\mathrm{c}$ is the velocity of light. The last term in Equation (4), called the distance Doppler coupling [35], is negligible:

$$\mathsf{\Delta }\psi \left(\mathrm{t}\right)=2\pi \left(\frac{2{\mathrm{f}}_0\mathrm{Y}}{\mathrm{c}}+\left(\frac{2{\mathrm{f}}_0\mathrm{V}}{\mathrm{c}}+\frac{2\mathrm{BY}}{{\mathrm{cT}}_{\mathrm{chirp}}}\right)\mathrm{t}\right)$$

(5)

The down-converted signal, which is also known as the intermediate frequency (IF) signal, is

$$\begin{array}{cc}\hfill {\mathrm{S}}_{\mathrm{IF}}\left(\mathrm{t}\right)& =cos\left(\mathsf{\Delta }\psi \right(t\left)\right)\hfill \\ & =cos(2\pi (\frac{2{\mathrm{f}}_0\mathrm{Y}}{\mathrm{c}}+(\frac{2{\mathrm{f}}_0\mathrm{V}}{\mathrm{c}}+\frac{2\mathrm{BY}}{\mathrm{c}{\mathrm{T}}_{\mathrm{chirp}}})\mathrm{t}))\hfill \end{array}$$

(6)

The sampling frequency of ${\mathrm{S}}_{\mathrm{IF}}\left(\mathrm{t}\right)$ is set to ${\mathrm{f}}_{\mathrm{s}}=1/{\mathrm{T}}_{\mathrm{s}}$. Then, the complex form of the IF signal is obtained using the Hilbert Transform [36], which will be used to estimate information about the target:

$$\begin{array}{c}\hfill \mathrm{S}\left(\mathrm{n}\right)={\mathrm{e}}^{\mathrm{i}·2\mathsf{\pi }\left(\frac{2{\mathrm{f}}_0\mathrm{Y}}{\mathrm{c}}+\left(\frac{2{\mathrm{f}}_0\mathrm{V}}{\mathrm{c}}+\frac{2\mathrm{BY}}{{\mathrm{cT}}_{\mathrm{chirp}}}\right){\mathrm{T}}_{\mathrm{s}}\mathrm{n}\right)}\end{array}$$

(7)

3.2. Video Data Pre-Processing

3.2.1. Lane Detection

In this section, an algorithm for detecting lanes that is both robust and efficient is employed, which relies on dependable lane markings. The schematic representation of the algorithm is illustrated in Figure 3.

The lane detection algorithm is specifically formulated in the following four steps:

(1) To exploit the parallelism of the channels, the raw image is converted into a bird’s eye view (BEV) by the inverse perspective mapping (IPM) transform, as shown in Figure 4a.

(2) The RGB image is represented as a mapping, ${\mathrm{I}}_{\mathrm{RGB}}:\mathrm{E}\times \mathrm{F}\times \{\mathrm{R},\mathrm{G},\mathrm{B}\}\to [0,1]$. ${\mathrm{I}}_{\mathrm{RGB}}(\mathrm{i},\mathrm{j},\mathrm{u})$ denotes the pixel value of the RGB image for row $\mathrm{i}$, column $\mathrm{j}$, and color $\mathrm{u}$. Here, $\mathrm{E}$ and $\mathrm{F}$ are the set of rows and columns, respectively. The greyscale image is represented as the mapping ${\mathrm{I}}_{\mathrm{Grey}}:\mathrm{E}\times \mathrm{F}\to \left[01\right]$. ${\mathrm{I}}_{\mathrm{Grey}}(\mathrm{i},\mathrm{j})$ represents the pixel intensity at the $\mathrm{ith}$ row and $\mathrm{jth}$ column of the greyscale image. Figure 4b illustrates the conversion of an RGB image to a greyscale image utilizing Equation (8):

$${\mathrm{I}}_{\mathrm{Grey}}(\mathrm{i},\mathrm{j})=0.299{\mathrm{I}}_{\mathrm{RGB}}(\mathrm{i},\mathrm{j},\mathrm{R})+0.587{\mathrm{I}}_{\mathrm{RGB}}(\mathrm{i},\mathrm{j},\mathrm{G})+0.114{\mathrm{I}}_{\mathrm{RGB}}(\mathrm{i},\mathrm{j},\mathrm{B})$$

(8)

After that, in order to reduce the amount of image processing and concentrate on lane marking, the ROI of the lane to be detected is selected from the greyscale image, as shown by the yellow bounding box in Figure 4c. Then, the transformation formula of binary image ${\mathrm{I}}_{\mathrm{Bin}}:\mathrm{E}\times \mathrm{F}\to \{0,1\}$ is

$${\mathrm{I}}_{\mathrm{Bin}}(\mathrm{i},\mathrm{j})=\left\{\begin{array}{c}1,\mathrm{if}{\mathrm{I}}_{\mathrm{Grey}}(\mathrm{i},\mathrm{j})\ge \mathrm{a}\\ 0,\mathrm{if}{\mathrm{I}}_{\mathrm{Grey}}(\mathrm{i},\mathrm{j})<\mathrm{a}\end{array}\right.$$

(9)

Here, the range of threshold a is $[0,1]$. Two binary images, ${\mathrm{I}}_{\mathrm{Bin}1}$ and ${\mathrm{I}}_{\mathrm{Bin}2}$, are obtained from the greyscale image ${\mathrm{I}}_{\mathrm{Grey}}$ by utilizing Equation (9). Subsequently, the binary images are merged using Equation (10):

$$\mathrm{I}(\mathrm{i},\mathrm{j})=\left\{\begin{array}{cc}0,\mathrm{if}\left(\sum _{\mathrm{m}=\mathrm{i}-\mathrm{h}}^{\mathrm{i}+\mathrm{h}}\sum _{\mathrm{n}=\mathrm{j}-\mathrm{h}}^{\mathrm{j}+\mathrm{h}}{\mathrm{I}}_{\mathrm{Bin}1}\left(\mathrm{m},\mathrm{n}\right)\right)\left(\sum _{\mathrm{m}=\mathrm{i}-\mathrm{h}}^{\mathrm{i}+\mathrm{h}}\sum _{\mathrm{n}=\mathrm{j}-\mathrm{h}}^{\mathrm{j}+\mathrm{h}}{\mathrm{I}}_{\mathrm{Bin}2}\left(\mathrm{m},\mathrm{n}\right)\right)=0\hfill & \\ & \\ 1,\mathrm{otherwise}\hfill \end{array}\right.$$

(10)

Here, $\mathrm{h}$ is called the proximity parameter. With these manipulations described above, we obtain the merged binary image I, as shown in Figure 4d. The white parts indicate lane markings, whose element value is 1. Furthermore, all other non-lane portions of the image are black with an element value of 0. At this point, we have completed the feature extraction of the lane lines.

(3) To determine the binary BEV used for lane detection, Figure 5 plots the histograms generated from the binary image $\mathrm{I}$. The histograms are separated into two regions centered on the vehicle. Since the closeness between lane pixel values is crucial for effective lane detection, it is necessary to calculate the standard deviation of histograms.

$${\mu }_l=\frac{2}{\mathrm{F}}\sum _{j=1}^{\mathrm{F}/2}\sum _{i=1}^{\mathrm{E}}\mathrm{j}·\mathrm{I}(\mathrm{i},\mathrm{j});{\sigma }_l=\sqrt{\frac{2}{\mathrm{F}}{\displaystyle \sum _{j=1}^{\mathrm{F}/2}}{\displaystyle \sum _{i=1}^{\mathrm{E}}}\mathrm{I}\left(\mathrm{i},\mathrm{j}\right){\left(\mathrm{j}-{\mu }_l\right)}^2}$$

(11)

$${\mu }_r=\frac{2}{\mathrm{F}}\sum _{j=\mathrm{F}/2+1}^{\mathrm{F}}\sum _{i=1}^{\mathrm{E}}\mathrm{j}·\mathrm{I}(\mathrm{i},\mathrm{j});{\sigma }_r=\sqrt{\frac{2}{\mathrm{F}}{\displaystyle \sum _{j=\mathrm{F}/2+1}^{\mathrm{F}}}{\displaystyle \sum _{i=1}^{\mathrm{E}}}\mathrm{I}\left(\mathrm{i},\mathrm{j}\right){\left(\mathrm{j}-{\mu }_r\right)}^2}$$

(12)

Here, ${\mu }_l$ and ${\mu }_r$ represent the mean values for the left/right areas of the histogram, respectively, while ${\sigma }_l$ and ${\sigma }_r$ denote the standard deviation.

The following criteria are used to determine which area to select for lane detection:

$$\left\{\begin{array}{cc}& \hfill \mathrm{Choose}\mathrm{left}\mathrm{region},\mathrm{if}(\lambda {\mathrm{P}}_{\mathrm{l}}-{\sigma }_l)\ge (\lambda {\mathrm{P}}_{\mathrm{r}}-{\sigma }_r)\hfill \\ & \hfill \mathrm{Choose}\mathrm{right}\mathrm{region},\mathrm{if}\left(\lambda {\mathrm{P}}_{\mathrm{l}}-{\sigma }_l\right)<(\lambda {\mathrm{P}}_{\mathrm{r}}-{\sigma }_r)\hfill \end{array}\right.$$

(13)

where ${\mathrm{P}}_{\mathrm{l}}$ and ${\mathrm{P}}_{\mathrm{r}}$ denote the maximum value within the histogram’s left and right segments. Furthermore, $\lambda$ is a balancing factor between the maximum value and standard deviation. Eventually, the positions of ${\mathrm{P}}_{\mathrm{l}}$ and ${\mathrm{P}}_{\mathrm{r}}$ on the column vector in the histogram are solved, which are the reference positions of carriageway markings used for testing.

(4) To precisely ascertain the shape of the lane pixels, we scan the binary image I line by line along the column indexes of ${\mathrm{P}}_{\mathrm{l}}$ or ${\mathrm{P}}_{\mathrm{r}}$. This step is shown in Figure 6. We control the size of the yellow rectangular box to be fixed and scan it from small to large along the Y-axis. Then, a second-order polynomial is used as the lane model:

$${\mathrm{pi}}^2+\mathrm{qi}+\mathrm{c}=\mathrm{j}$$

(14)

Considering the $\mathrm{F}$ measurements covered by the yellow rectangle as $\{({\mathrm{i}}_1,{\mathrm{j}}_1),({\mathrm{i}}_2,{\mathrm{j}}_2),\dots ,({\mathrm{i}}_{\mathrm{F}},{\mathrm{j}}_{\mathrm{F}})\}$, the least square estimate (LSE) algorithm is applied to all pixels overlaid by the rectangle. As a result, the parameters $\mathrm{p}$, $\mathrm{q}$, and $\mathrm{c}$ of the polynomial in the lane to be detected are calculated:

$$\overline{x}={\left(B^{T}B\right)}^{-1}{B}^T\overline{J}$$

(15)

$$\mathrm{B}=\left[\begin{array}{ccc}{\mathrm{i}}_1^2& {\mathrm{i}}_1& 1\\ {\mathrm{i}}_2^2& {\mathrm{i}}_2& 1\\ ⋮& ⋮& ⋮\\ {\mathrm{i}}_{\mathrm{F}}^2& {\mathrm{i}}_{\mathrm{F}}& 1\end{array}\right],\overline{x}=\left[\begin{array}{c}\mathrm{p}\\ \mathrm{q}\\ \mathrm{c}\end{array}\right],\overline{J}=\left[\begin{array}{c}{\mathrm{j}}_1\\ {\mathrm{j}}_2\\ ⋮\\ {\mathrm{j}}_{\mathrm{F}}\end{array}\right]$$

(16)

In this way, the square of the error vector is minimized:

$$\mathrm{e} = {\parallel \mathrm{B}\overline{x}-\overline{J}\parallel }_2^2$$

(17)

Subsequently, the lane polynomial to be computed is moved to the peak column position in another area to extract the subsequent lane marker.

3.2.2. YOLOv5 Target Detection Algorithm

YOLOv5 [37] is a one-stage target detection algorithm, which is classified into five versions: n, s, m, l, and x. Considering the model’s detection speed, accuracy, and size, YOLOv5s is chosen in this paper. Its structure is shown in Figure 7. YOLOv5 mainly consists of the backbone, neck, and head. The backbone network mainly consists of the focus structure, the cross-stage partial network (CSPNet), and the spatial pyramid pooling (SPP) structure. The focus structure adopts the slicing operation to reduce the loss of information caused by the downsampling. The CSPNet can improve the learning ability of the model and reduce the amount of data transfer and computation of the network. Furthermore, the SPP uses maximum pooling at different scales to improve the perceptual field of view. The neck network uses the pyramid attention network (PAN) to efficiently fuse semantic features by combining top-down and down-top approaches. The head network is used to detect objects at different scales and match the optimal target bounding box by non-maximal suppression (NMS).

After obtaining the bounding box of a target making use of the YOLOv5s algorithm, the pixel occupied by the target in the image will be used to estimate its actual distance. The traditional small-hole imaging method [38] for distance detection requires prior knowledge of the height of a target. However, determining the height of an object using a monocular camera is challenging. There is a data fitting algorithm that can be used to estimate the actual distance to the target. The basic principle of this method is to place the target 3.6 m directly in front of the camera and take a shot every 0.6 m of movement. Figure 8 shows a portion of images used to solve for distance.

Figure 9 illustrates the result of obtaining a distance fitting curve using a rational function.

The vertical coordinate $\mathrm{d}$ represents the practical distance of targets. Furthermore, the horizontal coordinate ${\mathrm{V}}_{\mathrm{pixel}}$ represents the vertical pixel positions, which are located at the center of the underside of bounding boxes. Eventually, the relationship between ${\mathrm{V}}_{\mathrm{pixel}}$ and $\mathrm{d}$ is

$$\mathrm{d}=\frac{2603}{{\mathrm{V}}_{\mathrm{pixel}}-698.2}$$

(18)

3.3. Spatiotemporal Calibration

3.3.1. Temporal Calibration

The millimeter wave radar was set to acquire 10 frames per second, while the camera was at 30 frames per second. Then, according to the relationship between the data acquisition frequencies of the two sensors, the data from the radar were processed frame by frame, and the data from the camera were processed every three frames. By performing this, we can make the data acquisition frequency of both sensors consistent. Finally, assuming that the camera is activated earlier than the radar by $\mathsf{\Delta }\tau$ seconds, the corresponding number of frames for early activation is obtained as

$$\mathrm{frameVid}=\mathsf{\Delta }\tau ·{\mathrm{V}}_{\mathrm{fps}}$$

(19)

Here, ${\mathrm{V}}_{\mathrm{fps}}$ is the data acquisition frame rate of the camera, whose value is 30 fps. Thereby, the first frame of data acquired by the radar is temporally aligned with the frame frameVid of the camera, as shown in Figure 10.

3.3.2. Spatial Calibration

Since the fusion is performed on a pixel coordinate system, the targets detected by the radar are projected onto the pixel coordinate system, which is called spatial calibration between multiple sensors [39,40]. Suppose $\mathrm{Q}$ is the target to be detected, and its coordinates in the world, radar, and camera are $({\mathrm{X}}_{\mathrm{w}},{\mathrm{Y}}_{\mathrm{w}},{\mathrm{Z}}_{\mathrm{w}})$, $({\mathrm{X}}_{\mathrm{r}},{\mathrm{Y}}_{\mathrm{r}},{\mathrm{Z}}_{\mathrm{r}})$, and $({\mathrm{X}}_{\mathrm{c}},{\mathrm{Y}}_{\mathrm{c}},{\mathrm{Z}}_{\mathrm{c}})$, respectively. To simplify the derivation process, it is assumed that the camera and world share the same reference coordinate system. Then, the position of point $\mathrm{Q}$ in each coordinate system is shown in Figure 11a. Using $\mathrm{R}$ and $\mathrm{T}$ to denote the angular rotation and translation information between the two sensors, respectively, the target detected by the radar could be transformed into the camera coordinate system using the following equation:

$$\left[\begin{array}{c}\mathrm{Xc}\\ \mathrm{Yc}\\ \mathrm{Zc}\\ 1\end{array}\right]=\left[\begin{array}{cc}\mathrm{R}& \mathrm{T}\\ 0^{\mathrm{T}}& 1\end{array}\right]\left[\begin{array}{c}\mathrm{Xr}\\ \mathrm{Yr}\\ \mathrm{Zr}\\ 1\end{array}\right]$$

(20)

Using the principle of similar triangles, the point $\mathrm{Q}({\mathrm{X}}_{\mathrm{c}},{\mathrm{Y}}_{\mathrm{c}},{\mathrm{Z}}_{\mathrm{c}})$ in Figure 11b is projected onto $\mathrm{p}(\mathrm{x},\mathrm{y})$, where o-xy represents the image coordinate system.

$${\mathrm{Z}}_{\mathrm{c}}\left[\begin{array}{c}\mathrm{x}\\ \mathrm{y}\\ 1\end{array}\right]=\left[\begin{array}{cccc}\mathrm{f}& 0& 0& 0\\ 0& \mathrm{f}& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}\mathrm{Xc}\\ \mathrm{Yc}\\ \mathrm{Zc}\\ 1\end{array}\right]$$

(21)

Here, $\mathrm{f}$ denotes the focal length of the optical sensor used in this study. The computer stores image data using a pixel-based coordinate system that takes the point ${\mathrm{o}}_0$ in the shaded portion of Figure 11b as the starting point. Suppose there is a pixel point $({\mathrm{u}}_0,{\mathrm{v}}_0)$ that corresponds to the home point of the image coordinate system. The transformation relationship from image coordinate to pixel is

$$\left[\begin{array}{c}\mathrm{u}\\ \mathrm{v}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{\displaystyle \frac{1}{\mathrm{dx}}}& 0& {\mathrm{u}}_0\\ 0& {\displaystyle \frac{1}{\mathrm{dy}}}& {\nu }_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}\mathrm{x}\\ \mathrm{y}\\ 1\end{array}\right]$$

(22)

In the above equation, $\mathrm{dx}$ and $\mathrm{dy}$ denote the pixel density along the horizontal and vertical axes in the image coordinate system, respectively. Ultimately, the conversion formula that links the radar coordinate to the pixel is presented as shown in the following mathematical expression:

$${\mathrm{Z}}_{\mathrm{c}}\left[\begin{array}{c}\mathrm{u}\\ \mathrm{v}\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{\mathrm{dx}}& 0& {\mathrm{u}}_0\\ 0& \frac{1}{\mathrm{dy}}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}\mathrm{f}& 0& 0& 0\\ 0& \mathrm{f}& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}\mathrm{R}& \mathrm{T}\\ 0^{\mathrm{T}}& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{r}}\\ {\mathrm{Y}}_{\mathrm{r}}\\ {\mathrm{Z}}_{\mathrm{r}}\\ 1\end{array}\right]={\mathrm{N}}_1{\mathrm{N}}_2\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{r}}\\ {\mathrm{Y}}_{\mathrm{r}}\\ {\mathrm{Z}}_{\mathrm{r}}\\ 1\end{array}\right]$$

(23)

$${\mathrm{N}}_1=\left[\begin{array}{cccc}\mathrm{f}/\mathrm{dx}& 0& {\mathrm{u}}_0& 0\\ 0& \mathrm{f}/\mathrm{dy}& {\mathrm{v}}_0& 0\\ 0& 0& 1& 0\end{array}\right]$$

(24)

$${\mathrm{N}}_2=\left[\begin{array}{cc}\mathrm{R}& \mathrm{T}\\ 0^{\mathrm{T}}& 1\end{array}\right]$$

(25)

The elements of matrix ${\mathrm{N}}_1$ are associated with the camera’s intrinsic properties, which are determinable through a calibration technique involving a checkerboard pattern. The elements of matrix ${\mathrm{N}}_2$ are determined by the mounting position of the camera, which can be obtained by measuring the spatial arrangement relative to the radar.

In this paper, we employ a checkerboard calibration board with $10 \times 8$ squares to adjust the intrinsic properties of camera. Here, the side length of each square is $20mm \times 20mm$. Firstly, 20 pictures of the checkerboard grid are taken from different angles using a camera, as shown in Figure 12a. Then, the corner points in the checkerboard grid pictures are extracted and corrected using the calibration toolbox. The findings are presented in Figure 12b. Where the arrows indicate the direction of increase in the X and Y axes.

Ultimately, the obtained internal parameter matrix ${\mathrm{N}}_1$ of camera is

$${\mathrm{N}}_1=\left[\begin{array}{cccc}2032.1& 0& 1338& 0\\ 0& 2033.2& 741.8& 0\\ 0& 0& 1& 0\end{array}\right]$$

(26)

The camera’s external parameter calibration determines the values of $\mathrm{R}$ and $\mathrm{T}$. Figure 13 depicts the placement of the sensors in relation to each other, where the camera is encased in a black box. The relative positions between the two sensors are adjusted so that the pitch and azimuth deflections are approximated to 0°. Therefore, the size of the rotation matrix between the two sensors is $\mathrm{R}=\mathrm{I}$. Then, the Z-axis offset between the center of the camera and the radar is measured to be $\mathsf{\Delta }\mathrm{Z}=170\mathrm{mm}$, the Y-axis offset is $\mathsf{\Delta }\mathrm{Y}=25\mathrm{mm}$, and the X-axis offset is 0 mm. The value of the translation matrix is thus obtained as $\mathrm{T}=[0-25{170]}^T$. Based on $\mathrm{R}$ and $\mathrm{T}$, the camera’s external parameter matrix ${\mathrm{N}}_2$ is expressed as

$${\mathrm{N}}_2=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& -25\\ 0& 0& 1& 170\\ 0& 0& 0& 1\end{array}\right]$$

(27)

3.4. Target Matching

In order to make consistent judgments on the targets detected by the radar and camera, it is necessary to generate the corresponding vehicle ROI based on each pixel point mapped by the radar in the image [41]. The vehicle’s external dimensions specify that the aspect ratio of the vehicle’s rear should range from 0.7 to 1.3. In this paper, the width of the vehicle is set to be 2.4 m, and the aspect ratio is taken to be 1.2. The formula to establish the pixel width (W) and height (H) of the target projection point detected by radar in the image is finally obtained as follows:

$$\left\{\begin{array}{cc}\hfill \mathrm{W}=2.4\times \frac{\alpha \mathrm{f}}{\mathrm{dx}·{\mathrm{Y}}_{\mathrm{r}}}\hfill & \\ & \\ \hfill \mathrm{H}=2.0\times \frac{\alpha \mathrm{f}}{\mathrm{dx}·{\mathrm{Y}}_{\mathrm{r}}}\hfill \end{array}\right.$$

(28)

Here, $\alpha$ is the ratio between the physical and pixel size.

In Figure 14, the red rectangle indicates the radar ROI with an area of ${\mathrm{S}}_{\mathrm{R}}$, and the green rectangle indicates the visual ROI with an area of ${\mathrm{S}}_{\mathrm{V}}$. In this manner, the IOU can be determined using the following equation:

$$\mathrm{IOU}=\frac{{\mathrm{S}}_{\mathrm{V}}\cap {\mathrm{S}}_{\mathrm{R}}}{{\mathrm{S}}_{\mathrm{V}}\cup {\mathrm{S}}_{\mathrm{R}}}$$

(29)

Then, target matching is performed based on the value of the IOU, and the corresponding fusion strategies are formulated based on the matching results. The specific fusion schemes are as follows:

If the value of the IOU is in the range $(0,1]$, it can be judged that the two sensors detect the same target. Considering the respective detection advantages of the two sensors, the distance, speed, and angle detected by the radar, as well as the category, bounding box, and lane detected by the camera, are output. If $\mathrm{IOU}=0$, it means that there is no overlap between the radar and vision bounding boxes. In this condition, it is essential to determine whether the values of ${\mathrm{S}}_{\mathrm{V}}$ and ${\mathrm{S}}_{\mathrm{R}}$ are 0. If ${\mathrm{S}}_{\mathrm{R}}=0$ and ${\mathrm{S}}_{\mathrm{V}}\ne 0$, it means that the radar fails to detect the target. At the same time, if the confidence score of the target detected by the camera is greater than or equal to 0.50, the camera is judged to have detected a valid target. Thus, the information detected by the camera is output. If ${\mathrm{S}}_{\mathrm{R}}\ne 0$ and ${\mathrm{S}}_{\mathrm{V}}=0$, it means that camera fails to detect the target. Then, target tracking is performed to determine if the radar detects a valid target. If it is a valid target, the information detected by the radar is output. Otherwise, the invalid target is discarded.

3.5. Evaluation Indicators

This paper evaluates the algorithm’s performance using precision, recall, and F1 score. The F1 score is also known as the balanced score, which takes into account both the precision and recall of the algorithm. The specific formulas are as follows:

$$\mathrm{precision}=\frac{{\mathrm{T}}_{\mathrm{P}}}{{\mathrm{T}}_{\mathrm{P}}+{\mathrm{F}}_{\mathrm{P}}}$$

(30)

$$\mathrm{recall}=\frac{{\mathrm{T}}_{\mathrm{P}}}{{\mathrm{T}}_{\mathrm{P}}+{\mathrm{F}}_{\mathrm{N}}}$$

(31)

$$F1=2\times \frac{\mathrm{precision}\times \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}$$

(32)

where ${\mathrm{T}}_{\mathrm{P}}$ denotes the number of targets correctly detected within lanes, ${\mathrm{F}}_{\mathrm{P}}$ denotes the number of false alarms, and ${\mathrm{F}}_{\mathrm{N}}$ denotes the number of missed detections.

4. Experiments and Results

4.1. Experimental Platforms and Environments

Information on the software and devices used in this study is shown in

Table 1. Information on the software and devices.

Software and Device	Version	Function
Operating system	Windows11	–
CPU	i5-11400	–
GPU	NVIDIA GeForce RTX 3060	–
CUDA	12.3	–
Pytorch	11.3	–
Python	3.8	–
Pycharm	2023	Running the YOLO algorithms
MATLAB	R2022b	Running the lane detection, radar, and fusion algorithms
Camera	Hewlett-Packard (HP) 1080p	–
Radar	AWR2243 [42]	–

.

In all experiments, the camera and radar were always placed on a tripod, as shown in Figure 13. The tripod was always placed on the ground, and the distance between the camera’s optical center and the ground was 1.28 m. We kept the roads on the campus as the experimental scenes except for special instructions.

4.2. Visual Detection Results

4.2.1. Validation of Lane Detection Algorithm

We placed the camera on four different roads to capture images with normal light, ground icon interference, ground shelter interference, and weak light. The results of the four environments for lane detection are shown in Figure 15. Where the detected lane lines are drawn in red, green and blue.

From the experimental result in Figure 15a, it can be verified that the lane detection algorithm can precisely detect the lane lines under normal lighting conditions. Even if there are some shadows near the lane lines, excellent detection results can be obtained. Moreover, the lane detection algorithm can also detect lane lines in complex situations, such as lines with ground icon interference, areas occupied by other vehicles, and situations with weak lighting. Examples of these situations and the correct detection results are given in Figure 15b–d.

4.2.2. Validation of YOLOv5 Algorithm

In order to verify the reasonableness of using the YOLOv5s model to detect targets in images, we analyzed its performance in three aspects: time, space, and accuracy. We used the pre-trained models of YOLOv5s, YOLOv7_tiny, YOLOv8s, and YOLOv9s [43] for comparison experiments. Their performance is shown in

Table 2. Performance of different models.

Model	Params/M	FLOPs/G	${\mathbf{AP}}_{50}^{\mathbf{val}}$/%	FPS $(\mathbf{bs}=32)$
YOLOv5s	7.2	16.5	56.8	106
YOLOv7-tiny	6.2	13.9	51.3	123
YOLOv8s	11.2	28.6	61.8	101
YOLOv9s	7.1	26.4	63.4	–

.

Where the values of Params, FLOPs, and ${\mathrm{AP}}_{50}^{\mathrm{val}}$ are official published results obtained on the COCO 2017 dataset. The FPS was obtained by taking a video with our camera and processing it using the four models mentioned above. Considering the limitations of the hardware, as well as the accuracy and real-time performance of the algorithm, YOLOv5s has apparent advantages. Therefore, the experiments justify using YOLOv5s to detect targets in images.

To verify the ability of the YOLOv5s model to recognize targets in natural environments, we captured images with normal light, weak light, intense light, and targets occluded by trees. The experimental results are shown in Figure 16. As we can see from the figures, YOLOv5s can accurately recognize targets even in harsh environments.

4.3. Validation of Radar Target Detection Algorithm

In order to verify the effectiveness of the radar target detection algorithm and the reliability of using lane lines to screen valid radar point clouds, we set up two targets on the road. Taking the radar center as the coordinate origin, the positions of these two targets are (−0.93 m, 21.62 m) and (0.50 m, 9.40 m), as shown in Figure 17, where line1, line2, line3, and line4, respectively, represent the lateral distance between the four lane lines and the center of the radar.

The original point clouds obtained after employing the 2D-FFT, CFAR, and DOA estimation algorithms are shown in Figure 18a. The two vertical lines in Figure 18a are the locations of line1 and line4, which were obtained based on the lane detection algorithm. Figure 18b shows valid point clouds after filtering the interfering targets outside lanes. The results of clustering point clouds within lanes using the DBSCAN algorithm and coalescing the coordinates of point clouds with the same category labels are, respectively, shown in Figure 18c,d. The experiments demonstrate that the radar sensor correctly detects two valid targets, and all of the interfering targets outside lanes are filtered out.

4.4. Validation of Information Fusion

4.4.1. Comparison with Fusion without Lane Detection

Keeping the positions of the sensors and targets the same as in Section 4.3, we performed comparative experiments between the proposed fusion algorithm with lane detection and the case without lane detection. The difference between the two algorithms is whether they contain lane detection, while the rest of the features are consistent. Figure 19 shows the results of the spatiotemporal calibration of the information detected by multiple sensors. The yellow bounding boxes indicate the targets detected by the camera, and the red bounding boxes indicate the targets detected by the radar. As shown in the experiments without lane detection in Figure 19a, neither of the interfering targets outside the lane detected by the two sensors are filtered out completely. This is because a fixed lane width needs to be assumed at this point, and this method is often inaccurate. In contrast, in Figure 19b, all of these disturbances outside lanes are accurately filtered.

Figure 20a,b show the fusion results without lane detection and with lane detection, respectively. In Figure 19a, the ROIs of different targets detected by different sensors in lane 2 and lane 3 overlap with each other, which leads to wrong target matching as shown in Figure 20a. On the contrary, in Figure 20b, we perform information fusion on the same lane, which can accurately fuse the information of the same target from different sensors.

4.4.2. Comparison with Other Related Studies

We collected information about the targets with radar and camera in weak, intense, and normal light. Furthermore, 199 frames of data were collected for each environment. The three scenarios are shown in Figure 21, Figure 22 and Figure 23. In the same experimental scenario, comparison experiments were performed between the results of camera-only detection, the proposed fusion algorithm, and another decision-level fusion method [41]. As the methods of camera-only detection and [41] need to pre-set the lane width to filter out the interfering targets, we set their lateral effective range to $[-6.0\mathrm{m},1.3\mathrm{m}]$. On the contrary, the proposed method does not require us to set the lane threshold. The performance of different algorithms is shown in

Table 3. Performance in different environments.

Environment	Algorithm	Precision	Recall	F1
Normal light	Camera-only detection	86.91%	88.44%	0.88
	Decision-level fusion	86.81%	90.95%	0.89
	Fusion (ours)	96.83%	99.75%	0.98
Weak light	Camera-only detection	79.18%	86.93%	0.83
	Decision-level fusion	82.50%	82.91%	0.83
	Fusion (ours)	96.32%	98.74%	0.98
Intense light	Camera-only detection	69.10%	97.59%	0.81
	Decision-level fusion	62.03%	97.81%	0.76
	Fusion (ours)	96.15%	98.46%	0.97

. The F1 score of the proposed algorithm is able to reach 0.98, which indicates that it has low false and missed detections. Moreover, the F1 score is almost unchanged under different lighting environments, which suggests that the performance of our algorithm is consistent under various environments. In addition, the proposed fusion algorithm outperforms [41] in all aspects. This is attributed to the fact that we used the lane detection results to assist the fusion of the radar and camera, whereas [41] had difficulty in solving the problem of mutual coverage of the bounding boxes of different targets on adjacent lanes.

Finally, in order to analyze the algorithm’s inference time and running memory, we processed 199 frames of data from the scene shown in Figure 23. After temporal-calibrating the radar and camera using the algorithm in Section 3.3.1, each frame of data was sampled at a period of 0.1 s. The time and space required for the algorithms to run are shown in

Table 4. Performance of different algorithms in time and space.

Algorithm	Inference Time/s	Running Memory/GB
Fusion (ours)	0.94	1.26
Decision-level fusion	0.90	0.99

. As a limitation of the software, the average inference time per frame of both algorithms is much larger than the actual acquisition time, which makes it difficult to achieve real-time processing. However, the inference time of the proposed algorithm increased by only 0.04 s compared to [41]. Therefore, combining Table 3 and Table 4, we can conclude that the proposed algorithm is able to achieve superior target detection performance at the expense of a certain time and space.

5. Conclusions

For the shortcomings of the low success rate of target matching and the imprecise accuracy of multi-target detection in decision-level information fusion, a robust target detection algorithm based on the fusion of FMCW radar and a monocular camera is proposed in this paper. The method utilizes a lane detection algorithm to process the image so as to obtain the actual distance of the lane lines concerning the sensors. Then, the lane lines are used to assist the radar and camera in filtering interferences outside the lanes and grouping the remaining valid targets. Spatiotemporal calibration of sensors is achieved through camera calibration, coordinate system conversion, and harmonization of sensor sampling moments. Finally, the information is fused using a decision-level fusion method for targets with the same lane markings detected by the radar and camera. Experiments show that the F1 score of the proposed algorithm can reach 0.98, which indicates that it has low false and missed detections. Moreover, the performance of our algorithm is consistent under various environments. However, the algorithm still presents significant challenges in terms of both time and space. In the future, we will aim to achieve superior detection accuracy while reducing the time and space expenses of the algorithm.