Part-Aware Refinement Network for Occlusion Vehicle Detection

Abstract

Traditional machine learning approaches are susceptible to factors such as object scale, occlusion, leading to low detection efficiency and poor versatility in vehicle detection applications. To tackle this issue, we propose a part-aware refinement network, which combines multi-scale training and component confidence generation strategies in vehicle detection. Specifically, we divide the original single-valued prediction confidence and adopt the confidence of the visible part of the vehicle to correct the absolute detection confidence of the vehicle. That reduces the impact of occlusion on the detection effect. Simultaneously, we relabel the KITTI data, adding the detailed occlusion information of the vehicles. Then, the deep neural network model is trained and tested using the new images. Our proposed method can automatically extract the vehicle features and solve larger error problems when locating vehicles in traditional approaches. Extensive experimental results on KITTI datasets show that our method significantly outperforms the state-of-the-arts while maintaining the detection time.

1. Introduction

Computer vision has been widely used in intelligent transportation fields [1,2,3,4,5,6,7,8] such as accurate identification of persons [9,10,11], reasonable allocation of traffic resources, and improvement of large-scale traffic linkage scheduling. Vehicle detection, as one of the important branches, provides basic support for the subsequent high-level tasks of vehicle decision planning and behavior control. Detecting vehicle type and location information from static images or dynamic videos is the main purpose of vehicle detection. Since 2005, vehicle detection in the traditional sense has been based on machine learning theory. First, vehicle features are extracted by Haar [12], HOG [13], SIFT [14], LBP [15] and other methods. Then, the extracted features are input to the support vector machine (SVM) [16], AdaBoost [17] and other classifiers for vehicle detection. Essentially, these methods rely on feature extraction models and are not effective for today’s complex road conditions. Therefore, vehicle detection methods based on deep learning are produced in accordance with the needs of the times. Recent advances in vehicle detection are mainly attributed to the application of convolutional neural networks based on region selection and region classification. As a representative of the two-stage object detection method, the R-CNN family of algorithms [18,19,20] includes two stages of region selection and object confidence calculation, which greatly improves the efficiency of multi-object detection. Selective search algorithm [21] is usually used to extract candidate regions in region selection, but it cannot meet the efficiency requirements in real-time detection on mobile devices (CPU). The proposal of the YOLO family of algorithms [22,23,24] brings another breakthrough to object detection. It converts the two stages of region selection and classification into a single-stage regression problem, and realizes fast detection. On this basis, YOLO is applied to the detection of road vehicles to achieve real-time detection. To this end, Qiu et al. [25] propose a vehicle recognition algorithm based on a convolutional neural network with fused edge features, which improves the recognition precision and the model’s convergence speed through a simple and effective edge feature fusion method. Luo et al. [26] present a model based on Faster R–CNN with neural architecture search (NAS) optimization and feature enrichment to realize the effective detection of multi-scale vehicle targets in traffic scenes. NAS is a technology for automatically designing neural networks, which can automatically design high-performance network structures based on sample sets through algorithms. The method proposed by Haritha et al. deals with the detection of vehicles and is capable of handling distant small-scale vehicles without bothering the image scalability. Image scalability refers to whether it can be further analyzed (identified, etc.). However, these methods are only for small-scale vehicle detection and cannot be robust to occlusion scenes. Occlusion robustness refers to the model achieving good performance in occlusion scenarios. Therefore, it is urgent to propose an efficient and effective vehicle detection algorithm under occlusion conditions. Moreover, there exist lots of YOLO extensions purport to handle small objects and occlusion. Li et al. propose the addition of an attention mechanism, a CIoU (complete intersection over union) loss function, Soft-NMS (non-maximum suppression), and depthwise separable convolution to handle the occlusion. Du et al. present an improved YOLO model for infrared occlusion object detection under confusing background. Ryu et al. propose a detection model of occluded object based on YOLO using hard-example mining and augmentation policy optimization. Most of these methods reduce the impact of occlusion on the model from the perspective of loss function and data augmentation, and our proposed method aims to achieve an occlusion-robust model from the network structure. Moreover, there exist many YOLO extensions that purport to handle small objects and occlusion. Li et al. [27] propose the addition of an attention mechanism, a CIoU (complete intersection over union) loss function, Soft-NMS (non-maximum suppression), and depthwise separable convolution to handle the occlusion. Du et al. [28] present an improved YOLO model for infrared occlusion object detection under confusing background. Ryu et al. [29] propose a detection model of occluded object based on YOLO using hard-example mining and augmentation policy optimization. Most of these methods reduce the impact of occlusion on the model from the perspective of loss function and data augmentation, and our proposed method aims to achieve an occlusion-robust model from the network structure.

There are two main challenges for vehicle detection in actual scenarios: (1) The previous single confidence value based vehicle detection method uses the anchor box generation strategy to obtain the final detection box, which does not perform well for datasets with large object scale spans. (2) The output tensor adopts the predicted single value of the single-state network, which cannot describe the confidence score of the vehicle’s components. For the occluded part, the normal single-state network [24] may assign low confidence to the occluded vehicle, resulting in missed detection and false detection. To cope with the above problems, we use the K-means clustering algorithm to generate a suitable anchor box size, adapt to the scale change of the dataset and improve the detection accuracy. Moreover, we optimize the model with the component confidence strategy, and use the confidence of the visible vehicle parts to correct the final detection confidence and improve the detection accuracy. In this paper, we choose the single-stage object detection method as the basic network structure. In order to eliminate the influence of scale and occlusion factors on vehicle detection, the KITTI dataset with our relabeling is used to train the deep neural network model. Combined with multi-scale training and component confidence generation strategy, the parameters are adjusted reasonably to transform the vehicle detection problem. For the multi-classification problem of vehicles, its efficiency and accuracy are further improved.

In brief, the major contributions of this paper are highlighted as follows.

• We propose a part-aware refinement network for occlusion vehicle detection, which optimizes the model with the component confidence strategy, and uses the confidence of the visible vehicle parts to correct the final detection confidence. Our proposed method can automatically extract the vehicle features, it has solved the problems of larger error when locating vehicles in traditional approaches, and improved the vehicle detection recall.
• we adopt the K-means clustering algorithm to generate a suitable anchor box size, adapt to the scale change of the dataset and improve the detection accuracy.
• we relabel the KITTI dataset, adding the detailed occlusion information of the vehicles. The new dataset can provide better supervision for occluded vehicle detection.

2. Proposed Method

2.1. Preliminary Knowledge on Single-Stage Object Detection

The algorithm proposed in this paper is based on a single-stage object detection framework to adapt to real-time vehicle detection in real-world scenarios. Thus this subsection gives preliminary knowledge on single-stage object detection. The single-stage object detection algorithm omits the selection steps of candidate regions, simplifies the network structure, and dramatically improves the detection speed of the algorithm. However, it meanwhile leads to insufficient feature extraction, sacrificing a certain degree of accuracy, and the appearance of detection objects.

In 2016, Redmond et al. proposed the YOLOv1 [22] for the first time, which inputs the raw image into the neural network to directly obtain the bounding box and classification results. Specifically, YOLO first divides the image into S × S squares. If the center point of an object falls in a square, N bounding boxes are predicted around the square, and one calculates the confidence of the object surrounded by each bounding box. Finally, the non-maximum suppression algorithm (NMS) [30] is used to remove the redundant bounding box of a single object, and the detection result is obtained. Compared with other algorithms, the biggest advantage of this algorithm is that the detection speed is fast. YOLOv1 can detect 45 frames per second while distinguishing the background and the object to be measured well. However, it loses a certain accuracy and the detection performance of small objects.

In order to solve the problem of low detection accuracy of the YOLOv1, YOLOv2 [23] adds BN (batch normalization) after each convolutional layer, applies multi-anchor training methods to increase the number of detectable objects. Moreover, YOLOv2 is fine-tuned on a pre-trained CNN using a high-resolution dataset. YOLOv3 [24] introduces a residual module and nine anchor boxes for detection, which improves the detection accuracy of small targets while ensuring the detection speed. Specifically, YOLOv3 first transforms the input image to a size of N × N (N is 416 by default), and uses Darknet-53 to extract image features. Then, the transformed image is divided into S × S equal-sized grids (S is 13, 26, and 52, respectively, representing three scales), and the network randomly selects a new image size every 10 batches. For each grid, three anchor boxes generate three bounding boxes, respectively. as shown in Figure 1, for each grid, the convolutional neural network predicts 4 values to represent each bounding box, denoted as $t_x,t_y,t_w,t_h$, ie the coordinates $(x,y)$ and the width w and height h of the object. If the object center is offset from the upper left corner of the image in the grid, the offset is assumed to be $(c_x,c_y)$, the height and width of the anchor box are recorded as $P_w$ and $P_h$. The bounding box is corrected as:

$$\begin{array}{cc}\hfill & b_x=\sigma \left(t_x\right)+c_x\hfill \\ \hfill & b_y=\sigma \left(t_y\right)+c_y\hfill \\ \hfill & b_w=p_w{e}^{t_w}\hfill \\ \hfill & b_h=p_h{e}^{t_h}.\hfill \end{array}$$

(1)

During training, the sum of squared errors is used to calculate the sum of the losses, including the localization loss and the confidence loss. Assuming that the real position coordinate is ${\hat{t}}^{*}$, the loss function is minimized by the method of gradient descent, where the gradient is the difference between the real value and the predicted value of the coordinate ${\hat{t}}^{*}-t^{*}$. For the confidence calculation of the predicted object, YOLOv3 converts it into a regression problem. First, the overlap rate between the anchor box and the real object bounding box is calculated [31], denoted as $A_{iou}$, and $A_{iou}^{max}$ is assumed to represent all anchor boxes and the real object bounding box. The maximum value of the overlap ratio, and then the probability $P_{object}$ that anchor box contains the object is calculated as follows:

$$P_{\mathrm{object}}=\left\{\begin{array}{cc}1\hfill & A_{iou}\ge A_{iou}^{max}\\ 0\hfill & A_{iou}>0.5 \mathrm{and} A_{iou}<A_{iou}^{max}\end{array}\right.$$

(2)

YOLOv3 assigns only one anchor box to each object. In particular, those anchor boxes that are considered to contain no objects are considered to have a loss value of 0. In order to enable YOLOv3 also to perform multi-label classification of an object, YOLOv3 simultaneously uses the binary cross-entropy loss [32] method for class prediction.

YOLOv4 adopts a better Mish activation function and introduces the SPP module to further improve the detection effect. YOLOv5 was proposed almost at the same time as the YOLOv4, which is slightly weaker than YOLOv4 in performance, but much stronger than the latter in flexibility and speed.

2.2. Part-Aware Refinement Network

2.2.1. Output Tensor Overview

One-stage vehicle detection can be viewed as a regression problem and generates tensors to predict the output. As shown in Figure 2, the width $\left(W\right)$ and height $\left(H\right)$ of the output tensor depend on the number of grids of the input image, and the depth $\left(K\right)$ depends on the number of anchor boxes per grid. The prediction output definition of each anchor box is shown in Figure 3. The box offset of the model is defined by the position and scale between the bounding box $x_{gt},y_{gt},w_{gt},h_{gt}$ and the matching anchor box $x_i,y_i,w_k,h_k$, where $i\in [1,W], j\in [1,H], k\in [1,K]$. The scale parameter $({\delta }_w,{\delta }_h)$ describes the scale difference between the prediction box and the anchor box:

$${\delta }_{w_{,}\left(ijk\right)}=log\left(\frac{w_{gt}}{w_k}\right), {\delta }_{h_i\left(ijk\right)}=log\left(\frac{h_{gt}}{h_k}\right).$$

(3)

For the position parameter $({\delta }_x,{\delta }_y)$, it corresponds to the relative position of the upper left corner in the prediction grid, and its boundary is [0, 1):

$${\delta }_{x,\left(ijk\right)}=\sigma \left(\frac{x_{gt}-x_i}{w_{grid}}\right),{\delta }_{y,\left(jjk\right)}=\sigma \left(\frac{y_{gt}-y_i}{h_{grid}}\right).$$

(4)

where $\sigma$ is the sigmoid function.

The single-stage vehicle detection method defines the confidence level of object existence in Equation (5). Then, the conditional probability of the class C object is defined in Equation (6). Finally, the final object probability is obtained by multiplying the object conditional probability by the object confidence.

$$c_{\left(ijk\right)}=P_{\left(ijk\right)}\left(\mathrm{Object}\right)\times {IOU}_{\left(ijk\right)}^{gt}$$

(5)

$$p_{m,\left(ijk\right)}=P_{\left(ijk\right)}(\mathrm{Class}=m\mid \mathrm{Object}),m\in [1,C]$$

(6)

2.2.2. Part-Aware Refinement Network

Our key idea for occlusion handling is to block the prediction confidence rather than represent it as a single value for an existing single-state network. A normal single-state network might assign low confidence to an occluded vehicle due to the occluded part, but our model can use the confidence of the visible parts of the vehicle to correct the final detection confidence of the vehicle.

We first introduce the concept of a part confidence map [33,34] represented by V, which is an $M\times N$ grid (generated by the sigmoid function) in the range [0, 1], as shown in Figure 4. The ground truth of the part confidence map is generated as follows. We identify a bounding box for the overall vehicle and divide it into an $M\times N$ grid. The YOLO algorithm determines whether it is an object box by the confidence of several prediction boxes. Our method aims to convert the confidence of each prediction box from a single value to a confidence map, on this basis, the confidence of the occluded part can be weakened to improve the vehicle prediction box. For each cell $(m,n)$, $m\in [1,M]$, $n\in [1,N]$, set $V_{gt}(m,n)=1$ if the cell area occupied by the vehicle exceeds ${\tau }_v$ times. In our experiments, we set $M=3$, $N=6$ and ${\tau }_v=0.5$.

For occlusion processing, we expand the output tensor to include predictions of the part confidence map $\hat{V}$ (see Figure 4). That is, the network predicts the $\hat{V}$ detection output, from which we compute the final occlusion-aware detection score for each anchor box. We adopt an end-to-end learnable scoring method [35], as shown in Figure 5. We define P blocks $W_p\in R^{M\times N}$, $p\in [1,P]$. We compute the intermediate part score by element-wise dot product prediction of the part confidence map $\hat{V}$:

$$S_p=\sum _{m=1}^M\sum _{n=1}^N\left(\hat{V}(m,n)·W_p(m,n)\right).$$

(7)

With $S=\left[s_1,s_2,\dots ,s_p\right]$, we compute the final vehicle prediction confidence through a multilayer perceptron (single hidden layer and ReLU layer):

$$S_{car}=\sigma \left(w_2^{T}max\left(0,w_1^{T}s\right)\right).$$

(8)

${\left\{W\right\}}_{p=1}^P$ and $w_{1,2}$ are parameters that need to be learned, where P represents the number of modes that occlude vehicles. We choose $P=40$ through experiments. The mode indicates the occlusion situation, such as occlusion above the vehicle, occlusion below, etc. Setting 40 modes means that there are 40 occlusion situations. At the same time, we tried different multi-layer perceptron structures and found that the effect in Equation (8) is the best.

2.2.3. Loss Function

The loss of the one-stage model consists of two parts: the localization loss $L_l$ and the confidence loss $L_c$. After introducing the part confidence map, we define the part confidence loss $L_p$ as follows:

$$L_p=\left({\lambda }_p^{+}{H}_{\left(ijk\right)}^{+}+{\lambda }_p^{-}{H}_{\left(ijk\right)}^{-}\right)\times \sum _{m=1}^M\sum _{n=1}^N{\left(V_{\left(ijk\right)}(m,n)-{\hat{V}}_{\left(ijk\right)}(m,n)\right)}^2$$

(9)

where $H_{\left(ijk\right)}^{+}=1$ indicates that the $\left(ijk\right)$ anchor box is a positive example but is occluded, and $H_{\left(ijk\right)}^{-}=1$ is a negative example, thus the final loss value is the weighted sum of 3 loss values:

$$L=\sum _{i=1}^W\sum _{j=1}^H\sum _{k=1}^K\left({\lambda }_l{L}_{l,\left(ijk\right)}+{\lambda }_c{L}_{c,\left(ijk\right)}+{\lambda }_p{L}_{p,\left(ijk\right)}\right),$$

(10)

where ${\lambda }_l,{\lambda }_c$ and ${\lambda }_p$ are hyperparameters to trade-off the loss.

3. Experimental Results

3.1. KITTI Dataset

The KITTI dataset [36] is the original dataset for training in this paper, which contains real image data collected under a variety of complex scenes (highways, urban streets, rural streets, etc.). Each of these images includes up to 15 vehicles with different motion states (stationary or moving) and different degrees (truncated or occluded). In the experiment, 5000 images with different characteristics were randomly selected and converted into COCO dataset format for training the model. At the same time, 2500 images are selected to test the trained model to verify the effectiveness of the algorithm.

The definition of occlusion in the KITTI dataset is 0 (unoccluded), 1 (partial occlusion), 2 (large area occlusion), and 3 (unknown). In order to get the true value of the part confidence in Equation (9), we preprocess the KITTI dataset here, as shown in Figure 6. For an image in the KITTI test set, we find the overlapping part of its annotation boxes, and determine which annotation box is occluded by the occlusion degree of each annotation box. Then, for the occluded bounding box, set $V_{gt}(m,n)=0$ in the overlapping part, and set $V_{gt}(m,n)=1$ in the non-overlapping part. This generates part confidence for each bounding box.

3.2. Implementation Details

The evaluation criteria of the validity test in the vehicle detection problem include recall and precision. The recall and precision used in this paper are in the range of [0, 1]. The precision is for the prediction result, it indicates how many of the predicted positive samples are true positive samples. Then, there are two possibilities for the prediction to be positive, one is to predict the positive class as the positive class ($TP$), and the other is to predict the negative class as the positive class ($FP$). That is

$$\mathrm{precision}=\frac{TP}{TP+FP}.$$

(11)

The recall is for the original sample, it indicates how many positive examples in the sample are predicted correctly. There are also two possibilities, one is to predict the original positive class as a positive class ($TP$), and the other is to predict the original positive class as a negative class ($FN$). The calculation method is as follows:

$$\mathrm{recall}=\frac{TP}{TP+FN}.$$

(12)

In this paper, all experiments are conducted on Intel(R) Core(TM) i5-6500 [email protected] GHz, 64 G RAM, TITAN X GPU. Every network is trained for 100 epochs, and the batch size is adjusted to 16 according to the GPU and network parameters. The gradually decreasing learning rate during the training process will be more accurate than the fixed learning rate. Therefore, this paper adopts the strategy of adjusting the learning rate at equal intervals. After each epoch, the learning rate is adjusted by 0.90 times, and the initial learning rate is set to 0.001. We choose YOLOv3 as our baseline model.

3.3. Multi-Scale Anchor Box Design

We use the K-means clustering algorithm to get the anchor box size, where the similarity is defined as the overlap ratio of the rectangular boxes (represented by $R_{IOU}$), and annotate the road vehicle target training set. The distance function of K-means clustering is

$$d(S,T)=1-R_{IOU}(S,T).$$

(13)

Among them, S and T represent the size and center of the rectangular frame of the object, respectively, and $R_{IOU}(S,T)$ represents the overlap ratio of the two rectangular frames. After weighing the average $R_{IOU}$ and the number of anchor boxes, as shown in Figure 7, take 15 anchor boxes, which are (10, 95), (13, 36), (19, 156), (22, 51), (29, 80), (34, 255), (39, 56), (44, 112), (60, 82), (64, 298), (74, 136), (108, 177), (119, 279), (175, 293), (175, 521). Each grid at each scale predicts five bounding boxes with five anchor boxes.

As shown in Figure 7, ideally, the larger the number of anchor boxes, the better. However, the number of anchor boxes affects the execution efficiency of the algorithm. We observed that when the number of anchor boxes reaches 15, we will continue to increase the number of anchor boxes. The improvement of the average intersection ratio is not obvious, so the average intersection ratio and the number of anchor boxes are weighed, and 15 anchor boxes are selected.

3.4. Ablation Studies

We adopt different strategies to train and test the neural network respectively, which further verifies the effectiveness of different strategies proposed in this paper. As shown in

Table 1. Detection results before and after algorithm training strategy improvement.

Model	Number of Anchors	Part-Aware Refinement	Recall (%)	Precision (%)
A	9	✗	94.24	84.24
B	16	✗	94.56	85.42
C	9	✔	96.12	86.26
D	16	✔	97.09	87.52

. It can be found that the detection precision is improved by 0.78% with the change of the number of anchor boxes. The reason is that there are nine anchor boxes in the original network, and vehicles with smaller sizes are not easy to be detected. Comparing strategies A and C in Table 1, it can be seen that the strategy of component confidence generation has obvious advantages. It solves the problem of vehicles being occluded in the road and improves the detection precision by 1.62 percentage points. This experiment fully proves the effectiveness of using multi-scale training and part confidence generation strategy to train the model, and improves the detection precision and recall of the model.

3.5. Compare the Detection Performance of the Two Algorithms in the Same Scene

The baseline model and our proposed method were trained using the KITTI dataset, and tested through the test set (only vehicle indicators were extracted). The precision and recall of the two algorithms are shown in

Table 2. Detection effect of our method and baseline model in the same scene.

Method	Recall (%)	Precision (%)	Average IOU
Baseline	94.24	84.64	72.38
Ours	97.09	87.52	78.03

, and the detection effect is shown in Figure 8.

From the data comparison in Table 2, it can be seen that the recall rate and precision of our model through multi-scale training and part confidence generation strategy are better than the original baseline model. Compared with the baseline model, the time performance is still good, the recall is increased by 2.85%, the precision is increased by 2.88%, and the average IOU is increased by 5.65%. As can be seen from Figure 8, our model can detect smaller-scale vehicles and partially occluded vehicles with good results. For sparse vehicle and dynamic pedestrian objects, the difference between our method and the two algorithms of the baseline model is small. For the case where some vehicles and pedestrian objects overlap, the original baseline model has missed detection, and our method can detect overlapping objects.

3.6. Comparison with SOTA Methods

We use the KITTI dataset to train SSD [37], RFCN [38], Faster R-CNN [20], YOLOv3 [24], FE-CNN [25], RIAC [26], MVD [39] and our method respectively. The accuracy and detection time are then evaluated on the testing set, as shown in

Table 3. Precision and time comparison between our method and existing algorithms for detecting single vehicle images.

Method	SSD [37]	RFCN [38]	Faster R-CNN [20]	YOLOv3 [24]	FE-CNN [25]	RIAC [26]	MVD [39]	Ours
Precision (%)	83.54	88.39	86.30	84.64	85.25	86.19	85.92	87.52
Recall (%)	94.93	97.62	96.34	94.24	94.87	96.55	96.14	97.09
IOU	74.39	76.11	75.99	74.26	72.38	76.11	75.96	78.03
Time (s)	0.177	0.281	0.293	0.082	0.083	0.151	0.097	0.091

. SSD combines the methods of Faster R-CNN and YOLOv1 to improve the detection accuracy. RFCN adopts the method of Faster R-CNN. Compared with Faster R-CNN, it improves the detection speed and reduces the amount of calculation. YOLOv3 converts the detection into a regression problem, the network structure and detection speed are improved, but the detection accuracy is relatively low. FE-CNN is based on a convolutional neural network with fused edge features, which improves the recognition precision and the model’s convergence speed through a simple and effective edge feature fusion method. RIAC is based on Faster R–CNN with NAS optimization and feature enrichment to realize the effective detection of multi-scale vehicle objects in traffic scenes. MVD deals with the detection of vehicles and is capable of handling distant small-scale vehicles without bothering the image scalability.

The advantage of this paper is that it combines the part confidence generation strategy and multi-scale training method to improve the accuracy based on YOLOv3. It can be seen from Table 3 that the method in this paper improves the accuracy while ensuring the faster detection rate of the YOLO series algorithm itself. It is more suitable for real-time detection systems. In summary, the method in this paper is an effective vehicle detection method.

4. Conclusions

This paper proposes a part-aware refinement network for occlusion vehicle detection, which combines multi-scale training and part confidence generation strategies to transform the vehicle detection problem in complex environments into an easy-to-implement regression problem. The model is trained on the KITTI dataset, and the parameters are adjusted reasonably. Experiments verify that for detection scenes of different scales and different degrees of occlusion, the trained models can extract vehicle features well and have good robustness.