Vehicle Detection and Classification via YOLOv8 and Deep Belief Network over Aerial Image Sequences

Abstract

Vehicle detection and classification are the most significant and challenging activities of an intelligent traffic monitoring system. Traditional methods are highly computationally expensive and also impose restrictions when the mode of data collection changes. This research proposes a new approach for vehicle detection and classification over aerial image sequences. The proposed model consists of five stages. All of the images are preprocessed in the first stage to reduce noise and raise the brightness level. The foreground items are then extracted from these images using segmentation. The segmented images are then passed onto the YOLOv8 algorithm to detect and locate vehicles in each image. The feature extraction phase is then applied to the detected vehicles. The extracted feature involves Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB), and KAZE features. For classification, we used the Deep Belief Network (DBN) classifier. Based on classification, the experimental results across the three datasets produced better outcomes; the proposed model attained an accuracy of 95.6% over Vehicle Detection in Aerial Imagery (VEDAI) and 94.6% over Vehicle Aerial Imagery from a Drone (VAID) dataset, respectively. To compare our model with the other standard techniques, we have also drawn a comparative analysis with the latest techniques in the research.

1. Introduction

In recent years, vehicle detection and classification has been an emerging research area due to its various applications in intelligent traffic management systems. Road Traffic management applications include congestion detection, categorizing the various vehicle types, recognizing doubtful vehicles on the road, and parking management system [1]. All these systems mainly depend on vehicle identification, which has become a significant and crucial issue in aerial imagery [2]. In conventional systems, vehicle detection was primarily conducted by estimating motion in the image pixels [3,4,5,6]. However, the methods are not efficient enough in remote sensing data because motion is also detected in pixels other than the targeted objects [7]. Recently, researchers have proposed many improved techniques, which include object segmentation [8], silhouette extraction [9], feature extraction, and classification [10], to enhance the object detection capabilities of a system [11,12,13,14,15,16].

Aerial images provide a better and broader view, thus providing significant information about the sensed environment [17]. These images are used in numerous applications, such as deforestation detection [18], agriculture field monitoring [19], and disaster management systems [20]. The aerial traffic data is also collected to do traffic analysis to efficiently use the road network, forecast forthcoming transportation requirements, and improve traveler protection [21].

In our proposed model, we have used aerial images to recognize and classify vehicles. In our model, the aerial videos are first converted into image frames. These frames are pre-processed for noise removal and brightness enhancement using defogging and gamma correction techniques, respectively [22,23,24,25]. Then, the images are segmented to reduce the background complexity using Fuzzy C Mean segmentation. To detect vehicles in each extracted frame, YOLOv8 is employed, which can detect small objects effectively. In the end, all the detected vehicles are subjected to SIFT, ORB, and KAZE feature extraction to classify them into multiple vehicle classes. For classification, we used the Deep Belief Network, which is a simple classifier that uses neural networks, thus providing better classification accuracy. Our accuracy has proven to be a result of an efficient model design. The following is our system’s primary contribution:

• Our model combines the pre-processing methodologies with the segmentation technique to prepare images before passing them to the detection phase to reduce model complexity.
• We used the newest YOLOv8, which has improved architecture to enhance vehicle detection in segmented images as it can effectively detect objects of varying sizes.
• To classify vehicles, multiple features, including SIFT, ORB, and KAZE features, are extracted. Combining scale and rotation invariant, 2D and fast and robust local feature vectors are effective in classifying vehicles in aerial images.
• The proposed system uses a deep learning-based DBN classifier to achieve higher classification accuracy.

The following is a list of the remaining sections of this article. Related work analysis of the current approaches is included in Section 2. The suggested system’s architecture is presented in Section 3. The experimental portion with a system performance evaluation is shown in Section 4. Section 5 presents the system’s conclusion and the direction of future efforts.

2. Related Work

In this section, we presented the most relevant and popular systems designed for vehicle detection and classifications.

Table 1. Related Work for Vehicle Detection and Classification.

Authors	Methodology
Arinaldi et al. [26]	The paper implements two different methodologies for vehicle detection and classification. The first method uses a Mixture of Gaussian (MoG), combined with a Support Vector Machine Classifier (SVM) classifier. The other method only uses faster Recurrent Convolutional Neural Network (RCNN). However, there was still a large number of vehicles that were left undetected.
Aqel et al. [27]	This study uses the background subtraction method to detect moving autos. To lower the occurrences of false positives, morphological corrections are performed. In the end, the classification is accomplished using the invariant Charlier moments. The method uses conventional image processing techniques, that limits its applicability to diverse traffic scenarios. Also, the background subtraction method will eliminate the cars which are not in motion, thus reducing the true positives.
Sarikan et al. [28]	The model uses a K-nearest neighbor classifier to automatically detect and classify vehicles. For feature extraction, windows and hollow areas of the vehicles are constructed to classify it as a motorcycle or car. The model is not applicable for broader views and dense traffic conditions.
Tan et al. [29]	The authors presented a method to classify vehicles using a Convolutional Neural Network (CNN). It uses an aerial image dataset. The proposed model firstly determines whether the area contains any vehicle or not by evaluating motion changes, feature matching and heat maps. Then, the classification is conducted using the classification layer of inception-v3 and AlexNet.
Hamzenejadi et al. [30]	This paper presents real-time vehicle detection solution based on Yolov5. The existing model is improved by adding attention mechanism and a new concept of ghost convolution. The experimental results prove the efficiency of the YOLO model in object detection models.
Ozturk et al. [31]	In this paper, a vehicle detection method has been presented. The vehicles are detected via miniature CNN architecture combined with morphological corrections. The model requires intensive post-processing to achieve good results. Also, the accuracy is not consistent on other datasets.
Roopa Chandrika et al. [32]	A model for vehicle recognition and classification has been presented. The model incorporates adaptive background subtraction along with binary label segmentation to locate vehicles. The approach is not suitable for stationary car detection or during traffic jam conditions.
Kumar et al. [33]	A new approach that uses You Only Look Once (YOLO) with Long Short-Term Memory (LSTM) to detect and classify vehicles. To reduce the model complexity, the images are segmented into binary labels in the pre-processing stage. The detected vehicles are also counted by counting the bounding boxes and classified into lightweight and heavy-weight vehicles.
Zhang et al. [34]	The paper proposes a method that uses an improved YOLOv3 algorithm to detect vehicles. The pre-trained YOLO network is trained with a new structure to improve the accuracy of the detection method. However, YOLOv3 is one of the oldest versions. The detection results can be improved by using the newest architectures.

presents the details of the different models proposed by the researchers in the literature.

Even though extensive research has been completed in the field of automated traffic monitoring systems, there is still room for improvement. The detection of vehicles in aerial images specifically in intensive traffic conditions requires efficient and specialized architectures to obtain good results. Machine learning methods are not good enough to differentiate between objects that have motion in their pixels [35,36]. Therefore, YOLOv8 is the newest and most effective object detector based on convolution layers [37,38]. Moreover, combining different feature sets to classify vehicles can contribute to reducing classification errors.

3. Proposed System Methodology

The proposed architecture identifies vehicles in the images and classifies them into multiple vehicle classes. Primarily, the videos are first converted into frames. Pre-processing procedures are applied to the images, i.e., defogging for noise reduction, and then Gamma correction is used to modify the intensity of the images for improved detection. On the filtered images, FCM segmentation is applied to separate the foreground and background objects [39,40,41]. The detection is performed using the YOLOv8 algorithm. After vehicle detection, SIFT, ORB, and KAZE features were extracted [42,43,44]. On this feature vector, the DBN classifier was trained to classify each detected vehicle into its corresponding class. The proposed system design is shown in Figure 1.

3.1. Images Pre-Processing

Noise reduction is required in the obtained image to remove additional pixel information, since the extra pixels make detection more difficult [45,46,47]. Only the most appropriate filter that incorporates defogging techniques is applied to the specific noise for good results [48]. The defogging method determines the amount of noise present in each pixel of the image, then eliminates it as follows:

$$G\left(x\right)=X\left(x\right)Y\left(x\right)+Z(1-p\left(x\right))$$

(1)

where x specifies the location of the pixel, Z is the fog density, and Y(x) is the transmission map. Figure 2 shows the defogged images.

In the next step, Gamma correction [49,50] is used to alter the denoised image’s intensity since the region of interest can be detected most effectively when the brightness is high [51]. The power-law for gamma correction is given as:

$$V_o=T{V}_I^{\gamma }$$

(2)

where T is a constant that is typically equal to 1, $V_I$ is the input’s non-negative values with power $\gamma$ whose range can be between 0 and 1. $V_o$ represents the resultant image [52,53,54,55]. Figure 3 displays the denoised, intensity-adjusted image with the plot. The gamma-corrected images are given in Figure 3.

3.2. Fuzzy C-Mean Segmentation

In this section, the foreground objects are separated from the background to reduce the complexity of the images. For this purpose, we used the FCM segmentation technique, that groups the image pixels into one or more clusters [56]. In FCM segmentation, the pixels which belong to more than one cluster are known as fuzzy logic [57,58]. While grouping the pixels, the objective function is optimized during numerous iterations of the process [59,60]. The clustering centers and membership degrees have been regularly changed during the iterations [61]. A finite collection of N elements Q = $q_1$, $q_2$, …, $q$ is divided into a set of M clusters via the FCM method. Each element of the vector $w_j$, where j = 1, 2, …, N, has n dimensions [62,63]. We define a technique to divide Q into M clusters using the cluster centers $c_1$, $c_2\dots ,c_m$ in the centroid set c [64]. In the FCM technique, h is a representative matrix that shows each element’s participation in each cluster [65,66]. It can be well-defined as:

$$h\left(j,y\right), 1\le j\le N;1\le y\le M$$

(3)

where the membership value of the element $q_j$ with cluster center $c_y$ is represented by $h\left(j,y\right).$ We are more certain that the element $q_j$ belongs to the y cluster if the value of $h\left(j,y\right)$ is higher [67,68]. Moreover, when calculating the performance index $L_{\mathrm{f}}$, the weighted sum of the distance between the components of the relevant fuzzy cluster and the cluster center is calculated [69,70].

$$L_{\mathrm{f}}=\left(h,c\right)={\displaystyle {\sum }_{i=1}^v} {\displaystyle {\sum }_{a=1}^y} h_{ia}^t{∥q_i-c_a∥}^2, 1 < t <\infty$$

(4)

where $c_a$ is the ath cluster center, $q_i$ is the ith pixel, v is the cluster number, y is the number of pixels, and t is the blur exponent [71,72,73,74,75]. The following formula is used to update the membership function:

$$h_{ia}^t=\frac{1}{{\displaystyle {\sum }_{h=1}^m} {\left(\frac{{dis}_{ia}^2}{{dis}_{ha}^2}\right)}^{\frac{2}{t-1}}}$$

(5)

where the distance between the cluster centroid $c_a$ and the pixel $q_i$ is supplied by ${dis}_{ia}^2$, and the membership matrix is represented by $h_{ia}^t$, which ranges (0, 1). The point of cluster centroid is calculated as follows:

$$c_a=\frac{{\displaystyle {\sum }_{j=1}^N} h_{ij}^t{q}_j}{{\displaystyle {\sum }_{j=1}^N} h_{ij}^t}$$

(6)

When a pixel gets close to the cluster center to which it belongs, it receives a high membership value, and vice versa. The result of FCM segmentation is seen in Figure 4.

3.3. Vehicle Detection via YOLOv8

For vehicle detection, we used the YOLOv8 algorithm. YOLOv8 is an efficient single-shot detector that can be used for detection, segmentation, and classification tasks [76]. Furthermore, it requires fewer parameters for training [77,78,79,80]. Based on the CSP concept, the C2f module replaces the C3 module, whereas the YOLOv8 backbone is mostly the same as the YOLOv5 backbone [81,82]. The C2f combines C3 and ELAN to create the C2f module, building on the ELAN concept from YOLOv7, so that YOLOv8 might continue to be portable while obtaining more comprehensive gradient flow information [83]. The SPPF module was still utilized at the end of the backbone, and three Maxpools of size 5 × 5 were sequentially applied before each layer was concatenated to ensure the precision of objects of varying scales while also maintaining a low weight [84].

The feature fusion approach still employed by YOLOv8 in the neck section is PAN-FPN, which improves the fusion and usage of feature layer data at numerous scales. The neck module is made up by combining the final decoupled head structure, numerous C2f modules, and two upsamplings [85,86,87,88]. The final component of the neck in YOLOv8 was constructed using the same concept as the head in YOLOx. It increased accuracy by combining confidence and regression boxes. Moreover, it is an anchor-free model which can directly detect the object’s center. In order to expedite Non-Maximum Suppression (NMS), a challenging post-processing step that sorts through potential detections following inference, anchor-free detection lowers the number of box predictions. The detected vehicles using the YOLOv8 are given in Figure 5.

3.4. Feature Extraction

This section describes a method for extracting various features. The feature set comprises three different features: SIFT, KAZE, and ORB.

3.4.1. SIFT Features

We used the Scale Invariant Feature Transform (SIFT) technique to obtain important features [89,90,91]. SIFT reduces an image’s information to a set of points that can be used to identify recurrent patterns in other pictures [92]. Scale and rotation invariant features are retrieved using SIFT [93,94,95]. Figure 6 shows the steps of SIFT feature extraction.

The extracted SIFT features are given in Figure 7.

3.4.2. KAZE Features

In order to extract KAZE features, a Gaussian kernel is convolved with an input image [96]. The convolved image is used to construct an image gradient histogram, and computer code is used to calculate the contrast parameters [97]. Values for the contrast parameter and evolution time are used to calculate the nonlinear scale space as follows:

$$t^{j+1}={\left(I-f_j+1-f_j{\displaystyle {\sum }_{i=1}^m} B_i\left(t^j\right)\right)}^{-1}{t}^j$$

(7)

to determine the response of the scale normalized determinant of the Hessian at various levels to identify interesting locations, we use the formula:

$$F_{Hess}={\sigma }^2\left(t_{xx}{t}_{yy}-t_{xy}^2\right)$$

(8)

The second-order cross-derivative is presented as $t_{xy}$, the second-order horizontal derivative as $t_{xx}$, and the vertical derivative is given as $t_{yy}$. The extracted KAZE features are shown in Figure 8.

3.4.3. ORB Features

The Oriented FAST and Rotated BRIEF (ORB) is an efficient feature extractor. To identify key points, it uses the FAST (Features from Accelerated Segment Test) keypoint detector [98,99,100]. It makes more complex use of the BRIEF (Binary Robust Independent Elementary Features) description. Additionally, it is dimensionally and rotationally invariant [101]. The patch moment is obtained as follows:

$$m_{uv}=\sum x^u{y}^{v}l(j,k)$$

(9)

where u and v represent the intensities of the picture pixels at the j and k locations. Moreover, the mass center is calculated by using the following formula.

$$W=\frac{m_{10}}{m_{00}}, \frac{m_{01}}{m_{00}}$$

(10)

The patch orientation is obtained by:

$$\theta =atan(m_{01},m_{10})$$

(11)

The final extracted feature is seen in Figure 9.

3.5. Classification via DBN

A Deep Belief Network (DBN) classifier is being used to classify vehicles. A deep neural network serves as a DBN’s building block, which is composed of layers of latent variables connected only between the layers as a whole and not between the units within each layer [102]. For the creation of DBN, Restricted Boltzmann Machines (RBN) act as the fundamental building blocks [103]. A layer of RBN’s visible and hidden units combine to form a two-layer structure [104]. The collective energy arrangement of the two units is calculated as:

$$\begin{array}{cc}\hfill Enr(MN,WM,\theta )& =-{\displaystyle {\sum }_{i=1}^D} r{M}_i{v}_i-{\displaystyle {\sum }_{j=1}^F} a_{Hj}{h}_j-{\displaystyle {\sum }_{i=1}^D} {\displaystyle {\sum }_{j=1}^F} s_{j}M{N}_{i}W{M}_j\hfill \\ & = > -r^{T}MN-a^{T}WM-M{N}^{T}MWM\hfill \end{array}$$

(12)

where $\theta =\{r{M}_i, a{W}_j, s{e}_{ij}\}$, $a_{Hj}$ and $r{M}_i$ stand for the bias conditions of the visible and hidden components, respectively. The hidden j and visible component i are given different weights by $s{e}_{ij}$. The following determines the combined unit’s configuration:

$$Pr(MN,WM,\theta )=\frac{1}{PC(\theta )}exp(-Enr(MN,WM,\theta ))$$

(13)

$$QC(\theta )={\sum }_{MN} {\sum }_{WM} Enr(MN,WM,\theta )$$

(14)

where $QC(\theta )$ denotes a regularisation constant. In the network, the energy function acts as a probability distribution, and Equation (12) can be used to modify the training vector. It is not recommended to use the RBN’s hidden layers alone to extract features from the data [105,106,107,108,109]. The output of the RBN from layer one serves as the input for layer two, and layer two’s output serves as the input for layer three. A hierarchical approach to DBN, which is created by the hierarchical layer-by-layer RBN structure, is more effective in extracting characteristics from the dataset [110,111,112]. The DBN architecture is displayed in Figure 10. Also, Algorithm 1 shows the steps in classification via DBN.

Algorithm 1: Classification via DBN

Input: $I:$ I = {i1, i2, ……, in}; image fames Output: C $=\left(n_0,n_1, \dots \dots ,n_N\right)$: the classification; $D$ ← []: Vehicle Detections $F$ ← []: Feature Vector Method: Video = VideoReader (‘videopath’) img_frame = read (video) for k = 1 to size (img_frame)     resize_img = imresize (img_framek, 768 × 768)     seg_img = FCM (resize_img)                   $D$ ← YOLOv8 (seg_img)                   for s = 1 to size D                                     F ← SIFT (Ds)                                     F ← KAZE (Ds)                                     F ← ORB (Ds)                                     veh-class = DBN (F)                   end for                  return veh-class return img_frame

4. Experimental Setup and Evaluation

Experiments were conducted on a computer with the specs Intel Core i5-7200U 2.30 GHz processor, 8 GB RAM, and x64-based Windows 10. Results were obtained using Google Colab. The system examined the proposed architecture’s performance on three benchmark datasets called: VEDAI, and VAID datasets. To evaluate the dependability of our suggested system, the k-fold cross-validation is applied on all three datasets. This section describes the dataset, details the trials, and compares the system to other state-of-the-art technologies.

4.1. Dataset Description

4.1.1. VEDAI Dataset

The VEDAI [113] is a public dataset for vehicle detection in aerial imagery. It was proposed in 2015. The collection aids researchers in locating cars in aerial photographs. The dataset contains miniature automobiles with a variety of properties, including variable lighting conditions, shadows, and obstructed objects. In this dataset, vehicles are classified into nine separate categories: “car”, “truck”, “pick-up”, “plane”, “tractor”, “boat”, “camping car”, “van”, and the “other” category. The average number of cars is 5.5, and they take up around 0.7% of the total number of pixels in each photograph. It also includes a common technique for replicating and contrasting the findings of other studies. Figure 11 shows some of the images from the VEDAI dataset.

4.1.2. VAID Dataset

The VAID dataset [114] included 6000 vehicle photos that were divided into seven categories, including minibus, truck, cement truck, sedan, pickup, bus, trailer, and truck. These images were taken by a drone in various lighting situations. The drone was positioned between 90 m and 95 m above the ground. Images taken at 23.98 frames per second have a resolution of 2720 × 1530. Ten locations in southern Taiwan’s traffic and road conditions are included in the dataset. The traffic images show an urban setting, a suburban city, and a university campus. Figure 12 shows sample photos from the VAID dataset.

4.2. Performance Metric and Experimental Outcome

The studies demonstrated the efficiency of the proposed system after we analyzed its performance across the two datasets. Figure 13 and Figure 14 represent the classification accuracies of both the datasets.

Table 2. Overall accuracy, precision, recall, and F1-score for vehicle detection over the VEDAI dataset.

Vehicle Class	Precision	Recall	F1-Score
Pickup	0.985	0.967	0.975
Tractor	0.991	0.987	0.988
Vans	0.941	0.958	0.949
s	0.907	0.910	0.908
Truck	0.934	0.971	0.952
Camping Car	0.956	0.945	0.950
Plane	0.977	0.936	0.956
Boat	0.965	0.971	0.968
Others	0.962	0.934	0.947
Mean	0.957	0.953	0.955

and

Table 3. Overall accuracy, precision, recall, and F1-score for vehicle detection over the VAID dataset.

Vehicle Class	Precision	Recall	F1-Score
Sedan	0.963	0.974	0.968
Minibus	0.986	0.965	0.975
Truck	0.975	0.989	0.982
PickupTruck	0.988	0.946	0.967
Bus	0.941	0.978	0.959
Cement Truck	0.944	0.912	0.927
Trailer	0.973	0.956	0.964
Car	0.945	0.901	0.922
Mean	0.964	0.953	0.958

demonstrated the vehicle detection accuracies, precision, recall, and F1-score.

Table 4. Confusion matrix for vehicle classification by proposed approach on the VEDAI dataset.

Vehicle Class	Pickup	Tractor	Vans	Car	Truck	Camping Car	Plane	Boat	Others
Pickup	0.98	0	0	0	0	0	0	0	0
Tractor	0.02	0.97	0	0	0	0.01	0	0	0
Vans	0	0.01	0.95	0.02	0	0.02	0	0	0
Car	0	0	0.04	0.93	0	0.02	0	0	0.01
Truck	0	0.03	0	0	0.97	0	0	0	0
Camping Car	0.02	0	0.03	0.02	0.01	0.92	0	0	0
Plane	0	0	0	0	0	0	0.96	0	0.04
Boat	0	0	0	0	0	0	0.01	0.95	0.04
Others	0	0	0	0	0	0	0.01	0.02	0.97
Mean = 95.6%

Highlights show the score for correct classification for each class.

and

Table 5. Confusion matrix for vehicle classification by proposed approach on the VAID dataset.

Vehicle Class	Sedan	Minibus	Truck	Pickup Truck	Bus	Cement Truck	Trailer	Car
Sedan	0.98	0.01	0.01	0	0	0	0	0
Minibus	0	0.95	0.02	0	0.03	0	0	0
Truck	0	0.01	0.99	0	0	0	0	0
Pickup Truck	0	0.01	0	0.96	0.02	0	0.01	0
Bus	0.01	0.02	0	0	0.97	0	0	0
Cement Truck	0.01	0	0	0	0	0.99	0	0
Trailer	0.01	0	0	0.01	0	0.01	0.98	0
Car	0.03	0.01	0.01	0	0	0	0.02	0.93
Mean = 94.6%

Highlights show the score for correct classification for each class.

illustrates the confusion matrices for the VEDAI and VAID dataset achieving an accuracy of 95.6% and 94.6%, respectively. The experiments were repeated to assess the effectiveness of the findings. The comparison of our system with other widely used research models is shown in

Table 6. Comparison of the proposed method with conventional systems over VEDAI and VAID Datasets.

Methods	VEDAI	VAID
Wang et al. [115]	93.96	-
Mandal et al. [116]	51.95	-
Terrail et al. [117]	83.50	-
Wang et al. [118]	91.27	-
Lin et al. [114]	-	89.3
Rafique et al. [1]	92.2	-
Hou et al. [119]	75.54	-
Our proposed Model	95.6	94.6

.

The results and comparison with other models show that our model performs well in detection and classification of the vehicles in aerial images. Additionally, YOLOv8 is an efficient algorithm in detecting objects of different sizes and appearance. Moreover, the classification accuracy can be improved further by extracting more useful features that are based on the texture and shape of the objects.

5. Conclusions

This study proposes an innovative method for identifying and categorizing vehicles in aerial image sequences. The model preprocesses the aerial images for noise removal before the detection phase. To reduce the complexity, all the images are segmented using the FCM segmentation technique. The vehicle detection task is accomplished using the YOLOv8 algorithm. All the detected vehicles are subjected to SIFT, KAZE, and ORB feature extraction. The extracted feature is then used to train the DBN classifier to classify vehicles into their corresponding classes. The proposed technique has produced promising results over both datasets. The accuracy attained over the VEDAI dataset is 95.6%, and on VAID it was 94.6%.

The proposed system needs to be trained with more vehicle classes. Also, more features can be added to improve the classification accuracy of the vehicles. In the future, to increase the efficiency of our system and make it a standard for all traffic environments, we intend to add more features and reliable algorithms.