Next Article in Journal
Error Correction of the RapidEye Sub-Pixel Correlation: A Case Study of the 2019 Ridgecrest Earthquake Sequence
Previous Article in Journal
Optically Transparent Frequency Selective Surfaces with Wideband Capability for IoT Applications: A Polarization-Independent Double-Layer Design
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Research on Vehicle Pose Detection Method Based on a Roadside Unit

by
Juan Ni
1,
Xiangcun Kong
1,
Bingchen Yan
2,
Shuzhe Si
3,
Shuyue Shi
1,
Dong Guo
1,
Pengwei Wang
1,
Lei Wang
3,4 and
Yi Xu
1,5,*
1
School of Transportation and Vehicle Engineering, Shandong University of Technology, Zibo 255000, China
2
Zibo Environmental Pollution Control and Prevention Center, Zibo 255000, China
3
Cestera Motor Co., Ltd., Zibo 255000, China
4
Realepo Technology Co., Ltd., Zibo 255000, China
5
Qingte Group Co., Ltd., Qingdao 266106, China
*
Author to whom correspondence should be addressed.
Sensors 2024, 24(14), 4725; https://doi.org/10.3390/s24144725 (registering DOI)
Submission received: 27 June 2024 / Revised: 17 July 2024 / Accepted: 18 July 2024 / Published: 21 July 2024
(This article belongs to the Section Vehicular Sensing)

Abstract

:
Vehicle pose detection plays a vital role in modern automotive technology, which can improve driving safety, enhance vehicle stability and provide important support for the development of autonomous driving technology. The current pose estimation methods have the problems of accumulation errors, large algorithm computing power, and expensive cost, so they cannot be widely used in intelligent connected vehicles. This paper proposes a vehicle pose detection method based on an RSU (Roadside Unit). First, the on-board GPS performs the positioning of the target vehicle and transmits the positioning information to the RSU through the UDP (User Data Protocol). Next, the RSU transmits a forward command to the OBU (On-board Unit) through the UDP. The OBU sends the command to the ECU (Electronic Control Unit) to control the vehicle forward. Then, the RSU detects and tracks the vehicle. The RSU takes pictures of two images before and after the movement and obtains the coordinates of the four angle points and the center point by image processing. The vehicle heading direction is determined by the moving direction of the center point of the front and rear two images. Finally, the RSU captures the vehicle images in real time, performs the process of tracking, rectangular fitting and pose calculation to obtain the pose information and transmits the information to the OBU to complete the whole process of vehicle pose detection and information transmission. Experiments show that the method can realize accurate and efficient detection of vehicle pose, meet the real-time requirements of vehicle pose detection, and can be widely used in intelligent vehicles.

1. Introduction

As a key vehicle perception technology, vehicle pose detection can accurately identify and understand the movement state of the vehicle, so that the driver or the intelligent driving vehicle can make more accurate and safe driving decisions. In the field of intelligent driving, vehicle pose detection can also provide key data support for automatic driving systems, and improve the perception of road conditions and obstacles, so as to better cope with the complex traffic environment. Therefore, the research on vehicle pose detection is necessary and meaningful. The camera has the advantages of easy installation, strong adaptability to the environment and low cost, so it has an important application value in vehicle pose detection. At the same time, with the rapid development of V2X (Vehicle to Everything) communication technology, RSU (Roadside Unit) plays an increasingly important role in intelligent transportation systems. Through its positional advantages, extensive coverage, and efficient data transmission capabilities, RSU provides strong support for realizing intelligent driving and intelligent transportation systems. The vehicle pose detection method based on the RSU can provide reliable pose information in real time due to the simplicity of its solution method and the high camera frame rate. With the advantages of low environmental requirements, strong adaptability, and low cost, this method can compensate for the disadvantages of easily losing GPS signals, the high cost of radar, and sensor drift, making it widely applicable.
This paper proposes a vehicle pose detection method based on the RSU, which can obtain vehicle pose information in real time and with high precision, laying a foundation for the vehicle in stability control, driving safety, and path planning. The remaining structure of the paper is as follows. The second section discusses the current vehicle pose detection methods. The third section details the rectangular fitting method, the vehicle pose calculation method, and the vehicle heading direction detection method. The fourth section shows the training results of target detection and the experiment results of this method in the real vehicle scenarios. In the fifth section, the proposed method is summarized and the subsequent research is also presented.

2. Related Works

Accurate vehicle pose information is the key component of autonomous driving systems. Scholars, both domestically and internationally, have conducted extensive research on vehicle pose detection methods. At present, the mainstream vehicle pose detection methods are the camera-based methods, the lidar-based methods, and the multi-sensor combination algorithms.
Because of the ability of cameras to capture rich image information and the increasing maturity of deep learning algorithms, camera-based vehicle pose detection methods are widely used. The camera-based methods can automatically extract and learn abstract features from the data, which have strong adaptability to the vehicle pose changes under different angles and light conditions. Xu et al. [1] integrated deep learning with enhanced digital maps to accurately estimate vehicle heading angles. Huang et al. [2] presented a lightweight deep learning framework for effective prediction performance in unobscured vehicle scenarios. Gupta et al. [3] introduced an efficient vehicle pose estimation architecture based on multi-scale deep learning for precise estimation with manageable network complexity. Zhao et al. [4] proposed a deep learning-based FPN method for monocular camera-based vehicle pose estimation to improve speed and accuracy. Lyu et al. [5] used the vehicle 3D priors to estimate the 6D pose and improved the detection accuracy and speed. The camera-based methods rely on large amounts of labeled data and have limited generalization ability in specific scenarios, while having a high demand for computational resources.
The lidar-based methods can provide a high precision distance measurement and can accurately capture the details of the vehicle surrounding environment, thus enabling 3D positioning and attitude detection of the vehicle. Wang et al. [6] introduced a lidar-based method for estimating the pose and velocity of the ego vehicle and surrounding moving obstacles simultaneously. Zhao et al. [7] presented a real-time tracking algorithm using L-Shape fitting, enhancing performance and accuracy. Yang et al. [8] proposed a vehicle pose estimation method based on edge distance for boundary rectangle-based pose estimation. Gu et al. [9] developed a registration algorithm with road information constraints, demonstrating accuracy and effectiveness in practical use. The lidar-based vehicle pose detection methods provide superior robustness in complex environments by exploiting 3D geometric information. However, the lidar-based methods may face limitations in some applications due to their uneven density, high processing complexity, and the dependence on precise registration and localization.
The multi-sensor combination algorithms have a wide range of applications, especially playing a key role in environmental perception as well as in navigation and positioning. By integrating data, more comprehensive, accurate, and real-time information acquisition from different sensors is enabled. Wu et al. [10] added a GPS heading angle to the measurement vector, and established its error model, which effectively improved the accuracy of the GPS heading angle. Balazadegan Sarvrood et al. [11] proposed a method that combined visual odometry, light detection and ranging odometry, and a simplified inertial measurement unit. This method achieved an accurate estimation of pose information in urban areas with surrounding buildings. Xu et al. [12] proposed a new multi-information fusion vehicle pose estimation method to achieve precise vehicle pose estimation even during GNSS interruptions. Yang et al. [13] proposed a method which used 4D radar and a camera to robustly estimate vehicle pose to use the complementary characteristics of each sensor. The method showed excellent performance and enhanced robustness. An et al. [14] fused inertial and multi-antenna GNSS measurements to provide higher precision for positioning and attitude estimates. To address the unique demands of safety-critical applications in challenging observation conditions, Li et al. [15] introduced an innovative tightly integrated RTK (Real-Time Kinematic)/INS (Inertial Navigation System) algorithm to ensure the continuous delivery of precise and dependable positioning outcomes. Bersani et al. [16] presented an innovative integrated algorithm that provided an accurate estimation of the vehicle pose for the panning algorithm in multiple challenging scenarios. To identify the pose accurately, CNS (Celestial Navigation System) is commonly combined into the INS/GNSS integration, leading to an INS/GNSS/CNS integrated navigation system [17,18,19,20,21]. The multi-sensor combination algorithms have the advantages of improving accuracy, enhancing robustness and adapting to multiple scenarios, but also face challenges of complexity, cost, and energy consumption.
Based on the above research, the current camera-based vehicle pose detection methods have limited generalization ability in specific scenes and have a high demand for computing resources; the lidar-based methods have a high cost, processing complexity, and severe dependence on accurate registration and positioning; and the multi-sensor combination algorithms have high complexity and are not widely used. In this paper, RSU is used to realize vehicle pose detection, UDP communication is used to realize information transmission between OBU and RSU, and the whole detection process is completed by controlling vehicle forward, vehicle heading direction detection, target detection and tracking, vehicle rectangle fitting, and pose calculation. This method solves the problems of high pose detection power, high cost, and weak real-time performance. With the development of Internet of Vehicles technology, this method has great practical application significance.

3. Method

The vehicle pose detection method based on the RSU can detect the vehicle pose without any prior information and meet the real-time requirements in the actual process of vehicle driving. Figure 1 shows the specific flow of the vehicle pose detection method based on the RSU.
(1)
In order to accurately identify the target vehicle in complex road conditions, the initial position information of the target vehicle is obtained by using the on-board GPS, and the information is transmitted to the RSU through UDP in the OBU.
(2)
After receiving the position information of the on-board GPS, the RSU camera takes pictures of the vehicle on the road and performs the target detection. Since the GPS position information of the vehicle has been obtained, the target vehicle can be extracted from the target detection.
(3)
Preliminary pose calculation is performed on the initial image P0 of the target vehicle to extract the center point coordinate, preparing for determining the vehicle heading direction.
(4)
The RSU sends forward command to the target vehicle through the UDP. After the OBU of the target vehicle receives the command, it sends the command to the ECU to control the vehicle forward.
(5)
The RSU tracks the target vehicle in real time, and performs a preliminary pose calculation for the real-time image Pt to obtain the center point coordinate.
(6)
The vehicle heading direction is judged according to the moving direction of the center point coordinate of the initial image P0 and the center point coordinate of the real-time image Pi after the vehicle advances.
(7)
The small movement distance between the two front and rear images may not be enough to determine the vehicle heading direction due to the high frame rate of the RSU camera. If the vehicle heading direction cannot be judged, return (5). otherwise, continue to track the vehicle in real time.
(8)
The RSU sends the pose information of the vehicle to the OBU in real time to complete the whole process of pose detection.

3.1. Data Interaction Based on the UDP

In this paper, data sharing between the RSU and OBU is realized through the UDP. The UDP has low resource consumption and fast processing speed, and provides connectionless datagram services to terminal devices. To enable real-time information transmission between RSU and OBU for transmitting small data volumes such as vehicle position and heading angle information, this paper employs UDP for data transmission. Firstly, the OBU transmits its own vehicle position information to the RSU through UDP for the RSU to locate the target vehicle for vehicle pose detection. Secondly, after the RSU locates the target vehicle, it uses UDP to send a forward command to the target vehicle. While sending the command, the RSU captures real-time images of the target vehicle to determine the vehicle heading direction. Finally, during the vehicle pose detection phase, the RSU continuously detects the vehicle in real time, calculates the pose, and then transmits the pose information to the OBU.
Before transferring data, the two communicating parties using the UDP do not need to establish a connection, so there is no delay in the connection establishment. With the development of network technology, especially in short-distance transmission areas, UDP data transmit well in real time relative to TCP. Figure 2 shows the format of the UDP message. The UDP message is divided into two parts: UDP message header and UDP data area. The header consists of the source port, the destination port, the message length, and the checksum. When the source host sends the data to the target host, it only needs to send the packet to the IP address and port number of the target host without any pre-connection. If the target host is ready to receive the data, it will receive and process the packet. The application layer protocol will guarantee the reliability of communication. At the same time, because UDP only provides a low level of error control, in the process of data transmission, the transmission delay and the system running memory is small, and the data transmission speed is high, which can be suitable for data transmission in scenarios with high real-time requirements.
Due to the lack of existing UDP communication programs in MATLAB (MathWorks, Natick, MA, USA), the programs are written on the computers of RSU end and OBU end to receive and transmit data. Figure 3 shows the architecture of the communication port. The vehicle pose detection algorithm is provided on the RSU terminal computer, and the OBU terminal computer provides the target vehicle ID, time, and GPS positioning information. All the information is transmitted through the communication port built by the UDP.

3.2. Vehicle Pose Detection

3.2.1. Vehicle Rectangular Fitting

In the top view, the shape of the vehicle is basically rectangular, so the external rectangle of its outline can accurately reflect the pose of the vehicle. Seeing the vehicle as a rectangle can simplify the process of pose estimation and reduce the calculation complexity. The heading angle and position of the vehicle are estimated more accurately by the boundary and position information of the rectangle. Moreover, the rectangular model is suitable for different types of vehicles and can provide stable fitting results at a variety of perspectives and distances. Figure 4 shows the flow chart of the target vehicle rectangular fitting. Figure 5 shows drawing effect of the target vehicle extraction. Figure 6 shows rectangle outline extraction of the target vehicle. Figure 7 shows rectangular fitting of the target vehicle.
(1)
The initial image P0 is used for canny edge detection [22,23,24]. Firstly, Gaussian blur is performed to reduce the effect of noise in the images on edge detection. Secondly, the gradient of the image is calculated using the Sobel operator. Thirdly, the image is processed by non-maximum suppression and refined in the gradient direction to make the edge more refined and accurate. Then, double threshold detection is performed to divide the edge pixels of the image into three categories: strong edge, weak edge, and non-edge. By setting two thresholds, the pixels in the gradient image are divided into high-threshold and low-threshold pixels, where the high-threshold pixels are strong edges and the low-threshold pixels are weak edges. Finally, the weak edge points are connected to connect them with the strong edge points to form the complete edges.
(2)
The image after edge detection is processed by closed operation. The image is corroded to eliminate small holes and small connection areas in the image, making the target more compact. After the corrosion operation, the image is expanded to increase the target in the image and fill the holes in the target to make the target more complete.
(3)
Fully fill in the image. The image after closed operation is completely filled, the convex packets and areas of the image are calculated, the area of the convex packets is calculated again, the target with the largest area is retained, the centroid is extracted for further target analysis and processing, the other targets are deleted, and finally, the filled image Pi of the vehicle is obtained.
(4)
Rotate Pi. Rotate Pi at different angles (from 1° to 90°) to find the minimum external rectangle.
(5)
The bounding box information is calculated for each rotated image, including the upper left coordinate (x, y), width (w), and height (h) of the bounding box.
(6)
Calculate the minimum area angle ω. The area of each rotated image is calculated based on the width and height of the bounding box to find the rotation angle that makes the area of the external rectangle minimum.
(7)
The image Pi is rotated according to the optimal angle found to obtain the final rotated image Pw.
(8)
The coordinates of the four corner points (xn, yn) (n = 1, 2, 3, 4) after rotation are calculated to complete the fitting process of the image minimum external rectangle.
(9)
The coordinates of the four corner points (xrn, yrn) (n = 1, 2, 3, 4) of the external rectangle in the initial image P0 are calculated based on the rotation angle ω, completing the final process of rectangular fitting.

3.2.2. Vehicle Pose Calculation Model

The roadside equipment end mainly detects the environment within its visual range through the camera loaded on the road experiment unit. The installation height and the pose angle of the camera can be set in advance. Figure 8 is the RSU camera model, which shows the position relationship between the RSU camera and the road plane. Xc, Yc, and Zc are the axes of the camera coordinate system. Xw, Yw, and Zw are the axes of the world coordinate system. The installation height of the RSU camera is h.
The vehicle heading angle refers to the angle between the center speed of the vehicle and the horizontal axis under the world coordinate system. At low speeds, the movement of the vehicle in the vertical direction is usually negligible, so the center velocity direction of the vehicle is consistent with the longitudinal central axis of the vehicle. After obtaining the vehicle rectangular fitting box, the coordinates of the four corner points in the image coordinate system can be obtained: A1 (xc1, yc1), B1 (xc2, yc2), C1 (xc3, yc3), and D1 (xc4, yc4). Converting the coordinates of the four corner points into the world coordinate system, A (xw1, yw1), B (xw2, yw2), C (xw3, yw3), and D (xw4, yw4) are obtained. Figure 9 shows the position of the four angle points of the vehicle in the world coordinate system. Through the coordinates of the four angle points, the angle between the central axis L and the horizontal coordinate axis, which is the calculated heading angle θ and the vehicle center point O, taken as the positioning point, can be obtained.
tan θ = y w 4 + y w 3 y w 2 y w 1 x w 4 + x w 3 x w 2 x w 1
θ = arctan y w 4 + y w 3 y w 2 y w 1 x w 4 + x w 3 x w 2 x w 1
x w , y w = x w 1 + x w 2 + x w 3 + x w 4 4 , y w 1 + y w 2 + y w 3 + y w 4 4
In practice, there is a difference between the calculated heading angle and the actual heading angle, so it is stipulated that when the vehicle heading orientation is consistent with the front half axis of the Xw in the world coordinate system, the heading angle is 0°, and the clockwise heading angle is positive. Then, the relationship between calculated heading angle θ and actual heading angle φ is shown in Figure 10.

3.2.3. Vehicle Heading Direction Detection Model

The judgment of the vehicle heading direction is an important process of the vehicle heading angle detection. In this paper, the forward command is sent to the vehicle through communication to control the vehicle forward and the vehicle heading direction is judged by the moving direction of the vehicle center point. When the vehicle takes the rectangular fitting, the movement direction of the center point of the vehicle may not be on the same straight line, that is, there is an angle. In this case, the movement direction of the center point of the vehicle cannot be simply used as the vehicle heading direction, and the vehicle heading direction needs to be accurately judged to calculate the pose. Figure 11 shows the specific steps. Figure 12 shows the schematic diagram of the vehicle heading direction judgment.
(1)
Calculate the initial image pose. Obtain the initial image P0, conduct the preliminary pose calculation for it, and obtain the center point coordinate C0.
(2)
Calculate the real-time image pose. Conduct the preliminary pose calculation for the real-time image Pt to obtain the center point coordinate Ct, and the four side midpoint coordinates C1, C2, C3, C4.
(3)
Calculate the displacement vector v0 and the side midpoint vector vn. The center point displacement vector v 0 = C t C 0 . Calculate the side midpoint vectors v1, v2, v3, v4 from the center point of the vehicle Ct obtained through the real-time bounding box to the midpoints of the four sides of the rectangle C1, C2, C3, C4.
v n = C n C t   ( n = 1 , 2 , 3 , 4 )
(4)
Calculate the minimum clip angle β. Calculate the angles a1, a2, a3, a4 between the displacement vector v0 and the side midpoint vectors v1, v2, v3, v4.
a n = arccos v 0 v n v 0 v n   ( n = 1 , 2 , 3 , 4 )
(5)
The minimum vector clip angle β is obtained:
β = min a 1 , a 2 , a 3 , a 4
(6)
Determine the vehicle heading direction. The vector direction corresponding to the minimum clip angle β is the vehicle heading direction.

4. Experiment and Results Analysis

4.1. Object Detection

4.1.1. Dataset Training

The vehicle pose detection method proposed in this paper calculates the vehicle pose by vehicle rectangular fitting. In order to identify the target vehicles in the traffic scenes, it is necessary to detect the vehicles from the overhead perspective. The current vehicle detection algorithms cannot meet the practical application requirements of the vehicle detection from the overhead perspective, so the dataset of the vehicles from the overhead perspective is collected and trained to realize the detection of the target vehicles. The dataset used in this paper consists of 3190 vehicle images captured by UAV, including images with occlusions and scale variations. The training set comprises 1724 images, the validation set comprises 175 images, and the test set comprises 1291 images. Part of the dataset images is shown in Figure 13.
During the training process, the model was optimized using the stochastic gradient descent optimizer. The initial learning rate is 0.01, the MiniBatchSize is set to 20, the MaxEpochs is set to 300, and the input resolution of the image is 1530 × 2720 [25,26,27]. Figure 14 shows the training loss for each iteration.

4.1.2. Training Accuracy

In order to evaluate the training effect more accurately, the evaluation indexes such as detection accuracy, true accuracy, and overall accuracy were used to evaluate the model performance. The calculation method is shown in Equations (7)–(9):
O T = F N + F P T P + F P + T N + F N
D T = F P T P + F P
R T = F P T N + F P
TP is the number of samples detected as vehicles and actually vehicles. FP is the number of samples detected as vehicles but not actually vehicles. TN is the number of samples detected as not vehicles but actually vehicles and FN is the number of samples detected as not vehicles and actually not vehicles. According to the detection results, the confusion matrix [28,29,30,31,32] was obtained as shown in Table 1.
It can be seen from Table 2 that the effect of the training for vehicles detection is remarkable, which can meet the needs of target vehicle identification for pose detection.

4.2. Pose Detection Experiment of the Real Vehicle Scenarios

The experiment was operated as follows. The UAV and the terminal computer form the RSU equipment, in which the UAV camera is the RSU camera, and the terminal computer realizes the image and data processing and communicates with the vehicle OBU. To enable the UAV to simulate the RSU as much as possible, the experiment was conducted in a windless environment. The UAV flies to a certain height and then hovers. The UAV camera should be kept parallel to the ground, namely, the pitch angle is 0°. When the vehicle enters the RSU perception area, the OBU sends the on-board GPS position information to the RSU through UDP. After the RSU accepts the information, the vehicle is obtained and the initial image obtained by the RSU is sent to the RSU through Raspberry PI for processing. The RSU computer first conducts the target detection. By matching the result of the target detection and the GPS information, the detection target is considered as the actual target vehicle. The preliminary pose calculation of the target vehicle was performed to obtain the coordinate of the vehicle center point. At this point, the RSU sends a forward command to the vehicle, and the OBU receives the command and forwards it to the ECU to control the vehicle to move forward. The RSU tracks the vehicle and completes the preliminary pose calculation of the vehicle in real time to obtain the coordinate of the vehicle center point. By comparing the initial center point coordinate of the vehicle with the real-time center point coordinate after moving forward, the vehicle heading direction can be determined. Subsequently, the RSU continues to track the vehicle, detect its pose, obtain the center coordinates of the vehicle and the heading angle information, and transmit the information to the OBU, completing the entire process of vehicle pose detection. To verify the effectiveness of the vehicle pose detection method based on the RSU under different driving modes, the experiments were conducted in both straight-line and turning driving states. Table 3 shows UAV parameters, Figure 15 shows experimental vehicle equipment, Figure 16 shows RSU equipment and Figure 17 shows the relative position of the UAV and the camera.
The vehicle heading direction detection experiments were conducted. Figure 18 shows the system equipment (the red represents device name and installation location, the blue represents communication mode, and the black represents data transmission content). The experiment scenarios are shown in Figure 19 (the red arrow indicates the driving direction of the vehicle). Figure 19a shows the actual experiment route of the straight-line scenario (where the green arrow indicates the heading direction and the red arrow is the moving direction of the center point); Figure 19b shows the map identification route of the straight-line scenario; Figure 19c shows the actual experiment route of the turning scenario; Figure 19d shows the map identification route of the turning scenario. The experiment results are shown in Figure 20 (the green arrow indicates the heading direction and the red arrow is the moving direction of the center point). Figure 20a–c show the detection effect of the head with no deviation between the moving direction of the center point and the heading direction; Figure 20d–i are the detection effect of the heading direction with different deviations. As can be seen from the figures, the heading direction detection method has a significant effect. Figure 21, Figure 22 and Figure 23 shows the results of target vehicle detection, target vehicle tracking and target vehicle pose calculation, respectively. Figure 24 shows the communication results.

4.3. Detection Speed Analysis

The average time used by the vehicle pose detection method based on the RSU and the GPS method was compared to obtain the pose detection data, and the detection speed of the two methods was compared. The detection time is shown in Table 4.
Table 4 shows that the vehicle pose detection method based on the RSU takes less time and detects faster when obtaining the pose detection data than the GPS-based method. This is because this method obtains many image data, and the pose solution process is simple, while data obtained by GPS have a low frequency, and there is a delay. Therefore, the detection speed of the vehicle pose detection method based on the RSU method is fast.

4.4. Detection Accuracy Analysis

The trajectory and heading angle plots obtained from the vehicle pose detection method based on the RSU and the GPS-based method are shown in the figure below. Figure 25 and Figure 26 shows the results of the straight-line scenario and the turning scenario, respectively. Due to the different data acquisition frequencies of this method, GPS, and RTK (which has a lower frequency), interpolation was performed on the RTK data to facilitate error analysis. As the data trend obtained in this process from RTK is relatively regular, linear interpolation is used to interpolate the data according to the frequencies of RSU and GPS to obtain the interpolated results. A comparison is then made between the interpolated results and the detection results from RSU and GPS, generating error plots. A detailed analysis of the errors is presented in the table below. The columns in the table represent max for maximum error, min for minimum error, mean for average error, std for standard deviation, and rmse for root mean square error. All values are in meters.
From the Table 5, it can be seen that whether in the straight-line scenario or the turning scenario, the performance of the vehicle pose detection method based on the RSU is overall better than the GPS-based method. This is because the vehicle pose detection method based on the RSU has strong adaptability to the environment and a faster detection speed, enabling it to capture the changes in the vehicle pose during driving in a timely manner. On the other hand, GPS has positioning delays, making it unable to reflect real-time changes in the vehicle pose. Additionally, during turning maneuvers where the environment changes significantly, GPS accuracy is affected by environmental factors, leading to decreased precision. The method presented in this paper exhibits some jitter phenomenon, which is attributed to the unstable process of rectangular fitting, particularly in the fitting of the rearview mirror. The instability occurs during the rectangular fitting process; where both rearview mirrors may be fitted, only one rearview mirror may be fitted, or no rearview mirror may be fitted, thereby affecting the precise detection of trajectory points. Furthermore, during the target detection process, parts of the environment that do not belong to the vehicle are also detected, leading to the extraction of other feature points during pose detection and impacting the final detection results. In summary, the vehicle pose detection method based on the RSU offers a higher accuracy and faster detection speed, making it suitable for real-world vehicle scenarios.

5. Conclusions

This paper presents a pose detection method based on the RSU method. In the process of vehicle pose calculation, the paper analyzes the pose information of vehicles with different heading orientations to avoid pose misjudgments caused by vehicle orientation. Comparison experiments were conducted outdoors between the GPS-based pose detection method and the method in this paper. The experimental results of the position and heading angle were compared using mean, max, min, std, and rmse as evaluation metrics. The results indicate that, due to the simplicity and stability of the detection method in this paper, it has a high accuracy. The detection speed of the GPS-based pose detection method was compared with this method, and the processing speed of this method is faster due to the high frame rate of the RSU camera.
The method presented in this paper has the advantages of high accuracy and fast processing speed, and is suitable for various vehicle scenarios such as parking lots, areas with dense and high buildings, and cargo yards. This paper is innovative in the pose detection field, using the method based on the RSU to realize the accurate detection of the vehicle pose. However, the security of the data transmission process in this paper is still insufficient. The subsequent research will focus on the security direction of the data transmission process, and ensure the security of the data transmission process by strengthening the encryption algorithm, adopting two-factor authentication, and establishing the secure data channels to confront the increasingly severe cyber security threats. The problems of increased transmission time brought by these measures will also be considered. At the same time, regarding the increased communication delay that may arise from a single RSU serving multiple OBUs, multi-channel and time allocation, priority and quality of service management, and channel management and resource allocation will be considered to improve the performance of the method proposed in the paper.

Author Contributions

Conceptualization, J.N. and Y.X.; methodology, J.N.; software, J.N., Y.X., S.S. (Shuzhe Si), X.K. and B.Y.; validation, J.N., B.Y., S.S. (Shuzhe Si), D.G. and P.W.; investigation, B.Y., S.S. (Shuzhe Si), L.W. and Y.X.; resources, J.N. and Y.X.; data curation, X.K., B.Y., S.S. (Shuyue Shi), P.W. and L.W.; writing—original draft preparation, J.N. and S.S. (Shuyue Shi); writing—review and editing, J.N. and Y.X.; supervision, J.N., B.Y., S.S. (Shuyue Shi), D.G. L.W. and Y.X.; project administration, L.W. and Y.X.; funding acquisition, L.W. and Y.X. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Shandong Province Major Science and Technology Innovation Project (grant number: 2023CXGC010111), Shandong Province Youth Innovation Team Plan for Universities (grant number: 2023KJ153), Small and Medium-sized Enterprise Innovation Capability Improvement Project (grant number: 2022TSGC2277), Intelligent Automotive Sensor Technology Graduate Project (grant number: 4053223015), Shandong Provincial Program of Introducing and Cultivating Talents of Discipline to Universities (grant number: 4012322006), Natural Science Foundation of Shandong Province (grant number: ZR2022MF230), National Natural Science Foundation of China (grant number: 52102465), Experiment technology upgrading project (grant number: 2022003).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

Author Lei Wang was employed by the company Cestera Motor Co., Ltd. and the company Realepo Technology Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Author Yi Xu was employed by the company Qingte Group Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Xu, Q.; Li, X.; Sun, Z.; Hu, W.; Chang, B. A novel heading angle estimation methodology for land vehicles based on deep learning and enhanced digital map. IEEE Access 2019, 7, 138567–138578. [Google Scholar] [CrossRef]
  2. Huang, W.; Li, W.; Tang, L.; Zhu, X.; Zou, B. A deep learning framework for accurate vehicle yaw angle estimation from a monocular camera based on part arrangement. Sensors 2022, 22, 8027. [Google Scholar] [CrossRef] [PubMed]
  3. Gupta, D.; Artacho, B.; Savakis, A. VehiPose: A multi-scale framework for vehicle pose estimation. In Proceedings of the Applications of Digital Image Processing XLIV, San Diego, CA, USA, 1–5 August 2021; pp. 517–523. [Google Scholar]
  4. Zhao, H.; Tao, B.; Huang, L.; Chen, B. A siamese network-based approach for vehicle pose estimation. Front. Bioeng. Biotechnol. 2022, 10, 948726. [Google Scholar] [CrossRef] [PubMed]
  5. Lyu, Y.; Royen, R.; Munteanu, A. Mono6d: Monocular vehicle 6d pose estimation with 3d priors. In Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 16–19 October 2022; pp. 2187–2191. [Google Scholar]
  6. Wang, Q.; Chen, J.; Deng, J.; Zhang, X.; Zhang, K. Simultaneous pose estimation and velocity estimation of an ego vehicle and moving obstacles using LiDAR information only. IEEE Trans. Intell. Transp. Syst. 2021, 23, 12121–12132. [Google Scholar] [CrossRef]
  7. Zhao, C.; Fu, C.; Dolan, J.M.; Wang, J. L-shape fitting-based vehicle pose estimation and tracking using 3D-LiDAR. IEEE Trans. Intell. Veh. 2021, 6, 787–798. [Google Scholar] [CrossRef]
  8. Yang, J.; Zeng, G.; Wang, W.; Zuo, Y.; Yang, B.; Zhang, Y. Vehicle Pose Estimation Based on Edge Distance Using Lidar Point Clouds (Poster). In Proceedings of the 22th International Conference on Information Fusion (FUSION), Ottawa, ON, Canada, 2–5 July 2019; pp. 1–6. [Google Scholar]
  9. Gu, B.; Liu, J.; Xiong, H.; Li, T.; Pan, Y. ECPC-ICP: A 6D vehicle pose estimation method by fusing the roadside lidar point cloud and road feature. Sensors 2021, 21, 3489. [Google Scholar] [CrossRef] [PubMed]
  10. Wu, Y.; Gao, T.; Ding, Y. A low-cost GPS/INS integrated vehicle heading angle measurement system. AIP Conf. Proc. 2018, 1955, 040097. [Google Scholar]
  11. Balazadegan, S.Y.; Hosseinyalamdary, S.; Gao, Y. Visual-LiDAR odometry aided by reduced IMU. ISPRS Int. J. Geo-Inf. 2016, 5, 3. [Google Scholar] [CrossRef]
  12. Xu, Q.; Tian, Y.; Ruan, G.; Chang, B. Vision-aided intelligent and adaptive vehicle pose estimation during GNSS outages. Meas. Sci. Technol. 2024, 35, 045106. [Google Scholar] [CrossRef]
  13. Yang, S.; Choi, M.; Han, S.; Choi, K.-H.; Kim, K.-S. 4D Radar-Camera Sensor Fusion for Robust Vehicle Pose Estimation in Foggy Environments. IEEE Access 2024, 25, 6223–6237. [Google Scholar] [CrossRef]
  14. An, X.; Bellés, A.; Rizzi, F.G.; Hösch, L.; Lass, C.; Medina, D. Array ppp-rtk: A high precision pose estimation method for outdoor scenarios. IEEE Trans. Intell. Transp. Syst. 2023, 12, 16178–16188. [Google Scholar] [CrossRef]
  15. Li, W.; Li, W.; Cui, X.; Zhao, S.; Lu, M. A tightly coupled RTK/INS algorithm with ambiguity resolution in the position domain for ground vehicles in harsh urban environments. Sensors 2018, 18, 2160. [Google Scholar] [CrossRef] [PubMed]
  16. Bersani, M.; Mentasti, S.; Cudrano, P.; Vignati, M.; Matteucci, M.; Cheli, F. Robust vehicle pose estimation from vision and INS fusion. In Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 20–23 September 2020; pp. 1–6. [Google Scholar]
  17. Hu, G.; Gao, S.; Zhong, Y.; Gao, B.; Subic, A. Modified federated Kalman filter for INS/GNSS/CNS integration. Proc. Inst. Mech. Eng. Part G-J. Aerosp. Eng. 2016, 230, 30–44. [Google Scholar] [CrossRef]
  18. Hu, G.; Gao, S.; Zhong, Y.; Gao, B.; Subic, A. Matrix weighted multisensor data fusion for INS/GNSS/CNS integration. Proc. Inst. Mech. Eng. Part G-J. Aerosp. Eng. 2016, 230, 1011–1026. [Google Scholar] [CrossRef]
  19. Hu, G.; Xu, L.; Gao, B.; Chang, L.; Zhong, Y. Robust unscented Kalman filter-based decentralized multisensor information fusion for INS/GNSS/CNS integration in hypersonic vehicle navigation. IEEE Trans. Instrum. Meas. 2023, 72, 8504011. [Google Scholar] [CrossRef]
  20. Gao, B.; Hu, G.; Gao, S.; Zhong, Y.; Gu, C. Multi-sensor optimal data fusion for INS/GNSS/CNS integration based on unscented Kalman filter. Int. J. Control Autom. Syst. 2018, 16, 129–140. [Google Scholar] [CrossRef]
  21. Gao, B.; Hu, G.; Zhong, Y.; Zhu, X. Distributed state fusion using sparse-grid quadrature filter with application to INS/CNS/GNSS integration. IEEE Sens. J. 2021, 22, 3430–3441. [Google Scholar] [CrossRef]
  22. Ansaripour, A.; Heydariaan, M.; Gnawali, O.; Kim, K. Viper: Vehicle pose estimation using ultra-wideband radios. In Proceedings of the 2020 16th International Conference on Distributed Computing in Sensor Systems (DCOSS), Marina del Rey, CA, USA, 25–27 May 2020. [Google Scholar]
  23. Trivedi, J.; Devi, M.S.; Dhara, D. Canny edge detection based real-time intelligent parking management system. Sci. J. Silesian Univ. Technol. Ser. Transp. 2020, 106, 197–208. [Google Scholar] [CrossRef]
  24. Zhang, T.; Gao, H. Detection Technology of Traffic Marking Edge. In Proceedings of the 2020 IEEE 2nd International Conference on Civil Aviation Safety and Information Technology (ICCASIT), Weihai, China, 14–16 October 2020; pp. 608–611. [Google Scholar]
  25. Zhang, J.; Huang, M.; Jin, X.; Li, X. A real-time Chinese traffic sign detection algorithm based on modified YOLOv2. Algorithms 2017, 10, 127. [Google Scholar] [CrossRef]
  26. Sang, J.; Wu, Z.; Guo, P.; Hu, H.; Xiang, H.; Zhang, Q.; Cai, B. An improved YOLOv2 for vehicle detection. Sensors 2018, 18, 4272. [Google Scholar] [CrossRef]
  27. Tang, T.; Deng, Z.; Zhou, S.; Lei, L.; Zou, H. Fast vehicle detection in UAV images. In Proceedings of the 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP), Shanghai, China, 18–21 May 2017; pp. 1–5. [Google Scholar]
  28. Liu, Y.; Wei, X. Track surface defect detection based on image processing. In Proceedings of the International Conference on Electrical and Information Technologies for Rail Transportation, Changsha, China, 20–22 October 2017; pp. 225–232. [Google Scholar]
  29. Pommé, L.E.; Bourqui, R.; Giot, R.; Auber, D. Relative confusion matrix: Efficient comparison of decision models. In Proceedings of the 2022 26th International Conference Information Visualisation (IV), Vienna, Austria, 19–22 July 2022; pp. 98–103. [Google Scholar]
  30. Riehl, K.; Neunteufel, M.; Hemberg, M. Hierarchical confusion matrix for classification performance evaluation. J. R. Stat. Soc. Ser. C-Appl. Stat. 2023, 72, 1394–1412. [Google Scholar] [CrossRef]
  31. Hong, C.S.; Oh, T.G. TPR-TNR plot for confusion matrix. Commun. Stat. Appl. Methods 2021, 28, 161–169. [Google Scholar] [CrossRef]
  32. Xu, J.; Zhang, Y.; Miao, D. Three-way confusion matrix for classification: A measure driven view. Inf. Sci. 2020, 507, 772–794. [Google Scholar] [CrossRef]
Figure 1. Flow chart of vehicle pose detection method based on the RSU.
Figure 1. Flow chart of vehicle pose detection method based on the RSU.
Sensors 24 04725 g001
Figure 2. UDP message format.
Figure 2. UDP message format.
Sensors 24 04725 g002
Figure 3. Communication port architecture.
Figure 3. Communication port architecture.
Sensors 24 04725 g003
Figure 4. Flow chart of the target vehicle rectangular fitting.
Figure 4. Flow chart of the target vehicle rectangular fitting.
Sensors 24 04725 g004
Figure 5. Drawing effect of the target vehicle extraction (the left, middle, and right images show the image processing effects in different directions).
Figure 5. Drawing effect of the target vehicle extraction (the left, middle, and right images show the image processing effects in different directions).
Sensors 24 04725 g005aSensors 24 04725 g005b
Figure 6. Rectangle outline extraction of the target vehicle.
Figure 6. Rectangle outline extraction of the target vehicle.
Sensors 24 04725 g006
Figure 7. Rectangular fitting of the target vehicle (the left, middle, and right images show the final effects in different directions).
Figure 7. Rectangular fitting of the target vehicle (the left, middle, and right images show the final effects in different directions).
Sensors 24 04725 g007
Figure 8. Schematic diagram of the road camera (RSU camera) model.
Figure 8. Schematic diagram of the road camera (RSU camera) model.
Sensors 24 04725 g008
Figure 9. Schematic diagram of the pose calculation.
Figure 9. Schematic diagram of the pose calculation.
Sensors 24 04725 g009
Figure 10. The schematic diagram of the conversion relationship between actual heading angle and calculated heading angle. (a) The schematic diagram when heading angle is 90°; (b) the schematic diagram when heading angle is 270°; (c,d) the schematic diagram of heading angle calculation in other cases.
Figure 10. The schematic diagram of the conversion relationship between actual heading angle and calculated heading angle. (a) The schematic diagram when heading angle is 90°; (b) the schematic diagram when heading angle is 270°; (c,d) the schematic diagram of heading angle calculation in other cases.
Sensors 24 04725 g010
Figure 11. Flow chart of the vehicle heading direction judgment.
Figure 11. Flow chart of the vehicle heading direction judgment.
Sensors 24 04725 g011
Figure 12. Schematic diagram of the vehicle heading direction judgment. (a), (b) and (c) represent the judgment of the heading direction in different situations, respectively.
Figure 12. Schematic diagram of the vehicle heading direction judgment. (a), (b) and (c) represent the judgment of the heading direction in different situations, respectively.
Sensors 24 04725 g012
Figure 13. Part of the dataset images.
Figure 13. Part of the dataset images.
Sensors 24 04725 g013
Figure 14. Training loss for each iteration.
Figure 14. Training loss for each iteration.
Sensors 24 04725 g014
Figure 15. Experimental vehicle equipment.
Figure 15. Experimental vehicle equipment.
Sensors 24 04725 g015
Figure 16. RSU equipment.
Figure 16. RSU equipment.
Sensors 24 04725 g016
Figure 17. The relative position of the UAV and the camera.
Figure 17. The relative position of the UAV and the camera.
Sensors 24 04725 g017
Figure 18. System equipment.
Figure 18. System equipment.
Sensors 24 04725 g018
Figure 19. Experiment scenarios. (a) shows the actual experiment route of the straight-line scenario; (b) shows the map identification route of the straight-line scenario; (c) shows the actual experiment route of the turning scenario and (d) shows the map identification route of the turning scenario.
Figure 19. Experiment scenarios. (a) shows the actual experiment route of the straight-line scenario; (b) shows the map identification route of the straight-line scenario; (c) shows the actual experiment route of the turning scenario and (d) shows the map identification route of the turning scenario.
Sensors 24 04725 g019
Figure 20. The vehicle heading direction detection. (ac) show the detection effect of the head with no deviation between the moving direction of the center point and the heading direction; (di) are the detection effect of the heading direction with different deviations.
Figure 20. The vehicle heading direction detection. (ac) show the detection effect of the head with no deviation between the moving direction of the center point and the heading direction; (di) are the detection effect of the heading direction with different deviations.
Sensors 24 04725 g020aSensors 24 04725 g020b
Figure 21. Target vehicle detection. (ac) represent the target vehicle detection results at different times, respectively.
Figure 21. Target vehicle detection. (ac) represent the target vehicle detection results at different times, respectively.
Sensors 24 04725 g021
Figure 22. Target vehicle tracking. (ac) represent the target vehicle tracking results at different times, respectively.
Figure 22. Target vehicle tracking. (ac) represent the target vehicle tracking results at different times, respectively.
Sensors 24 04725 g022
Figure 23. Target vehicle pose calculation. (ac) represent the target vehicle pose calculation results at different times, respectively.
Figure 23. Target vehicle pose calculation. (ac) represent the target vehicle pose calculation results at different times, respectively.
Sensors 24 04725 g023
Figure 24. The communication results diagram. (a) The vehicle ID and GPS information received by MATLAB in the RSU; (b,c) the information received by MATLAB in the OBU, which includes the forward command information and pose information, respectively.
Figure 24. The communication results diagram. (a) The vehicle ID and GPS information received by MATLAB in the RSU; (b,c) the information received by MATLAB in the OBU, which includes the forward command information and pose information, respectively.
Sensors 24 04725 g024
Figure 25. Results of the straight-line scenario. (a) The absolute trajectory comparison result of the straight-line scenario; (b) the absolute trajectory error comparison result of the straight-line scenario; (c) the heading angle comparison result of the straight-line scenario; (d) the heading angle error comparison result of the straight-line scenario.
Figure 25. Results of the straight-line scenario. (a) The absolute trajectory comparison result of the straight-line scenario; (b) the absolute trajectory error comparison result of the straight-line scenario; (c) the heading angle comparison result of the straight-line scenario; (d) the heading angle error comparison result of the straight-line scenario.
Sensors 24 04725 g025aSensors 24 04725 g025b
Figure 26. Results of the turning scenario. (a) The absolute trajectory comparison result of the turning scenario; (b) the absolute trajectory error comparison result of the turning scenario; (c) the heading angle comparison result of the turning scenario; (d) the heading angle error comparison result of the turning scenario.
Figure 26. Results of the turning scenario. (a) The absolute trajectory comparison result of the turning scenario; (b) the absolute trajectory error comparison result of the turning scenario; (c) the heading angle comparison result of the turning scenario; (d) the heading angle error comparison result of the turning scenario.
Sensors 24 04725 g026
Table 1. The confusion matrix of the detection results.
Table 1. The confusion matrix of the detection results.
TrueActualNone or False
Detection resultTrue1203 25
None or false29 34
Table 2. Detection accuracy.
Table 2. Detection accuracy.
OTDTRT
YOLOv2 detection0.9580.9800.976
Table 3. Table of UAV parameters.
Table 3. Table of UAV parameters.
CQ 360 UAV Development Platform
VersionCQ360-X8Endurance time25 min
Flying height 21 mPitch angle
Yaw angle5.32°Roll angle0.29°
Battery4 s (16.8 V)Remote control distance1500–2000 m
Dimension300 × 300 × 350 mmWind resistance levelLevel 3–4
Table 4. Comparison of the average detection time.
Table 4. Comparison of the average detection time.
MethodsRSUGPS
Detection time/s0.0630.235
Table 5. Error analysis.
Table 5. Error analysis.
MethodsMaxMinMeanStdRmse
Straight-line track (location) errorRSU0.7320.2130.4220.1430.446
GPS1.6310.751.1630.2261.184
Straight-line heading angle errorRSU3.5110.0141.3131.0381.426
GPS5.0151.8163.1000.6793.085
Turning track (position) errorRSU1.0630.2420.4550.2010.498
GPS2.3230.4951.2730.4171.340
Turning heading angle errorRSU5.6480.0372.3431.6422.060
GPS23.0440.6674.8074.1426.345
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Ni, J.; Kong, X.; Yan, B.; Si, S.; Shi, S.; Guo, D.; Wang, P.; Wang, L.; Xu, Y. Research on Vehicle Pose Detection Method Based on a Roadside Unit. Sensors 2024, 24, 4725. https://doi.org/10.3390/s24144725

AMA Style

Ni J, Kong X, Yan B, Si S, Shi S, Guo D, Wang P, Wang L, Xu Y. Research on Vehicle Pose Detection Method Based on a Roadside Unit. Sensors. 2024; 24(14):4725. https://doi.org/10.3390/s24144725

Chicago/Turabian Style

Ni, Juan, Xiangcun Kong, Bingchen Yan, Shuzhe Si, Shuyue Shi, Dong Guo, Pengwei Wang, Lei Wang, and Yi Xu. 2024. "Research on Vehicle Pose Detection Method Based on a Roadside Unit" Sensors 24, no. 14: 4725. https://doi.org/10.3390/s24144725

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Article metric data becomes available approximately 24 hours after publication online.
Back to TopTop