Perspective Transformer and MobileNets-Based 3D Lane Detection from Single 2D Image

Abstract

Three-dimensional (3D) lane detection is widely used in image understanding, image analysis, 3D scene reconstruction, and autonomous driving. Recently, various methods for 3D lane detection from single two-dimensional (2D) images have been proposed to address inaccurate lane layouts in scenarios (e.g., uphill, downhill, and bumps). Many previous studies struggled in solving complex cases involving realistic datasets. In addition, these methods have low accuracy and high computational resource requirements. To solve these problems, we put forward a high-quality method to predict 3D lanes from a single 2D image captured by conventional cameras, which is also cost effective. The proposed method comprises the following three stages. First, a MobileNet model that requires low computational resources was employed to generate multiscale front-view features from a single RGB image. Then, a perspective transformer calculated bird’s eye view (BEV) features from the front-view features. Finally, two convolutional neural networks were used for predicting the 2D and 3D coordinates and respective lane types. The results of the high-reliability experiments verified that our method achieves fast convergence and provides high-quality 3D lanes from single 2D images. Moreover, the proposed method requires no exceptional computational resources, thereby reducing its implementation costs.

1. Introduction

Large corporations, such as Tesla, Google, Uber, and Nvidia, have been developing autonomous vehicles. To explore different scenarios around these autonomous vehicles, they often employ several sensors, such as cameras, radar, sonar, and LiDAR. However, the main sensor, LiDAR, is quite expensive. Therefore, it is expected to be difficult to apply these systems widely in practice in the near future [1,2,3,4,5]. In contrast, camera-based autonomous driving systems are low-cost alternatives that use computer vision and deep learning techniques [6,7,8].

Three-dimensional (3D) lane detection from two-dimensional (2D) images is an important technique in camera-based autonomous driving applications. Additionally, 3D lane detection is also widely used in other applications including image understanding, image analysis, 3D scene reconstruction [6,7,8,9], and augmented reality applications of outdoor environments [10,11]. The 3D lanes and 3D-based object detection, localization, and tracking are applied in the perception and sensing systems of autonomous driving vehicles [9]. Several studies have investigated 3D lane estimation from single or multiple 2D images. However, these methods have low accuracy and high computational resource requirements (as discussed in Section 2).

MobileNets stands for efficient convolutional neural networks (CNN) for mobile vision applications [12]. This model is efficient for embedded and mobile vision applications. MobileNets, which are based on an elegant architecture, use depth-wise separable convolutions to construct lightweight deep neural networks. The model contains two global hyper-parameters that efficiently obtain a balance between latency and accuracy while choosing the right-sized model for their application relying on the problem constraint. Moreover, MobileNets require low computational resources, such as mobile devices, and exhibit high performance across various applications, including fine-grained classification, face attributes, object detection, and large-scale geolocalization.

Therefore, to overcome limitations such as low accuracy and high computational cost, we present a 3D lane detection method, which is inspired by PersFormer [13], using both a perspective transformer and MobileNets. The MobileNet-based spatial feature extraction reduces the high computational cost, and the spatial feature transformation module in the perspective transformer generates high-quality bird’s eye view (BEV) features from multiscale front-view features. The main contributions of our research are as follows: (1) a high-performance 3D lane detection method using medium hardware is proposed; and (2) the proposed method achieves fast convergence to reduce training time.

The remainder of this study is arranged as follows. A survey of literature related to 2D and 3D lane detection is discussed in Section 2. Section 3 details the proposed 3D lane detection method. The results of the high-reliability experiments and evaluations are presented in Section 4 consisting of both qualitative and quantitative measures. Finally, in Section 5, we provide conclusions and several potential directions for future work.

2. Related Works

The conventional 2D lane-detection method includes three stages. First, the semantic segmentation stage is employed to assign each pixel in a 2D image corresponding to a label to verify whether it places on a lane. Second, a spatial transformation stage projects the output of the image segmentation stage onto a plane surface. The third stage extracts 2D lanes using a lane model fitting with strong assumptions. The model fitting for quadratic curves is the most common model. Traditional 2D lane detection studies are based on image processing techniques, such as filtering [14,15] and clustering [16]. With developments in deep learning, several CNN-based approaches have significantly outperformed traditional methods. Many studies have considered 2D lane detection as a semantic segmentation problem [17,18,19,20,21]. For example, Neven et al. [19] introduced a binary segmentation method requiring a post-clustering process for lane instance discrimination. In addition, Lee et al. [17], Pan et al. [18], and Hou et al. [20] presented multiclass pixel-wise segmentation approaches that limit the maximum detection results in one frame but require large computational resources. Several lightweight methods have been proposed to reduce computational resources based on an effective grid [22,23,24,25] or anchor [26,27,28,29,30]. However, these methods are of poor quality. For example, grid-based research that finds lanes row-wise has a smaller resolution than that of the segmentation map. The model output is the probability of each cell being in a lane, and a vertical post-clustering process generates lane instances. In contrast, the anchor-based studies focus on the offsets optimization of the predefined line anchors, which is based on classical object detection. However, a critical problem which remains is defining these anchors. Chen et al. [26] selected vertical anchors, unsuitable for curving lane prediction, for overcoming this problem. In [27,29,30], anchors with slender tilt shapes were designed. In addition, the authors employed different anchors to improve the detection accuracy (albeit at the cost of increased computational resources).

For 3D lane detection, two trends exist. In the first trend, Nedevschi et al., Benmansour et al., and Bai et al. [31,32,33] employed multimodal or multiview high-cost sensors, such as a stereo camera or LiDAR, to obtain a 3D ground topology. The second involves adopting computer vision and deep learning using images captured from 2D cameras to reduce the cost. Based on the second trend, we put forward a novel method for predicting the 3D lanes.

The authors of [16,19,29] assumed that lanes in the BEV space are plane surfaces such that the lane height is zero. However, this assumption is inaccurate in most autonomous driving environmental scenarios that include uphill/downhill, bumps, and crush turns. Recently, some monocular methods [34,35,36,37] have employed inverse perspective mapping to predict lanes in 3D space from a single 2D image. For example, 3D-LaneNet [34] was a pioneering study with a simple end-to-end neural network. It uses a spatial transformer network [38] to create spatial feature projections but has moderate performance. Gen-LaneNet [35] builds on 3D-LaneNet consisting of a two-stage network for dividing the segmentation encoder and 3D lane generation. However, the Gen-LaneNet performance depends significantly on the binary segmentation stage accuracy, which degrades in extreme weather or night scenarios. 3D-LaneNet and Gen-LaneNet methods also exhibited low performance in curving or crush-turn scenarios. In contrast, the PersFormer method [13] provided better feature representation and optimized the anchor design to detect 2D and 3D lanes. PersFormer is an end-to-end monocular lane detector with a spatial-feature transformation module, that creates BEV features by attending to connected front-view local regions, using camera parameters as a reference. However, the PersFormer method has high computational resource requirements and long convergence times. To overcome the problems of PersFormer and maintain high performance, we propose a method that combines the perspective transformer with MobileNet.

3. Proposed Methods

3.1. System Overview

From an input image I ∈ R^H×W, the proposed method generates a set of 3D lanes as follows:

L_3D = {l₁, l₂, l₃, …, l_N},

(1)

where H × W is the resolution of the input image in pixels and N is the total number of 3D lanes in the predefined BEV range. Mathematically, each 3D lane l_i is represented by an ordered set of 3D coordinates as follows:

l_i = [(x₁, y₁, z₁), (x₂, y₂, z₂), …, (x_Ni, y_Ni, z_Ni)],

(2)

where i is the lane index and N_i is the total number of sample points in lane l_i.

The overall of the proposed framework is illustrated in Figure 1. The proposed framework includes three stages: feature extraction, perspective transformation, and lane detection postprocessing. The feature extraction stage uses single 2D images as the input. The input image is passed through a preprocessing step before being fed to the MobileNet model to obtain multiscale front-view features. In the second stage, the perspective transformer module uses the front-view features from the first stage to generate BEV features by attending to the local context and aggregating the surrounding features to form a robust representation. In this module, inverse perspective mapping from the front view to the BEV was performed. In the third stage, the lane detection module generates the coordinates, and types of 2D and 3D lanes. 2D lane detection is inherited from the LaneATT [30] model to increase the accuracy of 3D lane prediction. The 3D lane detection is referred to as the 3D-LaneNet [34] model, and it is obtained by modifying the structure and anchor design similar to PersFormer [13] method. After each iteration, the networks are fine-tuned based on PersFormer and we trained the system until 2D and 3D predictions are perfected. After training, the proposed system was used to retrieve high-quality 3D lanes from single red-blue-green (RGB) images. The main steps of the proposed method are detailed below.

3.2. Feature Extraction

The target of the feature extraction stage is similar to that of the backbone module in other methods [13,34,35]. However, we focus on a lightweight model with low computational resource requirements to maintain the front-view feature quality. Therefore, we designed the feature extraction stage differently from previous models by scaling up the input image size before feeding it to the MobileNet that serves as the backbone network. After moving through several convolution layers, the feature extraction stage generates four different scaled front-view feature maps with pixel resolutions of 180 × 240, 90 × 120, 45 × 60, and 22 × 30 pixels. Next, all front-view feature maps are transformed into the corresponding BEV-space feature maps using a perspective transformer.

3.3. Three-Dimensional Lane Detection

Most previous studies have focused on 2D and 3D lane detection separately. However, they are not mutually exclusive. Hence, we used 2D lane prediction to support 3D lane detection using the loss function. Based on PersFormer, the proposed method designs anchors to optimize feature sharing across 2D and 3D domains. In BEV space, a point (x, y) casts the corresponding point (u, v) in front view space through intermediate state (x₀, y₀). By learning offsets, the network learns target-reference points mapping from BEV space to front-view space. We employed seven groups of anchors sampled with different inclination angles in the BEV space and projected them onto the front view. The inclination angles consist of π/2, arctan (±0.5), arctan (±1), and arctan (±2), respectively.

3.4. Loss Function

We summarize the overall loss to fine-tune the networks in the proposed method. From the 2D input image and ground-truth labels, we computed the sum of losses of all of the anchors. The total loss ${\mathcal{L}}_{Total}$ is a combination of the 3D lane detection ${\mathcal{L}}_{3\mathrm{D}}$, 2D lane detection ${\mathcal{L}}_{2\mathrm{D}}$ and intermediate segmentation ${\mathcal{L}}_{seg}$ with different factors α, β, and γ, respectively, as shown in Equation (3). Since this study focuses on 3D lane prediction, α was the highest, followed by β and γ.

$${\mathcal{L}}_{Total}=\mathsf{\alpha }{\mathcal{L}}_{3\mathrm{D}}+\mathsf{\beta }{\mathcal{L}}_{2\mathrm{D}}+\mathsf{\gamma }{\mathcal{L}}_{seg}$$

(3)

${\mathcal{L}}_{3\mathrm{D}}$ and ${\mathcal{L}}_{2\mathrm{D}}$ comprises classification loss, regression loss, and visibility loss, as shown in Equations (4) and (5), respectively. $c_{3\mathrm{D}}^k$ and $c_{2\mathrm{D}}^k$ are the predicted lane categories in the 3D and 2D domains, respectively. Each loss type had a corresponding factor, $f_1$,$f_2$, and $f_3$ for 3D loss, and $g_1$,$g_2$, and $g_3$ for 2D loss. The classification loss is the cross-entropy loss ${\mathcal{L}}_{CE}$. The regression loss is the L1 loss, and the visibility loss is a binary cross-entropy loss ${\mathcal{L}}_{BCE}$.

$$\begin{gathered} {\mathcal{L}}_{3\mathrm{D}}={\displaystyle \sum }_k{f}_1{\mathcal{L}}_{CE}\left(c_{3{\mathrm{D}}_{pre}}^k,c_{3{\mathrm{D}}_{GT}}^k\right)+f_2{\mathcal{L}}_{L1}\left( {\left(x^k,z^k\right)}_{pre},{\left(x^k,z^k\right)}_{GT}\right) \\ +f_3{\mathcal{L}}_{BCE}\left(v3{\mathrm{D}}_{pre}^k,v3{\mathrm{D}}_{GT}^k\right) \end{gathered}$$

(4)

$$\begin{gathered} {\mathcal{L}}_{2\mathrm{D}}={\displaystyle \sum }_k{g}_1{\mathcal{L}}_{CE}\left(c_{2{\mathrm{D}}_{pre}}^k,c_{2{\mathrm{D}}_{GT}}^k\right)+g_2{\mathcal{L}}_{L1}\left({\left(u^k\right)}_{pre},{\left(u^k\right)}_{GT}\right) \\ +g_3{\mathcal{L}}_{BCE}\left(v2{\mathrm{D}}_{pre}^k,v2{\mathrm{D}}_{GT}^k\right) \end{gathered}$$

(5)

The segmentation loss is also a binary cross-entropy loss between the prediction $S_{pre}$ and ground-truth $S_{GT}$ segmentation maps, respectively, as follows:

$${\mathcal{L}}_{seg}={\displaystyle \sum }_k{\mathcal{L}}_{BCE}\left(S_{pre},S_{GT}\right)$$

(6)

4. Experiments

To verify the high reliability of the proposed 3D lanes detection method both qualitatively and quantitatively, we conducted several experiments. In addition, the performance of our 3D lanes detection method was compared with that of several ultra-modern studies.

OpenLane dataset [13] is the first real-world 3D lane dataset and currently the largest scale compared with the existing benchmarks. We used OpenLane for both qualitative and quantitative experiments. The OpenLane was constructed on top of the influential Waymo Open dataset [39] by following the same data format. It comprises 200,000 frames and over 880,000 high-accuracy annotated lanes with 14 category labels, including single white dash, double yellow solid, and left/right curbside. In addition, the OpenLane dataset contains six sub-datasets: curve case, up and down case, night case, extreme weather case, merge and split case, and intersection case.

We run the experiments on a medium server with four NVIDIA TitanX GPUs with the Ubuntu 16.04 operating system and Cuda 10.2 version. Moreover, we employed PyTorch 1.8.0 as a deep learning library. The Adam optimizer was used along with a batch size of 2. We scale up the input image size by 1.5 times before feeding to the MobileNet. Base on experiments, α, β and γ were set to 10, 4 and 1, respectively. Other parameters are based on the model in [13]. The training time for the OpenLane dataset was approximately 3 days with 10 epochs.

4.1. Qualitative Results

Figure 2, Figure 3 and Figure 4 show the 2D and 3D results of the six sub-datasets. In all six cases, the results were predicted accurately from the corresponding RGB images. Columns (a) and (c) show 2D results, and columns (b) and (d) show the 3D results compared with the ground truth of the 3D lanes. The resulting 3D lanes show that the proposed method has the ability to predict 3D lanes close to the ground-truth data. In some special cases, when the lanes are covered by vehicles or the lanes are not clear, the proposed method has some missing lanes.

Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 show qualitative comparisons between our method and ultra-modern PersFormer, 3D-LaneNet, and Gen-LaneNet methods for the six sub-datasets of OpenLane. In all cases, the proposed method performed significantly better than Gen-LaneNet and 3D-LaneNet. Besides, the proposed method can provide lane-type and subtle curbside information, which is missing in Gen-LaneNet and 3D-LaneNet. Moreover, the proposed method shows better qualitative performance than PersFormer which is the best current ultra-modern model. Under a straight-road scenario, PersFormer and the proposed method provided similar quality. Nevertheless, the proposed method shows improved performance for difficult cases, such as the up and down, curve, extreme weather, and merge and split cases.

4.2. Quantitative Results

To evaluate the 3D lane quality, we compared predicted data with ground-truth 3D lanes by calculating both accuracy and error values. We employed eight commonly used performance measures: F-score, recall, precision, category accuracy, error near/far of x and z, respectively. The F-score demonstrates the relationship between true positives, false positives, and false negatives. Relying on PersFormer, the match between the prediction and ground truth was built based on the edit distance. A generated lane is considered true positive only if more than 75% of the lane’s wrapped y-positions have a point-wise distance smaller than the maximum distance limit. In the experiments, the max-allowed distance was set to 1.5 m. Recall (Equation (7)) and precision (Equation (8)) were investigated by relying on the percentage of matched ground-truth and predicted 3D lanes. Using recall and precision, we calculate F-score (Equation (9)), which is the most important value for evaluating performance measures. Thus, the F-score illustrates the regression performance of deep learning models based on recall and precision values. In addition, the OpenLane dataset has category information per lane. Hence, we evaluated the accuracy of the matched lanes, referred as category accuracy, to show classification performance. Finally, the x and z error near are used to calculate the start and estimated end point errors of the predicted 3D lanes.

$$\mathrm{Precision}=\frac{\mathrm{TruePositive}}{\mathrm{TruePositive}+\mathrm{FalsePositive}}{,}_{}$$

(7)

$$\mathrm{Recall}=\frac{\mathrm{TruePositive}}{\mathrm{TruePositive}+\mathrm{FalseNegative}}{,}_{}$$

(8)

$$\mathrm{F}-\mathrm{score}=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}{.}_{}$$

(9)

Table 1. Results of eight quantitative measures of the proposed method.

Measure		Up and Down	Curve	Extreme Weather	Night	Intersection	Merge and Split
Accuracy (higher is better)	F-score (%)	42.8	53.6	49.8	45.0	37.6	45.4
	Recall (%)	45.5	54.2	46.6	40.2	38.2	38.8
	Precision (%)	40.5	53.0	53.5	51.2	37.0	54.7
	Category accuracy (%)	86.6	93.2	89.6	85.1	85.0	87.0
Error (lower is better)	x error near (m)	0.299	0.315	0.275	0.303	0.446	0.408
	z error near (m)	0.161	0.172	0.141	0.210	0.336	0.307
	x error far (m)	1.094	0.851	0.805	0.742	0.828	0.733
	z error far (m)	1.010	0.695	0.723	0.660	0.742	0.620

lists results of eight quantitative measures of the proposed method. The best F-score is obtained for the curved case (53.6%), followed by recall (54.2%) and category accuracy (93.2%). The merge and split case has the best precision (54.7%), x error far (0.733 m), and z error far (0.62 m). In contrast, the extreme weather case achieved the best x error near (0.275 m) and z error near (0.141 m). The intersection case has the worst F-score, recall, precision, category accuracy, x error near, and z error near measures considering the difficulty in predicting the 3D lanes. Among all of the measures, the up and down case obtained the two worst measures (x and z error far) due to the large height difference between the start and end points in one lane.

To verify the high reliability of the proposed method, we checked the quantitative performance of the proposed method against other ultra-modern methods on the OpenLane dataset.

Table 2. The comparison results in both accuracy and error values.

Method	Accuracy Measures (Higher is Better)		Error Measures (Lower is Better)
	F-Score	Category Accuracy	X Error Near	Z Error Near	X Error Far	Z Error Far
3D-LaneNet [34]	40.2	-	0.278	0.159	0.823	0.714
Gen-LaneNet [35]	29.7	-	0.309	0.160	0.877	0.750
PersFormer [13]	47.8	92.3	0.322	0.213	0.778	0.681
Proposed method	47.9	89.1	0.341	0.224	0.789	0.694

and

Table 3. The comparison results in accuracy measure. Proposed method achieves the best F-score on 4/6 scenario sets. The best scores are marked in bold.

Method	F-Score
	Up and Down	Curve	Extreme Weather	Night	Intersection	Merge and Split
3D-LaneNet [34]	37.7	43.2	43.0	39.3	29.3	36.5
Gen-LaneNet [35]	24.2	31.1	26.4	17.5	19.7	27.4
PersFormer [13]	42.4	52.8	48.7	46.0	37.9	44.6
Proposed method	42.8	53.6	49.8	45.0	37.6	45.4

illustrate the comparison results for the entire validation and six scenario sets, respectively. The scenario sets comprising the curve, up and down, night, extreme weather, merge and split, and intersection cases were selected from the validation set based on the scene tags of each frame.

The proposed method outperforms the Gen-LaneNet, 3D-LaneNet, and PersFormer methods in F-score by 18.2%, 7.7%, and 0.1%, respectively (Table 2). However, the proposed method overlooks the near and far error metrics on the x- and z-axis since it is more suitable for fitting the main body of a lane rather than the start and end points. Moreover, the proposed method also obtains the best F-score on 4/6 scenario sets, except for the night and intersection cases (Table 3).

3D-LaneNet and Gen-LaneNet failed to support category classification. Therefore, we only report the category accuracy of the proposed method and PersFormer. When we run with significantly lower computational resources than PersFormer, the category accuracy of our method is quite lower than that of PersFormer, as in

Table 4. Computational resources of proposed method and PersFormer.

Method	Number of GPUs	GPU Type	CUDA Cores	Tensor Cores	Memory
PersFormer [13]	8	Nvidia TeslaV100	5120	640	32 GB
Proposed method	4	Nvidia TitanX	3584	0	12 GB

. However, the category accuracy of our method was similar to that of PersFormer when using similar resources. In addition, the proposed method converged faster than the PersFormer, and the training time was 10 times shorter than that of PersFormer under similar conditions. Overall, the proposed method achieves the best performance in 3D lane detection compared with other researches.

5. Conclusions

We propose an approach to extract 3D lanes from a single 2D RGB image of an outdoor scene for applications, such as autonomous vehicles. To solve both the limitations of low accuracy and high computational cost, we implemented a three-stage PersFormer-based method using perspective transformation and MobileNets to generate 3D lanes. The results of the experiments verify that the proposed 3D lanes detection method has ability to create high-quality 3D data from a single 2D image. The efficiency of our method is evaluated on a common server with four TitanX GPUs. Compared with the ultra-modern 3D lane detection approaches, the proposed method obtains the best performance in quantitative analyses and visual assessment. Moreover, the proposed method limits the need for high computational resources, such as high-end GPUs. In some special cases, when the lanes are covered by vehicles or the lanes are not clear, the proposed method has some missing lanes. In the future, we will enhance the feature extraction stage to get higher category accuracy and expand the proposed model to build accumulated 3D lanes from multiple frames. Additionally, we will compare our method with more relevant works to verify the performance in both qualitative and quantitative terms.