Image-Aided LiDAR Extraction, Classification, and Characterization of Lane Markings from Mobile Mapping Data

Abstract

The documentation of roadway factors (such as roadway geometry, lane marking retroreflectivity/classification, and lane width) through the inventory of lane markings can reduce accidents and facilitate road safety analyses. Typically, lane marking inventory is established using either imagery or Light Detection and Ranging (LiDAR) data collected by mobile mapping systems (MMS). However, it is important to consider the strengths and weaknesses of both camera and LiDAR units when establishing lane marking inventory. Images may be susceptible to weather and lighting conditions, and lane marking might be obstructed by neighboring traffic. They also lack 3D and intensity information, although color information is available. On the other hand, LiDAR data are not affected by adverse weather and lighting conditions, and they have minimal occlusions. Moreover, LiDAR data provide 3D and intensity information. Considering the complementary characteristics of camera and LiDAR units, an image-aided LiDAR framework would be highly advantageous for lane marking inventory. In this context, an image-aided LiDAR framework means that the lane markings generated from one modality (i.e., either an image or LiDAR) are enhanced by those derived from the other one (i.e., either imagery or LiDAR). In addition, a reporting mechanism that can handle multi-modal datasets from different MMS sensors is necessary for the visualization of inventory results. This study proposes an image-aided LiDAR lane marking inventory framework that can handle up to five lanes per driving direction, as well as multiple imaging and LiDAR sensors onboard an MMS. The framework utilizes lane markings extracted from images to improve LiDAR-based extraction. Thereafter, intensity profiles and lane width estimates can be derived using the image-aided LiDAR lane markings. Finally, imagery/LiDAR data, intensity profiles, and lane width estimates can be visualized through a web portal that has been developed in this study. For the performance evaluation of the proposed framework, lane markings obtained through LiDAR-based, image-based, and image-aided LiDAR approaches are compared against manually established ones. The evaluation demonstrates that the proposed framework effectively compensates for the omission errors in the LiDAR-based extraction, as evidenced by an increase in the recall from 87.6% to 91.6%.

1. Introduction

Several transportation engineers conduct road safety analyses to explore ways to decrease accidents by correlating crash data with human, roadway, or vehicle factors. As shown in Figure 1, out of the 95% of accidents related to human factors, 41% are the result of roadway features (highlighted in lavender in Figure 1) [1], which include (i) road surface conditions (i.e., potholes or cracks), (ii) lane marking classification (i.e., color/pattern/location), (iii) lane marking retroreflectivity, (iv) roadway geometry, and (v) lane width [2,3,4]. Therefore, lane marking inventory documenting roadway factors could reduce accidents and facilitate road safety analyses. Furthermore, a practical reporting mechanism that can visualize such information is critical for comprehensive, easy-to-grasp lane marking inventory. Several researchers established inventory using MMS data through road surface identification, color/intensity enhancement, lane marking extraction, classification, and characterization, as well as reporting mechanisms, which are briefly explained below.

• Road surface identification: define road surface regions in imagery/LiDAR data;
• Color/intensity enhancement: enhance the utility of color/intensity information for road surface in imagery/LiDAR data;
• Lane marking extraction: detect lane markings in the enhanced road surface imagery/LiDAR data;
• Lane marking classification: assign varying labels based on color/pattern/location to detected lane markings according to the Federal Highway Administration (FHWA) standard [5];
• Lane marking characterization: derive lane marking attributes (e.g., visibility conditions of lane markings, intensity profiles, and lane width) using classified lane markings;
• Reporting mechanisms: visualize the derived lane marking results (e.g., extracted lane markings and their characteristics) based on imagery and/or LiDAR data.

It is important to acknowledge that while human, vehicle, and roadway factors play a significant role in accidents, other factors such as weather, lighting, and systemic transportation issues can also contribute to accidents. This study focused on the importance of human, vehicle, and roadway factors due to their notable influence on road safety analysis, as outlined by the Federal Highway Administration (FHWA) [6].

The majority of existing research has focused on steps a-d using either (1) imagery, (2) LiDAR, or (3) a combination of both. Image-based approaches [7,8,9,10,11] encompass road surface identification, color enhancement, and lane marking extraction. These approaches are reliable under reasonable weather and lighting conditions. On the other hand, LiDAR-based strategies [12,13,14,15,16] involve road surface identification, intensity enhancement, and lane marking extraction. LiDAR-based strategies demonstrate greater effectiveness in areas where there is a significant contrast between lane marking and pavement regions. Recently, several studies have investigated the potential of combining MMS imagery and LiDAR data for lane marking extraction. Huang et al. [17] proposed a lane marking extraction algorithm based on video and LiDAR datasets in urban areas. They used a 1D kernel filter to extract lane markings from each video frame and remove false positives using LiDAR-based road surface identification. Li et al. [18] utilized an MMS to detect the road surface by combining LiDAR and image data and subsequently extracted lane markings from the images. First, the road surface is determined using elevation changes in point clouds and color variability in RGB images. After that, road surface regions in all images are transferred into bird-eye-view imagery. Finally, Otsu thresholding and Hough transform strategies are applied to the bird-eye-view imagery for lane marking extraction. Shin et al. [19] presented a lane marking extraction approach based on imagery and LiDAR data, where road surface points are first identified by applying RANSAC-based plane fitting to point clouds. Next, lane marking points are extracted from road surface point clouds through intensity thresholding. Lane markings are also identified from images using a median filter and local thresholding. Finally, the LiDAR-based lane markings are projected onto images to eliminate false positives, considering only the lane markings detected by both LiDAR and images as correct detections. Gu et al. [20] proposed an algorithm to fuse imagery and LiDAR data for lane marking extraction. They identified road surface points based on the installation height of the used LiDAR scanner and projected them onto the corresponding images. In the road surface regions of each image, the RGB model is converted into HSV, and the saturation channel is replaced with LiDAR intensity. These road surface regions are used as input for training a convolutional neural network (CNN) model [21] for lane marking extraction. They labeled 30,000 images to train the model. Bai et al. [22] developed a CNN model for lane marking extraction using imagery and LiDAR data. They converted LiDAR point clouds into three channel images, with the first channel representing intensity and the second and third channels representing the elevations of the highest and lowest points within each discretization bin. After that, the images were fed to a ResNet50 model [23] to estimate road surface elevation. Based on the road surface elevation, all camera images are converted into bird-eye-view ones. Finally, LiDAR and camera images are used to train a CNN model for lane marking extraction.

Although the aforementioned approaches are capable of extracting lane markings, they have primarily been conducted or evaluated using an MMS equipped with a single camera and/or LiDAR unit within a limited study area (typically ranging from 1 to 25 km long). Furthermore, these approaches have mainly focused on extracting lane markings along the driving lane, with few studies addressing lane marking classification and characterization. In most image LiDAR-based approaches, imagery data have been predominantly used as the primary source, while LiDAR data are utilized to enhance lane marking extraction. Even when using imagery to enhance LiDAR-based extraction, the focus is mainly on addressing false positives in LiDAR-based lane markings. However, it is important to consider the strengths and weaknesses of both camera and LiDAR units when establishing lane marking inventory, as summarized in

Table 1. Advantages and shortcomings of using either camera or LiDAR units onboard an MMS for lane marking extraction.

Sensor	Pros	Cons
Camera	Color information is available Massive existing image-processing strategies Temporary lane markings might be detected	Images are affected by weather and lighting conditions Lots of occlusions due to neighboring traffic No 3D and intensity information Insufficient grinding of lane markings might not be detected
LiDAR scanner	Data collection is not affected by adverse weather and lighting conditions Minimal occlusions 3D and intensity information is available Insufficient grinding of lane markings might be detected	No color information Temporary lane markings might not be detected

. Images are susceptible to weather and lighting conditions, and lane markings can be obstructed by neighboring traffic. They also lack 3D and intensity information, although color information is available. On the other hand, LiDAR data collection is not affected by adverse weather and lighting conditions, and it has minimal occlusions. One should note that occlusion minimization in LiDAR data depends on the type of used scanner. A single 2D LiDAR system would suffer from occlusions as it scans a given object location only once. Implementing two 2D LiDAR units will have fewer occlusions as the two units will scan an area from two locations, which could minimize occlusions. When using several multi-beam spinning LiDAR units, occlusions are greatly reduced by the fact that a given object region is scanned from several locations by the same and different units onboard the mobile mapping system. LiDAR data also provide 3D and intensity information. Moreover, relying solely on imagery or LiDAR data presents challenges in assessing the insufficient grinding of lane markings and temporary pavement markers. Figure 2 displays a situation where inadequate grinding of previous lane markings can still be visible in a LiDAR point cloud but not in an image. Conversely, temporary lane markings can be observed in imagery but not in LiDAR data if glass beads are not applied, as shown in Figure 3. Considering the complementary characteristics of camera and LiDAR units, an image-aided LiDAR framework would be highly advantageous for lane marking inventory purposes.

For a reporting mechanism of lane marking inventory, several transportation agencies have utilized a geographic information system (GIS) [24,25]. These GIS-based reporting mechanisms involve overlaying various 2D bird-eye-view features on top of each other for visualization or spatial analysis [26]. However, with the advent of MMS platforms, existing GIS-based reporting mechanisms may not be adequate for displaying lane marking results derived from imagery/LiDAR data. Given that an MMS captures perspective-view images, the image-based lane marking results may require orthorectification before using them in GIS-based reporting mechanisms. However, the visualization of perspective-view images in a reporting mechanism raises another issue of integrating 2D imagery and 3D LiDAR data. This issue could be addressed through forward and backward projection. Forward projection allows the transformation from a 2D point on an image to the corresponding 3D point in LiDAR data. In contrast, backward projection enables the transformation from a 3D point in LiDAR data to the corresponding 2D point on an image. Additionally, forward/backward projection in a reporting mechanism is essential for effective quality control of the results obtained from lane marking extraction, as it allows for the assessment of the alignment of derived lane markings from multiple sensors or an MMS. Nevertheless, to the best of the authors’ knowledge, no study has been found that utilizes the projection capability to develop reporting mechanisms for lane marking inventory.

This paper addresses the aforementioned challenges by developing an image-aided LiDAR framework for establishing lane marking inventory. The main contributions of this study can be summarized as follows:

• Propose an image-aided LiDAR framework for the following:
• Lane marking extraction/classification/characterization;
• Identifying all lane markings visible in imagery/LiDAR data (not only along the driving lane);
• Handling multiple imaging and LiDAR sensors onboard an MMS;
• Evaluate the performance of the proposed strategies using an MMS equipped with multiple camera and LiDAR units along extended road segments;
• Develop a reporting mechanism for visualizing imagery and LiDAR data together with extracted lane markings, as well as their characteristics (e.g., visibility conditions of lane markings, intensity profiles, and lane width).

The remainder of this paper is organized as follows. Section 2 introduces the used MMS and imagery/LiDAR data. Section 3 presents the models used for forward/backward projection, the proposed image-aided LiDAR lane marking inventory framework, and metrics for performance evaluation. The experiment results are reported in Section 4. Finally, the conclusions and scope for future work are summarized in Section 5.

2. Data Acquisition Systems and Dataset Description

To propose a framework that can handle multiple imaging and LiDAR sensors with different positions and orientations, an MMS was deployed for data acquisition along extended road segments in this study. The following subsections outline the MMS specifications and provide details about the study site and acquired datasets.

2.1. Mobile Mapping System

To address the research objectives, this study adopted an MMS equipped with multiple imaging and LiDAR sensors; namely, the Purdue wheel-based mobile mapping system–high accuracy (PWMMS-HA), as displayed in Figure 4. The PWMMS-HA is equipped with four multi-beam LiDAR scanners: three Velodyne HDL-32E and one Velodyne VLP-16 Hi-Res. The HDL-32E comprises 32 vertically aligned laser rangefinders with a total vertical FOV ranging from $-30.67°$ to $+10.67°$. The VLP-16 Hi-Res consists of 16 laser rangefinders with a vertical FOV ranging from $-10°$ to $+10°$. The HDL-32E has a point capture rate of 700,000 points per second [27], while the VLP-16 Hi-Res has a point capture rate of 300,000 points per second [28]. In addition, three FLIR Grasshopper 3 9.1MP GigE cameras are installed on the PWMMS-HA: two forward facing and one rear facing. These cameras have a maximum image resolution of 9.1 MP (3376 column pixels $\times$ 2704 row pixels) and are synchronized to capture images at a rate of 1 frame per second per camera. The two front cameras are installed at 2.3 m height, with a slight downward angle of approximately $1°$ from the horizontal plane at their locations, allowing the bottom row of the image to capture the ground roughly 1.5 m ahead of the camera position. Similarly, the rear camera is positioned at the same height but at a slightly steeper angle of approximately $7°$, allowing the bottom row of the image to capture the ground roughly 1 m behind the camera position. The above sensors are directly georeferenced by an Applanix POS LV 220 GNSS/INS unit. The GNSS collection rate is 20 Hz, and the measurement rate of the IMU is 200 Hz [29].

After GNSS/INS post-processing, the positional accuracy is around ±2 cm, and the attitude accuracy is approximately ±0.020° [29]. The expected accuracy of the resulting point cloud, based on the LiDAR and navigation system specifications, is roughly 2–4 cm at a 30 m scanning range [30]. To achieve this expected accuracy, system calibration is conducted to estimate the mounting parameters relating the GNSS/INS unit to LiDAR and imaging sensors [31,32,33].

2.2. Study Site and Dataset Description

To demonstrate the capability of the proposed image-aided LiDAR framework, the PWMMS-HA is employed to collect a large set of imagery and LiDAR data. Specifically, a 110-mile section of I-465 in the United States, which includes both inner and outer loops, is surveyed. The trajectory and testing locations, where the performance of the proposed approaches (discussed later in Section 4.3) was evaluated, along I-465 are depicted in Figure 5. For the testing locations in Figure 5, locations 1–300 are situated on the inner loop, while locations 301–600 are located within the outer loop. The PWMMS-HA was able to capture up to five lanes per driving direction, denoted by location i in Figure 5. The specifications of the acquired imagery and LiDAR datasets along I-465 are provided in

Table 2. Description of the imagery/LiDAR data collected by the PWMMS-HA along I-465 in the United States.

Highway	Date	Average Driving Speed (mph)	Imagery Data (# of Images)	LiDAR Data (# of Points)
I-465	11 July 2023	inner loop: 50.6 outer loop: 50.4	22,428 1	42,000 million

¹ Each camera captured a total of 7476 images.

. The average local point spacing (LPS) of the road surface point clouds captured by the PWMMS-HA is 2.5 cm.

3. Methodology

To ensure consistent performance across datasets captured by multiple imaging and LiDAR sensors, this study proposes a geometric image-aided LiDAR framework. The geometric framework aims to overcome the potential inconsistency in performance that may arise from learning-based strategies when applied to datasets captured by different sensors [34]. The proposed image-aided LiDAR lane marking inventory framework is illustrated in Figure 6. This framework builds upon the forward/backward projection technique and previous research on LiDAR-based strategies for road surface identification, intensity normalization, and lane marking extraction/classification/characterization (interested readers can refer to previous studies [35] for more details regarding the above procedure). The subsequent subsections provide detailed explanations of the models used for the forward/backward projection between the imaging and LiDAR units onboard an MMS, the proposed image-aided LiDAR framework, and the metrics for performance evaluation.

3.1. Point Positioning Models for Forward/Backward Projection

For the forward/backward projection in this study, the point-positioning equations for camera and LiDAR units onboard an MMS are established, as graphically explained in Figure 7. First, an imaging or laser beam ray relative to its sensor coordinate system needs to be defined. This can be achieved based on the imagery/LiDAR measurements (i.e., the image coordinates for a camera as well as the scanning range and direction for a LiDAR unit) and the Interior Orientation Parameters (IOPs) of the used sensors (i.e., parameters describing principal point coordinates, principal distance, and distortion parameters for a camera, as well as encoder mechanism for a LiDAR unit). After that, the position and orientation of the imaging or laser beam ray relative to the mapping frame can be established based on the Exterior Orientation Parameters (EOPs), which describe the position and orientation of the sensor relative to the mapping frame.

According to Figure 7, the point-positioning model of a camera onboard an MMS can be described by Equation (1), where ${\lambda }_{(i,c,t)}$ is a scaling factor associated with point $i$ in the image captured by camera $c$ at time $t$ and $r_i^{c\left(t\right)}$ is the imaging ray for an image point $i$ relative to the camera coordinate system at time $t$. Here, $r_i^{c\left(t\right)}$ is defined by the image coordinates of a point $i$ (x_i and y_i) and the camera IOP, including the principal point coordinates (x_p and y_p) and principal distance ($f$), as well as distortions for the $x$ and $y$ coordinates (${dist}_{x_i}$ and ${dist}_{y_i}$). As presented in Equation (2) and Figure 7, the position and orientation information of the camera frame relative to the mapping frame ($r_{c\left(t\right)}^m$ and $R_{c\left(t\right)}^m$) can be derived using the GNSS/INS trajectory information ($r_{b\left(t\right)}^m$ and $R_{b\left(t\right)}^m$) and mounting parameters between the camera frame and GNSS/INS body frame ($r_c^b$ and $R_c^b$).

$$r_I^m=r_{c\left(t\right)}^m+{\lambda }_{(i,c,t)}{R}_{c\left(t\right)}^m{r}_i^{c\left(t\right)}, r_i^{c\left(t\right)}=\left[\begin{array}{c}{x}_i-x_p-{dist}_{x_i}\\ y_i-y_p-{dist}_{y_i}\\ -f\end{array}\right]$$

(1)

$$r_{c\left(t\right)}^m=r_{b\left(t\right)}^m+R_{b\left(t\right)}^m{r}_c^b \& R_{c\left(t\right)}^m=R_{b\left(t\right)}^m{R}_c^b$$

(2)

Similarly, the point-positioning model for a LiDAR unit onboard an MMS is described in Equation (3) and Figure 7. In Equation (3), $r_I^{lu\left(t\right)}$ represents the position of the footprint of a laser beam relative to the LiDAR unit frame at time $t$, while $r_{lu\left(t\right)}^m$ and $R_{lu\left(t\right)}^m$ are the position and orientation of the LiDAR unit frame relative to the mapping frame at time $t$. $r_I^{lu\left(t\right)}$ is determined based on the measurements of the LiDAR unit’s range and pointing direction, as well as its IOP. In the case of a spinning multi-beam laser scanner, each laser beam is fired at a fixed vertical angle $\beta$, and the horizontal angle $\alpha$ is determined based on the rotation of the LiDAR unit. The range $\rho$ is calculated based on the time it takes for the laser beam to travel from the firing point to the footprint. Accordingly, Equation (4) defines the coordinates of a 3D point relative to the LiDAR unit coordinate system. As shown in Figure 7, $r_{lu\left(t\right)}^m$ and $R_{lu\left(t\right)}^m$ can be estimated according to Equation (5), where $r_{b\left(t\right)}^m$ and $R_{b\left(t\right)}^m$ are the position and orientation of the GNSS/INS body frame relative to the mapping frame at time $t$ and $r_{lu}^b$/$R_{lu}^b$ represent the lever arm and boresight rotation matrix between the LiDAR unit and GNSS/INS body frame coordinate systems. Accordingly, for a LiDAR unit, the coordinates of an object point $I$ in the mapping frame ($r_I^m$) can be computed in Equations (3)–(5).

$$r_I^m=r_{lu\left(t\right)}^m+R_{lu\left(t\right)}^m{r}_I^{lu\left(t\right)}$$

(3)

$$r_I^{lu\left(t\right)}=\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=\left(\begin{array}{c}\rho \left(t\right)cos\beta \left(t\right)cos\alpha \left(t\right)\\ \rho \left(t\right)cos\beta \left(t\right)sin\alpha \left(t\right)\\ \rho \left(t\right)sin\beta \left(t\right)\end{array}\right)$$

(4)

$$r_{lu\left(t\right)}^m=r_{b\left(t\right)}^m+R_{b\left(t\right)}^m{r}_{lu}^b \& R_{lu\left(t\right)}^m=R_{b\left(t\right)}^m{R}_{lu}^b$$

(5)

Equations (1)–(5) indicate that accurate sensor measurements, trajectory, and system calibration parameters are crucial for obtaining properly georeferenced data from imaging and LiDAR units. In this study, a calibration procedure [32] is conducted to estimate the mounting parameters between the different LiDAR scanners and the GNSS/INS unit to reconstruct georeferenced, well-aligned point clouds. In addition, a simultaneous calibration [32,33] between the camera and LiDAR units is implemented to compute the mounting parameters of the onboard cameras, relative to the reference frame of the GNSS/INS unit. Based on the estimated trajectory and mounting parameters, forward and backward projection between point clouds and images can be established. Specifically, forward projection can identify the corresponding location in a LiDAR point cloud for a selected feature point in an image. On the other hand, backward projection allows for the determination of the corresponding point in an image where a 3D object point identified in a point cloud is selected. This forward/backward projection serves as the foundation for proposing an image-aided LiDAR lane marking inventory framework.

For forward projection, the corresponding object point coordinates to an image point (x_i, y_i) are estimated by finding the intersection between the imaging ray and a 3D LiDAR point cloud. To solve for the unknown scale factor ${\lambda }_{(i,c,t)}$ in Equation (1), an octree-based ray tracing algorithm [36] is adopted. This algorithm involves building an octree of the LiDAR points and generating a set of points at equal intervals along the imaging ray, denoted by $I_{ray,1}~I_{ray,8}$ in Figure 8. The closest LiDAR point to each point along the ray is identified, and the distance between the imaging ray and LiDAR points is computed. Starting from $I_{ray,1}$ (the closest point to the perspective center) in Figure 8, the first point (considered as the forward projected point) that satisfies the following two criteria can be determined: (i) the distance is smaller than a prespecified threshold (e.g., 0.2 m), and (ii) the distance is smaller than the distance for the next point. These criteria ensure the identification of the intersection between the imaging ray and the closest LiDAR surface (i.e., the visible surface). As shown in Figure 8, $I_{ray,3}$ is the first point that satisfies the criteria, with $L_3$ being the closest LiDAR point, while $I_{ray,4}$ and $I_{ray,7}$, although meeting the criteria, are not visible. By projecting $L_3$ onto the imaging ray (the yellow point in Figure 8), the closest LiDAR surface (i.e., the visible surface) can be determined. Finally, the forward projection solution can be obtained by the intersection between the projected point and the object’s surface.

For backward projection, the corresponding image point for an object point $I$ in a point cloud can be directly evaluated. The image point positioning equations—Equations (1) and (2)—can be reformulated as per Equation (6), expressing the image point coordinates as a function of known parameters, such as GNSS/INS trajectory information, camera IOP, camera mounting parameters, and ground coordinates of the object point, as well as the unknown scale factor ${\lambda }_{(i,c,t)}$. To eliminate this unknown scale factor, the first and second rows of Equation (6) are divided by the third row, resulting in Equations (7) and (8), which present the image point coordinates (x_i, y_i) corresponding to an object point $I$.

$$r_i^{c\left(t\right)}=\frac{1}{{\lambda }_{\left(i,c,t\right)}}\left[R_b^c{R}_m^{b\left(t\right)}\left(r_I^m-r_{b\left(t\right)}^m-R_{b\left(t\right)}^m{r}_c^b\right)\right]=\frac{1}{{\lambda }_{\left(i,c,t\right)}}\left[\begin{array}{c}{N}_x\\ N_y\\ D\end{array}\right]$$

(6)

$$x_i=-c\frac{N_x}{D}+x_p+{dist}_{x_i}$$

(7)

$$y_i=-c\frac{N_y}{D}+y_p+{dist}_{y_i}$$

(8)

3.2. Image-Aided LiDAR Lane Marking Inventory Framework

The image-aided LiDAR lane marking inventory framework section starts by describing the LiDAR-based, image-based, and image-aided LiDAR strategies. Subsequently, the developed Potree-based web portal is introduced.

3.2.1. LiDAR-Based Lane Marking Extraction

According to Figure 6, the adopted LiDAR-based lane marking extraction includes road surface identification, generalized intensity normalization, and geometric lane marking extraction. For road surface identification, the cloth simulation filtering algorithm [37] is adopted to separate bare earth from above-ground objects. After that, the bare earth point cloud is divided into road surface blocks using the block length ($L$) along the driving direction and width ($W$) across the driving direction. In this study, $L$—e.g., 12.8 m—is chosen based on the minimum radius of curvature required for designing highways [38], which ensures that the lane markings are straight along the driving direction. The block width ($W)$—e.g., 18 m (with a 9 m extent on the right and left sides of the vehicle)—is determined by the average LPS of a point cloud, which ensures that the points representing lane markings are sufficiently dense for their extraction. To assess the LPS of the lane markings, a hypothesized lane marking is derived by applying the percentile intensity thresholding (${Th}_{int}$)—e.g., top 95th—to a small bare earth point cloud along the driving direction. The LPS of the hypothesized lane marking point cloud is then evaluated using the approach proposed by Lari and Habib [39]. This analysis indicates that if the width of road surface blocks exceeds 9 m on either side of the vehicle, the LPS of hypothesized lane markings is larger than the standard width of lane markings (i.e., lane markings cannot be extracted from LiDAR data at such distance).

The original intensity values of these road surface blocks are then normalized using a generalized normalization approach [35], which reduces intensity variation within and across different LiDAR units (i.e., normalizing intensity across all laser beams of a LiDAR scanner as well as all LiDAR units onboard an MMS). Finally, lane markings are extracted through five steps: (i) top 95th percentile intensity thresholding, (ii) scan line-based outlier removal, (iii) density-based spatial clustering [40], (iv) geometry-based outlier removal, and (v) local and global refinement. For a given road surface block, as shown in Figure 9a, hypothesized lane markings after step (i) are illustrated in Figure 9b. Next, for step (ii), the scan lines (which represent a sequence of points emitted by a LiDAR unit, denoted by scan line in Figure 9b) within these hypothesized lane markings are removed if they exceed a certain length, as shown in Figure 9c. This removal approach is based on the assumption that scan lines within a lane marking should not exceed a pre-defined threshold (${Th}_{line}$), as lane markings have a specific width. The remaining hypothesized lane marking points are grouped into isolated segments using a density-based spatial clustering algorithm, as displayed in Figure 9d. After that, a geometry-based strategy is applied to remove non-linear segments and outlier points within linear segments, as shown in Figure 9e. Removing non-linear segments relies on each segment’s principal component analysis (PCA), while removing outlier points within linear segments is based on a straight line that best fits each segment and random sample consensus (RANSAC). To connect isolated linear segments, two refinement strategies are employed: local refinement aims to connect small segments within each block and identify undetected lane marking points between small segments, while global refinement focuses on combining the same lane marking segments in successive blocks, as shown in Figure 9f,g.

3.2.2. Image-Based Lane Marking Extraction

For images captured by an MMS, lane markings are typically found in the bottom half of an image, while the upper half contains elements, such as the sky and landmarks [41,42], in Figure 10. Thus, it is reasonable to focus on the bottom half of an image for image-based lane marking extraction. Nonetheless, merely applying a thresholding-based segmentation or straight line-based detection algorithm to the bottom half may not suffice due to several factors influencing the accuracy of image-based lane marking extraction. These factors include lighting conditions, lane marking geometry, and image resolution. For instance, lighting conditions may fluctuate within the bottom half of an image, leading to incorrect segmentation when using global thresholding. Although most lane markings close to the camera appear as straight lines in an image, their geometry may change to curved lines as they get further away. Additionally, as the distance from a camera to a lane marking increases, the image resolution at those lane markings decreases. To address these challenges, a region of interest (ROI) partitioning inspired by a prior study [43] is implemented. In this study, two ROIs—near and distant—within the bottom half of an image are established. The near ROI encompasses the area from the bottom row of an image up to a row above, denoted by a 10 m boundary, where a camera captures the road surface 10 m in front of the vehicle, as indicated by the red polygons in Figure 11. Conversely, the distant ROI covers the area beyond the 10 m boundary and extends to another row above, denoted by a 25 m boundary, where a camera captures the road surface 25 m in front of the vehicle, as represented by the blue polygons in Figure 11. One should note that the same ROI definition procedure applies to the rear camera, with the exception that the 10 m and 25 m distances are behind the vehicles. Applying different segmentation or detection algorithms to the two ROIs separately can reduce the impact of the aforementioned issues.

To define the 10 m boundary and 25 m boundary on an image, the LiDAR point cloud, as well as the position and orientation information of the used cameras (as described in Section 2.1), are utilized. For the 10 m boundary, the point along the road surface 10 m ahead of a camera is selected from the LiDAR data. This point is then backward projected (as described in Section 3.1) to find its corresponding point on an image, allowing for the determination of the 10 m boundary. The same procedure is implemented to determine the 25 m boundary. The boundaries, as described by their image rows, of the near/distant ROIs for each camera onboard the PWMMS-HA are listed in

Table 3. Boundary utilized for defining near and distant ROIs within the bottom half of images captured by the three cameras used in this study for image-based lane marking extraction.

Camera	Image Size (Pixel) #of Columns × # of Rows	Near ROI (Pixel) 10 m Boundary—Bottom Row	Distant ROI (Pixel) 25 m Boundary—10 m Boundary
front-left camera	3376 × 2704	1800–2704	1450–1800
front-right camera
rear camera		1600–2704	1250–1600

. One should note that the two front cameras are installed with similar downward angles, resulting in the same 10 and 25 m boundaries. However, the rear camera has a slightly steeper downward angle, leading to different 10 and 25 m boundaries. The choice of a 10 m boundary is based on a minimum design speed of 30 mph ($\approx$13 m/s) and a chord length of 10 m, corresponding to an arc length of approximately 10.01 m for the minimum radius of curvature of 231 ft ($\approx$70.4 m). Thus, the 10 m boundary ensures that lane markings are straight in a near ROI. In addition, the 25 m boundary guarantees sufficient overlap between successive images while the vehicle travels at speeds of approximately 50 mph ($\approx$22 m/s). Given that the PWMMS-HA can capture one image per second, the gap between two successive image positions is about 22 m, which is less than the specified 25 m (resulting in around 3 m overlap between successive images), as shown in Figure 10. This study also assumes that lane markings are straight in the distant ROI, as the chord length and arc length are around 14.97 m and 15.0 m, respectively, for a minimum design speed of 30 mph.

Next, according to the proposed framework (Figure 6), vanishing point generation is conducted in this study. Vanishing points are crucial in extracting lane markings from images, as they provide geometric information for delineating lane markings along road surfaces. The concept of vanishing points relies on the fundamental principle of perspective in images—parallel lines in the real world converge to a single point in an image. For images captured by an MMS, lane markings are typically represented by parallel lines. By identifying the vanishing points corresponding to lane markings, one can infer the geometry information of the lane markings, which aids extraction.

Because lane markings are assumed to be straight in near/distant ROIs, two distinct vanishing points are generated using LiDAR-based lane markings within the respective ROIs in this study. Vanishing point generation is achieved by backward projecting the LiDAR-based lane markings (as explained in Section 3.1) onto each ROI in the images. Subsequently, the intersection point, determined by the projected lane markings in each ROI, can be considered a vanishing point, as shown in Figure 12. According to Yang et al. [44], the two vanishing points can be utilized to remove outliers within their respective ROIs. One should note that in cases where lane markings are not detected in LiDAR data, the vanishing points from the previous frame will be used for the current frame.

Finally, for image-based lane marking extraction, color information of lane markings (e.g., white or yellow) is utilized, as they exhibit distinct hues compared to the surrounding pavement. According to Son et al. [10], lane markings exhibit consistent characteristics in the YCbCr model under different lighting conditions. Specifically, the Y layer can be utilized to identify most lane markings, as they consistently have higher values compared to other colors, regardless of lighting conditions. Conversely, the Cb layer can be used to distinguish non-white (e.g., yellow or red) lane markings, as they consistently have lower Cb values compared to other colors under various lighting conditions. Figure 13 illustrates an RGB image captured under poor lighting conditions and the corresponding Y and Cb layers. By taking advantage of the fact that lane markings are generally brighter than the surrounding pavement, most of them can be detected in the Y layer of an image, as shown in Figure 13a. Furthermore, the Cb layer can differentiate non-white lane markings, as shown in Figure 13b. One should note that the selection of the dark RGB image in Figure 13 was deliberate to illustrate the characteristics of lane markings in the Y and Cb layers under poor lighting conditions, and these images are not a consequence of printing issues.

Based on the ROIs, vanishing points, and characteristics of lane markings in Y/Cb layers, image-based lane marking extraction will be implemented through the following steps: (a) RGB-to-YCbCr conversion, (b) Y/Cb thresholding, (c) connected component labeling (CCL) [45], (d) contrast-based outlier removal, (e) vanishing point-based outlier removal, and (f) 3D width-based outlier removal. Figure 14a displays the Y and Cb layers overlaid with near/distant ROIs after step (a). Step (b) applies a Y percentile thresholding (${Th}_y$)—e.g., top 97th—to the Y layers within each ROI since lane markings have higher Y values. In addition, a Cb percentile thresholding (${Th}_{cb}$)—e.g., lowest third—is applied to the Cb layers within each ROI since lane markings have lower Cb values. The selection of the top ninety-seventh and lowest third thresholding is based on the experiments conducted by Son et al. [10]. The results after Y/Cb thresholding are shown as binary images in Figure 14b. Subsequently, the detected pixels in each binary image are grouped into several segments based on four connectivity-based CCL in step (c), as presented in Figure 14c. After that, step (d) identifies segments with low contrast compared to their neighboring area and eliminates them as false positives, as displayed in Figure 14d. In step (d), each lane marking segment is divided into smaller segments using a length parameter ($S$)—e.g., 100 pixels—along the column direction of an image, as shown in Figure 15. Then, a buffer is created around each small segment by a buffer parameter ($B$)—e.g., 20 pixels—along the row direction of an image, and the average Y/Cb value within the buffer is calculated. Additionally, the average Y/Cb value within each small segment is computed. If the difference between the Y/Cb averages of a small segment and its surrounding buffer is less than a pre-defined threshold (${Th}_{contrast}$), the small segment is deemed a false positive and subsequently removed. Step (e), as shown in Figure 14e, eliminates segments whose directional vector is not parallel to the vector formed by connecting its centroid to the corresponding vanishing point (the correspondence is determined based on near and distant ROIs). Step (f) involves checking the average 3D distance between the two edges across the main direction of each segment to remove outliers. The two edges are forward projected onto LiDAR data (as explained in Section 3.1), and the average distance between the projected edges is computed. Segments whose width does not meet the standard size of lane markings (${Th}_{3D}$) are removed. The lane marking segments after step (f) are illustrated in Figure 14f. Finally, the results from the Cb layers can be used to differentiate between white and non-white lane markings extracted from the Y layers.

3.2.3. Image-Aided LiDAR Lane Marking Extraction/Classification/Characterization

Based on the lane markings derived from the LiDAR-based and image-based strategies described in Section 3.2.1 and Section 3.2.2, respectively, image-aided LiDAR lane marking extraction can be conducted. First, all lane marking segments extracted from images are forward projected (as explained in Section 3.1) onto LiDAR data. In this study, the results obtained from all the images captured by the three cameras are projected onto LiDAR data. Figure 16 illustrates examples of the centerline points (at intervals of 10 pixels) of 2D image-based lane markings and the corresponding forward projected points onto the LiDAR data. Thereafter, considering the complementary characteristics of camera and LiDAR units (as mentioned in Section 1), LiDAR-based lane markings can be refined based on the correspondence between image-based and LiDAR-based lane markings. In this study, the correspondence between image-based and LiDAR-based lane markings is determined by a distance threshold (${Th}_{aid}$)—e.g., 20 cm—as follows:

• If projected image-based lane markings are within 20 cm of LiDAR-based lane markings, the LiDAR-based extraction will be colored according to the image-based results and FHWA standard colors [5]. For instance, if image-based lane markings are white, LiDAR-based ones will be colored using RGB values of 247, 241, and 227.
• If no projected image-based lane markings are within a 20 cm neighborhood of LiDAR-based lane markings (i.e., no corresponding LiDAR-based extraction in point clouds), the image-based extraction will be utilized to extract lane markings in point clouds. First, the top 95th percentile intensity thresholding is applied to a road surface point cloud to derive hypothesized lane markings. In the hypothesized lane marking point cloud, points within 20 cm (Th_aid) of projected image-based lane markings are extracted. The resultant lane markings will also be colored according to the abovementioned procedure. One should note that this study aims to utilize image information to refine LiDAR-based lane markings for establishing inventory, including intensity profiles for evaluating retroreflectivity. To prevent misrepresentation of the intensity profiles, areas with no intensity contrast in the LiDAR data will not be utilized to extract lane markings in point clouds.

Once the image-aided LiDAR extraction is completed, lane markings can be further classified according to their pattern/location information through three steps: (I) length-based classification, (II) spline-based clustering, and (III) reference-based clustering. For step (I), as depicted in Figure 17a, each lane marking is categorized as either part of a solid, dash, or dotted line based on the FHWA standard length [5]. Thereafter, lane markings of the same type are grouped based on their relative position information for step (II), as shown in Figure 17b. Spline fitting [46] is applied to each lane marking within a specific type, and if the distance between any two splines meets a pre-defined criterion (${Th}_{cluster})$, these two lane markings are grouped together. Here, the distance between two splines is estimated using an approach proposed by D’Errico [47]. In step (III), the lane marking clusters derived from step (II) are further grouped, as displayed in Figure 17c. Starting from the longest lane marking cluster, spline fitting is applied, and the spline is used as a reference. Spline fitting is then applied to the other lane marking clusters, and the distances between the reference spline and the splines from the other lane markings are estimated. If any splines from the other lane markings meet a pre-defined criterion (${Th}_{cluster})$, these lane markings are grouped together. Throughout the subsequent discussion, we will refer to the groups of lane markings derived from the classification as “lane marking clusters”.

After the classification, the resultant lane marking clusters can be utilized to derive intensity and lane width information. For intensity profile generation, each lane marking cluster (as shown in Figure 18a) is divided into 20 cm portions along the driving direction. The average intensity values are computed for each portion, and a graph can be created to display the average intensity against the traveled distance, as shown in Figure 18b. Lane width estimation starts with lane marking clusters. First, centerline points are generated at intervals of 20 cm along each cluster, as displayed in Figure 19a. In cases with missing lane markings for the gaps between dashed or dotted lines, a linear interpolation is performed to fill the gap between two consecutive centerline clusters, as shown in Figure 19b. One should note that this study adopts linear interpolation as per the FHWA standard [5], which stipulates that a straight line should connect two dashed/dotted lane markings. However, to avoid linear interpolation on curved roads, interpolation is not applied if the gap exceeds a pre-defined threshold (${Th}_{gap})$. Subsequently, lane width can be computed using opposite centerline points from different lane marking clusters, as presented in Figure 19c. Finally, a plot can be generated to display the lane width against the traveled distance, as shown in Figure 19d.

3.2.4. Potree-Based Web Portal Visualization

In this study, Potree (http://www.potree.org, accessed on 1 January 2023) [48] is adopted to create a prototype visualization web portal. The architecture of the developed web portal, as well as the forward/backward projection functions and intensity profile/lane width displaying tools, are discussed in the following paragraphs.

Potree-based web portal

The architecture of the established Potree-based web portal is depicted in Figure 20. The front end, which is the graphical user interface, is used to display georeferenced imagery/LiDAR data (such as satellite imagery and LAS/LAZ files). The back end consists of various functions that allow users to manipulate the georeferenced data. The imagery and LiDAR data are stored in a database. Figure 21 displays the image placeholders (which indicate the position/orientation of the images) and LiDAR point clouds captured by the PWMMS-HA on top of a Cesium base map (https://cesium.com/, accessed on 25 September 2023) at Exit 25 on I-465. Thanks to the georeferencing parameters obtained from the GNSS/INS trajectory and system calibration procedures, the images are properly positioned and oriented relative to the point clouds, denoted by the yellow ovals in Figure 22. The back end receives client requests from the front end and processes them by interacting with the database using visualization and/or computational functions. For instance, in this study, the back end facilitates forward/backward projection functions in coordination with the front end and database.

In addition, the Potree-based web portal provides users with built-in functionality for measuring distance, angles, and area in LiDAR data. Each measurement is visually represented by a sequence of user-selected vertices and labels, adhering to the International System of Units (SI). For example, a distance measurement can be represented by a vector connecting two vertices and a label showing the distance, as displayed in Figure 23a. On the other hand, an angle or area measurement necessitates a minimum of three vertices selected by a user, after which the angles or lengths of each edge and the area will be displayed in the web portal, as depicted in Figure 23b,c.

Forward/backward projection functions

To facilitate the visualization of corresponding features in imagery and LiDAR data, forward/backward projection functions (as explained in Section 3.1) are developed. As illustrated in Figure 24, the forward projection function projects a selected point from an image onto the corresponding LiDAR point cloud, with the blue placemark in the former appearing as a red dot (with a white-on-black label for the 3D coordinates) in the latter. Figure 25 shows that the backward projection function projects an object point (red dot with a white-on-black label for the 3D coordinates) in a point cloud onto the corresponding images, denoted by a blue placemark. These projection functions enable users to visualize georeferenced imagery/LiDAR data captured simultaneously or at different times by the same or various MMS. Additionally, this projection function can be employed to assess the accuracy of trajectory and system calibration. For instance, LiDAR-based lane markings can be backward projected onto an image to visualize any discrepancies between lane markings derived from multi-modal data.

Displaying tools for intensity profiles and lane width

To facilitate the visualization of the intensity profiles and lane width plots derived from the proposed lane marking characterization (as discussed in Section 3.2.3), tools for displaying these products are developed in this study. Figure 26 illustrates an intensity profile viewer within the Potree-based web portal. This viewer allows users to select a point of interest (crosshair cursor) in a profile, which will then display the corresponding point (red dot with a white-on-black label for the 3D coordinates) on the point cloud. Similarly, the lane width plot can also be visualized by a viewer, as shown in Figure 27. Users can select a point of interest (crosshair cursor) in lane width estimates, and then the corresponding point pair (connected by two red dots with a white-on-black label for the lane width) on the point cloud will be shown on the web portal, as shown in Figure 27. Once the points of interest are projected onto point clouds, users can further utilize the backward projection function to locate the corresponding points in an image.

3.3. Performance Evaluation

Given the minimal occlusions in LiDAR data compared to images, this study assumes that all lane markings are visible in point clouds. Consequently, this study manually annotated the point clouds captured by the PWMMS-HA to generate reference data for performance evaluation. Figure 5 shows the locations where LiDAR data were annotated. The annotated LiDAR data cover 600 road surface blocks (each block spans 12.8 m along the driving direction, covering a total area of approximately a 5-mile-long road surface), with 300 blocks located along the inner loop and another 300 blocks along the outer loop. As mentioned previously in Section 3.2.1, each LiDAR road surface block has a width of 18 m, which can capture up to five lanes (assuming a lane width of 3.6 m). However, not all 600 road surface blocks capture five lanes due to variations in road geometry, lane width, and the number of lanes on I-465. Additionally, even in a five-lane area, not all lanes can be captured if the vehicle is not driving in the middle lane. Thus, a total of 600 road surface blocks along I-465 are selected to ensure the evaluation of performance under different scenarios described above.

To compare the extracted lane markings with the reference LiDAR data, centerline points were created along each detected/annotated lane marking through the following steps. For the LiDAR-based extraction, each lane marking is divided into 3-meter-long segments to represent curved solid lines using straight segments. Centerline points are then computed along each segment at 20 cm intervals. The above procedure is also implemented to generate centerline points for the image-aided LiDAR and reference lane marking point clouds. On the other hand, for the image-based extraction, the lane marking segments in all images are forward projected onto LiDAR data (as explained in Section 3.1). Again, each projected lane marking is divided into 3-meter-long segments, and then 20-centimeter-interval centerline points are created along each segment.

Precision, recall, and F1-score are used as metrics to evaluate the performance of the LiDAR-based, image-based, and image-aided LiDAR lane marking extraction strategies. These metrics are calculated using Equations (9)–(11), where true positives, false positives, and false negatives are denoted by $TP$, $FP$, and $FN$, respectively. True positives refer to the lane markings correctly identified by a particular approach, while false positives (also known as commission errors) happen when a lane marking is mistakenly identified, even though it does not exist in the actual scene. False negatives (also known as omission errors) are the lane markings that an approach fails to identify. Accordingly, if an extracted centerline point is within 20 cm of a reference centerline point, it is considered a true positive. Conversely, if there is no reference point within the 20 cm neighborhood of an extracted point, it is considered a false positive. Similarly, if there is no extracted point within the 20 cm neighborhood of a reference point, it is considered a false negative. Precision measures the proportion of correctly detected lane markings out of detected ones, while recall presents the proportion of correctly detected lane markings out of reference ones. Lastly, F1-score, which provides an overall assessment of performance, is a combination of precision and recall using a harmonic mean.

$$Precision=\frac{TP}{TP+FP}$$

(9)

$$Recall=\frac{TP}{TP+FN}$$

(10)

$$F1-score=\frac{2\times Precision\times Recall}{Precision+Recall}$$

(11)

4. Experimental Results and Discussion

For the proposed image-aided LiDAR lane marking inventory framework, the thresholds/parameters used in this study are summarized in

Table 4. Thresholds/parameters used for LiDAR-based/image-based/image-aided LiDAR strategies in this study.

Threshold/ Parameter	Description	Strategy (Section)	Value
$L$	Length of road surface blocks	LiDAR-based lane marking extraction (3.2.1)	12.8 m
$W$	Width of road surface blocks	LiDAR-based lane marking extraction (3.2.1)	18 m
${Th}_{int}$	Percentile intensity threshold for lane marking extraction from point clouds	LiDAR-based lane marking extraction (3.2.1)	top 95th %
${Th}_{line}$	Length threshold for scan line-based outlier removal	LiDAR-based lane marking extraction (3.2.1)	25 cm
${Th}_y$	Percentile Y value threshold for lane marking extraction from images	Image-based lane marking extraction (3.2.2)	top 97th %
${Th}_{cb}$	Percentile Cb value threshold for lane marking extraction from images	Image-based lane marking extraction (3.2.2)	lowest 3rd %
$S$	Length for dividing a segment in an image	Image-based lane marking extraction (3.2.2)	100 pixels
$B$	Number of pixels for creating buffers around a segment in an image	Image-based lane marking extraction (3.2.2)	20 pixels
${Th}_{contrast}$	Y/Cb value threshold for contrast-based outlier removal in an image	Image-based lane marking extraction (3.2.2)	5 Y/Cb values
${Th}_{3D}$	Lane marking width threshold for 3D width-based outlier removal in an image	Image-based lane marking extraction (3.2.2)	15 cm
${Th}_{aid}$	Distance threshold for determining the correspondence between image-based and LiDAR-based lane markings, as well as extracting lane markings in point clouds using image-based results	Image-aided LiDAR lane marking extraction (3.2.3)	20 cm
${Th}_{cluster}$	Distance threshold for grouping splines	Image-aided LiDAR lane marking classification (3.2.3)	75 cm
${Th}_{gap}$	Gap threshold for avoiding linear interpolation on curved roads	Image-aided LiDAR lane marking characterization (3.2.3)	40 m

. These thresholds/parameters are applied consistently across imagery/LiDAR data obtained from different MMS sensors. The experimental results and discussion section begins by presenting the products (i.e., extracted lane markings, intensity profiles, and lane width estimates) generated by the image-aided LiDAR lane marking inventory framework. Subsequently, the performance of LiDAR-based/image-based/image-aided LiDAR lane marking extraction strategies is evaluated through qualitative and quantitative analyses. Finally, the products and performance evaluation are discussed.

4.1. Products from Image-Aided LiDAR Lane Marking Inventory Framework

The proposed image-aided LiDAR lane marking inventory framework was applied to mobile data spanning 110 miles.

Table 5. Processing time for LiDAR-based, image-based, and image-aided LiDAR strategies based on one-mile-long lane marking extraction.

Approach	Time Taken (Seconds) for One-Mile-Long Lane Marking Extraction	Platform
LiDAR based	~450 1	32 GB RAM computer
Image based	~5070 2	12.7 GB RAM (GPU) Google Collaboratory
Image-aided LiDAR	~5970 3	32 GB RAM computer and 12.7 GB RAM (GPU) Google Collaboratory

¹ Time is estimated using four LiDAR units. ² Time is estimated using three cameras, and each individual image requires approximately 45 s. ³ Time includes the duration required for LiDAR-based and image-based strategies.

lists the execution time for the different lane marking extraction approaches. Figure 28 demonstrates the Cesium base maps overlaid with the point clouds and image placeholders along I-465. The portal is able to render the LiDAR and imagery datasets for both inner and outer loops in around ten seconds. Furthermore, extracted lane markings can also be visualized through the portal, as shown in Figure 29. Users can interact with the rendered data using the built-in functions, such as rotation, zooming in/out, and panning, without experiencing any delays, as demonstrated by the red zoom-in boxes in Figure 28 and Figure 29. Additionally, the web portal allows users to visualize intensity profile/lane width plots along I-465. By selecting a specific portion based on mile markers, users can view the corresponding intensity profile/lane width plot in a viewer window, as shown in Figure 30.

4.2. Qualitative Evaluation Using Potree-Based Web Portal Visualization

Through the built-in functionality and tools developed in this study for the web portal, the extracted lane markings and intensity profile/lane width results, as well as the alignment of lane markings derived from imaging/LiDAR units, can be qualitatively evaluated. For evaluating LiDAR-based/image-based/image-aided LiDAR extraction strategies, LiDAR-based lane marking point clouds are imported into the web portal. The centerline points generated using the forward projected image-based lane markings (as explained in Section 3.3) are also imported. Finally, the image-aided LiDAR and reference lane markings are imported for evaluation. Figure 31 presents a region where lane markings were not detected using the image-based approach but were successfully identified using the LiDAR-based and image-aided LiDAR strategies. In contrast, Figure 32 demonstrates a region where the LiDAR-based approach failed but the image-based and image-aided LiDAR strategies were effective. Figure 33 and Figure 34 display the LiDAR-based and image-based lane marking centerline points as well as the corresponding images with the 2D extraction at the same locations as Figure 31 and Figure 32.

As depicted in Figure 33, the inability of the image-based strategy to extract lane markings is attributed to the excessive change in lighting conditions. To investigate the failure of the LiDAR-based lane marking extraction, the intensity profiles in the same locations as Figure 32 were examined using the intensity profile display tool in the web portal. The intensity values for the lane markings were not detected by the LiDAR-based approach, and their surrounding lane markings are depicted in Figure 34. The decrease in intensity values from 52 to 47, as shown in Figure 34, could potentially explain the failure of the LiDAR-based extraction. The lane markings with intensity values lower than the surrounding ones might not be extracted using the LiDAR-based approach. This finding also suggests that the incorporation of image information can enhance the extraction of lane markings that are not detected by the LiDAR-based approach. Furthermore, Figure 32 and Figure 34 serve as evidence that the proposed framework is capable of extracting lane markings within a five-lane region.

For the qualitative evaluation of intensity profiles, Figure 35 displays a worn-out lane marking region with an intensity value of 45, along with the corresponding extracted lane marking point cloud and image. Figure 36 shows another region with an intensity value of 55 for well-preserved lane markings and its corresponding point cloud and image. These intensity values are consistent with the lane marking conditions in the corresponding images. For the qualitative evaluation of lane width estimates, Figure 37 presents a region with a lane width estimate of 3.60 m (yellow oval) and the corresponding point cloud and image. The red dots within the yellow oval in Figure 37 are correctly positioned on the opposite lane markings, and the estimate is similar to the manual measurements of 3.61 m (cyan oval), which is close to the value of the estimate, obtained through the built-in functionality of the web portal. All the placemarks in all the camera images in Figure 35, Figure 36 and Figure 37 are derived by backward projecting the intensity profile/lane width points in LiDAR data.

To evaluate the alignment of lane markings derived from imagery and LiDAR data, Figure 38 displays the forward projection of two points along 2D image-based lane marking centerlines onto the corresponding LiDAR data. Figure 39 shows the backward projection function of two points along LiDAR-based lane markings onto the corresponding images. These figures can be used to assess the quality of the current trajectory and system calibration. It is noted that when an image-based object/LiDAR point is close to the camera, no significant discrepancies are observed between the image-based and LiDAR-based lane markings. However, as the distance between an image-based object/LiDAR point and the camera increases, slight discrepancies between the image-based and LiDAR-based lane markings become apparent.

4.3. Quantitative Evaluation

Based on the centerline points derived (as explained in Section 3.3) from the three lane marking extraction strategies—LiDAR based, image based, and image-aided LiDAR—the performance is evaluated by comparing them to the reference data. The evaluation metrics for the 600 locations, as shown in Figure 5, are summarized in

Table 6. Performance metrics for different lane marking extraction strategies.

Lane Marking Extraction	Precision (%)	Recall (%)	F1-Score (%)
LiDAR based	93.2	87.6	90.3
Image based	88.5	69.4	77.8
Image-aided LiDAR	93.4	91.6	92.5

. Figure 40, Figure 41, Figure 42, Figure 43 and Figure 44 display samples of centerline points and corresponding images for each approach. The main findings are categorized based on (1) LiDAR based, (2) image based, and (3) image-aided LiDAR.

LiDAR-based lane marking extraction

The LiDAR-based approach demonstrates slightly lower precision (93.2%), recall (87.6%), and F1-score (90.3%) compared to the image-aided LiDAR strategy. This is expected, as LiDAR sensors are not affected by adverse weather and lighting conditions and have minimal occlusions, allowing them to outperform the image-based extraction. However, there is room for improvement in refining some lane markings (denoted by the red ovals and blue ovals/placemarks in Figure 40 and Figure 41), where image-aided LiDAR performs better. Overall, the LiDAR-based extraction has few commission (as represented by the precision) and omission (as represented by the recall) errors, and most extracted lane markings are true positives.

Image-based lane marking extraction

The image-based approach exhibits the lowest precision (88.5%), recall (69.4%), and F1-score (77.8%) compared to the other two strategies. The significantly lower recall is caused by the inevitable influence of excessive change in lighting conditions (denoted by the blue ovals/placemarks in Figure 42) and/or traffic occlusions (denoted by the blue ovals/placemarks in Figure 43). Additionally, the low resolution of the images significantly limits the image-based extraction approach, making it challenging to identify dash/dotted lines located beyond the driving lane on either side (denoted by the blue ovals/placemarks in Figure 44).

Image-aided LiDAR lane marking extraction

The image-aided LiDAR approach achieves the highest precision (93.4%), recall (91.6%), and F1-score (92.5%) compared to the other two strategies. The recall increases from 87.6% (LiDAR based) to 91.6% (image-aided LiDAR), surpassing the improvement from 93.2% (LiDAR based) to 93.4% (image-aided LiDAR) in precision. This suggests that the enhancement in lane marking extraction is more pronounced when addressing the omission errors in the LiDAR-based approach (as shown in Figure 32, Figure 40, and Figure 41) rather than compensating for commission errors. The F1-score also shows an increase from 90.3% (LiDAR based) to 92.5% (image-aided LiDAR), indicating that the image-aided LiDAR approach is capable of extracting most lane markings, and the image information indeed enhances the LiDAR-based extraction, as per the discussion in Section 4.2.

In summary, the image-aided LiDAR lane marking extraction achieves the best performance, followed by the LiDAR-based approach. The image information is particularly effective in compensating for the omission errors in the LiDAR-based approach. These findings align with the complementary nature of camera and LiDAR units emphasized in this study.

4.4. Discussion

In this study, the processing time for image-aided LiDAR lane marking extraction approaches was approximately 100 min per mile (including the duration required for LiDAR-based—8 min per mile—and image-based—85 min per mile—strategies) of the point clouds from four LiDAR units and the images from three cameras. For instance, on a 100-mile-long highway, the extraction of lane markings using the image-aided LiDAR framework would take approximately 168 h (7 days), which is less than the estimated service times for lane markings (white: 10.4~22.6 months; yellow: 15.6~39.7 months) on highways [49]. Thus, the proposed framework has the potential to provide routine updates to lane marking inventory throughout the lifespan of lane markings. One should note that the processing time in this study was not executed in parallel, and the use of parallel processing techniques could significantly reduce processing time.

Overall, the image-aided LiDAR lane marking extraction (F1-score: 92.5%) outperforms the LiDAR-based (F1-score: 90.3%) and image-based (F1-score: 77.8%) approaches. Nevertheless, LiDAR-based and image-based approaches have strengths and weaknesses for various potential applications. Image-based approaches may have inferior performance compared to LiDAR-based strategies for establishing lane marking inventory. However, image-based approaches are more suitable for autonomous vehicle applications when only lane markings along the driving lane need to be identified, owing to recent advancements in machine learning technology and camera affordability.

5. Conclusions and Recommendations for Future Research

This paper presents an image-aided LiDAR framework for establishing lane marking inventory. The framework utilizes lane markings extracted from images to enhance the accuracy of LiDAR-based extraction. Thereafter, intensity profiles and lane width estimates can be derived using image-aided LiDAR lane markings. The proposed image-aided LiDAR framework can handle lane markings within a 9-meter-extent on either side of the vehicle, as well as multiple imaging and LiDAR sensors mounted on an MMS. Additionally, this study developed a Potree-based web portal for visualizing imagery/LiDAR data. A Potree-based web portal was developed to include a function that facilitates the projection between 2D images and 3D point clouds, as well as tools for displaying intensity profiles and lane width estimates.

The performance of the proposed framework was evaluated using a dataset of 22,428 images and approximately 42,000 million LiDAR points collected along I-465 in the United States. Lane markings spanning around 110 miles (55-mile-long inner and outer loops) were extracted using the image-aided LiDAR approach, requiring an average of 100 min per mile for processing. The proposed framework improves the performance of lane marking extraction, as evidenced by the highest F1-score (92.5%) of the image-aided LiDAR approach, outperforming the LiDAR-based (90.3%) and image-based (77.8%) ones. Specifically, the recall increase of 0.4%—from 87.6% (LiDAR based) to 91.6% (image-aided LiDAR)—surpasses the slight improvement in the precision of 0.2%—from 93.2% (LiDAR based) to 93.4% (image-aided LiDAR). These findings indicate that the enhancement in LiDAR-based extraction is more pronounced when addressing omission errors rather than compensating for commission errors. On the other hand, the web portal can render the LiDAR datasets along I-465 in around ten seconds and visualize intensity profiles and lane width estimates. Additionally, users can select points of interest in an intensity profile/lane width plot, which will then be highlighted as points in the corresponding LiDAR data. Furthermore, the highlighted points can be projected onto the corresponding images where they are visible.

Future work will focus on leveraging the lane markings derived across different imaging and LiDAR sensors to enhance the alignment of imagery and LiDAR data. The lane markings derived from LiDAR point clouds can be backward projected onto corresponding images to identify any discrepancy between the LiDAR-based and image-based lane markings. By minimizing the discrepancy between conjugate lane markings extracted from multiple modalities, it will be possible to enhance the trajectory information of an MMS, as well as the calibration parameters of imaging and LiDAR sensors. Furthermore, future efforts will seek to reduce the execution time by utilizing parallel processing techniques. Reducing the execution time is particularly crucial to ensure that the proposed framework can offer regular updates for lane marking inventory over the lifespan of pavement markers.