Author Contributions
Conceptualization, W.D. and S.C.; methodology, S.C. and Z.H.; software, W.D. and S.C.; validation, W.D. and S.C.; formal analysis, W.D. and S.C.; investigation, W.D. and S.C.; resources, W.D.; data curation, S.C.; writing—original draft preparation, S.C.; writing—review and editing, W.D., S.C., Z.H., Y.X. and D.K.; visualization, S.C.; supervision, W.D.; project administration, W.D.; funding acquisition, W.D. All authors have read and agreed to the published version of the manuscript.
Figure 1.
Intensity measurements. The attenuation in the traveling and the incident angle can influence the received intensity. Therefore, from different views, the intensity values are different for the same position.
Figure 1.
Intensity measurements. The attenuation in the traveling and the incident angle can influence the received intensity. Therefore, from different views, the intensity values are different for the same position.
Figure 2.
Densified intensity and depth maps obtained from multiple frames: (a) densified intensity (b) densified depth.
Figure 2.
Densified intensity and depth maps obtained from multiple frames: (a) densified intensity (b) densified depth.
Figure 3.
Distance compensation, incidence normalization, and multi-view fusion. The pixels on the image are enlarged five times to increase the visualization. The raw intensity measurement is compensated by the corresponding depth map and merged by poses to obtain . The incident angle is obtained from of the fused depth map to avoid the error caused by the normal estimation in the sparse point cloud. The multiple normalized intensity maps are fused to obtain a densified one.
Figure 3.
Distance compensation, incidence normalization, and multi-view fusion. The pixels on the image are enlarged five times to increase the visualization. The raw intensity measurement is compensated by the corresponding depth map and merged by poses to obtain . The incident angle is obtained from of the fused depth map to avoid the error caused by the normal estimation in the sparse point cloud. The multiple normalized intensity maps are fused to obtain a densified one.
Figure 4.
Inverse reproduction. The densified will be inversed to produce more dense artificial intensity maps. This process is the inverse process of the above steps except for the multi-view fusion.
Figure 4.
Inverse reproduction. The densified will be inversed to produce more dense artificial intensity maps. This process is the inverse process of the above steps except for the multi-view fusion.
Figure 5.
LiDAR-Net architecture. The pixels on the input are enlarged five times to increase the visualization. The output
and
of the backbone will be input for the inverse normalization network, as shown in the upper left corner. Because it is difficult to quickly estimate the normal using
, the inverse normalization network attempts to fit the inverse normalization process shown in Equation (
7) with several convolution layers and predicts the dense
supervised by
.
Figure 5.
LiDAR-Net architecture. The pixels on the input are enlarged five times to increase the visualization. The output
and
of the backbone will be input for the inverse normalization network, as shown in the upper left corner. Because it is difficult to quickly estimate the normal using
, the inverse normalization network attempts to fit the inverse normalization process shown in Equation (
7) with several convolution layers and predicts the dense
supervised by
.
Figure 6.
The overview of the dataset. Four different instances are demonstrated. Each pixel in (a,b) is enlarged five times to increase the visualization. Colder color in the depth map indicates a farther distance, while colder color in the intensity map indicates a weaker strength. The dataset consists of two inputs: sparse depth and sparse intensity. It has three ground-truth values: densified depth, artificial intensity , and normalized intensity . eliminates the influence of distance and the incidence angle, resulting in sharper edges between different materials and a better consistency within the same material: (a) sparse input depth ; (b) sparse input intensity ; (c) densified depth ; (d) artificial intensity ; (e) normalized intensity .
Figure 6.
The overview of the dataset. Four different instances are demonstrated. Each pixel in (a,b) is enlarged five times to increase the visualization. Colder color in the depth map indicates a farther distance, while colder color in the intensity map indicates a weaker strength. The dataset consists of two inputs: sparse depth and sparse intensity. It has three ground-truth values: densified depth, artificial intensity , and normalized intensity . eliminates the influence of distance and the incidence angle, resulting in sharper edges between different materials and a better consistency within the same material: (a) sparse input depth ; (b) sparse input intensity ; (c) densified depth ; (d) artificial intensity ; (e) normalized intensity .
Figure 7.
Trajectories of the two scenes in the dataset.
Figure 7.
Trajectories of the two scenes in the dataset.
Figure 8.
Two types of intensity ground truth (enhanced for visualization): (a) artificial intensity ; (b) normalized intensity . The boundaries of different materials of are much more distinguishable. In addition, the intensity of the same material shows consistency in . (a) . (b) .
Figure 8.
Two types of intensity ground truth (enhanced for visualization): (a) artificial intensity ; (b) normalized intensity . The boundaries of different materials of are much more distinguishable. In addition, the intensity of the same material shows consistency in . (a) . (b) .
Figure 9.
Histogram statistics before (left) and after (right) the incidence normalization. The red and green columns represent the statistical results of the lane and the road areas, respectively. Gaussian curves were used to fit their mathematical distributions. The smaller overlapping area and the sharper distribution indicate a better result after normalization.
Figure 9.
Histogram statistics before (left) and after (right) the incidence normalization. The red and green columns represent the statistical results of the lane and the road areas, respectively. Gaussian curves were used to fit their mathematical distributions. The smaller overlapping area and the sharper distribution indicate a better result after normalization.
Figure 10.
Input (a,b) and output (c,d). The proposed completion system takes the sparse depth and intensity from a LiDAR sensor as input (row 1) to obtain the dense completion result (row 2). Each pixel in (a,b) is enlarged five times to increase the visualization: (a) sparse input intensity; (b) sparse input depth; (c) intensity completion; (d) depth completion.
Figure 10.
Input (a,b) and output (c,d). The proposed completion system takes the sparse depth and intensity from a LiDAR sensor as input (row 1) to obtain the dense completion result (row 2). Each pixel in (a,b) is enlarged five times to increase the visualization: (a) sparse input intensity; (b) sparse input depth; (c) intensity completion; (d) depth completion.
Figure 11.
Comparison of intensity completion. Colder color in the intensity map indicates weaker strength: (
a) Ip-Basic [
51]; (
b) sparse-to-dense [
15]; (
c) pNCNN [
50]; (
d) ours.
Figure 11.
Comparison of intensity completion. Colder color in the intensity map indicates weaker strength: (
a) Ip-Basic [
51]; (
b) sparse-to-dense [
15]; (
c) pNCNN [
50]; (
d) ours.
Figure 12.
Qualitative analysis of ablation study: (a) the intensity map completed by the single input model supervised by ; (b) the completion result from using only the proposed completion backbone network supervised by and ; (c,d) are and completed by LiDAR-Net supervised by , , and ; (a) from onlyI (); (b) from DI-to-DI (+); (c) from LiDAR-Net (++); (d) from LiDAR-Net (++).
Figure 12.
Qualitative analysis of ablation study: (a) the intensity map completed by the single input model supervised by ; (b) the completion result from using only the proposed completion backbone network supervised by and ; (c,d) are and completed by LiDAR-Net supervised by , , and ; (a) from onlyI (); (b) from DI-to-DI (+); (c) from LiDAR-Net (++); (d) from LiDAR-Net (++).
Figure 13.
Convergence curves. LiDAR-Net (green) converges on intensity (left) faster than DI-to-DI (yellow) and onlyI (blue) and achieves better depth (right) completion performance.
Figure 13.
Convergence curves. LiDAR-Net (green) converges on intensity (left) faster than DI-to-DI (yellow) and onlyI (blue) and achieves better depth (right) completion performance.
Figure 14.
Lane segmentation results in good illumination conditions. Under normal lighting conditions, the dense intensity after completion can reach the same performance as RGB images in lane segmentation. It shows that traditional vision methods can be applied to the LiDAR intensity maps without modification: (a) RGB; (b) intensity.
Figure 14.
Lane segmentation results in good illumination conditions. Under normal lighting conditions, the dense intensity after completion can reach the same performance as RGB images in lane segmentation. It shows that traditional vision methods can be applied to the LiDAR intensity maps without modification: (a) RGB; (b) intensity.
Figure 15.
Lane segmentation results in complex illumination conditions. The upper and lower rows are the data under two different illumination conditions. The left and right images are the data from a visible light camera and LiDAR, respectively. Different colored lines indicate the segmentation results of different lanes. It shows that the LiDAR intensity maps have great potential for applications under adverse illumination conditions: (a) RGB; (b) intensity.
Figure 15.
Lane segmentation results in complex illumination conditions. The upper and lower rows are the data under two different illumination conditions. The left and right images are the data from a visible light camera and LiDAR, respectively. Different colored lines indicate the segmentation results of different lanes. It shows that the LiDAR intensity maps have great potential for applications under adverse illumination conditions: (a) RGB; (b) intensity.
Table 1.
Intensity consistency of the same object. The closer the value is to 1, the greater the similarity of the same object’s intensity distribution in two intensity maps.
Table 1.
Intensity consistency of the same object. The closer the value is to 1, the greater the similarity of the same object’s intensity distribution in two intensity maps.
|
I
| | |
---|
| 1 | 0.72 | 0.97 |
Table 2.
Intensity consistency of different objects. Similar values indicate that the relative intensity distributions of different objects in the two images are similar.
Table 2.
Intensity consistency of different objects. Similar values indicate that the relative intensity distributions of different objects in the two images are similar.
|
I
| | |
---|
| 0.396 | 0.317 | 0.407 |
Table 3.
Comparison of intensity completion accuracy. The results from state-of-the-art completion algorithms are shown in the bottom part. The best results are shown in bold; ’x’ denotes a failure.
Table 3.
Comparison of intensity completion accuracy. The results from state-of-the-art completion algorithms are shown in the bottom part. The best results are shown in bold; ’x’ denotes a failure.
| | | Scene 1 | Scene 2 | Mean |
---|
| | | Intensity | Intensity | Intensity |
Method | Input | Type | RMSE | MAE | RMSE | MAE | RMSE |
LiDAR-Net (Ours) | intensity + depth | learning | 20.332 | 13.449 | 28.137 | 18.392 | 24.234 |
Sparse-to-dense [15] | single intensity | learning | 20.676 | 13.696 | 28.570 | 18.767 | 24.623 |
SparseConvs [19] | single intensity | learning | 25.942 | 17.460 | 36.055 | 27.150 | 30.999 |
nConv-CNN [39] | single intensity | learning | x | x | x | x | x |
pNCNN [50] | single intensity | learning | 22.131 | 14.911 | 29.539 | 19.928 | 25.835 |
IP-Basic [51] | single intensity | non-learning | 28.725 | 17.957 | 56.374 | 35.784 | 42.550 |
Table 4.
Quantitative analysis of ablation study. The best results are shown in bold. The ablation study indicates that intensity–depth fusion and supervision with normalized intensity can improve the performance.
Table 4.
Quantitative analysis of ablation study. The best results are shown in bold. The ablation study indicates that intensity–depth fusion and supervision with normalized intensity can improve the performance.
| Scene 1 | Scene 2 | Mean |
---|
| Intensity | Intensity | Intensity |
Method | RMSE | MAE | RMSE | MAE | RMSE |
onlyI () | 20.676 | 13.696 | 28.570 | 18.767 | 24.623 |
DI-to-DI (+) | 20.454 | 13.556 | 28.237 | 18.582 | 24.346 |
LiDAR-Net (++) | 20.332 | 13.449 | 28.137 | 18.392 | 24.234 |
Table 5.
Comparison of depth completion accuracy. The results from state-of-the-art depth completion algorithms are shown in the bottom part. The best results are shown in bold. We use ‘i’ and ‘d’ to represent Lidar intensity and depth, respectively; ‘x’ denotes a failure.
Table 5.
Comparison of depth completion accuracy. The results from state-of-the-art depth completion algorithms are shown in the bottom part. The best results are shown in bold. We use ‘i’ and ‘d’ to represent Lidar intensity and depth, respectively; ‘x’ denotes a failure.
| | Scene 1 | Scene 2 | Mean |
---|
| | Depth [mm] | Depth [mm] | Depth [mm] |
Method | Input | RMSE | MAE | RMSE | MAE | RMSE |
LiDAR-Net (Ours) | i + d | 3822.5 | 1300.2 | 5093.0 | 1974.5 | 4457.8 |
Sparse-to-dense [15] | single d | 3900.1 | 1310.2 | 5226.3 | 2165.3 | 4563.2 |
SparseConvs [19] | single d | 7134.5 | 3162.3 | 9486.8 | 4271.21 | 8310.7 |
NConv-CNN [39] | single d | 5190.1 | 1725.2 | 6534.8 | 2425.7 | 5862.4 |
pNCNN [50] | single d | 3956.8 | 1110.4 | 5104.4 | 1816.0 | 4530.5 |
IP-Basic [51] | single d | 6645.9 | 1934.9 | 8521.6 | 2159.7 | 7583.8 |
Table 6.
Comparison of lane segmentation results. .
Table 6.
Comparison of lane segmentation results. .
Input Type | Precision | Recall | |
---|
RGB image from visible cameras | 0.957 | 0.612 | 0.746 |
from LiDAR-Net | 0.862 | 0.553 | 0.674 |