Multispectral LiDAR Point Cloud Segmentation for Land Cover Leveraging Semantic Fusion in Deep Learning Network
Round 1
Reviewer 1 Report (New Reviewer)
The authors have proposed a deep learning network leveraging semantic fusion to accomplish segmentation of large-scale point clouds. The data used in the paper is convincing and the method is adequately described. Furthermore, the conclusions are reliable.
In general, the paper is well-organized and is acceptable after minor revision.
1. What's the inadequacy of existing studies of multispectral LiDAR data segmentation?
2. How to ensure the semantic information accuracy of point cloud feature extraction in the CSF block?
3. In Figure 10, it's difficult for the readers to find the deatails of difference. More specific partial enlarged drawings should be given.
4. The following papers could be referenced.
a) Hepi H. Handayani, Arizal Bawasir, Agung B. Cahyono, Teguh Hariyanto & Husnul Hidayat (2022) Surface drainage features identification using LiDAR DEM smoothing in agriculture area: a study case of Kebumen Regency, Indonesia, International Journal of Image and Data Fusion, DOI: 10.1080/19479832.2022.2076160.
b) Xiangguo Lin & Wenhan Xie (2022) A segment-based filtering method for mobile laser scanning point cloud, International Journal of Image and Data Fusion, 13:2, 136-154, DOI: 10.1080/19479832.2022.2047801.
Author Response
Response Letter
Wish all is well. Thanks a lot for your professional and insightful comments which make this paper well revised and improved. The following is the response to comments on this paper.
Response：
1) What's the inadequacy of existing studies of multispectral LiDAR data segmentation?
Answer: First of all, due to the disorder, irregularity, non-uniformity and unstructured characteristics of 3D point clouds, it can not be directly applied to the traditional CNN framework to directly process point cloud. Researchers have been exploring deep learning networks that can be directly applied to 3D point cloud classification, and have made many meaningful achievements as described in this paper. Although these methods have achieved essential results in point clouds semantic segmentation tasks, some details of object boundaries are lost when segmenting objects, resulting in incomplet object segmentation. Especially, when two objects with similar structures are close, this problem is more serious. Thus, fine-grained segmentation of point cloud scenes from the multi-spectral LiDAR data is still a tough task.
2) How to ensure the semantic information accuracy of point cloud feature extraction in the CSF block?
Answer: In the CSF block, we encode and concatenate features separately, and output the fused features. Specifically, we use the center point coordinates, coordinate differences, center point features and feature point differences to encode respectively, and then concatenate them. CSF Block fuses local geometry and feature content based on 3D spatial geometry correlation, encodes the correlation of local geometry and feature content separately, and finally concatenates them together, which can provide certain sequential information. Finally, the CSF block ensure the accuracy of semantic information in this way.
3) In Figure 10, it's difficult for the readers to find the deatails of difference. More specific partial enlarged drawings should be given.
Answer: Thanks for your attentive and professional advice. We provide more specific partial enlargements in Figure 10.
4) The following papers could be referenced.
Answer: Thank you for your recommendation, which we have quoted in this paper reference[8][9].
Author Response File: 
Author Response.pdf
Reviewer 2 Report (New Reviewer)
Dear Authors:
Your article: “Multispectral LiDAR Point Cloud Segmentation for Land Cover Leveraging Semantic Fusion in Deep Learning Network” presents a study with emerging technologies: Multispectral Lidar and Deep Learning. The use of a single data source (Lidar) with the high accuracy of Deep Learning methods finds a very wide range of applications. This is even an aspect that can be better explored in chapter 1. Chapter 1 is well written and introduces well the challenges and approach to the problem. The use of the Singular Value Decomposition is one of the differentials of your article and that can be better explored in chapter 2.1. In this case, it would even be interesting to separate this topic into a specific chapter for it. Regarding the writing of the article, I pointed out comments in the digital file and request your attention to the following questions:
1) Abstract: Review the text. It would be interesting to cite the place of study and highlight the best result obtained.
2) Line 131: it would be interesting to mention the sensors used to capture the point clouds, as well as the characteristics of the flight and the point/density clouds. What is the total area of the study? What is the data acquisition date?
3) Are point clouds public? If yes, please quote the data repository.
4) Line 139: 8.59 million points? What is the density of the point cloud? It would be interesting to better explain the issue of the number of points because normally Lidar point clouds can contain billions of points.
5) It would be interesting to add a scale bar in figures 2, 7, 9, and 10 to facilitate the understanding of the dimensions of the areas.
6) Mention the terrestrial reference system and the cartographic projection used for the coordinates?
4) Lines 182 to 189: deepen the discussion of dimensionality reduction, which is an interesting point of the article.
5) Please detail the software resources used in the study.
6) About the developed code will be available on GitHub?
7) Line 352: Detail the processing of the ground truth
8) Chapter 3.2: It would be interesting to compare the results obtained in your study with those obtained by authors cited in the bibliographic references.
9) From Table 4, it is possible to see that there was an improvement in the results used in SVD. In terms of computational processing, there was a reduction in time?
10) Explore the issue of processing time in table 2.
11) Conclusions: please check if the proposed objectives were duly answered here.
I conclude by congratulating them for the high-level work carried out and for the presented version of the article.
Respectfully,
Comments for author File: Comments.pdf
Author Response
Response Letter
Wish all is well. Thanks a lot for your professional and insightful comments which make this paper well revised and improved. We responded to your comments one by one in the pdf-review. And submitted a revised manuscript. Here I would like to make a few important replies.
1.Abstract: Review the text. It would be interesting to cite the place of study and highlight the best result obtained.
Answer：Thanks for your attentive reading and professional advice. According to your suggestion, we have modified the highlighted part of the annotation and deleted unnecessary words like "above" and "below". Besides, we separate the topic of SVD into a specific section 2.2.
2.Line 131: it would be interesting to mention the sensors used to capture the point clouds, as well as the characteristics of the flight and the point/density clouds. What is the total area of the study? What is the data acquisition date?
Answer：The data was collected from the Titan Multispectral Airborne Lidar system, which contains three effective laser wavelengths of 1550nm, 1064nm and 532nm. Capable of capturing discrete and full-waveform data from all three wavelengths, the Titan system has a combined ground sampling rate up to 1 MHz. The scan angle varied between ± 20â—¦ across track from the nadir, and the Titan system acquired points at around 1075 m altitude with 300 kHz Pulse Repetition Frequency (PRF) per wavelength, and 40 Hz scan frequency. The total number of points in the entire scenario is 8.52 million, covering an area of about 25 square kilometers. The data was collected and labeled in 2021. We have added relevant descriptions.
3. Are point clouds public? If yes, please quote the data repository.
Answer：I'm sorry that the data is proprietary due to project requirements.
4. Line 139: 8.59 million points? What is the density of the point cloud? It would be interesting to better explain the issue of the number of points because normally Lidar point clouds can contain billions of points.
Answer：The actual LiDAR experimental data collected came from a small town (the center of latitude 44°02’25", longitude -79°17’00"). The average point density is about 3.6 points/m2. The total number of points in the entire scenario is 8.52 million, covering an area of about 25 square kilometers.
5. It would be interesting to add a scale bar in figures 2, 7, 9, and 10 to facilitate the understanding of the dimensions of the areas.
Answer：We added a scale bar to the lower right corner of the corresponding figures.
6. Mention the terrestrial reference system and the cartographic projection used for the coordinates?
Answer：The system collects the points about 1075m altitude from the terrestrial reference system. The location relationship between point clouds is used in the deep learning network, which is not affected by the the terrestrial reference system and the cartographic projection.
7. Lines 182 to 189: deepen the discussion of dimensionality reduction, which is an interesting point of the article.
Answer：We discussed dimensionality reduction more deeply in this paper.
8. Please detail the software resources used in the study
Answer：We added the description of software resources in Section 3.1.
9. About the developed code will be available on GitHub?
Answer：We will publish it on GitHub after authorization.
10. Line 352: Detail the processing of the ground truth
Answer：Through CloudCompare software, the groundtruth is manually marked point by point. We have added relevant descriptions.
11. Chapter 3.2: It would be interesting to compare the results obtained in your study with those obtained by authors cited in the bibliographic references.
Answer：Yes, we made detailed comparison in this chapter3.3.
12. From Table 4, it is possible to see that there was an improvement in the results used in SVD. In terms of computational processing, there was a reduction in time?
Answer：We use SVD results as input data to train the network. In fact, it is impossible to improve computing processing and reduce time without changing the network structure.
13. Explore the issue of processing time in table 2
Answer：In this paper, the trained model is used to predict. The processing time predicted by the models is within a few minutes, so there is no special record. We have added relevant explanations in the paper.
14. Conclusions: please check if the proposed objectives were duly answered here.
Answer：Thanks again for your professional and insightful comments. We responded to your comments one by one and marked them in red.
Author Response File: Author Response.pdf
Reviewer 3 Report (New Reviewer)
The manscript is related to multispectral Lidar point cloud segmentation using deep learning. It is interesting study. But it needs details in some points given below.
1. How was ground truth segmentation done. Please introduce them.
2. Page11/Line 397: Road and open ground confison on segmentation is probably related to the local geometry. Please discus this result on discussion section. Can the local geometry based segmentation be improved?
3. You applied SVD to remove redundancy of three point cloud. Please give details about determination of reflectance to point sets. Because you have three reflectances. How did you use these reflectances on classification.
Author Response
Response Letter
Wish all is well. Thanks a lot for your professional and insightful comments which make this paper well revised and improved. The following is the response to comments on this paper.
Response：
1. How was ground truth segmentation done. Please introduce them.
Answer: Through CloudCompare software, the groundtruth is manually marked point by point. We have added relevant descriptions in this paper.
2. Page11/Line 397: Road and open ground confusion on segmentation is probably related to the local geometry. Please discus this result on discussion section. Can the local geometry based segmentation be improved?
Answer: As you said, road and open ground segmentation may be related to local geometry. If special processing is designed for these two types of point clouds, it may be improved. However, further essential improvement may require more dimensional prior information. We have added relevant descriptions in the discussion section.
3. You applied SVD to remove redundancy of three point cloud. Please give details about determination of reflectance to point sets. Because you have three reflectances. How did you use these reflectances on classification.
Answer: We input the reflectances of the three channels into the deep learning network model as one of the multidimensional features of the point clouds for training, and finally achieve the prediction of the point cloud categories through the trained network model. We didn't take into account the determination of reflectance to point sets, but it also gave us new ideas. Thanks again for your professional and insightful comments.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report (New Reviewer)
Dear Authors
Analyzing the new version of your article: "Multispectral LiDAR Point Cloud Segmentation for Land Cover Leveraging Semantic Fusion in Deep Learning Network" I verified that you adequately implemented or justified my comments.
Thank you for sending the cover letter that helps in verifying the implemented changes.
Respectfully,
Author Response
Dear reviewer,
Thank you very much for your efforts and recognition of our paper. And wish you a happy New Year!
With best regards
This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.
Round 1
Reviewer 1 Report
Dear Authors.
First, I would like to thank the authors for their work and determination in carrying out this study. The manuscript entitled „Multispectral LiDAR Point Cloud Segmentation for Land Cover Leveraging Semantic Fusion in Deep Learning Network” contains:
- Introduction to the manuscript.
- Materials and methods (LiDAR, deep learning network.
- Experiments and Results.
- Discussion
- Conclusions.
The manuscript presents the case study, which deals with designing a deep learning network based on semantic fusion information for the semantic segmentation task of a multispectral LiDAR point cloud.
I have comments on the following manuscript:
The study area in Canada is incorrectly located. The location listed is for Almaty Province in Kazakhstan, east of Altyn Emel National Park. On p. 3. Fig. 1 b) has its center at latitude 44°02’25”, longitude “-“79° 17’00” (or add W before the value of longitude and add N before the value of latitude). Moreover, in the text, please add that Fig. 1 c) is study area 1. It would be appropriate to add a graphic scale to every graphical display.
The results of the tested areas 11,12,13 are shown in the manuscript. It would be suitable for fig. 7 to insert orthophotos of the selected study areas so that the reader can accurately compare reality vs. lidar results and below case study processing results. Are the chosen areas turned into a 3D view? Why are not the borders of areas rectangular (angles are not 90 degrees)?
Based on the analysis of the processing results presented in the tables, the proposed processing method provides one of the best results, even if it is not 100%. Still, the success of the results is already approaching this value.
The authors could also list the software in which they processed and visualized the results, e.g., I assume with high confidence that the tested areas 11-13 are displayed in the Cloud Compare software.
I miss a comparative discussion about the results achieved with previous scientific papers.
The manuscript has further formal mistakes or deficiencies, which will be removed in the eventual process of the check English and final editing of the paper by the editorial office according to the valid journal standards.
Best regards
Author Response
Wish all is well. Thanks a lot for your professional and insightful comments which make this paper well revised and improved. The following is the response to comments on this paper.
Response：
1) The study area in Canada is incorrectly located. The location listed is for Almaty Province in Kazakhstan, east of Altyn Emel National Park. On p. 3. Fig. 1 b) has its center at latitude 44°02′25″, longitude “-“79°17′00″(or add W before the value of longitude and add N before the value of latitude).
Answer: Thank you for your attentive and professional advice. We corrected the location of the study area and added a scale in the figure.
2) Moreover, in the text, please add that Fig. 1 c) is study area 1. It would be appropriate to add a graphic scale to every graphical display.The results of the tested areas 11,12,13 are shown in the manuscript. It would be suitable for fig. 7 to insert orthophotos of the selected study areas so that the reader can accurately compare reality vs. lidar results and below case study processing results. Are the chosen areas turned into a 3D view? Why are not the borders of areas rectangular (angles are not 90 degrees)?
Answer: Yes, we used Cloud Compare software to convert to 3D view. In the display figure, we use the default display Angle in cloudcompare software, and there is a certain Angle deflection, so the boundary of the display area is not rectangular. At the same time, the collected point cloud also has the characteristic of irregular distribution, and the boundary does not meet the distribution condition of strict rectangle is also one of the reasons.
3) Based on the analysis of the processing results presented in the tables, the proposed processing method provides one of the best results, even if it is not 100%. Still, the success of the results is already approaching this value.The authors could also list the software in which they processed and visualized the results, e.g., I assume with high confidence that the tested areas 11-13 are displayed in the Cloud Compare software.
Answer: Yes, as you mentioned, we used Cloud Compare software to convert to 3D view. We have presented the software for processing and visualizing the results. Thanks again for your professional comments.
Author Response File: Author Response.pdf
Reviewer 2 Report
Dear authors
congratluation for submitting an interesting manuscript ot the journal.
Your manuscript presents a well-formulated methodical comparison and methodical improvement of an existing deep learning network that leverages semantic fusion information for the multispectral LiDAR point cloud semantic segmentation task.
In detail, you have successfully worked on fusing local geometry and feature content based on 3D spatial geometric associativity and embed it into a backbone network. In addition, to cope with the problem of redundant point-cloud feature distribution found in your experiment, you smartly designed a data preprocessing with principal component extraction to improve the processing capability of the proposed network on the applied multispectral LiDAR data.
The manuscript is fluently written and well formulated and presents methodical and experimental findings on a high scientific level. You, the authors have set up a well-designed and standardized experiment to compare existent deep learning networks to semantically segments large scale point cloud features and received reproducible findings that are worth to publish.
Your paper does present novel and innovative improvement of method presented in addition to a well-designed empirical comparison of different methods in a standardized experiment. Findings of such comparative experiments have not been presented earlier yet, to my best knowledge.
However, the submitted manuscript requires several minor and a few major revision and improvement before it can be accepted for publication in the journal considered. In detail, I suggest to consider present a graphical workflow model of your data processing line, a strong revision of two major figures No 1 and No 2 and finally a full revision and reformulation of a real discussion section.
In addition, some sections like the introduction and the methods and material section can be improved, by adding further reference, where mentioned in my revision of your paper.
Please acknowledge my comments in the uploaded review document using the pdf-review function.
I wish you success in presenting an improved revised version within a few weeks.
All the best and good luck, take care.
Comments for author File: Comments.pdf
Author Response
Wish all is well. Thanks a lot for your professional and insightful comments which make this paper well revised and improved. We responded to your comments one by one in the pdf-review. And submitted a revised manuscript. Here I would like to make a few important replies.
1) About rewriting the discussion section. After careful consideration, we have rewritten the discussion section with reference to the format of previous work reference 7, which includes discussion withablation experiment. Thank you again for your professional advice.
Author Response File: Author Response.pdf
Reviewer 3 Report
Based on the deep learning method, this paper studies the classification of Titan multi-spectral lidar point cloud data. Firstly, SVD (singular value decomposition) is used to solve the problem of feature redundancy. Then the Contextual Semantic Fusion (CSF Block) is embedded to improve RandLA-Net and make it suitable for multi-spectral point cloud data.
We have several questions about this paper:
1. First, we want to confirm which features are displayed in the feature map in Fig. 2? Please give the explanation. We guess it is the 3 channel spectral value of the point cloud. Secondly, in line 158, the article proposed "the distribution of point cloud features of different categories shows features with high density and high similarity ", but we do not think that this conclusion can be reached by giving fig 2. Because the points in Fig. 2 are very dense and coincide with each other, so effective information cannot be obtained.
2. For the SVD algorithm, which features of the point cloud are extracted by principal components ? If the principal component extraction is carried out for two different types of information : coordinate and spectral information ( 6 dimensions ), is it logical ? Is it necessary if only principal component extraction is performed on spectral information ( 3 dimensions ) ? In remote sensing image processing, principal component extraction is generally carried out for hyperspectral images. Although the results show that it is effective, is it appropriate to use this method only for 3 spectral segments? In addition, we also want to know whether the feature map after extracting the principal components is obviously different from that in Fig. 2.
3. For RandLA-Net, the advantage of the network is that it can handle large-scale point clouds, which is matched with the characteristics of airborne lidar point cloud data. The specific patch size is not given in this paper. We believe it is a key parameter.
4. In RandLA-Net, the Divided Residential Block is composed of two groups of LocSE + AP, which is inconsistent with the statement in Fig4. Please give the explanation.
5. In the Contextual Semantic Fusion (CSF Block) proposed in this paper, the Relative Position and Relative Feature are encoded and concatenated respectively to output fused features. However, we note that this is basically similar to the Local Spatial Encoding (LocSE) in the original model. Whether it can be considered that the function of the Divided Resident Block + CSF Block is similar to that of the Divided Resident Block with three groups of LocSE + AP. The performance improvement of CSF block comes from the gradually expanding receptive field. If so, the contribution of this part of the paper can be ignored.
6. Most of the pictures in the paper are not clear. Table borders are not uniform ( table 2, table 3 ). Please check if the border of table 4 meets the requirements.
Author Response
Wish all is well. Thanks a lot for your professional and insightful comments which make this paper well revised and improved. The attachment is the response to comments on this paper.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Dear authors,
thank you for considerung about my review comments, remarks and suggetsions which you address more or less.
After your successful revision, I consider your manuscript academically ready for publication.
Thank you and congraluation for your success.
Reviewer 3 Report
First, the similarity presented in Figure 2 is subjective. Because the original data is directly input into the classification model, good accuracy can also be obtained. And, I noticed that after doing SVD, the "similarity" mentioned by the author is not resolved.
Second, I think the CSF block is very similar to the LocSE block, although the input feature types are different.
