Generalized Sparse Convolutional Neural Networks for Semantic Segmentation of Point Clouds Derived from Tri-Stereo Satellite Imagery

Round 1

Reviewer 1 Report

This paper study the application of GSCNN for segmentation of point clouds obtained by tri-stero satellite imagery. 

It's a good novel manuscript: provide sufficient background and include relevant and very recent references. The research design is appropriate and correctly described, also the presentation of results. Finally the discussion of results support the conclusions presented.

However it would be interesting and appropriate to use a multi-class extension of Matthew's correlation coefficient to summarize the confusion matrices (presented in Appendix B) into a single value in the [-1,1] range. Futhermore, in order to achieve a clear comparison between classifiers, normalized confusion matrices can be defined instead (see methodology section of doi:10.3390/s17030594 for how compute them).

Author Response

Dear Reviewer One,

thank you for taking time to review our paper! In particular, for your precise critique. Please find our response below.

Please note that all statements refer to the revised manuscript.

1) Normalized (along the rows) Confusion Matrices
We have added them in Appendix section B.

2) Matthew’s Correlation Coefficient
We have added them as well. In particular, they are now added to Table 3, and mentioned it in the introduction (Line Numbers 79-80), results (Line Numbers 501-503) and conclusion (Line Numbers 563-564).

Thank you again for your time and effort!

Yours Sincerely
Stefan Bachhofner (on behalf of all authors)

Reviewer 2 Report

In this paper, the authors proposed an approach based on a Generalized Sparse Convolutional Neural Networks for semantic segmentation using point clouds derived from tri-stereo satellite imagery. In fact, the 3D geometric information has been used as a useful feature to perform the semantic segmentation. The proposed approach has been validated on remote sensing images.

Generally, the proposed idea is very interesting, however, some minor revisions have to be made and some parts of the manuscript are not complete to claim the advantage of the proposed models :

1)  Could the authors explain what is the main contribution of the proposed approach than over existing methods of the state of the art?  Could the authors add more clarifications?

2)   Moreover, what is the motivation to use the Decision Tree model?

3)  In the experimental setup, did the authors randomly choose the training and testing samples?  If that is the case, what happens when you change the training samples?

4)  I suggest the authors add in the manuscript these references related to 3-D CNN, which aim to preserve the spectral and spatial feature of remote sensing images :

- Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network, Remote Sensing, 2017.

- Hyperspectral imagery classification based on semi-supervised 3-D deep neural network and adaptive band selection, Expert Systems with Applications, 2019.

  5)  The English and format of this manuscript should be checked very carefully.

Author Response

Dear Reviewer Two,

thank you for taking time to review our paper! In particular, for your precise critique. Please find our response below.

Please note that line numbers that refer to the submitted manuscript are prefixed with SM, while line numbers that refer to the revised manuscript are prefixed with RM.

1) The main advantages of a Deep Learning algorithm is three fold (among others). First, Deep Learning may require little or no feature engineering (compared to other approaches). Second, perhaps more importantly, Deep Learning offers the opportunity of transfer learning, which has been shown to give a performance boost on other tasks. Third, using Deep Learning in this use case would add another task for which it can be used. 

2) The motivation is to compare it to a supervised feature learning technique (SM: See line number 78-79). This approach uses exactly the same data structure (the point cloud) with exactly the same original observations (coordinates and color).

3) No, we did not. The data set was split such that the training and test set have the same number of occurrences for each class (SM: See line number 240-242). We agree with you that systematically altering this distribution is an interesting and important research. However, this is out of scope for the paper.

4) Thank you for the two references! After carefully studying them, we did come to the conclusion to include these works in our paper. We, in particular, added them in Section 2.2, Deep Learning on Satellite Images (RM: Line Numbers 139-144). By doing so, we also include work that is done with the cross-correlation operation in 3 dimensions where the gird-assumption does hold. We also achieve a smoother transition from Section 2.2 to Section 2.3 with these references.

5) The manuscript was carefully re-read. We improved the text at a number of locations. We are convinced that the English language usage is fine in the resubmitted version. Should the reviewer disagree, we will use an English editing service.

Thank you again for your time and effort!

Yours Sincerely
Stefan Bachhofner (on behalf of all authors)

Reviewer 3 Report

Authors have presented CNN method using semantic segmentation for satellite imagery. However, I would recommend them to study the results using the following methods and observe compare the same with their existing results. Also, it would be nice if you could study the results using the following methods applied for other applications. 

- https://ieeexplore.ieee.org/abstract/document/8556722

Author Response

Dear Reviewer Three,

thank you for taking time to review our paper! Please find our answer below.

Please note that line numbers that refer to the submitted manuscript are prefixed with SM, while line numbers that refer to the revised manuscript are prefixed with RM.

We carefully read and reviewed your recommended article. We agree that the presented approach to overcome class imbalance is interesting and may provide a performance boost on our data set. And it is also clear that it would add to the paper if we conduct additional experiments. However, the main objective of our paper is not to boost the performance as far as we possibly can with sophisticated techniques. Nor is the main objective to compare different types of approaches that combat class imbalance. In our paper, we contribute to the question if the semantic segmentation problem of tri-stereo derived point clouds is learnable at all, in particular for deep learning algorithms with the generalized sparse convolution. In the paper, we argue that our experiments indicate that this indeed seems to be the case, for the decision tree and the deep learning algorithm. Please also consider that we don’t have the resources to conduct additional experiments as we don’t have funding any more. For this reasons, we have decided to not conduct the presented approach in our paper. We have, however, decided to include this work as a reference in the discussion section as the class imbalance problem and the presented solution is a clear connection between the two works (RM: Line Numbers 534-537).

Thank you again for your time and effort!

Yours Sincerely
Stefan Bachhofner (on behalf of all authors)