Transferability of Recursive Feature Elimination (RFE)-Derived Feature Sets for Support Vector Machine Land Cover Classification

Round 1

Reviewer 1 Report

 

In this paper, the author discusses the influence of feature selection to the accuracy of supervised classification. Moreover, the author also designs a series of experiments to show the transferability of features between different images and training sets. The paper is well written and conclusions are useful for researches in this field.

 

The following are some detailed comments:

 

 

l  Since the author only discusses the results on SVM, he/she should change the “Supervised Machine Learning” to “SVM”

l  In section 2.3, the author should present more details on how he/she create the samples sets. Manually selected? Where is the ground truth from?

l  Line 120-121, 0.49nm, 0.56nm, 0.665nm, 0.842nm should be 0.49μm, 0.56μm, 0.665μm, 0.842μm

l  Equation (9-10), the expressions are not right.

l  There is an error in the subtitle of 2.6 in Line 275

l  When discussing the impact of the size of training samples, the author can refer to the following references:

Li, C. ,  Wang, J. ,  Wang, L. ,  Hu, L. , &  Gong, P. . (2013). Comparison of classification algorithms and training sample sizes in urban land classification with landsat thematic mapper imagery. Remote Sensing, 6(2).

Gong, Peng, Han Liu, Meinan Zhang, Congcong Li, Jie Wang, Huabing Huang, Nicholas Clinton, et al. 2019. "Stable classification with limited sample: transferring a 30-m resolution sample set collected in 2015 to mapping 10-m resolution global land cover in 2017."  Science Bulletin 64 (6):370-3. doi: https://doi.org/10.1016/j.scib.2019.03.002.

 

Author Response

Thank you again for your time and attention in reviewing this work.  Your comments and suggestions greatly strengthened this manuscript.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments to the Author:

Feature selection is a critical procedure for machine learning algorithms. This manuscript illustrated the transferability of recursive feature elimination (RFE) method for supervised machine learning land-cover classification. Although the interest is good, the manuscript is still unacceptable because the contribution is insufficient for this journal. In addition, the content of the manuscript is too redundant. The comments include some major and minor revisions, which reflected my concerns.

Major revisions:

1.     This manuscript investigated the transferability of a feature selection for a machine learning algorithm. However, as well known, machine learning algorithms are highly data-driven methods. Commonly, the feature set selected by a feature selection method could be different for different remotely sensed data, which is consistent with the conclusion of this manuscript. This experiment didn’t provide any exciting results. Thus, the study has little contribution to the remote sensing field, and is not appropriate to be published in this journal.

2.     Another major concern is the organization of this manuscript. Many representations are redundant or not professional.

(1) The introduction section should be improved. The author didn’t introduce GEOBIA, but unexpectedly mentioned it in the objectives. So, it is better to introduce GEOBIA ahead than in line 73-77. The objectives should be claimed clearer.  It is better to use sub-objectives after the total goal.

(2)Line 72-73, the sentence should be deleted. Feature selection is not only conducted in GEOBIA.

(3) For section 2.2, please use “remotely sensed data and preprocessing”. Please delete section 2.2.1 since there is no section 2.2.2.

(4) For section 2.3, the experimental design should be shortened. The procedures are introduced in sections 2.4 to 2.9. So, section 2.3 should be shortened.

(5) The methodology part is too long. Many techniques are not new. So, it is unnecessary to repeat it in detail. For example, the equations of NDVI, textural features, equations (2) to (10) could be deleted. The introduction of SVM should be shortened.

(6)  For the two flowcharts (Figure 2 and 3), they are too blurry. It is not necessary to show the SVM classification for each data set. The workflow is the same for the three datasets.

(7) For section 2.3, the sampling method is not clear enough. “Statistical sampling” is not sufficient. The sampling techniques (e.g., simple random sampling, or stratified random sampling) should be introduced. Why did you partition the sample like this (2103 v.s. 897)? Why did you design a large and a small training sizes?

(6) The conclusion should be shortened. It is too long.

 3. For feature extraction, the author only used NDVI as vegetation indices. Did you consider other vegetation indices, like EVI (Enhanced Vegetation Index)? In addition, in table 2, it is better to use vegetation indices for NDVI.

 

Minor revisions:

1. Line 48-49, please revise the sentence.

2. For figure 1, please use “study area and Sentinel-2” directly, and move the band combination ahead of (a),(b), and (c). The explanation is too redundant.

3. Line 110, please specify the “temporal window”.

4. Line 123, the information is not necessary. Line 125 to 129, they are too redundant.

5. Why were the table names beneath the tables?

 

 

Author Response

Thank you again for your time and attention in reviewing this work.  Your comments and suggestions greatly strengthened this manuscript.

Author Response File: Author Response.docx

Reviewer 3 Report

This article touches on a set of interesting and important issues with image classification, notably: the need for feature reduction, training set size, and transferability of results. There is definitely merit in illustrating these issues, as many researchers seem unaware of some important points associated with them. But at the same time key conclusions simply restate what is well-known. For example, it is well known that SVMs are sensitive to the curse of dimensionality, that larger training sets offer many advantages (especially with SVM classifiers) and that results need not transfer. Many of the points raised indeed simply show what is obvious: classification results are data set specific.

 A further concern is that differences in accuracy are compared directly with no account for the variation in estimates arising from basic issues connected to sample size. This is critical to this paper as many of the differences in accuracy reported are very small. The comparisons of accuracy statements should ideally be based on rigorous statistical testing; in many instances, the differences reported would not be significant. The comparisons are also made more complicated by some poor presentation. For example, Figures 6 and 8 show box and whisker plots but the reader has no idea what information is being shown. Normally, for example, the mean and inter-quartile range is summarised by the x and the box. But what do the whiskers show? Is it maximum and minimum (cannot be this as an outlier shown in Fig 8) so is it a confidence interval? If so at what level of significance? This is important. Just looking at the data for Delaware in Fig 8, the 95% confidence interval for SVM-All would be approximately 80-85%, an interval that overlaps with the 4 other plots – there is no significant difference to discuss.

 

There are other issues that could gain from attention. The meaning of the word ‘model’ is unclear, Figures such as Figure 4 need fuller explanation and the ‘large’ training set is actually quite a small one given the classifier selected and number of features available (classical guidance would suggest that a training set of approximately 5,000 pixels would be required, so ‘large’ would be something >5,000 – note also it is the smallness of the ‘large’ sample that explains much of the results as the Hughes effect can still be expected to occur).

Overall – an interesting article that has value in illustrating the size of the effect some variables have on a classification. But all of the issues are actually very well known. If to be revised it is essential to increase the rigour of the analyses and assess the statistical significance of differences.

Author Response

Many thanks to the reviewer for your time and attention in reviewing this work.  Your comments and suggestions greatly strengthened this manuscript.

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

The authors have addressed my main concern. Critically they now provide statistical testing of differences in accuracy. The approach used is a little odd (they test for the differences in the mean accuracy) but does fit with the method used. The authors could enhance the details in the Appendicies - for each test result they should state the calculated t statistic value AND indicate if significant or not at the stated level of confidence (probably easiest to state the t value and simply use an * to indicate significance/). 

Author Response

Thank you again for your time and attention to this manuscript.  Very much appreciate the constructive feedback and suggestions by the reviewer, as I do believe they strengthened the work.

Author Response File: Author Response.docx