Evaluation of Decision Fusions for Classifying Karst Wetland Vegetation Using One-Class and Multi-Class CNN Models with High-Resolution UAV Images

Round 1

Reviewer 1 Report

The authors have explored the classification performance of fusion models based on different CNN algorithms for karst wetland vegetation classification. The contributions to the literature, the detail level of the analysis, and the findings were satisfactory. However, some major and minor revisions are required for the publication. Detailed suggestions are as follows:

·       Some newly published references about wetland classification from remote sensing data using CNNs haven’t been quoted in the Introduction. With these studies, authors should expand on the literature review section and discuss the gaps in our understanding of this area.

·       The Introduction should include a clear explanation of why the study was necessary.

·       Some figure captions were written as "Fig" and some others were written as "Figure". Also, Figure 7 should be referred to in the text.

·       The theoretical background of the CNNs should be cited (Section 3.1.1).

·       Figure 4 and Figure 7 should include more details, such as Figure 5 and Figure 6. Filter sizes can be added to figures.

·       Training curves should be given to observe underfitting and overfitting.

·       I suggest that additional study areas be used to examine the classification performance of fusion models based on various CNN algorithms.

Author Response

Response to Reviewer 1 Comments

We thank the editors and reviewers for the valuable comments and suggestions. We have revised the paper substantially; and carefully made corrections according to the reviewers' comments and suggestions. Revised portions of manuscript are highlighted by red color. Our detailed point-by-point responses to the reviewers' comments are given below.

Point 1: Some newly published references about wetland classification from remote sensing data using CNNs haven’t been quoted in the Introduction. With these studies, authors should expand on the literature review section and discuss the gaps in our understanding of this area.

Response 1: Thank you for your comments. We have added two newly published literatures on wetland classification from remotely sensed data using CNN algorithms to the Introduction. The literatures are as following:

1. Fu, B.; He, X.; Yao, H.; Liang, Y.; Deng, T.; He, H.; Fan, D.; Lan, G.; He, W. Comparison of RFE-DL and Stacking Ensemble Learning Algorithms for Classifying Mangrove Species on UAV Multispectral Images. International Journal of Applied Earth Observation and Geoinformation 2022, 112, 102890, doi:10.1016/j.jag.2022.102890.
2. Pashaei, M.; Kamangir, H.; Starek, M.J.; Tissot, P. Review and Evaluation of Deep Learning Architectures for Efficient Land Cover Mapping with UAS Hyper-Spatial Imagery: A Case Study Over a Wetland. Remote Sensing 2020, 12, 959, doi:10.3390/rs12060959.

We rewrote the third and fourth paragraph in the Introduction section, expanded on the relevant literature review, and further discussed the gaps of current study on wetland vegetation classification. Please read the revised manuscript.

Point 2: The Introduction should include a clear explanation of why the study was necessary.

Response 2: We appreciate your careful comments and suggestions. We have revised the third, fourth, and fifth paragraphs of the Introduction section, and further provided a clear statement of the necessity for our study. Please read the revised manuscript.

Point 3: Some figure captions were written as "Fig" and some others were written as "Figure". Also, Figure 7 should be referred to in the text.

Response 3: Thank you for your comments. We have double-checked the manuscript and changed all "Fig" to "Figure". Also, we have corrected some errors, and referred Figure 7 in the text.

Point 4: The theoretical background of the CNNs should be cited (Section 2.3.1).

Response 4: We appreciate your careful comments and suggestions. We have added relevant references to all CNN algorithms in the Section 2.3.1. The details references are as followsing:

1. Badrinarayanan, V.; Kendall, A.; Cipolla, R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2017, 39, 2481–2495, doi:10.1109/tpami.2016.2644615.
2. Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid Scene Parsing Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, doi:10.1109/cvpr.2017.660.
3. Chen, L.-C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Computer Vision – ECCV 2018 2018, 833–851, doi:10.1007/978-3-030-01234-2_49.
4. Ni, Z.-L.; Bian, G.-B.; Zhou, X.-H.; Hou, Z.-G.; Xie, X.-L.; Wang, C.; Zhou, Y.-J.; Li, R.-Q.; Li, Z. RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments. Neural Information Processing 2019, 139–149, doi:10.1007/978-3-030-36711-4_13.

Point 5: Figure 4 and Figure 7 should include more details, such as Figure 5 and Figure 6. Filter sizes can be added to figures.

Response 5: Thank you for your comments and suggestions. We have added more detailed descriptions to Figures 4 and 7, and the revised figures are as follows:

Figure 4. Structure of the SegNet algorithm. Please see the attachment

Figure 7. Structure of the RAUNet algorithm. Please see the attachment

Point 6: Training curves should be given to observe underfitting and overfitting.

Response 6: We appreciate your careful comments and suggestions. We have plotted the training curves for all models, and add these figures in the appendix. The details are shown below:

Figure A1. Training and validation curves for multi-class classification. (a) training loss; (b) training accuracy; (c) validation loss; (d) validation accuracy. Please see the attachment

Figure A2. Training and validation curves for one-class classification of KH. (a) training loss; (b) training accuracy; (c) validation loss; (d) validation accuracy. Please see the attachment

Figure A3. Training and validation curves for one-class classification of KWP. (a) training loss; (b) training accuracy; (c) validation loss; (d) validation accuracy. Please see the attachment

Figure A4. Training and validation curves for one-class classification of EC. (a) training loss; (b) training accuracy; (c) validation loss; (d) validation accuracy. Please see the attachment

Figure A5. Training and validation curves for one-class classification of NN. (a) training loss; (b) training accuracy; (c) validation loss; (d) validation accuracy. Please see the attachment

Point 7: I suggest that additional study areas be used to examine the classification performance of fusion models based on various CNN algorithms.

Response 7: We appreciate your careful comments and suggestions. We additional selected a mangrove wetland to verify the applicability of the three decision fusion methods (Majority Voting Fusion (MVF), Average Probability Fusion (APF), and Optimal Selection Fusion, (OSF) proposed this paper. The mangrove wetland is located in Beibu Gulf, Guangxi, south China (Fig. 1), which was covered with Avicennia marina (AM), Kandelia candel (KC), and Aegiceras corniculatum (AC), and the edges of the area were also covered with Spartina alterniflora (SA). In addition, other land cover types in the area are Death mangrove (DM), Terrestrial vegetation (TV), Artifact (AF), Mudflat (MF), and Water Body (WB).

Fig. 1. Mangrove wetland of Beibu Gulf. Please see the attachment

We used DeepLabV3+ and HRNet algorithms with UAV multispectral imagery and digital surface models (DSM) to classify mangrove wetland community. We also used three decision fusion methods to fuse the multi-class classification models (MCC) of DeepLabV3+ and HRNet algorithms. We select overall classification accuracy (OA) and Kappa as the metrics for accuracy assessment. The specific classification accuracy showed in the Table 1.

Table 1. Classification accuracy of mangrove communities using DeepLabV3+ and HRNet algorithms

As seen in Table 1, Average Probability Fusion (APF) achieved the highest OA and Kappa of 92.93% and 91.68%, respectively, which provided better classification results than single DeepLabV3+ and HRNet. The other decision fusion methods (MVF and OSF) achieved similar classification accuracy with the single DeepLabV3+ and HRNet. These results demonstrated that decision fusions could improve the identification ability for mangrove vegetation compared to single CNN models.

Author Response File: Author Response.docx

Reviewer 2 Report

This manuscript is well presented the comparisons for different decision fusion strategies and different CNNs in different dataset. However, more could be done. 

Point 1：To be more clear for the abstract,three decision fusion methods including "Majority Voting Fusion(MVF), Average Probability Fusion(APF), and Optimal Selection Fusion(OSF)" can be introduced. 

Point 2：Some of citations are not cited with references. Please supplement the references in "SegNet in Line227 ， PSPNet in Line237 , DeeplabV3+ in Line 248, RAUNet in Line 258 , MVF in Line 314, APF in Line 326, OSF in Line338...".

Point3 : In Line 185 "the three different image datasets"  should be clearly stated in the text.

Point4:Why the quadrat size is set as 1m × 1m when resample the DOM and DSM to 0.1m spatial resolution? Some explaination should be added. 

Point5 : Have you considered about that the results could be reliable for other areas? I suggest to test for a different area. 

Author Response

Response to Reviewer 2 Comments

We thank the editors and reviewers for the valuable comments and suggestions. We have revised the paper substantially; and carefully made corrections according to the reviewers' comments and suggestions. Revised portions of manuscript are highlighted by red color. Our detailed point-by-point responses to the reviewers' comments are given below.

Point 1: To be more clear for the abstract, three decision fusion methods including "Majority Voting Fusion (MVF), Average Probability Fusion (APF), and Optimal Selection Fusion (OSF)" can be introduced.

Response 1: We appreciate your careful comments. We have added descriptions of the three decision fusion strategies in the Abstract section. Please read the revised manuscript.

Point 2：Some of citations are not cited with references. Please supplement the references in "SegNet in Line227, PSPNet in Line237, DeeplabV3+ in Line 248, RAUNet in Line 258, MVF in Line 314, APF in Line 326, OSF in Line338...".

Response 2: Thank you for your comments and suggestions. We have cited the relevant literatures in the Methods section. The specific literature is shown as follows.

1. Badrinarayanan, V.; Kendall, A.; Cipolla, R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2017, 39, 2481–2495, doi:10.1109/tpami.2016.2644615.
2. Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid Scene Parsing Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, doi:10.1109/cvpr.2017.660.
3. Chen, L.-C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Computer Vision – ECCV 2018 2018, 833–851, doi:10.1007/978-3-030-01234-2_49.
4. Ni, Z.-L.; Bian, G.-B.; Zhou, X.-H.; Hou, Z.-G.; Xie, X.-L.; Wang, C.; Zhou, Y.-J.; Li, R.-Q.; Li, Z. RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments. Neural Information Processing 2019, 139–149, doi:10.1007/978-3-030-36711-4_13.
5. Takruri, M.; Rashad, M.W.; Attia, H. Multi-Classifier Decision Fusion for Enhancing Melanoma Recognition Accuracy. 2016 5th International Conference on Electronic Devices, Systems and Applications (ICEDSA) 2016, doi:10.1109/icedsa.2016.7818536.

Point3: In Line 185 "the three different image datasets" should be clearly stated in the text.

Response 3: We appreciate your careful comments and suggestions. We have revised the sentence of “the three different image datasets ……” as following :

Finally, we combined DOM, DSM and TFs to obtain three image feature datasets (RGB, RGBS, and RGBST), and the detailed feature composition information is shown in Table 1.

Point4: Why the quadrat size is set as 1m × 1m when resample the DOM and DSM to 0.1m spatial resolution? Some explaination should be added. 

Response 4: Thank you for your comments and suggestions. The original spatial resolution of the UAV aerial images obtained in this paper is 0.05m-0.06m. In order to reduce the computational cost, we resampled the UAV image to 0.1m, which is still enough high spatial resolution for identifying each vegetation types.

In the field investigation, we built 1×1m sample plot in the area with the high spatial homogeneity of vegetation distribution, and record the geographical coordinates of the center point of each sample quadrat. These high quality of sample plots can help to create sematic labels for training CNN models.

Point5: Have you considered about that the results could be reliable for other areas? I suggest to test for a different area. 

Response 5: We appreciate your careful comments and suggestions. We additional selected a mangrove wetland to verify the applicability of the three decision fusion methods (Majority Voting Fusion (MVF), Average Probability Fusion (APF), and Optimal Selection Fusion, (OSF) proposed this paper. The mangrove wetland is located in Beibu Gulf, Guangxi, south China (Fig. 1), which was covered with Avicennia marina (AM), Kandelia candel (KC), and Aegiceras corniculatum (AC), and the edges of the area were also covered with Spartina alterniflora (SA). In addition, other land cover types in the area are Death mangrove (DM), Terrestrial vegetation (TV), Artifact (AF), Mudflat (MF), and Water Body (WB).

Fig. 1. Mangrove wetland of Beibu Gulf. Please see the attachment.

We used DeepLabV3+ and HRNet algorithms with UAV multispectral imagery and digital surface models (DSM) to classify mangrove wetland community. We also used three decision fusion methods to fuse the multi-class classification models (MCC) of DeepLabV3+ and HRNet algorithms. We select overall classification accuracy (OA) and Kappa as the metrics for accuracy assessment. The specific classification accuracy showed in the Table 1.

Table 1. Classification accuracy of mangrove communities using DeepLabV3+ and HRNet algorithms

As seen in Table 1, Average Probability Fusion (APF) achieved the highest OA and Kappa of 92.93% and 91.68%, respectively, which provided better classification results than single DeepLabV3+ and HRNet. The other decision fusion methods (MVF and OSF) achieved similar classification accuracy with the single DeepLabV3+ and HRNet. These results demonstrated that decision fusions could improve the identification ability for mangrove vegetation compared to single CNN models.

Author Response File: Author Response.docx

Reviewer 3 Report

This manuscript explores the application of deep learning in UAV image processing in remote sensing field. The manuscript uses different cnn models and different fusion algorithms to extract wetland vegetation. The effects of different combinations were studied and discussed. I think the manuscript needs to be improved in the following areas:

1. The title order of the article is wrong. Research methods and results are both the third title.

2. The introduction does not cite some newly published references on wetland classification from remote sensing data using CNN and model fusion methods. At the same time, there is a lack of introduction to why these four models and three model fusion methods are selected.

3. The article lacks the description of considering the balance of positive and negative samples. Please describe the distribution of positive and negative samples in detail.

4. The title of the picture is not uniformly named

5. The comparison between CNN model and traditional extraction methods can be added to show the advantages of deep learning

Author Response

Response to Reviewer 3 Comments

We thank the editors and reviewers for the valuable comments and suggestions. We have revised the paper substantially; and carefully made corrections according to the reviewers' comments and suggestions. Revised portions of manuscript are highlighted by red color. Our detailed point-by-point responses to the reviewers' comments are given below.

Point 1: The title order of the article is wrong. Research methods and results are both the third title.

Response 1: Thank you for your comments and suggestions. We have carefully checked and revised all titles of the article to ensure the accuracy of the rational title order.

Point 2: The introduction does not cite some newly published references on wetland classification from remote sensing data using CNN and model fusion methods. At the same time, there is a lack of introduction to why these four models and three model fusion methods are selected.

Response 2: We appreciate your careful comments and suggestions. We have added some references published in the last 5 years on wetland classification from remotely sensed data using CNN and model fusion methods in the Introduction section. The relevant references as following：

1. Fu, B.; He, X.; Yao, H.; Liang, Y.; Deng, T.; He, H.; Fan, D.; Lan, G.; He, W. Comparison of RFE-DL and Stacking Ensemble Learning Algorithms for Classifying Mangrove Species on UAV Multispectral Images. International Journal of Applied Earth Observation and Geoinformation 2022, 112, 102890, doi:10.1016/j.jag.2022.102890.
2. Pashaei, M.; Kamangir, H.; Starek, M.J.; Tissot, P. Review and Evaluation of Deep Learning Architectures for Efficient Land Cover Mapping with UAS Hyper-Spatial Imagery: A Case Study Over a Wetland. Remote Sensing 2020, 12, 959, doi:10.3390/rs12060959.
3. Hang, R.; Li, Z.; Ghamisi, P.; Hong, D.; Xia, G.; Liu, Q. Classification of Hyperspectral and LiDAR Data Using Coupled CNNs. IEEE Transactions on Geoscience and Remote Sensing 2020, 58, 4939–4950, doi:10.1109/tgrs.2020.2969024.
4. Zhang, C.; Sargent, I.; Pan, X.; Gardiner, A.; Hare, J.; Atkinson, P.M. VPRS-Based Regional Decision Fusion of CNN and MRF Classifications for Very Fine Resolution Remotely Sensed Images. IEEE Transactions on Geoscience and Remote Sensing 2018, 56, 4507–4521, doi:10.1109/tgrs.2018.2822783.
5. Hu, Y.; Zhang, J.; Ma, Y.; An, J.; Ren, G.; Li, X. Hyperspectral Coastal Wetland Classification Based on a Multiobject Convolutional Neural Network Model and Decision Fusion. IEEE Geoscience and Remote Sensing Letters 2019, 16, 1110–1114, doi:10.1109/lgrs.2018.2890421.
6. Meng, X.; Zhang, S.; Zang, S. Lake Wetland Classification Based on an SVM-CNN Composite Classifier and High-Resolution Images Using Wudalianchi as an Example. Journal of Coastal Research 2019, 93, 153, doi:10.2112/si93-022.1.

We rewrote the third, fourth and fifth paragraph in the Introduction section, and add the statements for explaining our choice of these four CNN algorithms and three decision fusion strategies. Please read the revised manuscript.

Point 3: The article lacks the description of considering the balance of positive and negative samples. Please describe the distribution of positive and negative samples in detail.

Response 3: Thank you for your comments and suggestions. For the image training dataset, the number of pixels for the four karst wetland vegetation species were 1,161,144,076 (Karst Herbages), 1,184,888,876 (Karst Woody Plants), 260,330,138 (Eichhornia Crassipes), and 140,572,566 (Nelumbo Nucifera). In contrast, the number of pixels for non-mangrove vegetation types was 2,495,944,344. For the weights of the loss function, the "balance" method of the scikit-image library was followed to solve for the weights of the different classes, as shown in the Table 1.

Table 1. Weight of each class.

Point 4: The title of the picture is not uniformly named

Response 4: We appreciate your careful comments and suggestions. We have carefully checked and revised all figure captions in the revised paper to ensure consistency of name.

Point 5: The comparison between CNN model and traditional extraction methods can be added to show the advantages of deep learning

Response 5: Thank you for your comments and suggestions. We have added the relevant comparison between CNN model and traditional extraction methods in the third paragraph of the Introduction section in the revised paper.

Author Response File: Author Response.docx

Reviewer 4 Report

The submission provides three decision fusion strategies and four convolutional neural networks (CNNs) for mapping wetland vegetations in Huixian Karst National Wetland Park, Guilin, south China. Moreover, in order to evaluated the effect of one-class and multi-class FCMs, the manuscript conducts a series of experiments and discussions. However, I have a number of major concerns with the manuscript, which are outlined below.

1-   In the introduction section, the authors present the advantages and disadvantages of different methods in OCC and MCC tasks, and also describes the effectiveness of fusion methods, but the reasons for choosing these 4 specific methods (SegNet, PSPNet, DeepLabV3+, and RAUNet) for detailed comparative analysis are ignored. It is necessary to summarize why these four CNN algorithms are selected in the proposed FCMs.

2-   In sections 2 and 3.1, the authors introduce the information of study area and dataset processing operations. In line 299, the paper says that: “The image data with a ratio of 3 was randomly cropped to 256 × 256 pixels and enhanced”. However, please provide the information whether there is overlap during the crop operation, and how much the overlap is? Meanwhile, the article only provides the amount of training set data. It is necessary to further present the number of pictures after cropping the entire picture to 256×256 pixels.

3-   In line 353, the paper describes that “the confusion matrix and five precision metrics (Precision, Recall, F1-score, Macro-average F1-score (Macro-F1), Intersection over Union (IoU), and Mean Intersection over Union (MIoU))” are used for evaluation. However, the indexes of Precision and Recall are ignored in experiments.

4-   In line 367, the paper describes that “while FP and FN denote the number of predicted vegetation types that do not match the actual vegetation types.” Please provide the detailed information of FP and FN.

5-   The technical route in Figure 3 needs to be explained in detail. Some modules are not described in the text.

Author Response

Response to Reviewer 4 Comments

We thank the editors and reviewers for the valuable comments and suggestions. We have revised the paper substantially; and carefully made corrections according to the reviewers' comments and suggestions. Revised portions of manuscript are highlighted by red color. Our detailed point-by-point responses to the reviewers' comments are given below.

Point 1: In the introduction section, the authors present the advantages and disadvantages of different methods in OCC and MCC tasks, and also describes the effectiveness of fusion methods, but the reasons for choosing these 4 specific methods (SegNet, PSPNet, DeepLabV3+, and RAUNet) for detailed comparative analysis are ignored. It is necessary to summarize why these four CNN algorithms are selected in the proposed FCMs.

Response 1: Thank you for your valuable comments. We rewrote the third, fourth and fifth paragraph in the Introduction section, and add the comparative analysis of the classification performance of four CNN algorithms (SegNet, PSPNet, DeepLabV3+, and RAUNet) in wetland vegetation mapping. Please read the revised paper.

Point 2: In sections 2 and 3.1, the authors introduce the information of study area and dataset processing operations. In line 299, the paper says that: “The image data with a ratio of 3 was randomly cropped to 256 × 256 pixels and enhanced”. However, please provide the information whether there is overlap during the crop operation, and how much the overlap is? Meanwhile, the article only provides the amount of training set data. It is necessary to further present the number of pictures after cropping the entire picture to 256×256 pixels.

Response 2: Thank you for your comments and suggestions. We used a sliding window crop with an overlap ratio of 0.5. The size of the cropped UAV images was 5248×5796, and 1800 images were obtained after cropping the windows of 256×256 size, and 100,800 images were obtained after data enhancement, and 100,000 of them were randomly selected as the training dataset. The data enhancement methods are rotation, flip and grid mask.

Point 3: In line 353, the paper describes that “the confusion matrix and five precision metrics (Precision, Recall, F1-score, Macro-average F1-score (Macro-F1), Intersection over Union (IoU), and Mean Intersection over Union (MIoU))” are used for evaluation. However, the indexes of Precision and Recall are ignored in experiments.

Response 3: We appreciate your careful comments and suggestions. The reason of adding Recall and Precision in the precision metrics is that the F1–score is calculated from the values of Recall and Precision. It was our oversight that we did not provide a clear description, so we have added some the description in section 2.3.4. of the revised paper to clarify the reasons for the addition of Recall and Precision.

Point 4: In line 367, the paper describes that “while FP and FN denote the number of predicted vegetation types that do not match the actual vegetation types.” Please provide the detailed information of FP and FN.

Response 4: Thank you for your comments and suggestions. We have added a more detailed description of FP and FN in the 2.3.4. section of the revised paper. Please read the revised paper.

Point 5: The technical route in Figure 3 needs to be explained in detail. Some modules are not described in the text.

Response 5: We appreciate your careful comments and suggestions. We have included a more detailed description of the technical route in the 2.3 section of the revised paper.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors have made all the corrections and explanations. It is appropriate for the study to be published in the journal as it is.

Reviewer 3 Report

This is an interesting study. It improves our understanding of the use of deep learning and fusion model strategies in the field of wetland vegetation classification. The research proves the effectiveness of the model fusion strategy. The text and diagram of the revised article are more standardized and accurate. After revision, this article has reached a publishing level. I hope everything goes well with you. Good luck