Assessing Land Cover Classification Accuracy: Variations in Dataset Combinations and Deep Learning Models

Round 1

Reviewer 1 Report (New Reviewer)

Comments and Suggestions for Authors

This study explored the relationships between the adjustments of deep learning models and the accuracy of land cover classifications using two U-Net models and four distinct datasets. The authors claimed that datasets comprising spectral, textural, and terrain information achieved the highest accuracy. Additionally, a combination of loss functions is better than a single loss function for the deep learning process.

 

The study is well designed and the manuscript is well written. However, the improvement of the deep learning model performance is limited with various adjustments.

 

Table 4: What is the validation accuracy? Is it overall accuracy?

 

Table 4: Why are the training accuracy numbers of model A much larger than those in model B? Compared with model B, why there is a big difference between the training and validation accuracy numbers in model A?

 

Lines 267-269: “Model A achieved the highest training and validation accuracies when utilizing only the spectral information from Dataset A. However, when textural and slope information were added, the accuracy decreased.” Why is the accuracy lower when more information is added?

 

Lines 269-272: “For Model B, adding textural information to the spectral information in Dataset B decreased the training and validation accuracies. However, when slope information was added to Dataset C, and textural and slope information were included in Dataset D, the accuracies were the highest.” Your accuracy numbers for Model B are all within the range of 85.3-86.2%. In other words, the differences are within 1%. It is not meaningful to differentiate the impacts of the four datasets.

Author Response

Q. Table 4: What is the validation accuracy? Is it overall accuracy?

In Table 4, the accuracy is reported as Overall Accuracy. Training Accuracy is calculated by comparing the labeled images of the training data with the outcomes of the deep learning segmentation. Validation Accuracy is derived from the comparison between the labeled images of the validation data and the results of the deep learning segmentation.

 

Q. Table 4: Why are the training accuracy numbers of model A much larger than those in model B? Compared with model B, why there is a big difference between the training and validation accuracy numbers in model A?

In the training process of deep learning models, overfitting is suspected when the training accuracy is high but the validation accuracy is significantly low. Overfitting means a phenomenon where the model fits the training data so well, resulting in a decrease in generalization performance on new data. In this study, Model A shows signs of overfitting, which are discussed in Lines 251-258. In Model B, there are no significant differences between the training accuracy and the validation accuracy, so overfitting is not observed.

 

Q. Lines 267-269: “Model A achieved the highest training and validation accuracies when utilizing only the spectral information from Dataset A. However, when textural and slope information were added, the accuracy decreased.” Why is the accuracy lower when more information is added?

The highest accuracy observed in Model A when solely utilizing spectral information may be attributed to variations in training methodologies. Model B improved its generalization capability through a hybrid loss strategy that concurrently considers multiple types of losses and an enhanced AdamW that reduces weight decay and restrains overfitting. Consequently, Model B, which is equipped with a sophisticated algorithm designed to process more complicated and diverse information, achieves the highest accuracy when it utilizes a combination of spectral, textural, and slope data. Model A, in contrast, shows the opposite results.

These findings are further elaborated in Lines 270-278.

 

Q. Lines 269-272: “For Model B, adding textural information to the spectral information in Dataset B decreased the training and validation accuracies. However, when slope information was added to Dataset C, and textural and slope information were included in Dataset D, the accuracies were the highest.” Your accuracy numbers for Model B are all within the range of 85.3-86.2%. In other words, the differences are within 1%. It is not meaningful to differentiate the impacts of the four datasets.

Although the improvement of accuracy in Model B according to the current dataset settings is modest, approximately within 1%, it is noteworthy that overfitting does not occur despite the use of more information, and a means to marginally enhance the segmentation accuracy is identified. Future research will study the utilization of various types of information that make significant differences in accuracy improvement.

Author Response File: Author Response.docx

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

The manuscript "Assessing Land-Cover Classification Accuracy: Variations in Dataset Combinations and Deep Learning Models" presents a valuable contribution to the field of remote sensing and land-cover classification.

The manuscript lacks detailed methodological descriptions and could benefit from more thorough explanations of the parameter tuning and pre-processing steps.

A comparative analysis with other established methods would provide a better context for evaluating the proposed approach's effectiveness.

The manuscript acknowledges the problems of misclassification, particularly for categories such as grassland and other land types. Conducting a more in-depth error analysis to understand the causes of these misclassifications and suggesting potential solutions or model improvements would add significant value.

While the manuscript touches on the practical applications of the results, a more detailed discussion of how these results can be applied in real-world scenarios, such as urban planning, environmental monitoring, and policy-making, would enhance the practical relevance of the manuscript. Providing case studies or examples of successful applications would be beneficial.

Author Response

1) The manuscript acknowledges the problems of misclassification, particularly for categories such as grassland and other land types. Conducting a more in-depth error analysis to understand the causes of these misclassifications and suggesting potential solutions or model improvements would add significant value.

I have additionally described potential solutions for misclassification cases through comparison with previous research examples. (Line 350-362)

 

2) While the manuscript touches on the practical applications of the results, a more detailed discussion of how these results can be applied in real-world scenarios, such as urban planning, environmental monitoring, and policy-making, would enhance the practical relevance of the manuscript. Providing case studies or examples of successful applications would be beneficial.

Applications of the land cover map developed in the study are additionally described in Section 3.5. (Lines 366-388)

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

The manuscript has undergone a notable improvement since the previous revision. However, there are still a few minor areas that could be enhanced, such as the clarification of the acronym LULUCF in line 375. 

Author Response

Comments : The manuscript has undergone a notable improvement since the previous revision. However, there are still a few minor areas that could be enhanced, such as the clarification of the acronym LULUCF in line 375. 

Response : We have accurately addressed the comments and revised the relevant sections accordingly.(Line 370~372)

Author Response File: Author Response.docx

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This study evaluates land-cover classification accuracy through adjustments to the deep learning model (DLM) training process, including variations in loss function, learning rate scheduler, and optimizer, along with diverse input dataset compositions. The U-Net model served as the baseline, with models A and B configured by adjusting the training parameters. Through comparison, it was observed that Model B achieved higher classification accuracy.

However, the technique used in this study is lack of novelty. Meanwhile, the experimental data used in this study is insufficient to support the author's conclusions. The manuscripts not suitable for publication. Comments in detail are listed as follows:

1、The background section in the introduction is not strongly aligned with the main focus of this study. It is suggested to make appropriate revisions for better coherence.

2、It is suggested to consider including a location map for the selected study area and providing additional geographical details such as latitude and longitude in the text. This would facilitate readers in gaining a better understanding of the study area.

3、It is suggested to evaluate the consistency between the training samples provided by the environmental department and the land types in the study area. Additionally, please consider the possibility of extracting samples directly from the study area for a more applicable model in this specific region.

4、Could the author provide examples or references to previous studies that have utilized the U-Net model for land-cover classification, if any? A brief exploration of existing literature on this topic would be beneficial.

5、The DLM is trained iteratively by adjusting parameters and configuring the optimizer based on the U-Net model, which is considered relatively simplistic, and the innovation aspect lacks clarity. It is suggested to explore more advanced or innovative methods for model training and highlight the unique contributions of the approach.

6、The analysis of research results appears to lack depth, and the significance, value, application domains, and future directions of the study are not clearly articulated. It is recommended to provide a more in-depth exploration of the findings, explicitly state the research's importance and potential applications, and outline future research directions for a more comprehensive contribution to the field.

7、It is suggested to provide explanations for all the formulas in the text, clarifying the meanings of each parameter to enhance reader understanding.

8、The experimental procedures exhibit a certain level of completeness, but the description of the experimental process is relatively concise. It is recommended to provide more detailed information for clarity.

9、The acquisition process of datasets A, B, C, and D could be described in more detail for better understanding.

10、It would be beneficial for the conclusion to provide specifically the study area.

11、When analyzing the results of land-cover accuracy for Models A and B, it would be beneficial to include insights into the reasons for misclassification of land-cover types.

12、The overall length of the article is relatively short, and the content on experimental procedures, results, and analysis appears simplistic. The research process lacks complexity and innovation, and there is potential to enhance the article by adding more substantial content to enrich the study. 

Comments on the Quality of English Language

Minor editing of English language required

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

The study presented in the paper evaluates land-cover classification accuracy through adjustments to the deep learning model (DLM) training process, including variations in loss function, learning rate scheduler, and optimizer. The methodology as well as the results are clearly presented.

Some aspects could be improved for example in Line 130 the authors present in Figure 2 the training dataset according to land-cover categories. For me it is not clear why training dataset areas by land-cover categories used are so big if the size of the RapidEye imagery has a spatial resolution of 5 m. For example, why choose for the wetland the training areas with forest and water?  If not the case the text of the legend of figure 2 should be rethink

It will be positive to see the spectral confusion between the training areas chosen.

 

 The spatial resolution of the DTM created should be presented?

 

Author Response

Q. Some aspects could be improved for example in Line 130 the authors present in Figure 2 the training dataset according to land-cover categories. For me it is not clear why training dataset areas by land-cover categories used are so big if the size of the RapidEye imagery has a spatial resolution of 5 m. For example, why choose for the wetland the training areas with forest and water?  If not the case the text of the legend of figure 2 should be rethink
It will be positive to see the spectral confusion between the training areas chosen.
 The spatial resolution of the DTM created should be presented?

A. The figure of the study area has been added, and as a result, the example image of the dataset has been changed to Figure 3. To avoid any confusion, Figure 3 has been updated. The spatial resolution of the land cover map based on the DLM corresponds to the resolution of the satellite imagery, which is 5 meters.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Line number  mentioned in the<author_response.docx  >failed to be alighed with the part of the revised manuscript.  I strongly suggest that the authors append the revised content same as that of the revised manuscript in the < author_response.docx> . Meanwhile, the text color of the point-to-point response should be modified, such as blue color, red color and the like.  Addtionally, what remote sensing image are used when the authors build Figure1?  The authors should described it in the title of Figure1.

Comments on the Quality of English Language

The quality of engish language should be improved.