Validation of Cloud-Gap-Filled Snow Cover of MODIS Daily Cloud-Free Snow Cover Products on the Qinghai–Tibetan Plateau

Round 1

Reviewer 1 Report

General Comments

The manuscript evaluates the performance of three cloud-gap-filled snow cover products based on daily MODIS data over the Tibetan Plateau. The most interesting aspect of the study is the separate evaluation of the products in cloudy conditions. Generally, the evaluation of this type of products is averaged on all sky conditions masking the performance of cloud-gap-filling procedures.

All in all, the manuscript is well structured, but clarifications and additional information are required to improve it, particularly in the method section. As an example, it is not clear how metrics for sub-regional analyses were extracted as the areas of “altitude zones” and “10x10 km” for the sub-regional analyses (lines 200-202) do not fit multiple of the TM image (about 170 x 180 km) reported in the equations 9-16. Similarly, equations 1-8 are confusing and contain many misprints.

Moreover, a general text revision is required to improve the manuscript's readability.

Specific comments (both functional and formal) are listed below.

 

Specific comments

Abstract

Line 16: The latest “cloud-free snow cover” seems redundant; only “MODIS daily products” is more fluent.

Line 21: there is a refused parenthesis close to ref. 1

Lines 22-23: The FS average score for all altitudes should be weighted by the area percentage of the altitude ranges.

Lines 23-24: Please better explain this sentence since also FS for SC_Huang and SC_Qiu show clustered areas (see Fig. 4). Do you mean that SC_MOYD provides clusters with higher FS values or that SC_MOYD clusters can be more easily interpreted?

 

 

Introduction

Line 59: National Snow and Ice Data Center; as the first time, please add the link.

Lines 80-82: I think these two concepts have to be inverted being the uncertainty notion more general. The sparse distribution of meteorological stations can induce significant uncertainties in the validation of snow-cover products and does not support the identification of regional differences in accuracy.

 

Study Area and Data

Figure 1: The red color of the stations is not clearly identifiable on pink and orange background; a more contrasting color is preferable (e.g., black triangles).

Section 2.2.1: As there are many acronyms and other words in capital letters that make hard the reading of results and discussion, I suggest avoiding SC_ to define the products. Alternatively, the authors can adopt the following definitions: SC1 for SC_MOYD, SC2 for SC_Huang, and SC3 for SC_Qiu.

Lines 120-124: Please describe the basic indices as previously indicated NDSI for MOD10A1F and MYD10A1F products.

Lines 126-128: Even if the reference is reported, you need to add information about how cloud persistence is determined. Cloud presence represents basilar information for your study.

Line 131: How many TM scenes were used for the same place (the same path and row) ? (average information on QTP)

Lines 152-154: Please better explain the use of AMSR-E data. What does mean “to clarify the cloud removal method” ?

 

Methods

Section Among the accuracy metrics, the authors selected Cohen’s kappa. Such a coefficient is not a measure of accuracy but of agreement beyond chance. An accurate thematic map could be associated with a very wide range of kappa values. For a detailed explanation of the unsuitability of the kappa coefficient in the assessment and comparison of the map accuracy, see https://doi.org/10.1016/j.rse.2019.111630. Therefore, such a metric can be eliminated.

A practical alternative metric for thematic map accuracy is the Balanced Accuracy that accounts for the unbalance class distributions, such as the snow and non-snow classes. Furthermore, on an imbalanced dataset, also the overall accuracy can present drawbacks leading to false conclusions. Smaller class performance is not shaded by the proportional influence of large classes. For details and applications on Balanced Accuracy, see https://doi.org/10.1109/ICPR.2010.764 , https://doi.org/10.1080/20964471.2019.1625528 , https://doi.org/10.3390/rs14205127 , https://doi.org/10.1016/j.deveng.2018.100039

Equations 1-8: It is a fairly confusing representation of accuracy metrics that makes them difficult to read and prompts errors (for example, in formula 1, NS should be in place of N; in formula 2, SN is missing). It is better to insert a standard formula notation with generic terms (e.g.,  https://doi.org/10.1016/j.rse.2014.02.015 ) or true positive (TP) and true negative (TN) (e.g.,  https://doi.org/10.1109/ICPR.2010.764) by explaining the terms for the contextual application to snow-cover products.

Equations 9-16: The areas of “altitude zones” and “10x10 km” for the sub-regional analyses (lines 200-202) do not fit multiple of the TM image (about 170 x 180 km). Therefore, accuracy metrics cannot be expressed as the average of TM scenes. Do you trim the areas and re-compute the confusion matrices? Please explain how the authors obtain the aggregate metrics.

Moreover, it is not necessary to repeat all the formulas, it is enough to explain the calculation method and show at most one example.

 

 

Results

Table 2: To make the table self-explicative, please add all the acronyms in the caption. Clear and cloudy skys do not represent a snow cover type; please change the column name (e.g., Sky condition) and eliminate the words “under” and ”sky” in the rows.

Figure 3: Do the values of the accuracy metrics represent the average accuracy of the maps available over the period 2002-2010 per altitude?

Figure 4: Maps of F-score differences (SC_MOYD-SC_Huang, and  SC_MOYD-SC_Qiu ) can be added to better support the performance discussion. Alternatively, you can move the original mas in the supplementary material and add the difference map in the text. A similar approach can be adopted for Figg. 5 and 6.

Line 272: In this sentence, “The higher accuracy of” can be better than “The improvement in accuracy of“.

Line 274: Please detail that it refers to Figure 4a (Figure 4b also includes two regions with a red frame).

Figures 5 and 6: It is unclear why the no-data areas change between the precision and recall maps of the same algorithm. If the snow cover maps from TM and MODIS products are defined, the corresponding accuracy metrics should have the same areas with no-data. Please verify the presence of errors in the figures or in the data analyses.

Discussion

Figures 7 and 8: To better support the discussion jointly with the TM RGB, it is useful to add also the “true” snow cover map from TM. If you prefer, it can be overlapped on the snow cover maps from the MODIS products.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Accurate daily snow cover extent is a significant input for hydrological applications in the Qinghai-Tibetan Plateau. This study assessed the accuracy of cloud-gap-filled snow cover from three open accessible MODIS daily cloud-free snow cover products based on snow maps retrieved from Landsat TM images. However, I think the results of this study is not convinced for me. There are some wrong knowledge bases for this study. Optic remote sensing sensors can not obtain surface snow cover, both for Landsat and MODIS sensor. How do you evaluate the cloud-gap-filled snow cover under without snow depth measurements?

However, the author gave a series evaluation under clear and cloudy sky conditions. I think this manuscript need a major revision to improve the quality of your works. In fact, your evaluation is focusing on cloud-free condition, not all-sky conditions. I think the structure of this manuscript and the experiments in the paper need significant restructuring and revision.

 

Following, I give some minor comments:

Lines 18-20: Delete “(global Level-3 … … data set (MOD10A1))”, this additional introduction is not necessary.

Line 29: “the best” ---> “a good”

Line 38-39: Please give its full name, such as Yangtze River, Yellow River, Langcang River.

Line 46: delete “Therefore,”

Line 52: What is “cloud shutter”? Did you mean “cloud obscuration”

Line 54: “close” ---> “remove”

Line 64 and Line 62: please show the details of [38], Who? Not only list the number of references in here. Please check the entire text, DO NOT only use the number of references instead the specific study information!!!

Line 70: “[39] assessed MODIS10A1F”, do you know which study? Who did this work?

Line 159: “F-score” ---> “F1-score”

Line 160-161: “OA and CK indicate … non-snow cover” this description is not correct. Please revised.

Line 163: “FS show … omission errors”. F1-score is the balance of precision and recall. Please the definition and description of these five metrics.

Eq. 2, Please check this equation.

Line 161: Here, what’s your meaning of “non-snow”? cloud, data gaps (non-data) are removed or belong to “non-snow”?  As you said in line 132, the cloud cover of Landsat scene is less than 10%. Cloud could exist in Landsat snow cover map, and how do you process cloud pixels or invalid pixels?

 

Table 2: there is a big problem about the validation. As we know, optic remote sensing data can’t obtain surface snow cover information due to cloud obscuration, both for Landsat series sensors and MODIS sensors (Aqua and Terra). And this study did not used the ground-based snow depth measurements to take part in the evaluation experiments. How do you obtain the evaluation results under cloud conditions?

Then the author faced another question how to evaluate the gap-filled snow cover without ground snow depth measurements. Please give the reasonable explanations for your evaluation results for “cloud-gap-filled snow cover” (lines 218-232).

In addition, based on the results of Table 2, I have very strong doubts about the data processing of this study and the fundamental knowledge for snow cover analysis.

 

The remaining analysis results on cloud-gap-filled snow cover are unnecessary to be read and reviewed.

 

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This study assessed the accuracy of cloud-gap-filled snow cover from three open accessible MODIS daily cloud-free snow cover products based on snow maps retrieved from Landsat TM images. The authors made an important contribution to the cloud screening over bright surfaces and I suggest the publication of this paper. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The authors assessed the accuracy of MODIS snow cover products over the Tibet region. The scope and merit of the manuscript are adequate for the journal. I have some comments, particularly about the abstract. 

The abstract reads too technical. There are too many acronyms that are hard to follow, and I am against using a reference in the abstract. The authors also say like x% and y% higher than... but I think it should be x% point and y% point. Please check (same in the main text. You should say "FS of SC_MOYD increased by 29.3% point, from 53.2% to 82.5%" instead of "FS of SC_MOYD increased by 29.3%, from 53.2% to 82.5%"). I recommend that the authors make the abstract more accessible to non-experts. 

I also think the introduction has too many short paragraphs. For example, lines 72-91 can be merged as one paragraph. 

I wonder if figures 4-6 can be fewer classes, like by 10%, not 5%. Is there a reason to have 20 classes? 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

General Comments

The changes made to the manuscript clarified the fundamental points of the method and made the text more readable.

To make the manuscript publishable in RS, further minor modifications are required as indicated in the comments below.

 

Specific comments

Introduction

Lines 55-56: MYD10A1F product is derived from MODIS Aqua MYD10A1 (https://nsidc.org/data/myd10a1f/versions/61). Please, add this information jointly with the already reported MODIS Terra specifications.

Line 64: “combination of the MODIS and SSM/I”. Please explicit also the SSM/I acronym.

Line 74: it is useful to specify “…..cannot be evaluated in this manner as it is biased by data from clear sky conditions.”

Line 81-82: The sentence can be more readable in this form “In addition, the snow cover in the low-altitude regions is less than in the higher-altitude regions”.

 

Study Area and Data

Line 149: (no-data) instead of (non-data)

 

Results

All figures: Please specify in the caption of all the figures the sky condition for the statistics (cloudy or all-weather conditions).

Figure 3: Please verify that the values actually correspond to weighted values by area percentage. Moreover, address the misprint of the labels (SC1 is missing) and follow the algorithm order to improve the readability (SC1, SC2, and SC3).

Lines 288-289. You can add the reference to the original map also for SC1 “…. FS was evidently lower than SC1 (Figure S1c).”  Moreover, the caption of Figure S1 reports an incorrect association of the maps (see the corresponding comment below).

Lines 290-295: To make the text more flowing, it might be helpful to specify the frames directly by avoiding the nesting of parentheses. As an example, “The first region is located between the Qaidam Basin and Tanggula Mountains (red frame 1), …”

Thus, for the frames in Figure4b (and the corresponding in Figure S1b), you can change the number (e.g., frames 3 and 4) or change the color (e.g., blue frame 1).

Lines 304-305: Similarly, to the previous comment, you can refer to the frame without the nesting of parentheses. “One region is in the Qaidam Basin (red frame 1), and the….”.

Lines 313-319: As for comments on figure 4, the author can change the numbers or the color of the frames in Figure 6b and refer in the text to the corresponding frame without the nesting of parentheses.

 

Discussion

Lines 332-334: If you adopt the extended names for a specific study in the introduction, as suggested by reviewer 2, you have to adopt this solution also for the other part of the text. See also lines 343, 345, 346, 348,349. Please check throughout the manuscript.

Lines 380-382: Please eliminate the nested brackets, e.g., (Figure 7a).

Lines 405: (Figure 5 and 6) should be (frame 3 in Figures 6a,b).

Lines 424-428: Similarly to Figure 7, please eliminate the nested brackets in comments on Figure 8, e.g., (Figures 8e and 8f) or (Figures 8e,f).

Section 5.4: A proposal to improve the performance of SC1 during the accumulation period could be to consider both the last and subsequent cloud-free observation for the terporal filter. In addition, among the integration with other satellite data, the combination with the high spatial resolution of Sentinel-1 SAR seems promising for snow melting and accumulation periods (see, e.g. https://doi.org/10.1007/s11629-019-5723-1, https://doi.org/10.3390/rs11161904).

 

Supplementary material

Figures S1-S2-S3: By comparing the old manuscript version, the correct association of the maps should be (a) SC2, (b) SC3, and (c) SC1. In any case, to avoid confusion, it is better to insert the map title directly on the figures as well.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

I am so sorry that the previous comments make you confused on “Optic remote sensing sensors can not obtain surface snow cover, both for Landsat and MODIS sensor.”. This sentence should be “Under cloud cover, optic remote sensing sensors can not obtain surface snow cover, both for Landsat and MODIS sensor”.

 

The revised manuscript has significant been improved than previous version. Based on your updated explanations, I have understood how do you perform the experimental test. However, some minor revisions are required before this work can be published.

 

1) Table 2: Change “Sky condition” to “sky condition for MODIS data”

2) Please show the specific numbers of records used under clear and/or cloudy conditions in Section 4.1

3) Figs. 4-6 and the Figs. Ss-S3: Why is a piece of snow information apparently missing in south-eastern Tibet? The missing piece appears to be the size of one view image of Landsat. How do explain the data missing in this region?

The reference data from Huang et al. (2014) and Qiu et al. (2016), both of them have the entire data coverage in Tibetan Plateau.

Author Response

Please see the attachment.

Author Response File: Author Response.docx