Landsat-8 Sea Ice Classification Using Deep Neural Networks

Round 1

Reviewer 1 Report

Decision: Major revision

1. Add some motivational sentences about the proposed algorithm in the abstract to avoid confusion for readers.
2. Introduction Section: The current challenges are not crystal clearly mentioned in the introduction section of this paper. I suggest adding a dedicated paragraph about the current challenges in this area followed by the authors’ contribution to overcoming those challenges.
3. The contributions of this research in the current manuscript are not clear. I strongly recommended improving and adding built-wise contributions in the manuscript. The authors can follow the introduction section for writing contributions to the article, for assistance: “doi: 10.3390/s21175892” and cite this regard behind the DL/CNN.
4. References are not enough, please add some of the references that you believe are suitable for your work.
5. In Section 3, the authors must introduce their proposed research framework more effectively. For example, the authors could consider some essential brief explanation compared to the text with a total research flowchart or framework diagram for each proposed algorithm to indicate how these employed models are working to receive the experimental results. It is not easy to understand how the proposed approaches work.
6. Explain the reasons that the suggested approach provides better performance compared to other previous models? I wonder if the proposed method can be applied to other regions with different systems and occupant profiles.
7. Comparison with SOTA is missing. I strongly recommend to add a dedicated table and comparing your method.
8. Add some visuals of the utilized dataset, the authors can follow the “Human action recognition using attention based LSTM network with dilated CNN features” and cite in this regard.
9. The limitations of the proposed system are missing, which should be included.
10. The readability and presentation of the study should be further improved. The paper suffers from language problems.

Author Response

Dear reviewer,
Thank you very much for your positive feedback as well as for your valuable recommendations. We have tried to address most of your points and we hope that our paper got some gain in quality.

1. Add some motivational sentences about the proposed algorithm in the abstract to avoid confusion for readers.
   Response: comment considered
2. Introduction Section: The current challenges are not crystal clearly mentioned in the introduction section of this paper. I suggest adding a dedicated paragraph about the current challenges in this area followed by the authors’ contribution to overcoming those challenges.
   Response: comment considered
3. The contributions of this research in the current manuscript are not clear. I strongly recommended improving and adding built-wise contributions in the manuscript. The authors can follow the introduction section for writing contributions to the article, for assistance: “doi: 10.3390/s21175892” and cite this regard behind the DL/CNN.
   Response: comment considered
4. References are not enough, please add some of the references that you believe are suitable for your work.
   Response: comment considered
5. In Section 3, the authors must introduce their proposed research framework more effectively. For example, the authors could consider some essential brief explanation compared to the text with a total research flowchart or framework diagram for each proposed algorithm to indicate how these employed models are working to receive the experimental results. It is not easy to understand how the proposed approaches work.
   Response: Updated version considered the comment.
6. Explain the reasons that the suggested approach provides better performance compared to other previous models? I wonder if the proposed method can be applied to other regions with different systems and occupant profiles.
   Response: For our solution we need to consider that the results should be accessible in near real time. That’s why we focused the study on DNN, but not CNN.
7. Comparison with SOTA is missing. I strongly recommend to add a dedicated table and comparing your method.
   Response: For now there is no established SOTA in this field of research as compared for object detection e.g.
8. Add some visuals of the utilized dataset, the authors can follow the “Human action recognition using attention based LSTM network with dilated CNN features” and cite in this regard.
   Response: We utilized data set are shown in the paper and can pe accessed by everyone for the USGS data portal for free. The proposed paper is quite interesting but not relevant for our paper.
9. The limitations of the proposed system are missing, which should be included.
   Response: Updated version considered the comment.
10. The readability and presentation of the study should be further improved. The paper suffers from language problems.

Response: Updated version considered the comment.

Thank you once again for your time and
kind regards

Egbert Schwarz

Reviewer 2 Report

The article describes the using of DNN for sea ice classification. The authors used Landsat-8 images and the BSH Ice charts. The approach is interesting despite of the fact that it has some limitations which were mentioned in the last sections. I think that the article can be published with some corrections.

 

 

1. Figure 4. Water mask as a separate figure is unreasonable.
2. Table 4, 2nd line. Misprint in the last column.
3. The authors use the TOA values, but the atmosphere affects image quality. I think it should be explained why the atmospheric correction was not applied.
4. Figure 6. There is no values and legend for the horizontal axis. And I think figures 6 and 7 can be combined.
5. Line 241. Incorrect reference.
6. The legend for figures 9-16 should be vertical, I mean each class on the separate row. Now it is difficult to read.
7. It is necessary to compare your results with the works of other scientists who study the sea ice using NN and optical images.

 

Author Response

Dear reviewer,

Thank you very much for your positive feedback as well as for your valuable recommendations. We have tried to address most of your points and we hope that our paper got some gain in quality.

1. Figure 4. Water mask as a separate figure is unreasonable.
   Response: We partly agree but as part of the workflow we would like to show the input data used.
2. Table 4, 2nd line. Misprint in the last column.
   Response:  Table is updated now.
3. The authors use the TOA values, but the atmosphere affects image quality. I think it should be explained why the atmospheric correction was not applied.
   Response:  With respect to the NRT functionality we would like to find a solution which is simple as possible. Considering this we could not access any ancillary data to performe a good atmospheric correction such as S6 or ATCOR. Image based atmospheric correction methods do not stable results from acquisition to acquisition. 
4. Figure 6. There is no values and legend for the horizontal axis. And I think figures 6 and 7 can be combined.
   Response: Figure legends are updated.
5. Line 241. Incorrect reference.
   Response: Reference corrected.
6. The legend for figures 9-16 should be vertical, I mean each class on the separate row. Now it is difficult to read.
   Response: Figure legends are updated.
7. It is necessary to compare your results with the works of other scientists who study the sea ice using NN and optical images.
   Response: We agree to this point but within the project life time we was not able to realise this but will consider this for the next project phase. Nevertheless till now we could not find a study which could be used to compare with our approach.
   
   Thank you once again for your time and kind regards
   Egbert Schwarz

Reviewer 3 Report

This paper introduces a method to use the deep neural network (DNN) tool to sea ice classification. They tested and assessed the results both qualitatively and quantitatively. The results showed
that the DNN model typically obtain promising classification results.  In the future work, the authors may compare other models such as convolutional neural network, XGBoost, random forest classifier...
In my opion, it's a good paper with the attached minor comments.

Minor:
Line 55:
"and applying" should be "and apply"

Line 193:
"categorial" should be "categorical"

Line 249:
"will produces" should be "will produce"

Line 284:
"which do not appear" should be "which does not appear"

Line  305:
"recommend to include " should be "recommend including"

Author Response

Dear reviewer,

Thank you very much for your positive feedback as well as for your valuable recommendations. We have tried to address most of your points and we hope that our paper got some gain in quality.

Thank you once again for your time and

kind regards,

Egbert Schwarz

Reviewer 4 Report

Review on “Landsat 8 Sea Ice Classification using Deep Neural Networks”, by Alvaro Cáceres, Egbert Schwarz and Wiebke Aldenhoff, submitted for publication in Remote Sensing.

General comments :

The study uses the German Federal Maritime and Hydrographic Agency (BSH) ice charts to train a Deep Neural Network (DNN) to recognize sea surface types from Landsat-8 imagery.

I am confused by the number of surface types. In the abstract, we are talking about 5 ice classes. In Table 1, we see a total of 7 surface types (6 ice + 1 ice free). In Figure 2, we see 5 surface types (4 ice + 1 open water). In Table 4, we see a total of 7 surface types (6 ice + open water). If some surface types are combined, please specify. Also, “ice free” and “open water” categories seems to be assumed to be the same in the paper, but in ice chart terminology, they refer to singly different surface types. Indeed, “ice free” is strictly zero ice concentration, whereas “open water” means there could be ice of concentration of less than 10%. If the same distinction is implied by the authors, please clarify in the paper. In Table 4, the ICESOD 81 and 84 are removed from the training set. That means only 5 categories remain (4 ice types + ice free). Are these the classes that are mentioned in the abstract? If that is the case, it should not be mentioned as “5 ice classes” in the abstract, since it is really 4 ice classes plus the “ice free” class.

Looking at figures 9 to 16, we often see obvious discrepancies between the BSH ice charts and the Landsat-8 image. I couldn’t find a single polygon with the ICESOD “1” (open water or ice free), while there are obvious large areas without ice in the image. Isn’t that a problem for the training and/or the validation and test results? That does not match the confusion matrix that is presented in figure 8. Please explain and reconcile your results. How could the model be 100% accurate over ice free (or open water) areas while the verification set (BSH ice charts) do not have this type?

On the other hand, the results of the classification (b panels in figures 9 to 16) seems to match pretty well the landsat-8 imagery, at least in terms of ice and no ice. Which make me think that the classification might be better than what the confusion matrix is indicating in figure 8. For sure, the classification model using Landsat-8 data greatly improves the resolution of features with respect to the BSH ice charts.

 

Specific comments :

1. The description of Table 1 should indicate that ice thickness ranges are between parentheses. Also, is the ice type corresponding to the thickest ice in the polygon, or to the dominant ice type in terms of partial ice concentration?
2. Line 90: “… ice charts with a standard deviation of one day…”. I would avoid using “standard deviation” in this context because it would mean that the difference between imagery and ice chart valid times could be more than one day. Is that the case? If not, reformulate and say the valid time difference is at most 1 day.
3. Line 101: “… low cloud cover …” Do you mean near surface cloud or almost clear sky?
4. Line 104: How is the false color image constructed? How the imagery triplet (green, red, NIR) matches the colors we see in Figure 1?
5. Figure 1: It seems to be a bad example of Landsat-8 coverage of the Baltic Sea, as most of the data is over land. Over water, what the black dots mean? I think the map is lacking of a colorscale.
6. Lines 128-130: I think this is the other way around: it is the reflected light that is affected by the surface type. Please rephrase.
7. Table 4 and 5: Please right-align the numbers in the column “Number of pixels” to improve readability.
8. Table 5: How the data split? Is the proportion of each ice types preserved in the training, validation, and test sets?
9. Line 184: The input and output of the DNN are clearly defined here. So the reader needs to read a long way into the paper to finally figure out that important detail. I suggest to specify that clearly earlier in the paper to avoid confusing the reader. See also the “general comments” section above.
10. Line 217: “The training time was 4 minutes and it was finished after 21 epochs.” Please add “and” between the 2 parts of the phrase, or rephrase.
11. Figure 6: Please change order in the caption: I think it should be “Loss vs epoch”. Please add labels on the horizontal axis. Also, for those not familiar with machine learning, I think it would be useful to have the mathematical expression of the “Loss”.
12. Figure 7: Please change order in the caption: I think it should be “Accuracy vs epoch”.
13. Line 237: Here “open water” is mentioned whereas Table 4 has “ice free”. Please be aware of the distinction between the two categories and avoid using them interchangeably.
14. Lines 276-277: It does not seem that this can be evidenced from the results shown in the paper. The classification results with the DNN show good agreement with the Landsat-8 imagery in terms of ice and no ice distinction, but the agreement with the BSH ice charts seems much lower. This is surprising since the BSH ice charts are the training data!
15. Line 278: Replace “;” with “,”.
16. Line 290: This is the first time that RGB is mentioned in the paper. It should have been introduced much earlier in the paper which of the Landsat-8 channel are associated with the 3 colors (Red Green Blue) to make the “false color” images.
17. Line 295: “… class 91, which is not part of the training data, is shown in the classification” Do you mean 81? Class 81 is excluded from the training set in Table 4, but class 91 is included. Please clarify what you mean.
18. Line 335: An improvement with respect to what reference?
19. Line 352: Please defined DLR as this is the first instance in the paper.

Author Response

Dear reviewer,

Thank you very much for your positive feedback as well as for your valuable recommendations. We have tried to address most of your points and we hope that our paper got some gain in quality. 

1. The description of Table 1 should indicate that ice thickness ranges are between parentheses. Also, is the ice type corresponding to the thickest ice in the polygon, or to the dominant ice type in terms of partial ice concentration?

Response: Table corrected.

1. Line 90: “… ice charts with a standard deviation of one day…”. I would avoid using “standard deviation” in this context because it would mean that the difference between imagery and ice chart valid times could be more than one day. Is that the case? If not, reformulate and say the valid time difference is at most 1 day.
   Response: Agree, sentence rephrased.
2. Line 101: “… low cloud cover …” Do you mean near surface cloud or almost clear sky?

Response: almost clear sky, clarified in the text

1. Line 104: How is the false color image constructed? How the imagery triplet (green, red, NIR) matches the colors we see in Figure 1?
   Response: clarified in the text
2. Figure 1: It seems to be a bad example of Landsat-8 coverage of the Baltic Sea, as most of the data is over land. Over water, what the black dots mean? I think the map is lacking of a colorscale.

Response: You are right, but we also would like to show the overlay with BSH ice chart. Black dots are open water

1. Lines 128-130: I think this is the other way around: it is the reflected light that is affected by the surface type. Please rephrase.
   Response: Agree, sentence rephrased.
2. Table 4 and 5: Please right-align the numbers in the column “Number of pixels” to improve readability.
   Response: corrected
3. Table 5: How the data split? Is the proportion of each ice types preserved in the training, validation, and test sets?
   Response: Data are split according the percentage, given in the table. The proportion is preserved.
4. Line 184: The input and output of the DNN are clearly defined here. So the reader needs to read a long way into the paper to finally figure out that important detail. I suggest to specify that clearly earlier in the paper to avoid confusing the reader. See also the “general comments” section above.

Response: Chapter updated.

1. Line 217: “The training time was 4 minutes and it was finished after 21 epochs.” Please add “and” between the 2 parts of the phrase, or rephrase.
   Response: corrected
2. Figure 6: Please change order in the caption: I think it should be “Loss vs epoch”. Please add labels on the horizontal axis. Also, for those not familiar with machine learning, I think it would be useful to have the mathematical expression of the “Loss”.
   Response: corrected, the single formula is not included otherwise there is a need increase the level of details about machine learning .which we wanted to avoid as this is rather a case study and not fundamental methodology discussion. Furthermore, this information is given in the references.
3. Figure 7: Please change order in the caption: I think it should be “Accuracy vs epoch”.
   Response: Corrected,
4. Line 237: Here “open water” is mentioned whereas Table 4 has “ice free”. Please be aware of the distinction between the two categories and avoid using them interchangeably.
   Response: Corrected
5. Lines 276-277: It does not seem that this can be evidenced from the results shown in the paper. The classification results with the DNN show good agreement with the Landsat-8 imagery in terms of ice and no ice distinction, but the agreement with the BSH ice charts seems much lower. This is surprising since the BSH ice charts are the training data!
   Response: This is clarified in the updated version. The discrepancy between BSH ice chart and Landsat image is because the BSH chart was produced based on another data set.
6. Line 278: Replace “;” with “,”.

Response: Corrected

1. Line 290: This is the first time that RGB is mentioned in the paper. It should have been introduced much earlier in the paper which of the Landsat-8 channel are associated with the 3 colors (Red Green Blue) to make the “false color” images.
   Response: Comment considered.
2. Line 295: “… class 91, which is not part of the training data, is shown in the classification” Do you mean 81? Class 81 is excluded from the training set in Table 4, but class 91 is included. Please clarify what you mean.
   Response: Class 81 is excluded from the training data set due to lack of training data. Clarified in the updated version.
3. Line 335: An improvement with respect to what reference?
   Response: Corrected

Line 352: Please defined DLR as this is the first instance in the paper.
Response: The information was given in the author affiliation

 

Thank you once again for your time and
kind regards

Egbert Schwarz

Round 2

Reviewer 1 Report

The authors successfully addressed my comments and suggestions. Good Luck!

 

 

Reviewer 4 Report

Review on “Landsat 8 Sea Ice Classification using Deep Neural Networks”, by Alvaro Cáceres, Egbert Schwarz and Wiebke Aldenhoff, submitted for publication in Remote Sensing.

General comments :

The study uses the German Federal Maritime and Hydrographic Agency (BSH) ice charts to train a Deep Neural Network (DNN) to recognise sea surface types from Landsat-8 imagery.

The authors addressed most of my concerns in report 1. I believe the manuscript could be published in the present form, after the authors consider the following remaining minor changes I suggest below.

 

Specific comments (line numbers are from the revised version of the manuscript) :

1. Line 21: Should be “Ice Free”.
2. Line 140: So the landsat-8 bands (Green, Red, NIR) match the (Red, Green, Blue) RGB colors in Figure 1 ? I think that could be said more clearly in the text because there are more than one way to display false color representation, although RGB is quite common. One question: why not matching red with red, and green with green (assuming RGB mapping) ?
3. Line 176: “gray” is used whereas “grey” appears in table 1. Please use the same throughout the text.
4. Figure 4: A space is missing between (a) and Landsat-8 in the caption.
5. Line 336: Should be “… results in an accuracy…”.