The Application of a Convolutional Neural Network for the Detection of Contrails in Satellite Imagery
Round 1
Reviewer 1 Report
The Application of a Convolutional Neural Network for the Detection of Contrails in Satellite Imagery:
· Add/Replace the name of the study area with the Keywords.
· The Introduction is too short and need to be expanded with relevant literatures.
· In the last paragraph of the Introduction, the authors should mention the weak point of former works (identification of the gaps) and describe the novelties of the current investigation to justify that the paper deserves to be published in this journal.
· “To a certain extent, non-cloud false contrail detections can be filtered out by using the ACHA product, however this does not remove all false contrail detection in cases where there are valid natural clouds that overlap false contrail detections.”. Explain.
· What is panel F in figure 4?! Add to the caption.
· Focus on the advantages/disadvantages of the proposed method concerning the obtained results.
· What are the strategies/recommendations to reduce uncertainties in this study?
· At the end of the manuscript, explain the implications and future works considering the outputs of the current study.
Acceptable.
Author Response
Reviewer notes in bold, author reply in italics
- Add/Replace the name of the study area with the Keywords.
The study area is neither a limiting nor a defining factor in the analysis. The GOES keyword does imply that study area is generally in the western hemisphere.
- The Introduction is too short and need to be expanded with relevant literatures.
There new sources have been referenced.
- In the last paragraph of the Introduction, the authors should mention the weak point of former works (identification of the gaps) and describe the novelties of the current investigation to justify that the paper deserves to be published in this journal.
A sentence has been added to clarify that this paper is not trying to present a novel detection method, but rather highlight how existing technologies in the form of an AI technique previously applied to sea ice lead detection can also be used for contrail detection. The primary novelty of the approach is the added value associating a cloud height with contrail detections.
- “To a certain extent, non-cloud false contrail detections can be filtered out by using the ACHA product, however this does not remove all false contrail detection in cases where there are valid natural clouds that overlap false contrail detections.”. Explain.
The text “Because ACHA height retrievals are limited to cloudy regions identified by the ABI Cloud Mask” has been added and this section has been edited for clarity.
- What is panel F in figure 4?! Add to the caption.
Figure caption has been updated for clarity.
- Focus on the advantages/disadvantages of the proposed method concerning the obtained results.
Text has been added: The primary advantage of the detection method is that it uses techniques previously demonstrated to be effective in satellite remote sensing applications [12, 13] and that it can be applied to this new application without changing the model archetecture. The technique also is also relatively easy to run in an operational sense because the detec-tions are based on a single brightness temperature difference image rather than more complex multi spectral time-series of images [18].
- What are the strategies/recommendations to reduce uncertainties in this study?
The draft has been updated to reflect a new detection model that was trained and validated against a newly available dataset with GOES imagery. The use of this new dataset significantly reduces the uncertainties of this study.
- At the end of the manuscript, explain the implications and future works considering the outputs of the current study.
The conclusion has been updated to reflect the premise of this study and a new sentence has been added: It will likely be possible to achieve better detection metrics with more advanced techniques, however there will likely be tradeoffs in terms developmental time, operation-al complexity and/or processing loads.
Reviewer 2 Report
1. In the abstract, you mentioned "longwave radiation scattered downward." Do you mean "emitted" instead of "scattered"?
2. I noticed that there have been several studies applying deep learning models to detect contrails in satellite images. For example, many papers have explored the climate impact of contrails while also utilizing machine learning for detection. It would be beneficial to include a more comprehensive discussion on these studies. Your existing discussion seems insufficient in terms of listing their accuracy or other relevant information. Additionally, it would be helpful to emphasize the innovation in your study.
3. The Materials and Methods section requires improvement. It appears that you used a manually identified contrails dataset as a training dataset, but your reference to the previous sea ice sheet method may confuse readers. Please clarify and streamline this section.
4. Although you have provided extensive details on preparing the training data, there is a lack of information on the specific architecture of your U-net model. This omission makes it difficult to evaluate your model properly.
5. Your paper showcases a good ROC curve and high scores on various indices. I would appreciate more insights into how you addressed the issue of overfitting in your model.
6. Another important consideration is imbalanced data. Given the nature of contrails detection, it is crucial to address the imbalanced data issue. However, I did not come across any mention of this in your study. Please provide an explanation of how you handled this challenge.
7. The contrails identified in Figures 3 and 4 appear wider than their actual appearance. Could you clarify whether the coloring of contrails was done manually or directly generated by the model?
8. In addition to the two examples presented in the manuscript, it would be beneficial to include some statistical data to provide a more comprehensive analysis.
9. It would be valuable to compare the accuracy of your method with other existing approaches.
I don't have any comments on the Quality of English Language.
Author Response
Reviewer notes in bold, author reply in italics
- In the abstract, you mentioned "longwave radiation scattered downward." Do you mean "emitted" instead of "scattered"?
To clarity of the point, the text has been changed to read: Contrails are found to have a net warming effect because the clouds prevent terrestrial (longwave) radiation from escaping the atmosphere. Globally, this warming effect is greater than the cooling effect the clouds have in the reduction of solar (shortwave) radiation reaching the surface during the daytime.
- I noticed that there have been several studies applying deep learning models to detect contrails in satellite images. For example, many papers have explored the climate impact of contrails while also utilizing machine learning for detection. It would be beneficial to include a more comprehensive discussion on these studies. Your existing discussion seems insufficient in terms of listing their accuracy or other relevant information. Additionally, it would be helpful to emphasize the innovation in your study.
Additional (3) references have been added.
A comparison of the AUC-PR scores has been added: 73.9 for this study vs 72.7 for [18].
The intention of this paper is to remain a brief demonstration of the application of a previously existing satellite based detection technology that can be applied in a new way. A climate scale study is beyond the scope of this study.
- The Materials and Methods section requires improvement. It appears that you used a manually identified contrails dataset as a training dataset, but your reference to the previous sea ice sheet method may confuse readers. Please clarify and streamline this section.
The draft has been updated to reflect a new detection model that was trained and validated against a newly available dataset with GOES imagery. The section has also been edited for clarity.
- Although you have provided extensive details on preparing the training data, there is a lack of information on the specific architecture of your U-net model. This omission makes it difficult to evaluate your model properly.
Because the model architecture did not change from a previously architecture, it was an intensions decision for the text to focus on the training of the model rather than the model architecture. The whole premise is that by only changing the training dataset and not the model architecture, an entirely new application can be achieved. Text has been added to the introduction to clarify this point.
- Your paper showcases a good ROC curve and high scores on various indices. I would appreciate more insights into how you addressed the issue of overfitting in your model.
A sentence has been added for clarity: To avoid overfitting, care was taken to avoid cross-contamination of the training, testing, and validation datasets.
- Another important consideration is imbalanced data. Given the nature of contrails detection, it is crucial to address the imbalanced data issue. However, I did not come across any mention of this in your study. Please provide an explanation of how you handled this challenge.
A sentence has been added: From the 10,000 cases, a ratio of 70%/20%/10% was used for training, testing, and validation.
- The contrails identified in Figures 3 and 4 appear wider than their actual appearance. Could you clarify whether the coloring of contrails was done manually or directly generated by the model?
The coloring is generated by the model.
Text has been added to expand on the discussion of contrail retrieval length and width:
One other point to make is that contrail detections are often longer and wider features than are detected in the cloud mask or cloud height products. This is because the cloud mask and height products detection techniques are based primarily on spectral test. In contrast, the U-Net used for contrail detection is able based on a combination of spatial and spectral characteristics. Features that have low patches of low spectral contrast may escape continuous detection in the cloud mask, but may be identified as a continuous contrail due to the spatial characteristics identified by the U-Net.
- In addition to the two examples presented in the manuscript, it would be beneficial to include some statistical data to provide a more comprehensive analysis.
The draft has been updated to reflect a new detection model that was trained and validated against a newly available dataset with GOES imagery. This includes updated statistical analysis in Table 1.
- It would be valuable to compare the accuracy of your method with other existing approaches.
A sentence has been added: Another way to assess skill is to measure the area under the curve (AUC-PR), and the results we achieved have a AUC-PR of 73.9, which compares favorably to the AUC-PR of 72.7 achieved by Ng et al. [18].
Reviewer 3 Report
The authors present an approach for the detection of contrails in satellite imagery using a convolutional neural network (CNN). The idea is interesting, but some important points must be considered carefully to improve paper quality. Also, some minor errors were found and are listed in sequence.
The authors mention reference [15] too suferficially. I understand the authors mention it was a work that was published very near the date this one was submitted, but it is important to consider and compare the proposed work with [15]. The main advantages and limitations between both must be clearly presented in order to show the novelty and contributions of this work.
According to reference [15], reference [8] also uses a U-Net architecture with ResNet-18 backbone. Since this work is already mentioned in the paper but no comparison is made, the authors must compare the differences and advantages of their proposed work to this already existing one. Such comparison is mandatory to show the novelty and how the proposed work advances the state of the art.
Other references should be added to the text, such as:
- Satellite-based detection of contrails using deep learning
- Atmospheric Contrail Detection with a Deep Learning Algorithm
- Contrail Recognition with Convolutional Neural Network and Contrail Parameterizations Evaluation
More general comments and minor errors are listed as follows.
"downward, typically" -> "downward typically"
"type of a CNN, a U-Net," -> "type of CNN, U-Net,"
"The U-Net can" -> "U-Net can"
"will be unable to enable" -> please rewrite
"Mannstein et al 1999 " -> "Mannstein et al."
" a U-Net [11]," -> "U-Net [11],"
"similar that that of" -> "similar to that of"
"by Hoffman et al 2019 and 2022 [12, 13]. " -> "by Hoffman et al. [12, 13]. "
"Ng et al" -> "Ng et al."
"GOES" -> please define the term on its first appearance in the text (it is definded only in page 5)
"Hoffman et al. 2021 [12] and updated in 2022 [13]." -> "Hoffman et al. in 2021 [12] and updated in 2022 [13]."
" a U-Net," - > " U-Net,"
"Ronneberger et al 2015 [11]." -> "Ronneberger et al. [11]."
"McCloskey et al" -> "McCloskey et al."
"McCloskey et al" -> "McCloskey et al."
"use a brightness" -> "use brightness"
"McCloskey et al" -> "McCloskey et al."
"Ng et al [15] " -> "Ng et al. [15] "
" however these " -> " however, these "
"McCloskey et al [16]" -> "McCloskey et al. [16]"
"positively the contrails" -> ?
" an examples where" -> " examples where"
" are can cause" -> "can cause"
"as contrail" -> "a contrail"
"However the single " -> "However, the single "
Author Response
Reviewer notes in bold, author reply in italics
- The authors mention reference [15] too suferficially. I understand the authors mention it was a work that was published very near the date this one was submitted, but it is important to consider and compare the proposed work with [15]. The main advantages and limitations between both must be clearly presented in order to show the novelty and contributions of this work.
The draft has been updated to reflect a new detection model that was trained and validated against a newly available dataset with GOES imagery. (The reference numbers have changed due to the addition of new references, what had been [15] is now [18])
- According to reference [15], reference [8] also uses a U-Net architecture with ResNet-18 backbone. Since this work is already mentioned in the paper but no comparison is made, the authors must compare the differences and advantages of their proposed work to this already existing one. Such comparison is mandatory to show the novelty and how the proposed work advances the state of the art.
A sentence has been added: Another way to assess skill is to measure the area under the curve (AUC-PR), and the results we achieved have a AUC-PR of 73.9, which compares favorably to the AUC-PR of 72.7 achieved by Ng et al. [18].
- Other references should be added to the text, such as:
- Satellite-based detection of contrails using deep learning
- Atmospheric Contrail Detection with a Deep Learning Algorithm
- Contrail Recognition with Convolutional Neural Network and Contrail Parameterizations Evaluation
References 15,16, & 17 have been added
More general comments and minor errors are listed as follows.
"downward, typically" -> "downward typically". Updated
"type of a CNN, a U-Net," -> "type of CNN, U-Net," Updated
"The U-Net can" -> "U-Net can" Updated
"will be unable to enable" -> please rewrite. Rewritten
"Mannstein et al 1999 " -> "Mannstein et al.". Here the year is relevant
" a U-Net [11]," -> "U-Net [11]," Updated
"similar that that of" -> "similar to that of" Updated
"by Hoffman et al 2019 and 2022 [12, 13]. " -> "by Hoffman et al. [12, 13]. " Here the year is relevant
"Ng et al" -> "Ng et al." Updated
"GOES" -> please define the term on its first appearance in the text (it is definded only in page 5) Updated
"Hoffman et al. 2021 [12] and updated in 2022 [13]." -> "Hoffman et al. in 2021 [12] and updated in 2022 [13]." Updated
" a U-Net," - > " U-Net," Updated
"Ronneberger et al 2015 [11]." -> "Ronneberger et al. [11]." Here the year is relevant
"McCloskey et al" -> "McCloskey et al." Updated
"McCloskey et al" -> "McCloskey et al." Updated
"use a brightness" -> "use brightness" Updated
"McCloskey et al" -> "McCloskey et al." Updated
"Ng et al [15] " -> "Ng et al. [15] " Updated
" however these " -> " however, these " Updated
"McCloskey et al [16]" -> "McCloskey et al. [16]" Updated
"positively the contrails" -> ? Changed to: positively identify the contrails
" an examples where" -> " examples where" Changed to: an example where
" are can cause" -> "can cause" Updated
"as contrail" -> "a contrail" Updated
"However the single " -> "However, the single " Updated
Round 2
Reviewer 1 Report
I appreciate the authors addressing the comments. The manuscript can be accepted in its current form. Congrats!
Reviewer 3 Report
Dear authors, thank you for addressing the aforementioned comments. I'm satisfied with the modifications performed and I believe the paper can be accepted now. Congratulations!
