Precipitation Nowcasting with Orographic Enhanced Stacked Generalization: Improving Deep Learning Predictions on Extreme Events

Round 1

Reviewer 1 Report

General comments: the paper can be published with some scientific and language modifications

Specific comments

Firstly the authors used in all text "we" even in abstract. For increase the quality of the paper and to use a scientific language it is necessary to use an expression of the results in the impersonal way. the abstract must be modified according to the upper observation and also in this part it is necessary to describe the results obtained in the paper Quoting Ciach [19] is not necessary to put in the text in the introduction part. Usually it need an interpretation with some critical or approving observation of the authors.
"Our contribution is threefold"...at the end of the introduction part usually the authors mentioned the principals objectives of the research not contributions (these are described in the conclusion part).
in the methodology part the first formula used must have a citation the discussion and conclusion part must be modified. The discussion must have an clearly part after results and the conclusion must describe the results obtained and integrated to the other researches

Author Response

[1] Firstly the authors used in all text "we" even in abstract. For increase the quality of the paper and to use a scientific language it is necessary to use an expression of the results in the impersonal way. the abstract must be modified according to the upper observation and also in this part it is necessary to describe the results obtained in the paper.

We agree with the reviewer. The paper has been edited accordingly, with the aim of avoiding the first person and maintaining an objective, impersonal tone. The abstract has been rephrased, with a focus on the study objectives and results.

 

[2] Quoting Ciach [19] is not necessary to put in the text in the introduction part. Usually it need an interpretation with some critical or approving observation of the authors.

Thank you for the advice. The relevant section has been thoroughly edited with a comment dedicated to the specified reference. 

 

[3] "Our contribution is threefold"... at the end of the introduction part usually the authors mentioned the principals objectives of the research not contributions (these are described in the conclusion part).

We thank the reviewer for the indication. The introduction has been edited with a focus on the main research objectives of our research, while the contributions have been moved in the conclusion section.

 

[4] in the methodology part the first formula used must have a citation the discussion and conclusion part must be modified. The discussion must have an clearly part after results and the conclusion must describe the results obtained and integrated to the other researches

Thanks for noting this. A specific reference has been added before the first formula to clarify the attribution. A separate Discussion section is now available. Finally, the analysis to further enrich the analysis we have added to the results the skill scores of a Lagrangian persistence model, to compare the proposed solution with classic extrapolation methods (Section 2.6, results and discussion)

 

Reviewer 2 Report

General comments:

By combining the multiple deep learning models, this study tries to improve the skill of nowcasting on extreme events. Authos collect ~ 9 years data from one operational radar site, and divided the data for training, validation and test to examine the performance of the nowcasting. The topic is interesting and very important for meteorology and computer science, and it is worth to publish. However, some additional works need to be addressed and explained before the manuscript being accepted. The following are my comments and suggestions:

 

Major comments:

 

1. The writing skill can be improved. There are many redundant words, typos and gramma issues in different sections (e.g.: line38; lines44; line120; line232-> what is ablation analysis; line 256-> mode than double than?). Some sentences are not clearly enough.
2. Abstract can be improved: the abstract should be concise and point out what are the major tasks and found for the current study.
3. Introduction should be improved: the literature reviews only cited previous studies without pointing out what they have done or improved (lines 31-39). In addition, authors need to identify what is the main purpose of this study (reduce condition bias? capture orographic effect? Improving DL schemes or extrapolation system? ) clearly. The improvement is based on comparing to the existing DL models, or existing echo extrapolation system?
4. Please give more details of how to generate the 4 ensemble members.
5. Current study only shows the results and inter-comparison among DL schemes. Readers will be curious that the improvement compared to traditional echo extrapolation algorithm.
6. Section 4 must be improved: I only see the conclusions briefly without seeing any discussions.

 

 Minor comments:

 

1. The information of abbreviations is not clear. For instance: line recurrent à RNN? Line 66: ConvSG, the full name should be introduced in the first time of the manuscript. Line 111: what are L1 and L2.
2. Table 2, (d) should be figure instead of table.
3. Captions of many figures could be improved: Fig. put all the details and statements in the captions; Fig. 7: what is the meaning of X- and Y- axis? Fig. 9: indentify the sub-figures in the captions;
4. Line 268: NMSE (Fig. 10d) should be (Fig. 10b). In addition, there is no explanation of the result in Fig. 10c.

Author Response

[1] The writing skill can be improved. There are many redundant words, typos and gramma issues in different sections (e.g.: line38; lines44; line120; line232-> what is ablation analysis; line 256-> mode than double than?). Some sentences are not clearly enough.

We have generally reviewed the text and addressed errors and relevant sections. We clarified that the ablation analysis is meant to show the contribution of each introduced feature (Rainfall Ensemble, Stacked Generalization and Orographic Enhancement) to the final result. We thank the reviewer for the indication.

 

[2] Abstract can be improved: the abstract should be concise and point out what are the major tasks and found for the current study.

We edited the abstract to better convey the study objectives and results; in particular we underline the goal of improving the skills of the model in particular for extreme rainfall regimes.

 

[3] Introduction should be improved: the literature reviews only cited previous studies without pointing out what they have done or improved (lines 31-39). In addition, authors need to identify what is the main purpose of this study (reduce condition bias? capture orographic effect? Improving DL schemes or extrapolation system? ) clearly. The improvement is based on comparing to the existing DL models, or existing echo extrapolation system

We thank the reviewer for the comment. We have extended the introduction, providing more context for the background research papers and made the objective of the study more explicit.

 

[4] Please give more details of how to generate the 4 ensemble members.

To better explain the architectural details of the model used for the generation of the ensemble we added a new Figure (Fig. 3), outlining the schema of the deep learning architecture adopted by TrajGRU. We added clarifications at the end of section 2.3, describing the training configurations for the members of the ensemble. Further, we have improved with details the choice of the thresholds.

 

[5] Current study only shows the results and inter-comparison among DL schemes. Readers will be curious that the improvement compared to traditional echo extrapolation algorithm.

This is an important remark. We added the application of the S-PROG (Section 2.6) echo extrapolation method based on lagrangian persistence to provide a comparison with traditional methods as suggested. We also extended the introduction and discussion section to highlight the intrinsic differences between the skill of the deep learning solution and the echo extrapolation.

 

[6] Section 4 must be improved: I only see the conclusions briefly without seeing any discussions.

We added a separate discussion section and moved the conclusion in a different section. Both sections are now substantially expanded.

 

[7] The information of abbreviations is not clear. For instance: line recurrent à RNN? Line 66: ConvSG, the full name should be introduced in the first time of the manuscript. Line 111: what are L1 and L2.

We agree: we fixed all the abbreviations and simplified the terms L1 and L2 by using the more common definition of MAE and MSE. All main abbreviations are listed.  

 

[8] Table 2, (d) should be figure instead of table.

Thank you; corrected.

 

[9] Captions of many figures could be improved: Fig. put all the details and statements in the captions; Fig. 7: what is the meaning of X- and Y- axis? Fig. 9: identify the sub-figures in the captions;

Thank you; captions for all the figures and subfigures have been updated to improve the descriptions.

 

[10] Line 268: NMSE (Fig. 10d) should be (Fig. 10b). In addition, there is no explanation of the result in Fig. 10c.

Thank you: corrected. Results and Figures are commented, expanding the Results and Discussion sections.

 

Reviewer 3 Report

Please see the attached review report.

 

Kind regards

Comments for author File: Comments.pdf

Author Response

[1] Regarding the benchmarking of the final model (ConvSG(Ens + Oro) in Table 3), it would be interesting to compare it with a ConvSG(ensemble simple average + ORO). Simple averages frequently beat more complex combinations (see forecast combination puzzle, Stock and Watson 2004, Timmermann 2006), while it is widely known that combining forecasts decreases generalization error (Bates and Granger 1969) compared to single methods. From Table 3 one can deduce that the largest improvements may be attributed to the inclusion of orography features, rather than the stacking algorithm (differences between ConvSG (Single + Oro) and stacking are small), while in my experience I would expect that ConvSG(ensemble simple average + ORO) would beat ConvSG (Single + Oro) (which is similar to majority voting).

We thank the reviewer for the insight. To further enrich the analysis we have added to the results the skill scores of a Lagrangian persistence model, to compare the proposed solution with classic extrapolation methods (Section 2.6, results and discussion). Additional evaluation will require an extensive round of further experiments on the deep learning stack which is beyond our budgetary limits in the short schedule of the review; however we definitely keep this as a next step to explore. 

 

[2] Besides Wolpert (1992) some important papers on stacking are Breiman (1996) and Van der Laan (2007).

We thank the reviewer and have integrated the suggested references in the introduction.

 

[3] In my opinion, the biggest gains are due to modifying the loss function. However, the choice of thresholds for the loss function seems arbitrary and depends on the knowledge of the full dataset (which was known to the authors at the time of computations).

While we agree with the reviewer that the choice of thresholds is dependent on a certain degree of knowledge of the dataset, we only considered the overall distribution values when choosing the thresholds (Figure 6). This information can be derived empirically (e.g. by a random sampling of the training set), thus validating if constant over time also for future predictions. Moreover, we expect that the chosen thresholds can be reused “as is”, at least on other Alpine radars, and with minor modifications in other continental areas. Indeed, the thresholds are computed on the actual rainfall rate calculated after the conversion from reflectivity, where all variabilities given from the physical characteristics of the radar (Marshall-Palmer parameters), background noise and environmental factors have already been taken into account and corrected.

 

[4] Furthermore, while improvements should be expected, the manuscript does not solve the problem of prediction of extreme events. Of course, this problem is one of the main focuses of the machine learning research community, however I do not think that it can be solved based on current knowledge on loss functions (see Brehmer and Strokorb 2019).

Thank you for the extremely useful insight. We agree with the reviewer’s comment that the proposed approach shows improvement but modifying the loss function alone does not solve the problem of extreme events and indeed the sole reliance on the loss function can be erroneous. We have expanded the introduction section, included the suggested reference, and remarked in the conclusions that there is still a long way to go before solving the problem of extreme events prediction. 

 

[5] References 20 and 39 are identical.

While the two references look similar, they point to different URLs / DOIs since the dataset is split into two parts because of its size. The two references point to the first and second part containing the years 2010-2016 and 2017-2019 respectively.

Round 2

Reviewer 1 Report

The authors have improved their manuscript and made the modification required, and for that the paper can be published in this form.

Author Response

We thank the reviewer for the help.

Reviewer 2 Report

Authors have significantly improved the manuscript based on reviewers suggestions. I am satisfied with the current version of manuscript. I only have minor suggestions for the authors which they may make the manuscript even more complete: 

1. Author may want to mention that there are only 4-member of the ensemble which is very limited in the current study. What do you expect when increasing the size of ensemble? 
2. Since traditional echo extrapolation skill is mentioned in the introduction, and the results also included the comparison of S-PROG Lagrangian extrapolation system, I suggest including some recent references which try to improve echo extrapolation scheme in the introduction to provide more complete knowledge of nowcasting system for the redears. 

     Nerini, D., Foresti, L., Leuenberger, D., Robert, S., and Germann, U. 2019: A reduced-space ensemble Kalman filter approach for flow-dependent integration of radar extrapolation nowcasts and NWP precipitation ensembles, Mon. Weather Rev., 147, 987–1006. 

     Ryu, S., G. Lyu, Y. Do, and G. W. Lee, 2019: Improved rainfall nowcasting using Burgers’ equation. J. Hydrol.,124140, https://doi.org/10.1016/j.jhydrol.2019.124140,

     Chung, K.-S., I.-A. Yao., 2020: Improving radar echo Lagrangian extrapolation nowcasting by blending numerical model wind information: statistical performance of 16 typhoon cases. Mon. Wea. Rev., 148, https://doi.org/10.1175/MWR-D-19-0193.1

     

Author Response

• Author may want to mention that there are only 4-member of the ensemble which is very limited in the current study. What do you expect when increasing the size of ensemble? 

Thank you for the useful remark. The conclusion section was expanded with consideration of this aspect (LINES 350 - 355). Indeed, the four ensemble members in the presented work are considered to be the minimum working example, and we expect that an increase in the number of members, along with the further study of the threshold behaviors, will lead to further skill improvements of the stacked generalization on the extreme rain rates.

 

• Since traditional echo extrapolation skill is mentioned in the introduction, and the results also included the comparison of S-PROG Lagrangian extrapolation system, I suggest including some recent references which try to improve echo extrapolation scheme in the introduction to provide more complete knowledge of nowcasting system for the readers. 

Thank you for the useful references. We added the proposed references in the introduction (LINES 31 - 33).