Development of a Comparison Framework for Evaluating Environmental Contours of Extreme Sea States

Round 1

Reviewer 1 Report

I'd like to thank the authors for providing reasonable responses to all my comments, and for the changes made to the text as a result of my comments. 

I recommend that the revised manuscript be accepted for publication.

Author Response

Thank you for your kind comments and for your consideration of our revised manuscript.

Reviewer 2 Report

Thank you very much for carefully considering my comments.

This revised paper addressed all my concerns appropriately. I find the paper very interesting and I believe that it pushes the question of how to evaluate different environmental contour methods, a very important current research topic. The paper is well-written, has high relevance and the results are novel. Of course one could discuss the pros and cons about your proposed metrics in depth, but I think that is a sign of the relevance of this work. I find the interpretation of the results is a bit difficult, because you used different contour construction methods for the IFORM, kernel density and empirical contours, but now you clearly describe how each contour is derived.

I have only one more minor comment.

Page 9, line 292: "This approach for construction empirical contours is akin to a HDC method, where the density level is unity and a binning approach is used to control convexity."

Isn't the density level 0 instead of unity (1)?

What does "a binning approach is used to control convexity" mean?

 

 

Author Response

Thank you for your kind comments and for your consideration of our revisions. We appreciate your interest in the paper and agree that there are important areas of further research and discussion that could follow this work.

We have made a minor revision in respond to your comment and to hopefully bring clarity. The sentence from page 9, line 292 has been broken into two and now states:

"This approach for constructing empirical contours is akin to an HDC method, where the contour is constructed to contain the full data density without overinflation. A binning approach is used to enforce this by allowing the contour to follow the data even in regions of concavity in which a convex contour representation would misrepresent the data density."

We have removed the discussion of the density level as we are not utilizing this density level in our construction, but instead assert that this approximation gives us a similar representation. Your correction on the density level is accurate. We added a better explanation of "controlling convexity" - we are ensuring that the contour is not always convex, which would make it overinflated in regions of convexity. We hope that this change resolves your question and comment.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

This paper considers the problem of prediction of environmental contour of extreme sea states. The authors try to make a comparative study of different algorithms. The paper includes 23 pages, 11 figures, and 39 references.

The topic has a big interest in the field, and I would appreciate the effort made by the authors. However,  the paper does not present any novelty. The methods used in this comparative study are not up to the task. Most methods are conventional as well as  Traditional I-FORM and  KDE.

In addition, 2 methods (Clayton and Gumbel) used in the comparision were not cited in the whole msnuscript. In page 5 and 6 (section 2.1.3), the authors  has cited two references, none of them present excpilicitly both methods.  Now, the question is,  which methods have been used in this study to define Clayton and Gumbel (proceeded in the comparison), Knowing that Reference [17] does not define any of those methods. 

besides concerning the other method, the authors compare with their own works  PCA/I-FORM. 

Many useless self references.

Since the topic is very interesting. I would like to see a strong improuvement of the paper by comparing the recent methods in the litterature. The methods should be clear, recent and refered in the results.

 

Other remarks: 

- List of abreviation should be made at the beggining of the article not at the end;

- Reference 31 is missing. It should be cited in the text, not in the footpage.

- Many references are not completed or do not respect the recomqnded format.

 

 

Reviewer 2 Report

Summary

This is an interesting paper, providing simple measures to estimate the quality of estimates of environmental contours.

I think the paper is worthy of publication after some corrections. I enjoyed reading the paper.

My main criticism of the paper is that it perhaps does not provide a sufficiently complete picture of current ideas around contours. I don't think it is reasonable or necessary to extend the calculations done, but I do think it's important to acknowledge (a) fundamental considerations and (b) the complexity of working with contours in general. A list of ideas to enhance the description is provided below.

The article is written in good English. Tables and Figures are good. References are in general OK, but a search of the most recent literature on contours would reveal some additional relevant material.

1) P2: A reference to the recent review paper by Ross et al (Ocean Engineering, v195, 2020) should be added. You might also reference the on-going contour comparison organised by Hasselsteiner (initiated in OMAE2019-96523, A Benchmarking Exercise on Estimating Extreme Environmental Conditions: Methodology and Baseline Results).

2) S2 (start): It feels a bit odd to dive straight into 2D contours here, and to talk immediately in terms for H_s and T_e. Perhaps a more general introductory paragraph might be added to S2, explaining contours in more general terms?

3) L112: I guess H_s and T_e need definitions

4) S2.1 It might be interesting here to point out (even before any analysis is done) some of the obvious pros and cons of the different approaches. For example, in S2.1.2, Haver-Winterstein makes assumptions regarding (a) the distribution of H_s, (b) the conditional distribution of T_e|H_s and critically (c) assumes that it is valid to extrapolate the parametric forms for mu and sigma beyond the sample. Therefore, any use of this approach must (at the very minimum) use appropriate diagnostics to assess whether these three assumptions appear to be valid for the sample under consideration. Similar comments can be made concerning all of the methods in S2. It would be useful for the reader to be made aware of these considerations.

5) L159: It's not completely obvious to me what C is. I guess you mean P(T_e<=t|H_s=h). If so, it would be useful to say this explicitly.

6) S2.1.3 Maybe the reader needs to be made aware that different copula methods in general make different assumptions about joint tail behaviour. This is incredibly important in the choice of appropriate copula to describe joint tails of H_s and T_E and other pairs of environmental variables. (You can really see this e.g. in your F5 with Clayton and Gumbel). A reference to Rodriguez (Measuring financial contagion: a copula approach, J. Empirical Finance v14 2007) might be useful here; this paper discusses the statistical properties of the copulas mentioned in the current work, and quantifies their tail dependence.

7) S2.2: Maybe point out that kernel methods assume the same kernel everywhere on the domain of interest. The optimal kernel width is determined by fitting to the sample (in some way). The optimal kernel width therefore does a good job of fitting the body of the data. It is not at all obvious however that the same kernel width will be good at describing the rarest events (or in other words for extrapolation to longer return periods). Indeed it's not obvious that the same kernel function should be used! Maybe you could adapt your fitting procedure to establish kernel widths which maximise predictive performance on the boundary of the sample (not in the body)? If you don't attempt this analysis, I think at least you should acknowledge that these problems exist.

8) S2: You probably need to talk about typical extreme value methods in S2 also (even if you don't include them in your comparisons). Appropriate choices of marginal distributions for H_s and T_e are important precursors in general to realistic contour estimation. For example, you might adopt (different) generalised Pareto distributions to describe marginal tails of both H_s and T_e, and then use an appropriate copula to describe the dependence structure. You should also probably reference Heffernan and Tawn

9) S3 (start): The case for the metrics proposed is clear and well made. However, the motivation for use of contours in structural design is often related to characterisation of structural failure. It would be useful if this was acknowledged, and reference to ISO or DNV design standards made.

10) S3.2: Maybe you could mention first transforming from H_s-T_e to H_s-Steepness as an alternative here; this seems natural when there is a known steepness boundary. See e.g. Mackay and Jonathan (OMAE2020-18308).

11) S3.3 I like this approach! You might reference bootstrapping here, since this provides even more general strategies for model assessment.

12) F5: The quality of figures throughout is excellent. Here, however, I'm a bit concerned about small font in places.

13) F5: The fits of the various contours are not that convincing. Perhaps it would be good to show two contours here, one corresponding to the period of the data (=5 years, used to fit the contour), and one (=10 years) corresponding to return period of interest.

14) F6: I think you should define your box-whisker plots. To what (quantiles, means, etc) to the various lines and boxes and "outliers" correspond?

15) General (probably in S6 I guess): It might be good to acknowledge that variables like H_s and T_e are non-stationary with respect to covariates (like season and direction). It therefore seems reasonable that models to describe the joint structure of H_s and T_e should also accommodate seasonality and directionality.

16) S5 and S6: The results and discussion here are reasonable but perhaps not surprising. My impression is that it's relatively difficult to draw strong conclusions from the work (which is not too surprising perhaps). Maybe S6 would benefit from a stronger summary: what have you learned from this work? What would you advise others regarding contour estimation based on your work? What are the practical difficulties you encountered?

 

 

 

 

 

 

 

Reviewer 3 Report

The paper entitled “Development of a comparison framework for evaluating environmental contours of extreme sea states” presents novel methods to evaluate the goodness of an environmental contour method and applies these methods to five contour methods to datasets of many sites that are located around the US coast. The environmental contour method is a widely used method to analyze marine structures and many variants of the method have been proposed in recent years. At the moment, there is no consensus among researchers how contour methods can be systematically evaluated. This paper proposes a framework for such evaluations and therefore touches a very important topic. While I like the general ideas of the paper, I am strongly concerned whether apples are compared with apples in the presented analysis.

The environmental contour method consists of two major steps: Modelling the joint distribution (for example with a global hierarchical model, a copula-based model or a KDE-based model) of the environmental variables and constructing the environmental contour (for example as an IFORM contour, an ISORM contour, a direct sampling contour or a highest density contour). This paper does not carefully differentiate between these two steps, which leaves important questions open to the reader.

It should be stated for each contour method:

1. What is the model structure that is used to describe the joint distribution
2. Which contour construction is used to calculate a contour based on the modelled joint distribution

While point 1 is described in the paper, point 2 is not well described. However, I suppose for “I-FORM, “PCA with updated I-FORM”, “Clayton Copula” and “Gumbel Copula” you constructed IFORM contours, I have no clue what kind of contours you constructed for the “empirical contour” and for the “Bivariate KDE”. Did you also construct IFORM contours? Or did you use another method such as ISORM, direct sampling or highest density?

While I do not know the answer, I would be concerned if different contour construction methods have been used, as the comparison would be somewhat between apples and oranges in that case. Some comparisons  between different contour construction methods are presented in the following three papers. They might be helpful to clarify the differences between different contour construction methods (as opposed to different models for the joint distribution) and might be useful to reference:

• Vanem, E., & Bitner-Gregersen, E. M. (2015). Alternative environmental contours for marine structural design - A comparison study. Journal of Offshore Mechanics and Arctic Engineering, 137, 51601–1 to 51601–51608. https://doi.org/10.1115/1.4031063
• Chai, W., & Leira, B. J. (2018). Environmental contours based on inverse SORM. Marine Structures, 60, 34–51. https://doi.org/10.1016/j.marstruc.2018.03.007
• Mackay, E., & Haselsteiner, A. F. (2021). Marginal and total exceedance probabilities of environmental contours. Marine Structures, 75. https://doi.org/10.1016/j.marstruc.2020.102863

It is especially important that you constructed the “empirical contours” using the same contour construction method that was used in the contour method that you compare it to.

I also have some smaller comments, which are given here in consecutive order:

1. Page 1, line 3-4: Please differentiate between the type of model that describes joint distribution and the method that is used to construct a contour based on a given joint distribution
2. Page, line 46: What about zero-crossing period?
3. Page 2 lines 54-55: I think the more important distinction between these two contour construction methods is not in the numerical methods that are used to calculate the contour (Monte Carlo versus the numerical methods that are typically used in IFORM) but that different regions in the variable space are used to define a contour
4. page 2 Lines 51-66: Please differentiate between the type of model that describes joint distribution and the method that is used to construct a contour based on a given joint distribution
5. Page 2 lines 74-75: Please give a reference that the most commonly contour method has a return period of 100 years or do not state this.
6. Page 3 lines 113 – 114: Please differentiate between the type of model that describes joint distribution and the method that is used to construct a contour based on a given joint distribution
7. Page 5 equations (2) and (3): These dependence functions are different from the dependence functions that most other authors used and that are recommended in DNV GL C205. Is there a reason you used these polynomials? Please state that this are different dependence functions that the ones that are usually used in this type of global hierarchical model.
8. Page 6 starting at line 172: Which method did you use to construct contours based on your estimated KDE joint model? The inverse first-order reliability method? Or as you stated earlier in the document that for KDE you did not use IFORM but constructed the contours directly: Did you construct highest density contour? Or direct sampling contours? Or some other type of contours in the original variable space?
9. Page 7 lines 231-233: You state that “the predictive accuracy metric is estimated by comparing the closeness with which the n-year contour captures n years of data”. This statement concerns me. An IFORM contour, of which you calculated many, should not capture n years of data. It should capture less than n years of data as an IFORM contour has many regions in the variable space that contain a probability of 1 / (return period * number of sea states per year). The publication “Marginal and total exceedance probabilities of environmental contours” (https://doi.org/10.1016/j.marstruc.2020.102863) analyzes the probabilistic properties of IFORM contours and might be helpful to clarify this point.
10. Page 8, Figure 2: How is the empirical contour calculated? Is it an IFORM contour based on the empirical joint distribution?
11. Page 9 line 268: I think the wave breaking limits for Hs-Te are estimates and not hard limits. Please add some language about his (whether you see them as hard limits or as estimates).