Sub-Ensemble Coastal Flood Forecasting: A Case Study of Hurricane Sandy

Abstract

In this paper, it is proposed that coastal flood ensemble forecasts be partitioned into sub-ensemble forecasts using cluster analysis in order to produce representative statistics and to measure forecast uncertainty arising from the presence of clusters. After clustering the ensemble members, the ability to predict the cluster into which the observation will fall can be measured using a cluster skill score. Additional sub-ensemble and composite skill scores are proposed for assessing the forecast skill of a clustered ensemble forecast. A recently proposed method for statistically increasing the number of ensemble members is used to improve sub-ensemble probabilistic estimates. Through the application of the proposed methodology to Sandy coastal flood reforecasts, it is demonstrated that statistics computed using only ensemble members belonging to a specific cluster are more representative than those computed using all ensemble members simultaneously. A cluster skill-cluster uncertainty index relationship is identified, which is the cluster analog of the documented spread-skill relationship. Two sub-ensemble skill scores are shown to be positively correlated with cluster forecast skill, suggesting that skillfully forecasting the cluster into which the observation will fall is important to overall forecast skill. The identified relationships also suggest that the number of ensemble members within in each cluster can be used as guidance for assessing the potential for forecast error. The inevitable existence of ensemble member clusters in tidally dominated total water level prediction systems suggests that clustering is a necessary post-processing step for producing representative and skillful total water level forecasts.

1. Introduction

Flood forecasting is important for timely evacuations and the preparation of strategic plans to mitigate infrastructural impacts [1]. Guidance for flood preparation and infrastructural damage mitigation plans can be provided through ensemble forecasting. Though having a numerical weather prediction origin [2,3], ensemble forecasting is also useful in flood forecasting because it provides uncertainty information together with a central prediction [1]. The central prediction or ensemble mean represents an outcome that performs more skillfully than any randomly sampled forecast associated with an individual ensemble member, at least when averaged over a set of forecasts [4,5]. The ensemble mean is defined as the average of all ensemble members and filters the unpredictable aspects associated with each individual member, leaving the forecast features on which most ensemble members agree.

Forecast uncertainty is captured by the ensemble spread, which is often related to the skill of the ensemble mean or control forecast, the so-called spread-skill relationship [6]. Another reason that ensemble forecasting is useful is that it can provide estimates of forecast probabilities [7,8], though such estimates may not reflect the actual uncertainty because of sampling errors [4] and small ensemble sizes [8]. In flood forecasting applications, estimates of event uncertainty are often reported as exceedance probabilities defined as the fraction of ensemble members exceeding a specified threshold.

Some ensemble systems contain bifurcations or bimodalities, which can arise in weather forecasting systems if weather regime changes are forecast or if some ensemble members forecast rapid cyclogenesis while other members forecast weak cyclogenesis [5,9]. The existence of ensemble system bifurcations has important implications for measuring the quality of forecasting systems. For weather forecasting systems, the ensemble mean will only improve a forecast during a forecast period when there are no meteorological regime changes inducing ensemble system bifurcations [10]. As shown in a previous study [11], different ensemble scoring rules behave differently in the presence of bimodality, which can lead to divergent viewpoints regarding the quality of forecasts. For a symmetric bimodal forecast distribution in which the outcome is located on the median of the distribution, the Continuous Ranked Probability score (CRPS; [12]) would highly rate the forecast. On the other hand, the Ignorance score [13], which is based on the probability mass assigned to the observation, would severely penalize such a forecast. Thus, the CRPS score would indicate a quality forecast while the Ignorance score would indicate a poor forecast.

It is known that the expected forecast error for a forecast is related to the uncertainty of the forecast. However, for consistently bimodal forecasts, ensemble mean error will also be introduced because the ensemble mean will consistently fall in a region of vanishing probability, but the observation will more frequently fall on the modes of the distribution located away from the ensemble mean. Thus, one may wonder: is the ensemble mean error related to the ensemble spread or the bimodality of the forecast distribution? This question is important to address because scoring rules like the root mean square error (RMSE; [11]) and a recently introduced spread-error score [14] are functions of the ensemble mean error.

Ensemble forecasting verification measures such as RMSE are pointwise measures that cannot identify how well the overall shape of an observed object is forecast [15]. For example, pointwise measures are unable to determine if the overall shapes and sizes of precipitation fields are skillfully forecast. Thus, pointwise metrics can severely penalize forecast features with correct structures and shapes if the locations of the features are not accurately forecast [15]. The limitations of conventional verification metrics in numerical weather prediction have been remedied through the application of cluster analysis, a set of statistical methods used for identifying structures in data [16,17,18]. Using cluster analysis, one can decompose forecast errors into multiple components such as shape, size, intensity, pattern, and timing. Such a decomposition allows sources of error in a forecasting system to be better diagnosed. Though the application of cluster analysis has extensively focused on precipitation forecast verification [19], the application of cluster analysis is also suitable for forecast verification of other atmospheric features and for identifying ensemble member clusters resulting from the generation of ensemble members from different numerical models [20].

While previous work has focused on evaluating the quality of coastal flood forecasting systems [21,22,23], no studies have focused on identifying potential sources of errors arising from bifurcations or clusters in such systems. The overall objectives of this study are thus to show that bifurcations and clusters in coastal flood forecasting systems are ubiquitous and to demonstrate the impact of the clusters on the representativeness of forecast metrics to be provided to decision makers. To account for such clusters, we develop a straightforward clustering algorithm and introduce two metrics for measuring forecast uncertainty and skill in the case of multi-modal ensemble systems.

The paper is organized as follows. The data used is presented in Section 2 and mathematical background is presented in Section 3. Discussed in Section 4 is the generation of ensemble member clusters in coastal flood forecasting ensemble systems and the development of a clustering algorithm that can be applied to such forecasts. The practical implementations of the proposed methodologies are presented in Section 5 and concluding remarks are provided in Section 6.

2. Data

In this study, total water level Sandy reforecasts at 13 stations (Figure 1) produced from the New York Harbor Observing Prediction System hydrodynamic model (NYHOPS; [24]) were analyzed. The Sandy 72-h reforecasts were initialized on 27 October 2012. The retrospective forecasts were generated using meteorological forcing from 21 reforecast ensemble members of the National Oceanographic Atmospheric Administration’s Global Ensemble Forecast System [25]. More details of the NYHOPS model configuration can be found in [24].

3. Mathematical Background

3.1. Sub-Ensemble Forecasting Theory

Suppose that a one-dimensional ensemble forecast is the set

$$\mathcal{X}=\left\{X_1\left(t\right),X_2\left(t\right),\dots ,X_N\left(t\right)\right\}$$

(1)

consisting of N ensemble members generated by the numerical model $ℳ$ [26]. Let ~ be an equivalence relation on the set $\mathcal{X}$. Then, the equivalence relation partitions $\mathcal{X}$ into a family of M distinct equivalence classes ${\mathcal{C}}_1, {\mathcal{C}}_2, \dots , {\mathcal{C}}_M$ such that

$$\mathcal{X}={\mathcal{C}}_1{{\displaystyle \cup }}^{}{\mathcal{C}}_2{{\displaystyle \cup }}^{}\dots {{\displaystyle \cup }}^{}{\mathcal{C}}_M$$

(2)

and

$${\mathcal{C}}_i{{\displaystyle \cap }}^{}{\mathcal{C}}_j=\varnothing$$

(3)

for $i\ne j$ [27]. In the context of data analysis, the ${\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_M$ are often referred to as clusters and such nomenclature is adopted in this paper for brevity. Assuming that the ensemble members composing $\mathcal{X}$ represent all possible outcomes for the observation (denoted by $X{\left(t\right)}^{*}$), $X{\left(t\right)}^{*}$ will fall into one of the M clusters. Because any two clusters are pairwise disjoint (Equation (3)), $X{\left(t\right)}^{*}$ can only fall into a single cluster. Probabilistically, $\mathcal{X}$ is the sample space comprising all possible outcomes and the clusters ${\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_M$ are probabilistic events embedded in $\mathcal{X}$. An event ${\mathcal{C}}_i$ will be said to occur if $X{\left(t\right)}^{*}$ becomes a member of ${\mathcal{C}}_i$. Thus, the occurrence of ${\mathcal{C}}_i$ renders ${\mathcal{C}}_i$ the verifying cluster ${\mathcal{C}}^{*}$. For an ensemble forecast containing clusters, there will be cluster uncertainty or the uncertainty about what cluster will become the verifying cluster (Section 4.1).

In the presence of clusters, one goal of an ensemble forecast is to estimate the probability that ${\mathcal{C}}_i$ will occur. Denote by $\mathit{Pr}\left(\varsigma \right)$ the estimated probability that the event $\varsigma$ will occur and assume that every ensemble member is equally likely, then the cluster probability is

$$\mathit{Pr}\left({\mathcal{C}}_i\right)=\frac{N_i}{N},$$

(4)

where $N_i$ is the number of ensemble members composing ${\mathcal{C}}_i$. Furthermore,

$$Pr\left({\mathcal{C}}_i{{\displaystyle \cap }}^{}{\mathcal{C}}_j\right)=0$$

(5)

for $i\ne j$ because $X{\left(t\right)}^{*}$ cannot belong to more than one cluster. Equation (2) implies that the events ${\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_M$ are exhaustive, i.e.,

$$Pr\left(\mathcal{X}\right)=\mathit{Pr}\left({\mathcal{C}}_1\right)+\mathit{Pr}\left({\mathcal{C}}_2\right)+\dots +\mathit{Pr}\left({\mathcal{C}}_M\right)=1.$$

(6)

Mutual exclusivity of the ${\mathcal{C}}_1,{\mathcal{C}}_2,\dots , {\mathcal{C}}_M$ makes it natural to define a sub-ensemble forecast. More precisely, the i-th sub-ensemble forecast embedded in an ensemble forecast is the subset ${\mathcal{C}}_i$. Any statistical quantity computed using only ensemble members belonging to ${\mathcal{C}}_i$ will be referred to as sub-ensemble means, medians, spread, etc.

3.2. Sub-Ensemble Mean

Associated with each ${\mathcal{C}}_i$ is a representative forecast trajectory, which will be termed a sub-ensemble mean. The sub-ensemble mean is given by

$${\overline{X}}^i\left(t\right)=\frac{\left(X_1^i\left(t\right)+X_2^i\left(t\right)+\dots +X_{N_i}^i\left(t\right)\right)}{N_i},$$

(7)

where the $X_1^i\left(t\right),X_2^i\left(t\right),\dots ,X_{N_i}^i\left(t\right)$ denote the $N_i$ ensemble members belonging to ${\mathcal{C}}_i$. The traditional ensemble mean is given by

$$\begin{gathered} \overline{X\left(t\right)}=\frac{\left(X_1\left(t\right)+X_2\left(t\right)+\dots +X_N\left(t\right)\right)}{N} \\ =\frac{N_1\left(X_1^1\left(t\right)+X_2^1\left(t\right)+\dots +X_{N_1}^1\left(t\right)\right)}{N_{1}N}+\frac{N_2\left(X_1^2\left(t\right)+X_2^2\left(t\right)+\dots +X_{N_2}^2\left(t\right)\right)}{N_{2}N}+\dots \\ +\frac{N_M\left(X_1^M\left(t\right)+X_2^M\left(t\right)+\dots +X_{N_M}^M\left(t\right)\right)}{N_{M}N} \\ =\mathit{Pr}\left({\mathcal{C}}_1\right){\overline{X}}^1\left(t\right)+\mathit{Pr}\left({\mathcal{C}}_2\right){\overline{X}}^2\left(t\right)+\dots +\mathit{Pr}\left({\mathcal{C}}_M\right){\overline{X}}^M\left(t\right) \end{gathered}$$

(8)

While ${\overline{X}}^i\left(t\right)$ is a representative scenario for ${\mathcal{C}}_i$, $\overline{X\left(t\right)}$ may not be a representative scenario of the ensemble system as a whole because it may not even fall into a cluster as a result of being a weighted average of sub-ensemble means.

To see how $\overline{X\left(t\right)}$ can be unrepresentative of the ensemble system, consider the ensemble forecast shown in Figure 2 consisting of N = 40 ensemble members. The r-th ensemble member is

$$X_r\left(t\right)=A_r{e}^{{\frac{-\left(t-{\sigma }_r\right)}{100}}^2}+\frac{{\epsilon }_r}{80}$$

(9)

where ${\sigma }_r$ = 25 if $1\le r\le 20$ and ${\sigma }_r$ = 75 if $21\le r\le 40$. The quantities $A_1,A_2,\dots , A_{40}$ with $A_r=A_{r+20}$ are random samples from an approximately normal distribution with mean 2.0 and standard deviation 0.4. The quantity ${\epsilon }_r$ is defined such that it equals r if $r\le 20$ and equals r − 20 if $r\ge 21$. In this case, the $X_1\left(t\right),X_2\left(t\right),\dots ,X_{20}\left(t\right)$ peak at $t_1=25$ and the $X_{21}\left(t\right),X_{22}\left(t\right),\dots ,X_{40}\left(t\right)$ peak at $t_2=75$ (Figure 2).

Now suppose in this case that any two ensemble members are equivalent if they peak at the same time. Then, the equivalence relation partitions the set of all ensemble members into two clusters, i.e.,

$${\mathcal{C}}_1=\left\{X_1\left(t\right),X_2\left(t\right),\dots ,X_{20}\left(t\right)\right\}$$

(10)

and

$${\mathcal{C}}_2=\left\{X_{21}\left(t\right),X_{22}\left(t\right),\dots ,X_{40}\left(t\right)\right\},$$

(11)

where $\mathit{Pr}\left({\mathcal{C}}_1\right)=\mathit{Pr}\left({\mathcal{C}}_2\right)=0.5$. As shown in Figure 2a, the sub-ensemble mean associated with ${\mathcal{C}}_1$ (thick red curve) has a single local maximum. Similarly, the sub-ensemble mean associated with ${\mathcal{C}}_2$ (thick blue curve) also has a single local maximum. On the other hand, $\overline{X\left(t\right)}$ has two local maxima, rendering its probability of occurrence zero, as it is not a member of any cluster.

3.3. Probability of Exceedance

While the ensemble mean provides an average value at each point in time, it does not make full use of the information provided by the ensemble members. Thus, another approach in ensemble forecasting is to compute the global probability of exceedance defined as

$$g_E=\frac{N_E}{N},$$

(12)

where $N_E$ is the number of ensemble members whose values exceed a threshold $T_f$ at least once in the forecast period. This measure provides information about the probability of a flood occurring but does not indicate when the flood is likely to occur.

To determine how likely a flood will occur at time t, one needs to compute the local flood exceedance probability given by

$$g_E^t=\frac{N_E^t}{N},$$

(13)

where $N_E^t$ is the number of ensemble members whose values at time t exceed $T_f$.

In the presence of clusters, however, the curve $g_E^t$ as a whole may not be representative of the information provided by the ensemble members. Consider the situation shown in Figure 2a in which all the ensemble members exceed $T_f$ = 1.0 at least once ($N=N_E=40$). Therefore, in this situation, $g_E$ indicates that there is a 100% chance of the observation exceeding $T_f=1.0$. Now suppose one computed $g_E^t$. Then, according to Equation (13), there is a 50% chance of $T_f$ being exceeded at $t_1=25$ because $N_E^{t_1}=20$. Similarly, there is also a 50% chance of $T_f$ being exceeded at $t_2=75$. Thus, despite a 100% global exceedance probability, another forecaster using local exceedance probabilities would infer that $T_f$ would be exceeded with at most 50% probability.

Now suppose that one instead calculated the conditional (sub-ensemble) exceedance probability associated with the i-th cluster given by

$$g_i^t=\frac{N_i^t}{N_i},$$

(14)

where $N_i^t$ is the number of the $N_i$ ensemble members in ${\mathcal{C}}_i$ whose values at t exceed the threshold $T_f=1.0$. The curves $g_i^t$ are collectively representative of the ensemble system as shown in Figure 2b. The curve $g_1^t$ depicts how the observation will have a 100% chance of exceeding $T_f$ = 1.0 if it peaks at $t_1=25$, but a 0% chance of exceeding 1.0 at $t_2=75$ if it peaks at $t_1$. The depiction provided by $g_2^t$ is similar, but the maximum in the exceedance probability is at $t_2$. The curve $g_E^t$ indicates that the observation could exceed 1.0 at $t_1$ and then again at $t_2$ even though not a single ensemble member indicates such an outcome. Therefore, this example demonstrates how a single probability of exceedance curve as a whole cannot represent the information provided by the ensemble members in the presence of clusters. However, locally, $g_E^t$ is representative because there is, indeed, a 50% chance of $T_f$ being exceeding at $t_1$, for example. Thus, traditional probability of exceedance is useful when one is interested in assessing how likely a flood will occur at a specific moment in time without regard to what could occur at other points in the forecast period. On the other hand, sub-ensemble exceedance probabilities are better suited for situations in which one is interested in how many times a flood could occur in the forecast period. One can think of $g_E^t$ as smoothing out probabilistic information associated with each cluster. In fact,

$$g_E^t={\displaystyle \sum }_{i=1}^M\frac{N_i{N}_i^t}{N_{i}N}=Pr\left({\mathcal{C}}_1\right)g_1^t+\mathit{Pr}\left({\mathcal{C}}_2\right)g_2^t+\dots +\mathit{Pr}\left({\mathcal{C}}_M\right)g_M^t$$

(15)

because $N_E^t=N_1^t+N_2^t+\dots +N_M^t.$ Equation (15) is the probabilistic analog of Equation (8).

4. Cluster Analysis Metrics

4.1. The Cluster Uncertainty Index

Although the unrepresentativeness of the traditional ensemble statistics can be depicted graphically, it is useful to quantify the representativeness so that differences among a large set of forecasts can be more easily compared. In addition, because ensemble member clusters are based on characteristics of the ensemble members, cluster uncertainty inevitably leads to uncertainty about the characteristics of $X{\left(t\right)}^{*}$. Thus, it is important to quantify the uncertainty in a forecast arising from the presence of clusters.

To measure the representativeness and cluster uncertainty, one can compute a cluster uncertainty index $\beta$ defined as follows:

$$\beta \equiv {\left(P_{prim}-1\right)}^2,$$

(16)

where $P_{prim}$ is a cluster probability among the set of M cluster probabilities such that there is no cluster probability greater than it. Note that the definition of $P_{prim}$ accounts for situations in which all cluster probabilities are equal. The cluster whose cluster probability is $P_{prim}$ is the primary cluster. In the case that more than one cluster has a cluster probability equal to $P_{prim}$, there will be more than one primary cluster. The reason for squaring $P_{prim}-1$ in Equation (16) is to make the relationship between $\beta$ and the maximal possible cluster forecast skill (Section 4.2) linear for fixed M. Furthermore, subtracting one from $P_{prim}$ is done to make the cluster uncertainty index increase with increasing cluster uncertainty.

The quantity $\beta$ is a measure of representativeness of traditional ensemble statistics. When $\beta <0.25$ ($P_{prim}$ > 0.5), the primary cluster contains the majority of the ensemble members and the traditional ensemble statistics will most closely resemble those of the primary cluster (Section 3.2 and Section 3.3). However, when $\beta \ge 0.25$ the ensemble members will generally be more evenly distributed across the M clusters in order to satisfy Equation (6). Therefore, according to Equations (8) and (15), the traditional ensemble statistics will smooth out the information provided by the individual clusters. For the ensemble forecast shown in Figure 2a, as an example, $\beta =0.25$ so that ensemble mean error is likely because such a trajectory has zero probability of occurring, as it is not a member of any cluster. A similar statement can be made for the local probability of exceedance.

The quantity $\beta$ also measures uncertainty arising from the presence of clusters. As $\beta$ increases ($P_{prim}$ decreases), the ensemble members will tend to become more evenly distributed across the M clusters. Therefore, there will be a weakening consensus regarding the cluster into which the observation will fall. One can regard the quantity $\beta$ as the cluster analog of the ensemble spread. Because the cluster uncertainty can be related to the uncertainty in the timing of peak intensity (Figure 2a), the probability of an observed value exceeding a threshold at a given time (Figure 2b), and the overall observed trajectory, it is a useful integrated metric for assessing many aspects of forecast uncertainty.

4.2. Cluster Forecast Skill

While $\beta$ measures some sources of uncertainty in a forecast with clusters present, it does not indicate how skillfully the verifying cluster is forecast. Thus, it is necessary quantify cluster forecast skill using an additional metric.

To measure cluster forecast skill, one can use the Brier score (BS; [28]), i.e.,

$$BS={\left(p^{*}-1\right)}^2$$

(17)

where $p^{*}=Pr\left({\mathcal{C}}^{*}\right)$ is the cluster probability associated with the verifying cluster. It is often useful to measure the accuracy of a forecast relative to a reference forecast. In the present case, it is natural to compare BS to

$$B{S}_R={\left(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$M$}\right.-1\right)}^2,$$

(18)

which is the Brier score for a random forecast in which the verifying cluster is randomly chosen. The Brier skill score or cluster skill score (CSS) for a single forecast is then given by

$$CSS=-\frac{BS-B{S}_R}{B{S}_R}$$

(19)

if M > 1 and assumed to equal 1.0 if M = 1. Furthermore, if the observation does not fall into a cluster, then $p^{*}$ = 0 and the lowest possible cluster forecast skill is achieved.

The case M > 1 is associated with three general possibilities. The first possibility CSS = 0 means that the accuracy of the forecast is identical to a random forecast. The second possibility CSS > 0 means that the forecast is more accurate than a random forecast. Finally, the case CSS < 0 means that the forecast is less accurate than a random forecast. The cluster forecast skill for the forecasting system as a whole is obtained by averaging CSS over a set of forecasts.

Naturally, cluster forecast skill is related to $\beta$ because $\beta$ measures uncertainty and more uncertain forecasts have a higher potential for forecast errors. In fact, for a given number of clusters, the maximum cluster skill score possible is obtained when $p^{*}=P_{prim}$ and $BS=\beta$. Furthermore, according to Equation (19), cluster forecast skill will generally decrease as $\beta$ increases ($P_{prim}$ decreases).

4.3. Sub-Ensemble Forecast Skill

Good cluster forecast skill does not imply good overall forecast skill because the within-cluster uncertainty will increase forecast error. Once the event ${\mathcal{C}}_i$ occurs, the forecast distribution associated with the verifying cluster should resemble the truth distribution for a skillful probabilistic forecast. The outcomes from the other events or clusters are no longer relevant because the occurrence of the event ${\mathcal{C}}_i$ excludes the possibility that the ensemble members belonging to the other clusters are outcomes. This theoretical idea suggests that forecast skill should be measured with respect to the sub-ensemble statistics associated with the verifying cluster. This idea leads to the concept of sub-ensemble skill scores.

Sub-ensemble skill scores can be defined in the same way as traditional ensemble skill scores. More specifically, a sub-ensemble skill score (SS) is given by

$$SS=1-\frac{SR}{S{S}_{ref}},$$

(20)

where SR is a scoring rule such as RMSE [11], Ignorance score [13], CRPS [12], spread-error score [14], or the Brier score [28] calculated using ensemble members belonging to the verifying cluster. The quantity $S{S}_{ref}$ is a reference score and $-\infty <SS<1$.

An example of $SR$ is the sub-ensemble spread-error score ($E{S}^{*}$) given by

$$E{S}^{*}=\left(s_{*}^2-e_{*}^2-e_{*}{s}_{*}{g}_{*}\right){}^2,$$

(21)

where $s_{*}^2$ is the sub-ensemble variance associated with ${\mathcal{C}}^{*}$, $g_{*}$ is the sub-ensemble skewness, and $e_{*}$ is the sub-ensemble mean error written explicitly as

$$e_{*}=\left(m_{*}-Z\right).$$

(22)

The quantity $m_{*}$ is the sub-ensemble mean corresponding to the verifying cluster, and $Z$ is the verification that follows the truth distribution. For a perfect probabilistic forecast, the moments in Equation (21) will equal those of the truth distribution.

The utility of $E{S}^{*}$ is demonstrated using a simple example (Figure 3). Suppose that a given forecast contains two well-defined mutually exclusive clusters, ${\mathcal{C}}_1$ and ${\mathcal{C}}_2$, such that ${\mathcal{C}}_1={\mathcal{C}}^{*}$. Let the forecast distribution associated with ${\mathcal{C}}_1$ at a time slice $t=t_0$ be a normal distribution with sub-ensemble mean −40 (vertical red line) and sub-ensemble standard deviation 10. Furthermore, let the forecast distribution associated with ${\mathcal{C}}_2$ at $t=t_0$ be a normal distribution with sub-ensemble mean 40 and sub-ensemble standard deviation 10. Thus, the two clusters result in a symmetric bimodal forecast at time $t_0$ as shown in Figure 3. The bimodal distribution results from the ensemble member trajectories passing through the time slice. The distribution corresponding to ${\mathcal{C}}_1$ describes how likely the observation will obtain a certain value given that ${\mathcal{C}}_1$ occurs. The distribution corresponding to ${\mathcal{C}}_2$ contains possible outcomes for the observation given that ${\mathcal{C}}_2$ occurs. Mutual exclusivity of the two clusters implies that the observation can be drawn from only one of the normal distributions. For this example, it is assumed that the truth distribution is identical to that associated with ${\mathcal{C}}^{*}$ at $t=t_0$. Also note that $m_{*}=-40$ because ${\mathcal{C}}_1$ is assumed to be the verifying cluster.

In the present example,

$$E{S}^{*}=\left(s_{*}^2-e_{*}^2\right){}^2$$

(23)

because skewness is zero for normal distributions. The spread-error score, $ES$, calculated using all ensemble members is similar to Equation (23) and is obtained by replacing sub-ensemble variance by traditional ensemble variance $s^2$ and sub-ensemble mean error by traditional ensemble mean error $e$.

For the case Z = −40, $E{S}^{*}$ is solely determined by $s_{*}^2$ because $e_{*}=0$ and $m_{*}=-40$. Consistent with how the forecast and truth distributions are identical, $E{S}^{*}$ evaluates this forecast most favorably because $E{S}^{*}$ is minimized. Moreover, $ES>E{S}^{*}$ because $0=e_{*}<e=40$ and because the variance of the bimodal distribution is greater than that of the single unimodal distribution associated with ${\mathcal{C}}_1$, as can be seen by an inspection of Figure 3.

Perhaps the case Z = 40 is most interesting. At a first glance it might appear that Z = 40 is in a region of high probability. However, the probability that the observation is in ${\mathcal{C}}_1$ $\left({\mathcal{C}}_1={\mathcal{C}}^{*}\right)$ and equal to Z = 40 at $t_0$ is forecast to be virtually zero so that the outcome Z = 40 is not a likely one. Thus, $E{S}^{*}$ penalizes the forecast ($e_{*}=-80)$, resulting in $E{S}^{*}$ being greater in the Z = 40 case than in the Z = −40 case. On the other hand, $ES$ is the same in both cases because $e^2$ and $s^2$ are unchanged. Thus, a skillful and unskillful forecast are evaluated identically by ES.

The discussion in this section and in Section 4.2 suggests that a perfect probabilistic forecast is achieved when there is a single cluster ($CSS$ = 1) and when the truth distribution is perfectly forecast. One can measure the deviation from this perfect forecast using the composite skill score, CS, defined as

$$CS\equiv 1-\sqrt{{\left(1-SS\right)}^2+{\left(1-CSS\right)}^2},$$

(24)

where CS reduces to SS if CSS = 1 (i.e., there are no cluster present). If CSS < 1, then $CS<SS$, which expresses the intuitive idea that the role of clusters is to reduce forecast skill.

4.4. Cluster and Sub-Ensemble Probability Estimates

Although sub-ensemble statistics were shown to be useful, the associated sub-ensemble forecast skill could be low if the verifying cluster has too few ensemble members. In particular, for ensemble systems comprising clusters, a large number of ensemble members may be necessary for producing sub-ensemble probability estimates that are representative of actual probabilities. To create larger ensemble sizes, one can run more numerical simulations, but such an approach can be computationally expensive.

Another approach to creating larger ensemble sizes was proposed by [26]. In the proposed approach (referred to as the phase-aware extension procedure, hereafter), the wavelet transform of the $X_1\left(t\right), \dots , X_N\left(t\right)$ are computed, where the wavelet transformation of the k-th ensemble member is given by

$${\mathcal{W}}_k\left(s,t\right)=\frac{1}{\sqrt{s}}{{\displaystyle \int }}_{-\infty }^{\infty }{X}_k\left(u\right){\psi }^{*}\left(\frac{u-t}{s}\right)du.$$

(25)

In Equation (25), s is wavelet scale, $\psi \left(t\right)$ is the Morlet wavelet function, t is a translation parameter, and the asterisk denotes the complex conjugate [29]. The Morlet wavelet is given by

$$\psi \left(t\right)={\pi }^{-1/4}{e}^{-i{\omega }_{0}t}{e}^{-t^2/2},$$

(26)

where ${\omega }_0=6$ is a dimensionless frequency [30]. The ${\mathcal{W}}_k\left(s,t\right)$ can be decomposed into phase ${\phi }_k\left(s,t\right)$ and modulus ${ℛ}_k\left(s,t\right)=\left|{\mathcal{W}}_k\left(s,t\right)\right|$ spectra, i.e.,

$${\mathcal{W}}_k\left(s,t\right)={ℛ}_k\left(s,t\right)e^{i{\phi }_k\left(s,t\right)}$$

(27)

where the phase spectra are given by

$${\phi }_k\left(s,t\right)={\tan }^{-1}\left(\frac{Im{\mathcal{W}}_k\left(s,t\right)}{Re{\mathcal{W}}_k\left(s,t\right)}\right).$$

(28)

The $Im{\mathcal{W}}_k\left(s,t\right)$ and $Re{\mathcal{W}}_k\left(s,t\right)$ are, respectively, the imaginary and real parts of the ${\mathcal{W}}_k$(s, t).

As noted by [26], the ${\mathcal{W}}_k\left(s,t\right)$ do not represent all possible outcomes because one ensemble member can predict the observed phase spectrum correctly, while another ensemble member can predict the observed modulus spectrum correctly. In other words, the set of all $N^2$ possible outcomes is given by

$$\mathsf{\Omega }=\left\{{\mathcal{W}}_{\mu \nu }\left(s, t\right)={ℛ}_{\mu }\left(s,t\right)e^{i{\phi }_{\nu }\left(s,t\right)}:\mu =1,\dots ,N, \nu =1,\dots ,N\right\}$$

(29)

in wavelet space. Taking the inverse wavelet transform of each member of $\mathsf{\Omega }$ results in a new set of $N^2$ outcomes in physical space denoted by ${\mathcal{X}}_{pw}=\left\{X_{\mu \nu }\left(t\right):\mu =1,\dots ,N, \nu =1,\dots ,N\right\}$.

As noted by [26], numerical errors can arise from the computation of inverse wavelet transformations. To correct for such errors, a wavelet error reduction procedure was performed. That is, a wavelet-derived ensemble member was first computed by taking the wavelet transform of an original ensemble member and subsequently its inverse wavelet transform. Then the difference between the original ensemble member and the wavelet-derived ensemble member was computed at each point in time, where the difference measured the numerical error introduced from the wavelet transformation. If the difference was zero at all times, then the original ensemble member was perfectly reconstructed from the inverse wavelet transformation. Repeating the procedure for all N ensemble members and computing the ensemble mean of the error at each point in time resulted in the average error introduced from the wavelet transformations at each point in time. The ensemble mean error was then subtracted from all $N^2$ wavelet-derived ensemble members to reduce the error. To further reduce errors arising from edge effects, the ends of the ensemble members were padded with a realization of a white noise process [26].

In the presence of clusters, one cannot compute all possible combinations of phase and modulus spectra using all ensemble members simultaneously. More specifically, modulus spectra of ensemble members from the j-th cluster cannot be paired with the phase spectra of ensemble members belonging to the i-th cluster for $i\ne j$ because of mutual exclusivity of the clusters. Instead, the phase-aware extension procedure should be applied to each cluster individually, resulting in M new sets denoted by ${\mathcal{X}}_1^{pw}$, ${\mathcal{X}}_2^{pw},\dots , {\mathcal{X}}_M^{pw}$ such that the ${\mathcal{X}}_i^{pw}=\{X{\left(t\right)}_{\rho \sigma }^i:\rho =1,\dots ,N_i, \sigma =1,\dots ,N_i\}$ contain $N_i^2$ ensemble members. The ensemble members composing ${\mathcal{X}}_i^{pw}$ are obtained by pairing the phase spectra of the ensemble members belonging to the i-th cluster with the modulus spectra of the ensemble members belonging to the same cluster and taking the inverse wavelet transformation of the results. The phase-aware extended ensemble forecast, ${\chi }_{pw}^C$, in the presence of clusters is then the union of the ${\mathcal{X}}_1^{pw}$, ${\mathcal{X}}_2^{pw},\dots , {\mathcal{X}}_M^{pw}$.

In some cases, an element of ${\mathcal{X}}_i^{pw}$ may more closely resemble elements in ${\mathcal{X}}_j^{pw}$. That is, the ${\mathcal{X}}_1^{pw}$, ${\mathcal{X}}_2^{pw},\dots , {\mathcal{X}}_M^{pw}$ may not be strictly clusters. This situation may arise as a result of numerical imperfections of the wavelet transformations. Thus, it is recommended that the cluster-tide algorithm be applied a second time but to the set ${\chi }_{pw}^C$. The application of the cluster-tide algorithm to ${\chi }_{pw}^C$ results in the clusters ${\mathcal{C}}_1^{pw}$, ${\mathcal{C}}_2^{pw},\dots , {\mathcal{C}}_{M_{pw}}^{pw}$, where M was found to equal $M_{pw}$ in this paper. The total number of ensemble members in a phase-aware clustered forecast is

$$N_{pw}={{\displaystyle \sum }}_{J=1}^{M_{pw}}{\mathbb{N}}_J={{\displaystyle \sum }}_{K=1}^M{N}_K^2,$$

(30)

where ${\mathbb{N}}_i$ is the number of ensemble members in ${\mathcal{C}}_i^{pw}$. Phase-aware cluster probabilities and sub-ensemble exceedance probabilities can be calculated using ${\mathbb{N}}_i$ and $N_{pw}$.

5. Sub-Ensemble Forecasting for Periodic Flows

5.1. Theoretical Background

Although previous sections described methods suitable for ensemble forecasts containing clusters, the practical situations in which clusters exist have not been discussed. Thus, in this section, the practical situation of coastal flood forecasting is discussed to provide practical applications of the proposed methods.

Consider a total water level ensemble forecast whose k-th ensemble member $W_k\left(t\right)$ describes a forecast total water level. That is,

$$W_k\left(t\right)=S_k\left(t\right)+T\left(t\right),$$

(31)

where the forecast storm surges $S_k\left(t\right)$ are responses to stochastic forcing (e.g., weather) and $T\left(t\right)$ is the tidal signal. In this case, the observation to be predicted is

W(t) = S(t) + T(t),

(32)

where W(t) is the observed total water level and S(t) is the observed storm surge. A practical example of a total water level forecast for New Port, Rhode Island is shown in Figure 4a.

An important quantity to forecast is the peak of W(t) and therefore it is useful to examine the probability distributions of the forecast peak timing and intensity of W(t). One way to visualize the peak timing and intensity uncertainty is to generate a scatter plot of the forecast timings ${\tau }_k^w$ and peak intensities $P_k^w$ (global maxima) associated with the $W_k\left(t\right)$. Together the ${\tau }_k^w$ and $P_k^w$ compose the points $p_k^w=\left({\tau }_k^w,P_k^w\right)$. An inspection of Figure 4b shows that the $p_k^w$ are clustered around Point 1 (29 October 12:35 UTC), Point 2 (30 October 00:25 UTC), and Point 3 (30 October 11:15 UTC) so that the distribution of the ${\tau }_k^w$ is trimodal. The reason for the trimodality is that the tidal signal modulates the probability of total water level reaching a global maximum at a given time. That is, global maxima are more likely to occur at high tide than at low tide so that total water level global maxima will be clustered around tidal local maxima as shown in Figure 4b. Time points around which the total water level global maxima are clustered for the other stations shown in Figure 1b are provided in

Table 1. Two main time points around which global maxima are clustered in the Sandy reforecasts.

Station	Point 1 (UTC)	Point 2 (UTC)
Atlantic City	10/29/2012 12:25	10/29/2012 23:45
Bergen Point	10/29/2012 13:55	10/30/2012 0:35
Bridgeport	10/29/2012 16:05	10/30/2012 3:35
Cape May	10/29/2012 13:15	10/30/2012 1:05
Kings Point	10/29/2012 16:25	10/30/2012 3:55
Lewes	10/29/2012 1:05	10/29/2012 13:25
Montauk	10/29/2012 13:45	10/30/2012 0:45
New Haven	10/29/2012 16:05	10/30/2012 3:35
New London	10/29/2012 14:25	10/30/2012 0:45
New Port	10/29/2012 12:35	10/30/2012 0:25
Ocean City	10/29/2012 12:55	10/29/2012 0:45
Providence	10/30/2012 0:35	10/29/2012 12:55
The Battery	10/29/2012 13:45	10/30/2012 0:35

.

5.2. Cluster Analysis for Periodic Flows

Given that clusters exist in total water level forecasts, it is important to develop an algorithm to effectively and easily cluster the ensemble members. Thus, in this section, a simplified clustering algorithm is developed in which the number of clusters is largely known and not estimated by additional procedures or statistics [31]. As shown in Figure 4b, total water level global maxima cluster around tidal local maxima, suggesting that a natural way to cluster the data is to use the time of occurrence of global maxima.

To cluster the total water level ensemble members based on time of occurrence, a set of rectangles that partitions the forecast period needs to be constructed. More specifically, let

$$R_q=({\tau }_q^{min}{\tau }_{q+1}^{min})\times \left[c d\right]$$

(33)

be a rectangle ($\times$ is the Cartesian product) centered on the q-th tidal local maximum (denoted by ${\tau }_q^{max}$), where ${\tau }_q^{min}$ is the tidal local minimum located immediately before ${\tau }_q^{max}$ and ${\tau }_{q+1}^{min}$ is the tidal local minimum immediately following ${\tau }_q^{max}$. Here, $0<q\le Q$, where Q is the number of local maxima. Furthermore, assume that the constant c is less than the lowest forecast total water level value and the constant d is greater than the largest forecast total water level value. An example of the rectangular partition is shown in Figure 4. Let $R_{q_1}, R_{q_2},\dots , R_{q_M}$ be a subcollection of the $R_1$, $R_2$, …, $R_Q$ such that each $R_{q_i}$ contains at least one peak, then the i-th cluster ${\mathcal{C}}_i$ will be defined as the set of all elements in $\mathcal{X}$ whose peaks are in $R_{q_i}$. Mathematically, each ${\mathcal{C}}_i$ can be written as

$${\mathcal{C}}_i=\left\{\gamma \in \mathcal{X}:p_{\gamma }^w\in R_{q_i}\right\},$$

(34)

where $p_{\gamma }^w=\left({\tau }_{\gamma }^w,P_{\gamma }^w\right)$ is the peak location of some element $\gamma \in \mathcal{X}$, ${\tau }_{\gamma }^w$ is the peak timing, and $P_{\gamma }^w$ is the peak value. Assuming that each element of $\mathcal{X}$ peaks somewhere, the ${\mathcal{C}}_1, {\mathcal{C}}_2, \dots , {\mathcal{C}}_M$ satisfy Equations (5) and (6) because global maxima can only occur once. Using this cluster algorithm (referred to as the cluster-tide algorithm, hereafter), the optimal number of clusters M is easily determined by finding the number of rectangles containing global maxima. For other clustering algorithms such as k-means clustering, one must use a criterion to determine M [32,33].

To quantify how similar a global maximum point associated with ${\mathcal{C}}_i$ is to other points associated with ${\mathcal{C}}_i$, one can compute a silhouette value [34] corresponding to the k-th global maximum point $p_k^w$. Silhouette values range from −1 to 1 and high values indicate that the point $p_k^w$ is well-matched within its respective cluster. Negative or near-zero silhouette values indicate that the points are not well-clustered, possibly because of too many clusters, too few clusters, or the non-existence of clusters. In the context of probabilistic forecasting, negative silhouette values indicate that the assumption of mutual exclusivity may not be a suitable one. Positive values indicate that one can treat the clusters as mutually exclusive events. In this study, the Euclidean metric is used to measure distance.

To show that the cluster-tide algorithm adequately partitions total water level global maxima, it was applied to the Sandy reforecasts at all 13 stations. The silhouette values were computed for all global maxima resulting in 273 silhouettes values. Figure 5 shows the distribution of silhouette values for the clustered total water level global maxima and indicates that most global maximum points have silhouette values close to one, confirming that the ensemble members are strongly clustered in the total water level Sandy reforecasts. The strong clustering was also found through the application of the commonly used k-means clustering algorithm (not shown). However, in general, other clusters may be present in total water level ensemble systems that can be detected by general clustering algorithms but not by the cluster-tide algorithm.

6. Practical Applications

6.1. Sub-Ensemble Forecast Example

The sub-ensemble means corresponding to the clustered New Port reforecast (Figure 4a) are shown in Figure 6a. The cluster forecast was generated using the cluster-tide algorithm. The sub-ensemble mean ${\overline{X}}^1\left(t\right)$ corresponding to Cluster 1 (red curve in Figure 6a) indicates that a likely outcome for W(t) is that it will peak around Point 1 before generally declining. The sub-ensemble mean ${\overline{X}}^2\left(t\right)$ corresponding to Cluster 2 (blue curve in Figure 6a) suggests that another likely outcome for W(t) is that it will generally increase until reaching a maximum value around Point 2. The thick purple curve represents a third possible scenario (Cluster 3) in which W(t) will generally increase until reaching a maximum value around Point 3. The third scenario is the most unlikely one because $Pr\left({\mathcal{C}}_3\right)$ < 0.05 < $Pr\left({\mathcal{C}}_j\right)$ for j = 1, 2. Whether the observation falls into Cluster 1, Cluster 2, or Cluster 3 is highly uncertain as indicated by the high cluster uncertainty index shown in Figure 7a for New Port. Figure 7a also shows that cluster uncertainty is high for other stations as well. Moreover, the results depicted in Figure 7a suggest that even for the same meteorological event, the cluster uncertainty at different locations may be different as a result of the tidal signals at the different locations being out of phase (not shown).

The full ensemble mean (black curve), as shown in Section 3.2, is a weighted average of the sub-ensemble means but itself is not a representative outcome. Each ensemble member in the reforecast indicates that the observation will peak once, but the ensemble mean suggests that W(t) will obtain nearly the same value at Points 1 and 2. In fact, the cluster uncertainty index for New Port is about 0.27, which implies that the ensemble mean is unrepresentative of the ensemble system as a whole.

6.2. Probability of Exceedance

The local exceedance probabilities for the New Port reforecast are shown in Figure 6b. The exceedance probabilities were calculated using the threshold $T_f$ = $P_m^w=1.6$ m, which is the median value of all 21 peak values (horizontal black line in Figure 6a). The global exceedance probability is approximately equal to 0.5 (horizontal black line in Figure 6b). The sub-ensemble local exceedance probabilities suggest that it is unlikely that the observation will exceed $P_m^w$ around Point 1 and then again at Point 2. Furthermore, the sub-ensemble exceedance probabilities indicate that the observation is more likely to exceed $P_m^w$ if it peaks around Point 2 than if it peaks around Point 1. Note that this relationship between flood potential and the potential timing of the peak in the observation cannot be depicted by the global probability of exceedance, which is nearly the average of the two global exceedance probabilities associated with the two main clusters (red and blue squares in Figure 6b).

As discussed in Section 3.3, the traditional local exceedance probability is a weighted average of sub-ensemble exceedance probabilities. Thus, the traditional local exceedance probabilities should under-estimate the exceedance potential at Point 1 (Point 2) if the observation falls in Cluster 1 (Cluster 2). As indicated by the high cluster uncertainty index shown in Figure 7a, the traditional exceedance probability curve as a whole is not representative of the information provided by the ensemble members. In fact, the traditional exceedance probability curve indicates that the observation could exceed $P_m^w$ at Point 1 and then again at Point 2 even though not a single ensemble member exceeds $P_m^w$ at both Points 1 and 2. The high cluster uncertainty indices displayed in Figure 7a imply that the traditional probability of exceedance curves will also not be representative of the ensemble system for most of the other 12 stations, especially for the New London and Providence stations where the cluster uncertainty indices are particularly high ($\beta$ > 0.2).

The results were found to be similar for the phase-aware ensemble forecast (Figure 8) corresponding to the ensemble forecast shown in Figure 4a. However, as shown in Figure 6c, the local sub-ensemble exceedance probability curves are smoother than those depicted in Figure 6b. Furthermore, the global exceedance probability associated with Cluster 2 (blue square in Figure 6c) is greater than that associated with the original ensemble forecast (blue square in Figure 6b).

6.3. Cluster Forecast Skill

Although the previous sections showed how traditional ensemble statistics can be unrepresentative of the information provided ensemble members, the ability of the NYHOPS model to forecast the verifying cluster has not been assessed. The cluster skill scores for all 13 stations were thus calculated for the Sandy reforecasts.

The individual cluster forecast skill scores shown in Figure 7b indicate that for 10 of the stations, the observation falls into the primary cluster, as reflected by the positive cluster skill scores. In other words, the general timing of the peak total water level is skillfully forecast and a majority of ensemble members have trajectories generally similar to the observation. For example, the observation corresponding to the New Port reforecast shown in Figure 6a peaks around Point 2, the point at which 48% of the ensemble members forecast an observed peak to occur. It is noted that for New Port $\beta$ > 0.25 (Figure 7a), which explains the modest cluster forecast skill for that station. A comparison of Figure 7a,b shows that stations with the larger $\beta$ are the stations with generally lower cluster skill. In fact, the Pearson correlation between CSS and $\beta$ is −0.55, indicating that $\beta$ may be a useful metric for assessing the likelihood of the verifying cluster being wrongfully forecast. The overall (average) cluster forecast skill for the Sandy reforecasts is 0.30, which is negatively impacted by the unskillful forecasts at the Bridgeport, Kings Point, and New Haven stations. For the phase-aware ensemble forecasts corresponding to the original 13 ensemble forecasts, the overall cluster forecast skill is similar and equal to 0.18.

Additional sources of error were identified by setting $SR$ in Equation (20) to the RMSE calculated between the sub-ensemble mean of the verifying cluster and the observation. The reference score $S{S}_{ref}$ was set to the RMSE computed between the observation and tide. The RMSE skill scores were plotted as a function of the cluster skill scores to demonstrate the importance of cluster forecast skill on the overall forecast quality. The first days of the forecast periods (time points before 28 October) were excluded from the RMSE calculations because the meteorological uncertainty is virtually non-existent during those periods.

Figure 9a indicates that the forecasts with the largest cluster skill are generally the forecasts with the highest RMSE sub-ensemble skill. In fact, the correlation between CSS and RMSE skill is 0.61 and thus RMSE error is related to the number of clusters and to the number ensemble members in the verifying cluster. The results for the corresponding phase-aware ensemble forecasts were found to be nearly identical (not shown). At least some of the remaining error is likely related to the sub-ensemble spread around the sub-ensemble means. These results suggest that the number of ensemble members in each cluster can be used to anticipate potential forecast errors because greater cluster forecast skill implies more ensemble members in the verifying cluster. The composite skill score averaged over all 13 stations in this situation is 0.25, indicating overall positive skill.

Given the importance of the peak total water level on the maximum potential of flood events, the spread-error skill score $E{S}^{*}$ was calculated using the distribution of peak values associated with the verifying cluster. The reference spread-error score in this analysis was computed using all peak values and thus $E{S}^{*}>0$ means that the sub-ensemble peak value distribution is a better representation of the truth distribution than is the distribution of all peak values. A CSS relationship with $E{S}^{*}$ was demonstrated by plotting $E{S}^{*}$ as a function of CSS.

As shown in Figure 9b, about 50% of stations have $SS>0$, suggesting that the sub-ensemble probabilistic peak forecasts associated with those stations are more skillful than the full ensemble peak distributions. The least skillful probabilistic forecasts are those with negative cluster skill scores, suggesting that the number of ensemble members within each cluster may be an indicator of potential forecast error. This result is not surprising because negative cluster skill scores imply that the verifying clusters comprise few ensemble members so that the distributions of peak values are unlikely to adequately represent the truth distributions. The composite skill score averaged over all 13 stations is −1.5 reflecting overall poor probabilistic skill when using the full ensemble as a reference forecast. However, the mean composite skill score calculated using the phase-aware ensemble forecasts corresponding to the original 13 ensemble forecasts is −0.42, suggesting that increasing the number of ensemble members could improve forecast skill.

7. Conclusions and Discussion

This paper demonstrated how clusters arise naturally in total water level ensemble forecasts as a result of the superposition of storm surge and tide. Clusters in total water level ensemble systems are determined largely by the temporal locations of ensemble member global maxima, the global maxima clustering around local maxima in the tidal signal. It was found that the existence of ensemble member clustering impacts the interpretation of common forecast metrics such as the ensemble mean and exceedance probability.

It was further shown that clusters represent an additional source of uncertainty in total water level forecasts and thus the presence of clusters should be considered when assessing the overall uncertainty of a forecast and potential forecast errors. Given that the clusters arise naturally from the interaction of the tide and storm surge, this source of uncertainty cannot be eliminated. Furthermore, forecast errors are related to forecast uncertainty so that the presence of clusters may limit coastal flood forecast skill even for forecast lead times of a few days. Future work could include understanding how the number of clusters in coastal flood forecasts changes with lead time. Such future work could identify a maximal lead time for which coastal flood forecasts can no longer be skillful.

Practical applications of the cluster-tide algorithm to total water level reforecasts for Sandy revealed that sub-ensemble statistics are globally more representative than traditional ensemble statistics. A cluster uncertainty index was used to show that the Sandy reforecasts contained considerable cluster uncertainty in some cases. The cluster skill score showed that the NYHOPS model has skill in predicting the cluster into which the observation will fall. A cluster skill score relationship with the cluster uncertainty index was identified, which is the cluster analog of the traditional spread-skill relationship. Thus, the cluster uncertainty index may be a useful index for assessing potential errors in a forecast. It was found that sub-ensemble forecast skill generally increases as the cluster forecast skill increases.

One limitation of the present study is that only clusters arising from the tidal signal were considered. Other ensemble member clusters may be present that cannot be extracted using the cluster-tide algorithm Thus, additional sources of uncertainty may arise from clusters not identified in this study. Future work could include the identification of other types of ensemble member clusters through the application of more general clustering algorithms to total water level forecasts.

While clustering was shown to be a necessary post-processing step for total water level forecasts, the step adds complexity to the conveyance of important flood information to the general public. Future work could thus include developing methods to better summarize the output from the cluster-tide post-processing method adopted in this study.