1. Introduction
Real-time predictions of oil spill transport rely on operational systems that integrate boundary forcing data, hydrodynamic models, oil transport models, and post-processing software. The predictions inevitably have uncertainties ascribed to empirical parameters of the numerical models (e.g., windage factor, oil diffusion coefficient), as well as forecast weather, offshore currents, oil spill conditions and model structural errors [
1]. The latter effects are primarily due to the coarse model grid that is required to get rapid results for emergency operations, although the type of numerical model, governing equations, etc. can also contribute. This issue is not unique to oil spill modeling, but also appears wherever fast models are required e.g., for forward integration of climate simulations over centuries. In climate science, quantifying/reducing uncertainty associated with downscaling from a coarse global-scale grid to a fine-scale local grid is an established research area (e.g., [
2]), but this area has been neglected in the development of operational oil spill models. Previous oil spill research focused on quantifying the uncertainties associated with empirical parameters or unknown forcing (i.e., forecast uncertainty), typically by applying stochastic methods (e.g., random walk) to account for subgrid processes [
3,
4] and/or using ensemble averaging as a means to provide greater confidence in a prediction [
5,
6]. Thus, previous research has not focused on the uncertainty arising from the structure of hydrodynamic models. Herein, we will focus solely on the uncertainty introduced into an oil spill transport prediction by the geometric approximations inherent in the coarse model grid, which has been defined as “geometric uncertainty” to differentiate it from broader range of uncertainty in model structure [
1]. Geometric uncertainty is arguably the most tractable of our uncertainty sources as it is entirely controlled by our modeling approaches and can be directly diagnosed by model-model comparisons. Our goal is to develop methods for quantifying the effects of geometric uncertainty that are computationally efficient and hence practical for operational oil spill modeling. Note that we are not claiming that geometric uncertainty is necessarily the largest or most important uncertainty in an oil spill modeling system. Our point is that this type of uncertainty can be diagnosed with model-model comparisons and hence is readily quantifiable
without physical drifter experiments or real-world spill observations. In contrast, diagnosing uncertainty through model comparisons to physical observations will integrate
all sources of uncertainty across a range of parameters and modeling assumptions. As physical experiments are expensive, it is difficult to obtain sufficiently differentiated data to separate different causes of uncertainty and identify remedies that are process-based rather than stochastic. By proposing an approach that isolates geometric uncertainty from other factors, the present work is a step forward that will hopefully provide foundations for future work that quantitatively addresses other uncertainty sources.
The approach we take is based on the proposition that oil spill response managers would like to have some measure of confidence (or its inverse, uncertainty) in a given oil spill transport prediction—but they want a reasonable prediction now rather than a better prediction tomorrow. Unfortunately, we do not see any fix for coarse-grid model errors that does not also require some form of fine-grid model for comparison and analyses. Ideally, an oil spill response manager would have high-resolution hydrodynamic models perpetually running over their entire domain of responsibility; however, coarse-resolution models are more practical for continuous operations. In this paper, we show how “on-demand” high-resolution models—which are only started in an area after a spill occurs—can be systematically combined with coarse-grid operational models to provide oil spill transport forecasts with improved confidence. This approach does not address all the sources of uncertainty, but is logically better than relying on coarse-grid models alone.
The principal goal of this research is to develop new approaches to quantify the geometric uncertainty that can be applied operationally to support oil spill decision-making. As such, we focus on how to apply available numerical resources combining coarse-grid and fine-grid models to improve uncertainty quantification. We use a matched set of coarse-grid and fine-grid models to quantitatively analyze errors introduced by the coarse-grid approximations on particle transport models used in oil spill transport prediction. We propose an approach to combine measures of particle separation and diffusion errors into a single confidence measure that represents the geometric uncertainty of the coarse-grid model. We also propose two new approaches to quantify the geometric uncertainty (increasing confidence) that can be readily applied in an operational oil spill system. The first approach uses historical simulations of the coarse and fine models to train a data-driven uncertainty model that provides an insight into the confidence measure. The second approach, a multi-model integration, provides rapid predictions from the coarse-resolution model and, as the slower fine-grid model results become available over time, creates an ensemble prediction that provides a direct expression of this uncertainty. These new tools quantify model performance based on Lagrangian transport behavior rather than Eulerian variables, and thus might have future applications for model designers who seek to develop a model mesh that is optimized for accurate Lagrangian transport.
The first goal of this paper is to establish a statistical relationship between coarse-grid and fine-grid models based on their historical outputs, and apply the data model to future projections, i.e., using the data-driven model-model comparison as a
priori estimate to assess the geometric uncertainty for real-time predictions. Such a data-driven model, when incorporated into the operational system, can provide emergency response managers guidance or diagnostics about the coarse-grid model reliability [
7]. Thus an operational system can be considered as a joint effort of the numerical models and the data-driven models. This work presents a data-driven model to predict the geometric uncertainty of an operational system and is the first study that applies modern machine learning techniques for addressing uncertainty issues in operational oil spill modeling.
Our second goal is to move beyond the use of a single-model predictions for operational decision-making. We seek methods for real-time generation of model ensemble members with perturbed parameters and/or forcing data that can be built into an operational forecast system. The ensemble results can be represented as a probability cloud of multiple spill trajectories, which allows uncertainty to be expressed directly within the context of an ensemble mean. The challenge addressed herein is in constructing the ensemble that matches the uncertainty problem for oil spills and explaining the probabilistic predictions from the results [
8]. These multi-plot approaches to visualizing uncertainty could provide new tools for operational managers, but there remain open questions as to how managers would perceive such data and whether the proposed formats are effective.
A principal limitation of this paper is associated with the state-of-the-art in validating hydrodynamic models. Such models are readily validated for tidal dynamics and salinity transport using sensors emplaced by operational agencies (e.g., [
9]). However, what is simply unknown is with what precision must a model capture a limited set of observed tidal elevations, salinity transport, and currents to
also get the Lagrangian drifter paths correct. Herein, our new methods are developed and tested with the assumption that a fine grid model that has been demonstrated to represent tidal fluxes and salinity transport is also sufficient for Lagrangian drifter transport. This assumption might be someday proven invalid, but the mechanics of the new methods will remain sound, regardless.
In this paper, we start from the background on geometric uncertainty, the data-driven model, and the multi-model integration in
Section 2. The uncertainty quantification and the proposed approaches are introduced in
Section 3. The corresponding results and the operational applications are discussed in
Section 4 and
Section 5, respectively. The conclusions are provided in
Section 6.
2. Background
Geometric uncertainty in modeling is not a new concept, but methods for quantification as part of oil spill operational applications are relatively recent [
1,
10]. This section describes some measures of uncertainty and discusses two approaches that have been previously used to quantify oil spill uncertainties: data-driven models and multi-model integration, which are based on
a posteriori analyses of historical data and real-time ensemble predictions, respectively.
The prediction, post-analysis, and uncertainty quantification of an oil spill build upon the near-surface water velocities provided by operational-scale hydrodynamic models [
11]. These predicted velocities are less accurate when the scales of velocity variability in the flow are poorly resolved by a coarse model grid, e.g., where complex boundary geometries redirect the flow and create sharp spatial gradients that are smaller than resolvable the grid scale. By definition, for our purposes a “coarse-grid” model is simply one that is unable to resolve small-scale flow features that affect velocities that, in turn, affect oil spill transport. The coarse-grid effects are quantifiable in discrepancies between models at different grid resolutions (e.g., [
9,
12,
13]). The term “geometric uncertainty” was used by [
1] to describe the uncertainty arising from use of a grid that is not sufficiently fine to represent the physical processes of interest for the numerical algorithms of a given model. As an example, the eddies formed by starting jet vortices around a narrow ship channel [
14] that are unresolved in a coarse-grid model is a predictable form geometric uncertainty that affects prediction of oil spill transport across a bay-coastal shelf interface [
15]. A sensitivity study of oil spill uncertainties was conducted in Galveston Bay, Texas (USA) that showed geometric uncertainty can be a significant contributor to uncertainty in estuarine oil spill modeling [
10].
Clearly, geometric uncertainty can be reduced (or possibly eliminated) by employing a sufficiently fine grid resolution to obtain a more accurate prediction of velocity fields [
12]. Unfortunately, such high-resolution models are not generally practical for operational applications over large coastal areas. Note that although oil spill transport in an estuary is often confined to near-surface regions, capturing the correct surface velocities will typically require three-dimensional (3D) numerical simulations—particularly in the presence of deep ship channels and horizontal salinity gradients [
9]. However, the critical issue for modeling the circulation is resolving the horizontal (barotropic) scales of motion. In general, the computational expense for a desired prediction interval increases by a factor of eight when the length scale resolved in the horizontal grid is reduced by 50%. This occurs because the number of grid cells is increased by a factor of four and the model time step must be cut in half. As an example, the fine-grid model in [
9] resolves a length scale of about 1/4 of the coarse-grid model of the same estuary but requires a time step of about 1/15 used at the coarser scale. Thus, the fine-grid model nominally requires about 240× the computations of the fine grid model for a given interval of time (the scaling is approximate due to the peculiarities of numerical stability for unstructured grid models). Thus, a coarse-grid model that can run in tens of minutes takes multiple days for the equivalent fine-grid model despite the relatively modest improvement in grid resolution. Furthermore, the fine grid resolution that is “good enough” for oil spill modeling remains an open question as it necessarily involves the integration of error in Lagrangian transport, which is more difficult that matching tidal elevations and currents. We take the view that geometric uncertainty will be with us for the foreseeable future and needs to be quantified for operational managers to understand its effect on predictions and model designers to evaluate the optimum mesh design.
A key challenge to quantifying geometric uncertainty is that it varies across a complex system such as a bay or estuary. A coarse-grid model often has a relatively uniform grid size throughout the domain, but a fine-grid model typically refines the regions with complex geometric features (e.g., deep ship channels) where the effect of small-scale physics is expected to be significant [
16]. Thus, there may be broad regions of a bay or estuary where a coarse-grid model introduces relatively little uncertainty and critical choke-points where the uncertainty is dramatically increased. For example, particles mixing within eddies at the tidal inlet can reduce the diffusion area of the predicted spill but this effect is neglected in coarse-grid models [
15]—an uncertainty that is irrelevant where such eddies cannot occur. The underestimation of such processes increases the local geometric uncertainty. Additionally, such uncertainty is likely to increase in time/space regions with strong flow dynamics as oil particles will transport and diffuse in a more rapid manner [
17].
Across a wide variety of disciplines, data-driven or statistical models have been used for uncertainty quantifications (typically for all sources of uncertainty rather than our specific focus on geometric uncertainty). While the numerical models are derived from the mathematical description of physics, the data models make predictions based on historical datasets by assuming the present state is similar to those in the future. These typically have lower computational costs than the numerical models used in nowcast/forecast simulations. Conventional statistical models have been employed for hazard mapping and risk assessment of oil spills to understand potential spill impacts based on past data [
18,
19]. A range of data-driven models have been built upon machine learning algorithms and computational intelligence that automatically learn the statistical relationship among variables. These include models in ecology [
20], hydrology, [
21] and oil spills [
22]. For oil spill analyses, the prior data-driven models were driven by
posteriori analyses of hypothetical spills and past spills to provide real-time predictions from present-state variables (e.g., flow conditions, atmospheric forcing). In operational applications, it is difficult for data-driven models to provide results as accurate as the numerical simulations for time-series predictions [
21]. However, we believe these models can be adapted to complement mechanistic models by quantifying the system uncertainty and diagnosing modeling errors.
Another approach to quantifying uncertainty is using predictions from multiple numerical simulations to create ensembles during real-time prediction [
23,
24]. Such multi-model ensembles have been constructed by using models with different grids, algorithms, physical parameterizations, data assimilation schemes, and/or forecast forcing conditions. The ensemble approach can be used to quantify the prediction uncertainty and determine the relative model performance or bias by displaying predictions with various uncertainty levels. With such tools operational decisions can consider the probabilistic expression of the uncertainty.
5. Operational Application to HysoPy
We use two case studies to demonstrate application of the data-driven uncertainty model and the multi-model integration in the HyosPy operational system. Both cases indicate that the proposed approaches were able to quantify the geometric uncertainty, which is critical for evaluating confidence in operational spill predictions. The maps with multiple trajectories or a probability cloud were visualized using new Google Map tools. Please contact the corresponding author for the Google map files.
Hypothetical spills are released at two locations with the spill information provided in
Table 1. The spill in Case 1 is initiated along the Houston Ship Channel, analogous to a midnight spill caused by a ship collision, and the spill in Case 2 is released in the upper Galveston Bay to provide a contrast in behaviors. Both spills were driven by the near-surface velocities of GB-C and GB-F models along with winds from the North American Regional Reanalysis (NARR) dataset [
37] with a windage of 0.01. The multi-model integration was applied by setting
, resulting in 6 different trajectories, which were specified as coarse-grid, 20% fine-grid, 40% fine-grid, 60% fine-grid, 80% fine-grid and fine-grid predictions. The corresponding simulation time for each ensemble member was 0.1, 1.08, 2.06, 3.04, 4.02 and 5 h, respectively.
It can be seen that the trajectory discrepancies (the separation distance and the diffusion error) are significant between the models, implying a critical geometric uncertainty in the GB-C prediction (
Figure 10). In Case 1, the GB-C trajectory extended towards the inner Galveston Bay, which was misleading given the particles beached on the eastern coastline as predicted by the GB-F. In Case 2, the GB-F predicted particles presented a higher probability of beaching on the Texas City Dike, whereas those driven by the GB-C advected towards the bay mouth.
The geometric uncertainty in the GB-C prediction was reflected by the time series of the confidence degree (
C) and the effective confidence (
E) estimated from the data-driven uncertainty model every 10 h. In Case 1, the confidence degree (
C) was low (∼20%) at the initial time step and then increased to a high level (>80%), whereas the integrative error of
E never recovered from the initial poor
C (
Figure 10a). These behaviors imply that the GB-C prediction is not reliable and has non-negligible geometric uncertainties at the beginning, leading to aggregated errors through the simulation period, which explained the overall poor model performance. In Case 2,
C fell between
and
, while
E slowly downgraded over time (
Figure 10b). The effective confidence suggests that the GB-C is effective for the first 10 h, after which fine-grid simulations should be applied to obtain a more accurate prediction.
The confidence measure emphasizes the need for real-time corrections based on the best available observations—i.e., when E gets low, the GB-C predictions have to be treated as inherently suspect unless re-initialized with new observations of the spill. That is, imagine that in Case 1 we received new data at 10 h as to the general location of this spill, the oil spill simulation could then be re-initialized at the new location with a larger spread. When coupled with the uncertainty model, the operational oil spill system (e.g., HyosPy) is capable of identifying the potential errors and reporting the intermediate model performance to the emergency response managers for decision-making.
The multi-model integration generates multiple trajectories (
Figure 11). The prediction generally improved when the GB-F had run a longer time and was employed for a larger portion of the oil spill simulation. The coarse-grid prediction accuracy was significantly increased by the 20% fine-grid prediction, in which the particles advected in a similar direction as the pure GB-F prediction (
Figure 11a). When the GB-F was employed for over 40% of the simulation, the particles beached at almost the same location. The predicted trajectories were similar among 40% fine-grid prediction to pure fine-grid prediction, while using the GB-F model for the first 20% did not improve much of the prediction (
Figure 11b).
In a real-world application, the prediction ensemble would grow over time and provide important information for making decisions within a limited response time. With the multi-model integration, we can identify the GB-C modeled errors at early stages without having to wait for the pure GB-F prediction to tell where and when the geometric uncertainty was high and the GB-C prediction is no longer adequate. For these experiments, the geometric uncertainty dominated at the beginning in Case 1 and after 20%∼40% of the simulation in Case 2.
We can also use the multiple trajectories to create “probability maps” of the multi-model ensemble as shown in
Figure 12. These maps can be used to visualize how the geometric uncertainty is spreading the solution as the ensemble evolves over time. Probability maps describe the normalized (
) probability of a point in space occurring in the ensemble. Thus, the red represents clustering of the predicted tracks in some area and
not necessarily a high probability that the particular area is the best prediction [
24]. In Case 1, the high probability (∼0.2) occurs along the fine-grid prediction and is away from the coarse-grid prediction over the entire simulation period. This result implies that the geometric uncertainty of the coarse-grid model was high from the beginning. In Case 2, the probability was high for all predictions during the first third of the simulation when we gained confidence that the coarse-grid prediction was adequate. The probability map coincides with the multiple trajectories (
Figure 11), but avoids single biased predictions. However, such maps must be used with caution as they are subject to misinterpretation; that is, we can imagine a case where a number of early tracks in the multi-model ensemble fall close together and the most recent track crosses an entirely different space due to a divergence in the latest fine-grid prediction. In such conditions, the early tracks would show up as high probability and the latest track as low probability, despite the fact that the latter is clearly the preferred estimate. Thus, these probability maps should be seen as a way of understanding the geometric uncertainty model behavior but there is a need for further consideration as to how such visualizations could be used in an operational context. In future work, we plan to evaluate a weighting method that ensures that ensemble members with a greater fraction of fine-grid simulation will dominate the probability map. Evaluating how operational managers would respond to such maps and their understanding of weighting functions remains a subject for future research.
The two approaches illustrated above can be used to quantify the geometric uncertainty and display the difference between coarse-grid and fine-grid models. The multi-model integration provides an explicit information of where and when the geometric uncertainty of the coarse-grid prediction is high during an event. The ensemble approach of the multi-model system allows generation of a new ensemble member when the fine-grid model has only run a portion (e.g., 20%) of the entire simulation. The ensemble set continues to grow during the fine-grid simulation run, which adds to our knowledge about the prediction uncertainty and provides oil spill response managers with greater insight into the predictions.
6. Conclusions
This work investigated geometric uncertainty of an oil spill modeling system designed for operation with combined coarse-grid and fine-grid models. The system was developed and demonstrated on Galveston Bay using the SUNTANS hydrodynamic model and the GNOME oil spill model within the HyosPy modeling system. The modeled difference in the separation distance and the diffusion error between coarse and fine-grid model results was used as an indicator of the geometric uncertainty for coarse-grid model predictions. The geometric uncertainty was influenced by both the model grid and the flow dynamics, and showed critical variability across the estuary at the tidal scale of hours. We proposed two approaches to operationally quantify geometric uncertainty: a data-driven uncertainty model and multi-model integration.
The data-driven uncertainty model provides an a priori estimate of the coarse-grid model reliability. We applied the GBRT machine learning algorithm to develop the uncertainty model trained on numerical Lagrangian experiments in Galveston Bay over a three-month period. A geospatial data clustering algorithm was derived to reduce the spatial dimension of the model so that data collection for model training was less computationally intensive. The uncertainty model was evaluated with hypothetical spills and showed satisfactory performance for predicting the effective confidence over ∼50 h.
The multi-model integration employs partial simulation results from fine-grid model that are available for early times after an event and are then extended to later times with the coarse-grid model. Model runs with different combinations of coarse and fine-grid simulations are used as part of a multi-model ensemble. The resulting ensemble of spill tracks can be visualized as multiple trajectories or as a probability map. Both trajectories and probability maps are integrated with Google Maps so that they can be projected on satellite images. This approach allows emergency response managers to obtain an increased understanding of the geometric uncertainty as an event is tracked and as new model results pare produced.
The uncertainty model and the multi-model integration were applied to the HyosPy system in two case studies; we initiated hypothetical spills at different locations to demonstrate the type of information on uncertainty that can be generated. The data-driven uncertainty model enables the HyosPy system to self-identify the uncertainty errors statistically, while the multi-model ensemble allows the emergency managers to quantify the geometric uncertainty without considering its variabilities. Both approaches identify the modeling errors of the coarse-grid prediction, i.e., where and when the coarse-grid model performance starts to decrease significantly.
The uncertainty quantification approaches proposed herein are not necessarily limited to oil spill modeling, and could be applied more broadly when more multiple models are available of a given process. The key challenge is to find the data model appropriate to the physical patterns of the system. Data-driven machine learning models can also be used to quantify other forms of uncertainty associated with forecasting or empirical parameters. In general, the above techniques can be adapted for use whenever historical assessment of model-model comparisons can be used to evaluate uncertainty.
The new methods quantify the error in Lagrangian particle transport associated with the design of a model grid mesh—i.e., the spatial/temporal geometric uncertainty introduced with a coarser grid. Traditional mesh design (e.g., [
9]) focuses solely on validating models against Eulerian observations or the grid convergence of Eulerian variables. For our test case in Galveston Bay, the results above show that the existing coarse-grid mesh provides low confidence for Lagrangian transport in the critical area of the ship channel entrance. We believe these tools hold promise for future model developers in designing and testing coarse unstructured meshes that are optimized for Lagrangian particle transport in areas where oil spills are likely to occur.