Protocols for Water and Environmental Modeling Using Machine Learning in California

Abstract

The recent surge in popularity of generative artificial intelligence (GenAI) tools like ChatGPT has reignited global interest in AI, a technology with a well-established history spanning several decades. The California Department of Water Resources (DWR) has been at the forefront of this field, leveraging Artificial Neural Networks (ANNs), a core technique in machine learning (ML), which is a subfield of AI, for water and environmental modeling (WEM) since the early 1990s. While protocols for WEM exist in California, they were designed primarily for traditional statistical or process-based models that rely on predefined equations and physical principles. In contrast, ML models learn patterns from data and require different development methodologies, which existing protocols do not address. This study, drawing on DWR’s extensive experience in ML, addresses this gap by developing standardized protocols for the development and implementation of ML models in WEM in California. The proposed protocols cover four key phases of ML development and implementation: (1) problem definition, ensuring clear objectives and contextual understanding; (2) data preparation, emphasizing standardized collection, quality control, and accessibility; (3) model development, advocating for a progression from simple models to hybrid and ensemble approaches while integrating domain knowledge for improved accuracy; and (4) model deployment, highlighting documentation, training, and open-source practices to enhance transparency and collaboration. A case study is provided to demonstrate the practical application of these protocols step by step. Once implemented, these protocols can help achieve standardization, quality assurance, interoperability, and transparency in water and environmental modeling using machine learning in California.

1. Introduction

Water and environmental systems comprise surface water bodies like rivers and lakes, along with groundwater aquifers, wetlands, and other components. Currently, these systems confront pressing challenges. These challenges include but are not limited to intensifying weather and hydrological extremes, habitat degradation, climate change, and unsustainable use of natural resources. California, in particular, is highly susceptible to these challenges due to its intricate and aging water infrastructure, along with a large population and economy that place significant strain on its resources and infrastructure [1,2]. Addressing these challenges requires a comprehensive understanding of complex environmental processes and their interactions, which is where water and environmental modeling plays a pivotal role.

Water and environmental modeling offers a multifaceted approach to comprehending complex systems by integrating diverse data sources, representing complex processes, and facilitating simulation and prediction. It also serves as a valuable tool for informing decision making across various sectors by providing insights into the current and projected conditions of the systems and their responses to different scenarios. Moreover, it facilitates communication and collaboration among stakeholders by providing a common platform for analyzing data, evaluating options, and fostering productive dialogue, ultimately contributing to more informed decision making [3,4].

California has a rich history of developing and employing models to plan and manage the State’s intricate water and environmental systems. For example, as far back as 1930, the California Division of Water Resources (now known as the California Department of Water Resources (DWR), the overarching mission of which is to sustainably manage the water resources of California in cooperation with other agencies) issued Bulletin 120, which provides forecasts for April to July runoff volume based on various variables, such as precipitation, snowfall, and runoff indices [5,6]. In 1931, the DWR devised the Bulletin 27 Salinity Intrusion Model to estimate the required flows for managing seawater intrusion in the Sacramento–San Joaquin Delta (Delta), the hub of California’s water transfer systems [7]. In 1978, the DWR introduced DAYFLOW, a straightforward mass balance model intended for determining the daily historical hydrology along the boundaries of the Delta [8]. Subsequently, in 1986, the DWR developed DWRSIM, a water planning model, to simulate the joint operation of the State Water Project (SWP) and Central Valley Project (CVP) [9]. Later on, the DWR and the U.S. Bureau of Reclamation collaborated to develop CalSim as a replacement for DWRSIM [10]. In the early 1990s, the DWR implemented the hydrodynamics and water quality model DSM2, designed to simulate variables such as flow, stage, salinity, water temperature, and other water quality variables in the Delta [11].

These models, encompassing statistical and physics-based approaches, play a vital role in various aspects of water management. They facilitate real-time flood and water supply forecasting [12,13,14,15,16,17], drought assessment and management [18,19], regulatory compliance [20], water rights management [21], climate change adaptation and mitigation planning [22,23], infrastructure investment and design [24], ecosystem restoration [25,26,27], and long-term water resource planning [28], among others. Given the interconnected nature of water systems, ecosystems, and human activities, computer simulations are indispensable to comprehensively understanding and predicting their interactions. Consequently, the majority of significant water projects in California rely on analyses conducted through modeling [9,10,28].

As models become more central to managing California’s water systems, scrutiny and controversy surrounding their use and development are also rising. It is widely understood among the State’s water community that the adoption of standardized modeling protocols is imperative. These protocols can help enhance the quality of models and modeling studies, while also bolstering the trust and confidence of decision makers and stakeholders who rely on model outcomes to inform their decisions. In light of this, the Bay-Delta Modeling Forum (now California Water and Environmental Modeling Forum (CWEMF)) developed a landmark document in 2000 outlining protocols for water and environmental modeling in California [3]. The CWEMF updated this document in 2021 to reflect advancements in the field, driven by new questions, technologies, and increased stakeholder engagement [4]. The updated document provides current best practices and highlights emerging techniques like machine learning, which could significantly enhance modeling capabilities. However, it does not offer specific guidance on developing and applying machine learning models. Machine learning techniques have advanced significantly in recent years, leading to a surge in their applications across California’s water and environmental sectors. These applications span surface water hydrology [29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44], groundwater hydrology [45,46,47,48,49,50,51,52], hydro-meteorology/hydro-climatology [53,54,55,56,57,58,59,60,61,62,63,64], water quality [65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85], ecology [86,87,88,89,90,91,92,93,94,95,96,97,98], and water management and operations [99,100,101,102,103,104,105,106,107,108,109], among others. Machine learning models offer a distinct advantage over traditional statistical methods by automatically discovering intricate patterns and interactions within data, without demanding predefined assumptions (e.g., linearity and normality) about the functional forms of relationships [110,111]. Furthermore, while physics-based models necessitate a thorough understanding of underlying physical mechanisms and precise parameterization, machine learning can extract valuable insights even when a complete, theoretical understanding of the system is lacking [112]. These capabilities render machine learning particularly effective in analyzing complex environmental systems, where variable interactions are frequently non-linear, interdependent, and challenging to express through conventional methods. By employing data-driven learning, machine learning can reveal obscured patterns, enhance prediction accuracy, and augment existing modeling approaches, especially in scenarios where traditional techniques encounter limitations. As a result, the protocols governing traditional models and machine learning models may not always be interchangeable.

This paper aims to supplement existing modeling principles and guidelines [3,4,113,114] by providing protocols for the development and implementation of ML models in water and environmental modeling in California. These protocols stem from practical insights gained from using conventional machine learning techniques in water and environmental modeling. Although generative artificial intelligence (GenAI) [115] represents the latest machine learning advancement, its application in California’s water modeling community remains limited. As such, the guidelines provided in this paper are not specifically tailored for GenAI models. More specifically, the protocols proposed in this study focus on conventional machine learning models that process quantitative and spatial water-related data, including numerical or categorical meteorological, hydrological, water quality, biological, and geospatial data. These protocols are not applicable to machine learning models designed for processing text, image, audio, or video data.

The remainder of the paper is structured as follows: Section 2 reviews machine learning in water and environmental modeling, covering the typical life cycle of a machine learning model, the roles that machine learning can play in water and environmental modeling, and the DWR’s three decades of experience with machine learning. Section 3 outlines the machine learning protocols based on the DWR’s experience and includes a case study to demonstrate these protocols step by step. Section 4 discusses the benefits and caveats of implementing these protocols. Section 5 concludes the paper and suggests future directions.

2. Machine Learning in Water and Environmental Modeling

Artificial intelligence (AI), a branch of computer science that emerged in the 1950s, aims to develop machines capable of simulating human intelligence [116]. Over time, its limitations spurred the growth of machine learning (ML), a subfield of AI that emphasizes statistical modeling and data-driven pattern recognition. ML has evolved significantly, gaining momentum with the advancement of neural networks since their inception [117], followed by the rise of deep learning (DL) and, more recently, generative AI (GenAI), fueled by increasing computational power and data availability. An overview of AI and ML history is provided in Appendix A. In addition, the terminologies used in this paper are defined in Appendix B. This section presents the lifecycle of typical ML models, their role in water and environmental modeling, and a recap of our three decades’ experience in ML model development and application in water and environmental modeling in California.

2.1. The Life Cycle of a Machine Learning Model

The life cycle of a typical ML model encompasses several key stages, including problem definition, data preparation, model development, and mode deployment (Figure 1). Data preparation can be divided into data collection and data pre-processing. Model development can be further divided into model selection, model training, and model evaluation. During the problem definition phase, the specific problem or task that the model aims to address is clearly defined, along with the objectives and desired outcomes. Following problem definition, the process moves to data collection, where relevant datasets are gathered from various sources, ensuring they are comprehensive and representative of the problem domain. Subsequently, data pre-processing is carried out to clean, transform, and prepare the data for analysis, addressing issues such as missing values, outliers, and noise. It is important to note that feature engineering, which involves extracting and transforming features from raw data for use in ML models, is typically considered part of the ML model life cycle [118]. However, for simplicity, this paper broadly categorizes it under data pre-processing.

Once the data are pre-processed, the next step is model selection, where the most appropriate ML algorithms are chosen based on the problem type, data characteristics, and desired outcomes. With the data pre-processed and the models selected, the models are trained and tested, through which they learn patterns and relationships between features (model input) and the label which is the target variable or the output that an ML model is trained to predict, referring to the definitions in Appendix B. Specifically, datasets are typically divided into training, validation, and test sets to ensure that a model generalizes well to unseen data. The training set is used to learn patterns, the validation set helps tune hyperparameters (configurable variables set before training that influence the model’s learning process but are not learned from the data) and prevent overfitting, and the test set evaluates final model performance. When datasets are small, cross-validation—where data are repeatedly split into training and validation sets—offers a more reliable estimate of model performance by reducing dependence on a single split. A key challenge in this process is overfitting, where a model learns noise or specific patterns in the training data rather than general trends, leading to poor performance on new data. Out-of-sample testing, such as evaluation on a separate test set, helps detect overfitting by revealing discrepancies between training and real-world performance. To mitigate overfitting, regularization techniques such as L1/L2 penalties [119,120], dropout in neural networks, pruning in Decision Trees, and data augmentation are commonly used to ensure the model captures meaningful patterns rather than memorizing the training data. Model performance can be evaluated via statistical metrics and/or visual inspection. Based on the evaluation results, adjustments may be made to the model or its parameters to optimize performance further. Finally, upon satisfactory evaluation outcomes, the model is deployed into production, making it available for real-world use, where it can provide insights, make predictions, or automate decision-making processes, thus completing the life cycle of an ML model. Throughout this cycle, iteration and continuous improvement are key, as models may require updating or retraining to maintain relevance and effectiveness over time. Further elaboration on each of these steps will be provided in later sections.

2.2. Roles of Machine Learning Models in Water and Environmental Modeling

The complexities of water and environmental systems along with a changing climate pose challenges for traditional modeling approaches. ML models have emerged as powerful tools, offering unique capabilities that complement and enhance existing methods (e.g., [121,122,123,124]). A number of roles that ML models play in water and environmental modeling are described as follows.

(1)

Enhanced simulation and prediction

ML models excel at identifying patterns and relationships within datasets, often improving prediction accuracy compared with traditional methods, even with a small-to-moderate number of data. However, DL models typically require very large datasets to achieve high performance. These capabilities enable ML to support accurate simulations and predictions of future environmental conditions. This includes tasks like the following:

(a) Hydrological forecasting: Forecasting precipitation (e.g., [55,56,58,125,126]), reservoir inflows (e.g., [40,127]) and outflows [43], water level (e.g., [128,129]), flood events and flood risks (e.g., [130,131,132,133,134]), flood inundation area (e.g., [135]), seasonal water supply forecast (WSF) [34,35,36,136], and drought events (e.g., [137]) with improved accuracy and lead time.

One prime example is the multi-model machine learning metasystem (M⁴), developed for the U.S. Department of Agriculture’s Natural Resources Conservation Service (NRCS) [34]. M⁴ enhances the traditional principal components regression approach used by the NRCS by incorporating supervised and unsupervised machine learning techniques, ensemble learning, and evolutionary optimization. A follow-up study [35] validated M⁴ by testing it across 20 hindcasting cases at 11 locations with diverse hydro-climatic conditions, including the Truckee River at Farad, California. The results demonstrated improved WSF accuracy, enhanced geophysical explainability, and strong operational feasibility, supporting M⁴’s adoption for large-scale water resource management applications.

Building upon the above work, Fleming et al. [36] investigated the integration of next-generation satellite remote sensing-based snow data into WSF models in coordination with the U.S National Aeronautics and Space Administration’s Jet Propulsion Laboratory in Pasadena, California. Specifically, the study assessed the spatially and temporally complete (STC) MODSCAG fractional snow-covered area (fSCA) product as a complement to traditional in situ snow data. Testing was conducted in four diverse watersheds, including one in California, using both conventional NRCS models and the M⁴ framework. Results indicated that incorporating fSCA data improved forecast accuracy, particularly for short-lead, late-season predictions when snowlines retreat above ground-based measurement sites. In addition, the ML-based M4 model is better equipped to leverage satellite data compared with traditional linear statistical models. The study further highlighted the operational advantages of integrating satellite-derived snow data into M⁴, with significant implications for enhancing water resource management in snow-dependent regions, including key headwater watersheds in California.

(b) Water quality modeling: Forecasting the spatial and temporal variations in water quality variables [138,139,140], including salinity (e.g., [66,67]), water temperature (e.g., [65,85,141]), ion concentrations (e.g., [75,76]), sediment (e.g., [73]), algal bloom [142,143], and dissolved oxygen [66], among others; identifying potential contamination sources; and assessing the effectiveness of mitigation strategies. To provide an example, Rath et al. [82] enhanced an empirical salinity model for the San Francisco Bay–Delta estuary by integrating a Bayesian ANN and new inputs. The hybrid model improved performance and characterized prediction uncertainty. This work demonstrated a practical method for integrating existing models with Bayesian ANNs, constrained by system behavior insights, for broad environmental modeling applications.

(c) Climate change analysis: Understanding the complex dynamics of climate systems and predicting future trends in hydro-climatic patterns (e.g., [144,145,146,147,148]). For instance, Wi et al. [42] evaluated the physical realism of ML models for projecting streamflow under climate change. By comparing a long short-term memory (LSTM) model to process-based models across California watersheds, the study found that while LSTM can capture hydrologic responses, these models may produce unrealistic runoff projections in warming scenarios. Incorporating process model outputs as additional inputs improves realism, but effectiveness depends on the process model used. The findings revealed both the potential and limitations of ML in hydrologic forecasting under a changing climate.

(2)

Uncovering hidden insights

ML algorithms can analyze a vast number of water and environmental data, including satellite imagery, sensor readings, and historical records. This ability to process complex data allows them to uncover hidden patterns and relationships that might be missed or even misrepresented by traditional methods. These insights can be crucial to the following:

(a) Identifying emerging environmental threats: Early forecasting and detection of flood or drought (e.g., snow drought) events and changes in water quality (e.g., [149,150,151]). To illustrate, Tanim et al. [107] developed an ML-based flood detection framework that integrated satellite imagery and change detection methods to identify urban flooding in San Diego, California. By combining Sentinel-1 satellite data with supervised and unsupervised classification techniques, the study effectively mapped flooded areas with high accuracy. The use of unsupervised methods enabled rapid flood identification with minimal data requirements. This approach enhanced early forecasting and detection capabilities, enabling city planners and emergency responders to mitigate flood risks and improve urban resilience to extreme weather events.

(b) Developing targeted conservation strategies: Identifying crucial habitats (e.g., [152]), predicting species (e.g., fish, birds, etc.) distribution (e.g., [153]), and informing conservation efforts (e.g., [154,155]). For example, Olivetti et al. [92] integrated computational fluid dynamics and ML to analyze how migratory Chinook salmon navigates complex flow environments. By decomposing movement into passive drift and active locomotion, the study improved predictions of migration trajectories and energy expenditure. The findings offered valuable insights for conservation strategies, enabling better habitat management, informed water flow regulation, and improved infrastructure design to support migratory species under changing environmental conditions.

(c) Optimizing resource planning and management: Understanding the factors influencing water availability and demand, system vulnerability, leading to more sustainable water use practices and water resource planning (e.g., [106,156]). As an example, Robinson et al. [106] developed an ML-based early warning system to detect long-term water supply vulnerabilities under climate change. Using historical and projected hydro-climatic data from Northern California’s reservoir system, the study applied classification models to predict future supply risks at lead times of up to 20 years. Results indicated that ML classifiers outperformed the benchmarks, with predictive accuracy declining at longer lead times. The findings underscored the potential for data-driven early warning signals to inform adaptive water resource planning.

(3)

Efficiency and scalability

ML models offer significant advantages in terms of efficiency and scalability. They can process large datasets rapidly, enabling model users to conduct simulations and explore various scenarios in a shorter timeframe. This allows for the following:

(a) Real-time operations and decision making: Providing near-real-time insights into water and environmental conditions, facilitating proactive responses to emerging issues (e.g., [157,158]). By way of example, Tillotson et al. [96] developed an ML-based forecasting tool to reduce the risk of entrainment loss for endangered salmonids at large-scale water diversions in California’s Sacramento–San Joaquin Delta. By using a quantile regression forest approach trained on historical data, the model predicted weekly entrainment risk. The study also developed an interactive web-based application that allowed for real-time operational adjustments, enhancing decision making for water management while balancing ecological conservation with water supply needs.

(b) Simulating large-scale water and environmental processes: Modeling complex interactions between different components of the environment, such as the impact of climate change on water resources or the spatial and temporal variations in critical water quantity or water quality variables (e.g., [159,160]). For instance, Kratzert et al. [38] applied ML, specifically LSTM networks, to improve regional rainfall–runoff modeling across 531 basins, including 29 in California. By leveraging large-sample datasets, the LSTM models effectively captured hydrologic variability and outperformed traditional models in streamflow prediction. The study advanced large-scale water and environmental modeling by demonstrating how ML can integrate meteorological, hydrological, and physical basin characteristics to improve regional hydrologic simulations.

(4)

Integration with traditional models

ML models are not necessarily intended to replace traditional models used in water and environmental modeling. Instead, they can serve as a complementary approach or, in some cases, provide an alternative when they offer improved performance. They can also be integrated with existing models for the following aims:

(a) Improve the accuracy, speed, and efficiency of simulations (e.g., [70,80,83]). For example, Qi et al. [80] developed an ML-based dashboard to mimic the DSM2 model for salinity simulation at critical locations in the Delta. This dashboard operates notably quicker than DSM2, and it is capable of running on a mobile phone with internet connectivity. The outcomes can be swiftly generated and displayed within seconds.

(b) Provide a deeper understanding of the complex interactions within environmental systems (e.g., [161,162,163,164]). For instance, Ransom et al. [81] developed a hybrid ML approach to predict and visualize nitrate contamination in the Central Valley aquifer, California, by integrating traditional physically based models with Boosted Regression Tree. By incorporating outputs from existing models, the ML approach provided deeper insights into the factors influencing nitrate concentrations in groundwater. The study highlighted the significance of redox conditions, nitrogen flux, and groundwater age in shaping contamination patterns. This integration of ML with traditional modeling frameworks improved predictive accuracy and supported more effective groundwater management strategies.

2.3. Machine Learning Experience of the California Department of Water Resources

The California Department of Water Resources (DWR) has been developing and applying ML tools for water and environmental modeling since the early 1990s. Initial efforts primarily focused on salinity modeling [165], as the salinity level is a critical factor influencing water operations in the Delta, the State’s central water hub. Recent work has expanded to include the modeling of specific ion constituents, including dissolved chloride, bromide, and sulfate [75]. Additionally, ML models have been applied to simulate other water quality variables, including water temperature [166], suspended sediment [73], trihalomethane [167], and flow-related parameters such as carriage water [168] and streamflow [169]. These applications are illustrated in Figure 2 and summarized as follows.

2.3.1. Applications in Modeling Salinity

The Delta plays a vital role in California’s water supply system, necessitating effective management of ocean-driven salinity intrusion to ensure water quality for municipal, agricultural, and environmental uses. Early applications of ML for salinity modeling in the Delta involved the development of separate Multi-Layer Perceptron (MLP) models—a type of Artificial Neural Network (ANN)—for each salinity monitoring location [96]. These models, with one or two hidden layers, demonstrated superior performance compared with traditional empirical methods. Subsequent enhancements focused on (a) improving model robustness by training them in diverse hydrological and operational scenarios [170], (b) optimizing input feature selection and training strategies [170,171,172], and (c) simplifying integration into operational planning tools [71,170,173,174].

A major advancement came with the adoption of multi-task learning (MTL), allowing a single model to predict salinity at multiple locations simultaneously [77,78,79]. The advent of deep learning further revolutionized salinity modeling with architectures such as Convolutional Neural Networks (CNNs) and Long-Short-Term Memory (LSTM) networks [70]. CNNs effectively captured peak salinity events, while LSTM excelled at managing long-term dependencies in time-series data [70]. Further refinements led to innovative architectures, including Residual Networks (ResNets), Res-Gated Recurrent Units (Res-GRUs), and Res-LSTM models [78,79]. To overcome data limitations and improve generalizability, data augmentation was employed to generate diverse datasets representing a wider range of hydro-climatic and operational conditions [80]. Transfer learning was also implemented, enabling models trained on simulated datasets to predict real-world salinity observations [80].

These advancements were complemented by the development of an interactive dashboard, designed to support decision making [80]. This dashboard offers an intuitive interface for water managers to visualize salinity levels and assess compliance in various planning scenarios, reflecting different operational strategies or hydro-climatic conditions.

2.3.2. Applications in Modeling Ion Constituents

Ion constituents are critical indicators of water quality, determining its suitability for various uses. Traditionally, regression models estimate ion constituent concentrations based on salinity, represented by electrical conductivity (EC). However, these models often struggle with ion constituents exhibiting strong non-linear relationships with EC. To address this, the DWR developed ML models as alternatives.

The first study [75] focused on simulating 12 ion constituents in the South Delta: total dissolved solids, dissolved chloride, dissolved sulfate, dissolved sodium, dissolved calcium, dissolved magnesium, dissolved nitrate, dissolved potassium, dissolved bromide, dissolved boron, alkalinity, and hardness. By using data from seven stations, the study compared different ML models, including Regression Tree (RT), Random Forest (RF), and ANNs. Random Forest generally outperformed the other models in the study.

Building on this foundation, the second study [76] expanded the scope to 30 stations across the Delta, incorporating additional predictors beyond EC, such as the Sacramento River X2 position, month of the year, and water year type, to improve prediction accuracy. This study evaluated RT, RF, Gradient Boosting, and ANNs, finding that ANNs performed the best overall. Additionally, an interactive, web-based dashboard powered by the ANN model was developed and hosted on Microsoft Azure. This dashboard allows users to modify input parameters and visualize resulting ion concentration predictions, providing a valuable tool for data-driven water quality management in the Delta. This effort will be discussed in further detail in Section 3.

2.3.3. Applications in Modeling Other Variables

In addition to salinity and ion constituents, the DWR has developed ML models to simulate minimum Delta outflow (MDO) and marginal export cost (MEC) [168,169,175,176,177]. MDO represents the freshwater flow required to maintain salinity standards at specific compliance locations in the Delta, particularly during increased water exports. MEC quantifies the additional flow needed to offset salinity increases caused by a unit of export. Traditional methods often oversimplified Delta hydrodynamics, struggling to capture the complex interplay of flows, gate operations, and salinity. The DWR’s ANN models address this by leveraging multiple inputs, including river inflows, export pumping rates, Delta Cross-Channel gate positions, and tidal influences, to predict salinity more accurately. These ANN-based approaches have been integrated into operational planning tools like DWRSIM and CALSIM [169,173,174], offering a faster and more precise alternative to traditional methods for simulating MDO and MEC.

ML models have also been developed to emulate traditional log-linear models for simulating trihalomethane (THM) formation in the Delta [167]. These models use inputs such as bromide concentration, dissolved organic carbon (DOC), UV-254, chlorine dose, ammonia, reaction time, temperature, and pH. Validation demonstrated their superior accuracy compared with traditional approaches.

Moreover, the DWR developed LSTM models to estimate suspended sediment concentration (SSC) at key locations in the Delta [73]. The LSTM models demonstrated better or comparable performance compared to the traditional process-based DSM2 model, particularly in capturing temporal variability and peak magnitudes in SSC. The models utilized flow, stage, and SSC data for training and testing, with additional analysis incorporating precipitation and wind speed. The study highlighted the potential for ML to complement existing process-based methods, offering faster and more flexible solutions for sediment management in complex estuarine systems like the Delta.

Additionally, ML models have been developed to estimate water temperature at boundary locations in the Delta using air temperature and flow as input variables [166]. In particular, MLPs with a single hidden layer were employed for this purpose. These models demonstrated a high level of accuracy in predicting water temperatures, effectively capturing the relationships between the input parameters and the observed outcomes.

Furthermore, a set of DL models were developed to forecast monthly reference evapotranspiration at 55 weather stations in California, some of which are located in the Delta. The work was built upon the studies [53,54] but extended beyond them in terms of (a) exploring the improvement of DL models in global versus local learning; and (b) automating hyperparameter optimization to achieve the most desirable forecasting accuracy.

ML models have also been applied to assess the risks of harmful algal blooms (HABs) in the Delta [178]. Specifically, five models were developed to simulate HAB risk levels (low or high) based on nine environmental variables across 16 locations. Among these models, the Random Forest model demonstrated the best performance, achieving the highest accuracy, precision, and recall.

The protocols presented in this paper are primarily based on our three decades of experience in this field, as mentioned above. However, we acknowledge that numerous other machine learning research efforts and applications in water and environmental modeling have been conducted in California outside of the DWR. Since the goal of this paper is not to provide an exhaustive review of all ML studies in the State, these studies are instead compiled and presented in Appendix C.

3. Machine Learning Protocols and Case Study

Drawing on the DWR’s extensive experience outlined in Section 2.3, this section provides an overview of the key principles and guidelines for the four main steps of ML model development and application: problem definition, data preparation, model development, and model deployment. A case study on using ML to model ion constituents is also included to illustrate these steps in practice.

3.1. Problem Definition

3.1.1. Problem Definition Protocols

Problem definition is the cornerstone of the successful application of ML in all fields, including water and environmental modeling. It acts as a compass, guiding the entire process and ensuring efficient resource allocation, optimal model selection, and ultimately, the development of appropriate tools for addressing critical water and environmental challenges. The typical protocols to craft a clear and well-defined problem statement involves the following:

(1)

Understand the Context

(a)

Identify the problem: Begin by clearly identifying the specific water-related or environmental issues to be addressed. Is it water data anomaly detection, water quality deterioration, flood, drought, and water supply forecasting, or others?

(b)

Stakeholder needs: Consider the needs and interests of stakeholders who might benefit from the model’s insights, such as policymakers, water managers, or the public.

(2)

Define the Scope

(a)

Specificity: Avoid broad and ambiguous statements like “improve water quality understanding”. Instead, focus on specific and measurable objectives.

(b)

Temporal and spatial considerations: Specify the desired prediction timeframe (e.g., hourly, daily, seasonal, or annual) and the spatial scope (e.g., a specific river, channel, an entire watershed, or groundwater basin).

(3)

Define the Desirable Outcome

(a)

Measurable goals: State what you want the model to achieve in a quantifiable manner. This could involve predicting specific water quality parameters, identifying areas of water quality degradation, or forecasting the probability of specific events.

(b)

Performance metrics: Define the metrics you will use to evaluate the model’s success in achieving the desired outcome. This could involve metrics including but not limited to accuracy, precision, and recall for classification tasks or bias, Mean Absolute Error, and the coefficient of determination for regression tasks.

(4)

Refine and Iterate

(a)

Seek feedback: Share the draft problem statement with colleagues or experts in the field and stakeholders to gather feedback and ensure clarity and feasibility.

(b)

Iterative process: Be prepared to make adjustments based on feedback and emerging information throughout the project.

3.1.2. Problem Definition Example

The composition of water in the Delta is significantly influenced by various salts and minerals, such as chloride, sulfate, sodium, magnesium, and potassium. Collectively identified as ions, these salts and minerals are pivotal to maintaining the ecological balance and agricultural viability and ensuring the safety of drinking water in the Delta.

The traditional approach to measuring these ion constituents involves infrequent (e.g., roughly monthly) on-site sampling (i.e., grab samples) at limited locations followed by laboratory analysis. This method, while accurate, is time-consuming and expensive and provides results that are not immediately available and lack spatial comprehensiveness. The necessity for an advanced approach is underscored by the challenges inherent in sampling methods, prompting the exploration of more efficient alternatives. Measurements of salinity (represented by EC) show great promise because they correlate with ions and are easily accessible from automatic sensors installed at key locations throughout the Delta. Historically, the DWR has utilized parametric regression equations for converting EC into ion levels [179]. However, the performance of these conversion equations is limited for ions having strong non-linear relationships with EC.

The objective of this case study is two-fold. Firstly, it seeks to create ML models that can mimic and possibly enhance the current regression equations for EC–ion conversion in terms of both goodness-of-fit metrics and visual inspection. Secondly, it provides a user-friendly interactive dashboard based on these ML models. Through its interactive features, the dashboard allows stakeholders, water managers, and the public to easily modify input parameters and quickly visualize ion concentrations for various input scenarios, without needing any programming skills.

3.2. Data Preparation

3.2.1. Data Preparation Protocols

Effective data preparation, including data collection and pre-processing, is indispensable to advancing ML model development in water and environmental modeling. By preparing diverse, high-quality datasets and adhering to best practices for data preparation, developers can build robust ML models that provide valuable insights into water and environmental processes and thus inform decision making. To ensure the quality and reliability of the collected data, it is essential to adhere to best practices throughout the data preparation process. The following outlines some typical practices:

(1)

Use of standardized protocols: It is necessary to adhere to standardized protocols and methodologies for data collection, measurement, model simulation, and sampling to ensure consistency and comparability across datasets.

(2)

Implementation of quality assurance/quality control (QA/QC) measures: It is necessary to implement rigorous QA/QC procedures to identify and rectify errors, outliers, and inconsistencies in the collected data, including the calibration of instruments, duplicate measurements, and cross-validation with reference data.

(3)

Selection of representative data: It is necessary to choose sampling locations that are representative of the spatial and temporal variability of the water and environmental processes under study, taking into account factors such as study purpose, study period, and data availability, among others.

(4)

Data accessibility and documentation: It is necessary to document metadata, data sources, and collection methods to facilitate data sharing, reproducibility, and transparency.

(5)

Data pre-processing: This process involves examining raw input data and transforming them—either directly or through feature extraction—into a form that ML models can effectively utilize. Typical techniques for pre-processing include normalization and standardization. Normalization scales the data to a range, typically [0, 1], by adjusting values proportionally based on the minimum and maximum of each feature. This is particularly useful when features have varying scales. Standardization transforms data to have a mean of 0 and a standard deviation of 1, making them suitable for ML algorithms that assume normally distributed features.

(6)

Data split: Once the data are prepared, they are typically divided into separate subsets. A common approach involves a three-way split into training, validation, and test sets, where the validation set is used for tuning hyperparameters and preventing overfitting. In some cases, cross-validation is employed instead of a fixed validation set to ensure robust model evaluation. These strategies help optimize the model’s parameters and assess its performance on unseen data.

3.2.2. Data Preparation Example

This case study combines data from three sources, spanning measurements from 1959 to 2022 across 30 monitoring locations in the Delta. Figure 3 shows these monitoring locations, which are grouped into three sub-regions based on their distinct hydrological and environmental characteristics: the Old-Middle River (OMR), the San Joaquin River Corridor, and the South Delta. This regional classification captures the varying water sources and quality impacts throughout the Delta.

The primary dataset originates from historical records utilized by [180] which have been quality-controlled and well documented, containing ion grab samples, EC, and X2 position data collected between 1959 and 2018 at 19 stations within the study area. The second dataset includes samples collected by the DWR between 2018 and 2020, focusing on seven stations in the South Delta. Finally, the third dataset, downloaded from the California Data Exchange Center (CDEC) website (https://cdec.water.ca.gov/, accessed on 1 March 2024), extends the geographical and temporal coverage by including samples from 13 stations within the interior Delta, collected from 2018 to 2022.

The integration of these datasets provides a comprehensive view of the water quality dynamics within the Delta, capturing a broad spectrum of hydrologic conditions and environmental changes over time. This approach ensures a robust analytical foundation, enabling the development of models to accurately simulate ion levels under a variety of conditions.

Table 1. Overview of study ion samples *.

Ion	Sample Size	Data Range	SD	Period	Units
TDSs	1466	49–2120	204	1968–2022	mg/L
Mg2+	1336	2–102	8.6	1959–2022	mg/L
Na+	1575	6–343	44	1959–2022	mg/L
Ca2+	1335	5.8–244	18	1959–2022	mg/L
Cl−	1972	4–775	77	1959–2022	mg/L
SO42−	1066	5–350	46.5	1959–2022	mg/L
Br−	1239	0.01–2.3	0.22	1990–2022	mg/L
Alkalinity	1036	26–198	27.6	1959–2020	mg/L as CaCO3
K+	1148	0.87–11	1.35	1959–2022	mg/L

* Adapted from [76].

presents a detailed overview of the ion constituents analyzed in this study, highlighting the sample size, range of data, and the standard deviation (SD) of the data for each ion constituent. This table illustrates the dataset’s composition and variability, offering insights into the distribution and concentration of ions across the Delta. The nine ions presented in the table each play a unique role in influencing the water quality and ecological health of the Delta.

The data pre-processing stage is crucial to preparing the dataset for ML analysis, ensuring that the input data are clean, normalized, and suitable for the ML algorithms to process efficiently. The following outlines the key pre-processing steps undertaken, including visualization, normalization, and the treatment of categorical data.

Visualization plays a pivotal role in understanding the underlying patterns and relationships within the dataset. Figure 4 illustrates the relationships between EC and the ion constituents across the three sub-regions of the Delta. These visual representations help identify the linear, bifurcated, and non-linear relationships between EC and ion levels, facilitating the categorization of ions into three distinct groups. This indicates that the relationship between EC and ion concentrations can vary significantly depending on the ion type and environmental factors. The groups are defined as follows:

Group 1: Ions with a linear relationship to EC, indicating a direct and proportional increase in ion concentration with EC. These ions comprise magnesium (Mg²⁺) and total dissolved solids (TDSs)).

Group 2: Ions exhibiting bifurcation or branching patterns in response to EC, suggesting varying ion concentration pathways under different conditions. These ions feature sodium (Na⁺), chloride (Cl⁻), and calcium (Ca²⁺).

Group 3: Ions with non-linear relationships to EC, reflecting complex dynamics that cannot be captured by a straightforward linear model. These ions consist of sulfate (SO₄²⁻), alkalinity, potassium (K⁺), and bromide (Br⁻).

The division into these groups allows for a more nuanced analysis, acknowledging the diverse behaviors of ion constituents in the Delta’s waters and ensuring that the ML models are optimally configured to reflect these complexities.

To accommodate the diverse range observed across the dataset, particularly in the numerical predictors of EC and X2 position, a normalization process is helpful. This normalization adjusts the dataset to a uniform scale, specifically between 0 and 1, for these predictors. The procedure puts all features on a comparable scale, crucial to preventing biases from differences in measurement units or ranges. High-value ions, or any disproportionately large measurements, could otherwise skew the ML models’ performance.

The incorporation of categorical variables, such as Water Year Type (WYT), month, and sub-region, requires appropriate encoding techniques to ensure efficient use by ML algorithms. A commonly used method is one-hot encoding [181], which represents each category as a unique binary vector. However, depending on the specific modeling requirements and dataset characteristics, alternative techniques, such as multi-hot encoding [182], may also be employed to capture additional categorical relationships more effectively. For instance, WYT, denoting the hydrological condition of a year with classifications ranging from “wet” to “critical”, required conversion into a computationally understandable format. To achieve this, one-hot encoding was utilized, transforming these categorical variables into binary columns [33,38]. With five water year types, we created four binary columns, where the fifth category (critical) is implicitly represented when all other columns are zero. Each column represents a potential category value, converting qualitative attributes into a quantitative framework suitable for analysis. For example, a “‘wet” year is encoded as [1, 0, 0, 0], a “dry” year as [0, 0, 0, 1], and a “critical” year as [0, 0, 0, 0]. This method allows for the seamless integration of essential hydrological, temporal, and spatial data into the models without compromising the numerical requirements of the ML algorithms.

3.3. Model Development

3.3.1. Model Development Protocols

Selecting the appropriate ML architecture is crucial to developing effective models in water and environmental modeling. Based on the DWR’s previous ML model development experience, we outline several best practices for ML architecture selection, ranging from simple models to ensemble and hybrid methods, tailored to the unique challenges and requirements of environmental applications.

(1)

Start Simple

It is recommended to begin with simple ML models, more interpretable models, such as basic versions of Multi-Layer Perceptron (MLP) models [183] or Decision Trees [184], to establish a baseline understanding of the data and the problem at hand. These models can help identify key relationships between input variables and the target variable while providing insights into feature importance.

(2)

Ensemble Methods

Ensemble methods can be considered to improve model performance and robustness. These techniques include Random Forest [185] and Gradient Boosting [186], which combine multiple base models to create a more generalized and accurate learner. Additionally, techniques like bootstrap aggregation (bagging) can be applied to MLPs and other models to enhance stability and reduce overfitting.

(3)

Hybrid Methods

Hybrid methods that integrate multiple ML architectures or combine ML with traditional modeling approaches should be explored [83,187,188,189]. Hybrid methods leverage the strengths of different techniques to address specific challenges in water and environmental modeling. For example, combining physics-based models with data-driven ML models can improve predictive accuracy while retaining interpretability and physical realism.

(4)

Consider Spatial and Temporal Dynamics

The spatial and temporal dynamics inherent in environmental data should be accounted for when selecting ML architectures. Convolutional Neural Networks (CNNs) [190] are well suited for spatial data analysis, capturing spatial patterns and relationships. Recurrent Neural Networks (RNNs) and long short-term memory (LSTM) networks [191] excel at modeling temporal dependencies, making them suitable for time-series data in water and environmental modeling.

(5)

Incorporate Domain Knowledge

Domain knowledge and expert insights should be integrated into ML architecture selection to ensure that the models capture relevant environmental processes accurately. Domain-specific constraints, physical laws, and expert understanding can guide the choice of architectures and enhance the relevance and applicability of the models.

Training and evaluating ML models in water and environmental modeling necessitates meticulous attention to multiple factors to guarantee their resilience, dependability, and precision. Drawing from the DWR’s prior experience, we deem the subsequent practices indispensable:

(1)

Sensitivity Analysis

Sensitivity analysis is a critical step in water and environmental ML modeling to identify the influence of individual input variables on model predictions. By systematically varying input parameters, the features that have the most significant impact on outcomes can be identified. This process helps prioritize data collection efforts, improve model robustness, and ensure interpretability in applications such as streamflow prediction, water quality assessment, and ecosystem modeling. Sensitivity analysis also aids in detecting potential overfitting to less important features, fostering more reliable and generalizable ML models.

(2)

Overfitting Prevention

Overfitting occurs when an ML model learns patterns specific to the training data, including noise or irrelevant details, leading to poor generalization to new, unseen data. Regularization methods, such as L1 and L2 regularization, dropout, pruning, data augmentation, bagging, and early stopping [119,120,192], help prevent overfitting by penalizing overly complex models. In water and environmental modeling, where datasets may be limited or noisy, regularization techniques ensure that models generalize well to unseen data. Regularization should be implemented to balance model complexity and performance, enhancing the model’s reliability and interpretability.

(3)

Ensemble Learning

Ensemble learning techniques combine multiple models to improve predictive performance and robustness. In water and environmental modeling, where uncertainties and complexities are prevalent, ensemble methods offer enhanced predictive capabilities. Ensemble techniques like bagging [192], boosting [193], or stacking [194] are utilized to leverage the diversity of individual models and achieve more accurate predictions across different environmental scenarios.

An emerging approach is the use of higher-level ensembles, where structurally diverse models are combined to improve predictive accuracy and robustness. One example is the previously discussed M⁴ metasystem [34,35,36]. Another example is the ensemble modeling framework introduced by Umirbekov et al. [195] for seasonal streamflow forecasting. Their approach integrates generalized linear models, Gaussian processes, support vector regression (SVR), and Random Forest. The predictions from these diverse models are then aggregated using an additional SVR model as a metalearner, enabling the ensemble to leverage the strengths of each modeling approach. By increasing structural variability among ensemble members, this method enhances stability and reduces the risk of overfitting to any single modeling paradigm.

(4)

Cross-Validation

Cross-validation is a valuable technique for assessing model performance and generalizability, especially when datasets are too limited to support traditional training–validation–testing splits. This is particularly relevant in water and environmental modeling, where data may be limited and often exhibit temporal or spatial dependencies. Methods such as k-fold cross-validation or stratified cross-validation [196] can be used to partition the data, providing an alternative form of out-of-sample testing. By leveraging cross-validation, ML models can be evaluated more robustly, helping to mitigate overfitting and provide reliable performance estimates across diverse environmental conditions.

(5)

Hyperparameter Optimization

Hyperparameter optimization [197] involves systematically tuning model hyperparameters to maximize performance while avoiding overfitting or underfitting. In water and environmental modeling, where model complexity may vary based on the intricacies of environmental processes, hyperparameter optimization is crucial to fine-tuning model performance. Techniques such as grid search, randomized search, gradient-based optimization, Bayesian optimization, evolution strategies, and hyperband [197,198,199,200,201,202,203,204,205] can be utilized to identify the optimal hyperparameter settings for ML models.

These best practices aid in streamlining the development and application of ML models for water and environmental modeling, ensuring their consistency, interoperability, and applicability in addressing the State’s water and environmental challenges.

3.3.2. Model Development Example

(1)

Model Selection

The primary objective of this case study was to predict ion concentrations in the Delta, a problem that inherently falls within the domain of regression analysis in ML. Regression problems require models that can predict continuous outcomes, such as ion concentrations, based on one or more predictor variables, such as EC and X2 position. Given this context, the model selection process was guided by the necessity to choose ML models adept at solving regression problems and capable of handling the complexity and non-linearity of environmental data.

Moreover, it is important to note that our dataset, derived from grab samples, does not constitute a time series. Grab samples provide a snapshot of water quality at specific points in time and locations but lack the sequential, time-dependent structure that characterizes time-series data. As a result, models that are good at time-series forecasting, such as LSTM networks, are not the most appropriate models for our dataset. LSTM and similar time-series models excel at capturing temporal dependencies and patterns over time, which our grab sample dataset does not present.

Given these considerations, the following ML models were selected for their proven effectiveness in regression problems and their suitability for the dataset’s characteristics.

Decision Trees: Decision Trees offer a straightforward, interpretable approach to regression, making them a useful starting point for understanding the relationships within our data [184,206]. They allow us to visualize how decisions are made based on the input features, although they may not capture complex relationships as effectively as more sophisticated models.

Random Forest (RF): As an ensemble of Decision Trees, Random Forest mitigate some of the overfitting issues associated with individual trees and are more capable of handling complex, non-linear relationships in regression problems [185]. Their robustness and ability to deal with high-dimensional data make them well suited for predicting ion concentrations.

Gradient-Boosting Machines (GBMs): GBMs build models sequentially to correct the residuals of prior models, gradually improving prediction accuracy [193]. This technique is highly effective for regression, especially when dealing with non-linearity and interactions between predictors, as is common in environmental datasets.

Artificial Neural Networks (ANN): The Artificial Neural Network (ANN) is capable of capturing complex and non-linear relationships within datasets [183]. At its core, a typical ANN consists of interconnected units called neurons, which are organized into layers: an input layer, several hidden layers, and an output layer. Neurons in these layers interact through connections that are weighted, with the network’s learning process revolving around adjusting these weights based on the input data.

Hidden layers are the ANN’s workhorse, allowing the model to learn features at different levels of abstraction. The activation function within each neuron determines how signals are transformed as they pass through, introducing non-linearity into the model and enabling it to learn complex patterns. Common activation functions include ReLU, sigmoid, and tanh, each bringing different properties to the model’s learning capability. Additionally, the learning rate is a crucial parameter that governs the step size during the weight update process, influencing the model’s convergence speed and accuracy.

In this study, the ANN architecture incorporates four hidden layers. This setup aims to leverage the ANN’s general abilities to process and analyze the data, utilizing neurons, activation functions, and an optimized learning rate to enhance model performance. A comparative summary of the ML models used in this study is presented in

Table A2. Overview of key features of selected machine learning models in the case study *.

Feature/Model	Decision Trees	Random Forest	Gradient Boosting	Artificial Neural Networks
Model type	Tree-based	Ensemble	Ensemble	Neural network
Basic unit	Decision Trees	Decision Trees	Weak learners	Neurons
Hidden layers	N/A	N/A	N/A	One or more
Loss function	Gini/Entropy	Gini/Entropy	Various	MSE, Cross-Entropy, etc.
Learning algorithm	ID3, CART, etc.	Bagging	Boosting	Gradient Descent, Adam, etc.
Regularization	Pruning	Voting/averaging	Shrinkage	Dropout, Weight Decay, etc.
Scalability	Moderate	High	Moderate to high	High
Robustness	Moderate	High	High	Varies
Interpretability	High	Moderate	Low	Low
Speed/efficiency (training)	Fast	Moderate	Moderate	Varies
Speed/efficiency (inference)	Fast	Fast	Fast	Fast
Applications	Classification and regression	Classification, regression, and anomaly detection	Classification, regression, and ranking	Classification, regression, NLP, and image processing

* Adapted from [76].

in Appendix D.

(2)

Model Training

In this case study, the input–output datasets are randomly divided, allocating 80% for training purposes and the remaining 20% for evaluation. This partitioning facilitates the assessment of each ML model’s predictive accuracy and generalization capability, providing a clear indication of performance on unseen data.

To prevent overfitting—a scenario where a model learns the training data too well, compromising its performance on new data—an early-stopping function is incorporated into the training process, with a threshold set to 50 epochs. Early stopping acts as a form of regularization, effectively halting the training when no improvement in the model’s performance on a validation set is observed for a consecutive number of epochs specified by the threshold. This technique ensures that the model maintains its ability to generalize from the training data to unseen data, thus safeguarding against overfitting.

The evaluation and comparison of the ML models are grounded in their performance metrics on the evaluation dataset. The models are assessed based on their R-squared (R²) and Mean Absolute Error (MAE), with a close examination of the convergence between their performance on the training dataset to further guard against overfitting. Such scrutiny ensures that the selected models not only exhibit strong predictive accuracy but also demonstrate robust generalization capabilities across different data samples.

This structured approach to data division, model evaluation, and the strategic incorporation of early stopping underscores the meticulousness of the study’s methodology. It aims to ensure the development of reliable and generalizable ML models for predicting ion concentrations in the Delta.

Among the ML models evaluated, the ANN model demonstrated superior performance in predicting ion concentrations, particularly for ions exhibiting non-linear relationships with EC. This finding led to the selection of the ANN model for comparison against classical methods.

Once the model is selected, the next step is the optimization of the ANN hyperparameters, which is a computationally demanding task, particularly when these ANN models incorporate multiple hidden layers. This study’s ANN models, which include four hidden layers, required a robust hyperparameter search to fine-tune each model’s settings for predicting various ion concentrations accurately.

Hyperparameter search is a pivotal step in ML that involves experimenting with different combinations of model settings. These settings, or hyperparameters, such as the number of hidden layers, neurons in each layer, activation functions, and learning rate, are crucial to defining the model’s capacity to learn from data. The optimal hyperparameters are those that contribute to the lowest evaluation error, indicating the model’s effectiveness in making accurate predictions.

Given the complexity of searching hyperparameters for ANN models with four hidden layers, the computational workload exceeds the capabilities of standard desktop computers. To address this challenge, this study leveraged Microsoft Azure cloud computing resources, utilizing an environment equipped with 96 CPU cores. This powerful computing infrastructure significantly accelerated the hyperparameter search process, allowing for more extensive and rapid exploration of the hyperparameter space. This expeditious approach was instrumental in identifying the configurations that most effectively minimized the prediction error.

The detailed outcomes of the hyperparameter search, particularly the optimal number of neurons and activation functions chosen for each ion constituent model, are presented in

Table A3. Optimized hyperparameters for each ion constituent ANN model *.

Hidden Layer	TDSs		Mg2+		Na+
	N	Act	N	Act	N	Act
1	30	elu	30	relu	30	tanh
2	30	sigmoid	30	elu	30	elu
3	30	elu	30	tanh	30	sigmoid
4	30	relu	30	relu	30	elu
Hidden Layer	Ca2+		Cl−		SO42−
	N	Act	N	Act	N	Act
1	40	elu	30	relu	44	relu
2	40	sigmoid	30	elu	44	relu
3	40	relu	30	sigmoid	44	relu
4	30	tanh	30	elu	22	relu
Hidden Layer	Br−		Alkalinity		K+
	N	Act	N	Act	N	Act
1	44	elu	30	tanh	44	relu
2	44	sigmoid	30	relu	44	relu
3	30	elu	30	tanh	44	relu
4	30	tanh	30	elu	22	relu

* Adapted from [76].

in Appendix D.

(3)

Model Evaluation

(a) K-fold Cross-validation

The K-fold cross-validation analysis was conducted to assess and evaluate the generalization capability of the developed ANN models. Unlike a simple train–evaluation split, K-fold cross-validation systematically uses different portions of the data for training and evaluation. This approach helps mitigate the risk of overfitting and ensures that the model’s performance is not dependent on a particular division of the data. It is particularly valuable in scenarios where the available dataset is not extensive, maximizing the use of data for both training and evaluation.

In this case study, the dataset was divided into five equally sized subsets, or folds (K = 5). The model training and evaluation process was then repeated five times, with each fold used exactly once as the evaluation set, while the remaining four folds were used for training. This method allowed every data point in the dataset to be used for both training and evaluation, providing a thorough evaluation of the model’s predictive accuracy.

By using a fixed random seed for reproducibility, the subsets were created randomly to ensure that each fold was a representative sample of the whole dataset. This randomized division is crucial to minimizing bias and variance in the model evaluation process.

The K-fold cross-validation indicates consistent accuracy of the ANN models in predicting ion concentrations, as evidenced by the evaluation metric MAE calculated for each fold in comparison to the corresponding MAE values of the benchmark model (Figure A2 in Appendix D).

(b) Sensitivity Analysis

Sensitivity analysis is a critical component in model validation, particularly for systems as complex and variable as ion constituent levels in the Delta. Figure 5 and Figure 6 showcase the model’s predictions for bromide (Br) and chloride (Cl) concentrations, respectively, across three sub-regions, i.e., Old-Middle River (OMR), San Joaquin River Corridor, and South Delta, under specified conditions (April, Wet Year, and Sacramento X2 = 75 km). These figures highlight the models’ ability to adjust simulations in response to changing levels of EC.

Both figures depict a smooth gradient in the simulated ion concentrations as EC values vary, reflecting the models’ nuanced response to input changes. This is indicative of an effective balance between sensitivity and stability; the models are responsive to EC changes without displaying erratic behavior for minor fluctuations. Such stability in the presence of varied input conditions is essential to obtaining reliable predictions in environmental contexts, where data can naturally exhibit variability.

The sensitivity analysis as demonstrated in these figures informs the practical application of these ANN models. For ML model users, the consistency and smoothness in predictions signify that the models can serve as dependable tools for ion simulation. These characteristics ensure that the outputs are accurate and interpretable, enabling informed decision making. The models’ generalizability, indicated by their performance under specific test conditions, is crucial to applications across the diverse ecological landscape of the Delta.

(c) Performance Evaluation

The performance of the ANN models is quantitatively superior to that of traditional parametric regression models, particularly for Group 3 ion constituents, which exhibit more complex and non-linear relationships with EC (Figure 7). The improvements are evident in both R² values and MAE. For Group 1 ions, the improvement in R² is limited due to the strong linear relationship between EC and the ion constituents, although the improvement in MAE is notable. For both Group 2 and Group 3 ions, improvements in both metrics are evident. The most remarkable improvements are observed for dissolved sulfate, with an 85% improvement in R² and a 59% improvement in MAE. This indicates that the ANN model provides much better simulations for ions where the traditional regression model struggles to capture the strong non-linear relationships between the predictand and the predictors.

3.4. Model Deployment

3.4.1. Model Deployment Protocols

Deploying ML models effectively is imperative for their successful application in water and environmental modeling. By adhering to protocols and best practices such as comprehensive documentation, hands-on training, open-source collaboration, engagement with the user community, publication in peer-reviewed journals, and maintenance and improvement, model developers can ensure transparency, collaboration, and innovation. These practices facilitate informed decision making, enhance stakeholder engagement, and contribute to the sustainable management of water resources and environmental conservation efforts. The following delineates some of these practices we consider essential, drawn from the DWR’s past experiences.

(1)

Comprehensive Documentation

Thorough documentation is paramount to understanding and replicating ML model deployment. It is recommended to provide a detailed record of the model architecture, data pre-processing steps, hyperparameters, and assumptions in a technical document. Additionally, a comprehensive user manual with detailed instructions is required for deployment, usage, and troubleshooting to aid users in effectively utilizing the ML model. Transparent documentation promotes collaboration and facilitates knowledge transfer among stakeholders.

(2)

Open-Source Collaboration

Open-sourcing the ML model code and associated documentation encourages collaboration and promotes transparency. Sharing code on open-source platforms like GitHub enables peer review, feedback, and contributions from the wider community. Open-source collaboration fosters innovation, accelerates model development, and ensures reproducibility in water and environmental modeling.

(3)

Hands-On Training

Hands-on training sessions are essential to educating users on model deployment, interpretation, and application. These interactive sessions offer practical experience and guidance for using ML models in real-world scenarios. Customizing training materials for different user groups—such as engineers, scientists, managers, and stakeholders—ensures relevance and maximizes effectiveness.

(4)

User Community Engagement

Engaging the user community can be facilitated through brown bag sessions and user group meetings focused on water and environmental modeling. These gatherings serve as platforms for knowledge sharing, collaboration, feedback collection, and networking among stakeholders.

(5)

Peer-Reviewed Journal Articles

Publishing research findings and methodologies in peer-reviewed journal articles disseminates outcomes and contributes to the scientific community. Peer-reviewed publications lend credibility and validation to ML model development and application in water and environmental modeling. It also shares insights, challenges, and lessons learned to advance the field and promote transparency in model deployment and communication.

(6)

Maintenance and Improvement

After deploying ML models, it is crucial to maintain and improve them. Maintenance includes updating the model with new data, retraining it when performance declines, and addressing any issues related to computational infrastructure or data pipeline failures. Improving the model requires ongoing evaluation and incorporation of user feedback, new methodologies, or additional features to enhance its accuracy and applicability. Regular audits should also be conducted to assess compliance with regulatory and ethical standards, ensuring the model remains relevant and trustworthy in addressing evolving challenges in California’s water and environmental systems. This proactive approach ensures the long-term success and sustainability of ML applications.

3.4.2. Model Deployment Example

We deployed the ML models developed in the case study through two parallel approaches. First, for technical users who wish to use and customize the models, we have made the source code of the models (in Python) and data available in a public GitHub repository (https://github.com/CADWRDeltaModeling/Ion_Study_Dashboard/tree/main, accessed on 1 March 2024).

Second, for users who prefer not to interact with the models directly, we designed and implemented an interactive ion simulation dashboard tool, as illustrated in Figure 8. This tool combines pre-trained ML models (RF, RT, GB, and ANN) with established regression methods to provide ion predictions across the three zones of the interior Delta. The deployment leverages a cloud-based workflow, with ML models maintained in a GitHub codebase and executed through Microsoft Azure’s computational infrastructure. The live platform is accessible via https://dwrdashion.azurewebsites.net/Dashboard (accessed on 1 March 2024). The interface prioritizes accessibility through intuitive design elements. Users interact with the platform by using dynamic controls, including sliding scales for EC values, contextual menus for site selection and water year classification, and additional parameter selectors. The “Compute” functionality generates instant visual outputs, displaying comparative predictions across all modeling approaches. This setup allows users to thoroughly explore various hydrological scenarios, examining how changes in EC, Sacramento X2, monthly variations, and water year type impact ion concentrations in the Delta.

The methodology and findings from the case study have been comprehensively documented, peer-reviewed, and published [75,76], ensuring transparency and accessibility for a wider audience. Both the ML models developed and the interactive dashboard tool have been showcased at multiple forums, including stakeholder meetings, technical conferences, and user group sessions, to promote awareness and engagement within the community. Based on the feedback received, we have updated the model and dashboard to enhance their practical applicability. We will continue monitoring their performance and making improvements as new questions, data, and techniques emerge. Additionally, a hands-on training session will be organized to guide users through effectively utilizing the dashboard tool. This session aims to provide practical experience, address user-specific questions, and demonstrate the tool’s capabilities in simulating and analyzing ion predictions for water and environmental modeling.

4. Discussion

As ML garners increasing attention and finds more applications within the California water community, the ML protocols proposed in the current study become imperative for several distinct benefits.

(1) Standardization: These protocols provide standardized guidelines and procedures for ML model development and deployment. Standardization ensures consistency and reproducibility across different projects and research efforts, facilitating comparison and collaboration among water and environmental model developers, users, and stakeholders in California.

(2) Quality assurance: These protocols help ensure the quality and reliability of ML models by specifying rigorous criteria for data quality, model performance, and validation methods. By adhering to established protocols, model developers can mitigate common pitfalls such as overfitting, resulting in more robust and trustworthy modeling outcomes.

(3) Interoperability: The protocols enhance interoperability by establishing standardized guidelines on data formats, model development, evaluation, and deployment. Following these guidelines ensures that ML models can seamlessly integrate with existing modeling frameworks, databases, and decision support systems. This promotes collaboration among agencies and researchers, improving the consistency, usability, and scalability of ML-based solutions for water and environmental modeling in California.

(4) Transparency and reproducibility: These protocols promote transparency and reproducibility by documenting the entire modeling process, including data sources, pre-processing steps, model architectures, hyperparameters, and evaluation metrics. Transparent and reproducible models allow stakeholders to understand and validate modeling decisions, ensuring accountability and trustworthiness in decision-making processes.

(5) Capacity building: These protocols facilitate capacity building by providing training materials, best practices, and guidelines for model developers, users, and stakeholders involved in ML-based water and environmental modeling. A training workshop is being scheduled to daylight the protocols and provide hands-on training on how to implement them. The workshop is expected to empower stakeholders with the knowledge and skills needed to effectively apply ML techniques in real-world water and environmental applications in California.

(6) Continual improvement: Protocols support continual improvement by fostering feedback mechanisms, community engagement, and peer review processes. Iterative refinement of protocols based on feedback from stakeholders and advances in ML research ensures that protocols remain relevant, adaptable, and responsive to evolving challenges and opportunities in water and environmental modeling.

While following these protocols benefits the broader water community in California when developing and applying ML models, there are several caveats to consider. First, California’s water and environmental systems are inherently complex, with numerous interacting variables. ML models may not always capture the full scope of these interactions, potentially leading to oversimplification or missed nuances in predictions. Therefore, domain knowledge is critical to informing the development of these models. In addition, ML models are optimized based on the data they are trained on, and their predictions can become unreliable if exposed to input data that fall outside of this range. In the context of California’s dynamic water and environmental systems, this could occur when models are applied to scenarios or conditions not well represented in the training data, such as extreme weather events or changes in operations. Using ML models in such situations may lead to inaccurate or misleading predictions, undermining their utility for decision making. Moreover, California’s water and environmental policies are highly regulated, with strict ethical and regulatory considerations. ML models must comply with these regulations, particularly regarding data privacy and equity. Finally, ML models, especially those requiring large datasets or complex algorithms, can demand significant computational resources. This can limit some studies and applications, particularly in resource-constrained settings, necessitating careful planning for infrastructure and costs.

5. Conclusions and Future Directions

The protocols presented in this paper provide a step-by-step guide for applying machine learning in water and environmental modeling. While they are informed by the DWR’s three decades of relevant experience in California, they are sufficiently general to be applicable to other regions beyond the State. The core of the protocols lies in careful pre-modeling considerations. This includes clearly defining the water and environmental problems one is trying to solve and collecting and preparing high-quality data. The protocols then delve into the model development process, from choosing the most suitable ML architectures for the specific task to train, evaluate, and identify the most desirable models. Finally, the protocols cover the deployment. They outline strategies for integrating the model into real-world applications and effectively communicating its results to ML model users. This paper also includes a case study demonstrating each of these steps.

This paper intends to serve as a dynamic roadmap for harnessing the power of ML in California’s water and environmental modeling endeavors. Conceived as a living paper, it will be continuously updated and expanded to address the State’s evolving challenges in water resource management and environmental protection.

First, as California grapples with new and ever-shifting water and environmental issues, the protocols will be tailored to tackle emerging threats and opportunities. This might involve incorporating novel data sources, like high-resolution satellite imagery, which are not addressed specifically in the current paper. Recent studies have shown that combining ML with satellite remote sensing data can advance water and environmental modeling. For example, Fleming et al. [36] enhanced seasonal water supply forecasts by integrating satellite-based snow cover data into AI-driven models, while Poon and Kinoshita [60] estimated post-fire evapotranspiration by using ML with satellite-derived land and vegetation indices. Additionally, communication strategies will be honed to effectively reach a wider range of stakeholders, encompassing policymakers, resource managers, and the public.

The second front for advancement lies within the burgeoning field of ML itself. The emergence of groundbreaking techniques like explainable AI and interpretable AI [207], digital twins (DTs) [208,209], and generative artificial intelligence (GenAI) [115] presents immense potential for water and environmental modeling. A critical barrier to the widespread adoption of ML in the operational community is its perceived lack of interpretability. Unlike traditional process-based models, which are grounded in well-established physical principles, many ML models function as black boxes, making it difficult to understand how inputs influence predictions. To address this challenge, researchers have developed various methods to enhance ML interpretability. One of the earliest approaches is graphical sensitivity analysis, which helps visualize how input variables influence model predictions. Cannon and McKendry [210] applied this technique to ANNs for Indian monsoon rainfall prediction. To further bridge the gap between data-driven ML models and physically interpretable scientific models, theory-guided data science (TGDS) has emerged as a promising approach. Karpatne et al. [211] introduced TGDS as a paradigm that infuses scientific knowledge into ML models, ensuring their predictions align with established physical laws. Building on TGDS, Physics-Informed Neural Networks (PINNs) [112,212,213,214] and differentiable modeling [215] provide more advanced frameworks for integrating ML with traditional physical models. Shen et al. [187] described differentiable modeling as a method that unifies ML and process-based models through the gradient-based optimization of physically constrained models. PINNs and differentiable geoscientific models are gaining increasing interest and have been successfully applied across various water and environmental studies, improving both interpretability and predictive accuracy (e.g., [83,189]).

DTs present an exciting opportunity to develop more holistic and dynamic representations of environmental systems (e.g., [216,217]). By coupling real-time sensor data with physics-based models and ML algorithms, DTs offer a comprehensive platform for simulating and optimizing environmental processes, thereby facilitating proactive and data-driven decision making in water resource management and environmental conservation efforts. The protocols presented in this paper are expected to be expanded in the future to include guidance on interpretable ML models, as well as the creation and utilization of DTs specifically tailored for water and environmental modeling in California.

GenAI has the potential to revolutionize environmental modeling by enabling the synthesis of realistic and diverse environmental scenarios, thus enhancing the robustness and adaptability of our models to unforeseen circumstances and novel challenges [115,218,219,220]. Looking ahead, GenAI can play a transformative role in water and environmental management in California, though these applications remain conceptual for now. One possibility is leveraging GenAI-powered chatbots as intelligent knowledge hubs, providing answers to inquiries about the State’s water resources, environmental issues, and related topics. These could include general information, historical data, scientific studies, regulations, policies, and legal frameworks—essentially functioning as advanced, domain-specific search engines. Such tools could support research, training, and public outreach efforts. Additionally, GenAI could be used to enhance water and environmental modeling by seamlessly integrating mechanistic and machine learning models to improve predictive accuracy. It might also assist in dynamically selecting the most suitable model for a given task. Another potential application involves deploying GenAI-driven agents for autonomous decision support. These agents could, for instance, process hydro-meteorological forecasts, use them to run hydrologic models for streamflow forecasting, and apply forecasted streamflow to guide reservoir operations or other operational decisions. In 2023, the Governor of California issued Executive Order (EO) N-12-23 to explore the development, application, and associated risks of AI technology, particularly GenAI, across the State. This initiative aims to establish a cautious and responsible approach to assessing and implementing GenAI within state governance, recognizing its significant impact. The protocols outlined in the current paper are not tailored specifically for GenAI. However, the guidelines and best practices on GenAI resulting from EO N-12-23 will be adapted and integrated into future iterations of these protocols.

Finally, ML models, particularly deep learning models for complex water and environmental systems like those in California, demand significant computing resources. While quantum computing has been explored for decades [221,222], recent breakthroughs [223] have unlocked its potential to revolutionize ML applications across various fields, including water and environmental modeling. Unlike classical computing, which processes data in binary digits (0s and 1s), quantum computing uses quantum bits (qubits), which can exist in multiple states simultaneously. This capability allows quantum computers to solve complex problems, process vast datasets, and perform high-dimensional calculations far more efficiently than traditional computers. In water and environmental modeling, quantum-enhanced ML could improve the accuracy and speed of simulations involving complex hydrological systems, optimize resource allocation, and refine predictive models for climate change impacts, droughts, water quality, and others. Additionally, quantum algorithms could enable the integration of diverse data sources—such as satellite imagery, sensor networks, and historical records—into unified models that better capture California’s complex water and environmental systems. While the technology is still in its early stages, ongoing advancements suggest that quantum computing could become a transformative tool for addressing California’s water and environmental challenges in the coming decades. This tool will be explored in the near future and its findings will be reported in subsequent updates to the protocols.