**4. Comparison of the SJR WARMF and Regression Model Applications**

A comparison of the SJR WARMF and Regression models was undertaken to evaluate the performance of the models for water quality forecasting. This evaluation initially compared differences between forecasted and observed water quality measured as EC at the monitoring station located at Vernalis (Figure 4). Similar analyses were performed in Excel using an algorithm that computed the difference (Δ) between the daily forecasted (FC) and observed (OBS) EC (Δ = FC − OBS) starting on the forecast day (FC Day + 0) and each consecutive day within the lead forecast time of 14 days (FC Day + 14). The analyses were conducted with observations and forecasts made between 22 February 2018 and 22 May 2020. During this period, a total of 820 EC observations were measured. However, for all the forecast lead times considerably fewer forecasts were actually made. In the case of the Regression model, the number of forecasts ranged from 399 for forecasts of less than 6 days (FC Day + 6) down to 347 forecasts for lead times of 7 days or more (FC + 7 to FC + 14). Forecasts were made only on regular workdays and were not conducted on certain days due to personnel availability and periods of downtime in the monitoring system. Forecasts for days 11 through 15 were simply repeats of the FC + 10 forecast given

that the California River Forecast Center (RFC) does not extend its daily forecasts, used by the WARMF and Regression models, past 10 days.

In the case of the WARMF model, there were even fewer forecasts throughout the evaluation period. The greater personnel time commitment to make WARMF model forecasts limited the forecast frequency to once per week, usually on a Monday. There were 131 forecasts for lead times from FC + 0 to FC + 7 and fewer forecasts for greater lead times. Table 2 presents the frequency count and statistics (mean and standard deviation) for the observations and model forecasts in the initial comparison of results produced by the Regression and WARMF models. Table 2 confirms that the Regression model forecasts were made approximately 3 times more often than those for the WARMF model.

In general, the Regression model forecasts had mean EC predictions that are approximately equal to the mean EC of the observations but increased to above the observation's mean EC after FC Day + 5 through the end of the forecast period. The WARMF model had slightly lower mean forecast EC values until FC Day + 4 after which they increased throughout the remainder of the forecast period. The observed EC, Regression and WARMF forecast mean EC values were compared in Figure 5 at each of the forecast lead times.

**Figure 5.** Means of the observed (OBS) EC and Forecast (FC) EC for the Regression and WARMF models for all forecast lead times between 22 February 2018 and 22 May 2020.


**Table 2.** Statistics of observed (OBS) and forecasted (FC) EC (μS/cm) for the Regression and WARMF models made between 22 February 2018 and 22 May 2020 by lead time.

> A comparison of the mean of differences between forecasted EC and observed EC for both Regression and WARMF models is shown in Table 3. For both models, the mean of the differences between forecasted EC minus observed EC was computed for the period between 22 February 2018 and 22 May 2020. For the Regression model, the differences were small (≤+5) for until Δ Day + 6. The mean EC differences increase to maximum of 15 μS/cm at Δ Day + 9. From Δ Day + 10 to the end of the forecast period, the mean EC differences decrease slightly to a value of 12 μS/cm. For the WARMF model, the mean of the EC differences were small, decreasing from +1 to −3 at Δ Day + 3. From Δ Day + 4 to Δ Day + 12, the mean of the EC differences increases consistently reaching a peak value of

+33 μS/cm at Δ Day + 12 after which there is a slight decrease to 30 μS/cm at the end of the forecast period. These results are illustrated in Figure 6.

**Table 3.** Comparison of mean differences (Δ) between forecasted EC and observed EC (μS/cm) for all model forecasts made between 22 February 2018 and 22 May 2020.


**Figure 6.** Comparison of mean differences in forecasted EC and observed EC for the Regression and WARMF models for the period between 22 February 2018 and 22 May 2020.

The forecast standard deviation is a measure of the dispersion of the forecast EC predictions around the mean EC value. Larger standard deviations imply a wider range of forecast predictions of EC and/or differences between forecasted EC values and observed EC. Figure 7 presents the standard deviations of the EC observations and EC forecasts for both models (Figure 7a) as well as the standard deviations of the EC differences between the forecasts minus observations (Figure 7b) over the forecast period. As illustrated, the standard deviations of the Regression model EC forecasts closely approximate the standard deviations of the EC observations at all lead times. In contrast, the standard deviations of the WARMF model EC forecasts are consistently less than standard deviations of the EC observations until lead time day 8 as shown in Figure 7a. The maximum difference (33 μS/cm) between forecast and observation standard deviations occurs at lead time day 2. In Figure 7b, the standard deviation of the differences between the EC forecasts minus EC observations for both models increase consistently with lead time indicating increasing uncertainty in the EC forecasts. Additionally, the WARMF model has consistently greater standard deviations in EC differences relative to the Regression model.

**Figure 7.** (**a**,**b**). Comparison of the standard deviations of forecasted EC and observed EC and standard deviations of differences between EC forecasts and EC observations for the Regression and WARMF models by lead time in the period between 22 February 2018 and 22 May 2020.

An additional evaluation was performed to determine the extent to which model bias affects the mean of differences between the forecasts and observations. For example, the models could forecast values significantly greater than the observations. However, a few large underestimates could potentially offset the positive bias and make the model appear to show better performance. In order to examine this effect, forecasts which were greater than the corresponding observations were examined separately from those in which the forecasts were less than the corresponding observations. After this sorting into positive (forecast >= observation) and negative (forecast < observation) bias groups, the means of the EC differences (forecast–observation) over the study period were calculated for each forecast lead time.

Figures 8 and 9 illustrate comparisons of the Regression and WARMF models for the positive and negative bias results, respectively. For the positive bias differences, the Regression model has lower differences at all lead times than the WARMF model.

**Figure 8.** Comparison of means of forecasted EC and observed EC for the Regression and WARMF models for the period between 22 February 2018 and 22 May 2020. Data censored to include only over (positive) predictions.

**Figure 9.** Comparison of means of forecasted and observed EC for the Regression and WARMF models for the period between 22 February 2018 and 22 May 2020. Data censored to include only under (negative)-predictions.

For the negative bias differences, the Regression model has lower negative mean differences than the WARMF model from Δ Day + 0 to Δ Day + 11 after which both models have nearly equal EC differences.

Another aspect of the potential bias introduced by these forecasting methods is how frequently do the overpredictions (positive) or underpredictions (negative) of mean differences in EC occur as a function of forecast lead times. For instance, the mean EC forecast bias could be overly influenced by a small number of very large EC discrepancies—either positive or negative. Figure 10 compares the percentages of positive bias differences in EC for both models.

**Figure 10.** Comparison of the percentages of higher (positive bias) EC forecasts for the Regression and WARMF models for the period between 22 February 2018 and 22 May 2020.

As illustrated above, both the Regression and WARMF models exhibit a slight positive EC forecast bias. The Regression model exhibits a higher frequency (65%) of positive EC forecast bias differences on Δ Day + 0 for the period between 22 February 2018 and 22 May 2020. From Δ Day + 1 to Δ Day + 4, the Regression model has a neutral EC forecast bias frequency of approximately 50%. From Δ Day + 5 to Δ Day + 8, the Regression model EC forecast bias becomes increasingly positive reaching a maximum of 60% before declining gradually to 55% by Δ Day + 14. The WARMF model exhibits a gradually increasing positive EC forecast bias from 53% on Δ Day + 0 to 58% on Δ Day + 6. Subsequently, the EC forecast bias declines slightly to Δ Day + 9.

In summary, the results of the model comparison analyses indicate that the Regression model EC forecasts were closer to the overall mean of the EC observations than the WARMF model forecasted EC (Figure 5). As illustrated by Figure 6, the Regression model provided EC forecasts with mean differences of less than or equal to 5 μS/cm for the first 7 days (Δ Day + 0 to Δ Day + 6). In comparison, the WARMF model provided EC forecasts with mean differences of less than or equal to 5 μS/cm for only 5 days (Δ Day + 0 to Δ Day + 4). Based on these measures of performance, the Regression model provided EC forecasts with reduced error relative to the WARMF model especially for the period from Δ Day + 4 to Δ Day + 6.

The standard deviations of Regression model EC forecasts closely approximated the standard deviations EC observations at all lead times. In contrast, the standard deviations of the WARMF model EC forecasts were consistently less than the corresponding standard deviations of the EC observations at lead time less than day (Figure 7a). For both models, the standard deviation of EC forecast differences steadily increased with forecast lead time, as expected, while the WARMF model had higher standard deviations of EC than the Regression model throughout the forecast period (Figure 7b).

When the EC forecasts were separated into those with overestimate (positive) and underestimate (negative) biases, the mean differences between the EC forecasts and observations were seen to increase predictably with forecast lead times. For both the positive and negative forecast EC mean differences, the Regression model performed better than the WARMF model for lead times from Δ Day + 0 to Δ Day + 10. From Δ Day + 12 to Δ Day + 14, the performance of both models was approximately the same.

As illustrated in Figure 10, both models have slightly positive EC forecast biases. With the exception of the high overprediction (positive) bias (65%) for the Regression model EC on Δ Day + 0, the Regression model predictions were relatively unbiased between Δ Day + 1 to Δ Day + 4 and subsequently remained slightly positively biased throughout the remainder of the forecast period. The WARMF model made consistently greater overpredictions (positive biases in EC) than the Regression model.

It is also important to note that the Regression model EC and WARMF model EC results were originally based on different forecasted flows. Up until mid-2020, the WARMF model used prior water year operations forecast for the 14 day flow forecast along the three major east-side tributaries. From July 2020 onward, the WARMF model has been using the same flow forecasts as the Regression model which come directly from the NOAA California-Nevada River Forecast Center. The analyst who makes these daily forecasts is in regular communication with reservoir operators at Modesto Irrigation District, Merced Irrigation District and the USBR Central Valley Operations Office who control releases and provide regular bulletins of changes in release schedules. Hence, any differences between the models are no longer a function of the flow release forecasts but rather the WARMF model's watershed simulation and prior knowledge of diversions and drainage inflow along each tributary research and along the mainstem of the SJR.
