4.1. Assessment of Representative Weather Inputs vs. FFE’s Defaults
Our expectation that representative weather scenarios would produce more accurate predictions was partially confirmed. The Mean Wind representative weather scenario resulted in the lowest mean error overall, and for most forest type groups and tree species. However, contrary to expectations, the default FVS-FFE moderate burn scenario (POTMod) had the lowest RMSEs, primarily by providing a better match than representative weather scenarios where observed mortality was low.
Accuracy of mortality prediction at stand scale was quite variable. Judged by mean absolute error (MAE), all five weather scenarios had success within five percentage points of our 30% error maximum target in at least two forest types while for nine of the forest type groups, MAE was below 33%. By contrast, RMSE of differences between predicted and observed exceeded 40% for all five weather scenarios. A key distinction between MAE and RMSE is that the latter more severely penalizes errors with the greatest magnitude [
36]. In all five scenarios, there were many stands for which discrepancy between predicted and observed mortality was very large, and this is apparent in the many cases with RMSE > 50%. Even with the best scenario, Mean Wind, predictions for 30% of stands were off by more than 50% in six of the forest type groups.
We expected that weather parameters more representative than those in FFE’s default scenarios would reduce prediction errors in fire behavior [
38,
39]. Errors in predicting fire behavior like flame length lead to errors predicting fire effects like crown scorch, and lead, in turn, to errors in predicting mortality. Flame length predictions from the three more temperate weather scenarios appear to be more accurate and credible, based on comparison with median bole char heights. However, relying on such an indirect indicator of fire intensity warrants caveats. While flame height suggested by bole char height may approximate flame length under low wind speed on gentle terrain [
40], departure from these conditions, for example, with respect to slope, fuel bed depth, and wind speed, can degrade this approximation. For example, bole char extends higher on steep slopes, irrespective of flame length [
41]. Nonetheless, much of the error in over-prediction in tree mortality in the MAXWind and POTSev scenarios stemmed from an over-estimation of fire intensity resulting from a combination of high winds and steep slopes. Our regression tree analysis highlighted greater over-prediction errors on very steep slopes (>68). The Rothermel surface fire model was originally intended to model surface fire spread on flat terrain, with later attempts to modify the model by increasing fire intensity on steeper slopes [
8,
42]. Our results suggest that the model as implemented in FVS-FFE has difficulty representing fires on steep slopes and under high winds.
Despite wind being represented in FFE as the strongest driver of fire behavior [
8], fuel moisture inputs appeared to play a greater role in our results than wind. Given the greater predictive power of the Mean Wind scenario, with POTMod and Min Wind not far behind, it appears that most of the benefits of using representative weather, rather than FFE defaults, can be attributed to the dead fuel moisture parameters and their influence on flame length prediction, rather than wind speed. At low wind speeds between 0 and 4.5 m/s, FFE, predicts little difference in fire intensity [
43]. The regression tree results provide additional support for the greater role of fuel moisture, since at very low moisture (<5% for 10-hr fuels), mortality was much greater (68%) than at higher fuel moisture. In FVS-FFE, low fuel moisture facilitates transition from surface to crown fire, even with low wind speeds, thus elevating the likelihood of mortality [
7,
8]. At the same time, in flatter stands (<43% slope), the regression tree provided evidence that high estimated fuel moisture resulted in under-estimation of mortality seen in the Min Wind scenario. Other limitations inherent in FVS-FFE’s fire behavior models may have also contributed to the underprediction by the Min Wind and POT Mod scenarios. Cruz and Alexander [
44] reported that rate-of-spread (ROS) predictions for surface fire (using the Rothermel model) and crown fire (using the Van Wagner model) were frequently underestimated. Moreover, the coupled Rothermel–Van Wagner models tend to under-predict transition of surface fire to crown fire in conifer forests [
45]. These known model biases have the potential to under-predict mortality by under-predicting fire intensity, and thus the crown scorch input to the mortality model. Surface fire ROS is important because flame length increases with ROS at low wind speeds before plateauing at higher wind speeds when fuel moisture is high [
46]. At the same time, underestimation of the transition of surface to crown fires might also be contributing to under-prediction in the Min Wind and POT Mod scenarios. Whether a fire is heading or backing up or down slope can also affect intensity and flame lengths [
47]; however, FVS-FFE only models head fires [
8].
Our discussion so far assumed that our “representative weather” wind estimates are accurate depictions of the fire environment when FIA plots were affected. Given that the nearest station is sometimes tens of miles away from the FIA plots, RAWS observations, especially wind speeds, are imperfect proxies for the weather on the plots when fire arrives. While it is at least theoretically possible to generate interpolated or meso-scale adjusted wind speed estimates using advanced models, topographic complexity—mountain ranges, elevation differences, wind-protected areas—pose challenges to accuracy. Page et al. [
48] tested the accuracy of the National Digital Forecast Database used by fire modelers with RAWS station measurements and found that the model tended to underestimate wind speeds when winds speed exceeded 4 ms
−1. More accurate downscaling might provide better weather inputs for future validation efforts. However, the temporal uncertainty on when, exactly, fire encountered the plot makes it difficult to pin down the best hour or hours of RAWS observations to use. Moreover, there may be limits to how well the models underlying FVS-FFE can use improved wind inputs as wind effects on wildfire rate of spread and flame lengths are modeled quite simply compared to other models (FARSITE e.g., [
49]) or latest fire physics models [
50].
Beyond weather inputs, fuel models are used to represent pre-fire conditions that influence fire behavior [
46]. In this study, we allowed FVS-FFE to select the fuel models, which vary by forest type. We did experiment with selecting fuel models based on fuel load data from the inventory plot, but the fuel models selected and results were typically not much different, which is consistent with other studies which report limited benefits to simulated fire behavior accuracy from customizing fuel models based on field-measured fuel loading [
51].
We found that the most predictive weather scenario varied by forest type group. The more severe weather scenarios produced the lowest prediction RMSEs in the Juniper/Pinyon and Pines forest type groups, though RMSEs exceeded 40%. Mortality has been observed to be highly variable in Pinyon-Juniper owing to high variability in fuel loads and vegetation composition [
52], and these species’ thin bark make them comparatively vulnerable to lethal cambial heating, though survival remains possible where surface fuels and/or tree cover are sparse [
53]. The Firs and Ponderosa pine forest type groups had the lowest prediction RMSE with the Min Wind scenario, possibly reflecting higher fuel moisture in these forest types than predicted.
The evaluation of FVS-FFE’s mortality predictions is complicated by challenges imposed by the spatial scale of the FIA plot. The bi-modal distribution of observed mortality (high frequencies of 0 and 100%) could reflect fire effects in within-stand patches rather than the mean effect across individual stands. Conditions adjacent to FIA plots may also introduce variation in fire effects, for example local topographic features that amplify (e.g., a canyon headwall acting as a chimney) or mitigate (e.g., a wind-protected spot) fire behavior. A strength of FIA data is that it reflects the full variation in forest conditions visited by fire, in proportion to how to their occurrence in the landscape. However, parameterization of the FFE was not based on a probability sample from the Pacific Northwest and California, so some FIA conditions may occur in what are essentially “gaps” in the continuum of forest conditions represented in the data from which the FFE model parameters were derived.
4.2. The Role of RA Equation and Tree Species Effects
As the only species-specific parameter in the Ryan and Amman [
54] mortality model (RA), the bark thickness coefficient (which is multiplied by tree diameter to estimate bark thickness) is intended to account for species-specific resistance to fire-induced cambial injury [
55] and can be interpreted as inversely correlated with probability of cambium death [
35,
54]. In an assessment of the RA equation on first-order mortality after prescribed fire on National Park lands in the western U.S., Kane et al. [
56] found that the RA equation over-predicted mortality for species with thin bark, under-predicted for trees with thick bark, and wasn’t very accurate for any tree with bark <1 cm thick. The most abundant species in this study included trees relatively resistant to fire (Douglas-fir, incense cedar, Ponderosa pine), moderately resistant (white fir, tanoak), and not resistant (California black oak, canyon live oak). Mortality prediction RMSE declined as bark thickness coefficient decreased such that canyon live oak had the lowest RMSE. Bark thickness appears to be a good predictor of first-order mortality, but the assumed linear relationship between bark thickness and tree diameter is not always correct [
55]. Zeibig-Kichas [
57] found FVS-FFE tended to under-predict bark thickness in California conifers, which would lead to over-prediction of mortality [
1,
56], but empirical evidence for this effect is lacking given limited data on bark thickness.
In this study, thin-barked canyon live oak was the only species for which mortality was over-predicted by all three representative weather scenarios. While errors in estimation of bark thickness or its effect on mortality may be partly responsible, hardwoods like this have the additional complication in top-killed trees with 100% crown scorch. Even if aboveground plant tissues are dead, new stems and leaves from underground plant tissue will commonly emerge [
58], but this does not convert the tree’s status to live under the FIA protocol. However, any tree that sprouts living tissue above the point of diameter measurement is considered a survivor under FIA measurement protocol (as opposed to basal sprouts, which count as new trees). Considering surviving trees included in this study that experienced any crown scorch, we found that 2.8% of the canyon live oak trees had 100% crown scorch, but most had recovered with live crown ratios over 10%. Fewer than 1% of the surviving burned trees of the other species discussed here had recovered from 100% crown scorch. Canyon live oak is increasing in basal area despite increased fire activity in California, possibly due to exploiting niches in unburned stands where fuels are sparse and to its shade-tolerance [
59].
Average observed mortality for the more fire-resistant California black oak was no less than for the thinner barked canyon live oak; other studies report comparable mortality rates of about 60% [
13,
60]. The best RMSE scenario slightly under-predicted mortality, suggesting that resprouting of heavily-scorched trees was not as strong a factor as in canyon live oak. Nevertheless, basal sprouting can greatly affect post-fire stand dynamics, with up to 70% of the top-killed black oak resprouting in one study [
13]. California black oak has been experiencing a decline in basal area in California [
59], in part due to conifer encroachment into black oak canopies resulting in greater crown-fire caused mortality [
61].
FFE’s mortality predictions have several limitations, some of which can be overcome. First, crown scorch is less effective than fire residence time in predicting mortality for species that re-sprout and for small trees [
55]. Second, the RA equation does not account for heat-induced root damage, which is a significant mortality mechanism in ecosystems like Ponderosa pine [
55,
62]. Third, the development and validation of the RA equation focused on conifers and largely excluded hardwoods [
1,
55], so expanding coverage of fire effects on species beyond major conifer species, such as Douglas-fir and Ponderosa pine, would be a major contribution to accuracy improvement and could build on the work now underway to improve mortality models for southern hardwoods [
63,
64]. Although the ability of the model to predict mortality for the major hardwood and softwood species was similar in this study, the route to improving model performance will likely require different variables for different species (e.g., root damage for Ponderosa pine as noted above).
4.3. Model Evaluation and Improvement in the Context of Forest Management
There are alternative ways to interpret model errors depending on objectives. Because it is based on a squared error term, RMSE gives greater weight to large errors than to small ones. The direction of error (under vs. over prediction) is also important. In some management contexts, over-prediction of tree mortality, as we observed occurring with the Max Wind and POTSev scenarios, might sometimes be preferred. Our results are particularly relevant to landscape-level assessments of fire-caused mortality used to support land management planning. Such assessments often focus on the extreme weather and fuel moisture (e.g., 90th or 97th percentiles) [
65,
66], such as those represented by the POTSev weather scenario. Our approach highlights some advantages of using a broader range of weather rather than relying on a single percentile. Over-reliance on the upper percentile wind values may lead to over-estimating fire effects like tree mortality, since stands might burn in light to no wind during otherwise extreme fire events. Given that differences between observed and predicted mortality were very high for a substantial fraction of cases for every forest type for every weather scenario (hence the large RMSEs), reliability of mortality prediction for an individual stand seems insufficient to support decisions concerning that stand. However, applying the mortality model using the weather scenario with the lowest mean error may nonetheless generate mortality predictions that, when considered in aggregate across multiple stands, prove useful in representing alternative outcomes, for example, under alternative forest management or fuel treatment scenarios. In a study of carbon recovery after the 2007 Angora fire the Lake Tahoe Basin, Carlson et al. [
67] found that getting accurate estimates of mortality was a key consideration in determining how fuel reduction treatments affected post-fire recovery of carbon stores. They found that FVS-FFE predicted basal area mortality for all stands examined to be within ±15% of observed mortality.
FFE-FVS provides many controls that may be deployed to localize, fine-tune and calibrate model inputs so as to improve mortality predictions [
63]). Moreover, some FFE model users possess expert knowledge in fire science and/or fire operations that may help them utilize even inaccurate model outputs to support decision-making, and guide modeling efforts, as is common practice [
68,
69]. However, extensive calibration effort introduces the potential to convert fire and mortality modeling to an exercise that does little more than confirm pre-existing assumptions. This study demonstrates very large discrepancies between modeled and observed tree mortality under a wide range of input scenarios guided by either FFE default assumptions or weather deemed relatively representative of when and where fire burned. While performance may approach what is acceptable for some purposes when considering accuracy on average, for every forest type group, for FIA conditions representing at least 20% of the forest (and as high as 35%–40% in some type groups), predicted differed from observed by more than 50 absolute percentage points. These errors have implications for decisions based on model output. For example, in a case where a stand has 100 percent mortality observed, predicted mortality could be less than 50%. The underestimation of mortality would present a misleading forecast of fire effects, potentially contributing to a lack of success in achieving desired management outcomes in cases where outcomes are affected by degree of mortality, such as when assessing recovery of carbon stocks after wildfire [
67].