Aviation Turbulence Forecasting over the Portuguese Flight Information Regions: Algorithm and Objective Verification

Abstract

Aviation turbulence remains one of the leading causes of weather-related aviation accidents. Therefore, turbulence prediction is a major concern of aviation forecasters. This paper describes the turbulence index (${\mathrm{TURB}}_{\mathrm{IPMA}}$) developed and used operationally at the Portuguese Institute of Sea and Atmosphere (IPMA), based on several diagnostics derived from ECMWF forecasts, using a new calibration approach. The forecast skill of the ${\mathrm{TURB}}_{\mathrm{IPMA}}$ and of individual diagnostics are evaluated using turbulence observations over the Portuguese Flight Information Regions and surrounding areas, for 12 months between February 2020 and March 2021 (excluding May and June). The forecasting skill of the predictors is discussed in terms of the Relative Operating Characteristic (ROC) curves, which is widely applied, but also in terms of novel measures such as the Symmetric Extremal Dependence Index (SEDI) and Symmetric Extreme Dependency Score (SEDS). The new measures are particularly relevant in assessing forecasts of rare events, such as moderate-or-greater turbulence. The operational index outperforms individual diagnostics (such as Ellrod) in terms of all verification measures. Furthermore, the use of a new Richardson number function was proven to be beneficial. Finally, the turbulence prediction by IPMA was comparable to that of the London WAFC for one turbulence episode.

1. Introduction

Aviation turbulence, experienced as in-flight bumpiness, is atmospheric turbulence caused by turbulent eddies with scales that can affect aircraft in flight. These scales range from about 100 m to 1 km and aircraft bumpiness is most pronounced when the size of the turbulent eddies encountered is about the size of the aircraft [1,2]. For commercial aircraft, this would correspond to eddy dimensions of approximately 100 m [2]. Turbulence remains a major aviation hazard as it is the leading cause of weather-caused accidents worldwide at cruise and descent phases [3]. For instance, from 2000 to 2011, in the USA, over 70% of weather-related accidents involving commercial jet aircraft at cruising levels are related to turbulence [4]. Furthermore, turbulence is responsible for tens of millions of losses for the aviation industry per year due to customer injury claims and aircraft damage [2].

Aviation turbulence can have different sources, namely convective clouds, upper-level fronts, mountain waves [1,5,6,7,8,9]. Turbulence not associated with convective clouds is referred to as clear-air turbulence (CAT) and is particularly hazardous to aviation because it cannot be detected by satellite or on-board radar [10]. Therefore, forecasting turbulence within stratiform clouds or cloud-free areas is of the utmost importance.

Turbulence scales affecting aviation are inferior to 1 km, which is smaller than a grid box of operational numerical weather prediction (NWP) models [2]. Thus, turbulence cannot be explicitly predicted, and, consequently, several diagnostic indicators have been used to predict the areas of the atmosphere where turbulence is likely to occur. In particular, Ellrod indices [2,10,11,12,13] and the Richardson number [5,10,14] have been widely used. These and other turbulence indicators have been utilized by the two World Area Forecast Centres (WAFCs)—London (Met Office) and Washington (National Oceanic and Atmospheric Administration, NOAA)—that are responsible for providing operational turbulence forecasts to Meteorological Watch Offices (MWOs), which are used by pilots and flight planners around the world [15,16]. Until recently, these turbulence forecasts were provided worldwide with a horizontal grid spacing of 1.125°. Since November 2020, these forecasts are provided with a horizontal grid spacing of 0.25° up to 36 h forecast with 3 h steps, for nine vertical layers above 10,000 ft.

Instituto Português do Mar e da Atmosfera (IPMA), as MWO, is responsible for preparing and disseminating Significant Meteorological Information (SIGMET: information issued by a meteorological watch office concerning the occurrence or expected occurrence of specified en-route weather and other phenomena in the atmosphere that may affect the safety of aircraft operations) to two Flight Information Regions (FIR): Lisbon (EUR Region) and Santa Maria Oceanic (NAT Region), which cover an area of about 6 million sq. km (see Figure 1). In particular, SIGMET of turbulence is issued when severe turbulence has occurred, or it is expected to occur [16]. Therefore, the accuracy of turbulence forecasts based on NWP outputs is of the utmost importance.

The first goal of the present paper is to describe the algorithm of the turbulence index, based on an integrated approach, developed at IPMA (${\mathrm{TURB}}_{\mathrm{IPMA}}$). This index uses forecasts of the European Centre for Medium-Range Weather Forecasts (ECMWF) deterministic model and provides to the Portuguese MWO operational hourly turbulence forecasts up to 48 h and tri-hourly forecasts up to 60 h. The availability of hourly forecasts is considered a significant asset by forecasters. Although the early version of ${\mathrm{TURB}}_{\mathrm{IPMA}}$ was implemented in 2015 at the Portuguese MWO, it has been documented only at IPMA to date. Aviation forecasters use ${\mathrm{TURB}}_{\mathrm{IPMA}}$ and the WAFC products as guidance tools to forecast severe turbulence conditions and to issue SIGMET information if it is justified. The use of ECMWF may be an advantage because this model outperforms the NWP global models operational at both WAFCs [17] and for consistency reasons, as forecasters at IPMA diagnose the synoptic environment primarily by relying on ECMWF forecasts. The second aim of this study is to provide a performance assessment of several turbulence diagnostics, and of ${\mathrm{TURB}}_{\mathrm{IPMA}}$, using novel verification measures.

This paper is organized as follows: Section 2 describes the data and the verification methodology used in this study. The results are presented and discussed in Section 3. This section provides characteristics of the turbulence data and analysis of the distribution of model turbulence indicators. Besides, Section 3 provides the assessment of the performance of several turbulence indicators and the description of the turbulence algorithm applied operationally at the Portuguese MWO. Additionally, the performance of this index is analyzed, and the comparison between this index and the WAFC product is presented for one turbulence episode as an example. Finally, Section 4 highlights the main conclusions.

2. Materials and Methods

2.1. Turbulence Data

This study uses special air-reports (pilot’s reports, AIREP hereafter) and derived equivalent vertical gust velocity (DEVG) from Aircraft Meteorological Data Relay (AMDAR) for the period from February 2020 to March 2021, excluding May, June, and 20 days of April. The DEVG is a turbulence indicator estimated from vertical acceleration, which also depends on aircraft mass, equivalent air speed, and empirical constants [18,19]. Sherman [18] proposed using 9 m s⁻¹ as the criterion for severe (SEV) turbulence. Truscott [19] classified the turbulence severity in four classes. Null (NON) turbulence when DEVG < 2 m s⁻¹; Light (LGT) turbulence is defined as 2 ≤ DEVG < 4.5 m s⁻¹; moderate turbulence is defined as 4.5 ≤ DEVG < 9 m s⁻¹; severe turbulence for DEVG ≥ 9 m s⁻¹ [19]. The DEVG indicator has been utilized in statistical analyses on aviation turbulence, e.g., [10,20,21], and in evaluations of the performance of turbulence forecasts based on NWP models, e.g., [13,20,22,23]. However, due to uncertainties in the empirical parameters used in the DEVG computation, DEVG can contain misleading values, mostly during the ascent and descent phases [19,20]. Therefore, DEVG data is used only for levels above FL150, in agreement with previous studies, e.g., [20,22,24]. Note that the flight level (FL) is a surface of constant atmospheric pressure (expressed as pressure altitude in hundreds of feet) which is related to 1013.2 hPa, based on the assumption of the International Standard Atmosphere [16].

The Eddy Dissipation Rate (EDR) is also used as a measure of turbulence and is the International Civil Aviation Organization (ICAO) standard for reporting turbulence [16]. However, in the study area, EDR data is not available and the AIREPs contains only information about moderate and severe turbulence. Therefore, the use of DEVG was the only option to complement the AIREP data.

2.2. Forecast Data and Turbulence Indicators

This study uses hourly forecasts from the ECMWF deterministic model for the period from February 2020 to March 2021, excluding May, June, and 20 days of April. The model uses a cubic-octahedral spectral transform discretization, which corresponds to a grid spacing of approximately 9 km [25]. It has 137 vertical levels and the lowest level is located at nearly 10 m above ground. In the troposphere, the vertical grid-spacing increases from 20 m near the surface to 290 m above 6 km. At IPMA, operationally, the turbulence indicators are computed using the ECMWF forecasts in a small domain covering mainland Portugal with a resolution of 0.1° and on a large domain with a horizontal resolution of 0.2°, including the two Portuguese FIRs and surrounding areas (see Figure 1). The turbulence diagnostics tested in this study are described in Appendix A. These diagnostics are calculated at ECMWF model levels for levels above 800 hPa, and, prior to their calculation, the model fields were smoothed using a 9-point smoothing filter. All 9 points were multiplied by their weights and summed, then divided by the total weight to obtain the smoothed value. The centre point received a weight of 1.0, and the points on either side and above and below received a weight of 0.5, and the corner points received a weight of 0.3. The turbulence forecast at the lower levels is beyond the scope of this study and will be the focus of forthcoming research. In the lower troposphere, it may be better to calculate turbulence diagnostics at constant heights, rather than at model levels, where shear may be created only by terrain gradients.

2.3. Verification Methodology

The use of contingency tables has been widely applied in the evaluation of forecasts of turbulence [10,22,23,26], where forecasts and observations are expressed in binary terms, by defining an event (yes) if turbulence was observed and a non-event (no) if there was no turbulence. A simple 2 × 2 contingency table (

Table 1. A simple 2 × 2 contingency table, expressing the joint occurrence of a turbulence index greater than or less than some threshold value f_i and an observation greater or less than threshold value o_i.

Forecast	Observation
	Obs < oi (No)	Obs ≥ oi (Yes)
Index < fi (No)	NN	NY
Index ≥ fi (Yes)	YN	YY

) contains four elements: hits (YY), false alarms (YN), misses (NY), and correct negatives (NN): where YY are the number of correctly predicted events; NY denotes the number of events that occurred but were not predicted; YN are the number of predictions of an event when the event was not observed; and NN represents the number of times an event was not observed and not predicted.

Several scores can be determined from a contingency table [27]. The bias rate (BIAS) is (YY + YN)/(NY + YY). The fraction of correctly forecast events, POD = YY/(YY + NY), is defined as the probability of detection [27,28,29] or as hit rate [22,23,30]. The probability of false detection (also known as false alarm rate) is defined as the ratio of false alarms to the total numbers of nonevents: POFD = YN/(YN + NN) = 1 − POD_NO, where POD_NO = NN/(YN + NN) is the probability of detection of non-events. The True Skill Statistic (TSS), also known as Hanssen–Kuipers (H-K) Discriminant, is defined as TSS = POD − POFD.

Another measure commonly used to assess the quality of a forecast system [10,14,22,23,28,29] is the relative operating characteristic (ROC; Mason [31]). For each 2 × 2 contingency table, the corresponding POD and POFD can be determined for different thresholds of the model predictors. Then, the ROC curve is defined by a set of POD and POFD values for these thresholds. If a forecast system is skillful (POD > POFD), the ROC curve will lie above the 45° line from the origin and the total area beneath the ROC curve (hereafter AUC) will be greater than 0.5 [31]. Therefore, for a perfect forecast, AUC will be equal to 1; on the other hand, an ROC curve coincident with the diagonal line indicates no skill and AUC = 0.5. The ROC curve can also be useful to determine what POD threshold would yield an acceptable POFD.

However, some previous studies have shown that many measures based on 2 × 2 contingency tables (e.g., POD, POFD, TSS, Heidke skill score) converge to trivial values (either to 0 or 1) as the rarity of the event increases, i.e., when the correct negative term dominates the contingency table [32,33]. Thus, to overcome some of the shortcomings of these verification measures, new scores have been derived. Namely, Hogan et al. [33] proposed the Symmetric Extreme Dependency Score (SEDS), defined as:

$$SEDS=\frac{log\left(q\right)-\log \left(POD\right)}{\log \left(BR\right)+\log \left(POD\right)}$$

(1)

where $q=\left(\mathrm{YY}+\mathrm{YN}\right)/\mathrm{N}$ is the relative frequency with which the event was predicted and $BR=\left(\mathrm{Y}\mathrm{Y}+\mathrm{N}\mathrm{Y}\right)/\mathrm{N}$ is the base rate and N is the sample size. Thus, BR is the relative frequency of occurrence of the event and therefore rare events have low base rates.

More recently, Ferro and Stephenson [30] proposed the symmetric extremal dependence index (SEDI):

$$SEDI=\frac{log\left(POFD\right)-log\left(POD\right)+log\left(1-POD\right)-log\left(1-POFD\right)}{log\left(POD\right)+log\left(POFD\right)+log\left(1-POD\right)+log\left(1-POFD\right)}$$

(2)

Goecke and Machulskaya [26] also applied this score to evaluate the performance of turbulence forecasts. The use of SEDI for this purpose was also suggested by [1] (pp. 272–273).

3. Results and Discussion

The performance of the ECMWF-based turbulence predictors, presented in this section, is evaluated against DEVG and AIREP observations, made over 12 months between February 2020 and March 2021 (excluding May and June).

3.1. Characterization of Turbulence Observations

In the study period, the turbulence data included 19,120 DEVG observations and 222 AIREPs, totaling 19,342 observations. The location of these observations over the Portuguese FIRs and surrounding areas is shown in Figure 2, displaying AIREPs dispersedly distributed and DEVG exceeding 4.5 m s⁻¹ more concentrated over land. For levels above FL280, the 90th and 99th percentile are 0.4 and 1.9 m s⁻¹, respectively. These values are smaller than those found by Kim et al. [24] for a 39-month period from Hong Kong-based airlines, where, for instance, the 90-percentile varied between 0.63 and 1.3 m s⁻¹, depending on the aircraft type.

Null cases represent about 92.4% and are mostly distributed over sea, while LGT turbulence represents 7.2% of the DEVG data. Moderate turbulence cases represent approximately 1.4% of total data and 0.44% of the DEVG data. Severe turbulence represents 0.2% of the total data (see

Table 2. Absolute frequency of DEVG and AIREP data. The turbulence observations total 19,342 cases.

AMDAR DEVG (m s−1)				AIREP
NON (0–2)	LGT (2–4.5)	MOD (4.5–7)	MOD-SEV (7–9)	MOD	SEV
17,659	1377	76	8	185	37

). The majority of observations were recorded in December and January, comprising 40% of the data. Another 40% of the data is concentrated between August and November (Figure 3a). Most of the data is registered between FL300 ($~$30,000 ft) and FL390, totaling almost 60% (Figure 3b).

Moderate or greater turbulence (MOG) was observed most frequently in autumn and winter (Figure 3c,d). It is also relevant to note that the monthly relative frequency (turbulence events divided by the number of observations in each month) of MOG turbulence is maximum in February and March (≥3.1% for MOD and 0.7–0.9% for SEV turbulence). The observations of MOD and SEV turbulence have the maximum absolute frequency in FL360/390 layer, registering 90 and 16 encounters, respectively (Figure 3e,f). Besides, the relative frequency (turbulence events divided by the number of observations in each layer) of MOG turbulence peaks in the FL390/430 layer (6.2% for MOD and 1.3% for SEV turbulence) and presents the second maximum in the FL360/390 layer (compare Figure 3e,f and Figure 3b). Moreover, moderate turbulence has a second maximum (50) in the layer between FL150 and FL200 (Figure 3e), corresponding to a monthly relative frequency of 1.6% (Figure 3b,e).

3.2. Distribution of Turbulence Indicators

Figure 4a depicts the box-plots of the distribution of turbulence observations for four turbulence intensity classes (NON, LGT, MOD, and SEV). This box-plot uses only DEVG data for NON and LGT turbulence, but also uses AIREP data for MOD turbulence. The largest DEVG value is less than 9 m s⁻¹, and therefore the SEV observations contain only AIREP data. Each AIREP reporting moderate and severe turbulence was assigned a DEVG value of 5.5 and 9.5 m s⁻¹.

Figure 4a also shows that most of the NON turbulence events have DEVG less than 0.5 m s⁻¹. The median value of DEVG is 2.4 m s⁻¹ and 5.5 m s⁻¹, respectively, for LGT and MOD turbulence. The box-plots of the distribution of turbulence forecast indicators for four classes of turbulence intensity (NON, LGT, MOD, and SEV) are shown in Figure 4 and Figure 5. The values of the turbulence prediction indicators correspond to their maximum value at the eight grid points (including two vertical levels) closest to the location and time (within a time window of $\pm$25 min) of the observation for each observed turbulence category.

In general, the NWP turbulence indicators show similar distributions for NON and LGT events, depicting slightly lower values for LGT events. This result indicates that these turbulence indicators have difficulty in distinguishing between null and light turbulence encounters (Figure 4 and Figure 5). In addition, uncertainties in DEVG (referred by Kim et al. [24]) may also contribute to this outcome.

The distribution of the Richardson number ($R_i$) reveals lower values for SEV and MOD turbulence than for the other classes, with a 75th percentile of 4.4 and 2.3, respectively, for MOD and SEV turbulence (Figure 5c). It is relevant to remember that Kelvin–Helmholtz instability (KHI) is a known source of CAT [34,35], and that this instability is favored when $R_i$ is less than the critical Richardson number (Ric), close to 0.25 [34,36,37,38]. However, due to the relatively coarse resolutions from the NWP models, the Richardson number computed using outputs from these models rarely reached values lower than 0.5 [39] and therefore the thresholds of Ric may be larger than the theoretical values, as suggested by Figure 5c.

The other turbulence indicators present higher values as the turbulence severity increases. For instance, the 25th percentile of SEV encounters is slightly higher than the median of the MOD distribution, except for the DEF indicator (Figure 4 and Figure 5). Moreover, the median of the MOD encounters is clearly higher than the 75th percentile of the LGT distribution.

3.3. Description and Evaluation of the Operational Turbulence Index

This section describes the methodology applied to generate the operational turbulence index used at the Portuguese MWO.

3.3.1. The Turbulence Predictors

Previous studies have shown that combining several turbulence predictors, rather than using only one predictor, improved the forecasting skill [2,10,13,26]. This approach, where an index results from a linear combination of the individual predictors, is known as the integrated approach. However, the units and magnitudes of each turbulence diagnostic are different from each other, so some normalization is required. In this study, the turbulence diagnostics are normalized using Equation (3). The conversion coefficients (${\mathit{bb}}_i$ and $f$i), presented in

Table 3. Parameters and functions used in the regression for each turbulence indicator in Equation (3).

$\mathit{T}{\mathit{u}}_{\mathit{i}}$	$\mathit{f}\mathit{u}\mathit{n}$	$\mathit{b}{\mathit{b}}_{\mathit{i}}$	$\mathit{c}{\mathit{c}}_{\mathit{i}}$	${\mathit{a}}_{\mathit{i}}$	${\mathit{f}}_{\mathit{i}}$
EE	log	4.313	1	1	1.604
ELLROD1	log	4.603	1	1	0.601
ELLROD2	log	4.107	1	1	0.658
VWS	SQRT	7.239	100	0	−3.995
DUTTON	SQRT	0.547	1	0	−0.131
CAT1	log	3.533	1	1	0.488
DEF	SQRT	2.14	1	0	−2.773
GRADT	SQRT	4.697	100	0	−2.549
CAT2	SQRT	2.335	100	0	−1.503

, are obtained from the best-fit between the quantiles of observations and each turbulence predictor. The best-fit was obtained using a simple linear regression model, where the coefficients ${\mathit{bb}}_i$ and $f$i were determined using the method of least squares. In this procedure, the predictors could be the turbulence indices or a function of a given turbulence index. The tested functions were the logarithmic (log) and square root (SQRT). The use of these functions produces smoother fields. The SQRT function provides better performance for five predictors, whereas the logarithm provides better skill for the other four predictors (not shown). The regression fits for SQRT (VWS) and log(CAT1 + 1) are illustrated in Figure 6. Table 3 also indicates the function used for each turbulence indicator.

$$I{T}_i=b{b}_i\times fun\left(\max \left(0,T{u}_i\times c{c}_i+a_i\right)\right)+f_i$$

(3)

3.3.2. Evaluation of the Individual Turbulence Predictors

In this section, the assessment of the performance of the turbulence indicators expressed in Equation (3) (see also Appendix A) and two additional turbulence diagnostics based on $R_i$ (RICH1 and RICH2) are presented. These indexes are defined as:

$$\mathrm{R}\mathrm{I}\mathrm{C}\mathrm{H}1=MIN(10,MAX\left(-0.01,aa\right));aa=5.6-2.2\log \left(MAX\left(R_i,0.09\right)\right)$$

(4)

$$\mathrm{R}\mathrm{I}\mathrm{C}\mathrm{H}2=MAX\left(-0.01,aa\right);aa=10\left[\left(1-R_i/10\right)\right]$$

(5)

Sharman and Pearson [40] used the inverse of the Richardson number. Figure 7 compares the decrease of 1/$R_i$, RICH1, and RICH2 with $R_i$, illustrating that RICH1 decays with $R_i$ more steeply than RICH2, but more slowly than 1/$R_i$.

The forecasting performance of all turbulence predictors is evaluated against the DEVG and AIREP data for 12 months. The verification data utilized here is independent of the training data (used in Section 3.1), though cover the same period and have similar characteristics (not shown).

The performance skill of the turbulence indicators to discriminate between MOG turbulence and weaker (NON and LGT) turbulence is depicted in Figure 8. The ROC curves show that five indicators (VWS, DUTTON, ELLROD indices, and EE) have the highest area under the ROC curve (with AUC varying between 0.730 for ELLROD2 and 0.756 for VWS). This indicates that these turbulence predictors perform similarly, outperforming the other predictors (Figure 8a). On the other hand, CAT2 and GRADT have an ROC curve closer to the diagonal, demonstrating the worst performance, with AUC = 0.620 and AUC = 0.655, respectively. In terms of ROC curves, RICH2 and CAT1 reveal an intermediate skill, with AUC$~$0.71.

Nonetheless, it is worth mentioning that other verification measures are more suitable for assessing the forecasting skill of rare events, namely SEDI and SEDS (see Section 2.3). These scores confirm that VWS and DUTTON have the best forecasting performance. The ELLROD and EE indices have slightly lower skill than VWS and DUTTON. The poor performance of CAT2 and GRADT is also confirmed by SEDI and SEDS (Figure 8b,c). However, there are two interesting differences between the SEDI and SEDS scores. The first difference is related to the choice of the optimal thresholds of the turbulence predictors. SEDI peaks at threshold values between 3 and 5.5 for most predictors (Figure 8b). In contrast, in general, SEDS increases as the threshold increases up to 8 to 10 (Figure 8c). This increase is accompanied by a decrease in POFD (Figure 8d) and BIAS (not shown). For thresholds lower than 4.5, BIAS has values greater than 7, revealing a large overestimation of MOG events (not shown). In the case of RICH1, SEDS achieves the maximum value of 0.25 at a threshold of 8 and SEDI reaches the maximum value of 0.45 at a threshold of 3. The second difference between SEDI and SEDS concerns the evaluation of the RICH2 skill. In terms of SEDI, RICH2 shows an intermediate performance, while in terms of SEDS, RICH2 (along with GRADT) performs the poorest. Note that RICH2 has the highest POFD values, revealing the highest tendency to over-predict MOG events (Figure 8d). These differences between SEDI and SEDS suggest that SEDS penalizes over-prediction more than SEDI. This result is consistent with Goecke and Machulskaya’s [26] study, stating that SEDI favors POD more than it penalizes false alarms.

3.3.3. Combination of Turbulence Diagnostics

As mentioned above, an integrated approach reveals a considerable improvement of the skill of turbulence forecast [2,10,13,26]. Moreover, turbulence predictors have been normalized by the local Richardson number [15] because previous studies found that the introduction of this normalization leads to a better agreement between forecasts and observations, e.g., [40]. These turbulence diagnostics can be combined into a single turbulence index using a weighted average of several indices, where the weights are given by the AUC, following the approach of Sharman et al. [2] and Sharman and Pearson [40]:

$$\mathrm{MULTI}={{\displaystyle \sum }}_{i=1}^N{w}_i\max (0,I{T}_i),\hspace{1em}{w}_i=AU{C}_i^2/{{\displaystyle \sum }}_{i=1}^{N}AU{C}_i^2$$

(6)

where $N$ = 9 and $I{T}_i$ are defined in Equation (3) and Table 3, using the calibration approach described in Section 3.3.1.

The performance of a combined index depends on the predictors used. Therefore, a sensitivity study is given in this section.

Table 4. Combined turbulence indices and their weighting factors.

Predictor	AUC	Combined Turbulence Index
		MULTI	MULTI6	MULTI5	MULTI3
EE	0.744	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2
ELLROD1	0.735	$w_{i }$ = AUC2	$w_{i }$ = 0	$w_{i }$ = AUC2	$w_{i }$ = 0
ELLROD2	0.730	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2
VWS	0.756	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = 0
DUTTON	0.746	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = AUC2
CAT1	0.703	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = 0	$w_{i }$ = 0
DEF	0.669	$w_{i }$ = AUC2	$w_{i }$ = AUC2	$w_{i }$ = 0	$w_{i }$ = 0
GRADT	0.655	$w_{i }$ = AUC2	$w_{i }$ = 0	$w_{i }$ = 0	$w_{i }$ = 0
CAT2	0.620	$w_{i }$ = AUC2	$w_{i }$ = 0	$w_{i }$ = 0	$w_{i }$ = 0

presents the different combined indices. The MULTI6 index combines six indices, excluding the two worst-performing indices and ELLROD1 (which is highly correlated with ELLROD2). The MULTI5 index combines the five best-performing indices. The MULTI3 index combines only the EE, ELLROD2, and DUTTON indices.

The indices described in Equation (6) and Table 4 can also be combined with RICH1 or RICH2 as follows:

$$\mathrm{MULTI}-\mathrm{RI1}=\left(1-coef\right) \mathrm{M}\mathrm{U}\mathrm{L}\mathrm{T}\mathrm{I}+coef\times \mathrm{R}\mathrm{I}\mathrm{C}\mathrm{H}1$$

(7)

$$\mathrm{MULTI}-\mathrm{RI2}=\left(1-coef\right) \mathrm{M}\mathrm{U}\mathrm{L}\mathrm{T}\mathrm{I}+coef\times \mathrm{R}\mathrm{I}\mathrm{C}\mathrm{H}2$$

(8)

$$\mathrm{MAX}-\mathrm{RI2}-\mathrm{m}=min\left(max\left(I{T}_i\right),\mathrm{R}\mathrm{I}\mathrm{C}\mathrm{H}2\right)$$

(9)

Figure 9a shows the performance of the combined indices in terms of SEDI and SEDS scores for unbiased forecasts (BIAS$~$1). The importance of comparing unbiased predictions when using SEDI and SEDS to evaluate forecasting skills was stressed by Ferro and Stephenson [30]. This is particularly important for SEDI because this measure penalizes underprediction more than overprediction. Figure 9a reveals different outcomes. First, MULTI3 index performs similarly to MULTI5, MULTI6, and MULTI indices, indicating that adding other highly correlated indices or indices with a worse performance has no positive impact on forecasting skill. Secondly, the use of the Richardson number is beneficial. Moreover, the use of RICH2 is more advantageous than the use of RICH1 and this benefit is more noticeable as more predictors are utilized. Thus, MULTI-RI2 and MULTI6-RI2 outperform all other indices. This result was also verified in terms of the ability to predict SEV turbulence (not shown).

Figure 9b shows the performance of the combined indices using RICH1 and RICH2 (see Equations (7)–(9)) and depicts their skill as a function of the weighting coefficient (coef). It is clear that, as the coef increases from 0.1 to 0.25, the forecasting skill of the indices increases, reaching its maximum for coef = 0.25. This result was also confirmed by the SEDS and AUC scores (not shown).

Figure 10 shows the performance of the best-combined turbulence indices to correctly capture MOG turbulence. It also compares their skill to the DUTTON index (one of the most skillful turbulence diagnostics). The area under the ROC curve is considerably higher for the combined indices using R_i than for the DUTTON index, illustrating the higher skill of these indices when compared to the individual turbulence diagnostics. The other verification measures confirm this result (Figure 10b–d). In terms of SEDI and TSS scores, the use of RICH1 and RICH2 appears to be equally beneficial. However, SEDS reveals that the use of RICH2 is more advantageous than the use of RICH1 (Figure 10d).

Figure 10 also shows that the optimal threshold depends on the verification measure. SEDI and TSS peak for the same thresholds for all turbulence indices (Figure 10b,c). In contrast, SEDS reaches the maximum values for higher thresholds than the other scores (Figure 10d). As mentioned in Section 3.3.2, this reflects the fact that SEDS penalizes overprediction more than SEDI. This is discussed in more detail in the next section.

3.3.4. The Operational Turbulence Index and Its Forecasting Skill

In the previous section, it was shown that MULTI6-RI2 and MULTI-RI2 outperform the other indices. Thus, considering the tradeoff between performance and efficiency, the turbulence index which is operationally used at IPMA (${\mathrm{TURB}}_{\mathrm{IPMA}}$) is the MULTI6-RI2 index with coef = 0.25 (see Section 3.3.3). In this section, the performance of ${\mathrm{TURB}}_{\mathrm{IPMA}}$ is discussed in detail.

Figure 11 compares the scores that evaluate the performance of ${\mathrm{TURB}}_{\mathrm{IPMA}}$ concerning the forecast of MOG and SEV turbulence. From this figure, it is evident that the optimal threshold depends on the verification measure, as shown in Figure 10. It is noteworthy that, when using the TSS score, the optimal threshold is the same for both MOG and SEV turbulence classes (Figure 11a), which is not acceptable in an operational forecasting system. Moreover, in terms of TSS, ${\mathrm{TURB}}_{\mathrm{IPMA}}$ appears to be considerably more skillful at correctly distinguishing between SEV turbulence and other classes than it is at discriminating between MOG and other turbulence classes. This may be explained by the fact that, as the rarity of the predicted event increases (the base rate decreases), the contingency table becomes overwhelmingly dominated by the correct predictions of non-events. In this case, TSS can thus be maximized by maximizing the POD [27], regardless of the bias rate. This applies here in the evaluation of severe turbulence forecasts, for which the base rate is 0.002. Note that, for a threshold of 4 to 4.25, TSS is maximum ($~$0.92) and POD $\ge$0.97 (Figure 11a,c), but the BIAS $\ge$ 30 (

Table 5. Scores of ${\mathrm{TURB}}_{\mathrm{IPMA}}$ concerning the forecast of MOD and SEV turbulence.

Scores	Thresholds for MOD Turbulence
	4	4.5	5 1	5.25	5.5
POD	0.73	0.64	0.52	0.47	0.41
POFD	0.08	0.03	0.02	0.01	0.01
BIAS	5.42	2.58	1.53	1.20	0.97
SEDI	0.82	0.80	0.76	0.74	0.70
SEDS	0.48	0.60	0.64	0.66	0.65
Scores	Thresholds for Severe Turbulence
	4.25	7.0	7.5	7.75	8
POD	0.97	0.47	0.31	0.25	0.19
POFD	0.05	0.00	0.00	0.00	0.00
BIAS	30	1.42	0.61	0.44	0.31
SEDI	0.98	0.81	0.74	0.71	0.68
SEDS	0.45	0.74	0.75	0.74	0.74

¹ Threshold used for MOD turbulence.

), revealing a large overestimation of severe turbulence events.

Figure 11 also shows that, concerning the forecast of MOG turbulence, SEDS peaks at the threshold of 5.25 when POD is 0.47 and the BIAS is 1.2. In contrast, TSS peaks at the threshold of 4 (Figure 11a), when BIAS $~$ 5.4 (Table 5). In addition, according to SEDS, the optimal threshold for forecasting severe turbulence is 7.5 (Figure 11b). In this case, BIAS $~$ 0.6 and POD = 0.31. In the operational practice, a forecaster can use a slightly lower threshold, for instance 7, which guarantees a higher POD (0.47), with a BIAS $~$ 1.4 (Table 5).

It is important to note that the choice of the optimal threshold should take into account that, for certain meteorological phenomena, the cost of a false negative (miss) is worse than a false alarm. In this case, it is better to have a moderately biased forecast, which guarantees a certain probability of detection. Following this reasoning, operationally, for the forecast of moderate turbulence, the threshold chosen is 5 (see the corresponding

Table 6. A 2 × 2 contingency table for the verification of the index ${\mathrm{TURB}}_{\mathrm{IPMA}}$ for a threshold of 5. MOG turbulence is considered an event. In this case, the base rate is 0.016.

Forecast	Observation
	No	Yes
$\mathrm{T}\mathrm{U}\mathrm{R}{\mathrm{B}}_{\mathrm{I}\mathrm{P}\mathrm{M}\mathrm{A}}$ < 5 (No)	18,727	148
$\mathrm{T}\mathrm{U}\mathrm{R}{\mathrm{B}}_{\mathrm{I}\mathrm{P}\mathrm{M}\mathrm{A}}$ ≥ 5 (Yes)	310	158

). For this threshold, SEDI and SEDS values are close to their maximum values and, simultaneously, POD = 0.52 and BIAS $~$ 1.5. For a higher threshold, POD is too low, while, for lower threshold values, BIAS is too high (see Table 5).

3.4. The Operational Turbulence Index

In the operational practice, a final step is applied to the ${\mathrm{TURB}}_{\mathrm{IPMA}}$ index. The maximum value of ${\mathrm{TURB}}_{\mathrm{IPMA}}$(TURB_MAX) and average value for ${\mathrm{TURB}}_{\mathrm{IPMA}}\ge 5$ (TURB_MEAN) in a given layer is calculated. The final index (TURB_LAYER) is the average of TURB_MAX and TURB_MEAN. On 20 February 2021, the Canadian and Portuguese MWOs issued turbulence SIGMETs, respectively, for Gander and Santa Maria Oceanic FIRs. These turbulence zones lie on the western edge of an upper-level trough (not shown). Figure 12 shows the TURB_LAYER and turbulence product from London WAFC, both for the FL300/390 layer. Note that the scales of these products differ since the latter product was calibrated with EDR data [15]. An EDR greater than 0.22 and 0.34 m^2/3 s^–1 indicates moderate and severe intensities of turbulence for mid-size aircraft [15]. Both products predict a large area of turbulence associated with the upper-level trough, which extends to middle levels (not shown). Despite the similarities between the spatial patterns of these two indices, there are some differences regarding turbulence intensities. For example, ${\mathrm{TURB}}_{\mathrm{IPMA}}$ predicts favourable conditions for severe turbulence northwest of the Azores and over the central Azores archipelago. In contrast, the WAFC EDR product predicts only severe turbulence in two small areas, one over the Canary Islands and one northwest of the Azores archipelago in the FL240/300 layer (not shown) and no severe turbulence in the FL300/390 layer. It is also worth mentioning that both products appear to underestimate the severity of the turbulence reported in the northeastern region of the Iberian Peninsula and also in the Atlantic west of the Azores. One of the severe turbulence AIREPs in the Iberian Peninsula was due to mountain wave turbulence. It is expected that the IPMA index underestimates mountain wave turbulence because it does not include specific mountain wave diagnostics (as proposed by Kim et al. [14,15]). Moreover, diagnostics derived from a non-hydrostatic model with higher resolution (grid spacing < 3 km) would be more suitable for this purpose.

4. Conclusions

The performance of several turbulence diagnostics derived from ECMWF forecasts are evaluated over Portuguese flight information regions (FIR) and surrounding areas for the period February 2020 to March 2021, excluding May and June. In addition, the algorithm developed and used operationally by aviation meteorologists at IPMA to forecast moderate and severe turbulence over Portuguese FIRs is also discussed. The forecasts were compared with turbulence observations from special air reports and DEVG data from AMDARs received at the Portuguese MWO.

Previous studies have shown that a combined turbulence index, using multiple turbulence diagnostics instead of using only one turbulence diagnostic, leads to a considerable improvement in the turbulence forecasting skill [2,10,13,26]. The present study uses a new approach to combine different NWP-based turbulence diagnostics to obtain the turbulence index used operationally in IPMA (${\mathrm{TURB}}_{\mathrm{IPMA}}$). This index combines six turbulence diagnostics (VWS, DUTTON, ELLROD2, EE, CAT1, and DEF) with a new function of Richardson number (RICH2). The choice of these predictors and of the weight given to RICH2 were established through a sensitivity analysis. The use of RICH2 has proven to be beneficial, even when compared to another Ri function. The VWS, DUTTON, EE, and ELLROD2 indices outperform the other turbulence diagnostics. Thus, the weight coefficient for these indicators is higher than for the other two predictors, as has been shown in previous studies [2,10,13,26].

The objective verification approach in this paper uses not only the Relative Operating Characteristic curves but also novel measures such as the recently proposed Symmetric Extreme Dependence Index (SEDI) and Symmetric Extreme Dependence Index (SEDS). These measures are particularly suitable for assessing the forecasting skill of rare events [30,33] such as moderate or greater turbulence, which accounts for 1.6% of the total data.

The prediction of moderate and severe turbulence depends on the choice of the optimal threshold. However, this optimal threshold varies with the verification measure used. The results show that TSS and SEDI achieve a higher value for lower thresholds compared to SEDS. This is because, when the contingency table becomes dominated by the correct predictions of non-events, both TSS and SEDI penalize under-prediction more than over-prediction. The referred drawback of SEDI can be minimized by forcing the forecast to be unbiased (BIAS = 1), as suggested by Ferro and Stephenson [30]. In this regard, the use of SEDS is beneficial, especially as the rarity of the event increases. To the author’s knowledge, this is the first study using SEDS to evaluate the performance of turbulence forecasts. The properties of SEDS allow its application to assess the forecast performance of severe turbulence (a rare event), which is usually not addressed. In general, previous studies evaluate the forecasting skill of forecasts of moderate-or-greater turbulence [2,10,15,22,23]. However, this information is insufficient for aviation forecasters, who must issue SIGMETs when severe turbulence is expected to occur.

Studies comparing the performance of the ECMWF model with other models in terms of turbulence diagnostics are still lacking. Therefore, it would be worthwhile to compare the IPMA (ECMWF-based) and WAFC turbulence products for a sufficiently long period (such as two years). Furthermore, in the future, it would be relevant to investigate the impact of using other turbulence diagnostics, such as divergence and vertical velocity, and adding to the operational index mountain wave turbulence diagnostics as used by Kim et al. [15].