A Runway Overrun Risk Assessment Model for Civil Aircraft Based on Quick Access Recorder Data

Abstract

The quick access recorder (QAR), as an airborne device used to monitor and record flight parameters, has been widely installed on various types of aircraft. Based on QAR data, research on runway overrun, a typical flight safety incident, has attracted widespread attention in recent years. However, existing runway overrun risk models generally suffer from oversimplified risk metrics or insufficient consideration of risk dynamics. In this paper, we propose a new dynamic runway overrun risk assessment model based on QAR data. We first consider the noise of aircraft trajectory data in the QAR parameters and present a landing trajectory correction method combining ground speed and runway position information. Second, to improve the accuracy of the risk assessment model, we design an algorithm to automatically recognize the aircraft autobrake level during the landing phase, based on which a new dynamic risk assessment model is developed. Finally, feature engineering is performed to extract the relevant contributing factors of runway overrun risk, based on which classification and regression models are applied to identify risky flights and predict the risk values. The proposed risk assessment model was evaluated using QAR data from an airline in China. The results show that the automatic deceleration rate, the way that the aircraft approaches the runway, the touchdown distance, and the kinetic energy at 50 ft are key factors in the risk of runway overrun during the landing phase.

1. Introduction

Flight safety is a primary focus of the civil aviation industry [1,2]. According to Boeing’s statistical summary of commercial jet airplane accidents from 1959 to 2018 [3], the approach and landing are the flight phases during which safety incidents are most likely to occur. The landing phase only accounts for an average of 1% of the total flight time; however, the occurrence rate of flight incidents for this phase is as high as 24%. Therefore, the landing phase is a critical phase for ensuring flight safety.

Runway overrun is a flight safety incident in which an aircraft fails to slow down in time after landing, resulting in it running off of the runway. This may cause serious economic losses to airlines and even endanger the lives of passengers in severe cases. According to the runway safety accident analysis report of the International Air Transport Association (IATA) [4] published in 2015, runway overrun incidents accounted for 44% of the 78 runway excursion incidents that occurred from 2010 to 2014, and they accounted for 100% of the five fatal accidents. Therefore, it is crucial to study the risk of runway overrun for ensuring civil aviation safety.

The QAR (quick access recorder), as an airborne data recorder, can collect and store multidimensional flight parameter data during a flight. Currently, QAR devices have been widely installed on various types of aircraft. It has been proven by practical experience that the QAR can effectively collect thousands of real-time parameters during flight. These flight parameters can reflect the condition of the external environment, flight attitude, aircraft status, pilot operations, etc. Therefore, they provide objective evidence for flight technical evaluations, safety incident investigations, and the elimination of potential flight safety hazards, which can yield better flight safety. Consequently, research related to civil aviation flight safety based on QAR data has attracted significant attention from researchers in recent years [5,6,7].

Regarding the risk of runway overrun for commercial aircraft, existing research can be broadly categorized into two types. The first type of research focuses on post hoc analysis of runway overrun incidents based on historical accident data [8,9,10,11,12]. These studies mainly employ probabilistic statistical analysis methods, combined with accident investigation reports, to analyze the causes of runway overrun accidents and propose corresponding improvement measures. The main problem with this type of research is that there is a limited number of accident data samples. Therefore, it is difficult to fully explore valuable information within vast QAR data, thus leading to significant limitations. The second type of research focuses on flights where runway overrun incidents did not occur and uses QAR data to analyze the relative risk levels of different flights. The main issue with this type of research is that the risk assessment metrics are oversimplistic and fail to consider the dynamic runway overrun risk due to the pilot’s deceleration operation after touchdown.

To address the aforementioned issues, we propose a new dynamic risk assessment model for runway overrun in this paper. This model fully considers the dynamic risk from the moment the aircraft touches down until it exits the runway, including the impact of the pilot’s braking operations. It provides a more comprehensive approach to assess runway overrun risk. We first correct the landing trajectory data of the aircraft so that the aircraft position relative to the runway can be accurately determined. Then, we design an algorithm to recognize and extract the autobrake level of the aircraft during the landing phase. Based on the recognized autobrake level, a dynamic assessment model for runway overrun risk is constructed. Then, relevant risk factors for runway overrun are extracted by applying feature engineering methods. Furthermore, classification and regression models are employed to classify risky flights and predict risk metrics. Finally, we use a real QAR dataset provided by a domestic airline to validate the effectiveness of our model.

The rest of this paper is organized as follows: Section 2 briefly reviews the related work; Section 3 introduces the correction of landing trajectory data; Section 4 illustrates the dynamic runway overrun risk model; in Section 5, we analyze the runway overrun risk with classification and regression models; and Section 6 concludes this paper.

2. Related Work

2.1. Runway Overrun

As mentioned above, current studies on the risk of runway overrun can be roughly divided into two categories, i.e., studies based on real historical accident data and studies based on QAR data. In this section, we give a brief overview of the related work in these two categories.

For studies based on historical accident data, Kirkland et al. [8] attempted to normalize historical accident data to facilitate future research. They further adopted bivariate analysis to build a probabilistic model for the risk factors of runway overrun incidents [9]. Their model mainly considers factors such as the aircraft weight, tailwind, light conditions, weather, approach speed, and touchdown point. Valdés et al. [10] added factors such as the aircraft type, airport elevation, and safety area length to their probabilistic model. Ayres et al. [11] built a probabilistic model that considers the spatial distribution of accident locations to describe runway overrun and excursion accidents. Wagner et al. [12] studied the severity of accident consequences using logistic regression and Bayesian logistic regression methods based on over 1400 runway overrun and excursion data from an ACRP database from 1970 to 2009, which included five types of accidents: runway overrun and runway excursion in both takeoff and landing phases and undershoot.

For studies based on QAR data, machine learning and risk assessment models are mainly employed to analyze QAR data, thereby identifying key factors contributing to the runway overrun risk. Wang et al. [13,14] analyzed long landing incidents with a risk of runway overrun and used variance analysis and linear regression (LR) methods based on QAR data to analyze different factors of long landing. Subsequently, Wang et al. [15,16] proposed a runway overrun risk assessment model based on QAR data, defining the long landing risk as the product of the probability of a certain landing distance and the severity of risk corresponding to that landing distance. Kang et al. [17,18] investigated the long landing problem and proposed a deep sequence-to-sequence model to predict the landing speed and distance. Lv et al. [19] defined runway overrun risk as a function related to the remaining runway distance and touchdown speed, dividing flights into high-risk and low-risk flights based on the magnitude of the risk indicator. They ultimately employed machine learning algorithms for high-risk flight classification. Ayra et al. [20] took the remaining runway distance when the aircraft’s ground speed reaches 80 knots as a measure of runway overrun risk. They used Bayesian networks to analyze the influencing factors of runway overrun risk, including the crosswind, tailwind, surface contamination, approach mode, autobrake usage, and entry altitude. To address the problem of a lack of positive samples for runway overrun, Koppitz et al. [21] employed subset simulation methods to calculate the changes in the probability of runway overrun incidents based on selected factor distributions, thereby identifying relevant risk factors.

The limitations of the existing studies are as follows: For studies based on historical accident data, they can only provide coarse-grained information, such as the weather condition, aircraft weight, aircraft age, etc. Therefore, compared to QAR data, the available information from historical accident data are very limited, and it is difficult to uncover valuable in-flight information to support flight safety analysis. For studies based on QAR data, the current risk assessment metrics are usually oversimplistic and they failed to consider the dynamic runway overrun risk of the pilot’s deceleration operation after touchdown.

2.2. Other QAR-Based Flight Safety Studies

In addition to studying runway overrun, some other studies have been performed regarding flight safety issues based on QAR data, among which the main category is exceedance events. Qi et al. [22] proposed a new method for partitioning the risk subspaces of exceedance events based on rough set theory and a particle swarm multiobjective optimization algorithm. Liu et al. [23] developed a risk assessment model for exceedance events, defining exceedance risk as the product of the probability of exceedance events and the severity of the events. They developed a quality assessment system for pilot operations based on their model. Wang et al. [24] investigated the relationship between the risk cognition of pilots and exceedance events based on QAR. Li et al. [25] investigated the tail strike risk and proposed an unsupervised learning method to discover patterns of unsafe pilot stick operations during the landing stage.

Hard landing is another typical flight safety incident that researchers are concerned about. Hu et al. [26] designed a prediction model based on a support vector machine (SVM) for hard landing. Qiao et al. [27] employed RBF neural networks and the K-means clustering algorithm to predict hard landing. Tong et al. [28] addressed the problem of hard landing using a deep learning framework. Considering the temporal characteristics of QAR data, they proposed a hard landing prediction framework based on long short-term memory (LSTM) networks. Additionally, they applied LSTM networks to predict aircraft landing speeds [29]. Chen et al. [30] used scalar measurements and aggregated QAR data to detect the influential features of hard landing. Recently, Li et al. [31,32] performed automatic classification and identification for the causes of hard landing using the K-means clustering algorithm. Chen et al. [33] proposed a deep learning neural network model with time-aware attention for interpretable hard landing prediction. Jin et al. [34] developed transfer learning methods for high-dimensional quantile regression and applied the methods to solve the problem of determining the hard-landing risk for flight safety.

In terms of abnormal flight analysis, Li et al. [35,36,37] applied clustering and outlier detection methods to identify abnormal flights from massive QAR data. Later, Li et al. [38] improved their model to detect the specific flight phase of abnormal flights in which the QAR parameters deviate from their normal behavior.

3. Landing Trajectory Data Correction

3.1. Data Preprocessing

The dataset used in this paper consists of QAR data for aircraft of the same model (A320) landing at one airport in China. The QAR data have been decoded and each flight corresponds to a CSV file, with a total of 180 flights. Each CSV file contains multiple rows, with each row corresponding to one second, covering multiple QAR parameters. In this paper, we first preprocess the data through the following steps:

Data cleaning: In the original QAR data, there are some decoding or sensor errors, which may cause some incorrect parameters or information loss. Therefore, it is necessary to filter out these fields or complete missing information. In addition, some CSV files are incomplete and do not cover the entire flight process from takeoff to landing, while others are flight training data with the same departure and arrival airports. These flights are filtered out through the data cleaning step.

Conversion of discrete state parameters: Some QAR parameters are discrete state variables. For example, the landing gear state is either ‘AIR’ or ‘GROUND’, corresponding to the air–ground switching of landing gears. AIR represents that the landing gear is in the air, while GROUND represents that the landing gear is on the ground. To show the landing gear state of the aircraft, these two states need to be converted into binary values. In this paper, to obtain more intuitive results, AIR is represented as 1, and GROUND is represented as 0. Similar operations are performed for other discrete state parameters, such as the activation of the autopilot and the flap position.

Parameter transformation: In addition to the aforementioned data processing, some parameters require advanced transformations, such as the wind speed and wind direction. The crosswind and tailwind during the landing phase are important factors related to flight safety incidents. However, the crosswind and tailwind information were not included in the original QAR parameters. Hence, it is necessary to combine the aircraft magnetic heading (flying direction) with wind speed and wind direction parameters to calculate the effect of wind on the aircraft. Given the wind direction $\beta$ and aircraft magnetic heading $\alpha$, we apply trigonometric functions to calculate the components of wind speed that are parallel and perpendicular to the magnetic heading of the aircraft, as shown in the following equation:

$$\theta =\beta -\alpha$$

(1)

$$WI{N}_{CRS}=WI{N}_{SPD}\times sin\theta$$

(2)

$$WI{N}_{ALG}=WI{N}_{SPD}\times cos\theta$$

(3)

where $\alpha$ is the magnetic heading of the aircraft, $\beta$ is the wind direction, $WI{N}_{SPD}$ represents the wind speed, $WI{N}_{CRS}$ represents the crosswind component, and $WI{N}_{ALG}$ represents the headwind component ($WI{N}_{ALG}<0$ corresponds to the tailwind).

3.2. Landing Interval Extraction

The landing phase only accounts for approximately 1% of the entire flight process. It is necessary to extract the landing interval from the QAR data for more efficient analysis. Therefore, we first identify the touchdown moment (i.e., when one of the landing gears first touches the ground) during the aircraft landing process. Then, a 120 s interval is selected by taking 20 s before the touchdown moment and 100 s after it, which serves as the landing interval, as shown in Figure 1.

3.3. Landing Trajectory Correction

We first visualize the landing trajectory of a flight with its latitude and longitude parameters. Figure 2a shows the visualization of the trajectory obtained with the Folium map visualization tool (https://python-visualization.github.io/folium/, accessed on 25 August 2023). In the visualization, the green location marker represents the touchdown moment of the aircraft, and the red location marker represents the last trajectory point of the 120 s interval. The trajectory points are sampled at a frequency of 1 Hz, meaning that the time interval between any two adjacent points is 1 s. Generally, the overall shape of the landing trajectory fits the runway. However, when the trajectory is enlarged, as shown in Figure 2b, it can be observed that there are local deviations from the runway in terms of latitude and longitude. Since the landing trajectory information is crucial for the analysis of runway overrun risk, it is necessary to correct the landing trajectory to meet the analysis requirements. In this paper, we correct the landing trajectory data through the following steps.

3.3.1. Trajectory Correction Based on Projection

Based on the prior knowledge that an aircraft must run along the runway, we first project the aircraft’s landing trajectory onto the centerline of the runway. Specifically, given the latitude and longitude coordinates of the two ends of the runway centerline, the runway direction is calculated first based on these coordinates. Then, the latitude and longitude coordinates of the aircraft’s starting point and end point on the runway are denoted as ($x_1$, $y_1$) and ($x_2$, $y_2$), respectively, where x represents the latitude, y represents the longitude, and both are converted to radians. The calculation for the runway direction is as follows:

$$\alpha =arctan\frac{sin\Delta y\times cos{x}_2}{cos{x}_{1}sin{x}_2-sin{x}_{1}cos{x}_{2}cos\Delta y}$$

(4)

where $\alpha$ represents the angle between the runway direction and the north direction taken clockwise and $\Delta y=y_2-y_1$ is the difference in longitude. Additionally, the distance between any two points with latitude and longitude coordinates ($x_1$, $y_1$) and ($x_2$, $y_2$) can be calculated as follows:

$$d=2Rarcsin\sqrt{{sin}^2\frac{\Delta x}{2}+cos{x}_{1}cos{x}_2{sin}^2\frac{\Delta y}{2}}$$

(5)

where R is the radius of the Earth (approximately 6371.393 km), and $\Delta x=x_2-x_1$ is the difference in latitude. Based on the above formulas and the projection formula in vector space, a point on the aircraft’s landing trajectory can be projected onto the runway centerline. The effect of trajectory correction based on projection is shown in Figure 3, where the original trajectory is represented in blue and the projected trajectory is represented in red. It can be observed that the red trajectory aligns perfectly with the centerline of the runway.

Although the trajectory correction method based on projection addresses the lateral offset error of the landing trajectory on the runway, it can be seen that the distance between adjacent projected points along the runway direction fluctuates significantly, and it cannot reflect the actual landing situation of an aircraft, as the speed of the aircraft during the landing process will not fluctuate as dramatically.

3.3.2. Trajectory Correction Based on Projection and Ground Speed

Based on the above analysis, we further combine the runway projection and ground speed information during the landing phase to improve the correction of the aircraft’s landing trajectory. First, the touchdown point is selected and projected onto the runway as a reference point. Second, based on the ground speed, the distance between the next trajectory point and the reference point is calculated. Then, combining this distance with the runway direction, the coordinates of the next trajectory point are determined. Additionally, the trajectory point where the aircraft’s ground speed is reduced to 30 knots is projected onto the runway. Finally, the trajectories of the other points between the two reference points are corrected based on the ground speed. The final correction results are shown in Figure 4, where the corrected trajectory aligns with the runway and exhibits a more uniform variation, which is more consistent with actual situations.

4. Dynamic Runway Overrun Risk Model

4.1. Basic Idea

Research on the risk of runway overrun for civil aircraft based on QAR data faces the main challenge of defining the risk. This is mainly because real runway overrun accidents are rare, while the QAR device has been widely used only in recent years. Therefore, to enable early warning, researchers tend to focus on flights with a high risk of runway overrun, even if there is no actual runway overrun accident.

Considering that the risk of runway overrun is mainly determined by the position and ground speed of the aircraft at the moment of touchdown, we develop a risk assessment model for runway overrun based on risk areas. It assumes that, after landing, the aircraft decelerates at a fixed rate until it stops (Runway Excursion Risk Assessment Diagram, https://flightsafety.org/files/iass/2011-proceedings/Fabregas.pdf, accessed on 25 August 2023). Specifically, as shown in Figure 5, we first construct a two-dimensional diagram, where the origin point corresponds to the end of the runway, the horizontal axis represents the remaining runway length at touchdown for a specific flight, and the vertical axis represents the ground speed at touchdown. Second, the position and ground speed of the aircraft at the touchdown moment are extracted based on the QAR data. In this way, each flight is represented as a data point $(x,y)$ on the diagram, where x is the remaining runway length at touchdown and y is the aircraft ground speed at touchdown, as shown by the circular markers. Then, starting from the origin, a curve with constant acceleration is drawn. The horizontal axis of the curve represents the distance, and the vertical axis represents the speed. Obviously, the red curve in Figure 5 is a quadratic curve of distance with respect to speed. The physical meaning of this curve is as follows. First, if the touchdown point of a specific flight is located on the curve, then if the aircraft decelerates at the rate given by the curve it will stop exactly at the end of the runway when its speed reduces to zero. That is, if the touchdown point of a flight is above the curve then decelerating at the rate given by the curve will not allow the aircraft to stop at the end of the runway; the aircraft will run off the runway, resulting in a runway overrun accident. Therefore, the area above the curve indicates that a runway overrun incident may occur, while the area below the curve is safe. The distance between a sample point and the curve reflects the degree of runway overrun risk. Clearly, the smaller the distance is, the greater the risk.

It is worth noting that this curve is often not fixed and depends on factors such as the runway conditions, the weather conditions, and the aircraft’s autobrake level. For example, the deceleration rate on a wet and slippery runway is often smaller, resulting in a larger corresponding risk area for runway overrun. In this section, the risk of runway overrun is studied by drawing inspiration from the above method.

4.2. Automatic Extraction of the Deceleration Rate

To plot the risk curve mentioned above, it is necessary to determine the deceleration rate of the aircraft under different conditions. Since the QAR data used in our study do not include the runway or weather condition information, the focus here is mainly on the deceleration rates under different autobrake levels. Specifically, after touchdown, the autobrake system is activated, and then the aircraft enters the autobrake phase and maintains a relatively constant deceleration rate before manual braking is applied. Therefore, by identifying the autobrake phase, the autobrake level and deceleration rate can be determined based on the average deceleration rate of this phase.

Based on the above considerations, we design an algorithm to extract the deceleration rate during the autobrake phase, which not only identifies the autobrake level but also determines the deceleration rate. The algorithm flowchart is shown in Figure 6, and the specific steps of the algorithm are as follows:

Step 1: Identify the aircraft’s touchdown point based on the landing gear state parameters and determine the time of manual brake intervention based on the brake pedal parameters. In addition, when the parameter value of the brake pedal is particularly small, there may be situations where the autobrake release cannot be triggered. Therefore, in this paper, the time of manual brake intervention is defined as any value of the brake pedal parameters greater than or equal to 4.

Step 2: Starting from the touchdown point, find the time point $t_{start}$ when the engine power N11 first reaches a maximum value of 90%. Choosing 90% instead of the moment of the maximum speed ratio improves the stability of the algorithm.

Step 3: Starting from $t_{start}$, find the first time point when the deceleration rate is stable and replace $t_{start}$ with this time point. The stability of the deceleration rate at any time t is defined as follows. Starting from time point t, take the two preceding deceleration rate values and the three subsequent deceleration rate values and add the deceleration rate value at time point t itself to form an array [$a_{t-2}$, $a_{t-1}$, $a_t$, $a_{t+1}$, $a_{t+2}$, $a_{t+3}$]. Calculate the standard deviation $\sigma \left(t\right)$ of this array, and when it satisfies the condition $\sigma \left(t\right)\le \theta$ the deceleration rate is considered stable; otherwise, it is considered unstable. Here, $\theta$ is a stability threshold parameter, which is set to 0.15 in this paper.

Step 4: Determine whether $t_{mbk}>t_{start}$, i.e., whether manual braking has already been applied before the deceleration rate becomes stable. Here, $t_{mbk}$ is the time point when manual braking begins. If manual braking has already been applied, end the program; otherwise, go to Step 5.

Step 5: Traverse each point starting from $t_{start}$ until a time point with an unstable deceleration rate is found, and mark this point as $t_{end}$.

Step 6: Mark [$t_{start}$, $t_{end}$] as an autobrake phase.

Step 7: Calculate the average value of ground speed (GS) during [$t_{start}$, $t_{end}$] as the automatic deceleration rate.

Step 8: Determine whether there is a next deceleration phase. If so, let $t_{start}=t_{end}+1$ and return to Step 3 to find the next deceleration phase; otherwise, proceed to Step 9.

Step 9: Output all [$t_{start}$, $t_{end}$] as automatic deceleration phases.

We apply the above autobrake level extraction algorithm to the 180 flight samples and record the distribution of automatic deceleration rates for each flight, as shown in Figure 7. From Figure 7, it can be observed that the automatic deceleration rates are concentrated at approximately 1.6 m/s${}^2$ and 2.9 m/s${}^2$, which aligns with the experience of flight experts. The concentration of the distribution also indirectly reflects the accuracy of the autobrake identification algorithm. The differences in automatic deceleration rates may be related to factors such as runway and weather conditions, aircraft, and aircraft weight.

4.3. Establishment of the Risk Assessment Model

Based on the idea of the risk assessment model described in Section 4.1, for a specific flight, we first extract the automatic deceleration rates to plot the theoretical risk boundary curve, as shown by the purple curve in Figure 8. The green vertical line corresponds to the touchdown moment, and the red vertical line corresponds to the moment when the ground speed of the aircraft drops to 30 knots (which is regarded as a safe speed). In the risk model described in Section 4.1, each flight is represented by a single point on the figure, corresponding to the ground speed and remaining runway distance at touchdown, and the risk of runway overrun can be assessed based on the distance from that point to the theoretical curve. The main drawback of this assessment method is that the influence of the pilot’s deceleration operations after touchdown on the risk is not considered. To address this problem, we propose a dynamic risk assessment model. Specifically, we use the modified trajectory points to calculate the remaining runway distance and plot the dynamic ground speed with respect to the remaining runway distance, as shown by the blue scattered points in Figure 8. Then, the distance from each point to the theoretical curve is calculated, and the dynamic risk of runway overrun is measured based on this distance. We select the phase from the touchdown moment to the moment when the ground speed decreases to 30 knots, and we use a weight function of the distance between the ideal curve and the actual aircraft trajectory as the evaluation indicator for the runway overrun risk, which is calculated as follows:

$$risk=\frac{{\sum }_{t=t_1}^{t_2}w\left(v_t\right)·f\left(d_t\right)}{{\sum }_{t=t_1}^{t_2}w\left(v_t\right)}$$

(6)

where $t_1$ and $t_2$ represent the touchdown moment and the moment when the speed drops to 30 knots, respectively. The weight and risk calculation functions are as follow:

$$w\left(v_t\right)=1-{\left(\frac{1}{e}\right)}^{\frac{v_t}{a}}$$

(7)

$$f\left(d_t\right)={\left(\frac{1}{e}\right)}^{\frac{d_t}{b}}$$

(8)

Equation (7) is a velocity-based weight function, which indicates that as the speed v increases the weight function value also increases, and the confidence level of the calculated risk is higher. This is because when the aircraft’s speed is higher it is usually a critical phase for the pilot to perform deceleration operations, and the calculated risk indicator at this time is more reliable. However, when the aircraft’s speed drops to a certain level (such as 40–50 knots), the pilot may intentionally maintain speed to quickly exit the runway, and the confidence level of the risk assessment is lower. Here, a is a control parameter determined by experiments, which is set to 85 in this case.

Equation (8) is the risk calculation function, which indicates that when the distance from a point to the theoretical curve is negative, i.e., the actual trajectory point is to the left of the ideal curve, the risk value is greater than 1. When the distance is positive, i.e., the actual trajectory point is to the right of the ideal curve, the risk value is less than 1. The greater the distance, the lower the risk. Here, b is a control parameter determined by experiments, which is set to 500 in this case.

Figure 9 shows the variation in the risk and weight functions over time for the flight corresponding to Figure 8, from the touchdown moment to the moment when the ground speed decreases to 30 knots. With the original automatic deceleration rate, the aircraft cannot exit from the middle of the runway, and the aircraft needs to turn around at the end of the runway. Therefore, the pilot needs to lightly tap the brakes to deactivate the autobrake and maintain a certain speed to taxi to the end of the runway. Then, the pilot can perform a manual brake to turn around at the end of the runway. If the velocity weighting is not used, there will be a high risk at this time, as shown by the solid red line in Figure 9. However, this does not align with our subjective experience. The dynamic risk curve after weighting is shown by the dashed red line, which is more consistent with the actual situation.

5. Classification and Regression of the Runway Overrun Risk

In this section, we mainly conduct regression analysis and classification research. The former focuses on predicting the risk metrics of runway overrun, while the latter focuses on classifying high-risk flights.

5.1. Regression Analysis of the Runway Overrun Risk

5.1.1. Feature Extraction

Before conducting feature extraction, we preprocess the features, including transforming the discrete variables, filling in missing values, and normalizing features.

Next, the variance of each feature is calculated, and features with a variance of 0 are removed. Then, various methods, including Pearson correlation analysis, model-based feature ranking, and recursive feature elimination, are employed to select the nine features shown in

Table 1. Features related to runway overrun.

No.	Name	Physical Meaning	Correlation Coefficient
1	LEAVE_MANNER	Leave from the middle or end of runway	−0.71
2	ABK_RATE	Automatic deceleration rate	−0.65
3	MBK_POS	Manual brake position	0.48
4	IS_EXIST_HMBK	Is the emergency brake present?	−0.38
5	HMBK_RE_DIS	Remaining runway distance when emergency braking	−0.44
6	TD_POS	Touchdown position	0.33
7	50FT_KE	Kinetic energy at 50 feet	0.34
8	GS_LEAVE_10	Maximum ground speed in the last 10 s before reaching 20 knots	−0.36
9	FLARE_TIME	Time from 50 feet to touchdown	0.35

.

5.1.2. Regression Model and Evaluation Metrics

Regression analysis is performed with Equation (6) as the prediction target, using the nine features from

Table 2. Regression results for runway overrun.

No.	Regression Model	MAE	RMSE	MAPE
1	LR	0.0295 ± 0.0025	0.0416 ± 0.0036	0.1712 ± 0.0242
2	NN	0.0206 ± 0.0024	0.0279 ± 0.0039	0.1129 ± 0.0217
3	DT	0.0401 ± 0.0053	0.0585 ± 0.0081	0.1908 ± 0.0356
4	RF	0.0298 ± 0.0036	0.0426 ± 0.0065	0.155 ± 0.0253
5	SVR	0.0464 ± 0.0063	0.0588 ± 0.0097	0.2853 ± 0.0509

. The machine learning models applied include LR, neural networks (NNs), decision trees (DTs), random forests (RFs), and support vector regression (SVR).

The evaluation metrics include the mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE), which are calculated as follows:

$$\mathrm{MAE}=\frac{1}{n}\sum _{i=1}^m\mid y_i-\widehat{y_i}\mid$$

(9)

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum _{i=1}^m{(y_i-\widehat{y_i})}^2}$$

(10)

$$\mathrm{MAPE}=\frac{1}{n}\sum _{i=1}^m\mid \frac{y_i-\widehat{y_i}}{y_i}\mid$$

(11)

where $y_i$ and $\widehat{y_i}$ represent the ground truth and predicted values of the model, respectively.

5.1.3. Regression Results

In this paper, 60% of the flights are used as the training dataset, and 40% are used as the test dataset. The results are shown in

Table 3. Classification results for runway overrun.

No.	Classification Model	Accuracy	Precision	Recall	F1-Score
1	LR	0.8326 ± 0.0376	0.7536 ± 0.0824	0.7625 ± 0.085	0.7522 ± 0.0513
2	NN	0.899 ± 0.0331	0.8671 ± 0.0637	0.8312 ± 0.0971	0.8439 ± 0.0552
3	DT	0.8056 ± 0.0444	0.7153 ± 0.0836	0.7135 ± 0.0802	0.71 ± 0.0614
4	RF	0.8372 ± 0.0425	0.8103 ± 0.0877	0.6792 ± 0.0916	0.7341 ± 0.07
5	SVR	0.8576 ± 0.0451	0.8018 ± 0.0904	0.7833 ± 0.0773	0.787 ± 0.0562

. Considering the randomness of the algorithms, each algorithm is run 40 times, and the mean and standard deviation of the evaluation metrics are provided in Table 2. The best results are highlighted in bold.

From Table 2, it can be observed that the neural network model has better prediction accuracy than the other algorithms, with an average MAPE of approximately 11.29%. The worst performing algorithms in terms of prediction accuracy are DT and SVR, with average MAPEs of 0.1908 and 0.2853, respectively.

Furthermore, the random forest algorithm is used to provide a feature importance ranking, as shown in Figure 10. The automatic braking rate (ABK_RATE) feature has the highest weight (close to 0.32), followed by the leaving manner (LEAVE_MANNER) and touchdown position (TD_POS) features, with approximate weights of 0.3 and 0.2, respectively. The importance of the other features is below 0.1, which is consistent with previous correlation analysis results.

5.2. The Classification of Flights with Runway Overrun Risk

5.2.1. Classification Model and Evaluation Metrics

For the classification task, we still use the nine features from Table 1. In this paper, all flights are sorted based on their risk values. The top 35% of flights with the highest risk values are selected as positive samples, while the remaining 65% of flights are considered negative samples. The threshold for distinguishing positive and negative samples is set at risk = 0.30. The following classification models are employed: LR, NNs, DTs, RFs, and SVM.

The evaluation metrics include accuracy, precision, recall, and F1-score, with the following formulas:

$$Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$$

(12)

$$Precision=\frac{TP}{TP+FP}$$

(13)

$$Recall=\frac{TP}{TP+FN}$$

(14)

$$F1\text{-}Score=\frac{2Precision·Recall}{Precision·Recall}$$

(15)

where TP, TN, FP, and FN represent the numbers of true positive samples, true negative samples, false positive samples, and false negative samples, respectively.

5.2.2. Classification Results

The classification results are shown in Table 3. Considering the randomness of the algorithms, each algorithm is run 40 times, and the mean and standard deviation of the evaluation metrics are provided. Among them, the NN model again achieves the best classification performance.

Similarly, the random forest algorithm is further used to determine the feature importance ranking. The results are shown in Figure 11. The feature ABK_RATE still has the highest weight (approximately 0.25), followed by the 50FT_KE (50-feet kinetic energy) and TD_POS features, with weights of approximately 0.18 and 0.16, respectively. The importance ranking of the other features has changed compared to that in Figure 10. Additionally, in regression problems the importance of the top three features is significantly higher than that of the others, while in classification problems, the weight differences between different features decrease.

5.3. Further Discussion of the Results

Through the study of various machine learning models, we identified several key factors that influence the risk of runway overrun during the landing phase, including the automatic deceleration rate, runway exit mode, 50-feet kinetic energy, and touchdown position. To the best of our knowledge, the factor of energy has rarely been considered in previous research. To reduce the risk of runway overrun, pilots should pay attention to the aircraft’s kinetic energy when it reaches a height of 50 feet. Additionally, during the landing phase, an excessive runway distance at touchdown should be avoided depending on the situation. At the same time, when necessary, a higher automatic deceleration rate can be selected to reduce the risk of runway overrun.

6. Conclusions

In this paper, we investigated the runway overrun problem based on QAR data and proposed a dynamic risk model. To this end, we first proposed a landing trajectory correction method by combining ground speed and runway position information. To improve the accuracy of the risk assessment model, we designed an algorithm to automatically recognize the aircraft autobrake level during the landing phase, based on which a new dynamic risk assessment model was developed. Finally, we applied classification and regression models to identify risky flights and predict the risk values. The results showed that the automatic deceleration rate, the way that the aircraft exits the runway, the touchdown distance, and the kinetic energy at 50 feet are key factors for the risk of runway overrun during the landing phase.

There are still some limitations of this study, and we will consider them in the future. First, in this paper, we did not fully consider environmental factors, such as weather conditions, contamination of the runway, terrain, etc., and these factors will be considered in our future work. Second, our risk model was only evaluated on one airport, since this airport has a shorter runway length than other airports. In the future, we will consider applying our risk model to more airports. Third, the runway design (e.g., runway length and direction) can also impact the runway overrun risk. For example, the lighting at sunset may affect the pilot’s eyesight, which may increase the runway overrun risk, and this will also be considered in our future study.

Practical operationalization and technology transitions are very important aspects for the flight safety area, and these aspects were not referred to in this paper. In the future, we will consider how to translate the model to usable tools for operators.