Trends in Autonomous Vehicle Performance: A Comprehensive Study of Disengagements and Mileage

Abstract

This study explores the trends and causes of disengagement events in Autonomous Vehicles (AVs) using data from the California Department of Motor Vehicles (CA DMV) from 2019 to 2022. Disengagements, defined as instances where control transitions from the AV to a human driver, are crucial indicators of the reliability and trustworthiness of Autonomous Driving Systems (ADS). The analysis identifies a significant correlation between cumulative mileage and disengagement frequency, revealing that 77% of disengagements were initiated by safety drivers. The research categorizes disengagements into system-initiated, driver-initiated, or planned for testing purposes, highlighting that environmental factors and interactions with other road users are the primary causes attributed to the AV system. The findings indicate a downward trend in the ratio of disengagements to mileage, suggesting improvements in AV technology and increasing operator trust. However, the persistent rate of manual disengagements underscores ongoing challenges regarding driver confidence. This research enhances the understanding of ADS performance and driver interactions, offering valuable insights for improving AV safety and fostering technology acceptance in mixed-traffic environments. Future studies should prioritize enhancing system reliability and addressing the psychological factors that influence driver trust in ADS.

1. Introduction

The advent of Autonomous Vehicles (AVs) represents a transformative shift in the transportation landscape, promising enhanced safety, efficiency, and convenience [1,2,3]. However, the deployment of AV technology is accompanied by critical challenges, particularly concerning the reliability of Autonomous Driving Systems (ADS) [4] and their interaction with human drivers [5,6]. These challenges include ensuring the systems’ performance in diverse driving conditions and understanding how human drivers perceive and respond to ADS [7]. Additionally, comprehensive reviews have highlighted significant obstacles in the deployment of autonomous vehicles, emphasizing the need for robust solutions to address these issues [8,9,10]. Disengagement events, where control is transferred from the AV to a human driver, serve as a significant indicator of the performance and trustworthiness of these systems [11,12]. Understanding the patterns and causes of disengagements is essential for improving AV technology and ensuring public safety [13].

Since 2015, the California Department of Motor Vehicles (CA DMV) has maintained a comprehensive database documenting disengagements and mileage related to AV testing [14]. This database is particularly relevant for the Autonomous Vehicle Testing (AVT) program, which aims to evaluate the performance and safety of self-driving technologies. This dataset provides a valuable opportunity to analyze trends in disengagements alongside the operational performance of various vehicle manufacturers [15]. Previous research has indicated a positive correlation between cumulative disengagement frequency and cumulative mileage, highlighting the importance of monitoring these metrics for evaluating AV safety [16,17]. Studies have categorized disengagements into three types: automated, manual, and planned, based on the initiator of the disengagement event [13,18,19]. Notably, Boggs et al. (2020) reported that 25% of disengagements were initiated by human operators, reflecting the critical role of human oversight in AV operations [16]. Furthermore, the relationship between disengagements and crashes has been a focal point of research, with findings suggesting that a notable percentage of AV crashes occur following disengagement events [19].

Despite the growing body of literature, significant gaps remain in understanding the evolving dynamics of ADS performance and driver trust. Many studies have primarily focused on correlational analyses without adequately addressing the implications of these trends on public safety and technology acceptance [20]. The necessity for this research arises from the need to systematically evaluate the relationship between increasing operational mileage and the corresponding disengagement trends. This study aims to fill the gap by providing a comprehensive analysis of disengagements relative to mileage, contributing to a deeper understanding of the factors influencing driver interactions with ADS and the overall safety of AVs in mixed-traffic environments [21].

In summary, this research aims to deepen the understanding of AV disengagements and their implications for safety and technology acceptance. By utilizing extensive datasets from the CA DMV, the study will provide insights into AV performance and the critical factors influencing disengagement rates and driver trust. Analyzing disengagement events and mileage is crucial as it helps identify the frequency and causes of disengagements, leading to targeted improvements in AV algorithms and systems. This analysis also aids in classifying and understanding the challenges faced by AVs, setting industry benchmarks, and developing regulatory frameworks for safe AV deployment.

Furthermore, understanding disengagement rates and their causes is vital for building driver trust in AV technology. High disengagement rates or frequent safety-critical events can erode public confidence, hindering AV adoption. By addressing these issues, this research seeks to contribute to the broader acceptance and successful implementation of AV technology. Ultimately, the analysis of disengagement events and mileage is essential for advancing AV technology, ensuring safety, and fostering public trust, providing valuable insights for future advancements in AV technology and regulatory frameworks.

2. Literature Review

The CA DMV database is crucial for analyzing trends in disengagements alongside the performance metrics of various vehicle manufacturers [22]. Prior studies have consistently demonstrated a positive correlation between cumulative disengagement frequency and cumulative mileage, indicating that as AVs accumulate more miles, the frequency of disengagements also increases [16,17].

Disengagements are typically classified into three categories: automated, manual, and planned. This classification is based on the initiator of the disengagement event—whether it is the AV itself, the test operator, or a pre-programmed error occurrence [16,23]. Favaro et al. (2018) reported that approximately 25% of disengagements were initiated by human operators, reflecting the critical role of human oversight in the operation of AVs. Moreover, the monthly trends show that the count of automated disengagements is closely related to the count of manual disengagements, suggesting a complex interplay between the operators’ trust in the AVs and the systems’ perceived reliability [13].

The CA DMV does not provide predefined classifications for the factors causing disengagements, leading researchers to adopt various categorizations from previous studies for their analyses [20]. Boggs et al. (2020) found that system issues accounted for 89% of disengagements, with control discrepancies (7%), hardware and software discrepancies (26%), perceptual discrepancies (21%), and planning inconsistencies (35%) also contributing significantly to these events [16].

The relationship between disengagements and crashes has also been a focal point of research. Banerjee et al. (2018) indicated that 23% of AV crashes involved prior disengagements, although only a small fraction (0.8%) of disengagements resulted in crashes [19]. Similarly, Favaro et al. (2018) reported that out of 178 disengagements, only one led to an accident. These findings underscore the necessity for a nuanced understanding of how disengagements relate to overall safety outcomes in AV operations [13].

Zhang et al. (2021) developed an automated method for analyzing disengagement data, highlighting the importance of further research into human trust and comfort levels with AVs to better understand the dynamics of manual disengagements. In a comprehensive review of the DMV AV Testing program in California [17], Guo and Zhang (2022) examined the growth of AV testing by manufacturers and identified hardware and software mismatches and planning discrepancies as predominant causes of disengagements. Their analysis suggested that manufacturers have made significant technological advancements, as evidenced by a decrease in disengagements attributed to perceptual inconsistencies and hardware/software issues [21].

Correlation Analysis Methods

In this section, we provide a brief overview of various methods for analyzing correlations, particularly those utilized in existing literature:

Pearson Correlation Coefficient: This method measures the linear relationship between two continuous variables. Its primary advantage lies in its simplicity and effectiveness in analyzing the relationship between mileage and disengagement events. By employing Pearson’s correlation, we can easily quantify the degree to which changes in mileage are associated with changes in disengagements, making it particularly suitable for our dataset.

Spearman’s Rank Correlation: This non-parametric method assesses how well the relationship between two variables can be described using a monotonic function. It is particularly useful for ordinal data or non-linear relationships.

Regression Analysis: This encompasses various techniques, including linear and logistic regression, to determine the impact of one or more independent variables on a dependent variable. Logistic regression is especially relevant for analyzing binary outcomes, such as whether a disengagement was initiated by the system or the driver.

Given the nature of our data, the use of the Pearson correlation coefficient is justified due to its ability to provide clear insights into the linear relationships present in our mileage and disengagement dataset.

Despite the existing literature on AV disengagements and mileage, significant gaps remain in understanding the evolving dynamics of ADS performance and driver trust. Previous studies have primarily focused on correlational analyses without adequately addressing the implications of these trends on public safety and technology acceptance. The necessity for this research arises from the need to systematically evaluate the relationship between increasing operational mileage and the corresponding disengagement trends. This study aims to fill the gap by providing a comprehensive analysis of disengagements relative to mileage, contributing to a deeper understanding of the factors influencing driver interactions with ADS and the overall safety of AVs in mixed-traffic environments.

3. Materials and Methods

3.1. Conceptual Framework

The process of aggregating and classifying disengagement and mileage data is illustrated in Figure 1. This framework serves as a guide for systematically analyzing the relationship between disengagements and operational mileage of AVs.

3.2. Data Collection

Study Area

While several states in the United States conduct AV testing on public roads, California stands out as the primary location for comprehensive data collection on autonomous vehicle crashes. The CA DMV provides detailed reports on AV incidents, making it a vital resource for this study. Notably, major AV technology companies operate extensively in urban areas such as San Francisco, Mountain View, Santa Clara, San Jose, and Los Angeles [12]. AV operators are required to complete form OL 316 for each accident involving an AV within 10 days of the incident, which is then submitted to the CA DMV. Figure 2 shows the scatter plot of 606 AV crashes that occurred from 2014 until November 2023.

3.3. Data Sources

3.3.1. Disengagement Data

The primary data source for this study is the CA DMV Disengagement Reports, published annually and covering December to November of the reporting year. Available in Excel and PDF formats, these reports document disengagements initiated by drivers, operators, and the AV system, including their locations and causes [15]. Due to inconsistencies in data from 2015 to 2018, this study focuses on reports from 2019 onwards to identify the most prone causes and locations for disengagements [13].

This dataset includes nine Excel files for disengagements with operators and four for full AVs, aggregated into two files for 2019 to 2022. The data allow for the extraction of a classified dataset for further analysis, with summary statistics presented in

Table 1. Frequency of dataset variables disengagement and mileage.

Feature Name		Class/Value	Disengagement	Percentage	Mileage
Year		2019	18,344	46.71	2,888,593
		2022	8303	21.14	5,105,949
		2020	6934	17.66	1,986,361
		2021	4709	11.99	4,091,517
		2018	983	2.50	21.36
Disengagement Initiated By		Test Driver	30,413	77.44	-
		AV System	8860	22.56	-
Location		Street	32,160	81.89	-
		Highway	7088	18.05	-
		Parking Facility	25	0.06	-
Permit Number	Toyota Research Ins	AVT050	9274	23.61	46,300.55
	Apple	AVT030	7696	19.60	185,445.8
	Mercedes-Benz	AVT002	7024	17.89	309,427.3
	Lyft	AVT045	3626	9.23	173,492.8
	Nvidia Corporation	AVT020	1731	4.41	30,910.71
	AImotive	AVT036	1170	2.98	34,411.76
	Udelv	AVT029	1078	2.74	3,593,333
	Waymo LLC(Google)	AVT003	1016	2.59	9,236,363
	Valeo	AVT017	863	2.20	1666.02
	EasyMile	AVT058	700	1.78	639.097
	Qualcomm Tech	AVT047	668	1.70	14,334.76
	Ghost Autonomy	AVT073	448	1.14	14,545.45
	SF Motors	AVT049	402	1.02	8040
	Ridecell	AVT062	384	0.98	307.84
	Aurora Innovation	AVT037	374	0.95	76,326.53
	Intel Corporation	AVT052	320	0.81	2922.37
	Zoox	AVT012	273	0.70	1,365,000
	Cruise	AVT008	241	0.61	11,234.44
	Imagry	AVT061	204	0.52	650.09
	Pony.AI	AVT032	158	0.40	1,580,000
	Nuro	AVT028	151	0.38	377,500
	Drive.AI	AVT013	150	0.38	115,384.6
	Nissan North A	AVT007	144	0.37	8275.86
	Bosch	AVT006	141	0.36	780.73
	Nullmax	AVT038	140	0.36	1456.23
	Motional.AD	AVT004	135	0.34	4945.05
	Box Bot	AVT057	109	0.28	462.45
	WeRide Corp	AVT034	91	0.23	303,333.3
	Phantom.AI	AVT046	86	0.22	284,344.3
	SAIC Innovation C	AVT035	80	0.20	4469.27
	Apex.AI	AVT051	63	0.16	448.40
	Woven	AVT077	63	0.16	6407.76
	Gatik.AI	AVT054	61	0.16	15,250
	Ambarella Corp	AVT053	52	0.13	3151.5
	QCRAFT	AVT067	29	0.07	19,333.33
	ThorDrive	AVT064	27	0.07	2547.2
	BMW of North A	AVT009	22	0.06	286.83
	DiDi Research A	AVT055	16	0.04	80,000
	Baidu USA	AVT015	14	0.04	92,857
	AutoX Technologies	AVT021	13	0.03	144,444.4
	Telenav	AVT019	10	0.03	31.86
	Atlas Robotics	AVT068	10	0.03	47.40
	Argo.AI	AVT066	5	0.01	41,666.66
	Deeproute.AI	AVT069	4	0.01	25,000
	PlusAI	AVT027	4	0.01	25,474
Class		Operator Takeover	15,155	38.59	-
		Planning Discrepancy	6822	17.37	-
		Environment and Other Road Users	6352	16.17	-
		Control Discrepancy	6262	15.94	-
		Perception Discrepancy	3804	9.69	-
		Hardware and Software Discrepancy	878	2.24	-

.

3.3.2. Mileage Data

Complementing the disengagement data are the CA DMV Mileage Reports, which provide essential operational mileage information for AVs. Published concurrently with the disengagement reports, these cover the same periods and are also available in Excel and PDF formats. This study utilizes mileage reports from 2018 to 2022, focusing on monthly mileage for each test vehicle and total disengagements [21]. The completed dataset can be downloaded in xlsx and csv formats from https://github.com/kohanpour1/AVs (accessed on 9 November 2024). The article by Kohanpour et al. (2024) also supports these data [24].

The mileage dataset includes nine Excel files aggregated for 2019 to 2022, detailing miles driven and total disengagements per vehicle without distinguishing between manual and automated disengagements. These data will be merged with disengagement data for statistical analysis to identify causes, consisting of unique identifiers, manufacturers, monthly mileage, and total disengagements for each vehicle.

3.4. Data Analysis

After classification, we utilized descriptive statistical methods, Sankey diagrams, and cross-tabulation tables to analyze the results and identify the principal causes of disengagements. The trends in AV disengagements were evaluated over multiple years, providing insights into the evolving dynamics of AV technology and operational safety. The process of aggregating disengagement and mileage data and modeling is illustrated in Figure 1.

3.4.1. Dependent Variable

For modeling disengagement data, the variable “cause of disengagement” was considered as the dependent variable. This variable included two classes: (1) disengagement by the safety driver or remote operator, and (2) disengagement by the AV system. Our goal with this modeling was to identify factors that led to unwanted disengagements by the ADS or, in other words, when the autonomous system failed.

3.4.2. Study Population and Sample

The study population included all AVs registered in California. For accident data, the sample encompassed all available data related to AV crashes from 2014 to September 2023. For mileage and disengagement data, the sample included records from 2019 to 2022. This methodology treated the population size and sample size as equivalent, ensuring that all available data were analyzed without sampling bias, which was critical for achieving high accuracy in the findings.

3.4.3. Data Preprocessing

Data preprocessing involved several steps to ensure the integrity and usability of the datasets. Initially, the datasets were cleaned to remove duplicates and correct inconsistencies. Missing values were addressed through imputation or exclusion, depending on the extent of the missing data. The cleaned datasets were then merged based on unique identifiers, allowing for a comprehensive analysis of disengagements relative to mileage.

The data were relatively clean, with 18 entries completed through explanations and other variables. However, two records related to Tesla were excluded due to missing data.

Variable Classification

To classify the causes of disengagements, we employed Python programming and text-mining techniques. The process involved aggregating data from all years into a single file, extracting unique causes, and identifying keywords or phrases that corresponded to specific classes of disengagement [16]. This iterative process continued until all causes were classified, with any unclassifiable causes categorized under “other causes”. The classification yielded approximately 40,000 causes assigned to six major classes, aiding in the identification of prevalent disengagement factors. Figure 3 illustrates this classification process.

3.5. Statistical Algorithms

In this section, we explored the statistical algorithms employed to analyze the data collected from the CA DMV reports. These algorithms were essential for uncovering patterns and relationships within the dataset, particularly in understanding the causes of disengagements in AVs. By utilizing tools such as contingency tables and chi-square tests, we aimed to provide insights into how various factors, such as location, cause, and disengagement factors, influenced disengagement occurrences. This analytical approach enhanced our understanding of the underlying dynamics affecting autonomous vehicle performance.

Contingency Table

To assess the relationships between the dependent variable and other factors, a contingency table was constructed. This two-dimensional table examined the interaction between the dependent variable (cause of disengagement) and independent variables (e.g., vehicle type, location, cause, and disengagement factors). The construction of the contingency table utilized the Pandas library in Python, facilitating the examination of variable relationships, frequency distributions, and patterns within the data. The chi-square test was employed to evaluate the significance of the associations observed in the contingency table. The chi-square statistic was calculated using the Equation (1):

$$\mathit{\chi }²=\sum \left(\right(\mathit{O}-\mathit{E})²/\mathit{E})$$

(1)

where:

$\mathit{\chi }²$: Represents the chi-square test statistic

$\mathit{O}$: Represents the observed frequency in the contingency table

$\mathit{E}$: Represents the expected frequency under the null hypothesis

Degrees of freedom are calculated according to Equation (2) as follows:

$$\mathit{d}\mathit{f}=(\mathit{r}-1)\times (\mathit{c}-1)$$

(2)

where $r$ is the number of rows and $c$ is the number of columns. In this study, the contingency table is calculated separately for the dependent variable of causes of disengagements.

4. Results

4.1. Accident Analysis

To investigate the crashes involving AVs, we created a comprehensive diagram (Figure 4) illustrating the number of crashes reported by various manufacturers under different driving conditions. Notably, Cruise, Waymo, and Zoox have recorded the highest number of crashes. This trend correlates with the mileage data, which indicates that these companies have also logged the greatest distances traveled. Therefore, a high number of reported crashes is expected, as increased exposure to traffic typically corresponds with a higher likelihood of incidents [12].

In the subsequent sections, we will delve into the analysis of disengagements and mileage covered, providing a clearer understanding of the performance metrics across different manufacturers.

4.2. Disengagement Dataset

Disengagements of the Autonomous Driving System (ADS) can occur due to actions taken by safety drivers, the AV system itself, or through pre-programmed testing scenarios [13]. In this study, disengagements are categorized into two classes: manual and ADS. The locations of these disengagements, as depicted in Figure 5, primarily include streets, highways, and parking spaces. The classification of disengagement locations is divided into three categories: (1) Parking spaces, (2) Highways (including freeways and interstates), and (3) Streets (all other road types).

4.2.1. Classification of Disengagement Causes

To analyze the causes of disengagements, we classified thousands of unique causes into six main categories using text-mining techniques in Python. The dataset from 2019 to 2022 was aggregated, identifying unique causes for 39,273 disengagements across 46 manufacturers. Approximately 1400 of these causes were manually reviewed to identify keywords and phrases that could effectively categorize each cause into the appropriate classes. Following this initial manual review, the process was automated using Python (version 3.9) based on the extracted keywords. The results of the automated classification were then compared with the manual classifications to ensure accuracy. After confirming the correct functionality of the code, the detailed coding process was repeated multiple times, employing over 300 keywords, until no cause remained unclassified, some of which are illustrated in Figure 6. This iterative approach ensured a high level of precision in the classification of causes. This classification builds on the foundational work of Boggs et al. (2020) [16].

The identified causes of disengagement were categorized into six classes:

• Operator Takeover
• Planning Discrepancy
• Control Discrepancy
• Perception Discrepancy
• Hardware and Software Discrepancy
• Environment and Other Road Users.

4.2.2. Distribution of Disengagement Classes

The comprehensive analysis of the disengagement data, as presented through the three figures, provides valuable insights into the operational dynamics of ADS. The Sankey diagram in Figure 7 illustrates the distribution and sample size of each class, with the thickness of each variable representing its approximate percentage of total disengagements. Figure 7 further details the distribution of disengagement causes based on location and the initiating factor, revealing that the safety driver accounted for 15,155 of the 39,273 disengagements, making it the leading cause. Following this, planning issues and environmental conditions emerged as significant factors, particularly in street and parking locations. Conversely, for highways and freeways, perception errors and environmental-related issues ranked as the second and third leading causes of disengagements, suggesting that AVs frequently encounter planning issues on streets while control issues are more prevalent on highways.

The cross-table heatmap in Figure 8 presents a more quantitative analysis of the relationships between location, disengagement causes, and the contributing factors. The data indicate that the most common disengagement scenarios involve test driver-initiated events on streets, with factors such as Planning Discrepancy, Control Discrepancy, and Environment and Other Road User being the primary contributors. The heatmap also reveals the relative importance of different factors across various locations and disengagement causes, providing valuable insights for targeted improvements in autonomous vehicle technology.

Overall, Figure 7 and Figure 8 offer a comprehensive analysis of the disengagement data, highlighting the key factors, locations, and entities involved in the disengagement events. The insights gained can help autonomous vehicle manufacturers and researchers identify areas for improvement, such as enhancing perception algorithms, improving vehicle control systems, and optimizing decision-making processes to enhance the safety and reliability of autonomous driving technologies.

4.2.3. Evaluation of Disengagement Results

To evaluate the relationship between disengagement causes and locations, a statistical analysis was conducted using the chi-square test. The null hypothesis $H0$ posits that there is no significant relationship between disengagement causes and locations at the significance level of 0.05. The results are summarized in

Table 2. Chi-square test results for disengagement causes and locations.

Variable	$\mathit{\chi }2$	$\mathit{P}$	$\mathit{d}\mathit{f}$	Result
Location of Disengagement	619.22	3.45 × 10−135	2	$\mathrm{Reject}H0$
Cause of Disengagement	8975.41	0.00	5	$\mathrm{Reject}H0$

below:

Given that the p-values are significantly lower than the significance level of 0.05, we reject the null hypothesis. This indicates a significant relationship between the causes and locations of disengagements, suggesting that different contexts can influence the frequency and nature of disengagements by the AV system. The validity of these findings is supported by the statistical tests performed.

The contingency table heatmap in Figure 9 presents a more quantitative analysis of the relationships between location, disengagement causes, and contributing factors. The heatmap shows the frequency of disengagements for each combination of location, cause, and factor, allowing for a deeper understanding of the underlying patterns.

The data indicate that the most common disengagement scenarios involve test driver-initiated events on streets, with factors such as Planning Discrepancy, Control Discrepancy, and Environment and Other Road User being the primary contributors. Figure 9 also reveals the relative importance of different factors across various locations and disengagement causes, providing valuable insights for targeted improvements in autonomous vehicle technology.

4.3. Mileage Dataset

This dataset contains information such as the monthly mileage for each test vehicle and the total number of disengagements for each vehicle. By aggregating the mileage and disengagement datasets, the number of disengagements by the AV system and manual disengagements for each manufacturer can be determined for each year. Additionally, the total mileage for each manufacturer can be calculated. These preprocessing steps were performed in Python, resulting in a dataset that can be used to analyze the trends of disengagements and mileage over the years for each manufacturer. Understanding which manufacturers perform better and the overall trend among all manufacturers is crucial for analysts and safety organizations. The total mileage from 2019 to the end of November 2022 for 42 manufacturers is approximately 14 million miles.

Figure 10 displays the monthly trend of disengagements and mileage from 2019 to 2023. To facilitate comparison, the number of disengagements was multiplied by 10. Following the outbreak of COVID-19 and the imposition of restrictions, the trend of mileage decreased and nearly halted until mid-2020; however, testing of these vehicles resumed afterward. Starting from early 2021, changes in the number of disengagements relative to mileage have been observed. The results indicate that following 2020, improvements in autonomous vehicle performance have led to a reduction in ADS disengagements.

Analysis of the Monthly Disengagement and Mileage Trend in Figure 10 shows COVID-19 restrictions may have led to a reduction in testing and evaluation of AVs, which could mean less available data for more thorough analysis. The performance improvement after 2020 might be due to changes in algorithms, hardware, or other aspects of autonomous vehicle technology that are not explicitly evident in the data.

Figure 11 presents the trend in mileage traveled by the top ten manufacturers with the highest recorded distances. Notably, Cruise, Waymo, Zoox, and Apple have logged the most miles. The increase in disengagements illustrated in Figure 10 is attributed to the escalation in testing activities and the greater mileage accumulated by specific companies.

For a clearer understanding of these changes, Figure 12 illustrates the precise number and trend of disengagements by safety drivers and the AV system. It is evident that after 2020, the performance of the AV system improved, resulting in fewer disengagements despite an increase in mileage. The increase in mileage after 2020 may be due to more testing and evaluation in real-world environments, indicating greater trust in autonomous vehicle technology. The reduction in Autonomous Driving System disengagements could be attributed to improvements in perception algorithms, path planning, and vehicle control.

The analysis of the annual trend of the disengagement rate per one million miles traveled in Figure 13 indicates a decreasing trend from 2019 to early 2023. This consistent reduction suggests improvements in autonomous vehicle technology and increased safety measures, reflecting a growing confidence in the system’s performance over the years.

To enhance the clarity of the statistics presented thus far and to facilitate a more effective comparison of various manufacturers based on disengagement rates per mile, Figure 14 is included.

This chart illustrates the ratio of total disengagements to total mileage traveled from 2019 to early 2023 for each manufacturer. According to the data presented, EasyMile (AVT058), Ridecell (AVT062), and Valeo (AVT017) exhibit the highest disengagement rates, indicating suboptimal performance. In contrast, Cruise (AVT008), Waymo LLC (Google) (AVT003), and Toyota Research Institute (AVT050) demonstrate the lowest disengagement rates per mile, reflecting superior performance.

The overall trend of disengagements relative to mileage, as illustrated in Figure 15, indicates that as mileage increases, the total number of disengagements decreases. According to the data, manufacturers have gradually succeeded in reducing this ratio over time by minimizing system errors, which has resulted in fewer disengagements by the AV system. Although this chart reflects the average performance of all manufacturers, some manufacturers demonstrated poorer performance. In the study by Boggs et al. for the data from 2015 to 2019, approximately 160,000 disengagements were recorded over 3.5 million miles [16], resulting in a disengagement-to-mileage ratio of 0.046. In our study, this ratio for the years following 2019 is:

$$Disengagementspermile={\displaystyle \frac{\mathrm{39,000}}{\mathrm{14,000,000}}}=0.0028$$

• Disengagements (Numerator): This represents the total number of disengagement events recorded during the specified period (2019 to 2022). In this case, the total is 39,000 disengagements.
• Mileage (Denominator): This is the total distance traveled by the vehicles during the same period. Here, it amounts to 14,000,000 miles.

The result of 0.0028 disengagements per mile indicates that for every mile driven, there are approximately 0.0028 instances where the autonomous vehicle system disengaged. This low ratio reflects a significant improvement in the disengagement-to-mileage ratio over the subsequent four-year period compared to the previous one, highlighting advancements in autonomous vehicle (AV) technology and operational safety. These findings suggest enhanced performance and reliability of AV systems, demonstrating the ongoing progress in the field.

5. Discussion

The analysis of disengagement and mileage datasets through statistical models and descriptive statistics provides significant insights into the operational dynamics of ADS. Findings derived from the Sankey diagrams, along with the examination of disengagements and mileage ratios, reveal critical patterns that align with and expand upon previous research in this field.

5.1. Disengagement Patterns and Causes

Results indicate that different locations are predisposed to various causes of ADS disengagements. Consistent with previous studies, disengagements by the ADS system occur more frequently on streets and roads than on highways, freeways, or state routes [16,25]. This trend underscores the complexity of urban environments, where factors such as dense traffic, unpredictable interactions with pedestrians, and varying road conditions can challenge the ADS’s operational capabilities.

Interestingly, research reveals that perceptual system failures are the predominant cause of ADS disengagements on highways, while manual disengagements are more prevalent in urban settings. This distinction highlights the need for targeted improvements in ADS technology, particularly in perception algorithms that must adapt to the specific challenges presented by different driving environments [13].

The analysis also shows that disengagements initiated by safety drivers are often due to route planning errors or failures in vehicle control systems. In contrast, disengagements attributed to the AV system are primarily caused by environmental factors and interactions with other road users. Notably, perceptual errors and hardware/software malfunctions were identified as the least frequent causes associated with the AV system. This finding contrasts with previous studies by Boggs et al. and Sinha et al., which indicated that perceptual discrepancies and hardware/software issues were the main factors leading to disengagements [16,23]. The discrepancies may be attributed to the evolving nature of AV testing and deployment, particularly as manufacturers transition from controlled testing environments to real-world applications [12].

Data indicate that 23% of disengagements were attributed to the AV system, while 77% were initiated by safety drivers. Boggs et al. reported a similar figure of 75%, whereas Zhang (2021) noted that 80% of disengagements were driver-initiated. These variations may reflect differences in testing conditions, the level of trust in AV systems among operators, and the operational maturity of the vehicles being tested [16,26]. Trust is a critical factor; as operators gain confidence in the technology, they may be less inclined to intervene, potentially leading to a decrease in manual disengagements over time [13].

5.2. Trends in Disengagements and Mileage

The integrated analysis of disengagement and mileage data reveals a positive trend: as mileage increases, the number of disengagements attributable to the AV system has shown a downward trajectory since 2020. This suggests that the average performance of autonomous vehicle manufacturers has improved in mitigating disengagements caused by failures in prediction, planning, perception, and environmental factors. However, it is important to note that this improvement is not uniform across all manufacturers, indicating that some companies may still face significant challenges in optimizing their ADS [23].

The ratio of disengagements to mileage has also decreased dramatically, from 0.37 in 2019 to 0.04 by the end of 2022. This substantial reduction indicates a trend toward fewer disengagements relative to the increasing mileage, suggesting that manufacturers have made significant strides in enhancing the reliability and safety of their systems. The mileage increased from 0.25 million miles in 2019 to 4.7 million miles in 2022, reflecting an expanded operational footprint of AVs and increased confidence in their deployment [16].

It was posited that as AVs develop over the years, the number of disengagements by the AV system would decrease. Results support this hypothesis, indicating that advancements in automotive technology and artificial intelligence have enabled the AV system to perceive and react more effectively in complex traffic conditions. Boggs et al. emphasized that improvements in technology likely contribute to increased operator trust in the system, further supporting the observed trends [16].

5.3. Safety Implications and Future Directions

Results indicate that out of 39,273 disengagements from December 2018 to November 2022, only 0.22% resulted in a collision. This low collision rate suggests that while disengagements are a critical safety concern, the actual incidence of crashes remains relatively low compared to the total number of disengagements. This finding aligns with the broader narrative that, despite the challenges posed by disengagements, AV technology is progressing toward safer operational standards [13].

However, several challenges and limitations must be acknowledged. The reliance on historical data may not fully capture the dynamic nature of ADS technology and its evolving challenges. Additionally, variations in reporting standards across different manufacturers can lead to inconsistencies in the data. Future research should focus on standardizing data collection methods to ensure comparability across studies.

5.4. Suggestions for Future Development of ATO Algorithms

Based on our data analysis, we propose the following suggestions for the future development of ATO algorithms:

Identification of High-Accident Scenarios: It is crucial to focus on urban environments with high pedestrian traffic and complex intersections, as these areas have shown a higher incidence of disengagements. Rigorous testing in these scenarios is essential to enhance safety and reliability. Enhancing Vehicle–Driver Interactions: Scenarios where the AV system requires sudden driver intervention can undermine trust. Developing better communication protocols and feedback mechanisms between the vehicle and the driver can improve user confidence in the system.

Looking forward, there are several avenues for future research and development. First, manufacturers should enhance the perception capabilities of their systems, particularly in urban environments where interaction complexity is highest. Second, further studies should explore the psychological factors influencing operator trust and disengagement behavior, as understanding these dynamics can inform better training and operational protocols. Finally, ongoing data collection and analysis are essential to monitor the evolving landscape of autonomous vehicle technology and its implications for traffic safety.

6. Conclusions

The analysis of disengagement events in Autonomous Driving Systems (ADS) indicates a significant trend: while the frequency of these events is decreasing, advancements in system performance and reliability are evident. The evolution of artificial intelligence (AI) has enabled ADS to perform driving tasks previously considered challenging, thereby enhancing operational efficiency [12,16].

Despite this reduction in ADS disengagements, manual disengagements have not significantly declined, suggesting a persistent lack of trust among drivers in ADS technology. As operators gain experience, they may better understand scenarios where ADS struggles, leading them to disengage manually in complex situations while allowing the system to manage routine tasks [13]. This highlights the critical need to foster trust in ADS for broader acceptance in everyday traffic.

Variations in disengagement rates may arise from differences in testing locations and conditions, necessitating caution in drawing conclusions about technological improvements based solely on disengagement data. An analysis of the ratio of disengagements to mileage reveals a downward trend from 0.3 in 2019 to 0.04 by the end of 2022, while total mileage increased from 2.9 million to 4.6 million miles. This suggests that as ADS covers greater distances, their relative disengagement frequency decreases, supporting the hypothesis of improved performance [26].

For definitive insights into ADS advancements, future research should extend evaluations over varied testing environments and assess driver trust and comfort levels in relinquishing control. Collaboration with manufacturers and testing companies can facilitate engagement with drivers who have experienced disengagement incidents.

Addressing the causes of ADS disengagement is essential. Manufacturers must enhance AI and control algorithms, particularly in their responsiveness to environmental conditions and interactions with other road users. Notably, disengagement causes differ between highway and urban settings, with perceptual errors—such as failures to recognize traffic signals—linked to collisions. Some incidents have shown ADS either failing to proceed or accelerating slowly after a traffic light turns green, leading to rear-end collisions with non-AV vehicles. Further investigation is required to confirm these hypotheses [23].

In summary, while ADS shows promise in enhancing safety within mixed traffic environments, both the technology and traditional vehicle manufacturers must adapt to the evolving landscape. Governments and safety agencies should develop supportive infrastructure and training programs, focusing on integrating autonomous aerial and ground transportation systems. This multifaceted approach aims to improve safety and foster public acceptance of autonomous technologies in everyday transportation.