Exploration of Traffic Accident-Based Pilot Zones for Autonomous Vehicle Safety Validation

Abstract

Recently, the commercialization of autonomous vehicles has increased the importance of verifying vehicle safety through autonomous trials. Autonomous driving trials are conducted in limited areas within artificially constructed test roads and pilot districts and directly explore road sections and areas with similar environments to ensure the safety of AVs driving on real roads. Many previous studies have evaluated the complex response potential of AVs by deriving edge scenarios to ensure their safety. However, difficulties arise in exploring real roads with traffic accident factors and configurations similar to those in edge scenarios, making validation on real roads uncertain. This paper proposes a novel method for exploring pilot zones using traffic accident data to verify the safety of autonomous vehicles (AVs). The method employs a CNN + BiGRU model trained on DMV dataset data to classify traffic accidents as AV- or human-caused. The model’s classification accuracy was evaluated using recall, precision, F1 score, and accuracy, achieving 100.0%, 97.8%, 98.9, and 99.5%, respectively. The trained model was applied to the KNPA dataset, identifying 562 out of 798 cases as AV-like, indicating potential areas of high accident density due to AV operation. Outlier detection and DBSCAN clustering were utilized to identify compact pilot zones, effectively reducing the area size compared to raw data clusters. This approach significantly lowers the cost and time associated with selecting test roads and provides a viable alternative for countries lacking real AV accident data. The proposed method’s effectiveness in identifying pilot zones demonstrates its potential for advancing AV safety validation.

1. Introduction

As the likelihood that ordinary drivers can purchase vehicles equipped with autonomous driving systems at the commercialized level rather than in test drives and research increases, interest in their safety also rises [1,2,3]. Therefore, technological development and research to verify the safety of driverless autonomous vehicles (AVs) have been actively conducted in recent years [4,5,6,7,8]. In order to verify the safety of AVs on real roads, it is necessary to conduct real-world verification. Currently, AVs are temporarily permitted by the government to operate through a ‘temporary operation permit’, the primary purpose of which is to test and demonstrate the developed technology [9,10,11]. Test operation of experimental AVs is carried out in a limited scope, such as driving on test roads and pilot driving zones artificially created for safety reasons. The current standards for issuing AV test operation licenses need to be revised to verify the safety of AVs that will be driven on real roads [12]. Therefore, an evaluation method is required to confirm the safety of various situations that may occur when AVs drive on real roads [13]. However, finding a road section or area with an environment similar to the traffic accidents caused by AVs is difficult.

Therefore, many studies have focused on deriving edge scenarios to ensure the safety of AVs [14,15,16,17,18,19,20,21,22,23]. By deriving the patterns of existing traffic accidents, it is possible to evaluate whether AVs can respond to complex situations [23]. However, the scenario-based safety verification approach has various limitations. It is challenging to develop scenarios, and it is difficult to generalize about accident situations because there are many factors involved in traffic accidents [11]. An additional limitation is that verification on real roads is still being determined [11]. Therefore, it is necessary to develop scenarios based on accurate data. Recently, the use of real-world traffic accident data has been increasing in traffic accident research [24,25,26]. This method can overcome the problem of scenario representativeness by generating scenarios using actual accident data and machine training. However, there is a prerequisite that scenarios can only be generated if requirements such as data purification and artificial intelligence are met [27]. Moreover, obtaining actual AV accident data is difficult and expensive [28]. While simulations and edge-case scenarios can assist in minimizing validation costs, they are likely to provide a sufficient degree of assurance for full-scale deployment with a more sophisticated approach to validation data collection and safety analysis [29].

In this study, pilot zones were explored in the pilot operation district to verify the safety of AVs. There is a need for an evaluation method to verify AVs’ safety in various situations that may occur when they drive on roads [13]. A strategy for AV safety verification based on road testing in a brute-force manner is not feasible [29]. Our strategy is to determine pilot zones to verify the safety of AVs based on traffic accident data of human drivers. Our strategy is to determine pilot zones to verify the safety of AVs based on traffic accident data from human drivers. If we can identify AV-like data that are similar to traffic accidents caused by AVs during self-driving from actual traffic accident records, areas with a high density of such accidents can be considered vulnerable to autonomous driving. Consequently, these vulnerable areas can be selected as pilot zones for road testing. Using AV-like data instead of actual traffic accident data raises concerns about reliability, as some argue that traditional vehicle accident data may not be applicable due to the differences between AVs and human-driven vehicles [30]. However, despite these differences, AVs and human-driven vehicles share similar environments and face overlapping challenges. Thus, analyzing traditional vehicle accident data can provide valuable insights into human error, environmental factors, and road design, helping to predict potential risks for AVs. Since AVs still interact with human drivers, pedestrians, and cyclists, understanding traditional accident patterns is essential for improving AV algorithms and ensuring safer operations across diverse situations.

We generated a model to classify whether each traffic accident data point is AV-caused or human-caused using a Bi-directional Gated Recurrent Unit (BiGRU) [31] combined with Convolution Neural Networks (CNNs) [32]. Additionally, we focused on “Sejong-si”, a smart city in the pilot operation district in South Korea, and selected cases from actual traffic accident data similar to those caused by AVs. Finally, based on the selected traffic accident data locations, we recommend pilot zones within Sejong-si by clustering according to the distribution of traffic accidents.

This paper is organized as follows. In the Related Work section, we review previous research to ensure the safety of AVs. In the Motivation section, the factors of traffic accidents are divided into AV and human perspectives. In the Methodology section, we construct a dataset of traffic accidents involving AVs and humans and a dataset of traffic accidents involving AVs and humans in Korea to explore the actual verification area of AVs. We tokenize the text data of AV and human traffic accidents and construct an embedding vector space using Global Vectors for Word Representation (GloVe) [33]. The embedding vectors are trained using CNN + BiGRU and classify actual traffic accident overviews into AV and human classes. Outliers in the data classified as AVs are detected and clustered using Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [34]. The clustered pilot zones are visualized using a polygon on the map. In the Conclusions section, we analyze the advantages and limitations of the proposed methodology and suggest plans.

2. Related Work

The testing of AVs is critical to ensuring the safety and reliability of autonomous vehicles. Scenario generation for AV testing has been an active area of research. A study was conducted to automatically generate test scenarios for AVs, combining a simple trajectory planner and a feasibility checker to generate a diverse set of efficient test scenarios [15]. A graphical scenario modeling language modeled as a graph based on behavioral trees was proposed. The graphical framework allows for the creation and design of new scenarios and examination of the generated scenarios by machine training algorithms [16].

In addition, research has been conducted on the generation of test scenarios with a low probability of occurrence, which are not typical accident situations. One study focused on developing scenarios for AVs based on real-world accident data that consider unpredictable accidents [17]. The researchers used unstructured text data based on the Pegasus layer model to set up accident scenarios. They classified them into three types (typical traffic, critical traffic, and edge cases). Topic modeling was applied to edge cases—those least likely to occur and hardest to predict—to develop risk scenarios for AVs. To generate risky scenarios for virtual testing, Gao et al. explored the creation of rare event scenarios [18]. In their study, they proposed a two-stage heterogeneous driver model that alters the number of dangerous scenarios in the scenario space to generate more realistic and efficient scenarios [18].

Various studies have analyzed and generated test scenarios. One study aimed to include a variety of scenarios while reducing repetitive scenarios to ensure economical and safe vehicle behavior [19]. Vehicle behaviors were clustered and characterized based on trajectory data to achieve this. To predict the outcomes of untested scenarios, Batsch et al. studied the efficient determination of performance boundaries that can clearly distinguish safe scenarios from unsafe ones [20]. They utilized a Gaussian process-based model to simulate scenarios. They presented an iterative method for the optimization of the parameter space, focusing on obtaining the most significant scenarios at the performance boundary. Liu et al. explored critical AV crash causes by analyzing the differences between AV and conventional vehicle crashes [21]. Using the U.S. Department of Transportation’s pre-crash scenario types, they identified crash patterns and applied statistical analysis to determine the differences between pre-crash scenarios for AVs and traditional vehicles [21]. They found that differences in perception-reaction times between AVs and human drivers, inaccurate identification of lane-changing intentions of other vehicles, and insufficient path planning by AVs were key contributing factors. Yu et al. investigated AVs’ operational safety risk factors based on risk scenarios with causal mechanisms consistent with crashes [22]. To this end, contributing factors regarding vehicle kinematics and the traffic environment were extracted to develop a model analyzing the variability of route scenarios and the probability of hazardous scenario co-occurrences, considering their variability characteristics.

Scenario-based testing for AVs faces several limitations that hinder its effectiveness in the comprehensive development of autonomous driving systems. First, the limited variety of scenarios means that these tests only cover predefined conditions, making it difficult to account for unexpected real-world variables such as sudden weather changes or unpredictable behavior of road users, which might not be reflected in controlled environments.

Secondly, there is a significant gap between the complexity of real-world driving and what scenario-based tests can replicate. Real-world driving involves intricate elements like navigating congested urban areas or interacting with diverse road users—challenges that are difficult to simulate comprehensively in a controlled scenario. As a result, unforeseen challenges in real-world conditions might not be adequately tested.

Thirdly, the cost and time constraints of designing and testing every possible traffic condition or event make it impractical. Simulating diverse traffic, road, and weather conditions requires substantial resources, making the process time-consuming and expensive.

Finally, the limited scope of interaction testing restricts the ability to evaluate how AVs respond to other road users, such as pedestrians, cyclists, and human drivers. Real-world driving involves complex interactions that are often challenging to capture in a controlled testing environment.

Given these limitations, scenario-based testing alone is insufficient for the complete development of AVs. Combining more complex and diverse scenarios supplemented with real-world testing is necessary.

The methodology proposed in this paper focuses on analyzing actual traffic accident data rather than relying solely on predefined scenarios. This approach significantly reduces the costs associated with scenario development and improves time efficiency. Focusing on real-world accident locations accounts for complex and unpredictable factors, making the selection of pilot zones more realistic. Furthermore, real-world validation enables meaningful interaction with various road users, providing a more comprehensive assessment of AV performance.

3. Motivation

Traffic accident factors include human factors, environmental factors, weather conditions, animals on the road, etc. [24]. Human factors associated with drivers encompass physical, cognitive, and psychological aspects. Physical factors include the driver’s health condition, sleep deprivation, physical discomfort, and illnesses [35]. Cognitive factors encompass the driver’s attention, concentration, experience, and judgment. Psychological factors involve emotions like stress, anxiety, depression, and anger, among others. These factors interact in intricate ways, ultimately heightening the potential for accidents. Traffic accidents can also be influenced by various environmental factors. Road-related issues, such as the absence of road markings, malfunctioning signs or traffic lights, slippery road surfaces, and inadequate lighting, can contribute to accidents [36,37,38,39,40].

Additionally, road conditions, including width, length, slope, and intersection placement, can impact the likelihood of accidents. Weather conditions like rain, snow, fog, strong winds, and dust can reduce driver visibility and make vehicle control more challenging, increasing accident risks [36,38,40]. Furthermore, encountering animals on the road, especially at night, can create unexpected situations for drivers, further elevating the risk of accidents.

Let us categorize the causes of traffic accidents into those caused by humans and those caused by AVs. Assuming AVs operate typically without malfunctions, traffic accidents involving AVs are less likely to be related to human factors. The Venn diagram presented in Figure 1 show that AV-caused accidents are included in the human-caused category. In other words, human-caused traffic accidents involve a broader range of factors than those caused by AVs. If we selectively identify human-caused traffic accidents related to non-human factors, we can categorize these as AV-like traffic accidents. This approach can be considered reasonable, especially when real traffic accident data involving AVs are limited.

Therefore, data related to human factors in human-caused traffic accidents are more related to AV-caused traffic accidents. We can classify real-world traffic accidents as similar to AV accidents. Based on these data, we can explore pilot zones. We utilize various machine learning techniques to select AV-like data from actual traffic accidents.

4. Methodology

To validate the safety of AVs, it is essential to consider a wide range of potential accident scenarios that might occur in real-world situations. However, given the current lack of sufficient actual AV accident data, utilizing alternative datasets is crucial. The primary purpose of leveraging these datasets is to improve the accuracy and robustness of AV safety validation by simulating various real-world accident scenarios and identifying risks that AVs may face on the road.

First, the Department of Motor Vehicles (DMV) dataset (https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports (accessed on 2 August 2024)), which contains records of both AV-caused and human-caused accidents, is used to train the classification model. This dataset helps the model learn to differentiate between accidents caused by autonomous vehicles and those caused by human drivers. Next, we perform the classification process on the Korean National Police Agency (KNPA) dataset, which consists entirely of human-caused accidents. The classification model is used to identify human-caused accidents that exhibit patterns similar to those of AV-caused accidents, which are defined as AV-like data. This process allows us to simulate potential accident scenarios that AVs could encounter in real-world conditions. Consequently, it addresses the gap caused by the limited availability of actual AV accident data, providing a broader and more comprehensive validation of AV safety through the use of alternative datasets. Ultimately, this approach contributes to improved safety testing and validation for autonomous vehicles, ensuring they are prepared to handle a wide range of scenarios on the road. Finally, the geographic coordinates (latitude and longitude) of the traffic accidents classified as AV-like data are mapped on a geographical map, allowing us to explore potential pilot zones for verification of AV safety. The overall flow chart of this methodology is depicted in Figure 2.

• Data Preparation: The accident summaries (text) included in the training dataset (DMV dataset) and the test dataset (KNPA dataset) are standardized in English.
• Data Tokenization: The text data are tokenized using Natural Language Processing (NLP) techniques. In this step, the Natural Language Toolkit (NLTK), a tool well-suited for processing English text, is used to remove unnecessary words and to tokenize the text into a word-by-word format.
• Word Embedding: The word embedding process is then applied. In this step, the GloVe embedding technique converts words into high-dimensional vectors, providing numerical representations from which the model can learn.
• Model Training: The CNN + BiGRU model is trained using transformed vector data. This model combines CNNs with BiGRU to effectively learn patterns in the text data and capture various features, making it suitable for classification tasks.
• Classification: The classification process is conducted to identify AV-like data from the test dataset that exhibit patterns similar to those caused by autonomous vehicles.
• Outlier Detection: AV-like data are subjected to an outlier detection process. In this stage, outliers are removed based on the geographical coordinates of the accidents, thereby improving the density of the data distribution. A distance metric is used to measure the density around each data point’s coordinates, allowing for the detection and removal of outliers.
• Pilot Zone Exploration: In this phase, the DBSCAN clustering algorithm is applied to explore potential pilot zones. This process identifies pilot test zones within suitable areas where AV safety can be validated.

Figure 2. Flow chart of the proposed method.

Figure 2. Flow chart of the proposed method.

4.1. Experimental Setup

The experimental environment was set up using an NVIDIA GeForce RTX 4080 GPU, an Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz, and an AMD Ryzen 9 5950X 16-core processor, along with 32 GB of DDR4 RAM. Python version 3.9.1 and Java version 1.8.0 were used as the primary software tools. Morphological text analysis was performed with NLTK version 3.7. The pre-trained GloVe model [33] (Common Crawl, 840B tokens, 2.2M vocabulary, case-sensitive, 300-dimensional vectors) was employed for word embedding. The CNN + BiGRU model for text classification was implemented using TensorFlow version 2.5.0. DBSCAN was applied to cluster accident points via scikit-learn version 1.5.1. Map visualization was handled with folium version 0.14.0, and scipy version 1.13.1 was utilized to compute the convex hull to draw polygons.

4.2. Data Preparation

4.2.1. Training Dataset

This study utilizes the DMV’s “Autonomous Vehicle Collision Reports” dataset, which contains data from both categories, to analyze the patterns that indicate the differences between AV-caused and human-caused traffic accidents.

Figure 3 shows a sample form from the DMV dataset. The Accident Details Description provides an overview of accidents and is divided into autonomous and conventional modes. The accident overviews in the DMV dataset are shown in Figure 3a, and they are divided into various traffic accident factors for AVs and humans. The traffic accident factors in the DMV dataset are shown in Figure 3b and include Weather, Lighting, Roadway Surface, Roadway Conditions, Movement Preceding Collision, and Type of Collision.

The Weather field is classified into the following seven types: clear, cloudy, rainy, snowy, fog/visibility, other, and wind. The Lighting field includes the following five factors: daylight, dusk/dawn, dark with street lights, and not functioning. The Roadway Surface is described as dry, wet, snowy–icy, or slippery (muddy, oily, etc.). The Roadway Conditions field encompasses the following eight elements: holes, deep ruts, loose material on roadway, obstruction on roadway, construction/repair zone, reduced roadway width, flooded, other, and no unusual conditions. The Movement Preceding Collision field includes 18 elements, namely stopped, proceeding straight, ran off the road, making a right turn, making a left turn, making a U-turn, backing up, slowing/stopping, passing other vehicle, changing lanes, parking maneuver, entering traffic, other unsafe turning, crossing into opposing lane, parked, merging, traveling the wrong way, and other. The Type of Collision field includes head-on, sideswipe, rear-end, broadside, and hit object categories.

The DMV dataset was compiled using web crawling to automatically collect data for which PDF report files exist for 2019–2023. During the data cleaning stage, the report files were converted to images, and text data were extracted using Optical Character Recognition (OCR). The text in the Accident Details Description field was extracted to obtain an overview of AV accidents, with specific labels categorized into autonomous and conventional modes. The training dataset contains 192 records comprising 147 labels for autonomous modes and 45 for conventional modes.

4.2.2. Test Dataset

Although traffic accident data collection in Asia is still in its early stages, the KNPA in South Korea collects various types of information related to traffic accidents [24,41]. The actual information recorded by the KNPA on traffic accidents [24] was used as test data to identify pilot zones for the operation of AVs.

The KNPA dataset consists of fields such as Weather, Lighting, Roadway Surface, Type of Accident, Location, and Accident Overview. The Weather field categorizes conditions as clear, rainy, and cloudy. The Lighting field differentiates between daylight and darkness, indicating visibility conditions. The Roadway Surface field classifies conditions as either paved–dry or paved–wet/moist. The Type of Accident includes car-to-car, car-to-person, and vehicle-only categories.

The Location of the accident is specified using the Universal Transverse Mercator (UTM) coordinate system, which provides precise latitude and longitude coordinates. The Accident Overview summarizes all vehicles involved in the accident, focusing on each vehicle’s role. Given that the DMV dataset documents only single-vehicle accidents, cases in the KNPA dataset involving more than two vehicles were excluded from the experiment to maintain consistency. Unlike the DMV dataset, the KNPA dataset does not include information on Roadway Conditions and Movement Preceding Collision.

A total of 798 real traffic accident data points were collected from “Sejong-si”, a pilot zone for AVs in South Korea. The contents of the Accident Overview field, initially recorded in Korean, were translated into English to facilitate comparison with the DMV dataset.

4.2.3. Data Tokenization

To construct a cleaned version of the data, we tokenized the collected data. Tokenization, as part of the data preprocessing process, reduces the dimensionality of the dataset and enhances the model’s performance.

Table 1. Summary of unnecessary part-of-speech tags.

POS	Meaning	Examples
PRP	Personal pronoun	it, he, they
IN	Preposition and subordinating conjunction	on, at, in, into, with, of, between, behind, without, from
NNP	Proper noun	California, Octavia, Cruise, Ponyai, April, PM, Francisco, Ford
CC	Coordinating conjunction	and, or, but
CD	Cardinal numbers	2023, 26th, 21st, 4-way, 296 mph, two
DT	Determiner	a, the, another, both, no, this, either, an
Punctuation	Symbols	:, ., ;, -, ?, !, “, ‘, (, _

explains unnecessary Part-of-Speech (POS) tags.

General sentences often contain numerous unnecessary elements. Elements that do not significantly contribute to the meaning of a sentence, such as prepositions and subordinating conjunctions (IN), coordinating conjunctions (CC), determiners (DT), cardinal numbers (CD), and various punctuation marks, should be removed. Additionally, significant differences in regional and building names exist because the data were independently collected in California and South Korea. Proper nouns (NNPs) were excluded from the analysis to resolve these discrepancies. Personal pronouns (PRPs) were also removed, as they are irrelevant in data targeting autonomous vehicles.

Figure 4 illustrates the tokenization process, where unnecessary words and punctuation were eliminated. In this example, PRPs such as “it” are removed, along with INs like “in” and “on” and CCs like “and”. Additionally, DTs like “the”, NNPs like “California”, and CDs like “29th” were excluded. These elements were removed because they do not carry significant meaning in the context. As a result, the total number of words was reduced from 1439 to 805 across the entire dataset (both training and test datasets).

4.2.4. Word Embedding Using GloVe

The text dataset was vectorized to improve the classification model’s performance. Many previous studies have utilized GloVe in the embedding phase to improve the training model’s performance [42,43,44,45].

GloVe is a word embedding algorithm employed for various NLP tasks such as text processing, document classification, information retrieval, machine translation, and sentiment analysis. It trains distributed representations of words to capture semantic similarities and identify relationships among them. By calculating the co-occurrence probabilities between word pairs in a large text corpus, GloVe models these relationships and learns embedding vectors that effectively represent word meanings. Similar words have similar vectors in this embedding space, enabling the computation of cosine similarity to evaluate semantic relationships. While the initial training can be computationally expensive for large corpora, subsequent training is faster because the number of entries in the non-empty matrix is much smaller than the total number of words in the corpus. Despite its high computational complexity, GloVe prevents high-frequency word pairs from dominating the model, thereby preserving the information of sparse word pairs [46].

Therefore, we chose the GloVe approach to transform the preprocessed text data into an embedding vector space. GloVe’s primary goal is to represent words as fixed-size vectors, which numerically express word meanings. Words with similar meanings are positioned close to each other in the vector space. First, a co-occurrence matrix ($\mathbf{P}$) is constructed based on the frequency of words appearing together in the entire corpus. GloVe minimizes the cost function of a weighted least squares regression model, which applies weights to reduce the influence of word pairs that co-occur rarely in the corpus. The objective function (J) for the weighted least squares regression model on $\mathbf{P}$ is expressed as follows.

$$J=\sum _{i=1}^K\sum _{j=1}^{K}f\left(P_{ij}\right){({\mathbf{v}}_i^T{\tilde{\mathbf{v}}}_j+b_i+{\tilde{b}}_j-log{P}_{ij})}^2$$

(1)

where $P_{ij}$ represents the number of times the center word (i) and the surrounding word (j) co-occur within a window, while $\mathbf{v}$ denotes the word vector that captures the meaning of a specific word in a fixed-dimensional real space. ${\mathbf{v}}_i$ and ${\tilde{\mathbf{v}}}_j$ represent the vectors of the center word and the context word, respectively. ${\mathbf{v}}_i^T{\tilde{\mathbf{v}}}_j$ measures the similarity between the two words by computing the dot product between the center word vector and the context word vector. $b_i$ and ${\tilde{b}}_j$ are bias terms for the center and context words, respectively, correcting for the deviations in word frequency. In other words, they serve as frequency adjustment factors to accurately learn the co-occurrence probability between the center and surrounding words. The J function represents the sum of all possible word pairs for the center word (i) and the context word (j). The weighting function ($f\left(P_{ij}\right)$) reduces the importance of word pairs that infrequently co-occur while ensuring that frequently co-occurring words are not overemphasized. The f function should be relatively small for large values of P while ensuring it does not diminish too quickly for smaller values of P. Considering f as a continuous function, when $P\to 0$, $f\left(0\right)=0$, and $f\left(P\right)$ should converge to zero fast enough such that $f\left(P\right){log}^2\left(P\right)$ remains finite.

$$f\left(P\right)=\left\{\begin{array}{cc}{(P/P_{max})}^{\alpha },\hfill & P<P_{max}\hfill \\ 1,\hfill & P\ge P_{max}\hfill \end{array}\right.$$

(2)

In Equation (2), the model’s performance depends weakly on the cutoff, with $P_{max}$ fixed at 100 in all experiments. $\alpha$ is determined as a heuristic parameter, and based on empirical observations, $\alpha =3/4$ showed better performance than the linear version where $\alpha =1$, so $\alpha$ was fixed at this value [46]. $f\left(P\right)$ adjusts the weight to 1 when P reaches or exceeds $P_{max}$, preventing frequent co-occurrences from being overemphasized.

We utilized the pre-trained GloVe model to project each word in a sentence into a 300-dimensional embedding vector space. Then, we generated inter-sentence embedding vectors as the average of the matched values in the same feature space. The AV traffic accident overview data and the actual traffic accident overview data were projected onto a 300-dimensional pre-trained GloVe model consisting of $\mathbf{w}\in {\mathbb{R}}^{300}$ vectors.

4.3. Classification Based on CNN + BiGRU

Previous studies have primarily utilized the CNN model to classify sentences or documents [47,48,49,50]. A nonlinear sliding CNN and an N-gram language model were used to classify short texts effectively in [47]. CNNs with Word2Vec embeddings were used to classify news articles and tweets in [48]. TextConvoNet was used to classify text using two-dimensional multi-scale convolutional operations instead of the traditional one-dimensional convolving filters in [50]. There have been attempts to enhance text classification performance by combining deep learning models based on embedding techniques. A Long Short-term Memory (LSTM) model with GloVe embeddings was used to classify text in a large dataset in [51]. Chowdhury et al. combined an LSTM-CNN model with GloVe to classify Bangla newspaper articles into ten categories [43]. A CNN-GRU model was used to classify sentiment in Twitter data in [52]. GloVe and CNN-BiLSTM models were used to analyze extended and short-text sentiment information in [44].

We trained the CNN + BiGRU model on the overview data from the DMV dataset to classify it into “AV” (autonomous vehicle) and “human” (conventional mode) classes. A structural diagram of the CNN + BiGRU model is shown in Figure 5. The CNN + BiGRU model trains the spatial and sequential characteristic information from the matrix (${\mathbf{W}}_{tr}$), which contains $N_{tr}$ embedding vectors with a dimensionality of 300. A 1D convolutional layer in the CNN architecture extracts various patterns and features from the text, and a pooling layer extracts important information and reduces dimensionality. A fully connected layer is then utilized to perform classification or similarity measures. The CNN trains by updating weights to minimize the loss function, a strategy commonly used for classification tasks.

The hyperparameters used in the experiments include the Rectified Linear Unit (ReLU) activation function and the Adam optimizer. Binary cross-entropy is used as the loss function. The hyperparameters used in the CNN + BiGRU classification model are detailed in

Table 2. Hyperparameters used in the CNN + BiGRU classification model.

Epoch	$\mathit{\eta }$	Optimizer	Activation Function		Loss Function	Metric
			Hidden	Output
100	0.001	Adam	ReLU	sigmoid	binary cross-entropy	accuracy

.

The trained model’s classification performance was evaluated using recall, precision, F1 score, and accuracy. The formulas for these evaluation metrics are expressed as follows.

$$Recall={\displaystyle \frac{TruePositive}{TruePositive+FalseNegative}}$$

(3)

$$Precision={\displaystyle \frac{TruePositive}{TruePositive+FalsePositive}}$$

(4)

$$F1-Score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(5)

$$Accuracy={\displaystyle \frac{TruePositive+TrueNegative}{TruePositive+TrueNegative+FalsePositive+FalseNegative}}$$

(6)

where $TruePositive$ indicates cases where the prediction is positive and the actual class is positive, $FalsePositive$ occurs when the prediction is a positive class but the actual class is negative, $FalseNegative$ refers to cases where the prediction is a negative class but the actual class is positive, and $TrueNegative$ indicates cases where the prediction is negative and the actual class is negative. In this experiment, “AV” was designated as the positive class (1), and “human” was designated as the negative class (0).

A similarity value (ranging from 0 to 1) is generated in the output layer using a CNN + BiGRU model trained on two classes. To obtain a binary class result of 0 or 1, we classified the data based on a threshold of 0.8 and measured the accuracy metric. Figure 6 presents the training results by epoch. In deep learning, an epoch refers to one complete pass through the entire training dataset by the model. During an epoch, the model processes all data samples and updates its weights accordingly. Multiple epochs are often used to improve model accuracy, as each epoch allows the model to learn and refine its predictions based on the data. Figure 6a shows the accuracy for each epoch, and when the epoch is around 80, the change is significantly reduced. Figure 6b illustrates the loss value between epochs. First, it showed a rapid loss rate. However, when the epoch was 100, it gradually began to show a constant loss rate and a meager loss value, so it designated 100 as the optimal epoch value. For the final model, classification performance was evaluated using recall, precision, F1 score, and accuracy, with values of 100.0%, 97.8%, 98.9, and 99.5%, respectively.

Finally, when the final CNN + BiGRU model was applied to the KNPA dataset, 562 out of 798 cases were classified as “AV” and 236 as “human”. Using the trained classification model, we identified traffic accidents similar to AV-like accidents, which are less likely to be caused by human factors.

4.4. Optimal Pilot Zone Search Result

The distribution of coordinates $(latitude,longitude)$ in traffic accident data classified as AV-like can be utilized to identify pilot zones. If a polygon encompassing the entire dataset is designated as a pilot zone, the resulting area may become excessively large, leading to potential cost issues. Therefore, clustering was performed to divide the pilot zone into more localized areas. The optimal pilot zones can be identified by generating polygons for multiple smaller clusters. In this study, we employed the DBSCAN method [34], which allows for optimal clustering without specifying the number of clusters in advance.

4.4.1. Global Outlier Detection

In this paper, we proactively perform outlier detection to avoid the pilot zone becoming very large due to the availability of a small amount of data. For each data point ($\mathbf{X}\left[i\right]=(latitude,longitude)$) in a dataset ($\mathbf{X}\in {\Re }^{N\times 2}$) that consists of two-dimensional data containing latitude and longitude coordinates, we count the number of neighboring data within a radius (r) based on a distance metric, and if the number is less than a given threshold (k), the data point is determined to be an outlier. In particular, the distance function ($distance\left(\mathbf{X}\right[i],\mathbf{X}[j\left]\right)$) computes the sum of the lengths of line segments projected over the data points on a two-dimensional coordinate axis containing latitude and longitude using Manhattan distance ($=\left|\right|\mathbf{X}\left[i\right]-{\mathbf{X}\left[j\right]\left|\right|}_1$). The array ($\mathbf{is}\_\mathbf{outlier}$) stores the outlier status for each piece of data. Algorithm 1 shows the overall outlier detection process.

Algorithm 1 Outlier Detection in Spatial Data

1: input: Dataset $\mathbf{X}$ (latitude and longitude coordinates), radius r, neighbor count threshold k 2: output: Array $\mathbf{is}\_\mathbf{outlier}$ indicating whether each data point is an outlier 3: $\mathbf{is}\_\mathbf{outlier}\leftarrow$ initialize an array of size N (all elements set to False) 4: for $i\leftarrow 1$ to N do                     ▹N is the number of data points 5:     $count\leftarrow 0$                         ▹ Initialize neighbor count 6:     for $j\leftarrow 1$ to N do 7:         if $distance\left(\mathbf{X}\right[i],\mathbf{X}[j\left]\right)<r$ then 8:            $count\leftarrow count+1$                  ▹ Increment neighbor count 9:         end if 10:     end for 11:     if $count<k$ then 12:         $\mathbf{is}\_\mathbf{outlier}\left[i\right]\leftarrow$ True                ▹$\mathbf{X}\left[i\right]$ is marked as an outlier 13:     end if 14: end for 15: return $\mathbf{is}\_\mathbf{outlier}$     ▹Return an array indicating outlier status for each data point

To calculate the time complexity of this algorithm, we analyze the operations at each step. The input dataset ($\mathbf{X}$) has a size of N, with each data point represented as 2D coordinates. The outer loop iterates N times from $i=1$ to N. The inner for loop also iterates N times for each i, where the distance between two points is computed. The calculationi of the Manhattan distance between two points takes a constant time of $\mathcal{O}\left(1\right)$. The number of neighbors is counted inside the inner loop, and the algorithm checks whether the count is below the threshold (k). These operations are also performed in constant time ($\mathcal{O}\left(1\right)$). Therefore, the complexity of the inner loop is $\mathcal{O}\left(N\right)$, and since the outer loop runs N times, the overall time complexity of the algorithm becomes $\mathcal{O}\left(N\right)\times \mathcal{O}\left(N\right)\times \mathcal{O}\left(1\right)\times \mathcal{O}\left(1\right)=\mathcal{O}\left(N^2\right)$.

In the experiment, parameters were set as $r=0.005$ (about 0.56 km) and $k=7$. The experimental results estimated 190 traffic accidents as outliers among cases classified as AV-like, and 372 of 562 cases were selected. As shown in Figure 7, outlier detection increases the data’s distribution density, which can be beneficial for the designation of pilot zones. Additionally, the outlier detection method can help avoid shallow density distributions in pilot zones by excluding the reference point when no adjacent points exist.

4.4.2. Pilot Zone Exploration Based on DBSCAN

After the removal of outliers, the distribution of AV-like data still covers a large area. For a more compelling demonstration, several pilot zones were estimated by dividing the areas with dense distributions of traffic accidents into clusters. To identify areas with a high density of traffic accidents, we utilized DBSCAN, a clustering method based on the density of the core data. Clustering using DBSCAN means that data in a cluster indicate high surrounding density, while data not included in a cluster are considered noise and indicate low density. Based on the radius ($ϵ$) of a given cluster, neighboring points within the radius are extracted. If a cluster’s minimum number of points is not reached, it is determined to be noise. If the point has a sufficient number of close neighbors, it is the first point in a new cluster, and all points within the radius ($ϵ$) are assigned to the same cluster. The clustering process is repeated until all points have been determined, and the number of clusters is automatically determined. In the experiment, we set the $ϵ$ value for DBSCAN to 0.3 and the minimum number of outliers to 5 for the local outlier removal process.

Figure 8 visualizes the polygons generated based on the AV-like data coordinates. These polygons were created using the convex hull algorithm, which forms the smallest possible polygon encompassing all the given points. Figure 8a shows a single polygon covering all 372 cases after removing outliers from a global perspective, with an area of 8283.88 ${\mathrm{km}}^2$. This polygon is referred to as $Cluster1$. Figure 8b presents the clustering results after applying the DBSCAN algorithm, which generated two cluster regions. The areas of $Cluster1$ and $Cluster2$ are 3324.66 ${\mathrm{km}}^2$ and 567.73 ${\mathrm{km}}^2$, respectively, resulting in a total area of 3892.38 ${\mathrm{km}}^2$ when the two clusters are combined. Figure 8c illustrates the results after applying DBSCAN with an additional local outlier removal process. In this case, the following three clusters were formed: $Cluster1$, $Cluster2$, and $Cluster3$, with areas of 1759.04 ${\mathrm{km}}^2$, 510.02 ${\mathrm{km}}^2$, and 493.84 ${\mathrm{km}}^2$, respectively. The total area encompassing all three clusters was reduced to 2762.91 ${\mathrm{km}}^2$.

Each cluster represents a data group that shares specific patterns or similarities. Therefore, analyzing and comparing the areas and the significance of each cluster is crucial for effectively defining and exploring pilot zones. $Cluster1$ represents the primary region where AV-like behavior patterns are concentrated, indicating a key area for AV-related activity. Initially, the area was 8283.88 ${\mathrm{km}}^2$, but after applying DBSCAN, it was reduced to 3324.66 ${\mathrm{km}}^2$ and further decreased to 1759.04 ${\mathrm{km}}^2$ after local outlier removal. This demonstrates that the DBSCAN algorithm can more precisely explore pilot zones, minimizing the area without significant data loss. $Cluster2$ represents a medium-sized AV activity region, with its area slightly decreasing from 567.73 ${\mathrm{km}}^2$ to 510.02 ${\mathrm{km}}^2$ after local outlier removal. The consistency in cluster patterns suggests that this area could be an optimal choice for a pilot zone, especially under limited budget constraints. $Cluster3$, which emerged after the additional local outlier removal process, has an area of 493.84 ${\mathrm{km}}^2$. This cluster indicates that local outlier filtering identified a more specific area of AV-like activity, providing deeper insights into AV behavior.

In conclusion, using the DBSCAN algorithm for clustering enables a more efficient reduction of the area while still accurately identifying pilot zones. Initially, the polygon generated from globally filtered AV-like data covered 8283.88 ${\mathrm{km}}^2$, which was reduced to 3892.38 ${\mathrm{km}}^2$ after applying DBSCAN. With the inclusion of local outlier removal, the area was further reduced to 2762.91 ${\mathrm{km}}^2$. This demonstrates that DBSCAN, coupled with outlier removal, allows for more focused analysis, yielding cost savings and improved resource efficiency for pilot zone exploration.

5. Conclusions

Searching for adequate pilot zones for AV safety verification in various pilot districts is costly and time-consuming. In addition, because the demonstration roads are searched after the pilot districts are selected, roads are selected in a limited range, which is ineffective for AV safety verification. This study analyzed traffic accident overviews to explore effective pilot zones for AV safety verification. We aimed to identify AV-like data within human-caused accidents. AV-like data were used to identify areas that could experience a high density of traffic accidents due to the operation of AVs in the future. The CNN + BiGRU model was trained using the DMV dataset to accurately classify traffic accident data into AV and human classes in the experiment. The test data were collected from “Sejong-si”, an autonomous driving pilot district, and consisted of 798 sampled cases, all caused solely by human drivers. The total number of test data points classified as AV-like was 562. To explore optimal pilot zones, we clustered 372 cases, excluding outliers, using density-based DBSCAN.

This paper proposes a novel methodology for the exploration of pilot zones using traffic accident data. The proposed method significantly reduces costs and time requirements by semi-automatically identifying pilot zones. Through experiments, it was confirmed that a minimum of two to a maximum of three pilot zones can be explored, and the total area of these zones can be reduced by approximately 2.13 to 2.99 times compared to the original, resulting in a more compact outcome. As a result, these advantages can simplify the policy-making process for the selection of pilot zones. Furthermore, from a data perspective, the proposed methodology can be considered a viable alternative for countries lacking traffic accident data involving AVs. Meanwhile, Level 4 autonomy in self-driving vehicles refers to a high level automation, where vehicles can operate independently under specific conditions while ensuring passenger safety [53]. At this level, a human driver is not required to be on standby for emergencies as long as the vehicle operates within its defined boundaries. Considering that most autonomous vehicles under the jurisdiction of the California DMV are classified as Level 4, the proposed methodology is expected to be useful for future pilot operations involving Level 4 autonomous vehicles with limited Operational Design Domains (ODDs).

However, the method proposed in this paper has several limitations. Firstly, it categorizes documents with features similar to traffic accidents based only on a summary. This approach may only partially capture the diverse factors involved in traffic accidents, as it relies on limited information. Secondly, the size of the training dataset is relatively small, with only 192 cases, which raises concerns about the reliability of the training outcomes and the potential for overfitting. This limitation stems from the scope of the currently available datasets. However, this is expected to be addressed soon, with the anticipated collection of large-scale data on AV traffic accidents. Finally, the method selects AV pilot zones with a dense distribution of traffic accidents. This selection bias occurs because more data are available from these high-accident areas.

Subsequent research will focus on overcoming the limitations identified in the proposed method. The primary focus will be on expanding the dataset to enhance the robustness of the training model and address the issue of overfitting. We aim to incorporate additional factors such as regional infrastructure data and population density to minimize the likelihood of traffic accidents during empirical tests, thereby facilitating the extraction of safer test roads for evaluating the efficacy and safety of AVs. Furthermore, we will delve into connecting traffic accidents within the test areas, comparing a broader range of factors, including infrastructure, population density, and diverse traffic accident elements, to more accurately estimate AV testing routes. It is also crucial to determine test routes within AV pilot zones to trial and validate developed autonomous driving technologies, mainly focusing on sections of roads prone to frequent accidents involving conventional vehicles. This approach will enable AVs to be tested in challenging situations, enhancing their ability to avoid dangers. However, the current methodology for selecting AV pilot zones in South Korea, which prioritizes sections with clear visibility and uncomplicated lane configurations due to concerns over potential traffic accidents, presents limitations in thoroughly testing and affirming AV safety. Therefore, the approach outlined in this paper is anticipated to be beneficial in choosing more effective and comprehensive pilot zones for AV demonstrations, thereby significantly contributing to the advancement of autonomous driving technology.