1. Introduction
The International Disaster Database, Université Catholique de Louvain—Brussels—Belgium, shows that roughly 300–400 natural disasters have occurred annually since 2000 [
1]. Natural disasters such as hurricanes, earthquakes, wildfires, and volcanic activity can cause extreme destruction of property and infrastructure, personal injury, and death. A particular threat to Floridians and other coastal states is hurricanes. It has been projected that the combined forces of coastal development and climate change will increase the amount of hurricane damage in the future [
2,
3,
4,
5,
6]. One of the undeniable aftermaths of hurricanes is the damage and obstructions to roadways, which can inhibit emergency vehicles from rescuing people. These obstructions may include: flooding, uprooted vegetation, sinkholes, structural damage to the road, and other debris. Injuries that are sustained from a hurricane impact may be life-threatening and require immediate emergency medical services (EMS) [
7]. An EMS driver does not have time to deal with blocked or defective roadways. It is imperative that an EMS driver takes the fastest traversable route in order to maximize the probability of saving a person’s life.
The use of satellite imagery and unmanned autonomous vehicles equipped with cameras permits the mapping and exploration of territories that may be inaccessible and unsafe. For instance, a swarm of unmanned aerial vehicles (UAVs) can be deployed after a disaster to assess damage to critical infrastructure. Similarly, a UAV can stream video to a high-performance server with dedicated algorithms used to assess infrastructure damage and the traversability of roads after a major hurricane. Mainly, the first step in assessing the traversability of roads is to identify and extract the roads from images or video streams. The difficulty of extracting continuous road segments can indicate that the road contains debris and is potentially not traversable. After extraction, road segments can be fed into classification algorithms to further assess traversability.
In this research, high-resolution UAV imagery was taken from the aftermath of hurricane Sally at the University of West Florida (UWF). The images have been stored in a custom dataset. Ground truth masks were created for images in this dataset so that the quality of subsequent road extraction on these images could be assessed. A new approach was developed for road extraction using a particular set of pre-processing techniques, a modified version of Efficient Graph-Based image segmentation, and a novel post-processing heuristic.
The original contributions of this study can be summarized as: (i) the use of custom aerial footage data taken from the UWF campus after hurricane Sally, (ii) the development of ground truth masks to measure the quality of road extraction, (iii) the development of a new post-processing heuristic which can identify and connect segments of the road after image segmentation and help address one of the shortcomings of Efficient Graph-Based image segmentation, and (iv) performing a hue and saturation analysis for the segments neighboring road segments so that the segments that are similar enough can be merged. The intention of this paper is to present our modified Efficient Graph-Based image segmentation framework design with analysis and to show proof-of-concept results of the entire pipeline on custom aerial footage data recorded after hurricane Sally in 2020. Further analysis, such as robustness to noise and comparison with other methods, are not included in the scope of the originally intended contribution in this article, and they are left for a future work. Additionally, the planned future work also includes another important aspect of comprehensive computational time analysis.
The remainder of the paper is organized as follows. Related work on image segmentation is discussed in
Section 2, followed by the details of Efficient Graph-Based Image Segmentation in
Section 3. Road Extraction is presented in
Section 4, followed by the results and discussion in
Section 5. Conclusions are presented in
Section 6.
2. Related Work
Image segmentation is a computer vision technique that divides an image into several specific components with unique attributes. Modern image segmentation algorithms are usually categorized as semantic (pixel-wise association with class label), instance (accurate delineation of each object in an image), and panoptic (assigning class labels to objects in images). Early image segmentation methods include threshold-based, centroid-based, density-based, graph-based, fuzzy theory-based, hierarchical, and distribution-based methods. The goal of this section is to outline some of the most popular techniques that are well-suited to solving the problems of road extraction and traversability assessment. A comprehensive survey of clustering algorithms can be found in [
8]; furthermore, a survey of clustering algorithms used in Image segmentation can be found in [
9].
Centroid-based image segmentation and clustering involve finding an arbitrary number of centroids in a dataset and then grouping together data points with the smallest distance to a particular centroid. One of the most algorithms, K- means, is a centroid-based clustering algorithm [
10]. However, K-means has significant drawbacks as it lacks flexibility in the shape of the clusters, and there are no probabilities associated with cluster assignments [
11]. In other studies, the K-means algorithm is used to segment images; of particular relevance, in [
12], the road is extracted from images using the K- means algorithm and morphological operations. The K-means algorithm is used to generate segments, and then a simple geometric post-processing heuristic is applied, which attempts to identify the segments that are road.
Hierarchical clustering has also been used to segment images, both as a standalone segmentation method and in combination with other segmentation methods. In [
13], images were first pre-segmented with the well-known graph-based segmentation method of Normalized Cuts. Afterward, hierarchical segmentation was applied. In [
14], the study proposes an agglomerative hierarchical clustering-based high-resolution remote sensing image segmentation algorithm. The algorithm showed favorable results over standard K-means image segmentation. In [
15], an agglomerative clustering technique within a feature matrix is presented. It is shown to compare favorably with the image segmentation algorithm and has the advantage of not needing to know the number of clusters in advance.
The fuzzy sets-based method is another approach presented in the literature for image segmentation [
16]. In [
17], the authors developed a fuzzy system to identify roads in aerial images with five fuzzy membership functions (Good values, Up-Probable values, Down-Probable values, Up-Bad values, and Down-Bad values), which help classify pixels as either being road or non-road with a certain probability. Some limitations of this method are that it is designed for 8-bit images, cannot handle regions of road less than 5 pixels in width, relies on hard-coded values, and cannot handle shadows. In [
18], the authors develop an approach based on Fuzzy C-Means (FCM) to extract roads from foggy aerial images. The authors point out that aerial image quality is susceptible to weather conditions, variations in lighting, and properties of the ground. Fog can obscure the gray scale difference between road and non-road regions in an image; thus, a defogging procedure should be applied if fog is present. Other segmentation methods include density-based algorithms [
19,
20], Mean Shift-based methods [
21,
22,
23,
24,
25,
26], and Gaussian mixture models (GMM)-based approaches [
27,
28,
29,
30,
31,
32].
In graph-based image segmentation methods, images must first be converted into a graph where each pixel is a node in the graph. Next, a decision must be made as to which edges to add to the graph. One option is to add edges between a pixel and all other pixels that are in its neighborhood. Two obvious neighborhoods that can be used are the Von Neumann 4-pixel neighborhood and the Moore 8-pixel neighborhood [
33,
34]. In [
35], Normalized Cuts is used to segment high-resolution satellite images. Although the experiments showed “good operability,” the slow running time of the algorithm was discouraging. In [
36], the Normalized Cuts method was used for the detection of roads in aerial images. The aerial image is first segmented into 20 components. Next, color and shape are used to determine which components are roads.
In terms of road extraction, there are several studies with various methods presented in the literature, some of them worth mentioning here. The U-net neural network is one of the methods presented for road extraction [
37]. In [
38], the authors propose an object-based classification approach for automatic road detection from orthophoto images. Additionally, a deep convolutional neural network (CNN)-based framework for road detection and segmentation from aerial images is presented in [
39]. Similarly, another advanced algorithm, namely Fully Convolutional Network (FCN) and conditional Generative Adversarial Networks (GAN), are used for road extraction using RGB images captured by a UAV [
40]. In [
41], the author used the Mask-R-CNN neural network to detect flood water on roads, and in [
42], a Mask-R-CNN approach was used to segment images in an effort to monitor the road surface condition. There are numerous other high-performing neural network architectures used for image segmentation presented in the literature. The lack of availability of large labeled datasets and the unsuitability for sparse scenes limit the effectiveness of neural network-based methods [
43]. A comparison of a large number of image segmentation methods, including neural network and non-neural network-based methods, is conducted in [
43].
There are also other notable techniques that have been used to segment images. In the Affinity Propagation method, real number messages are exchanged between data points until a high-quality set of clusters gradually emerges [
44,
45]. In [
46], the authors present MRF (Markov Random Fields) models that can accurately capture road network topologies in synthetic aperture radar (SAR) images. In [
47], the author proposes a genetic algorithm to find the initial contour points so that an Active Contour Model can be used to find the road in images. In [
48], ant colony optimization is used for parameter selection of fuzzy object-based image analysis to extract the roads from remotely sensed images.
Although significant research has been conducted on image segmentation, road extraction based on image segmentation after a major disaster using aerial footage is still an open area of research. Little attention has been imparted to the development of custom pre- and post-processing steps. In this paper, an Efficient Graph-Based Image Segmentation approach with improvements performed via developed algorithms is introduced. This approach shows advantages over K-mean clustering with a higher Intersection over Union (IoU) score.
3. Efficient Graph-Based Image Segmentation
The image segmentation approach that was used in this study to isolate segments of traversable road is based on the Efficient Graph-Based image segmentation algorithm. In this section, an overview of Efficient Graph-Based image segmentation (EGS) and the modifications that are made is presented. The word efficient in the name of the algorithm refers to the algorithm’s fast running time.
Consider an undirected graph
, with
being a set of vertices of and
being the set of edges consisting of two vertices (
) and a weight
. In EGS, edge weights represent how dissimilar two pixels in an image are. Similarity can be measured in terms of pixel attributes, including hue and intensity, and a distance function such as the
L2 norm can be used for that purpose. EGS constructs a segmentation
by partitioning
into
connected components
(|●| represents the cardinality, numbers of elements in that set). Next, the issue of deciding if there is evidence for a boundary between components is considered. EGS introduced a novel pairwise region comparison predicate that measures the similarity between boundary elements of two components relative to the similarity inside of each component. The difference inside of each component is quantified by
, which is the largest edge weight in the minimum spanning tree (MST) of the component [
34].
In Equation (1), if
e is a graph’s edge,
w(
e) returns the weight of that edge. The difference between components is captured by the term
, which represents the smallest edge weight connecting the components [
34].
In Equation (2),
min(
w(
e)) refers to this minimum edge weight, that is, the smallest (minimum) edge weight that exists connecting two distinct components
C1 and
C2. The minimum edge weight must be used because using some quantile, such as the median, produces an NP-hard computational problem. The region comparison predicate of EGS is defined to be [
34]:
where the minimum internal difference,
, is defined as [
34]:
where
is a threshold function used to control how different two components must be from each other in order for a boundary to exist between them. In Equation (4),
Int(
C) is the largest edge weight in the MST. The default threshold function given in the EGS algorithm is defined as [
34]:
where
is the lone parameter of EGS and can be thought of as the affinity for larger components or, more succinctly, as a parameter to control scale. Any function that is not negative can be used as
.
The EGS algorithm works by first constructing a graph,
G, from the image with
n vertices and
m edges. The output of the algorithm is a segmentation of the vertices into
k components. The following steps define the algorithm [
34]:
Sort the edges by non-decreasing edge weight.
Start with segmentation where each vertex is its own component.
Construct the q-th segmentation from by doing the following. Let be the vertices connected by the q-th edge. If are not in the same components and the weight is small compared to the internal difference of then join the two components; otherwise, take no action.
Repeat (3) for each .
Return the final segmentation as .
There are several highly appealing features of the EGS algorithm. Reference [
34] emphasizes that in order for a segmentation algorithm to be of practical real-time use, it should run in time approximately linear in the number of image pixels. The default implementation given by the author runs in
time where
n is the total number of pixels in the image; however, this can be improved to
by using integer edge weights and a constant time sorting algorithm. For images, the use of integer edge weights is a great option because the color of each pixel in an image is usually represented by one or more integers. Further details about the implementation of EGS on aerial images can be found in [
49].
The results section (
Section 5) will show the application of EGS in road extraction that the higher number of unique edge weights generated from the Euclidean distance is not worth the sacrifice of computational speed. A lookup table can be pre-computed to increase computational time efficiency if needed. Alternatively, using a sorting optimization can decrease the computational time. However, even without using any sorting optimization, the EGS method is incredibly fast and possibly the fastest method that can be used to segment an image while considering all of the pixels in the image.
In this study, original contributions include the changes that were made to the default algorithm as: (
i) a simple way of automatically computing the parameter
k and (
ii) a new adaptive isoperimetric quotient threshold function designed for road extraction. The basis for this formula is that
k should increase with image size to prevent tiny segments from forming and attenuate the burden of post-processing.
The variable
z captures the elevation of the UAV. The constant
is some real number that controls the degree to which the elevation of the UAV affects the size of the road segments. Presumably, the higher the elevation, the smaller the road segments will be. In this study,
was set equal to 2.5. Additionally, a new threshold function is introduced in this study, which has been given the name
adaptive isoperimetric quotient threshold function
where
p is the perimeter of a segment,
r is the radius of a circle that encloses the segment, and
Aiso is the isoperimetric area of the circle that encloses the segment. The key feature of this threshold function is that more compact circular components are less favorable than long and skinny geometries. This provides a great improvement in the algorithm’s ability to extract paths and roads. Further modifications to the EGS algorithm can be found in [
49].