A Morphological Feature-Oriented Algorithm for Extracting Impervious Surface Areas Obscured by Vegetation in Collaboration with OSM Road Networks in Urban Areas

Abstract

Remote sensing is the primary way to extract the impervious surface areas (ISAs). However, the obstruction of vegetation is a long-standing challenge that prevents the accurate extraction of urban ISAs. Currently, there are no general and systematic methods to solve the problem. In this paper, we present a morphological feature-oriented algorithm, which can make use of the OSM road network information to remove the obscuring effects when the ISAs are extracted. Very high resolution (VHR) images of Wuhan, China, were used in experiments to verify the effectiveness of the proposed algorithm. Experimental results show that the proposed algorithm can improve the accuracy and completeness of ISA extraction by our previous deep learning-based algorithm. In the proposed algorithm, the overall accuracy (OA) is 86.64%. The results show that the proposed algorithm is feasible and can extract the vegetation-obscured ISAs effectively and precisely.

1. Introduction

Impervious surface refers to the surface covered by natural or artificial impervious materials that can prevent surface water from penetrating the soil. With global urbanization, a vast number of impervious surface areas made of cement and asphalt have replaced natural surface areas dominated by the flora and bare soil in recent decades, resulting in urban environmental problems. Early study has shown that the regional environment changes as the percentage of impervious surface increases [1]. In recent years, more and more environmental issues (e.g., the heat island effect [2] and urban floods [3]) related to urbanization have received significant attention from researchers, and studies have shown that ISAs cause many environmental problems. Therefore, how to accurately estimate the ISAs in urban areas has become an essential topic in the urban remote sensing field. After more than 20 years of development, the studies on impervious surface remote sensing have made significant progress and many innovative remote sensing technologies and methods for retrieving impervious surface information have been proposed [4]. In recent years, machine learning has been successfully applied to high-resolution images to optimize the extraction of ISAs [5,6,7]. Shao et al. [5] fused optical and SAR data at the decision level to estimate urban impervious surface areas based on random forest (RF) classifier and evidence theory and analyzed the uncertainty levels of the impervious surface areas. However, none of the conventional or cutting-edge methods in the past can achieve good results when the ISAs are obscured by vegetation. Due to the obscuring effect, ISAs are often misclassified as vegetation areas in the classification results. The area of the impervious surface extracted by the algorithms is less than the actual area of the impervious surface. Akbari et al. [8] discovered the obscuring effect of ISAs extraction when studying the land cover. The results from the study show that in many cities, a large part of ISAs is not extracted, and in most of these cases, trees cover streets, parking lots, grasslands, and sidewalks. Among them, in most non-residential areas, the paved areas account for 50–70% of the areas under the tree canopy. In residential areas, on average, paved roads cover about 35% of the areas under the tree canopy. Gray and Finster’s investigation [9] also demonstrates that the impervious covering rate varies dramatically in congested urban areas. This is due to the fact that canopy coverage is not always defined by pervious planting zones, such as street trees, which cover areas beyond the area they are planted. A study by van der Linden et al. [10] quantified the occlusion effect of vegetation in the entire study area in the metropolitan area of Berlin, Germany. The study found that more than 30% of the street pixels in the study area were classified as vegetation. These studies show that the proportion of ISAs obscured by vegetation is not low in urban areas, and the under-extraction will reduce the completeness of the ISAs extraction results, which is an obstacle to the analysis of urban environmental issues.

At present, many studies have been published on the association between vegetation obstruction and ISAs estimation, exposing the correlation between ISAs extraction and seasons. Based on the research results, many methods have been proposed to use seasonality to reduce vegetation obscuring effects. A study found that there is an evolutionary inverse relationship between impervious and vegetation coverage [11]. As a result, in the study area, seasonal change in plant quantity has a considerable impact on impervious surface estimation [12]. Gong et al. [13] used seasonal sensitivity as an influencing factor for impervious surface estimation and compared the ISAs extraction of Pascagoula City along the Mississippi Gulf coast in the United States in different seasons. The results show that the summer model significantly reduces the average error (AE) of ISAs estimation, indicating that the sensitivity of regional images to seasons can be detected by calculating AE. The influence of seasons on the extraction of ISAs is reflected in that when vegetation degrades with seasonal changes, the vegetation that was originally obscured on ISAs disappears, and these exposed ISAs are easier to be correctly classified than before [14]. This series of studies indicates that the influence of obstruction can be reduced or removed if the images used to estimate the ISAs are obtained in vegetation degradation seasons. Nevertheless, the method does not work when there are no images available during vegetation degradation seasons, such as in countries or regions with subtropical or tropical climates where vegetation grows all year. That is why a generalized method is needed.

The systematic removal of the obscuring effect of vegetation on ISAs necessitates the use of external data [15]. Hung [15] argues that remote sensing mapping methods based on optical and near-infrared imagery will have significant difficulties when the surface is under a dense vegetation canopy. He proposed using LiDAR data to distinguish the canopy of buildings and vegetation and extract the buildings covered by vegetation. However, collecting LiDAR data is costly and time-consuming for a large area. Zhang et al. [16] provide us with a simpler idea to solve the obscuring effect. To solve the misclassification issue of ISAs and street trees, his study extracted the road map in Nanjing by combining roads from traffic maps and obtained road widths from VHR images visually. The street trees are extracted from the images by supervised classification and then overlaid with the road map to detect whether the roads are misclassified. The study indicates a strong correlation between the distribution of street trees and road networks in urban images, which facilitates the extraction of vegetation-obscured ISAs. Although Zhang’s study reveals the correlation, he does not extract vegetation-obscured ISAs. So far, there is still no universal method to extract vegetation-obscured ISAs.

This paper proposes an algorithm to extract vegetation-obscured ISAs in urban areas and overcomes the limitations of our previous deep learning-based methods. The proposed algorithm can complement the results obtained by deep learning for extracting impervious surfaces covered by vegetation. The algorithm is under the assumption that there is a correlation between road network information and street trees. The proposed algorithm consists of three parts: vegetation detection, processing with mathematical morphology operations, and feature mining. Experiments showed that the proposed algorithm can extract vegetation-obscured ISAs effectively and precisely.

2. Study Area and Data Used

The vegetation obscuring effect is a common problem globally, as shown in Figure 1. In cities with different climates, vegetation-obscured ISAs, especially roads, are common in urban areas. In the follow-up study, we chose Wuhan, China, as the study area.

2.1. The Study Area

Wuhan, one of China’s megacities, is located between 113°41′–115°05′ E and 29°58′–31°22′ N with a humid sub-tropical monsoon climate. Wuhan is located in the middle reaches of the Yangtze River and has a lot of lakes and vegetation. The Yangtze River and its tributary, the Han River, divide Wuhan into three towns, Wuchang, Hankou, and Hanyang. Wuhan has a large-scale road network and a 12 million population. In the urban area of Wuhan, street trees planted along both sides of the road are one of the critical green areas, which leads to significant vegetation obscuring effects.

2.2. Data Used

2.2.1. GF-2 Remotely Sensed Imagery

GF-2 is a civil land observation satellite with sub-meter spatial resolution developed by China, carrying two high-resolution multispectral cameras that can achieve high positioning accuracy. GF-2 was successfully launched on 19 August 2014.

Table 1. The parameters of the GF-2 used in this paper.

Satellite Name	Spatial Resolution (m)	Number of Bands	Coordinate System	Range of Study Area	Acquisition Season
GF-2	0.8	3 (Red, Green, Blue)	WGS84	30.50–30.54° N 114.35–114.42° E	Summer, 2020

shows the parameters of the images. Notice that the mosaic data set is used.

2.2.2. OpenStreetMap Road Network

OpenStreetMap (OSM) is an open-source geographic data platform with WGS84 (https://www.openstreetmap.org, accessed on 14 September 2021). In this study, the road network of Wuhan comes from OSM. However, the quality of the data from OSM is not high enough in some cases. Jasmeet [17] concluded the parameters of OSM, such as positional accuracy, completeness, attribute accuracy, and topological consistency, are not accurate. From this perspective, here are some obvious issues:

• The road network is complete at a large scale; however, as the scale decreases, the completeness of the road network decreases, which means some roads totally disappear on low scales.
• Although some roads do not disappear, the road information is incomplete on a low scale.

To solve the above issues, we proposed feature mining to find the complete road information so that we can use them in the extraction of impervious areas covered by vegetation.

3. Methodology

The proposed algorithm aims at dealing with the challenges faced by deep learning in extracting urban vegetation-obscured ISAs, shown in Figure 2. The main idea is as follows: based on the existing ISAs extracted by our previous deep learning, the proposed algorithm integrates the OSM road network to extract vegetation-obscured ISAs from VHR images and fuses all ISAs. The extraction of vegetation-obscured ISAs consists of three steps. Firstly, we detect vegetation such as street trees, under which the roads are likely to exist. Secondly, we thin the regions into skeletons while preserving morphological features. Thirdly, we combine the features from the OSM road network with skeletons and retain some important information in the skeletons with road features. Buffers are generated based on the retained ones as vegetation-obscured ISAs. The specific methods used in the three steps will be introduced in the following.

3.1. Vegetation Detection Based on Image Segmentation and Color VI

Detecting vegetation on VHR images is the basis for identifying vegetation-obscured ISAs. Vegetation indices (VI) are often used to detect vegetation from images. However, calculating a VI using pixel information is inefficient, especially from VHR images. Image segmentation can improve detection efficiency because it divides pixels into regions. Therefore, our vegetation detection involves two steps: image segmentation and computation of VI. Image segmentation helps improve detection efficiency.

Since a VHR image contains more detail than a low-resolution image, object-based segmentation is proved to be better [18]. In image segmentation, we use the tools provided by ENVI, an image processing software, where the object-based segmentation consists of two parts: segmentation and merging. In the segmentation part [19], Sobel edge detection is first used to calculate the gradient image. Then, the watershed algorithm [20] is applied to the gradient image, and the algorithm floods from the uniform part of the object to the edge to identify the boundary with obvious features. Merging is used to solve the problems of feature misclassification and over-segmentation in segmentation [21]. The merging part combines spectral and spatial information and iteratively merges adjacent segments. Object-based image segmentation can segment the objects from different scales. However, the choice of the parameters in the segmentation is very important. If the choice of the parameters is not good, the segmentation results will be bad and will have a bad impact on the extraction of ISAs. The exact parameter values should be chosen based on the growth and distribution of vegetation in the study area. In general, the segmentation parameters should be set to a large value and the merge parameter to a relatively small value to portray the objects in as much in detail as possible.

After we segment the objects, we will compute VI for each object. Many VIs were defined in the past and used to extract vegetation from images [22], most of which are related to the near-infrared (NIR) band. The most used VI is NDVI, which is essentially a mathematical operation at the pixel level. In this study, due to the lack of NIR band in the images, Color VI will be used to replace NDVI to detect vegetation. Color VI is usually used for plant detection, and it was originally proposed to use RGB bands to identify living plants in bare lands, wheat straw, and other residues [23]. However, owing to its ability to highlight a specific color, such as the green of plants, we use Color VI for vegetation detection in the remote sensing images. For each object extracted, we calculate the mean value of the RGB bands of the pixels and calculate the Color VI for that object based on the mean value. Color VI for each object is computed as [24]:

$$\mathrm{VI}=\left(2\times \mathrm{g}-\mathrm{r}-\mathrm{b}\right)-\left(1.4\times \mathrm{r}-\mathrm{g}\right)$$

(1)

where $r$, $g$ and $b$ were defined using the mean values ($\overline{R}$,$\overline{G}$,$\overline{B}$):

$$r=\frac{\overline{R}}{\overline{R}+\overline{G}+\overline{B}}$$

(2)

$$g=\frac{\overline{G}}{\overline{R}+\overline{G}+\overline{B}}$$

(3)

$$b=\frac{\overline{B}}{\overline{R}+\overline{G}+\overline{B}}$$

(4)

The Color VI values of objects are used to detect vegetation. When performing vegetation detection, the thresholding method is used to determine whether an object is a vegetation region or not. If the Color VI of an object is bigger than the threshold, then the object is determined as a vegetation region. Although the probability that woodland or grass obscures ISAs is low, the vegetation detected using this simple thresholding method possibly contains some woodlands and grasslands. Thus, the vegetation areas obtained by the simple thresholding need to be post-processed to remove as many non-street trees as possible and retain street trees.

Post-processing includes area thresholding and rectangular fitting. The area thresholding method can remove objects with a large area, such as woodlands. The ISAs under large areas have little influence on the whole area; instead, retaining the large-area objects will disturb the morphological study of street trees and finally result in a deviation of ISAs. Rectangular fitting describes how well an image object fits into a rectangle of similar size and proportions. With rectangular fitting, it is possible to remove objects such as football fields and so on.

3.2. Skeleton Extraction

After we obtain the vegetation regions by segmentation, we will thin the vegetation regions into skeletons. The skeleton is extracted by mathematical morphology operations. Mathematical morphology operations are often used in road extraction from remote sensing images [25,26]. Since vegetation-obscured ISAs have a great correlation with road information in morphology, we use road morphology to design a mathematical morphology-based framework to extract skeleton lines of vegetation regions (see Figure 3). We used bwmorph function provided by MATLAB to achieve our goal.

3.2.1. Data Preprocessing

Before thinning the detected regions, we design a part named data preprocessing (see the top part of Figure 4) to avoid unnecessary deviation and fracture of the skeleton lines. In data preprocessing, different morphological operations are used to smooth the boundary of vegetation regions, fill the holes in the regions, and connect the cracks between the regions. First, area thresholding is used to remove the isolated vegetation regions with small areas. The threshold in area thresholding is determined by the type and growth status of local vegetation in the study area. In this study, the threshold was set at the average size of the local individual tree canopy, and a region with an area below this value will be removed. Next, a spur removal operation is used to remove the small branches in the regions or lines. Then, regional dilation and erosion are further used to process the results. Regional dilation makes the boundary of the region expand outwards, thus filling the void and connecting the fracture. On the contrary, erosion makes the boundary of the region shrink inward, preventing some regions from sticking together due to dilation and ensuring that the extracted skeleton line does not produce a large deviation. Properly selecting the parameters for dilation and erosion will achieve a good result. For dilation parameters, it is necessary to ensure that the dilation parameters are as small as possible under the premise of filling the gullies and connecting fractures while avoiding excessive dilation. As for the erosion parameters, the purpose is to restore the dilated regions to the previous size as much as possible to prevent positional deviation. Consequently, the difference between the erosion parameters and the dilation parameters should not be too significant. Between the regional dilation and erosion, we add majority judgment to fill the small holes inside the region. In the 8-neighborhood of a pixel, if there are five or more pixels in its neighborhood equal to 1, then the value of the pixel is set to 1.

3.2.2. Skeleton Extraction

After preprocessing, we use thinning algorithm to extract the skeleton for each vegetation region. Thinning can eliminate the influence of irregular contours, which is suitable for dealing with the irregular boundary of vegetation regions. In addition, during thinning, the morphological features of vegetation regions can be retained. Thinning algorithm iteratively removes successive layers of pixels on the boundary till the width of the skeleton is one pixel. The deletion of a pixel is determined by its neighborhood. As shown in Figure 4, let P be the pixel which we need to decide whether to delete or not, and X₁, X₂, …, X₈ constitute its neighborhood, and in a binary image, the value of the pixel is 0 or 1. The thinning algorithm consists of two sub-iterations, which alternately detect the boundary pixels located in the north and east, then south and west. In the first sub-iteration, P will be deleted when by checking whether it meets some conditions using procedures C1, C2 and C3 (see the explanation below), and the second iteration will not be performed on P. If P is preserved in the first sub-iteration, then P will be determined to be deleted or preserved in the second sub-iteration. In the second sub-iteration, P will be deleted by checking the conditions using procedures C1, C2, and C3′ are met. The conditions are shown below [27], where $\wedge$ and $\vee$ refer to logical AND and OR, respectively:

Procedure C1: Procedure C1 calculates the crossing number XH(p) defined by Hilditch for P (Hilditch 1969). If XH(p) equals 1, P is set to be on the boundary of the region using the following equation:

$$XH\left(p\right)={\displaystyle \sum }_{i=1}^4{b}_i$$

(5)

where

$$b_i=\left\{\begin{array}{c}1, \mathrm{if} x_{2i-1}=0 \mathrm{and} \left(x_{2i}=1 \mathrm{or} x_{2i+1}=1\right) \\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{otherwise}\end{array}\right.$$

(6)

After that, we will go to procedure C2 or stop here.

Procedure C2 is used for endpoint check. In procedure C2, we first compute $n_1\left(p\right) and n_2\left(p\right)$ using Equations (6) and (7), and then each breaks the neighbor of P into four pairs of adjoining pixels. After that, we count the number of the pair containing 1 or 2 pixels equaling 1.

$$2\le min\left\{n_1\left(p\right),n_2\left(p\right)\right\}\le 3$$

(7)

where

$$n_1\left(p\right)={\displaystyle \sum }_{k=1}^4{x}_{2k-1}\vee x_{2k}$$

(8)

$$n_2\left(p\right)={\displaystyle \sum }_{k=1}^4{x}_{2k}\vee x_{2k+1}$$

(9)

If the above condition is met, P is not an endpoint.

Procedure C3 is used to check the connectivity of east and north of P using the following equation.

$$\left(x_2\vee x_3\vee {\overline{x}}_8\right)\wedge x_1=0$$

(10)

where ${\overline{x}}_8=1-x_8$. If the above condition is met, then P is located in the east or the north of the region, and the connectivity of P’s east and north is 0.

Procedure C3′ is similar to C3. If the following condition is met, then P is located in the west or the south of the region, and the connectivity of P’s west and south is 0:

$$\left(x_6\vee x_7{\vee }_{x_4}^{-}\right)\wedge x_5=0$$

(11)

where ${\overline{x}}_4=1-x_4$.

In MATLAB, we generate a binary region and extract the skeleton of it by using the thinning algorithm from bwmorph. As Figure 5 shows, the thinning algorithm ensures that the deletion of P does not change the connectivity of the image while using a single-pixel width to generate the result.

However, when the vegetation region is too large, the thinning will generate a lot of small branches. Though the truth of vegetation-obscured ISAs is unknown, the main direction of the vegetation region is most likely to be the main direction of the ISA. In general, vegetation and ISAs are planted or built in the same direction. To remove these branches, the spur removal operation from the previous summary is used again.

3.3. Feature Mining

In Section 2.2.2, we have discussed the advantages and disadvantages of the OSM road network. To make up for the incomplete information on some roads in OSM, ISAs need to be extracted by analyzing the road-like structure of skeleton lines. Therefore, feature mining is used to achieve the goal. Feature mining involves two ways to extract vegetation-obscured ISAs: one combines the feature of skeleton lines with OSM, and the other only relies on skeleton lines. The one with OSM is called feature consistency, and the other is called feature recognition.

3.3.1. Feature Consistency

Feature consistency consists of position consistency and direction consistency. By comparing the two consistencies of an OSM road network with the same skeleton lines under the same coordinate system, the skeleton lines consistent with the OSM roads are retained. The retained ones are assigned with the grade of the OSM roads to build a buffer for ISAs.

The consistency of the position removes the skeleton lines with a large deviation from the OSM road network. The main steps of position consistency are as follows: We first calculate the minimum distance from a skeleton line (corresponding to a vegetation region) to the OSM roads. Then, determine whether this minimum distance is lower than a preset threshold, and if so, keep the skeleton line; otherwise, remove it. For the computation of the minimum distance from the skeleton to the road network, we design a method used for calculating the minimum distance between two polylines, which are shown in Figure 6. The method is described as follows: For two polylines P and Q, we first calculate the distance from each point on polyline P to each point on polyline Q. Similarly, perform the same calculation from each point on polyline Q to polyline P. Finally, take the minimum value of all the distances as the distance between the two lines.

Direction consistency calculates the azimuth of the endpoints of the polyline as the main direction in the projected coordinate system and classifies the direction into classes: horizontal and vertical. If the class of a skeleton line is the same as a road line (the selected road line is the one closest to the skeleton line), the skeleton line is retained; otherwise, it is removed. The range of the values for the two directions is shown below, where x is the azimuth of the endpoints of the polyline and 0 represents the horizontal direction, and 1 represents the vertical direction. The computation of the direction angle is obtained using the following Equation.

$$ori=\left\{\begin{array}{c}1, \mathrm{if} \left(x\ge 135\mathrm{and}x<225\right) \mathrm{or} \left(x\ge 315\mathrm{and}x<360\right) \mathrm{or} \left(x\ge 0\mathrm{and}x<45\right)\\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{if} \left(x\ge 45\mathrm{and}x<135\right) \mathrm{or} (x\ge 135 \mathrm{and} x<315)\end{array}\right.$$

(12)

Additionally, the range of classification is shown in Figure 7, and the class of Polyline A is vertical.

As shown in Figure 8, only the skeleton lines that meet both position consistency and direction consistency will be retained. For example, skeleton line A does not satisfy direction consistency, and skeleton line D does not satisfy position consistency. Thus, both are removed.

Based on the retained skeleton lines, buffers are built as the vegetation-obscured ISAs. Since vegetation is widely distributed and has different sizes, the width of an ISA obscured is different. The road grade of an OSM provides a reference for setting buffer width. The road grade of an OSM and the corresponding buffer width in our study area is shown in

Table 2. Road grade attribute of OSM and corresponding buffer width.

Num	Field Name	Description	Buffer Width (m)
1	Motorway	Mainly high-grade roads, on which vegetation usually covers the road shoulders or non-motorized lanes attached to high-grade roads. The coverage of vegetation is not broad.	1.25
2	Trunk
3	Secondary
4	Tertiary
5	Trunk_link
6	Secondary_link
7	Cycleway
8	Unclassified	Contains some special road types. It should be noted that “Unclassified” does not mean that the classification is unknown. It refers to the least important roads in a country’s system.	4.5
9	Residential
10	Living_street
11	Service
12	Null	Roads without grade
13	Pedestrian	Only for people to walk.	1
14	Footway
15	Steps
16	Path	A non-specific path. It can be used as a cycleway or sidewalk.
17	Construction	Roads under construction, do not study	0

. The width setting refers to the relevant Chinese road regulations, and different countries and regions should be adjusted according to local regulations. Notice that the buffer width of the roads without a grade is 4.5 m. In fact, most of these roads are residential or living streets.

3.3.2. Feature Recognition

Feature recognition analyzes the features of the skeleton lines and retains the skeleton lines conforming to road network structure. According to prior knowledge, roads have an obvious network structure. As street trees are vegetation planted on both sides of the roads, the network structure of street trees is consistent with that of roads. Figure 9 shows the network structure of street trees, which is drawn by combining a GF-2 image and a ground survey. From Figure 9, we can find that when the vegetation is similar to the structure of the road network, the vegetation is likely to be the street trees.

We use two features to determine whether a skeleton line is a part of the network structure of street trees. They are the number of connections of the skeleton line node (endpoint) to other skeleton lines and the angle at which the skeleton line intersects with other skeleton lines at the node. As shown in Figure 10, the number of connections for both node A and node B is 3. Additionally, the angle between two polylines is the acute angle. Let the angle between Polyline $P_1$ and Polyline $P_2$ is α (see Figure 10), where the coordinates of the endpoints of $P_1$ are ($x_1$, $y_1$) and ($x_2$, $y_2$) and the coordinates of endpoints of $P_2$ are ($x_3$, $y_3$) and ($x_4$, $y_4$).

Under the projection coordinate system, the angle $\mathsf{\alpha }$ is calculated as follows:

$$\mathsf{\alpha }=\arctan \left|l\right|$$

(13)

where $l$ is defined as:

$$l=\frac{K_1-K_2}{1+K_1\times K_2}$$

(14)

where $K_1,K_2$ are defined as:

$$K_1=\frac{y_1-y_2}{x_1-x_2}$$

(15)

$$K_2=\frac{y_3-y_4}{x_3-x_4}$$

(16)

In general, the number of connections at a node is set to be ≥3 to describe the network structure, and the range of the angles is set to be as close to 90° as possible (for example, the range could be set to be [80, 90]). In fact, not all road networks are approximately checkerboard-shaped; hence, the range of angles depends on the study area.

Based on the skeleton lines conforming to the network structure of street trees, buffers are built as the vegetation-obscured ISAs too. The width of buffers is set according to the grade of roads obscured by street trees in the study area. Generally, they are residential or living streets.

4. Results

4.1. Vegetation Detection from the GF-2 Image

In order to clearly show the experimental results, as shown in Figure 11, an area was selected from the whole study area located in the urban of Wuhan. Figure 12a shows an original GF-2 subimage from the study area. Figure 12b shows the result of image segmentation. In the study, image segmentation was performed using the tool called Segment Only Feature Extraction Workflow in ENVI, where the scale level of the edge algorithm for segmentation is set to 35%, and the level of the full lambda schedule algorithm for merging is set to 96%. After segmentation, the vegetation objects are appropriately bordered by other feature objects (buildings, water, etc.). However, inside a vegetation object, different vegetation types (street trees, woods, grass) possibly merge into one object. After segmentation, we calculated the Color VI of each object and selected an appropriate threshold to extract vegetation. The result is shown in Figure 12c. In the study, Color VI was set to be more than −0.12. By analyzing the results, we could find that vegetation accounted for most of the retained objects. However, some misclassifications were still included in the result, such as buildings with high brightness. We applied post-processing to solve the problems above, and the result is presented in Figure 12d. In the post-processing, the area threshold was set to be 80,000 pixels (if the number of the pixels is more than the threshold, the object will be removed) and the threshold for the rectangular fit degree was set to be 0.9 (if the degree is more than 0.9, the object will be removed). The results show that the objects without obscuring effects, such as large-area woodland, farmlands, and buildings, have been removed.

4.2. Skeleton Extraction Using Mathematical Morphology

After data were preprocessed, we needed to extract the skeleton for further processing. Before skeleton extraction, to reduce the information loss during dimension reduction, we performed some morphological operations on the segmentation result. First, we removed the isolated regions from the binary image of the vegetation regions (as shown in Figure 13a). In this step, a region with small areas will be removed. The threshold of the area for a region to be removed was determined by the canopy size of the street trees. In our study area, the canopy size was like a square with an area of 9 m² (15 pixels) based on a field survey. The result after area threshold is shown in Figure 13b. Then, we used morphological operations to remove the spurs. The results are shown in Figure 13c. Next, regional dilation was used to expand the regions to connect the cracks between the regions and smooth the boundary of regions. The result is shown in Figure 13d. For dilation operation, the kernel size was set to 3 × 3, and we performed dilation three times for each image. After the dilation operations, majority judgment (introduced in Section 3.2.1) was performed to further bridge the gaps without over-filling the holes. The result is shown in Figure 13e. Finally, regional erosion was performed to recover the size of the regions. For regional erosion, we used the same kernel size while the times of erosion were 4. The result is shown in Figure 13f.

Thinning was performed on the image after regional erosion to obtain the skeleton. However, the skeleton obtained using the above processing still included a lot of spurs (as shown in Figure 14a). We performed spurs removal again to remove the spurs from the skeleton, and we obtained the final skeleton, as shown in Figure 14b.

4.3. Feature Mining

Feature mining involves two parts: feature consistency and feature recognition. Figure 15 shows the result of feature consistency. By matching the position features and direction features of the skeleton lines with the OSM road network (the red part in Figure 15), we kept the skeleton lines as the centerlines of vegetation-obscured ISAs. In Figure 15a, the red polylines are OSM road networks, and the black polylines are skeleton lines. In fact, some of the OSM roads and the roads in GF-2 images do not match exactly. Therefore, to eliminate the error in the study area, we selected some intersections of OSM roads and measured their real coordinates using the real-time kinematic (RTK) technique. The error was the deviation distance value from OSM roads to the real ones, which was calculated by the average error of the point coordinates of intersections of OSM roads and real ones. After calculating the average deviation distance, we added the deviation distance value to the distance threshold when performing position consistency. Figure 15b shows that there were 8687 polylines, and 2521 polylines were kept after feature consistency matching.

Although some skeleton lines associated with the OSM road network were retained, due to the poor completeness of OSM, some skeleton lines belonging to the regions of vegetation-obscured ISAs were removed. Thus, we further used feature recognition to find more vegetation-obscured ISAs. Figure 16 shows the result obtained using feature recognition. In feature recognition, skeleton lines with street tree structures were determined whether they are vegetation-obscured ISAs using the criteria described in Section 3.3.2. In the study area, we set the number of connections of nodes >=3, and the range of the angles of the skeleton lines is [80, 90]. In Figure 16a, there were 8687 skeleton lines. In Figure 16b, 3794 polylines were kept, whereas 1092 polylines are duplicated in Figure 15b.

By setting the retained lines as the centerlines, we generated buffers for vegetation-obscured ISAs. Different widths of buffers were set according to Table 2 in Section 4.1. After generating the buffers, we merged the results of feature consistency and feature recognition as the results of the proposed algorithm. As Figure 17 presents, we combined ISAs obtained by deep learning [28] and the proposed algorithm. Figure 17a shows the ground surface from the GF-2 image, and Figure 17b shows vegetation-obscured ISAs merged with the results from the proposed algorithm and the results from our previous deep learning algorithm.

To show the effectiveness of the proposed algorithm, we evaluated the producer’s accuracy (PA), user’s accuracy (UA), and overall accuracy (OA) of the proposed algorithm for extracting the vegetation-obscured ISAs. As shown in Figure 18, ground truth represents the vegetation-obscured ISAs in the study area, checked from the ground survey. From the result, the effectiveness of the previous deep learning algorithm in extracting vegetation-obscured ISAs is almost zero. Then, we calculated the accuracies of the proposed algorithm, and the results are shown in

Table 3. Accuracies for the proposed algorithm.

	Proposed Algorithm
PA	79.98%
UA	56.08%
OA	86.64%

.

The result demonstrates that our previous deep learning algorithm was not able to extract the vegetation-obscured ISAs from the optical remote sensing images because spectral features were weak in describing the phenomenon of multi-level distribution of features. Apparently, the proposed algorithm has good effectiveness.

5. Discussion

5.1. Deviation of Extracted Vegetation-Obscured ISAs

Since the features of different vegetation types (street trees, woods, grass) are very similar, it is easy to classify them into one object in the image segmentation stage. Such kinds of objects possibly lead to the deviation of the extracted vegetation-obscured ISAs from the real vegetation-obscured ISAs. As shown in Figure 12b, the grass, woodland, and street trees were merged into one object. Figure 14 shows that these objects led to the deviation in the thinning. In order to remove deviated skeleton lines after thinning, in feature mining, position consistency removed lines with large deviation by matching them with the OSM road network, which improves the accuracy. However, some skeleton lines with small a deviation were not removed, resulting in a lower UA.

5.2. Network Structure of Extracted Vegetation-Obscured ISAs

In feature recognition, the network structure of street trees is the reference for the extraction of vegetation-obscured ISAs. Therefore, the extracted vegetation-obscure ISAs should have an obvious network structure. In fact, in Figure 17, some of the ISAs extracted by the proposed algorithm lost their network structure and were not connected to other ISAs. The reason is that in the process of feature mining, the description of the network structure of street trees is based on local features (the number of connections of a skeleton line node to other skeleton lines and the angle that the skeleton line intersected with other skeleton lines at the node) rather than global features. In order to retain more network information of the extracted vegetation-obscured ISAs, more knowledge about network features need to be added to feature recognition, such as graph theory, etc.

5.3. Overestimation of Extracted Vegetation-Obscured ISAs

Although the proposed algorithm has proved its effectiveness in extracting ISAs, the poor recall ratio means some extracted ISAs were not in existence. As a result, the extracted ISAs were overestimated. Analysis shows: (1) areas obscured by vegetation similar to street trees are not always roads. Sometimes it is just soil underneath or some other pervious surface. Even for roads, the roads are likely to be constructed using sand and mud, as well as other pervious materials. Therefore, it is possible to overestimate ISAs. (2) ISAs were extracted from grass and woodland. As analyzed in Section 5.1, deviation happens for some ISAs. Some of these ISAs were not removed due to low deviation. However, the deviation still leads to the overestimation of ISAs, which can be observed with the spatiotemporal-spectral observation model [29] or observations from UAV and mobile mapping vehicles for better urban mapping [30].

6. Conclusions

The obscuring effect of vegetation often affects the extraction of ISAs from remote sensing images. Obscuring effect reduces the accuracy and connectivity of the extracted ISAs. When the structure of vegetation-obscured areas is similar to street trees, it is feasible to extract the ISAs under street trees. In this paper, we propose a morphological feature-oriented algorithm. The algorithm forms a processing system, including vegetation detection, mathematical morphology processing, and feature mining. The algorithm extracts vegetation-obscured ISAs by analyzing the morphological features of the OSM road network and street trees. The experiments prove the effectiveness of the proposed algorithm in the extraction of the vegetation-obscured ISAs. The overall accuracy was 86.64%.

From the analysis, the deviation and disconnectedness of the extracted ISAs reduce the precision and recall ratio. Therefore, the proposed algorithm needs further improvement: the features used in this paper are local features. The scope of the feature calculation could be extended to improve the accuracy of the extraction of vegetation-obscured ISAs. For example, knowledge of graph theory and other aspects and the network structure of features could be extracted in depth, etc. Additionally, in this paper, we evaluated the proposed algorithm to extract the vegetation-obscured ISAs with linear elements (i.e., roads). In the future work, we will exploit prior knowledge, deep learning, etc. to extend the proposed algorithm to more spatial elements, such as the ISAs of squares covered by tree canopy.