Road Extraction in SAR Images Using Ordinal Regression and Road-Topology Loss

Abstract

The road extraction task is mainly composed of two subtasks, namely, road detection and road centerline extraction. As the road detection task and road centerline extraction task are strongly correlated, in this paper, we introduce a multitask learning framework to detect roads and extract road centerlines simultaneously. For the road centerline extraction problem, existing works rely either on regression-based methods, or classification-based methods. The regression-based methods suffer from slow convergence and unsatisfactory local solutions. The classification-based methods ignore the fact that the closer the pixel is to the centerline, the higher our tolerance for its misclassification. To overcome these problems, we first convert the road centerline extraction problem into the problem of discrete normalized distance label prediction, which can be resolved by training an ordinal regressor. For the road extraction task, most of the previous studies apply pixel-wise loss function, for example, Cross-Entropy loss, which is not sufficient, as the road has special topology characteristics such as connectivity. Therefore, we propose a road-topology loss function to improve the connectivity and completeness of the extracted road. The road-topology loss function has two key characteristics: (i) The road-topology loss function combines road detection prediction and road centerline extraction prediction to promote the two subtasks to each other by using the correlation between the two subtasks; (ii) The road-topology loss can emphatically penalize gaps that often appear in road detection results and spurious segments that easily appear in centerline extraction results. In this paper, we select the AdamW optimizer to minimize the road-topology loss. Since there is no public dataset, we build a road extraction dataset to evaluate our method. State-of-the-art semantic segmentation networks (LinkNet34, DLinkNet34, DeeplabV3plus) are used as baseline methods to compare with two kinds of method. The first kind of method modifies the baseline method by adding the road centerline extraction task branch based on ordinal regression. The second kind of method uses the road topology loss and has the same network architecture as the first kind of method. For the road detection task, the two kinds of methods improve the baseline methods by up to $3.51\%$ and $11.98\%$ in $IoU$ metric on our test dataset, respectively. For the road centerline extraction task, the two kinds of methods improve the baseline methods by up to $8.22\%$ and $10.9\%$ in the Quality metric on our test dataset.

1. Introduction

Synthetic Aperture Radar(SAR) can penetrate clouds and collect data under any weather at any time. Therefore, it has been widely used in various fields. With the rapid development of SAR technology in decades, high-resolution synthetic aperture radar images have become an essential source of information due to their broader coverage and increased ground details. Roads are an indispensable part of the modern transportation network, and they have a pivotal position both geographically and economically. Automatic road extraction from SAR images has become a research hotspot. It has a wide range of urban planning applications, disaster prevention and mitigation, and geographic information system updates.

In recent decades, many methods of road extraction from SAR images have been proposed [1,2,3,4,5,6,7]. The road extraction task can be divided into two subtasks: road detection and road centerline extraction. The road extraction methods in previous works separated these two tasks, which can be roughly summarized into three steps. Firstly, the road in the SAR image is detected according to the extracted image features. Then, the road centerline is obtained by skeletalizing the road detection result. Finally, the road centerline is converted into a graph, which then is topologically optimized to obtain the road centerline network. The methods that separate the two tasks ignore the correlation between the two tasks, and the errors of road detection affect the result of road centerline extraction. The work [8] achieved a better performance using a conditional random field model to simultaneously perform the three steps: road detection, skeletonization, and topology optimization.

Since deep convolutional neural network (DCNN) has shown an outstanding performance in the 2012 ImageNet image classification challenge, DCNN [9] has made great progress in various traditional computer vision tasks such as image classification [10,11,12,13,14] and image semantic segmentation [15,16,17,18,19]. Deep learning has been widely used in the fields of remote sensing [20,21,22,23,24]. However, the works that applied deep learning to SAR images [25,26] are relatively few. One of the reasons for this is that the unique characteristics of SAR images make labeling time-consuming and labor-intensive. Reference [27] proposed a method to automatically label buildings from SAR images, which significantly reduces labeling difficulty. At the same time, reference [27] also applied the architecture combined with FCN used to extract features with CRF-RNN that exploited spatial information to extract buildings from SAR images. Although there have been fewer minimal works that apply deep learning for SAR imagery to date, it is undeniable that deep learning has colossal application potential for SAR image processing. A DisDBN which combined ensemble learning with a deep belief network to learn discriminant features is proposed in [28] for SAR image classification. Reference [29] let extracted six-channel covariance matrix fed into DCNN for PolSAR image classification. Reference [30] proposed a deep supervised and contractive neural network for SAR image classification. Reference [31] extended the semantic segmentation network from the real-value domain to the complex-valued domain, which can exploit the unique information of SAR data. Reference [32] proposed a new fully convolutional neural network that can be trained in an end-to-end for semantic segmentation of polarimetric SAR imagery in the complex land cover ecosystem. Reference [33] realized smooth classification with a small training set with four classes by deep transfer learning. Reference [34] extracted multiscale trained features using a multiscale CNN model to detect built-up areas from SAR images. Reference [35] firstly applied a deep fully convolutional neural network to segment road from SAR images.

There are two challenges that restrict the accuracy of road extraction from SAR images: (1) the road in SAR images is usually modeled as dark elongated areas surrounded by bright edges which are easily confused with other objects such as rivers, shades of trees, shadows of buildings, etc. (2) Speckle noise in SAR images seriously degrades image quality and interferes with road extraction from SAR images. A deep convolutional neural network with a large receptive field can effectively extract contextual information. This characteristic can be used to distinguish the road from other similar objects. A deep convolutional neural network reduces the influence of speckle noise by continuously performing convolution operations on the input images. Therefore, in this paper, we will introduce a deep learning framework to extract roads from SAR imagery.

The road extraction task includes two subtasks: road detection task and road centerline extraction task. Most of the previous works relied on multistage-based learning methods to extract roads. These methods obtain road centerlines by a post-processing step of thin road detection predicted by CNN. The disadvantage of these methods is that imperfect road detection results lead to road centerline extraction results with low connectivity. There is a symbiotic relationship between the road detection task and road centerline extraction task. The road detection task and road centerline extraction task can promote each other. The road detection task can provide detection cues for the road centerline extraction task to constraint road centerlines, which can avoid spurious parts. The road centerline extraction task can motivate the road detection task to pay more attention to the key points of roads, which can enhance the road connectivity. In order to make full use of this relationship, our proposed network learns the road detection task and road centerline extraction task simultaneously under the multitask learning scheme.

For road centerline extraction from raw image data, the previous works can be roughly divided into two categories: one is based on classification methods, and the other is based on regression methods. Reference [7,36,37] extracted road centerline by a classification-based method. The features of pixels on the road centerline are similar to features of pixels adjacent to the centerline, and the features of pixels far from the centerline of the road are completely different from the pixels in the centerline of the road. However, the errors caused by the misclassification of pixels adjacent to the centerline of the road are the same as those caused by the misclassification of pixels far from the centerline of the road, which is unreasonable. In practice, the closer the pixel is to the centerline of the road, the more we can tolerate its classification errors. This contradiction makes it difficult for the classification-based to cause the to network converge to a better result. Reference [38] first proposed method-based regression, which learned a designed function whose returns value decreases with the distance from the pixel to the centerline. However, due to abnormal, i.e., annotation errors, a deep network for regression is relatively unstable, and the network trained by MSE loss will not converge to a satisfactory global solution. Reference [39] learned the map of distance from each pixel to the nearest boundary by training a multi-class classification network, which ignores the ranking relation between different distance classes. To avoid the above problems, we exploit the method based on ordinal regression to learn the discrete normalized distance labels. We use the ordinal loss, minimized to learn network parameters of the road centerline task.

In the real world, the roads have unique topology properties. Previous works usually applied topology prior to using variational and Markov random field-based methods [36,40,41,42]. Reference [40] imposed a topology constraint by high-order CRF, in which high-order cliques connect superpixels of the road network. Reference [36] represented the road network as a sequence of graph structures and found an optimal subgraph by integer programming. These previous works generally employed road topology optimization as post-processing, which cannot remove large spurious parts and connect large gaps. Recently, some approaches guaranteed a perfect topology of the extracted road by minimizing a topology-preserving loss function [43,44]. Reference [43] stated that only using pixel-wise loss functions is not enough for curvilinear detection and proposes the topology-aware loss function defined by the selection of the filter responses of pretrained VGG19 to penalty topology errors. Reference [44] adopted a loss function based on Persistent Homology, which is continuous. Neither VGG19 nor Betti are specifically designed for road extraction tasks, so their penalty for topology errors of extracted road network is limited To solve the above-mentioned problems, a new road-topology loss is specially designed for the road extraction task, which can reduce the topology errors. Our main contributions are as follows:

• Different from the previous methods for road extraction from SAR imagery, we detect the road and extract road centerline simultaneously. This multitask learning scheme can exploit the correlation between road detection task and road centerline extraction task;
• For the road extraction task, we build our dataset with TerraSAR-X images, which cover urban, suburbs and rural areas. Our experiments are carried out based on this dataset. Our experimental results show that our proposed framework can achieve a better road extraction performance;
• For the road centerline extraction task, we first convert the road centerline extraction problem into the problem of discrete normalized distance label prediction, which can be solved by training an ordinal regressor;
• Consider the special topology feature of road network, we propose a new road-topology loss which is designed to reduce the topology error of road extraction including spurious parts and gaps.

The remainder of our paper is organized as follows. We present the proposed method in detail in Section 2. In Section 3, we quantitatively and qualitatively analyzed the superior performance of our method compared with baseline methods. We discuss the stability of different methods with various binarization thresholds in Section 4. Finally, we conclude the whole paper in Section 5.

2. Materials and Methods

Figure 1 illustrates our road extraction framework. For road extraction, our proposed network learns the road detection task and the road centerline extraction task jointly under a multitask learning scheme. As shown in Figure 1, our framework has two branches: the road detection branch and the road centerline extraction branch. The encoder of the two branches is shared for feature extraction, which establishes a connection between the two branches. In this section, we first separately introduce how our network performs road detection tasks and road centerline extraction tasks. Next, the definition of our first proposed road-topology loss function is given. Finally, we introduced how our multitask learning framework simultaneously learns the road detection task and road centerline extraction task based on ordinal regression, using the road-topology loss that we first proposed.In the following discussion, we let $I\in R^{H\times W}$ be the $H\times W$ input image, let $Y\in {\{0,1\}}^{H\times W}$ be the corresponding ground truth, with 1 indicating pixels on the road and 0 indicating background pixels, and let $\widehat{Y}\in {\{0,1\}}^{H\times W}$ be the predicted probability map of the road. We let the mini-batch be B, let i be a pixel in I, $y_i$ is label of pixel i. The predicted probability that pixel i is on the road is denoted by $\widehat{y_i}$.

2.1. Road Detection

The road detection task aims to detect the roads from SAR imagery. The output of the road detection task is the binary image, in which the pixels seen on the roads are 1 and the others are 0. In practice, most pixels of the SAR imagery belong to the non-road regions. As a result, there is a label-unbalancing problem in the road detection task. To  overcome this problem, we use weighted cross-entropy loss proposed in [45]. The weighted cross-entropy loss of I is

$$L_{rs}=-\frac{1}{H\times W}\sum _{i\in I}{w}_1{y}_{i}log\widehat{y_i}+w_0(1-y_i)log(1-\widehat{y_i}).$$

(1)

Next, we will present the weights in the weighted cross-entropy loss. Let $\left|B\right|$ be the number of the images. The loss weight for road pixels is $w_1=\frac{1}{\left|B\right|\times H\times W}{\sum }_{I\in B}{\sum }_{i\in I}1(y_i==0)$ and the loss weight for non-road pixels is $w_0=\frac{1}{\left|B\right|\times H\times W}{\sum }_{I\in B}{\sum }_{i\in I}1(y_i==1)$.

2.2. Road Centerline Extraction

For road centerline extarction, the classification-based approach learn a function $y(·)$ such that

$$y\left(f_i\right)=\left\{\begin{array}{cc}1,\hfill & i \mathrm{is} \mathrm{on} \mathrm{centerline},\hfill \\ 0,\hfill & i \mathrm{is} \mathrm{not} \mathrm{on} \mathrm{centerline},\hfill \end{array}\right.$$

(2)

where $f_i$ is the feature of pixel i. The method based on regression is to learn a regressor $y(·)$ whose values decrease monotonically as the distance of i to the centerline increase. Especially in [38], the regressor $y(·)$ is such that

$$y\left(f_i\right)=\left\{\begin{array}{cc}1-\frac{D_C\left(i\right)}{d_M},\hfill & \mathrm{if} D_C\left(i\right)\le d_M,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(3)

where $D_C\left(i\right)$ is the metric distance from pixel i to the closest the pixel on the centerline and $d_M$ is $s/2$, where s is the size of local neighbourhoods that are used to compute feature vector $f_i$. Our proposed method is based on ordinal regression, which is different from both. In the remainder of this subsection, we first model the road centerline extraction problem as the discrete normalized distance label prediction problem. We then describe how to predict discrete normalized distance labels by learning an ordinal regressor.

Roads are surrounded by bright edges in high-resolution SAR images. As a result, we can predict the distance $d_i$ from any pixel i to the nearest road edge. However, the probability that the pixel i is on the centerline of the road is not proportional to the distance from this pixel i to the nearest road edge $d_i$. This is due to the width of the road is various, which is depicted in Figure 2. As a result, we predict the normalized distance from pixel i to the nearest road edge. The normalized distance $d{n}_i$ is defined as follows

$$d{n}_i=\left\{\begin{array}{cc}2\ast \frac{d_i}{w_i},\hfill & i \mathrm{is} \mathrm{on} \mathrm{road},\hfill \\ 0,\hfill & i \mathrm{is} \mathrm{not} \mathrm{on} \mathrm{road},\hfill \end{array}\right.$$

(4)

where $w_i$ is the road width of pixel i. In particular, $d{n}_i$ is proportional to the probability of i on the centerline of the road. Meanwhile, if i is on the road centerline, $d{n}_i$ is the local maxima value along the direction that is perpendicular to the direction of the road. We further quantize each $d{n}_i$ using the thresholds $\{t_0,t_1,\dots ,t_{K-1}\}$ into one of the $K+1$ intervals. The reason we quantify the normalized distance $d{n}_i$ is that the direct training of the deep network for regression is relatively unstable, because outliers( annotation errors) cause large error terms, making it difficult for the network to converge and lead to unstable predictions [39]. After quantization, each i is given a discrete normalized distance label $l_i$, such as

$$l_i=\left\{\begin{array}{cc}0\hfill & d{n}_i\le t_0.\hfill \\ k+1\hfill & t_k\le d{n}_i<t_{k+1}k=(0,1,2,\dots ,K-2).\hfill \\ K\hfill & t_{K-1}\le d{n}_i.\hfill \end{array}\right.$$

(5)

We can predict discrete normalized distance label prediction by a typical method based on multi-class classification. However, the ordinal information between discrete normalized distance labels will be ignored. [46] first combines ordinal regression with DCNNs to address the age estimation, which transforms an ordinal regression problem into a series of binary classification sub-problems to achieve sequential age regression tasks, thus taking accounting for the fact that the set of ages is well-ordered. Therefore, we use ordinal regression in [46] to solve the discrete normalized distance label prediction problem and modify the ordered loss to adapt to the road centerline extraction task.

Next, we will introduce the ordinal regression and the ordinal loss used in this paper in detail. Let $\phi$ denote the feature extractor of network that is used to extract the road centerlines and the parameters of $\phi$ are denoted by $\mathsf{\Phi }$. $I\in R^{H\times W}$ is input image. One of the pixels in I is denoted as i. The feature map of I and the feature vector of pixel i for the road centerlne extarction task are $F=\phi (I,\mathsf{\Phi })$ and $f_i\in F$, respectively. $\psi$ is the last layer of the network for the road centerline extarction task, which is used for ordinal regression. Its parameters are given by $\mathsf{\Theta }=\{{\theta }_0,{\theta }_1,\dots ,{\theta }_{2K-1}\}$ where ${\theta }_j(j\in \{0,1,\dots ,2K-1\})$ is the weight vector. Ordianl output of I and ordianl output vector of i are presented by $O=\psi (F,\mathsf{\Theta })$ and $o_i\in O$ ($o_i=(o_i^0,o_i^1,\dots ,o_i^{2K-1})$), respectively. With the softmax activation function that is also used in [46], the probability $\widehat{p_i^k}=P(\widehat{l_i}>k|F,\mathsf{\Theta })(k\in \{0,1,\dots ,K-1\})$ that the predicted label of i is greater than k is calculated as

$$\begin{array}{cc}\hfill & \widehat{p_i^k}=\frac{e^{o_i^{2k+1}}}{e^{o_i^{2k}}+e^{o_i^{2k+1}}}.\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & o_i^{2k}={\theta }_{2k}^T{f}_i,\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & o_i^{2k+1}={\theta }_{2k+1}^T{f}_i.\hfill \end{array}$$

(8)

According to the method of calculating the ordinal loss in [46], the pixel-level ordinal loss of pixel i is then given by

$$\mathsf{\Psi }(f_i,\mathsf{\Theta })=-(\sum _{k=0}^{l_i-1}{\omega }_1^{k}log\left(\widehat{p_i^k}\right)+\sum _{k=l_i}^{K-1}{w}_0^k(log(1-\widehat{p_i^k}))),$$

(9)

where ${\omega }_1^k=\frac{1}{\left|B\right|\times H\times W}{\sum }_{I\in B}{\sum }_{i\in I}1(l_i\le k)$ and ${\omega }_0^k=\frac{1}{\left|B\right|\times H\times W}{\sum }_{I\in B}{\sum }_{i\in I}1(l_i>k)$ are used to solve the unbalanced classes problem.

The ordinal loss of I is defined as the sum of the ordianl loss of each i in image I and given by

$$L_c(F,\mathsf{\Theta })=\frac{1}{H\times W}\sum _{i\in I}\mathsf{\Psi }(f_i,\mathsf{\Theta }),$$

(10)

The advantage of the ordinal loss is that the greater the difference between the predicted label $\widehat{l_i}$ and the true label $l_i$, the greater the ordinal loss. We use iterative optimization algorithm to minimize $L_c(F,\mathsf{\Theta })$. The partial derivative of $L_c(F,\mathsf{\Theta })$ with respect to ${\theta }_j(j\in \{0,1,\dots ,2K-1\})$, is

$$\frac{\partial L_c(F,\mathsf{\Theta })}{\partial {\theta }_j}=\frac{1}{H\times W}\sum _{i\in I}\frac{\partial \mathsf{\Psi }(f_i,\mathsf{\Theta })}{\partial {\theta }_j},$$

(11)

$$\frac{\partial \mathsf{\Psi }(f_i,\mathsf{\Theta })}{\partial {\theta }_{2k+1}}=-\frac{\partial \mathsf{\Psi }(f_i,\mathsf{\Theta })}{\partial {\theta }_{2k}},$$

(12)

$$\begin{array}{cc}\hfill \frac{\partial \mathsf{\Psi }(f_i,\mathsf{\Theta })}{\partial {\theta }_{2k}}& =-({\omega }_1^k{f}_i\eta (l_i>k)(\widehat{p_i^k}-1)\hfill \\ \hfill & +{\omega }_0^k{f}_i\eta (l_i\le k)\widehat{p_i^k}),\hfill \end{array}$$

(13)

where $k\in \{0,1,\dots ,K-1\}$ and $\eta (·)$ is the indicator function, where $\eta \left(true\right)=1$ and $\eta \left(false\right)=0$. We can update the parameters of the network for road centerline extraction task through backward propagation.

In the test phase, we calculate $\widehat{p_i}$, which is the mean of $\widehat{p_i^k}$ for each pixel i. $\widehat{p_i}$ is given by

$$\widehat{p_i}=\left(\sum _{k=0}^{K-1}\widehat{p_i^k}\right)/K.$$

(14)

We observe that $\widehat{p_i}$ is proportional to the normalized distance $d{n}_i$. We can regard $\widehat{p_i}$ as the predicted probability that the pixel i is on the road centerline. Let $\widehat{P}$ be the centerline probability map of I, where $\widehat{p_i}\in \widehat{P}$. According to Formula (4), we know that, from the normalized distance map $D_N$, the necessary and sufficient condition for pixel i on the road centerline is that $d{n}_i$ is a local maximum along the direction perpendicular to the direction of the road. However, if only by judging whether $p_i$ is a local maximum along the direction perpendicular to the direction of the road to infer whether pixel i is on the road centerline, some non-road regions will be extracted as the road centerline. As a result, we first set the value of $\widehat{P}$ that is less than T to zero, then we apply the canny-like non-maximum suppression algorithm to $\widehat{P}$ to obtain the road centerline.

2.3. Road-Topology Loss

In practice, the cross-entropy loss is widely used in various segmentation tasks, such as semantic segmentation and instance segmentation. Cross-entropy loss is a pixel-wise loss, which is completely local and does not take the special and complex topological characteristics of the road into account. Such a loss penalizes the mistake of each pixel equally and independently, regardless of the effect of the error on geometry. However, in practice, we find that the pixels closer to the centerline of the road are more important. because the misclassification of these pixels will cause serious topology errors, such as gaps and spurious parts, To penalize the gaps in the prediction of road detection and the spurious parts in the road centerline extraction prediction, we propose a new road-topology loss $L_T$.

Next, we will give the details defining the road- topology loss. To measure the connectivity of road detection prediction, we define the connectivity metric as

$$T^{con}=\frac{|\widehat{Y}\circ L|}{\left|L\right|},$$

(15)

where $\widehat{Y}\in {[0,1]}^{H\times W}$ is a prediction map of road detection, $L\in {\{0,1,\dots ,K\}}^{H\times W}$ ($l_i\in L$) is discete normalized distance label of I and $\left|\right|$ is an operation to calculate the sum of matric. Similarly, we define the differentiable correctness metric to measure the correctness of road centerline extraction prediction as

$$T^{cor}=\frac{|Y\circ \widehat{P}|}{|\widehat{P}|},$$

(16)

where $Y\in {\{0,1\}}^{H\times W}$ is ground truth of road detection, $\widehat{P}\in {[0,1]}^{H\times W}(\widehat{p_i}\in \widehat{P})$ is predicted road centerline probability map of I. We observe that the measure $T^{con}$ is susceptible to gaps in the road detection prediction while the measure $T^{cor}$ is susceptible to spurious parts in the road centerline extarction prediction. Finally, we define road-topology metric $T^{Road}$ as the harmonic average between connectivity metric $T^{con}$ and differentiable correctness metric $T^{cor}$ as

$$T^{Road}(\widehat{Y},\widehat{P},Y,L)=\frac{2·T^{con}·T^{cor}}{T^{con}+T^{cor}}.$$

(17)

The road-topology metric measures the connectivity and correctness of the road extraction result at the same time.

In order to maximize the road-topology metric in CNNs in an end-to-end manner, we define our road-topology loss $L_T$ as

$$L_T=-log\left(T^{Road}\right)$$

(18)

$L_T$ is calculated directly from the raw prediction $\widehat{Y}$ and $\widehat{P}$ without thresholding. As a result, $L_T$ is differentiable over the prediction $\widehat{Y}$ and $\widehat{P}$ which can be integrated into CNN. In this paper, we use AdamW optimizer to minimize the road-topology loss.The partial derivatives of loss $L_T$ over network activation $\widehat{Y}$ and $\widehat{P}$ at the location of pixel i are

$$\begin{array}{cc}\hfill \frac{\partial L_T}{\partial \widehat{y_i}}& =-\frac{1}{T^{Road}}\frac{\partial T^{Road}}{\partial \widehat{y_i}}\hfill \\ \hfill & =-\frac{2}{T^{Road}}(\frac{T^{con}}{T^{cor}+T^{con}}-\frac{T^{con}·T^{cor}}{{(T^{con}+T^{cor})}^2})·\frac{\partial T^{con}}{\partial \widehat{y_i}}\hfill \\ \hfill & =-\frac{1}{T^{con}(T^{con}+T^{cor})}·\frac{\partial T^{con}}{\partial \widehat{y_i}}\hfill \\ \hfill & =-\frac{1}{T^{con}(T^{con}+T^{cor})}·\frac{l_i}{\left|L\right|}.\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \frac{\partial L_T}{\partial \widehat{p_i}}& =-\frac{1}{T^{Road}}\frac{\partial T^{Road}}{\partial \widehat{p_i}}\hfill \\ \hfill & =-\frac{2}{T^{Road}}(\frac{T^{cor}}{T^{cor}+T^{con}}-\frac{T^{con}·T^{cor}}{{(T^{con}+T^{cor})}^2})·\frac{\partial T^{cor}}{\partial \widehat{p_i}}\hfill \\ \hfill & =-\frac{1}{T^{cor}(T^{con}+T^{cor})}·\frac{\partial T^{cor}}{\partial \widehat{p_i}}\hfill \\ \hfill & =-\frac{1}{T^{cor}(T^{con}+T^{cor})}·\frac{y_i·|\widehat{P}|-|Y\circ \widehat{P}|}{|\widehat{P}{|}^2}.\hfill \end{array}$$

(20)

2.4. Multitask Learning

In our road extraction framework, the input image I feeds into the shared encoder to extract features. The feature maps are, respectively, input into the decoders corresponding to the two tasks to obtain road detection prediction and road centerline extraction prediction. As shown in Figure 1, the prediction of the road detection task $\widehat{Y}$ and the ground truth Y are used to calculate the weighted cross-entropy loss that can be minimized to update the parameters of the road detection network. The prediction of the road centerline extraction task $\widehat{Y}$ and the discrete normalized distance label map L are used to calculate the ordinal loss that can be minimized to update the parameters of the road centerline extraction network. Y, $\widehat{Y}$, $\widehat{L}$, and L are used to calculate our proposed road-topology loss that combines the prediction of road detection and the prediction of road centerline extraction. Our proposed road topology loss makes full use of the correlation between the two tasks and can be minimized to make the two tasks promote each other. The entire loss function is the sum of the cross-entropy loss, the ordinal loss, and road-topology loss. By minimizing the entire loss function, the parameters of the road detection network and the parameters of the road centerline extraction network can be updated simultaneously, which means our framework learns teh road detection task and road centerline extraction task. The loss of mini-batch is calculated by

$$L=\frac{1}{\left|B\right|}\sum _{I\in B}(L_{rs}+L_c+L_T).$$

(21)

3. Experiments

3.1. Dataset

In this subsection, we aim to present the dataset used in this paper. There is no public dataset that is applicable to our research. We create our dataset by using high-resolution TerraSAR-X images that are obtained by striped mode. As shown in

Table 1. Metadata of the TerraSAR-X Images Used in Our Data Set.

Beijing, China
Size	21,800 × 15,500 px
Range Sample Distance	0.909627 m/px
Azimuth Sample Distance	1.848561 m/px
Spatial Resolution	3 m/px
Center Coordinate	[39.8798466,116.4503446]
Polarization	HH
Date	2013-05-09
Beijing, China
Size	27,600 × 18,700 px
Range Sample Distance	0.908790 m/px
Azimuth Sample Distance	1.888833 m/px
Spatial Resolution	3 m/px
Center Coordinate	[39.9571641,116.6996268]
Polarization	HH
Date	2013-03-04

, we label the roads in two SAR images, the coverage areas of which include urban, suburb, and rural areas. The Google Earth Maps of study area are shown in Appendix A. As the dataset is applied to extract roads, we only label all roads in the region where the road network is dense. Our labeled area was split into a training and test set as follows: the upper 80 % of the area ($435\times 1024\times 1024$ pixels) was used for training, and the lower 20% ($104\times 1024\times 1024$ pixels ) for testing. We use the raw SAR intensity image without any preprocessing. There are speckle noise in SAR images. Our train set contains raw SAR patches, the ground truth of road, and the ground truth of discrete labels. Algorithm 1 describes how road centerline and road width ground truth can be outlined. Figure 3 shows a sample in the train set. There is the only ground truth of road and ground truth of road centerline in the test set.

Algorithm 1 Obtain Road Centerline and Road Width Ground Truth

Input: road ground truth Y

Output: Road centerline ground truth C, Road width ground truth W

1:  Calculate the distance map D 2:  From distance map D, calculate the horizontal gradient map $D_X=D⨂f$, where $f=[-1,2,-1]$ 3:  From distance map D, calculate the vertical gradient map $D_Y=D⨂f^T$, where $f=[-1,2,-1]$ 4:  Calculate the orientation map $OR=ta{n}^{-1}(DY,DX)$ that is perpendicular to road direction. 5:  for (do $i\in I$ and $y_i==1$) 6:      if $d_i$ is local maximum along the direction $o{r}_i$ then 7:          i is on the road centerline: $c_i=1$ 8:          the road width of pixel i: $w_i=2\ast d_i$ 9:      else 10:          i is not on the road centerline: $c_i=0$ 11:          the road width of pixel i: $w_i=2\ast d_j$, where $d_j$ is the local maximum starting from i along the direction $o_i$ in the distance map D.

3.2. Evaluation Metrics

To assess the qualitative performance in both road detection task and road road centerline extraction task, we apply the metrics that are introduced in [47]. In this paper, Our framework can detect road and extract road centerline simultaneously. To highlight the characteristics of our method, we propose a series of metrics to evaluate the performance of road extraction task.

3.2.1. Road Detection

The metrics that are employed to evaluate the performance of our approach for road detection task are precision, recall, F1-score, and intersection over union (IoU) that is the ratio of the intersection of prediction and groundtruth to the union of prediction and groundtruth. Precision (P) measures the ratio of the number of the pixels which are labeled as road pixels in the ground truth and are predicted as road pixels to the number of pixels that are inferred as road pixels. Recall (R) calculates the ratio of the number of the pixels which are labeled as road pixels in the ground truth and are predicted as road pixels to the number of pixels that are labeled as road pixels. F1-score($F{1}_{rd}$) is used to balance precision and recall, which is a harmonic average between precision and recall. IoU is the ratio of the intersection and union of the true label and predicted result, which can trade-off between recall and precision. Specifically, the four metrics are defined as:

$$\begin{array}{c}P=\frac{TP}{TP+FP},\end{array}$$

(22)

$$\begin{array}{c}R=\frac{TP}{TP+FN},\end{array}$$

(23)

$$\begin{array}{c}F{1}_{rd}=\frac{2\ast Precision\ast Recall}{Precision+Recall},\end{array}$$

(24)

$$\begin{array}{c}IoU=\frac{TP}{TP+FN+FP},\end{array}$$

(25)

where $TP$ is true positive, $FP$ is false positive and $FN$ is false negative.

As there is a deviation between the manually labelled roads and the real roads, we relax metrics using the buffer method given in [48]. Specifically, if the regions in the prediction result are within the $\rho$ pixels range, they are regarded as matching regions. In this paper, we set $\rho =2$.

3.2.2. Road Centerline Extraction

Due to the differences between the road detection task and road centerline extarction task, it is better to use different metrics for the road centerline task. Next, we present the metrics that are used for the road centerline extarction task. We calculate completeness, correctness, quality, and F1-score to assess the performance of our approach for the road centerline extraction task. As it was difficult to directly compare the pixel difference between the extracted centerline and the ground truth, we introduced a buffer-based evaluation for the road centerlines. Completeness (COM) is the ratio of the length of reference road centerline that lies within the buffer around the around the extracted centerline to the length of reference centerline. Correctness (COR) is the ratio of the extracted road centerline that lies within the buffer around the reference centerline to the length of the extracted centerline. Quality (Q) is a comprehensive metric that combines the completeness and correctness. F1-score ($F{1}_{rce}$) is used to balance COM and COR, which is a harmonic average between COM and COR. The four metrics can be calculated as

$$\begin{array}{c}COM=\frac{\mathrm{length} \mathrm{of} \mathrm{matched} \mathrm{reference}}{\mathrm{length} \mathrm{of} \mathrm{reference}}\approx \frac{TP}{TP+FN},\end{array}$$

(26)

$$\begin{array}{c}COR=\frac{\mathrm{length} \mathrm{of} \mathrm{matched} \mathrm{extraction}}{\mathrm{length} \mathrm{of} \mathrm{extarction}}\approx \frac{TP}{TP+FP},\end{array}$$

(27)

$$\begin{array}{c}Q=\frac{\mathrm{length} \mathrm{of} \mathrm{matched} \mathrm{extarction}}{\mathrm{length} \mathrm{of} \mathrm{extarction} + \mathrm{length} \mathrm{of} \mathrm{unmatched} \mathrm{reference}}\approx \frac{TP}{TP+FN+FP},\end{array}$$

(28)

$$\begin{array}{c}F{1}_{rce}=\frac{2\ast COM\ast COR}{COM+COR}\approx \frac{2\ast P\ast R}{P+R}.\end{array}$$

(29)

3.2.3. Road Extraction

From the definition, we know that the metrics of road detection: precision (P), recall (R), IoU, F1-score ($F{1}_{rd}$) correspond to the the metrics of road centerline extraction: correctness (COR), completeness (COM), quality (Q), F1-score ($F{1}_{rce}$), respectively. As a result, we design four metrics: precision for road extraction ($P_{re}$), recall for road extraction ($R_{re}$), quality for road extraction ($Q_{re}$), F1-score for road extraction ($F{1}_{re}$) to evaluate the performance for road extraction. The metrics for road extarction are given as

$$\begin{array}{c}{P}_{re}=\frac{\alpha \ast P+\beta \ast COR}{\alpha +\beta },\end{array}$$

(30)

$$\begin{array}{c}{R}_{re}=\frac{\alpha \ast R+\beta \ast COM}{\alpha +\beta },\end{array}$$

(31)

$$\begin{array}{c}{Q}_{re}=\frac{\alpha \ast IoU+\beta \ast Q}{\alpha +\beta },\end{array}$$

(32)

$$\begin{array}{c}F{1}_{re}=\frac{\alpha \ast F{1}_{rd}+\beta F{1}_{rce}}{\alpha +\beta },\end{array}$$

(33)

where $\alpha$ and $\beta$ can be set according to the importance of the road detection task and the road centerline extraction task. If the evaluator pays more attention to road detection task, then $\alpha >\beta$. Otherwise, we set $\alpha <\beta$. If the road detection task and road centerline extraction task are equally important, then we set $\alpha =\beta$.

3.3. Implementation Details

In this subsection, we will specifically present the values of all hyperparameters in our experiment. We adopt the Pytorch framework to implement networks trained on a single NVIDIA Tesla V100 with 16G memory using a batch size of one. We train the networks with AdamW optimizer with the initial learning rate of 0.001, and we drop the learning rate by the factor of 0.1 at every ten epochs. We applied data augmentation to the training set with image rotation and horizontal and vertical flips. The augmented training set is composed of 3480 SAR images, the size of which is $1024\times 1024$. For road centerline extraction, we set the threshold T as 0.25. As T only works in the preprocessing step of our test stage, this will not affect the final road centerline extraction result too much. If the parameter T is set too large, there will be more discontinuities in the road centerline extraction results. In practice, we do not know the optimal value of T for a test image without ground truth, and users can set T according to their needs. If users do not want spurious parts in the results, they can set a larger T. Conversely, if the user prefers to ensure the completeness of the road centerline extraction result, T can be set to a smaller value. In this paper, we set the T to 0.25 that is neither too big nor too small to remove some spurious parts and guarantee the connectivity of the road centerlines.

3.4. Results

In this subsection, we first introduce the networks that are used as baseline methods in this paper. We choose three fully convolutional neural networks (FCNNs), including LinkNet34 [48], DLinkNet34 [49], and DeeplabV3plus [50] as baseline methods to study the performance of our method. These three networks were implemented to extract roads from optical remote sensing imagery. To adapt to the size of images in our dataset, we set the dilation rates of dilated conventional operations in DeeplabV3plus to $[2,4,8,16]$. To verify that both the proposed multitask architecture and loss function are effective, we will perform three sets of comparative experiments. Each set of comparative experiments contains three methods, including the baseline method, the method I, the method II. As shown in Figure 4, (a) is the network architecture of the baseline method, while (b) and (c) are the network architectures of the method I and the method II, respectively, which both have two branches: road detection branch and road centerline extraction branch. The methods I in the three sets of comparative experiments are, respectively, abbreviated as LinkNet34+, DLinkNet34+, and DeeplabV3plus+, which are obtained by modifying the networks of the baseline methods with dual and identical decoders having shared encoders. The methods II in the three sets of comparative experiments are, respectively, abbreviated as LinkNet34++, DLinkNet34++, and DeeplabV3plus++, which use the road topology loss to update the network parameters and have the same network architecture as the methods I.

Comparative Evaluation on Road Detection

To compare the performance of road detection, all the methods are evaluated based on the test samples in the test set for road detection. For qualitative comparison, we show the results produced by all methods based on example images depicted in Figure 5. The quantitative comparisons are reported in

Table 2. Comparative quantitative Evaluation Among Different Methods for Road Detection on our dataset. It should be noted that the results are the average performance of all images in the test set. With the best results marked in bold.

Methods Names	P	R	IoU	F1
DeeplabV3plus	0.9153	0.6341	0.5629	0.7378
DeeplabV3plus+	0.9039	0.6666	0.5980	0.7534
DeeplabV3plus++	0.8691	0.7673	0.6827	0.8069
DLinkNet34	0.9295	0.6300	0.5666	0.7382
DLinkNet34+	0.9314	0.6478	0.5899	0.7507
DLinkNet34++	0.8922	0.7424	0.6707	0.8009
LinkNet34	0.9252	0.6565	0.5918	0.7554
LinkNet34+	0.9097	0.6897	0.6216	0.7724
LinkNet34++	0.8948	0.7415	0.6693	0.8007

.

As shown in Figure 5, the methods with multitask learning architecture and using road-topology loss function generally perform better than the baseline methods. The baseline methods miss road regions in many places, i.e., the false negative (green) part is large, which leads to the poor connectivity of road detection result. With the learning road detection task and road centerline extraction based on ordinal regression task jointly, the false negative part is smaller. By further using the road-topology loss function that can penalize discontinuous parts of the detected road, the performance of road connectivity is improved. Table 2 presents the comparative quantitative evaluation measured in terms of P, R, IoU, F1. In Table 2, the best value of performance is presented with bold style. We observe from the Table 2 that the proposed methods outperform baselines in terms of three metrics, i.e., R, IoU, F1. Though the precision of proposed methods is less than the baseline, the slight decrease in P is insignificant compared to the large increase in the other three metrics.

3.5. Comparison of Road Centerline Extraction

In this subsection, we present the result of a comparative evaluation based on the centerline extraction task. For LinkNet34, DLinkNet34, DeeplabV3plus, we applied the morphological thinning algorithm [51] to the road detection results so as to extract road centerline. Figure 6 illustrates the centerlines identified by different methods. From Figure 6, we can see that more discontinuities are observed in road centerline extraction results of baseline methods. As shown in Figure 6, by learning the road centerline extraction task based on ordinal regression to extract road centerline, false negative parts in the road centerline extraction results decrease, and the connectivity of the centerline network is enhanced. Figure 6 also shows that By using road topology loss, the quality of road centerline extraction results has been further improved. The one reason is that our method can learn centerline extraction task that is based on ordinal regression and the road detection task simultaneously under a multitask learning scheme, whereas the prediction of one subtask can bootstrap the performance of solving another subtask. The other reason is that minimizing our firstly proposed road-topology loss helps eliminate topology errors of road extraction results, which include spurious parts and gaps.

Table 3. Comparative quantitative Evaluation Among Different Methods for Road Centerline Extraction on our dataset. It Should Be Noted That the Results are the Average Performance of all Images in Test Set. With the best results marked in bold.

Methods Names	COR	COM	Q	F1
DeeplabV3plus	0.9532	0.7320	0.6374	0.8192
DeeplabV3plus+	0.9244	0.7860	0.6880	0.8379
DeeplabV3plus++	0.9138	0.8346	0.7360	0.8658
DLinkNet34	0.9670	0.6929	0.6187	0.7955
DLinkNet34+	0.9350	0.7839	0.6976	0.8456
DLinkNet34++	0.9150	0.8276	0.7277	0.8618
LinkNet34	0.9668	0.7102	0.6363	0.8076
LinkNet34+	0.9063	0.8237	0.7185	0.8561
LinkNet34++	0.9171	0.8265	0.7255	0.8623

summarized the results evaluated by COM, COR, Q, and F1 metrics. From Table 3, we can see that the methods with centerline extraction based on ordinal regression perform better than baseline methods. The methods with centerline extraction based ordinal regression and road-topology loss achieve the best performance.

3.6. Comparison of Road Extraction

In this subsection, we present the result of a comparative evaluation based on the road extraction task. To compare the performance of road extraction qualitatively, we adapt our proposed metrics with different values of $\alpha$ and $\beta$ to evaluate different methods. For qualitative comparison, we show the results produced by all methods based on example images depicted in Figure 6. The quantitative comparisons are reported in

Table 4. Comparative quantitative Evaluation Among Different Methods for Road Extraction on our dataset. With the best results marked in bold. $\alpha =1,\beta =1$.

Methods Names	${\mathit{P}}_{\mathit{r}\mathit{e}}$	${\mathit{R}}_{\mathit{r}\mathit{e}}$	${\mathit{Q}}_{\mathit{r}\mathit{e}}$	$\mathit{F}{1}_{\mathit{r}\mathit{e}}$
DeeplabV3plus	0.9342	0.6831	0.6002	0.7785
DeeplabV3plus+	0.9141	0.7263	0.6430	0.7957
DeeplabV3plus++	0.8915	0.8010	0.7094	0.8364
DLinkNet34	0.9483	0.6615	0.5926	0.7669
DLinkNet34+	0.9332	0.7159	0.6437	0.7981
DLinkNet34++	0.9036	0.7850	0.6992	0.8313
LinkNet34	0.9460	0.6833	0.6141	0.7815
LinkNet34+	0.9080	0.7567	0.6701	0.8143
LinkNet34++	0.9060	0.7840	0.6974	0.8315

,

Table 5. Comparative quantitative Evaluation among Different Methods for Road Extraction on our dataset. With the best results marked in bold. $\alpha =2,\beta =1$.

Methods Names	${\mathit{P}}_{\mathbf{re}}$	${\mathit{R}}_{\mathbf{re}}$	${\mathit{Q}}_{\mathbf{re}}$	$\mathit{F}{1}_{\mathbf{re}}$
DeeplabV3plus	0.9279	0.6668	0.5877	0.7649
DeeplabV3plus+	0.9107	0.7064	0.6280	0.7816
DeeplabV3plus++	0.8840	0.7897	0.7005	0.8265
DLinkNet34	0.9420	0.6510	0.5839	0.7573
DLinkNet34+	0.9326	0.6932	0.6258	0.7823
DLinkNet34++	0.8998	0.7708	0.6897	0.8212
LinkNet34	0.9391	0.6744	0.6067	0.7728
LinkNet34+	0.9086	0.7344	0.6539	0.8003
LinkNet34++	0.9022	0.7699	0.6880	0.8213

and

Table 6. Comparative quantitative Evaluation Among Different Methods for Road Extraction on our dataset. With the best results marked in bold. $\alpha =1,\beta =2$.

Methods Names	${\mathit{P}}_{\mathbf{re}}$	${\mathit{R}}_{\mathbf{re}}$	${\mathit{Q}}_{\mathbf{re}}$	$\mathit{F}{1}_{\mathbf{re}}$
DeeplabV3plus	0.9405	0.6994	0.6126	0.7921
DeeplabV3plus+	0.9175	0.7462	0.6580	0.8098
DeeplabV3plus++	0.8989	0.8122	0.7183	0.8462
DLinkNet34	0.9545	0.6720	0.6013	0.7764
DLinkNet34+	0.9338	0.7386	0.6617	0.8140
DLinkNet34++	0.9074	0.7992	0.7087	0.8415
LinkNet34	0.9529	0.6923	0.6215	0.7902
LinkNet34+	0.9075	0.7790	0.6862	0.8282
LinkNet34++	0.9097	0.7982	0.7068	0.8418

. From Figure 7, we can see that the poor road detection results of the baseline method lead to many discontinuities in the road centerline extraction results. As shown in Figure 7, by learning the road detection task and the road centerline extraction task based on ordinal regression to extract the road centerline, the performances of both tasks were improved. Figure 7 also shows that, by using our first proposed road-topology loss, the continuity of road detection result is enhanced, and the quality of road detection result is increased. This indicates that minimizing the road-topology loss helps eliminate topology errors in the road extraction results, which include spurious parts and gaps. Table 4, Table 5 and Table 6 summarized the results evaluated by $P_{re}$, $R_{re}$, $Q_{re}$, and $F{1}_{re}$ metrics. We can observe that whether the evaluator pays more attention to the road detection task or the road centerline extraction task, the method II that learns road detection task and road centerline extraction task simultaneously and adapt our proposed road-topology loss can achieve the best performance.

4. Discussion

In this section, we will discuss the robustness of different methods when the binarization threshold is changed. When we set different binarization thresholds, the road detection results will be different. However, in practice, we do not know the optimal binarization threshold. Figure 8 reflects the statistical characteristics of IoU and F1 across different binarization thresholds. The IoU and F1-score results of baseline methods(LinkNet34, DLinkNet34, DeeplabV3plus) are unstable with the vary of the binarization thresholds, as evidenced by the large variance in IoU and F1. In contrast, the F1 and IoU of our methods show better stability. Figure 8 also reports that, for F1 and IoU metrics, the means of the baseline methods are less than our proposed methods’ means. This is due to the use of a multitask learning scheme that can make two tasks mutually promote each other, and a road-topology loss that is minimized to eliminate topology errors. In conclusion, our methods can achieve a high performance under a wider range of binarization thresholds than baseline methods.

5. Conclusions

In this paper, we have learned the road detection task and the road centerline extraction task jointly with a multitask learning scheme to solve the problem of the road extraction from SAR imagery. To eliminate topology errors in road extraction results, we have specially designed a road-topology loss function for road extraction, which is differentiable. Different from the centerline extraction method based on regression method or classification method, we have adapted ordinal regression to learn discrete distance labels and trained the network by minimizing ordinal loss. By using the road centerline extraction method based on ordinal regression, the network is not sensitive to incorrect labels and can converge to a satisfactory result. Using multitask learning architecture, we have made full use of the correlation between the road detection task and the road centerline extraction task by learning the two tasks at the same time. The performance of one task has been improved under bootstrapping by the other task. The test result has shown that the networks, modified as multitask architecture, perform better than baseline methods. Considering the unique topological characteristics of the road, we have proposed a new road-topology loss function to penalize spurious parts in centerline extraction results and gaps in road detection results. The results show that the proposed road-topology loss function improved the connectivity and completeness of road networks. Finally, we discussed the robustness of our method, and the result has shown that our method not only greatly improves the performance of road extraction, but is also more stable than the baseline methods.

6. Future Work

Although our proposed method improves the performance of road extarction task, there are still some false detections. There are two main reasons for the false detection. One reason is that the noisy speckle nature of sar images, the layover and the shadowing effects of imaging destroy the roads in the SAR image. This leads many road parts to be misssed. The other reason is that the roads in sar images can often be confused with other targets, such as railway tracks, rivers, or even tree hedges. This leads some non-road regions to be detected as the roads. In the future, we will further improve our network to eliminate the influence of these two factors.