Dual-Dense Convolution Network for Change Detection of High-Resolution Panchromatic Imagery

Abstract

This paper presents a robust change detection algorithm for high-resolution panchromatic imagery using a proposed dual-dense convolutional network (DCN). In this work, a joint structure of two deep convolutional networks with dense connectivity in convolution layers is designed in order to accomplish change detection for satellite images acquired at different times. The proposed network model detects pixel-wise temporal change based on local characteristics by incorporating information from neighboring pixels. Dense connection in convolution layers is designed to reuse preceding feature maps by connecting them to all subsequent layers. Dual networks are incorporated by measuring the dissimilarity of two temporal images. In the proposed algorithm for change detection, a contrastive loss function is used in a learning stage by running over multiple pairs of samples. According to our evaluation, we found that the proposed framework achieves better detection performance than conventional algorithms, in area under the curve (AUC) of 0.97, percentage correct classification (PCC) of 99%, and Kappa of 69, on average.

1. Introduction

Change detection is a challenging task in remote sensing for identifying changed areas between two images acquired at different times from the same geographical area. It has been widely used in both civil and military fields such as agricultural monitoring, urban planning, environment monitoring, and reconnaissance. In general, change detection is performed in three steps. First, a preprocessing step is commonly used to conduct registration of two images and to correct geometric and radiometric distortions. In the second step, a feature map is extracted, for example, a difference image is computed in order to generate change features with the assumption that two images are not perfectly registered for all of the pixels. Lastly, a classification or clustering algorithm is driven in order to distinguish changed pixels and unchanged pixels based on statistical characteristics.

For change detection, many manually designed features such as a difference image (DI) [1,2,3,4,5,6,7], local change vector [8], and texture vector [9,10,11] have been proposed. In further classification analysis, an unsupervised change detection was proposed based on fuzzy c-mean (FCM) clustering [12,13]. The optimization algorithm based on Markov random field (MRF) and genetic algorithm was employed so as to optimize the FCM. On the other hand, a supervised learning algorithm was presented based on an active learning and MRF in order to detect change areas [14]. In addition, a support vector machine (SVM) has widely been used to perform binary classification based on texture information and change vector analysis [9,15,16,17]. Since the classification process mainly depends on extracted features, the selection of handcrafted features for effective image representation is known to be crucial. In general, handcrafted features in change detection are sensitive due to geometric and radiometric distortions, as well as imperfect registration. All of those mentioned classification algorithms would be reasonably good for training data sets. However, those algorithms are not able to incorporate accurate and reliable statistical characteristics for a huge amount of data sets, and thus would not yield good detection performance for new data sets.

Recently, a deep convolution neural network (DCNN) was developed to produce a hierarchy feature-maps via learned filters, and it can automatically learn a complex feature space from a huge amount of image data. The DCNN can achieve superior performance compared to conventional classification algorithms with handcrafted features. Recently, several change detection methods using deep learning algorithms have been proposed [18,19,20]. A difference image is fed into the deep neural networks as input data [18]. In addition, the neighboring features on each pixel on the difference image are taken as inputs. The restricted Boltzmann machine (RBM) is used for pre-training and is then unrolled in order to create a deep neural network. On the other hand, the change detection is performed by combining a sparse autoencoder, convolutional neural network (CNN), and unsupervised clustering algorithm [19]. In addition, a log-ratio map was used and transformed by a sparse autoencoder into a suitable feature space. A change detection map is directly extracted from the two images using a pre-trained CNN [20]. A unique higher dimensional feature map is produced by the CNN through different convolutional layers. The change map is computed using pixel-wise Euclidean distance of hyper dimensional features. Another change detection algorithm has also been proposed that adopts a log-ratio difference [21]. It is used as a feature input for detecting changes between multi-temporal synthetic aperture radar (SAR) images. In addition, a deep neural network was developed by stacking RBMs to learn and recognize changed pixels and unchanged pixels. In addition, a combined algorithm with the deep belief networks (DBNs) and change analysis are presented to highlight changes [22]. The presented algorithm merges and vectorizes local pixel features into DBN inputs. Then, the DBN model is established in order to capture key information for discrimination and to suppress irrelevant variations. An unsupervised clustering algorithm is then used to classify changed and unchanged pixels. Another approach utilizes joint features for change detection [23]. This work proposed an efficient change rule with a reliable expression of difference information. It learns the reliable change rule by recording the change information for a long-term sequence of remote sensing data with long short-term memory (LSTM) model. As mentioned above, all of the deep learning-based change detection algorithms yield relatively good performance. However, most of them still rely on the difference image as a feature input of their networks, resulting in them being sensitive to noisy conditions caused by geometric, radiometric distortions, and different viewing angles. In order to solve these problems, an alternative approach for change detection was developed by measuring similarity. A Siamese convolutional network was proposed to detect changed areas for optical aerial images [24]. The Siamese convolutional network with shared weights learns to extract features directly from image pairs. This work uses shared weights that are dependent from those of two branch networks. The shared weights can reduce parameters to be optimized, resulting in faster convergence. However, this model is also less flexible, which leads to overfitting due to shared weights with some other neurons.

In order to overcome the problems described above, this paper proposes a dual-dense convolutional network for recognizing pixel-wise change based on dissimilarity analysis of neighborhood pixels on high resolution panchromatic (PAN) images. In this proposed algorithm, two fully convolutional neural networks are employed to measure dissimilarity of neighborhood pixels. Furthermore, dense connection in convolution layers is applied to reuse preceding feature maps by connecting them to all subsequent layers. It is proposed to enhance a feature-map representation. While the conventional change detection algorithm [24] and conventional Siamese network use shared weights, the proposed algorithm removes shared weights in order to obtain independent optimal weights for two points of input data. So, each network can independently learn for optimal weights, called the “dual-dense convolutional network (dual-DCN)”. During its training, the dual-DCN is driven to learn more robust different representations to better distinguish different types of changes. The proposed algorithm gives better performance compared to conventional methods in qualitative and quantitative evaluation. It yields AUC of 0.97, PCC of 99%, and Kappa of 69 on average.

The rest of this paper is organized into five sections: In Section 2, the conventional convolutional neural network and problem statements will be described. Section 3 presents the proposed algorithm in detail. Section 4 will present and analyze experiment results. Finally, we conclude it in the last section.

2. Convolutional Neural Network and Problem Statement

The convolutional neural networks (CNNs) are a category of neural networks which are very effective in image recognition, classification, and so on [25]. The CNN is one of the deep learning approaches that is composed of multiple convolutional and nonlinearity layers with optional pooling, followed by fully-connected layers, as shown in Figure 1.

Let $I$ be an image ($m\times n\times c$) to be input, where $m$, $n$, and $c$ are the height, width, and channel numbers of the image, respectively. In the convolutional layers, $I$ is convolved by 2D $k$ kernels and mapped by a nonlinearity function, called rectified linear units (ReLU), to build $k$ feature-maps ($F$). The feature-map output of the lth layer is connected to the input of the (l + 1)th convolution and pooling layer. The final feature-maps are connected to a fully-connected layer. The last layer of fully-connected layer produces the class probability output $\left(P_{class}\right)$. A cross-entropy function is then used to compute an objective loss. All of the weighting parameters of the network can be trained using the backpropagation algorithm.

Changes on remote-sensed images can be detected by analyzing two registered images over the same geographical area. For change detection, CNN could be employed to learn changed image characteristics and detect changed areas on remote-sensed imagery. However, the difference image (DI) or the feature fusion (FF) is widely used as an input feature of CNN, as shown in Figure 2. The DI is extracted by image subtraction or log ratio. Then, the FF is constructed by concatenating the two images. Note that these approaches are sensitive to noise as direct pixel-wise comparison features; thus, the traditional CNNs with DI or FF features could be weak to distorted data. In practice, distorted images and data are common in the remote sensing field. This distortion can be caused by not only radiometric but also geometric and viewing angle factors. In general, a geometric distortion is generated when satellites or aircrafts acquire images. In addition, image registration is required to align two images, even over the same geographical area, in a pre-processing stage. However, it is almost impossible to perfectly achieve distortion correction through automated methods. In addition, a viewing angle difference in acquisition is another challenging issue in registration and change detection. This problem cannot be resolved without precise 3-D building models, complicated algorithms, and manual intervention. For robust change detection, a robust and stable classification model is required that resolves all of the problems described above.

3. Change Detection with the Proposed Dual-Dense Convolutional Neural Network

In general, generic change detection algorithms consist of two phases: pre-processing and change detection. Figure 3a depicts a general procedure of the conventional change detection system. The pre-processing stage performs radiometric correction, geometric rectification, and image registration. The registered images are then fed into a change detection algorithm in order to identify changed areas with feature vectors, for example, a difference image. In the general change detection systems, the radiometric correction and image registration stages are important and indispensable for better performance. The radiometric correction is performed in order to alleviate distortion for radiometric consistency. Then, the geometric correction is performed by aligning the global earth coordinates with the corresponding image points. Even though two images are compensated using multiple steps, they are still not perfectly registered, as they are independently processed with many error factors. Thus, an additional registration between two images is frequently required in order to reduce mis-alignment.

In urban and mountainous areas in particular, most automatic image registration methods remain ineffective. It would degrade performance of change detection by direct pixel-wise comparison using the difference image. In order to resolve imperfect registration impacting on the change detection, a dual-DCN, as shown in Figure 3b is proposed by employing a dissimilarity distance in order to overcome the mis-alignment problem for better performance of change detection even without a perfect image registration. The proposed algorithm employs two deep convolution networks to keep all the information of the original data. The generic characteristics of the CNN handle some local distortion and alignment, thus, the proposed algorithm absorbs the misalignment problem. In addition, the dense connectivity in the convolution layer is introduced by reusing all preceding feature-maps to enhance the feature-map representation.

3.1. Pre-Processing for Change Detection

As mentioned previously, an atmospheric correction is required to remove scattering and absorption effects from the atmosphere to characterize the surface reflectance effects for a time-series image analysis. This work uses KOMPSAT-3 images with product level 1G. In these images, the radiometric correction has been done by converting the image pixel values (Digital Numbers/DNs) to surface reflectance values. It involves the conversion of DNs to a radiance value, and then to top-of-atmosphere (TOA) radiance. On the other hand, gain and offset values are provided by KOMPSAT-3 to derive the TOA reflectance values [26]. After the atmosphere errors are corrected, the geometric correction is performed in order to ensure that the pixels in the image are in their proper geometric positions on the Earth’s surface. For our test images, geo-rectification and orthorectification are each conducted. For the geo-rectification, ground control points (GCPs) are identified in an unrectified image and correspond to their real coordinates to estimate the parameters (polynomial coefficients) of polynomial functions by the least square fitting. In addition, orthorectification can partly correct the image for image distortions caused by variations in the terrain topography in tandem with non-optimal satellite sensor viewing angle. Optical distortions are corrected, and terrain effects are corrected using coarse digital elevation model (DEM), namely shuttle radar topography mission DEM (SRTM DEM) for KOMPSAT-3 imagery [26].

In general change detection systems, an image registration is applied in order to ensure that two images become spatially aligned. Even though the correction of geometric distortion is performed, the spatial alignment of two images could contain a relatively large error of up to $\pm 6$ pixels. In order to overcome this distortion, automatic image registration is widely used. However, it requires high computational load, and is furthermore not easy to obtain perfect registration. They impact the performance of change detection algorithms, resulting in the possibility that a great deal of false change areas could occur. The proposed dual-DCN is proposed so as to handle distortion problems and simplify image registration. The dissimilarity distance of local characteristics is measured in order to identify a change with the dense dual-DCN model.

3.2. Dual-Dense Convolutional Neural Network for Change Detection

In order to achieve accurate change detection without a perfect registration, this paper proposes a dual-dense convolutional network (dual-DCN) with two deep convolutional networks, as shown in Figure 4. This proposed network identifies the change areas by measuring the dissimilarity distance of two inputs at the last stage for use of all the information of the two input images. Two branch networks, $N^1$ and $N^2$, handle two input images acquired at different time instances, respectively. The proposed network is based on CNN, thus, it can robustly conduct a pixel-wise change detection by inspecting the neighboring pixels.

A pair of images is cropped into two patches ($40\times 40$) by sliding in the raster scan order and two cropped patches ($I^1$ and $I^2$) are fed into the proposed network. The center pixel of the cropped patch is identified as changed or unchanged with the presence of a single dissimilar value between the cropped two patches. The Siamese network proposed by Reference [24] extracts features from an image pair. The pair of the convolutional networks is used to capture similarity characteristics by sharing the weights of the two network paths.

However, the shared weights of Siamese network reduce the parameters optimized during training for fast convergence. However, it is known to be frequently overfitted. Thus, the proposed network does not employ the shared weights to provide more flexible optimization than a restricted Siamese network. The parameters of each network branch of the proposed algorithm can be independently optimized in order to avoid early overfitted convergence. In addition, the proposed network employs dense connection [27] in the convolutional layers by reusing all the preceding feature-maps, in order to enhance representation capability of the feature-maps, as shown in Figure 4. The preceding feature maps are directly connected to all of the subsequent layers. The traditional CNN connects the output of the (l − 1)th layer as input to the lth [28]. In the proposed dual-DCN model, the lth layer receives all of the preceding feature-maps. The feature map of the lth layer at the rth dense block and the ith network can be computed by

$$F_{l,d_r}^i=H_{l-1,d_r}^i\left(\left[F_{0,d_r}^i{F}_{1,d_r}^i,\dots ,F_{l-1,d_r}^i\right]\right),r=0,1;i=1,2$$

(1)

where $\left[F_{0,d_r}^i,F_{1,d_r}^i,\dots ,F_{l-1,d_r}^i\right]$ indicates concatenation of the feature-maps of all of the previous layers, layer 0, …, layer ($l-1$). Each dense block is a group of convolution layers with the dense connectivity to avoid variant sizes of the feature maps. $H(·)$ plays a role in batch normalization (BN) [29], $3\times 3$ convolution, and ReLU. The BN is used to normalize parameters change of the preceding layers. The ReLU is used by thresholding at zero following $3\times 3$ convolution. The convergence of the stochastic gradient descent algorithm can be accelerated. $G$ including $3\times 3$ convolution followed by ReLU is employed before a dense block in order to generate the feature-map $F_0$. In the proposed architecture, each dense block contains three $H(·),$ including 64 feature maps of each layer. After a dense block is performed, a down-sampling operation is applied to produce variety scales with $2\times 2$ maximum pooling. Furthermore, the feature maps at the last convolutional layer are vectorized and fed into the fully-connected layer consisting of 64 neurons and 0.5 drop-out. The probability output, $O^i,$ at the last stage is computed by the sigmoid function. Furthermore, Euclidean distance ($D$) is employed in order to measure the dissimilarity between $I^1$ and $I^2$ computed by

$$D=O^1-O^2{}_2$$

(2)

When the value of $D$ approaches 1, the center pixel of the 40 × 40 patch is set to a changed one, otherwise, it is set to unchanged.

3.3. Training of the Proposed Dual-DCN for Change Detection

Given a training set consisting of image pairs, the proposed network can be end-to-end trained by the backpropagation algorithm. For each image pair, let $Y$ be a binary label of the ground truth in which $Y$ = 0 if both inputs are similar, and $Y$ = 1 if both inputs are dissimilar. The proposed dual-DCN is trained based on dissimilarity by computing the contrastive loss $L\left(D,Y\right)$ as an objective function [30]. This loss function employs a partial loss function for similar and dissimilar of a pair image. It produces a low value of D for unchanged pixels pair and high value for a pair of change pixels.

This proposed network is optimized using the stochastic gradient descent (SGD) optimizer. Each mini-batch arises from a single image pair that contains many changes and many absences of changes. The proposed algorithm randomly initializes all new layers by drawing weights from Glorot uniform [31]. The learning rate, decay rate, and momentum are set to 0.01, 1 × 10⁻⁶ and 0.9, respectively. The epoch number is set to 30.

4. Experimental Evaluation and Discussion

This paper uses a KOMPSAT-3 image dataset that was captured over South Korea. The KOMPSAT-3 image data set is provided by the Korea Aerospace Research Institute. Note that panchromatic band images which provide 0.7 m GSD are used for change detection. Figure 5 shows the example of an overlapped panchromatic images ($1214\times 886$) of the training dataset. These images were cropped by $40\times 40$ sliding patch. The labels for the dataset were manually constructed for all of the center pixels of cropped patch pair.

Figure 6 shows the two panchromatic images of ($29,368\times 27,388$) and ($29,188\times 28,140$) used in our experiments, which were acquired by KOMPSAT-3 on March 2014 and October 2015, respectively. These two images were acquired not only at different time instances, but also with different viewing angles. They have geometric misalignment of approximately $\pm 6$ pixels for overlapped area.

Figure 7 shows four selected urban areas from Figure 6, and they contain changed areas due to building construction. In Area 1, there are two types of construction changes, under construction changes and completed construction changes. Moreover, in certain areas, there are tall buildings, which could lead to false changes in change detection due to differing viewing angles. Rather than construction changes and tall buildings, we can find a forest area in Area 2. There are many tall buildings in Area 3, and accurate detection is not easy due to a large different viewing angle.

Area 4 is used to assess the influence of change detection due to differing seasons for a forest area. This case is challenging because the change due to the season should be disregarded for practical applications. Note that the labels for four areas were manually obtained, as shown in Figure 7.

In order to evaluate the change detection performance of the proposed algorithm and conventional algorithms, several metrics are used in this study, including receiver operating characteristic (ROC) curve, area under the curve (AUC), percentage correct classification (PCC), and Kappa coefficient [32]. For existing algorithms, DI + CNN, FF + CNN, and Siamese network were implemented. This CNN architecture includes 8 depth convolutional, 2 pooling, and 2 fully connected layers. For fair comparison, the same parameters of training parameters, the number feature maps, and training dataset were used in our evaluation.

Figure 8 shows detection results for four areas with the exiting algorithms and the proposed algorithm. As shown in Figure 8, the proposed algorithm and FF + CNN generate better detection accuracy for Area 1. On the other hand, DI + CNN and Siamese network produce many false positives for the area. For urban surfaces, it is relatively difficult to handle the misalignment and the different viewing angle impacts because there exists tall buildings and complex constructions, resulting in the fact that false detections are likely to be performed. For Area 2, FF + CNN and DI + CNN yield more false positives, particularly in forest and urban areas. Moreover, Siamese net achieves a better detection result than other conventional algorithms. However, many false positives are still detected in certain areas. Overall, the proposed dual-DCN gives proper change detection performance, even in different viewing angle conditions. For Area 3, the proposed algorithm is still able to properly detect the changes.

The other algorithms result in more false positives. Note that input images for Area 4 were acquired in difference seasons for a forest area. For the test data, Siamese net produces some false positives. As shown in Figure 8, the proposed algorithm yields a better detection result with the proposed dual-DCN. The proposed algorithm can alleviate the impacts of distortions caused by imperfect geometric correction and different viewing angles. As mentioned previously, the proposed dual-DCN was designed to learn the dissimilarity of two local images in order to avoid false changes. That is why the false positive rate is relatively lower by the proposed algorithm. In contrast, DI + CNN and FF + CNN yield higher false positive rates, particularly for Areas 2 and 3. Moreover, the Siamese network produces higher false positives in Area 1, due to less optimized parameters. Figure 9 shows that the proposed algorithm can give better ROC than the conventional algorithms. According to ROC curves, the proposed dual-DCN shows better quantitative detection performance in AUC of 0.97, on average, as tabulated in

Table 1. Quantitative assessments of the existing and proposed algorithms.

Metrics	Algorithms	Area 1	Area 2	Area 3	Avg
AUC	FF + CNN	0.95	0.95	0.98	0.96
	DI + CNN	0.70	0.68	0.88	0.75
	Siamese net	0.96	0.92	0.91	0.93
	The proposed	0.99	0.93	0.99	0.97
PCC (%)	FF + CNN	97	92	98	96
	DI + CNN	94	97	97	96
	Siamese net	96	98	99	98
	The proposed	98	99	99	99
Kappa	FF + CNN	78	19	47	48
	DI + CNN	30	32	28	30
	Siamese net	52	35	68	52
	The proposed	78	60	69	69

. FF + CNN is slightly better in AUC than the proposed dual-DCN for Area 2, because it has better true positive for this case. However, the proposed algorithm has a lower false positive rate than FF + CNN. Table 1 summarizes the PCC and Kappa values of different methods for the three areas. As shown in Table 1, the proposed algorithm achieves higher PCC and Kappa values. We can say that the proposed dual dense convolutional network architecture has the ability to identify both changed and unchanged areas by disregarding irrelevant variations and false changes, even in cases of complicated urban surfaces, geometric distortion, and different viewing angles.

Regarding time complexity, the proposed DCN requires more computational complexity than the single architecture using FF + CNN and DI + CNN by a factor of approximately two with sequential machines. However, the proposed dual-DCN can work in parallel, thus, throughput can be enhanced with a parallel machine such as GPU. In addition, the proposed also takes about 20% more running time than the Siamese network because it includes additional preceding of feature maps.

5. Conclusions

In this paper, we presented a robust change detection algorithm for high-resolution panchromatic imagery. The proposed algorithm learns and analyzes the dissimilarity of two input images with the densely convolutional network by incorporating local information. We found that the proposed algorithm achieves higher detection accuracy, even with noisy conditions such as geometric distortion and different viewing angles in qualitative and quantitative analysis. Further work can be conducted to extend the framework for other modalities such as multi-spectrum images and SAR data.