A Lightweight Siamese Neural Network for Building Change Detection Using Remote Sensing Images

Abstract

Building change detection (BuCD) can offer fundamental data for applications such as urban planning and identifying illegally-built new buildings. With the development of deep neural network-based approaches, BuCD using high-spatial-resolution remote sensing images (RSIs) has significantly advanced. These deep neural network-based methods, nevertheless, typically demand a considerable number of computational resources. Additionally, the accuracy of these algorithms can be improved. Hence, LightCDNet, a lightweight Siamese neural network for BuCD, is introduced in this paper. Specifically, LightCDNet comprises three components: a Siamese encoder, a multi-temporal feature fusion module (MultiTFFM), and a decoder. In the Siamese encoder, MobileNetV2 is chosen as the feature extractor to decrease computational costs. Afterward, the multi-temporal features from dual branches are independently concatenated based on the layer level. Subsequently, multiscale features computed from higher levels are up-sampled and fused with the lower-level ones. In the decoder, deconvolutional layers are adopted to gradually recover the changed buildings. The proposed network LightCDNet was assessed using two public datasets: namely, the LEVIR BuCD dataset (LEVIRCD) and the WHU BuCD dataset (WHUCD). The F1 scores on the LEVIRCD and WHUCD datasets of LightCDNet were 89.6% and 91.5%, respectively. The results of the comparative experiments demonstrate that LightCDNet outperforms several state-of-the-art methods in accuracy and efficiency.

1. Introduction

Buildings are a particularly significant type of human-made feature on Earth and can be inventoried using remote sensing techniques. Building specifications change over time with the influence of human activities. Such alterations can also be obtained from remotely sensed data, especially high-spatial-resolution RSIs. Thus, physical and social processes such as urban sprawl, heat island, and disaster impact [1,2,3,4] can be better understood. It is not surprising, then, that BuCD using RSIs has become a hot topic in remote sensing [2,5,6,7,8,9,10].

According to the methods of feature extraction, methods of BuCD using RSIs can be divided into two categories: traditional BuCD and deep-learning-based BuCD. Traditional BuCD approaches utilize man-designed features, such as textures and spectral features [11,12,13]. Conversely, deep-learning-based BuCD algorithms generally learn features from large amounts of high-spatial-resolution RSIs [14,15,16,17,18,19].

Traditional BuCD methods can be split into two categories according to the processing units, namely pixel-based and object-oriented BuCD. Pixel-based BuCD methods usually begin with a difference computation of multi-date RSIs, followed by change detectors. Simple differences, such as the absolute value of the subtraction of the spectra between multi-temporal images, likely lead to inconsistent results. To characterize the spectral difference of multi-date RSIs, various algorithms were devised, such as the computation of the ratio, the Euclidian distance of multi-date RSIs, and the synthesis of several results after difference computation [20]. Nevertheless, the spatial information from high-spatial-resolution RSIs is difficult to capture using only the spectral difference features because of the increase in the spatial resolution of RSIs. Hence, BuCD algorithms employ well-designed features, spectral–spatial integration, and morphological operators, for instance, to collect more geometrical and context information [12,21]. Afterward, the images containing varied features are further processed by feature-level or decision-level fusion to generate the final change map. Despite these efforts, the lack of contextual information affects the performance of traditional pixel-based BuCD methods using high-spatial-resolution RSIs.

Contrary to the pixel-based BuCD, the processing units of object-oriented BuCD methods are images objects. They are usually generated by the segmentation algorithms from high-spatial-resolution RSIs. Accordingly, the generation of image objects is fundamental in object-oriented BuCD algorithms. A common solution for object-oriented BuCD is to detect changed information after image segmentation. The input data used in segmentation algorithms typically comprise (1) bitemporal composite images [22,23,24], (2) auxiliary data, e.g., digital surface model [25], and (3) separate multi-date RSIs [26]. Approaches using the first two kinds of input data provide one segmentation result for the subsequent change-detection phase. Consequently, these approaches have difficulty identifying geometric changes using the multi-date RSIs. Approaches that utilize the separate multi-date RSIs usually generate multi-temporal image objects according to the dates of the RSIs. Zhang et al. [26] reported that the change detection method using multi-temporal image objects from separate segmentations performed better than that using one segmentation result. Among BuCD algorithms employing multi-temporal images objects, co-segmentation with decision-level fusion performs better [26,27]. Overall, object-based BuCD can incorporate more spatial features into the algorithm compared with pixel-based BuCD. Nevertheless, the dependence on manually designed characteristics remains in object-oriented BuCD.

With the burgeoning of deep-learning technology, deep neural networks (DNNs) have contributed much to various remote sensing applications, such as disaster detection/assessment, land cover/land use classification, and BuCD [7,28,29,30,31,32,33,34,35]. Specifically, the DNNs frequently used to extract features from RSIs are convolutional neural networks (CNNs). CNNs can naturally compute multilevel features on account of the network structure. Among these multilevel features, the lower levels primarily describe the characteristics of edges and locations, while the higher-level ones amplify the semantics [36]. To efficiently carry out pixel-wise training and prediction, CNNs were improved. In detail, the convolutional layers were used to replace the fully connected ones in CNNs, called fully convolutional neural networks (FCNs) [37]. Most of the existing BuCD research takes advantage of the structure of FCNs, thus accomplishing pixel-wise prediction directly. From the perspective of network structures, early studies of BuCD took inspiration from tasks such as semantic segmentation. Unlike semantic segmentation, which usually takes a single-temporal image as input, the BuCD input data are multi-temporal RSIs. As a result, for BuCD, it is necessary to modify the existing networks to match the multi-temporal RSIs input. For instance, Daudt et al. [38] presented the fully convolutional early fusion network (FCEFN). They adjusted the structure of U-Net [39] to FCEFN by using the input from the multi-date RSIs [38]. Additionally, Peng et al. [29] combined multi-date RSIs as one input in their improved U-Net++ architecture. They adopted the multiple side-output fusion strategy to combine multilevel features [29]. In summary, the input for the above networks is the concatenation of multi-date RSIs. Such architectures often lack the direct expression of differences from multi-date RSIs.

In contrast, networks with the Siamese architecture can process two input images with two separate branches [40,41]. This architecture naturally fits the multi-temporal input for BuCD. For instance, Daudt et al. [38] proposed the fully convolutional Siamese concatenation network (FCSiCN) and the fully convolutional Siamese difference network (FCSiDN). Both FCSiCN and FCSiDN are modified from U-Net. In the encoder, they both contain two streams with shared weights but differ in how to design the skip connection. To be specific, FCSiCN directly connects the encoding features from two branches, whereas FCSiDN joins the absolute value of their difference [38]. FCSiDN generally outperforms FCSiCN, according to [38]. In this type of network architecture, effectively extracting and combining features from multi-date RSIs is crucial. To achieve this aim, attention modules, feature pyramid modules, and transformers have all been included in recent BuCD studies [7,17,42]. For example, Zhang et al. [14] developed the image fusion network (IFN). In IFN, deep features were separately extracted by two branches from multi-temporal images. They were further fused with difference features by attention modules. Liu et al. [17] incorporated a dual attention module into the Siamese network for BuCD, thereby improving feature discrimination. Shen et al. [43] presented a semantic feature-constrained change detection network (SFCCDN). SFCCDN combined high-level and low-level features (LoLeFs) using a global channel attention module. Additionally, multiscale and multi-temporal features were fused using the attention mechanisms [43]. Along with attention modules, BuCD networks frequently embed the feature pyramid module. For example, Dong et al. [5] devised a multiscale context aggregation network (MCAN). To combine multiscale features, they integrated a feature pyramid module and a channel-spatial attention module in MCAN. Liu et al. [7] designed local and global feature extraction modules (LGFEM) in the BuCD network on the basis of U-Net. They combined the pyramid and attention modules in LGFEM to learn local and global representation [7]. In addition, transformers have been incorporated into BuCD networks to manage long-range feature dependencies. For instance, Feng et al. [42] introduced the intra-scale cross-interaction and inter-scale feature fusion network (ICIFNet). ICIFNet extracted local and global features simultaneously using CNNs and a transformer. Li et al. [44] presented a hybrid transformer model, TransUNetCD, which embedded a CNN-Transformer encoder to compute global context features.

Although the above approaches have made significant progress, issues remain in BuCD concerning the high-spatial-resolution RSIs. First, because of the remote sensor systems considerations, such as varied viewing angles, and environmental variables, such as atmospheric conditions, pseudo predictions of building changes are demonstrated in the existing methods. Second, because of the detailed and complex backgrounds shown in the high-spatial-resolution RSIs, the boundaries of changed buildings may be mispredicted. Third, most prior DNNs for BuCD involve very deep convolutional layers and various feature extraction modules, thus demanding heavy computing resources.

Considering the above issues, in this paper, we propose the LightCDNet. The multilevel and multiscale features can be efficiently exploited and fused in LightCDNet. The Siamese encoder based on MobileNetV2 [45], a memory-efficient network, was created to extract multilevel features. Then, a MultiTFFM combined the two-branch multilevel features to gather the localization and semantics information. Finally, the decoder received the fused features to gradually recover the varied buildings. The following are the main contributions of this study:

• A lightweight Siamese network LightCDNet using RSIs is proposed for BuCD. The LightCDNet consists of a memory-efficient encoder to compute multilevel deep features. Correspondingly, it utilizes a decoder with deconvolutional layers to recover the changed buildings.
• The MultiTFFM is designed for exploiting multilevel and multiscale building change features. First, it fuses low-level and high-level features (HiLeFs) separately. Subsequently, the fused HiLeFs are processed by the atrous spatial pyramid pooling (ASPP) module to extract multiscale features. Lastly, the fused low-level and multiscale HiLeFs are linked to produce the final feature maps containing the localizations and semantics of varied buildings.

2. Materials and Methods

2.1. Datasets for BuCD

The proposed network, LightCDNet, was tested using two public BuCD datasets, LEVIRCD [46] and WHUCD [47].

2.1.1. LEVIRCD

The public BuCD dataset, LEVIRCD, comprises 0.5 m images from Google Earth. It covers several regions in Texas and the United States of America, such as Austin, Lakeway, and others. Additionally, the satellite images in the LEVIRCD dataset were taken between 2002 and 2018. Hence, seasonal variations and illumination differences can be observed in the LEVIRCD dataset, which may result in the diverse appearance of buildings. Additionally, various types of building changes that have occurred over time can be observed in the LEVIRCD dataset (Figure 1). More specifically, both newly constructed and demolished buildings can be seen in the multi-date images (Figure 1a–e). The LEVIRCD dataset also includes multi-temporal RSIs that display unchanged regions with different colors (Figure 1f). Furthermore, there are 637 multi-date image pairs with 1024 × 1024 pixels in the original LEVIRCD [46]. The training, validation, and testing sets adhere to the settings in [46]. Moreover, the original 1024 × 1024 pixels multi-date RSIs were all cropped to 256 × 256 pixels (

Table 1. An overview of multi-date image pairs (ImgPs) with 256 × 256 pixels in LEVIRCD and WHUCD datasets.

Dataset	Number of Training ImgPs	Number of Validation ImgPs	Number of Testing ImgPs
LEVIRCD	7120	1024	2048
WHUCD	5376	768	1536

) because of the configuration of the graphics processing unit (GPU) employed.

2.1.2. WHUCD

The WHUCD dataset covers parts of the region in New Zealand that was subjected to a 6.3-magnitude earthquake in 2011. The dataset includes 0.2 m bitemporal aerial images acquired in 2012 and 2016. As a result, building variations, particularly renovations, may be discerned between images collected in 2012 and those captured in 2016 (Figure 2). Specifically, there are 12,796 buildings in the 2012 images and 16,077 buildings in the 2016 images [47]. The majority of buildings in the WHUCD dataset are low-rise structures (Figure 2). In addition, the original bitemporal images provided by [47] have 32,507 pixels in the column and 15,354 pixels in the row. Considering the GPU used, the 2012 and 2016 images were also cropped to 256 × 256 pixels as those in the LEVIRCD dataset. Additionally, a 7:1:2 ratio of training, validation, and testing sets was created using a random division of the cropped bitemporal image pairs (Table 1).

2.2. Methods

The proposed LightCDNet consists of a lightweight Siamese encoder, a MultiTFFM, and a decoder that predicts the changed buildings (Figure 3). Specifically, LightCDNet utilizes a pair of RSIs collected on different dates, T₁ and T₂, as the input. Using two parallel branches, multilevel features are computed from bitemporal RSIs in the encoder (Figure 3a). Afterward, the MultiTFFM (Figure 3b) fuses multilevel features from two branches, depending on their level. LoLeFs F_L1 and F_L2 and HiLeFs F_H1 and F_H2 are taken as separate groups, and each is fed into the MultiTFFM for further processing. Finally, the decoder (Figure 3c) receives the combined feature maps from the MultiTFFM to reconstruct the changed buildings.

2.2.1. Siamese Encoder

On the one hand, the structure of the Siamese network corresponds precisely to the input bitemporal RSIs of BuCD. As a result, we utilized two weight-shared branches in the encoder module. On the other hand, DNNs have demonstrated superiority over feature learning, most likely at the cost of quantities of computational resources [48,49,50]. To decrease the number of computing resources needed for BuCD and still maintain accuracy, we chose MobileNetV2 [45] as the backbone for the Siamese encoder. MobileNetV2 is designed for environments with limited resources, such as mobile phones. The fundamental component of MobileNetV2 is the inverted residual bottleneck (InvResB) [45]. Thanks to this basic structure, MobileNetV2 can outperform the most cutting-edge technology [45]. Additionally, the InvResB module can significantly lower the memory occupancy required during inference, making it well-suited for mobile applications [45]. Please see [45] for further details on MobileNetV2. It is worth noting that a pretrained MobileNetV2 (Appendix A) was utilized in the training process.

Suppose the input of the encoder is bitemporal RSIs I(T₁) and I(T₂), I(T₁), and I(T₂)$\in {\mathcal{R}}^{h\ast w\ast b}$, where h, w, and b represent the height, width, and number of bands of the input RSIs, respectively. In other words, bitemporal images I(T₁) and I(T₂) are fed into the dual-branch encoder separately (Figure 3a). Subsequently, I(T₁) and I(T₂) are, respectively, processed by a convolutional layer and 17 InvResB modules (Figure 3a). For BuCD feature extraction, dense feature maps are required, analogous to the work of semantic segmentation. For clarity, the output stride (OtS) is defined first. The OtS describes the ratio of the input images’ spatial resolution to the output images’ resolution. To obtain dense feature maps, OtS is commonly set to 16 or 8 for semantic segmentation [45]. Setting OtS to 16 is more efficient when using MobileNetV2 as feature extractors for semantic segmentation [45]. Therefore, the OtS is set to 16 in the Siamese encoder of LightCDNet. According to the above OtS setting, the first convolutional layer has a stride of 2. Additionally, the stride of the second, fourth, and seventh InvResB modules is set to 2, and the rest are set to 1 (Figure 3a).

To account for both localization and semantic information, we demonstrate how to select HiLeFs (F_H1 and F_H2) and LoLeFs (F_L1 and F_L2), respectively. According to the application of MobileNetV2 for semantic segmentation [45], the penultimate feature maps are chosen as the HiLeFs to consider both efficiency and accuracy. Hence, we utilized the penultimate layers of MobileNetV2 as the HiLeFs (F_H1 and F_H2) in the dual branches (Figure 3a), i.e., $F_{H1},F_{H2}\in {\mathcal{R}}^{\frac{h}{16}\ast \frac{w}{16}\ast 320}$. LoLeFs, which are four times larger than HiLeFs, can be used to capture precise localization information, according to [51,52]. As a result, we choose the output of the third InvResB module of each branch as F_L1 and F_L2 (Figure 3a), i.e., $F_{L1},F_{L2}\in {\mathcal{R}}^{\frac{h}{4}\ast \frac{w}{4}\ast 24}$.

2.2.2. Multi-Temporal Feature Fusion Module

In the Siamese encoder (Figure 3a), features related to localization (F_L1 and F_L2) and semantics (F_H1 and F_H2) are exploited from the original bitemporal RSIs I(T₁) and I(T₂). The MultiTFFM fuses two-stream features to improve the representation of building change information (Figure 3b). Initially, the multi-temporal LoLeFs, F_L1 and F_L2, and the HiLeFs, F_H1 and F_H2, are concatenated independently on the basis of corresponding levels (Equations (1) and (2)):

F_LC = Concat(F_L1, F_L2),

(1)

F_HC = Concat(F_H1, F_H2),

(2)

where $F_{LC}\in {\mathcal{R}}^{\frac{h}{4}\ast \frac{w}{4}\ast 48}$ is the concatenated LoLeFs, $F_{HC}\in {\mathcal{R}}^{\frac{h}{16}\ast \frac{w}{16}\ast 640}$ is the concatenated HiLeFs, and Concat is the concatenation operation. Subsequently, two 3 × 3 convolutions are applied individually on F_HC and F_LC to decrease the number of channels by half; thus, reducing the amount of redundant information. Accordingly, assuming that the intermediate outputs after the above two 3 × 3 convolutions are F_HC1 and F_LC1, the channels of F_HC1 and F_LC1 are 320 and 24, respectively. Afterward, F_HC1 is processed by the ASPP module [53,54] to compute multiscale contextual features, yielding F_HASPP$\in {\mathcal{R}}^{\frac{h}{16}\ast \frac{w}{16}\ast 256}$. Inspired by [52], F_HASPP is fourfold up-sampled and joins the features with abundant localization information. To this aim, a 1 × 1 convolution is applied on F_LC1 to balance the channels of F_HASPP and F_HC1, obtaining F_LC2$\in {\mathcal{R}}^{\frac{h}{4}\ast \frac{w}{4}\ast 48}$. Finally, F_LC2 and fourfold up-sampled F_HASPP are combined as F_MC$\in {\mathcal{R}}^{\frac{h}{4}\ast \frac{w}{4}\ast 304}$.

2.2.3. Decoder

The decoder’s purpose is to reconstruct the appearance of altered buildings from course to fine (Figure 3c). Initially, two 3 × 3 convolutions are performed on the fused features F_MC from the MultiTFFM, yielding F_D1$\in {\mathcal{R}}^{\frac{h}{4}\ast \frac{w}{4}\ast 256}$. Subsequently, repeated deconvolutional layers [55] are employed to recover the feature maps from the spatial dimension of F_D1, $\frac{h}{4}\times \frac{w}{4}$, to that identical to the bitemporal RSIs I(T₁) and I(T₂), $h\times w$. A 1 × 1 convolution is then applied to compute the final score map, $F\in {\mathcal{R}}^{h\ast w\ast 2}$. Assuming that 1 and 0 represent the varied buildings and the background, respectively, the BuCD result $I_{cd}$ can be computed as follows:

$$I_{cd}\left(i\right)={}_{c\in \left\{0,1\right\}}{}^{argmax}F\left(i,c\right),1\le i\le h\ast w,$$

(3)

where $I_{cd}\left(i\right)$ denotes the ith pixel of $I_{cd}$.

2.2.4. Loss Function

In LightCDNet, BuCD is regarded as a binary classification problem. Hence, the cross entropy (CE), L_CE, is utilized:

$$L_{CE}=\frac{1}{H\ast W}{{\displaystyle \sum }}_{h=1,w=1}^{H,W}{L}_{ce}\left(P\left(h,w\right),Y\left(h,w\right)\right),$$

(4)

$$L_{ce}\left(P\left(h,w\right),Y\left(h,w\right)\right)=-{{\displaystyle \sum }}_{i=1}^c{Y}_i\left(h,w\right)log(P_i\left(h,w\right)),$$

(5)

where H and W are the image height and width, respectively, c is the class number, which is 2 here, and Y is the ground truth. Additionally, P represents the output of the Sigmoid function applied on the score map F.

On the other hand, the altered buildings often only make up a small part of the RSIs. This imbalance problem may increase the difficulty of identifying changed buildings using the BuCD network. To tackle this problem, we selected the Dice loss [56], L_Dice (Equation (6)):

$$L_{Dice}=1-\frac{2\times Y\times Y^{\prime }}{Y+Y^{\prime }} ,$$

(6)

where $Y$ is the ground truth, and $Y^{\prime }$ denotes the predicted BuCD result.

Therefore, the loss function used in LightCDNet, L_BUCD, is defined as follows,

L_BUCD = L_CE + L_Dice.

(7)

2.2.5. Implementation Details

The codes to carry out the BuCD experiments were written in Python 3.6 and run on the platform PyTorch v1.9. (Our code is available for potentially interested users. Please see Appendix A for more information.) Additionally, we chose Adam [57] for stochastic optimization because of its low memory requirement. The learning rate (LR) was initially set to ${10}^{-4}$. When no improvements in the loss L_BUCD were observed after 5 epochs, LR was adjusted by a factor of 0.9 during the training process. Furthermore, data augmentation strategies were utilized since they can preclude DNNs from overfitting. The data augmentation techniques include random flip (vertical or horizontal flip), random rotation (rotation by 90, 180, and 270 degrees), random resize (resize between 0.8 and 1.2), random crop, random Gaussian blur, and random color jitter (brightness = 0.3, contrast = 0.3, saturation = 0.3, and hue = 0.3).

Moreover, our LightCDNet was trained on the computer with an Intel Xeon X5570 processor and one NVIDIA GeForce GTX 1080 Ti GPU. As a result, the batch size was assigned to 8 in consideration of the GPU memory limitations. In addition, the network was trained for 100 epochs. Information about the computation time is shown in Section 4.3.

2.3. Accuracy Assessment

To evaluate the accuracy of LightCDNet, we utilized five indicators: Precision, Recall, F1 score (F1), overall accuracy (OA), and Intersection over Union (IoU). Assuming that TP, TN, FP, and FN stand for true predictions on changed buildings, true predictions on the background, false predictions on changed buildings, and false predictions on the background, respectively, the five metrics can be computed as follows:

$$Precision=\frac{TP}{TP+FP} ,$$

(8)

$$Recall=\frac{TP}{TP+FN} ,$$

(9)

$$F1=\frac{2\ast Precision\ast Recall}{Precision+Recall} ,$$

(10)

$$OA=\frac{TP+TN}{TP+FN+FP+TN} ,$$

(11)

$$IoU=\frac{Precision\ast Recall}{Precision+Recall-Precision\ast Recall} .$$

(12)

We can notice that Precision quantifies the percentage of accurate classification of altered buildings (Equation (8)). Recall measures the missed proportion of altered buildings (Equation (9)). Compared with Precision and Recall, F1 (Equation (10)), OA (Equation (11)), and IoU (Equation (12)) are more comprehensive.

2.4. Comparative Methods

To demonstrate the effectiveness of LightCDNet, four state-of-the-art BuCD approaches were chosen for comparison. These methods are briefly described below.

• Cross-layer convolutional neural network (CLNet) [18]: CLNet was adjusted from U-Net. The key component of CLNet is the cross-layer block (CLB), which can combine multilevel and multiscale features. To be specific, CLNet contains two CLBs in the encoder. It is worth noting that the input of CLB is the concatenated multi-date RSIs, i.e., images with six bands.
• Deeply supervised attention metric-based network (DSAMNet) [16]: DSAMNet utilized two weight-sharing branches for feature extraction. Hence, DSAMNet can separately receive multi-date images. Moreover, DSAMNet included the change decision module (CDM) and deeply supervised module (DSM). CDM adopted attention modules to extract discriminative characteristics and produced output change maps. Additionally, the DSM can enhance the capacity of feature learning for DSAMNet.
• Deeply supervised image fusion network (IFNet) [14]: IFNet adopted an encoder-decoder structure. In the encoder, IFNet utilized dual branches based on VGG16. The decoder consists of attention modules and deep supervision. Specifically, the attention modules were embedded to fuse raw convolutional and difference features.
• ICIFNet [42]: ICIFNet, unlike the previous three, contained two asymmetric branches for feature extraction. Specifically, ResNet-18 [49] and PVT v2-B1 [58] are employed separately by two branches in ICIFNet. Four groups of features brimming with local and global information are produced from the dual branches. The attention mechanism was then applied to fuse multiscale features. Finally, the output was generated from the combination of three score maps.

To provide a fair comparison, data augmentation and hyperparameter settings utilized in the four comparative networks are consistent with the original sources and their published codes (Appendix A).

3. Results

The proposed LightCDNet and the comparative approaches, CLNet [18], DSAMNet [16], IFNet [14], and ICIFNet [42], were evaluated quantitatively and qualitatively using the LEVIRCD and WHUCD datasets.

3.1. Results on LEVIRCD Dataset

Six groups of bitemporal images were picked from the testing set of the LEVIRCD dataset to graphically demonstrate the results (Figure 4). The multi-date images (Rows Image 1 and Image 2 in Figure 4) displayed the regions transformed from shrubs into buildings (Figure 4a,b) or from bare lands into buildings (Figure 4c–f). In general, ICIFNet [42] and our LightCDNet performed better than CLNet [18], DSAMNet [16], and IFNet [14] visually. We observed that CLNet [18] failed to detect the newly built structures (Figure 4e,f). In addition, the newly constructed circular roads were falsely flagged by CLNet [18] (Figure 4a,b). Furthermore, we found that the unchanged land was falsely flagged by DSAMNet [16] and IFNet [14] because of the color variation (Figure 4c–e). Notably, DSAMNet [16] mistakenly identified most of the shadows as altered buildings. Moreover, by comparing the predicted maps of our LightCDNet and ICIFNet [42], we observed that our method illustrated fewer false identifications and more complete buildings. It is also worth mentioning that five networks, including ours, could not identify the changed rooftop completely in the yellow boxes of Figure 4e. From Rows Image 1 and Image 2, Column (e) of Figure 4, we can see that trees covered the upper corner of the rooftop. Thus, the omission shown in the upper corner of the rooftops is understandable and consistent with the input images. However, the missed parts on the right indicate that the five approaches are more or less affected by differences in rooftop color.

Let us briefly recall that there are 10,192 multi-date image pairs with 256 × 256 pixels in the LEVIRCD dataset, including 2048 pairs in the test set (Table 1). The quantitative analysis of the LEVIRCD dataset (

Table 2. Quantitative analysis of the LEVIRCD dataset. The highest metrics are in bold.

Methods	Precision (%)	Recall (%)	F1 (%)	IoU (%)	OA (%)
CLNet	83.2	75.4	79.1	65.4	98.0
DSAMNet	80.0	90.6	85.0	73.9	98.4
IFNet	84.7	85.5	85.1	74.0	98.5
ICIFNet	89.6	84.3	86.8	76.8	98.7
Ours	91.3	88.0	89.6	81.2	99.0

) demonstrates that our LightCDNet outperforms CLNet [18], DSAMNet [16], IFNet [14], and ICIFNet [42] from the overall metrics, F1, IoU, and OA. ICIFNet [42] is second-ranked among the five methods, which is 2.8%, 4.4%, and 0.3% lower than our network in F1, IoU, and OA. This is consistent with the observation in Figure 4. Additionally, we found that CLNet [18] had the lowest F1, IoU, and OA among the five approaches. These findings imply that networks with two branches for receiving multi-date RSIs separately perform better than one branch processing the combination of multi-temporal RSIs. Moreover, DSAMNet [16] has the highest Recall (90.6%) while providing the lowest Precision (80.0%). From the perspective of overall evaluation indicators, our LightCDNet achieves 4.6%, 7.3%, and 0.6% improvements in F1, IoU, and OA, respectively, compared with the DSAMNet [16] metrics.

3.2. Results on WHUCD Dataset

Six pairs of multi-temporal RSIs from the WHUCD dataset were chosen to display the final output maps of five networks (Figure 5). As introduced in Section 2.1.2, we can see the reconstruction of various buildings from 2012 to 2016 (Rows Image1 and Image2 of Figure 5). For example, Figure 5a,d,f depict the ground’s transformation from bare soil to buildings. Figure 5b illustrates that the land was converted from a mixture of shrubs, grass, and human-made facilities to buildings. From the predicted maps in Figure 5, we found that our LightCDNet demonstrated more complete and regular rooftops than CLNet [18], DSAMNet [16], IFNet [14], and ICIFNet [42]. Additionally, conspicuous green regions were observed in the results of CLNet [18], IFNet [14], and ICIFNet [42] (Figure 5b,e,f), indicating that these models omit the new colorful rooftops. Moreover, falsely flagged altered buildings (red regions) are exhibited in the results of DSAMNet [16], mainly caused by shadows in multi-temporal RSIs. This observation is consistent with the visual results on the LEVIRCD dataset. Furthermore, from Column (e) of Figure 5, we can notice that omissions of altered buildings (green regions) are displayed on the top and left of images from all five approaches. This is probably because of the absence of contextual information in the original multi-temporal RSIs caused by image cropping performed in dataset preparation.

From the quantitative evaluation of the WHUCD dataset (

Table 3. Quantitative analysis of the WHUCD dataset. The highest metrics are in bold.

Methods	Precision (%)	Recall (%)	F1 (%)	IoU (%)	OA (%)
CLNet	88.8	61.6	72.7	57.1	98.0
DSAMNet	79.0	91.9	85.0	73.9	98.6
IFNet	94.9	77.0	85.0	74.0	98.8
ICIFNet	88.0	74.8	80.9	67.9	98.5
Ours	92.0	91.0	91.5	84.3	99.3

), we find that our method shows higher F1 (91.5%), IoU (84.3%), and OA (99.3%) compared with CLNet [18], DSAMNet [16], IFNet [14], and ICIFNet [42]. Additionally, CLNet [18] has the lowest F1 (72.7%), IoU (57.1%), and OA (98.0%) among the five approaches. These numerical results are consistent with the LEVIRCD dataset (Table 2). In addition, IFNet [14] ranks second among the five networks. It has the highest Precision (94.9%), but its Recall is relatively low (77.0%). Our LightCDNet obtained 6.5%, 10.3%, and 0.5% improvements in F1, IoU, and OA compared with IFNet [14]. The third place among the five approaches belongs to DSAMNet [16]. Although DSAMNet [16] gains the highest Recall (91.9%), it has the lowest Precision (79.0%). Additionally, its F1 is 6.5% lower than that of our LightCDNet. The above quantitative analysis is consistent with the visualization in Figure 5.

4. Discussion

4.1. Ablation Study for Siamese Encoder

To demonstrate the effectiveness of the lightweight feature extractor used in our LightCDNet, architectures using different backbones in the Siamese encoder were tested on the WHUCD dataset. In detail, we employed two more complex models, ResNet-50 [49] and Xception [59], as the backbones. The predicted maps (Figure 6) illustrate that our LightCDNet with MobileNetV2 as the encoder outperforms networks with ResNet-50 and Xception as the encoder, respectively. The quantitative results (

Table 4. Quantitative analysis of LightCDNet with different settings in the Siamese encoder on the WHUCD dataset. The highest numbers are in bold.

Siamese Encoder	Precision (%)	Recall (%)	F1 (%)	IoU (%)	OA (%)
ResNet-50	90.2	88.6	89.4	80.8	99.1
Xception	89.5	78.7	83.7	72.0	98.7
Ours (MobileNetV2)	92.0	91.0	91.5	84.3	99.3

) also confirm the above observation. In terms of F1, IoU, and OA, our network with MobileNetV2 is improved by 2.1%, 3.5%, and 0.2%, respectively, compared with the network using ResNet-50 as the backbone. Additionally, compared with the network with Xception as the backbone, LightCDNet exceeded it by 7.8%, 12.3%, and 0.6% based on F1, IoU, and OA. The above visual and numerical results imply that the designed encoder of our LightCDNet can effectively compute multilevel features for the following MultiTFFM and decoder.

4.2. Ablation Study for Decoder

The precise masks of targets of interest can be progressively recovered using a series of deconvolutional layers [55]. To verify the deconvolutional operations embedded in the decoder, the architecture employing bilinear up-sampling operations instead of deconvolution in the decoder was implemented on the WHUCD dataset. From the predicted maps (Figure 7), we can observe that our LightCDNet with the decoder using deconvolutional operations illustrates more precise varied building boundaries compared with the network with the decoder using the up-sampling strategy (red boxes in Figure 7). In addition, numerical evaluation (

Table 5. Quantitative evaluation of networks with different settings in the decoder used on the WHUCD dataset. The highest numbers are in bold.

Decoder	Precision (%)	Recall (%)	F1 (%)	IoU (%)	OA (%)
Ours (Deconvolution)	92.0	91.0	91.5	84.3	99.3
Upsampling	91.0	89.3	90.1	82.0	99.2

) further demonstrates that the decoder with deconvolutional operations performs better than that with up-sampling operations, achieving 1.4%, 2.3%, and 0.1% improvements based on F1, IoU, and OA. These findings indicate the applicability of the decoder in our LightCDNet.

4.3. Efficiency Test

We will compare the computational efficiency of CLNet [18], DSAMNet [16], IFNet [14], ICIFNet [42], the networks for ablation experiments, and our LightCDNet in this section. To be specific, four indicators, floating point operations (FLOPs), the number of parameters of DNNs (NOPs), average training time (ATrT), and average testing time (ATeT), are computed for comparison. The ATrT and ATeT for DNNs depend heavily on the settings of hyperparameters and the running environment. Therefore, the four indicators are all computed in the computer mentioned in Section 2.2.5. The hyperparameter settings followed the original paper mentioned in Section 2.4. It should be noted that, unlike the setting in Section 2.2.5, the batch size is set to 4 in all the efficiency tests because the network using an encoder with Xception [59] in the ablation study cannot run on the above computer when batch size is 8 due to GPU memory limitation. Additionally, four indicators were calculated on the WHUCD dataset. All networks were trained using five epochs, and the average time per epoch was calculated as ATrT. Furthermore, we repeatedly predicted the testing set five times and took the average as ATeT.

Table 6. Comparison of the complexity of networks and computation cost on the WHUCD dataset.

Methods	FLOPs (M)	NOPs (M)	ATeT (s/Image)	ATrT (s/Epoch)
CLNet	16.202	8.103	0.031	204.12
DSAMNet	301,161.792	16.951	0.045	842.20
IFNet	329,055.161	50.442	0.060	604.36
ICIFNet	101,474.886	23.828	0.129	1041.75
Siamese Encoder (ResNet50)	283,461.036	155.725	0.077	738.29
Siamese Encoder (Xception)	289,171.335	168.750	0.118	725.05
Decoder (Upsampling)	48,801.779	10.405	0.052	294.18
Ours	85,410.156	10.754	0.056	303.57

shows that the FLOPs, NOPs, ATrT, and ATeT of CLNet [18] were less than all the other methods. Nevertheless, CLNet [18] provided the lowest accuracy among these methods on the WHUCD dataset (Table 3, Table 4 and Table 5). Compared with our LightCDNet, F1 of CLNet [18] is 18.8% less (Table 3). In addition, the FLOPs and NOPs of the proposed LightCDNet are lower than those of DSAMNet [16], IFNet [14], and ICIFNet [42]. Moreover, we can observe that the lightweight encoder of LightCDNet dramatically reduces the FLOPs, NOPs, ATrT, and ATeT, proving the encoder’s efficiency. Lastly, the up-sampling strategy in the decoder reduces the FLOPs, NOPs, ATrT, and ATeT. However, the Precision, Recall, F1, IoU, and OA (%) of the network using the up-sampling strategy in the decoder decreased by 1.0%, 1.7%, 1.4%, 2.3%, and 0.1%, respectively, compared with those of LightCDNet (Table 5). These findings indicate that our LightCDNet considers both efficiency and accuracy for BuCD.

4.4. Perspectives

To improve the accuracy and reduce the computation cost of BuCD, we designed a lightweight Siamese neural network, LightCDNet. Results (Table 2, Table 3, Table 4, Table 5 and Table 6) show that LightCDNet reduces the model complexity and the computation cost while improving accuracy. This indicates that LightCDNet befits resource-limited environments.

In future research, we will extend our method in two aspects. First, we will directly integrate boundary information into our network to generate outlines of the changed buildings in vector files because the vectors of altered buildings can be further applied to geographic information systems. Secondly, we will develop our network from binary BuCD to multi-class change detection using high-spatial-resolution RSIs.

5. Conclusions

In this paper, a lightweight Siamese neural network, LightCDNet, is proposed for BuCD using high-spatial-resolution RSIs. In detail, a Siamese encoder, the MultiTFFM, and a decoder comprise the LightCDNet. Firstly, the multi-temporal images are processed by the Siamese encoder, with the lightweight MobileNetV2 separately extracting multilevel features. Subsequently, the multi-temporal features are fused progressively on the basis of corresponding levels in the MultiTFFM, thus producing multilevel and multiscale fusion features. Finally, the fused features are fed into the decoder to gradually recover the changed buildings using the deconvolutional layers. LightCDNet and comparative approaches were tested on the LEVIRCD and WHUCD datasets. The proposed LightCDNet demonstrated its superiority over comparative networks in terms of both accuracy and efficiency.