Attention-Enhanced Region Proposal Networks for Multi-Scale Landslide and Mudslide Detection from Optical Remote Sensing Images

Abstract

Detecting areas where a landslide or a mudslide might occur is critical for emergency response, disaster recovery, and disaster cost estimation. Previous works have reported that a variety of convolutional neural networks (CNNs) significantly outperform traditional approaches for landslide/mudslide detection. These approaches always consider features from the local window and neighborhood information. The CNNs mainly focus on the features derived at a local scale, which might be inefficient for recognizing complex landslide and mudslide scenes. To effectively identify landslide and mudslide risks at a local and global scale, this paper integrates attentions into the architecture of state-of-the-art CNNs—including Faster RCNN—to develop an attention-enhanced region proposal network for multi-scale landslide/mudslide detection. In detail, we employed the attentions to process the region proposals generated by a region proposal network and then combined the results obtained from the attentions and region proposal network to identify whether the object included in a region proposal was a landslide/mudslide. Based on our developed dataset and the Bijie dataset, the experimental results prove that: (1) although the state-of-the-art CNNs for object detection can precisely detect landslides and mudslides, they are inadequate in dealing with similarity to non-landslide/non-mudslide regions; and (2) the proposed method, which integrates global features from attention layers into local features derived from CNNs, outperforms the unmodified CNNs in detecting non-landslides and non-mudslides. Our findings prove that the representations at the local and global scale might be significant for precise landslide and mudslide detection.

1. Introduction

Landslides and mudslides are significant natural disasters occurring in a wide range of areas with special topography and landforms [1,2,3,4,5]. Due to their speedy flow rate, massive flow volume, and ability to carry solid debris, landslides and mudslides have a large capacity for severe destruction. According to a report published by the Second Global Forum on Landslides, landslides have led to costs of over EUR 6 billion per year in damage in industrialized countries (https://www.isprambiente.gov.it/contentfiles/00010100/10185-second-world-landslide-forum-press-release.pdf, accessed on 16 November 2022). Recognizing and localizing a landslide or mudslide is critical for emergency response, disaster recovery, and disaster cost estimation. However, landslides and mudslides can look significantly different in various areas. Considering these characteristics, remote sensing has potential as a technique for mudslide and landslide detection, supporting this task in terms of accelerated and automatic identification for a large-scale area [6,7,8,9,10].

Although topographical properties and spectral information are useful for determining landslide/mudslide risks, the implementation of digital elevation models (DEMs) and multispectral remote sensing images might be restricted due to their limited spatial resolution. Economically assessed DEMs and multispectral remote sensing images generally have a spatial resolution ranging from 15 m to 100 m, allowing small-scale landslides/mudslides to be visually recognized. Currently, high-resolution optical remote sensing images provide significant data for detecting and monitoring landslides and mudslides. Previous works have reported a variety of approaches enhanced by visual features and semantics, such as image morphology [11], object-based image analysis [12], bag of visual words [13], and machine learning techniques [14]. To detect landslides/mudslides at different scales, these approaches require users to select a parameter or a scale so that successive detection and recognition can be achieved. This indicates these approaches might be challenging to use to automatically detect multi-scale landslides/mudslides in a large region.

Recently, the efforts toward employing a series of convolutional neural networks (CNNs) have attracted considerable attention from the research community [15,16]. Among these CNNs, Faster R-CNN is considered a state-of-the-art approach for imagery-based object detection and recognition [17,18]. Based on the backbone of a CNN that can obtain high-level representations from local neighbors, Faster R-CNN innovatively incorporates a region proposal network (RPN) into the architecture of a CNN to generate a number of region candidates more rapidly than other conventional ways. The backbone of an RPN has been successfully implemented into numerous networks in studies on landslide/mudslide detection and recognition [19].

However, Faster R-CNN mainly focuses on the features derived from a local window. Its detection of landslides and mudslides relies on discovering their features and their background information. In other words, any possible solution for precise landslide and mudslide detection should take advantage of the representations of landslides and mudslides at the local and global scale. This makes it difficult for CNNs such as Faster R-CNN to obtain features and information from the whole image. Moreover, Raghu, Unterthiner, and Kornblith [20] reported that the attentions in ViT [21] could derive similarities from multiple layers to achieve global representations of data, supporting the obtainment of more spatial details. For high-resolution remote sensing images, the self-attention within multiple heads, which is developed based on the structure of a transformer, aims to process the representations derived from various patches, or different parts of a remote sensing image [22].

Considering the difference between attention and CNNs in feature learning and extraction, researchers have explored how to effectively deal with the features from every patch and local window [22]. The currently proposed solution tackles this through combining Faster R-CNN and attentions [23]. The features derived from RPNs and attentions need to be addressed in multilayer feature processing. Thus, this paper integrates the critical feature layers of attentions into the architecture of Faster R-CNN to develop an attention-enhanced region proposal network for multi-scale landslide and mudslide detection. This paper is organized as follows, following this introduction. The second section compares Faster RCNN and attentions with respect to feature extraction from optical remote sensing images. The third section presents the details of our proposed attention-enhanced landslide/mudslide detection model. The fourth section reports and discusses the experimental results. Lastly, the fifth section summarizes the highlights and findings of our work.

2. RPNs and Attentions

2.1. RPNs for Landslide/Muslide Feature Learning and Detection

RPNs are used in structures to generate a series of proposals in one layer; essentially, they comprise the last convolutional layer of the backbone as the input, and the specific location of the proposal acts as the output of the neural network. An RPN mainly carries out the following steps [17]: anchor box generation, foreground or background determination of the anchor box, and regression for object localization. Anchor box generation focuses on image regions that possibly contain a target object. Foreground or background determination of an anchor box identifies whether an anchor box contains foreground or background information. When foreground or background information is available, regression for object localization generates the position difference between the anchor box and the true object in order to obtain the precise location of the object.

The general workflow of a CNN for landslide/mudslide detection and recognition includes: (1) generating a series of candidate regions (or proposals); (2) determining whether a candidate region contains the target object; and (3) if a candidate region includes the target object, regressing to modify the geometry of this candidate region to generate a bounding box for the target object. The strength of RPNs in remote-sensing-based landslide/mudslide detection has attracted significant attention. Since the original version [24], a variety of improved RPNs have been proposed, including those combining a scale pyramid with an RPN [25], employing various backbone nets [26], integrating decision trees [27], etc. Although successive landslide detection has been attempted, these improved RPNs mainly focus on local features, which might be inefficient to represent landslides/mudslides at different scales.

2.2. Attentions for Landslide/Muslide Feature Learning and Detection

A number of researchers have reported that the series of attentions developed based on the backbone of a transformer outperform the CNN models in a variety of tasks, such as scene classification, object detection, and segmentation [15,16].The backbone of attention, or multi-head attention, was initially proposed by Vaswani et al. [18] for extracting and learning sequential features in natural language processing. Furthermore, Dosovitskiy et al. [19] integrated visual features into the architecture of a transformer to develop the attentions, including a patching and position embedding part, a transformer encoder, and multi-head classification. The patching and position embedding part focuses on dividing the entire image into various patches, and building a linear projection to address the features of these patches. The transformer encoder develops a multi-head attention layer for feature extraction and learning.

In comparison to CNNs, attention-enhanced neural networks (NNs) have the following advantages:

• Scalability. The feature learning of attention-enhanced NNs is completed by an attention mechanism, which is a fully connected graph structure that helps discover the features and their relationships through nodes and vertexes [10]. Moreover, since the graph structure allows computing for heterogeneous nodes, an attention-enhanced NN can map the heterogamous data into a similar feature space to identify the corresponding inherited features. The landscapes of landslides and mudslides vary significantly, and thus their shapes and textures appear considerably different in various remote sensing images [20]. The scalability of attention-enhanced NNs might be useful for addressing the heterogeneities in visual features from optical remote sensing images;
• Global-view-based feature extraction. Graph-structure-based feature learning in attention-enhanced NNs focuses on extracting the features from the entire image [21]. Otherwise, convolution extracts the features from a local image region, including a pixel and its neighborhoods [22];
• Adaptive feature learning. Features derived from the convolutional layers in a CNN are generally followed by the same filter. Otherwise, the attention mechanism in attention-enhanced NNs could vary the filter for feature learning according to the pixels at a global scale; and
• Semantic-driven feature understanding. The origin of the transformer makes attention-enhanced NNs competitive in dealing with the semantics inherited in an image. Moreover, previous works have proved that sequential information can be useful.

Specifically, previous works have reported that a spatio-contextual approach overperforms others in landform element extraction [24,25]. This indicates that the information derived from neighborhood pixels is critical for detecting landform objects such as landslides and mudslides. Currently, many works are focused on segmentation, rather than identifying and localizing landslide and mudslide regions. Few references have reported employing ViT and the improved ViT called SWIN to conduct landslide detection [28,29].

3. The Proposed Approach

3.1. Architecture

Figure 1 illustrates the architecture of the proposed approach, which includes three main layers: a feature map generation layer, an attention-enhanced region proposal extraction layer, and an ROI classification layer. The feature map generation layer, which was developed based on the backbone of VGGNet [25], was derived from Faster RCNN. This layer focuses on extracting potential features from the input remote sensing image. The attention-based feature extraction layer conducts foreground determination to identify whether a region contains a target object, and region proposal regression to determine the anchor box that perfectly covers the identified object. In foreground determination, we combine the RPN and multi-head attentions to generate region proposals based on global and local features, specifically focusing on extracting the features, the region proposal network layer, the feature classification layer, and the region proposal regression layer.

3.2. Feature Map Generation Layer

The architecture of the generation layer focuses on generating the feature maps using the backbone of VGGNet. More details can be found in Reference [17]. The architecture of VGGNet includes 13 convolution operations, 13 ReLU activations, and 4 pooling operations. Pooling operations are max pooling, and the structure of convolution includes Conv1-Conv2-Conv3-Conv4, the structure of which is shown as follows:

• Conv1: {1 × 1, 64}, {3 × 3, 64}, and {1 × 1, 256};
• Conv2: {1 × 1, 128}, {3 × 3, 128}, and {1 × 1, 512};
• Conv3: {1 × 1, 256}, {3 × 3, 256}, and {1 × 1, 1024};
• Conv4: {1 × 1, 512}, {3 × 3, 512}, and {1 × 1, 2048}.

The feature map, which is generated by Conv4, holds a dimension of 2048. This feature map is integrated into the results given by the attention-enhanced region proposal extraction layer.

3.3. Attention-Enhanced Region Proposal Extraction Layer

The architecture of the attention-enhanced region proposal extraction layer is shown in Figure 1. Based on the backbone of an RPN, we designed the attention-enhanced region proposal extraction layer as a workflow including two independent processing stages: foreground determination for determining whether the object included in a region proposal is true, and region proposal regression for determining the bounding box that perfectly contains the object.

3.3.1. Foreground Determination

Foreground determination employs multi-head attention for determination by Faster RCNN, which is shown in Figure 2.

(1)

Two-dimensional positional embedding

We assume a remote sensing image as $\left[x,y,3\right]$, where $x$ and $y$ refer to the horizontal and vertical dimension of this image, respectively. Specifically, we define $x=y$, so the embedding can equally divide the whole image into various tokens. $c$ refers to the channel. We set the dimension of a token as $\left[16,16,3\right]$. Then, we have the number of tokens as ${\left(x/16\right)}^2$ and the dimension of tokens as $16\times 16\times 3=768$.

(2)

Encoding

Encoding is completed by a variety of encoding blocks, including layer normalization ($LN$), a multi-head attention ($MHA$), a droppath ($DP$), and a multi-layer perceptron ($MLP$). The structure of an encoding is expressed by Equation (1):

$$LN\to MHA\to DP\oplus LN\to MLP\to DP,$$

(1)

Layer normalization computes the average and variance for every feature map of one sample, and then normalizes the average and variance results. The droppath randomly inactivates the layers in a neural network, preventing the co-adaption of parallel paths in the neural network. The multi-layer perceptron creates a non-linear activation function to predict the classification result with a structure involving a fully connected layer, a GeLU activation function, and DropOut. The mechanism of the multi-head attention, which is presented in the following, is a complicated procedure that is critical for encoding.

The structure of a multi-head attention includes query ($q$), key ($k$), and value ($v$), which are designed to match other parameters to be matched and to be extracted. These three parameters can be obtained using the following equation:

$$\{\begin{array}{c}{q}_{i,h}=w_{q_i,h}\times I_i\\ k_{i,h}=w_{k_i,h}\times I_i\\ v_{i,h}=w_{v_i,h}\times I_i\end{array},$$

(2)

where $I$ refers to the input vector, $i$ refers to the index of the input vector, $h$ refers to the dimension of the head in the attentions, and $w$ refers to the weight.

Then, we can obtain the attention ($a_i$) with the query ($q$) and key ($k$) using the following equation:

$$a_{i,h}=\left(q_{i,h}\times k_{i,h}\right)/\sqrt{di{m}_{q_{i,h}\times k_{i,h}}},$$

(3)

where $i$ and $j$ refer to the index of the query ($q$) and the key ($k$), and $di{m}_{q_{i,h}\times k_{i,h}}$ refers to the dimension of the query ($q$) and key ($k$). Generally speaking, $q_{i,h}\times k_{i,h}$ might generate a vector with a high dimension. Thus, the square root $(\sqrt{di{m}_{q_{i,h}\times k_{i,h}}}$) can effectively decrease the dimensional number. Normalization can also be used to replace the square root.

Then, we use the Softmax function to process the attentions to enhance their scalability in linear and non-linear spaces. The Softmax function is expressed in Equation (4):

$$a{s}_i=exp\left(a_{i,h}\right)/{{\displaystyle \sum }}_{h}exp\left(a_{i,h}\right),$$

(4)

Based on this equation, we can obtain the output result (O_i), as shown in Figure 2, using Equation (5):

$$O_i={{\displaystyle \sum }}_i{{\displaystyle \sum }}_{h}a{s}_i\times v_{i,h},$$

(5)

$O_i$ refers to the final features (or the tokens), which are given to the ROI classification layer to identify whether an object in a regional proposal is the target one.

For landslide and mudslide detection, the position of an object is also a critical parameter. Thus, we use the cosine function to create a positional vector ($P_i$) that has the same dimensions as $I_i$, whose positional encoding is expressed by Equation (6):

$$\{\begin{array}{c}{q}_{i,h}=w_{q_i,h}\times I_i+w_{p_i,h}\times P_i\\ k_{i,h}=w_{k_i,h}\times I_i+w_{p_i,h}\times P_i\\ v_{i,h}=w_{v_i,h}\times I_i+w_{p_i,h}\times P_i\end{array},$$

(6)

From Equation (6), we obtain the $q_{i,h}$, $k_{i,h}$, and $v_{i,h}$ within the position and features. Then, we put these three parameters into Equations (1)–(5) to obtain the final features (or the tokens) within the position and features. Positional tokens undergo region proposal regression to generate the perforce anchor box (or region proposal) for a landslide and mudslide.

3.3.2. Region Proposal Regression

Region proposal regression aims to integrate a predicted region proposal into a new region proposal similar to the true one. This operation focuses on modifying the position of the anchor box by moving and scaling. We assume the region proposal as $\left[x,y,w,h\right]$, referring to the coordinate of the center pixel of this region proposal. $x$ and $y$ refer to the width and height of the center pixel, respectively. Then, we assume the true region proposal as $\left[x_t,y_t,w_t,h_t\right]$, and the predicted region proposal as $\left[x_p,y_p,w_p,h_p\right]$.

The regression includes translation and scaling, which are shown in Equation (7):

$$\{\begin{array}{c}∆x=\left(x_t-x_p\right)/w_p\\ ∆y=\left(y_t-y_p\right)/h_p\\ ∆w=log\left(w_t/w_p\right)\\ ∆h=log\left(h_t/h_p\right)\end{array},$$

(7)

where $\left[∆x,∆y,∆w,∆h\right]$ refers to the offset between the true and the predicted region proposal. Then, the loss function is:

$$L={{\displaystyle \sum }}_{∆,p}S{L}_1\left(∆-p\right),$$

(8)

where $∆\in \{∆x,∆y,∆w,∆h$}, $p\in \{x_p,y_p,w_p,h_p$}, and $S{L}_1$ refer to smooth L1 loss function.

The above calculation is carried out by object detection with CNNs. In the proposed method, since we have the positional tokens generated by multi-head attentions, we design an additional method to integrate the moving and scale results into the result of positional tokens.

Based on Equation (6), we can obtain the query (qP), key (kP), and value (vP) of the position, rather than input features.

$$\{\begin{array}{c}q{P}_{i,h}=w_{p_i,h}\times P_i\\ k{P}_{i,h}=w_{p_i,h}\times P_i\\ v{P}_{i,h}=w_{p_i,h}\times P_i\end{array},$$

(9)

Then, we transfer the $qP$, $kP$, and $vP$ from Equation (2) to Equation (5) to obtain the positional tokens ${\widehat{O}}_{dim}$, where $dim$ refers to its dimension. In addition, we define the true and predicted positional tokens as ${\widehat{Ot}}_{dim}$ and ${\widehat{Op}}_{dim}$, respectively. Using the smooth L1 loss function, we can obtain the difference between the true and predicted positional tokens using Equation (10):

$$L={{\displaystyle \sum }}_{dim}S{L}_1\left({\widehat{Ot}}_{dim}-{\widehat{Op}}_{dim}\right),$$

(10)

Next, we combine the results of Equation (8) with the results of Equation (10), and then integrate the combined results into the ROI pooling and identification layer to regress the position of possible landslide and mudslide objects again.

3.4. ROI Pooling and Identification

The region proposals generated by the attention-enhanced region proposal extraction layer might have different dimensions, making the traditional classification impossible to deal with. ROI pooling converts these proposals into proposal feature maps by rescaling the original dimension of a region proposal. Then, in the classification stage, the proposal feature maps are fully combined into Softmax to determine the category of the proposal. In addition, regression is used for the bounding box again to obtain the final exact position of the region proposal.

4. Experiment

4.1. Benchmark Dataset

To test the proposed approach for landslide and mudslide detection, we integrated our developed dataset into a state-of-the-art benchmark dataset called the Bijie Landslide Dataset [26]. Figure 3 illustrates the selected samples in this dataset. The dataset includes landslide and mudslide samples, and false samples that contain a scene similar to landslide and mudslide scenes (we call these non-landslide and non-mudslide samples), which total 770 and 2300, respectively. We selected the non-landslide and non-mudslide samples by (1) increasing the diversity among different landslides and mudslides, and (2) providing different scales of samples for the same landslide or mudslide object.

To further expand the dataset, we designed a variety of data augmentation techniques for each sample. The details of the data augmentation process are given in

Table 1. Data augmentation.

Data Augmentation	Methods
Rotation	Generating 7 new images through rotating the original image 45 degrees.
Flip	Generating 2 new images through flipping the original image over the horizontal dimension and vertical dimension.
Scale	Generating 4 new images through scaling the original image by 1:4, 1:2, 2:1, and 4:1.
Contrast	Generating 4 new images through modifying the original image with separate weighting parameters of AGCWD*: 0.2, 0.4, 0.6, and 0.8.
Brightness	Generating 4 new images through randomly modifying the brightness of the original image.
Fog noises	Generating 4 new images through randomly adding different cloud noises to the original image.

Efficient Contrast Enhancement Using Adaptive Gamma Correction with Weighting Distribution (AGCWD).

.

4.2. Experimental Analysis

The region proposal network was accessed at: https://pytorch.org/vision/main/models/faster_rcnn.html, accessed on 16 November 2022. To test the scalability of the proposed method, we created ten experimental groups in total. Figure 3 illustrates the selected samples from the integrated benchmark dataset.

For each experimental group, we randomly selected 300 samples of landslides and mudslides and non-landslides and non-mudslides from the integrated benchmark dataset to pretrain the proposed model, and used the remaining data to test the detection results.

Table 2. Detection results for landslides and mudslides and non-landslides and non-mudslides.

	Faster RCNN		The Proposed Method
	Landslides and Mudslides	Non-Landslides and Non-Mudslides	Landslides and Mudslides	Non-Landslides and Non-Mudslides
Group 1	384/400 0.9600	1161/2000 0.5805	385/400 0.9625	1321/2000 0.6605
Group 2	379/400 0.9475	1140/2000 0.5700	380/400 0.9550	1346/2000 0.6730
Group 3	390/400 0.9750	1125/2000 0.5625	390/400 0.9750	1318/2000 0.6590
Group 4	385/400 0.9625	1098/2000 0.5490	388/400 0.9700	1240/2000 0.6200
Group 5	388/400 0.9700	1075/2000 0.5375	394/400 0.9850	1199/2000 0.5995
Group 6	385/400 0.9625	1197/2000 0.5985	387/400 0.9675	1372/2000 0.6860
Group 7	377/400 0.9425	1158/2000 0.5790	377/400 0.9425	1333/2000 0.6665
Group 8	382/400 0.9550	1146/2000 0.5730	383/400 0.9575	1409/2000 0.7045
Group 9	387/400 0.9675	1137/2000 0.5685	390/400 0.9750	1350/2000 0.6750
Group 10	389/400 0.9725	1141/2000 0.5705	390/400 0.9750	1367/2000 0.6835

lists the detection results for landslides and mudslides.

4.3. Discussion

From the results, several conclusions could be reached.

• Both Faster RCNN and the proposed method perform well in detecting landslides and mudslides. This proves that the state-of-the-art CNN techniques for object recognition have great potential for visually detecting landslides and mudslides from remote sensing images. Although the land surface surrounding a landslide or mudslide can vary, CNN methods can effectively discover and learn the high-level representations of landslides or mudslides.
• However, the recall of detection of this method is poor. In the benchmark dataset, we selected the non-landslide and non-mudslide samples based on expert knowledge, ensuring these images were similar to true samples. In other words, these samples included a potential landslide or a mudslide region, although the landslide or mudslide was not visually recognizable in these images. This issue generates considerable interference in state-of-the-art CNN techniques for object recognition. The selected incorrect results for non-landslide and non-mudslide detection are shown in Figure 4.
• The recall of detection results generated by the proposed method outperformed those generated by Faster-RCNN. In comparison to the architecture of Faster R-CNN, the architecture of the proposed method added attentions for classifying the features derived from the global image. Thus, we assume that features extracted from every patch of an image can avoid interference by non-landslide and non-mudslide regions. The features at a global scale might be sufficient to enhance the identification of true landslide and mudslide features from optical remote sensing images.

5. Conclusions

Landslide and mudslide detection from optical remote sensing images is a main task of remote sensing in landform and geological disaster monitoring. Since landslides and mudslides are distributed across a wide range of areas, and developed in multiple scales, labor-based monitoring might not precisely and completely detect all landslides and mudslides. Remote sensing provides a comprehensive dataset allowing landslide and mudslide detection to be accomplished with multi-scale monitoring and no physical contact. High-resolution images can provide spatial details of a landslide and a mudslide, requiring an approach to effectively address the complicated background and neighborhood of landslides and mudslides [30,31].

State-of-the-art CNNs for object recognition (Faster RCNN) can extract and discover the representative features of landslides and mudslides. Thus, a number of previous works have employed a variety of CNNs to identity landslide and mudslide regions from optical remote sensing images. However, these models mainly focus on learning the representations at local scales, which might lead to the exclusion of surrounding information. As reported in the experimental results, Faster RCNN could not efficiently identify irrelevant regions. The attentions extract and discover the representative features from every part of an image. These representative features at a global scale might be useful for distinguishing between a true landslide and mudslide and a false one. The proposed method incorporates the global features derived from attentions into the architecture of Faster RCNN, allowing the features to be used at global and local scales. The experimental results prove that the proposed method can outperform Faster RCNN in filtering out irrelevant regions while retaining high precision in recognizing landslides and mudslides. Several aspects are worthy of attention in future works. First, landslides and mudslides are represented at different scales in optical remote sensing images. Adding scale-independent parameters could significantly improve attention-enhanced feature learning. Moreover, the integration of attentions into a YOLO (You Only Look Once) model—another state-of-the-art CNN for object detection—represents a potential technical solution.