Robust Landslide Recognition Using UAV Datasets: A Case Study in Baihetan Reservoir

Abstract

The task of landslide recognition focuses on extracting the location and extent of landslides over large areas, providing ample data support for subsequent landslide research. This study explores the use of UAV and deep learning technologies to achieve robust landslide recognition in a more rational, simpler, and faster manner. Specifically, the widely successful DeepLabV3+ model was used as a blueprint and a dual-encoder design was introduced to reconstruct a novel semantic segmentation model consisting of Encoder1, Encoder2, Mixer and Decoder modules. This model, named DeepLab for Landslide (DeepLab4LS), considers topographic information as a supplement to DeepLabV3+, and is expected to improve the efficiency of landslide recognition by extracting shape information from relative elevation, slope, and hillshade. Additionally, a novel loss function term—Positive Enhanced loss (PE loss)—was incorporated into the training of DeepLab4LS, significantly enhancing its ability to understand positive samples. DeepLab4LS was then applied to a UAV dataset of Baihetan reservoir, where comparative tests demonstrated its high performance in landslide recognition tasks. We found that DeepLab4LS has a stronger inference capability for landslides with less distinct boundary information, and delineates landslide boundaries more precisely. More specifically, in terms of evaluation metrics, DeepLab4LS achieved a mean intersection over union (mIoU) of 76.0% on the validation set, which is a substantial 5.5 percentage point improvement over DeepLabV3+. Moreover, the study also validated the rationale behind the dual-encoder design and the introduction of PE loss through ablation experiments. Overall, this research presents a robust semantic segmentation model for landslide recognition that considers both optical and topographic semantics of landslides, emulating the recognition pathways of human experts, and is highly suitable for landslide recognition based on UAV datasets.

1. Introduction

Landslides are one of the most common geological disasters in the world. When a landslide occurs, it often carries a large amount of soil, rock, or a mixture of both, moving downhill along the sliding surface formed on the slope [1]. As a result, the built environment (including houses, factories, roads, etc.) and the natural environment (including farms, forests, grasslands, etc.) on the slope are partially or completely destroyed [2,3]. In some extreme cases, landslides and their secondary disasters can cause considerable damage. For example, the 2014 Abe Barek landslide in Badakhshan Province, Afghanistan, resulted in nearly 2700 fatalities [4]; the 2018 Baige landslide in Tibet, China, blocked the main stream of the Jinsha River and formed a huge barrier lake, which breached, causing floods that affected areas as far downstream as Shangri-La and Lijiang, nearly 600 km away, resulting in losses exceeding USD 500 million in Lijiang alone [5,6]. However, landslide disasters are not entirely uncontrollable. Effective monitoring of landslide-prone areas and research into landslide initiation mechanisms, motion processes, and risk management, based on monitoring data, can lead to a comprehensive understanding of landslides. This, in turn, allows for the implementation of early measures to reduce the losses from landslide disasters and even prevent them through certain slope reinforcement measures [7,8,9].

In practice, the effective monitoring of landslide areas is not an easy task. After a major landslide disaster, detailed field investigations, sampling, and experimental work are typically required [10]. However, the mountainous distribution characteristics of landslides often make it difficult for researchers and monitoring equipment to quickly reach the disaster site, thus limiting the possibility of immediate landslide research post-disaster. Fortunately, the recent popularization of low-cost unmanned aerial vehicles (UAV) has greatly facilitated the monitoring of landslides. UAV, with their convenience, speed, proximity, and lightweight dataset [11], perfectly meet the needs for landslide monitoring. The use of UAV photogrammetry technology allows researchers to easily obtain extensive landslide photographs from a distance. Then, combined with structure-from-motion (SfM) technology, very high resolution (VHR) landslide image datasets can be quickly constructed, which can be used as the basic data for landslide identification and monitoring [12].

Following the deformation process of the landslide, a number of physical changes can be observed through photogrammetry. These include alterations in topography [13], vegetation cover [14], color (spectrum) [15], and cracks [16]. Therefore, based on the aforementioned multifaceted information, landslide recognition can be indirectly achieved from the landslide image dataset. Landslide recognition based on the image dataset can be divided into four methods: manual interpolation method, object-oriented method, object-based deep learning (DL) method, and pixel-based DL method. Figure 1 provides an intuitive illustration of the results of these four landslide recognition methods. The manual interpolation method (Figure 1a) employs direct observation of landslide features in images [17], or alternatively, the selection of a threshold through trial and error, followed by semi-automated recognition of landslide areas in images based on this threshold [18,19]. The object-oriented method (Figure 1b) often employs the multiresolution segmentation approach, which first divides the image into multiple non-overlapping objects based on homogeneity by selecting the optimal segmentation scale, and then extracts the landslide objects by combining the spectral, textural, and morphological characteristics of landslides [20,21]. The object-based deep learning (DL) method (Figure 1c) uses algorithms to detect objects in images by determining bounding boxes and classes, either in one or two stages. The YOLO series models are key examples of this method [22,23]. The pixel-based method (Figure 1d) relates to semantic segmentation tasks, which classify each pixel in the image using deep learning techniques, considering both local and regional context [24,25]. Overall, the object-based methods focused on object detection and the pixel-based methods focused on semantic segmentation have come together as the dominant methods for landslide recognition [26,27].

In contrast to object detection, semantic segmentation could provide more physical information about landslides, including boundaries and areas. While current popular semantic segmentation network models such as U-Net [28], Mask R-CNN [29], and DeepLabV3+ [30] can accurately perform landslide recognition tasks, they encounter significant difficulties in complex scenarios. Since these popular models usually use only RGB optical images as inputs, they may experience interference from factors such as vegetation, shading, and human activities in real landslide identification scenarios [31,32,33]. To address this problem, we hypothesize that the introduction of a corresponding DSM obtained from SfM may be able to ameliorate this problem [34]. Since the DSM essentially represents a 2.5D reproduction of a real-world 3D model, its introduction into the semantic segmentation model can be interpreted as improving the ability to identify landslides from an engineering geological perspective, in a manner similar to the empirical judgement of human experts during landslide field investigations.

This study will reference the architecture of DeepLabV3+ to re-design and develop a new semantic segmentation model named DeepLab for Landslide (DeepLab4LS) for landslide recognition that integrates optical and topographic information, which will be validated on landslide UAV datasets from the Baihetan reservoir. Additionally, this paper will investigate the issue of the loss function during model training to determine the most suitable loss function scheme for landslide recognition through detailed experiments. Finally, comparison and ablation experiments will be conducted on the novel semantic model to explore the role of various details in the architecture and training of the semantic model in landslide recognition.

2. Study Area and Dataset

2.1. Study Area and Data Acquisition

The Baihetan reservoir was formed after the Baihetan hydropower station began storing water in April 2021, with a total storage capacity of 20.627 billion m³ [35]. The storage of water has changed conditions such as the groundwater level and pore water pressure on the slopes [36], leading to the continuous development of a series of landslides within the reservoir area [16,37,38]. The activity of the landslides has affected the production and traffic of towns along the river within the reservoir area, and the potential impulse wave risk of the landslides also poses a significant threat to the downstream Baihetan hydropower station. In this study, a UAV image dataset obtained from the Baihetan reservoir in the lower reaches of the Jinsha river in China is employed as the data source for model training and validation.

Therefore, to thoroughly investigate the distribution of landslides within the reservoir area, a VHR UAV image dataset was collected between February and September 2023. This dataset, which covers an area of approximately 65 km², encompasses a stretch of Jinsha river within the reservoir area measuring approximately 50 km in length (Figure 2). The UAV photogrammetry employed the Feima D2000 four-rotor UAV, equipped with a SONY DOP 3000 camera (Shenzhen Feima Robotics Co., Ltd., Shenzhen, China) (Figure 3). The UAV flew in a fixed flight height mode at an altitude of approximately 150 to 200 m above the ground, ensuring that the resolution of the collected photos was stable at approximately 0.03 m. The heading overlap and the side overlap were 80% and 65%, respectively. In total, data collection for the UAV photogrammetry took approximately 1 month.

The original photos captured by the UAV and the corresponding ground control points (GCPs) were input into Pix4D software (4.9.0 trial version), where they underwent the SfM process. As a result, the orthophoto (Figure 2) and DSM of the study area were generated. It is worth mentioning that the orthophoto and DSM were subsequently downsampled to 0.1 m to avoid potential issues during subsequent deep learning training. At a resolution of 0.03 m, a 1024 × 1024 size cropped image would only represent less than 1000 m² of surface area. In many cases, this would only cover a small part of a landslide, preventing the network model from effectively processing the landslide’s regional context information [39]. Meanwhile, using a resolution of 0.03 m and cropping to 1024 × 1024 size would result in a dataset with over 50,000 images, which would be a significant burden for training. Therefore, using a dataset with a resolution of 0.1 m could effectively address the two aforementioned issues.

2.2. Dataset Creation

There are three types of image data used as inputs for the semantic segmentation model, which can be summarized as follows: optical semantic data, topographic semantic data, and label data. The optical semantic data are represented by orthophoto, while the label data are obtained by landslide experts who observe the orthophoto and annotate it using ArcGIS software. Here, it is worth mentioning the selection of topographic semantic data. We ultimately chose relative elevation, slope, and hillshade as the three types of data to collectively constitute the topographic semantic data used for input (Figure 4).

Relative elevation measures the degree of elevation difference within a region, numerically represented by the absolute elevation difference between the highest and lowest points in the area. The relative elevation metric directly affects the gravitational potential energy level. The greater the maximum value of relative elevation within an area, the higher the potential energy at that point, indicating an increase in the driving force of the landslide [40,41]. Although the concept of relative elevation is intended to be used, in practice, when composing topographic semantic data, we directly use elevation (i.e., the DSM itself) as input, because neural networks inherently perform normalization operations when processing data inputs, and the processed elevation becomes relative elevation.

Slope represents the angle between the vertical height of a point and its horizontal projection on a slope, intuitively reflecting the degree of slope inclination, which is also importantly related to landslides. An increase in the slope means that the component of gravity projected onto the potential sliding surface of the landslide also increases [42]. According to statistical data, most landslides have a slope between 10° and 30° [43].

By simulating the position and angle of the sun’s light source, hillshade calculates the illumination of each pixel to reflect the undulations and terrain of the landscape. Hillshade does not have a strong correlation with landslides on a geological level, but it has unique advantages when analyzing landslides using UAV datasets: it can reduce the impact of illumination differences in UAV data at different times [44]. In fact, the DSM exported by SfM can also mitigate the effects of illumination differences, but compared to the RGB image, the DSM loses a lot of shape and texture information. However, the hillshade calculated from the DSM can supplement a significant amount of image shape and texture, making it an ideal source of topographic semantic data for landslides. In this study, both slope and hillshade were directly exported from the DSM using ArcGIS software version 10.8. The solar altitude angle and the orientation of hillshade in this manuscript are 45° and 90°, respectively.

Furthermore, it is important to note that in subsequent sections of this paper, the term “topographic image” is used as a generic term for the DSM, hillshade, and slope, while the term “optical image” refers to the orthophoto. Both the terms precisely convey the core content of these two types of data, and it is helpful for understanding the data usage in this study.

To ensure that the model training could run smoothly under limited hardware conditions, the orthophoto, DSM, slope, hillshade, and label data were then cropped to a size of 1024 × 1024 and the cropped image set was filtered. The filtered images met at least one of the following conditions: (1) the proportion of pixels with null values exceeded 90%; (2) there were no pixels labeled as landslides. After the above operations, a dataset composed of 633 images of 1024 × 1024 size was obtained. The dataset was randomly divided into a training set and a validation set in an 8:2 ratio, which were used for model training and validation, respectively.

The entire dataset creation process for this section is shown in Figure 5, which also includes the mathematical expression of the filtering conditions for the image set.

2.3. Dataset Augmentation

In order to expand the dataset, avoid model overfitting, and enhance the generalizability of the model, data augmentation (DA) methods were used to improve the training set [45]. Specifically, four types of DA operations were applied: stretching, flipping, Gaussian blur, and rotation (Figure 6). We applied these four transformations using an on-the-fly data augmentation approach [46]. During the training process, these DA operations were applied randomly before the dataset was loaded into memory. Theoretically, this means that the training set for each epoch is different during the training process, increasing the diversity of the data and helping the model generalize better to unseen data.

3. Methods

3.1. DeepLabV3+

3.1.1. Model Architecture

DeepLabV3+ is a deep learning semantic segmentation model developed by Google, which has been widely applied in fields such as medical image segmentation [47], remote sensing image analysis [48], and autonomous driving [49]. Compared to previous models in the DeepLab series, DeepLabV3+ introduces a decoder module to refine segmentation results, especially along the boundaries of target objects, making it also a typical semantic segmentation model with an encoder-decoder architecture. DeepLabV3+ also upsamples the output feature maps from the encoder through the decoder module and fuses them with low-level features, thereby restoring detailed information about the targets and improving the accuracy of segmentation boundaries. A simplified diagram of the overall architecture of DeepLabV3+ is shown in Figure 7.

3.1.2. The Dilated Convolution & ASPP Optimizations

The two most important optimizations in the DeepLabV3+ encoder are dilated convolutions and Atrous Spatial Pyramid Pooling (ASPP), which indirectly contribute to DeepLabV3+’s excellent performance in multi-scale object recognition. By using dilated convolution operations instead of pooling operations, it is possible to obtain a receptive field larger than the size of the convolutional kernel without reducing the size of the feature map, while keeping the number of parameters manageable. For a standard convolution process, it can be described as:

$$y\left[i,j\right]={\displaystyle {\sum }_{m=-M/2}^{M/2}{\displaystyle {\sum }_{n=-N/2}^{N/2}x\left[i+m,j+n\right]·w\left[m,n\right]+b}},$$

(1)

However, in dilated convolution, the distance between adjacent pixels in the image corresponding to the convolutional kernel is not constantly 1, which means that:

$$y\left[i,j\right]={\displaystyle {\sum }_{m=-M/2}^{M/2}{\displaystyle {\sum }_{n=-N/2}^{N/2}x\left[i+r·m,j+r·n\right]·w\left[m,n\right]+b}},$$

(2)

where y is the element of the output feature map, x is the element of the input feature map, w is the weight of the convolution kernel, b is the bias, m and n are the indexes of the convolution kernel, and r is the dilated ratio, which is an integer greater than or equal to 1.

Dilated convolution can effectively improve the ability of the model to capture the features of large targets, such as landslides. As shown in Figure 8, when the dilation rate of the 3 × 3 convolution kernel is 4, the receptive field of the convolution operation is expanded from a space of 9 pixels to a space of 81 pixels, which already contains almost the whole landslide area.

Relying solely on dilated convolution can effectively expand the receptive field, but it does not achieve unified feature extraction for multi-scale objects. To address the issue of multi-scale feature extraction, Spatial Pyramid Pooling (SPP) has provided an excellent practical approach—establishing multiple parallel branches and using pooling operations with different kernel sizes in each branch to extract and ultimately integrate multi-scale features [50]. On the other hand, as mentioned above, dilated convolutions can be seen as a non-downsampling alternative to pooling operations. Naturally, the idea arises to use dilated convolution operations to replace the pooling operations in some branches of SPP, giving rise to Atrous Spatial Pyramid Pooling. Since dilated convolutions do not change the size of the feature map, this substitution allows the output results of each branch to be directly concatenated in the channel dimension (except for the pooling branches, which require upsampling), resulting in a multi-scale feature map that fully accounts for the features of objects at different scales. Figure 9 shows the basic structure of the ASPP module.

DeepLabV3+ has shown good performance on several open-source datasets, but these datasets are based on RGB optical images obtained from common imaging devices.

Therefore, the topographic information in the UAV datasets would be lost by using the general DeepLabV3+. For this reason, in order to achieve a landslide semantic segmentation model with good generalization performance, we re-designed a new DeepLabV3+-based semantic segmentation model for landslide recognition that includes topographic information.

3.2. DeepLab4LS: A Dual-Encoder DeepLab Model for Landslide

3.2.1. Basic Design of Model

In order to design a landslide recognition model that fuses both optical and topographic semantics, it is first necessary to consider the fusion strategy. In deep learning tasks for computer vision, common data fusion methods can be categorized into three types: simple fusion, channel fusion, and dual-encoder fusion (Figure 10). Simple fusion is considered the most basic form of data fusion, where two sets of data are directly added or multiplied together. This method necessitates high consistency between the two datasets and is typically used for fusing intermediate features within the network [51,52,53]. Channel fusion concatenates optical and topographic data along the channel dimension before its input into the model, thus better preserving the original characteristics of the data [54,55,56]. Dual-encoder fusion (also known as dual-backbone fusion) goes a step further by feeding each data set into an encoder (which is primarily composed of a backbone) and the features outputted by the encoders are then fused at the model level (usually through direct channel fusion) [57,58,59]. Compared to the other methods, dual-encoder fusion is the most complex, but it allows the data to undergo several layers of neural network processing both before and after fusion, thereby enabling a true deep fusion of data.

Therefore, to achieve better performance, this study employs a dual-encoder fusion approach for the model design. Overall, the network receives two types of inputs: orthophotos as optical features, and DSM, slope, and hillshade images as topographic features. Initially, two separate encoders are used to extract preliminary optical and topographic semantic features. Subsequently, these extracted semantic features are combined through channel fusion, followed by channel adjustment, size adjustment, and SoftMax processing to produce the final segmentation results. Throughout this process, the subsampling rates of the backbone within the encoders are appropriately adjusted to unify the subsampling multiples of the dual encoders.

3.2.2. Encoders Design

Considering the increase in the number of parameters introduced by the new encoders, the optical feature extraction backbone adopts the lightweight MobileNetV2 [60]. MobileNetV2 achieves model miniaturization and efficient inference speed through its unique inverted residual structure combined with depthwise separable convolutions. In numerous experiments [61,62,63], MobileNetV2, as a backbone, has been able to achieve results comparable to or even surpassing those of ResNet50 [64], with the former having only 1/7 to 1/8 of the latter’s parameters. Overall, MobileNetV2 can be regarded as an excellent lightweight backbone in terms of both computational accuracy and parameter volume.

With regard to the topographic feature extraction backbone, no single network emerges as a clear standout. In order to identify the most suitable network for this purpose, this study tested five different networks for their ability to extract topographic semantics, including VGG-16 [65], ResNet-18, ResNet-50, ShuffleNetV2 [66], and MobileNetV2. Among these, ShuffleNetV2 is a lightweight network structure similar to MobileNetV2. In ShuffleNetV2, the introduction of channel shuffle operations and pointwise group convolution techniques significantly reduces the model’s parameter count while ensuring high accuracy. Regardless of which topographic feature extraction backbone is used, from the perspective of controlling the overall model’s parameter volume, none employs the deep and shallow feature fusion method found in DeepLabV3+. Instead, they directly process based on the final output.

3.2.3. Model Architecture

With this understanding, we referenced the architecture of DeepLabV3+ and redesigned and implemented a recognition semantic segmentation model for landslide recognition—DeepLab for Landslide (DeepLab4LS) (Figure 11), composed of four modules: Encoder1, Encoder2, Mixer, and Decoder. The detailed structural design of the four modules is as follows:

(1)

Encoder1

Encoder1 serves as the optical feature extraction module, comprising the optical feature extraction backbone and its associated components. Encoder1 takes RGB images (orthophotos) as input and normalizes them to a range of [0, 1] before entering the network. The optical feature extraction backbone is based on the MobileNetV2 model, with the original model’s downsampling factor adjusted to 8× to prevent data distortion caused by excessive downsampling. The final output of Encoder1 is divided into two parts: 48-channel 8× downsampled shallow optical semantic features and 256-channel 8× downsampled deep optical semantic features.

(2)

Encoder2

Encoder2 acts as the topographic feature extraction module, comprising the topographic feature extraction backbone and its associated components. Encoder2 accepts a 3-channel image composed of stacked single-channel images of DSM, slope, and hillshade as input, and normalizes it to a range of [0, 1] before entering the network. In order to align with the deep optical semantic features outputted by Encoder1, the downsampling factor of Encoder2 is also set to 8x, and the final channel count of topographic semantic features is fixed at 256. This allows the shallow optical semantic features, deep optical semantic features, and topographic features to be fused at a ratio of 48:256:256.

(3)

Mixer

The Mixer serves as the multimodal feature fusion module. Within the Mixer, the task is to further achieve cross-semantic fusion of optical and topographic semantics, building upon the foundation of fusing shallow and deep optical semantic features. Before fusion, deep optical semantic features and topographic semantic features are upsampled by a factor of 2 using bilinear interpolation to return to a 4× downsampled size. The three sets of features are then channel-stacked and immediately processed through two 3 × 3 convolution layers. The first convolution layer transforms the 256 + 48 + 256 channel features into 256 channels of fused semantics, while the second convolution layer further refines the fused semantics without changing the number of channels. As a result, the Mixer outputs a 256-channel feature map downsampled by 4×, representing the outcome of deep multimodal feature fusion.

(4)

Decoder

The Decoder is the conversion module for segmentation results. The main function of the Decoder is to convert the feature map output from the Mixer into the final segmentation results. Therefore, its network structure is relatively simple, consisting of a 1 × 1 convolution layer, a bilinear interpolation module, and a final SoftMax module. The 1 × 1 convolution layer is used for channel adjustment (outputting 2 channels, representing the landslide foreground and background), the bilinear interpolation adjusts the image size to match the input image size, and the subsequent SoftMax transformation yields pixel classification probabilities. Finally, the category with the highest probability is selected as the pixel class to produce the final segmentation results.

It is important to emphasize that in the aforementioned Encoder1 and Encoder2 modules, adjustments to the downsampling multiples of each model are made by modifying the stride of the convolutional or pooling layers accordingly. Another modification to these models is the removal of the classifiers, so that they exist only as a backbone for a semantic segmentation model, rather than as complete classification models.

3.2.4. Improved Loss Function

In this study, another significant endeavor is the formulation of a loss function tailored for landslide recognition, which is then applied to the DeepLab4LS model. Given the relatively low proportion of landslide pixels in the image, a novel loss function, positive enhanced loss (PE loss), is introduced. This loss function is designed to enhance the model’s ability to comprehend positive samples. The PE loss is defined by the following equation:

$$PE loss=1-\frac{1}{\left|C\right|}\frac{{\displaystyle \sum _{pixels}{y}_{\mathrm{true}}{y}_{\mathrm{pred}}}}{{\displaystyle \sum _{pixels}\left(y_{\mathrm{pred}}+y_{\mathrm{true}}-2y_{\mathrm{true}}{y}_{\mathrm{pred}}\right)}},$$

(3)

Here, y_true is equal to 1 when a pixel is a positive sample and 0 otherwise, while y_pred is equal to 1 when a pixel is predicted as a positive sample and 0 otherwise. The value of |C| is the total number of classes, which is consistently equal to 2 in this study.

During the training process of the DeepLab4LS model, the loss function is composed of both Positive Enhanced loss and Cross Entropy loss (CE loss), which is:

$$loss=PE Loss+CE Loss,$$

(4)

CE loss is a commonly used loss function in deep learning. The official implementation of DeepLabV3+ employs CE loss as its loss function [67]. The incorporation of the PE loss component into the loss function serves to reinforce the gradient stability and to enhance the model’s probability estimation capabilities.

3.3. Training Strategy

To facilitate model convergence and maintain the stability of learning low-level features, this study adopts a strategy of using pre-trained weights combined with a freeze-training approach. Given that the DeepLab4LS model’s backbones are based on widely-used feature extraction networks that have been slightly adjusted for downsampling factors, it is possible to apply the pre-trained weights from these networks, which have been trained on various open-source datasets, as the initial weights for DeepLab4LS’s backbone components. The initial weights, w_i, of the non-backbone parts of DeepLab4LS are initialized to follow a normal distribution with a mean of zero and a standard deviation of 0.02.

$$w_i~N\left(0.0,{0.02}^2\right),$$

(5)

The model training was conducted using an NVIDIA GeForce RTX 3060 graphics card with 12G of dedicated VRAM. The training phase was set to a total of 500 epochs, with the first 50 epochs designated for freeze training. During the freeze phase, the batch size was set to 8, and for the unfreeze phase, it was reduced to 4. This approach ensured that the total number of training steps exceeded 79,000, guaranteeing full convergence of the model. In the initial freeze training phase, the weights of the backbone components (Encoder1 and Encoder2) were fixed, allowing for rapid adjustment of the Mixer and Decoder weights, which quickly elevated the model to a higher level of accuracy. In the subsequent unfreeze phase, the weights of the entire model, including the backbone networks, were fine-tuned, enabling the model to gradually develop the capability for deep extraction and inference of landslide semantics.

Furthermore, to minimize VRAM consumption, the training process employs PyTorch’s mixed-precision training strategy. This strategy can boost training efficiency by integrating half-precision (float16) with single-precision (float32) floating-point numbers, thus striking a balance between performance and memory usage. Additionally, key hyperparameters that influence the training dynamics, including optimizer and learning rate, are concisely detailed in

Table 1. The training parameters set for DeepLab4LS.

Environment and Parameters	Name and Value
Development Environment	Python 3.7.13/PyTorch 1.10.1/NVIDIA CUDA 11.3
Epoch	50 (freeze) + 450 (unfreeze)
Batch Size	8 (freeze)/4 (unfreeze)
Optimizer	Stochastic gradient descent
Momentum	0.9
Weight Decay	10−4
Learning Rate	7 × 10−3 (beginning)/7 × 10−5 (minimum)
Learning Rate Decay	Cosine annealing

.

3.4. Evaluation Metrics

In this study, mIoU (mean Intersection over Union), mPA (mean Pixel Accuracy) and Accuracy were introduced to evaluate the accuracy of the semantic segmentation model in landslide recognition tasks. mIoU is a crucial metric for almost all semantic segmentation models and indicates the percentage of intersection over the union of the predicted and ground truth regions, providing a quick assessment of the model’s performance across all classes. mPA is derived by averaging the percentage of correctly classified pixels within each class, placing more emphasis on the model’s performance in each individual class. Accuracy, which is more intuitive than the first two, represents the proportion of correctly predicted pixels out of the total pixels, reflecting the model’s performance across the entire dataset. Specifically, the calculation methods for these three evaluation metrics are as follows:

$$\mathrm{mIoU}=\frac{1}{N}{\displaystyle {\sum }_{i=1}^N\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FP}}_i+{\mathrm{FN}}_i}},$$

(6)

$$\mathrm{mPA}=\frac{1}{N}{\displaystyle {\sum }_{i=1}^N\frac{{\mathrm{TP}}_i}{{\mathrm{TP}}_i+{\mathrm{FN}}_i}},$$

(7)

$$\mathrm{Accuracy}=\frac{{\displaystyle {\sum }_{i=1}^N{\mathrm{TP}}_i+{\mathrm{TN}}_i}}{{\displaystyle {\sum }_{i=1}^N{\mathrm{TP}}_i+{\mathrm{TN}}_i+{\mathrm{FP}}_i+{\mathrm{FN}}_i}},$$

(8)

where N is the total number of classes, and TP_i, FP_i, TN_i, FN_i are the true positive, false positive, true negative, and false negative counts for the i-th class, respectively.

4. Results

4.1. Segmentation Results and Comparison Experiments

The segmentation results of DeepLab4LS using different topographic semantics to extract the backbone are shown in

Table 2. Segmentation results of DeepLab4LS under different topographic semantic extraction backbones.

Optical Semantic Extraction Backbone	Topographic Semantic Extraction Backbone	mIoU (%)	mPA (%)	Accuracy (%)
MobileNetV2	MobileNetV2	76.0	85.3	92.3
	VGG-16	72.6	86.3	90.3
	ShuffleNetV2	75.5	83.7	92.4
	ResNet-18	74.1	84.1	91.5
	ResNet-50	74.8	83.7	92.0

. It is evident that no single topographic semantic extraction backbone excels in all three evaluation metrics simultaneously. However, in the overall evaluation, MobileNetV2 emerges as the superior topographic feature extraction backbone network. It has the highest mIoU and the second highest mPA and precision, demonstrating its robust ability to extract landslide semantics from topographic features. Somewhat surprisingly, VGG-16 achieved the highest mPA of the five topographic semantic extraction backbones, but scored the lowest in both mIoU and accuracy. This suggests that the model may have a tendency to predict pixels as landslides, or it may not have effectively learned the boundary features of landslides. Overall, considering both number of parameters and performance, MobileNetV2 stands out as a relatively optimal backbone for topographic semantic extraction.

Further, the DeepLab4LS model was compared with five other popular semantic segmentation models, and the results are shown in

Table 3. Comparison experiments between DeepLab4LS and other popular semantic segmentation models.

Model	mIoU (%)	mPA (%)	Accuracy (%)
DeepLab4LS (with MobileNetV2 as the topographic backbone)	76.0	85.3	92.3
DeepLabV3+	70.5	80.1	90.4
PSPNet	70.5	81.2	90.1
SegFormer	70.1	77.8	90.8
U-Net	67.6	77.5	89.3
HRNet	69.6	80.0	89.8

. It is quite apparent that the DeepLab4LS model has an absolute advantage in all evaluation metrics, with its mIoU surpassing the second place by 5.5 percentage points, indicating the significant gain brought by the integration of topographic features in landslide recognition. Among the other five models, DeepLabV3+ and PSPNet exhibited similar performance and were superior to the other three semantic segmentation models. Notably, U-Net, which excels in medical imaging datasets, performed moderately on the landslide drone dataset. In fact, models like DeepLabV3+ and PSPNet [68] have adopted a degree of multi-scale optimization to enhance the model’s multi-scale target perception capabilities; SegFormer [69] also employed a Transformer architecture for more effective regional feature modeling. In contrast, U-Net has a simpler architecture and lacks sufficient multi-scale target extraction capabilities, which may be the main reason limiting its landslide recognition ability.

In order to visually analyze the performance enhancement in landslide recognition exhibited by DeepLab4LS compared to the DeepLabV3+ model, we selected a subset of images from the validation set for comparative analysis, as shown in Figure 12. It can be observed that DeepLab4LS shows a significant enhancement in landslide recognition performance over the DeepLabV3+ model. For landslides with distinct boundary information, both DeepLab4LS and DeepLabV3+ can roughly identify the extent of the landslides (as in rows 3, 5, and 8 of Figure 12); however, for those landslides with less distinct boundaries (as in rows 6 and 7 of Figure 12), DeepLabV3+ often struggles to produce reliable results. It is also noteworthy that even for landslides with distinct boundaries, the edges recognized by DeepLabV3+ tend to be coarse. In contrast, DeepLab4LS is able to delineate the true boundaries of the landslides with greater precision.

4.2. Ablation Experiments

In this study, we designed the DeepLab4LS model, which consists of Encoder1, Encoder2, Mixer and Decoder modules, and introduced the PE loss function to enhance the model’s ability to understand positive samples. Through a series of ablation experiments, we aim to demonstrate the contributions of these modules or the loss function. Specifically, we aim to demonstrate:

(1)

The contribution of the new combination of loss functions.

(2)

The contribution of the Mixer module.

(3)

The contribution of the dual-encoder architecture.

The design and results of the ablation experiments are presented in

Table 4. Ablation experiments based on the DeepLab4LS model, each experiment differing in module composition or loss function.

Base Model	Encoder1	Encoder2	Mixer	Loss Functions	mIoU (%)	mPA (%)	Accuracy (%)
DeepLab4LS (with MobileNetV2 as the topographic backbone)	✔	✔	✔	CE 2 + PE 3	76.0	85.3	92.3
	✔	✔	✔	CE	72.2	81.2	91.1
	✔	✔	✔	PE	75.6	85.0	92.2
	✔	✔		CE + PE	74.4	81.7	92.2
		✔		CE + PE	63.9	76.4	83.9
	✔ 1			CE + PE	75.0	84.0	92.1

¹ The DeepLab4LS model degrades to DeepLabV3+ model in this scenario (but with different loss functions). ² CE: Cross Entropy (loss). ³ PE: Positive Enhanced (loss).

, revealing significant insights. The loss function in this study is composed of the CE and PE terms, and the removal of either significantly diminishes the performance of the model. Both the Mixer and the dual-encoder play a crucial role in the model; removing the Mixer and Encoder2 results in a decrease of 1.6% and 1.0% in mIoU, respectively. The contribution of Encoder1 is even more pronounced; without it, the model’s mIoU drops by 12.1%, which is clearly unacceptable. Nevertheless, the ablation experiments have successfully demonstrated the vital contributions of the combined loss functions, the Mixer module, and the dual-encoder architecture to the model.

5. Discussion

5.1. Topographic Features in Landslide Segmentation

One of the major enhancements that DeepLab4LS offers over DeepLabV3+ is the incorporation of topographic images as part of the input. As indicated in Figure 12, the introduced topographic information aids in bolstering the model’s comprehension of landslide semantics, particularly when landslide boundaries are indistinct. This outcome is unsurprising; as a geological entity, analyzing the topographic characteristics of landslides is crucial for their effective differentiation. Indeed, many features of landslides, such as cracks, fissures, accumulation bodies, and sliding surfaces, are more pronounced in topographic images [70,71], allowing the model to learn these features easily from the topographic images.

However, despite the significant assistance that topographic features in images provide for landslide recognition, ablation experiments (Table 4) indicate that relying solely on topographic images for recognition yields less accurate results. This phenomenon could be related to various factors, including digital imaging, neural networks, and the inherent characteristics of landslides themselves. Here, we propose a possible explanation: features in optical images are “easier” for neural networks to learn than those in topographic images. Therefore, when both optical and topographic images are used, the model can initially obtain some “hints” of landslide features from the optical images, which then synergistically promotes a deeper learning of landslide features from both optical and topographic images. In contrast, when only topographic images are used, the model is immediately confronted with challenging landslide features to learn, resulting in suboptimal training results.

5.2. Mixer of DeepLab4LS Model

The ablation experiments (Table 4) have confirmed the significant role of the Mixer in the model. It is noteworthy that when employing the dual-encoder structure without the Mixer, the mIoU is 74.4%, which is even lower than the 75.0% obtained when using only Encoder1. This indicates that merely introducing Encoder2 does not aid in a better understanding of the landslide semantics, but rather may somewhat impede the model’s comprehension of the optical landslide semantics.

To gain a better understanding of the Mixer’s function, we visualized the feature maps at five key points in DeepLab4LS by averaging the values across channels, as shown in Figure 13. It is readily observed that in the feature maps F1 and F2 output by Encoder1, the extent of landslide and background can be roughly discerned. However, in the feature map F3 output by Encoder2, the extent information is not as clear. This actually corroborates our conjecture in Section 5.1. that the topographic features of landslides are more challenging to learn. Once the three feature maps are stacked into F4, the network possesses both optical and topographic semantics of the landslide, but the features within the landslide extent remain somewhat disordered. Feature map F5 is the output of F4 after Mixer processing. After this, feature map F5 is considerably smoother, with features within the landslide extent exhibiting high consistency. Finally, the Decoder captures such a feature distribution and outputs reliable segmentation results. In summary, the Mixer effectively facilitates the deep integration of optical and topographic semantics in the model.

5.3. Loss Function in Landslide Segmentation

In this study, we attempted to enhance the model’s comprehension of positive samples by introducing a novel loss function—PE loss. The results of the ablation experiments (Table 4) indicate that PE loss effectively achieved our goal. Although the standard loss function in this study was designed as a combination of CE loss and PE loss, the ablation results show that even when using PE loss alone, there is a significant performance enhancement over using CE loss alone. We can try to explain this phenomenon based on the formula of PE loss itself. Observing the fractional term of PE loss, the numerator represents the correctly predicted positive samples, while the denominator accounts for all incorrectly predicted samples. Compared to Soft IoU loss (which will be discussed in detail later), the denominator of PE loss lacks the correctly predicted positive samples, which means that PE loss will easily attain a small value when a large number of positive samples are successfully predicted and few positive and negative samples are mispredicted, thereby amplifying the penalty in the opposite scenario. In essence, through PE loss, we aim to instruct the model to focus its efforts on the current positive sample.

In fact, the design of PE loss in this study was inspired by another similar loss function—Soft IoU loss (SI loss) [72]. Compared to PE loss, SI loss includes correctly predicted positive samples in the denominator, making it “less aggressive” than PE loss. The classic form of SI loss is:

$$classic SI loss=1-\frac{1}{\left|C\right|}\frac{{\displaystyle \sum _{pixels}{y}_{\mathrm{true}}{y}_{\mathrm{pred}}}}{{\displaystyle \sum _{pixels}\left(y_{\mathrm{pred}}+y_{\mathrm{true}}-y_{\mathrm{true}}{y}_{\mathrm{pred}}\right)}},$$

(9)

We can also logarithmize the fractional term, resulting in:

$$logarithmic SI loss=1-\log \left(\frac{1}{\left|C\right|}\frac{{\displaystyle \sum _{pixels}{y}_{\mathrm{true}}{y}_{\mathrm{pred}}}}{{\displaystyle \sum _{pixels}\left(y_{\mathrm{pred}}+y_{\mathrm{true}}-y_{\mathrm{true}}{y}_{\mathrm{pred}}\right)}}\right),$$

(10)

Clearly, logarithmic SI loss makes SI loss more “aggressive” in a more direct manner. Here, the meanings of the terms in Equations (9)–(11) are the same as those in Equation (3).

We applied the above two forms of SI loss to the training process, and the results are shown in

Table 5. Segmentation results of DeepLab4LS under different loss functions.

Base Model	Loss Functions	mIoU (%)	mPA (%)	Precision (%)
DeepLab4LS (with MobileNetV2 as the topographic backbone)	CE 1	72.7	81.5	91.3
	CE + PE 2	76.0	85.3	92.3
	CE + dice	72.9	81.5	91.5
	CE + classic SI 3	74.0	84.3	91.4
	CE + logarithmic SI 3	72.2	82.5	90.8

¹ CE: Cross Entropy (loss). ² PE: Positive Enhanced (loss). ³ SI: Soft IoU (loss).

. Additionally, we also tried to introduce dice loss [73], which is often used to address the issue of imbalance in samples, to see if it could be profitable for the training. The dice loss is defined as follows:

$$dice loss=1-\frac{1}{\left|C\right|}\frac{2{\displaystyle \sum _{pixels}{y}_{\mathrm{true}}{y}_{\mathrm{pred}}}}{{\displaystyle \sum _{pixels}\left(y_{\mathrm{pred}}^2+y_{\mathrm{true}}^2\right)}}$$

(11)

Generally, the ranking of the loss functions in terms of performance gain for the model is as follows: PE loss (Equation (3)) > classic SI loss (Equation (9)) > dice loss (Equation (11)) > logarithmic SI loss (Equation (10)). PE loss, dice loss, and classic SI loss all improve the model’s ability to recognize landslides to some extent. However, logarithmic SI loss resulted in a negative optimization, which may be due to its “excessive” enhancement of penalty, causing the model to easily fall into local optima. In summary, our practice has proven that PE loss can effectively enhance the performance of landslide recognition models.

6. Conclusions

In this study, we designed and implemented the DeepLab4LS model, a novel semantic segmentation model for landslide recognition based on DeepLabV3+. We introduced a dual-encoder design and established a model architecture composed of Encoder1, Encoder2, Mixer, and Decoder modules. Within this framework, the model receives both optical and topographic images as inputs and achieves deep integration within the model itself. This endows the model with the ability to infer both optical and topographic semantics of landslides, aligning more closely with the approach human experts use to recognize landslides. Another significant optimization in this study is the introduction of a new loss function term—Positive Enhanced loss (PE loss)—to enhance the model’s comprehension of positive samples.

A series of comparison and ablation experiments were conducted on optical and topographic UAV datasets of the Baihetan reservoir. The results show that the DeepLab4LS model achieved an mIoU of 76.0%, outperforming DeepLabV3+’s mIoU of 70.5% and significantly surpassing other popular models such as SegFormer. These results are closely related to the introduction of the dual-encoder architecture and the Mixer module. Our experiments also found that optical and topographic images play a mutually reinforcing role, and that inputting only one type of image would lead to a decline in model performance. With only topographic images as input, the mIoU could drop by as much as 12.1%, which is completely unacceptable. The introduction of PE loss is also crucial for enhancing model performance, increasing the mIoU by approximately 3.8 percentage points. In fact, PE loss allows the model to focus more on improving the recognition quality of current positive samples, which is evidently beneficial for landslide recognition tasks against complex backgrounds.