Improved Shallow Landslide Susceptibility Prediction Based on Statistics and Ensemble Learning

Abstract

Rainfall-induced landslides bring great damage to human life in mountain areas. Landslide susceptibility assessment (LSA) as an essential step toward landslide prevention has attacked a considerate focus for years. However, defining a reliable or accurate susceptibility model remains a challenge although various methods have been applied. The main purpose of this paper is to explore a comprehensive model with high reliability, accuracy, and intelligibility in LSA by combing statistical methods and ensemble learning techniques. Miyun country in Beijing is selected as the study area. Firstly, the dataset containing 370 landslide locations inventories and 13 conditioning factors were collected and non-landslide samples were prepared by clustering analysis. Secondly, random forest (RF), gradient boosting decision tree (GBDT), and adaptive boosting decision tree (Ada-DT) were selected as base learners for the Stacking ensemble method, and these methods were evaluated using measures like area under the curve (AUC). Finally, the Gini index and frequent ratio (FR) were combined to analyze the major conditioning factors. The results indicated that the performance of the Stacking method was enhanced with an AUC value of 0.944 while the basic classifiers also performed well with 0.906, 0.910, and 0.917 for RF, GBDT, and Ada-DT, respectively. Regions with a distance to a stream less than 2000 m, a distance to a road less than 3000 m, and elevation less than 600 m were susceptible to the landslide hazard. The conclusion demonstrates that the performance of LSA desires enhancement and the reliability and intelligibility of a model can be improved by combining binary and multivariate statistical methods.

1. Introduction

Landslides are a common natural phenomenon and may cause unpredictable damage to human beings and property worldwide, especially in China where geohazards are enormously occurring and widely distributed [1]. Generally, damages can be decreased or mitigated by predicting the area prone to landslides [2,3]. Therefore, landslide susceptibility mapping (LSM), which predicts the spatial distribution of the likelihood of a landslide occurring, is significant and worthwhile for the reduction of hazards.

How to improve the quality of a model is always the focus of attention and discussed by researchers although related studies have been conducted on improving the predictive accuracy [4,5]. The effectiveness of LSM depends greatly on the models adopted [6], which can be roughly divided into knowledge-based and data-driven methods [7]. Conventional knowledge-based methods as a heuristic, are subjective and limited to be applied in small-scale areas. Conventional statistical methods, like logistic regression (LR) and principal component analysis, are popular due to their simplicity. Nevertheless, the mechanism of a landslide is complicated and usually involves multiple factors. Nevertheless, conventional statistical methods fail to deal with nonlinear problems [8,9]. Geographic information systems (GIS) and computing techniques are increasingly developing so that machine learning techniques (MLT) translated from statistical methods have achieved promising performance for LSM [10]. However, the inductive preference is inevitable for MLT and the performance of different MLT fluctuates as the data change. Besides, the performance of a model will be affected by the purity of the samples. Accordingly, controversy continues over which method is the best and which is feasible for further improvement [11]. A single classifier applied to form an ensemble is called the “base learner”. Base learners are not limited to homogeneous but also can be heterogeneous. Bagging and boosting are two of the most popular ensemble techniques of homogeneous and have been applied to LSM by some researchers [12]. While the stacking ensemble method, which is heterogeneous, has seldom been applied to LSM and needs more exploration [13].

Data-driven methods are binary classification processes and are sensitive to the quality of training data, which require a data set consisting of an equal amount of both disaster presence and absence observations in LSM [14]. In terms of landslide presence data, it is obtained from the landslide inventory, which was achieved through historical records, remote sensing technology, and extra field investigations. While absence samples were not available, they are usually selected randomly or subjectively from the “safe area” based on the acknowledgment and experience of the experts [15]. Nevertheless, it is controversial and difficult to implement especially for a large area. We could not identify the area with low susceptibility based on the historical records because these landslide-free areas may contain locations prone to landslides, which have not been recorded in the past. Unreliable sampling strategies may bring the noise to the data and eventually, a false assessment of the models. This study applies clustering analysis to improve the quality of samples and the performance of models [16]. Two initial landslide susceptibility maps are made by k-means clustering and Fuzzy c-means (FCM) and the reasonability of the initial maps are compared and finally determined the better one. Accordingly, the non-landslide samples are selected from the very low susceptibility area.

Previous studies have emphasized the importance of accuracy and regarded it as the only indicator for evaluating a model. However, emphasizing accuracy is not enough for the requirement of prevention and control. Communication between theory and practice can be improved through a better understanding of major variables. The bivariate statistical method as FR is commonly applied to explore the relationship between conditioning factors and the occurrence of landslides by calculating the FR values of factors in a certain interval [17]. While the bivariate statistical method fails to determine the relative importance among different factors and Gini index (the larger the value indicates the greater the contribution to the occurrence of landslides) makes up for it [18].

The current study aims to explore a model with high reliability, accuracy, and intelligibility on LSM. Three ensemble techniques were evaluated by 5-fold cross-validation according to the Receiver Operating Characteristic (ROC) curve and statistical indexes. RF, GBDT, and Ada-DT are selected as the candidate base learners of the stacking method and LR as the meta-learner. The stacking method is explored as a potential application to LSM and is compared to other ensemble methods. The purity of the samples is improved by selecting the non-landslide samples in a more reliable way. The Gini index and FR were combined to identify and analyze the major conditioning factors to improve intelligibility. Miyun country, China, where shallow landslides occurred frequently, was selected as the study area and a comparison of the ensemble methods above was made.

2. Materials

2.1. Study Area

Miyun country located in Beijing, China, extends from longitudes of 116°39′ E to 117°30′ E and latitudes of 40°13′ N to 40°47′ N (Figure 1). It has a population of more than 470,000 and occupies an area of about 2229.45 km², which is composed of mountainous areas (82.9%), cultivated land (8.3%), and reservoirs, roads, and villages (collectively 9.8%). The average annual precipitation is 663.1 mm (1981–2012) mainly concentrated in summer (76.4%) and it is a continental monsoon semi-arid climate.

The study area is part of the transition zone between the North China Plain and the Yanshan mountains, which leads to a series of large fold and fault structures. The faults are large in scale and widely distributed, mainly in the Northeast and north-south directions. The elevation ranges from 45 m to 1750 m above mean sea level with a slope angle between 10–45°. The strata are mainly composed of Archaean (Ar), Proterozoic (Pt), Mesozoic Jurassic (J), and Quaternary (Q). Three types of lithology are usually exposed in our investigation: gneiss from Middle Archean (ArXdgn), dolomites from Proterozoic (Pt22w), and siltstone from Mesozoic Jurassic (J2z). Magmatic intrusive rocks are widely distributed, accounting for nearly one-third of the total area and are exposed discontinuously in the northeast direction.

Road traffic is developed, and human activities are intensive in the study area, involving mining, reservoir, and power station projects. The disasters are various and frequent, mainly rain-induced landslides, which has affected the normal life of the local villagers.

2.2. Data Preparation

2.2.1. Landslide Inventory

The statistically based models follow a crucial assumption: future landslides have more chances to occur again in the places with the conditions which cause the landslides once and present [18,19]). Accordingly, the landslide inventory map as the initial source is essential and was depicted according to related records (from 1970–2010), field surveys (from 2016–2017) (Figure 2 and Figure 3), and Google Earth satellite images interpretation (May 2018) (Figure 4). Ultimately, 620 landslide locations were identified, including soil slides (370), rockslides (6), and falls (244) [18]. It is accepted that different type of landslides has a different mechanism of occurrence. Soil slides were only considered in our work and were represented as points shown in Figure 1. Landslides occurred during or after heavy rainfall. Based on field investigation, remote sensing interpretation and relevant records, the scale of landslides in the study area is generally small, accounting for about 80%. The area of landslides ranges from 3.6 km² to 300 m² while the depth of most landslides is less than 4 m, belonging to shallow landslides.

2.2.2. Choice of Mapping Units

The selection of mapping units should be determined in advance for LSM [20]. Another piece of literature discussed and compared the difference among mapping units, such as grid cells and slope units [21]. To better predict or identify the locations of landslides, slope units were applied in our work, which describes the topographic and geomorphic conditions of landslides integrally. Finally, the area was divided into 8736 slope units using the hydrologic analysis tool in ArcGIS and indispensable artificial corrections according to remote sensing images. Detailed division steps and discussion can be referred to in other literature [22].

2.2.3. Conditioning Factors

Factors responsible for a landslide are various and there is no consensus on the choice of number and types of factors. It is commonly accepted that landslide is controlled by topographical, geological, and triggering factors. However, data availability, reliability, and accuracy should be given priority [23] and finally, 13 conditioning factors were selected. Detailed information on conditioning factors is shown in

Table 1. Landslide conditioning factors in this study.

Category	Conditioning Factors	Type	Data Source	Values
Topographical	Elevation (m)	Continuous	SRTM	(1) <200; (2) 200–400; (3) 400–600; (4) 600–800;
				(5) >800
	Plan curvature	Continuous	SRTM	(1) <0; (2) 0–0.01; (3) 0.01–0.02; (4) 0.02–0.03;
				(5) >0.03
	Profile curvature	Continuous	SRTM	(1) <0; (2) 0–0.01; (3) 0.01–0.02; (4) 0.02–0.03;
				(5) >0.03
	Slope angle (°)	Continuous	SRTM	(1) <10; (2) 10–20; (3) 20–30; (4) >30
	TWI	Continuous	SRTM	(1) <6.5; (2) 6.5–7; (3) 7–7.5; (4) 7.5–8;
				(5) 8–8.5; (6) >8.5
	MED (m)	Continuous	SRTM	(1) <100; (2) 100–200; (3)200–300; (4) 300–400;
				(5) 400–500; (6) >500
	Slope aspect	Categorical	SRTM	(1) north; (2) northeast; (3) east; (4) southeast; (5) south; (6) southwest; (7) west; (8) northwest
Geological and Geomorphological	Distance to faults (m)	Continuous	Geological map	(1) <1000; (2) 1000–2000; (3) 2000–3000; (4)3000–4000; (5) >4000
	Distance to streams (m)	Continuous	DNRB	(1) <1000; (2) 1000–2000; (3) 2000–3000; (4)3000–4000; (5) >4000
	Lithology	Categorical	Geological map	(1) Gneiss; (2) Dolomites; (3) Siltstone (4) Granite;(5) Limestone; (6) Conglomerate
Triggering factors	Maximum 24 h rainfall (mm)	Continuous	BHM	(1) <270; (2) 270–280; (3) 280–290; (4) >290
	Maximum 7 days rainfall (mm)	Continuous	BHM	(1) <320; (2) 320–330; (3) 330–340; (4) >340
	Distance to roads (m)	Continuous	DNRB	(1) <1000; (2) 1000–2000; (3) 2000–3000; (4)3000–4000; (5) >4000

and Figure 5a–m. A brief description of each controlling factor is given below.

Topographic-related factors were derived from the DEM (Digital Elevation Model) with a resolution of 30 m (http://www.gscloud.cn, accessed on 4 April 2022) originally sourced from the Shuttle Radar Topography Mission (SRTM) data. Elevation affects slope instability and precipitation properties and was frequently applied to LSM [24,25]. Landslides are likely to occur as slopes become steep and vice versa [26]. Maximum elevation difference (MED) reflects the potential energy of a slope and was calculated in ArcGIS [27]. Topographic wetness index (TWI) and Curvature reflect topographic relief [28]. TWI was reclassified into six classes (Figure 5g) and the related algorithm is as follows:

$$\mathrm{TWI}=\ln \left(\frac{{\mathrm{A}}_{\mathrm{s}}}{\tan \mathsf{\beta }}\right)$$

(1)

where, A_s is the specific catchment area, $\mathsf{\beta }$ is the slop angle.

The plan curvature (Figure 5g) and profile curvature (Figure 5g) are both the most extensively used predisposing factors, which reflect the changes in terrain [29]. The slope aspect map was reclassified into eight classes according to the eight cardinal directions (Figure 5g).

Fault information (Figure 5i) was collected from a geological map of which the ratio was 1:50,000. Faults decrease the rock strength, which acts as potential weak planes in slopes. It was produced by the spatial distance analysis tool in ArcGIS. Similarly, the distance to roads (Figure 5h) and distance to rivers (Figure 5j) were both constructed based on the data from the Department of Natural Resources of Beijing (DNRB).

Shallow landslides are mainly caused by heavy or continuous rainfall [30]. Consequently, both the maximum 24 h rainfall (Figure 5l) and maximum seven days of rainfall (Figure 5m) were selected based on the data (1981–2000) from Beijing Hydrology Manual (BHM) using the kriging interpolation coordinated with elevation in ArcGIS and 11 precipitation stations nearby were taken as reference. Rainfall was regarded as the natural trigger while the distance to the road was the human factor.

Factors were reclassified into four to eight classes and the mean value was regarded as the statistic value of slope units.

3. Methods

3.1. Sampling Strategy

3.1.1. K-Means Clustering

K-means comes out to be a well know clustering method due to its efficiency and feasibility [31]. It is applied to divide n observations into k clusters, where each sample is allocated to the cluster based on the closest Euclidean distance, thus considered as the centroid of the cluster [32]. The procedure is then repeated until the change of the cluster seed from one stage to the next is negligible. The main equation involved in k-means is as follows:

$$\frac{\left|u_{n+1}-{ u}_n\right|}{u_{n+1}} \le \epsilon$$

(2)

where u_n+1 represents the sum of squares of distances from each point to the cluster center after the nth clustering; $\epsilon$ represents the precision value.

3.1.2. FCM Algorithm

The fuzzy c-means method is a soft clustering method developed by Dunn [33] and it is different from K-means (hard clustering). It has been widely used for statistical analysis of geological problems because of its flexibility and rationality [34]. Its core idea is to assign the objects to the corresponding clusters according to the degree of membership. The function of the FCM clustering is defined by the equation:

$$C_{i }={\displaystyle \sum }_{j=1}^n{\mu }_{ij}^m{x}_j/{\displaystyle \sum }_{j=1}^n{\mu }_{ij}^m$$

(3)

$$J={\displaystyle \sum }_{j=1}^N{\displaystyle \sum }_{i=1}^C{\mu }_{ij}^m{d}^2\left(X_j{,V}_i\right)$$

(4)

$${\mu }_{ij}=1/{\displaystyle \sum }_{k=1}^C{\left(\frac{d_{ij}}{d_{kj}}\right)}^{2/\left(m-1\right)}$$

(5)

where C_i represents the cluster centers, C represents the number of centers, u_ij represents the membership matrix; m represents the degree of fuzziness; J is the objective function and n is the number of objects in the database; d² is the Euclidean distance between the ith clustering center and the jth sample [35].

Two parameters as m and C are required to determine in advance. C is determined by the cluster validity function [36] and m is equal to 2 referred to in most applications in this study.

Machine learning methods need both positive and negative datasets. Three-hundred-seventy positive samples (that is, landslide locations) were set as “1” and the same number of negative samples with the value of “0”, which were selected based on the result of K-means and FCM in this study. As the purity of absent samples increases, it is more likely to reflect the characteristics of non-landslide areas. Accordingly, the critical value of the model results distinguishing landslides and non-landslides is 0.5.

3.1.3. Frequency Ratio

The equation for determining the FR value of a certain level of conditioning factor is defined below [16]:

$${\mathrm{FR}}_i=\frac{\frac{{\mathrm{landslide}\_\_\mathit{cell}}_{s_i}}{{\mathrm{landslide}\_\_\mathit{cell}}_{s_{\mathit{tot}}}}}{\frac{{\mathrm{total}\_\_\mathit{cell}}_{s_i}}{{\mathrm{total}\_\_\mathit{cell}}_{s_{\mathit{tot}}}}}$$

(6)

where i indicates the i-th class for each variable considered.

An FR_i greater than 1 manifest that there exists a close relationship between landslide occurring and variable class, and if the values are less than 1 then a weak correlation is reflected. Continuous variables are required to be reclassified into classes before application, as Table 1 showed.

3.2. Modeling Landslide Susceptibility

3.2.1. LR Model

LR establishes a non-linear probability function model, trying to find appropriate regression coefficients to express the correlation between the independent variable and the dependent variable [37]. The LR model is constructed as the equation below:

$$p=\frac{1}{{1+e}^{-y}}$$

(7)

where p is the probability of a landslide occurring; y is a linear combination function as Equation (7).

$${y=b}_0{+b}_1{x}_1{+b}_2{x}_2{+b}_3{x}_3{+\dots b}_n{x}_n$$

(8)

where b₀ is the constant value, and b₁, b₂, …, b_n refer to each significant input variable (x₁, x₂,..., x_n) causing the landslide.

The forward7 stepwise method was adopted to screen variables during LR modeling in SPSS software.

3.2.2. RF

RF belongs to a family of ensemble methods based on the decision tree and Bagging technique and it was first introduced by Breiman [17]. The bagging technique, which is also called bootstrap aggregation, is applied to selecting variables and samples randomly as the training data for modeling. Unused observations are applied to calculate the classification error. Consequently, there are two powerful ideas of RF: random feature selection and Bagging [38]. More details about RF can be found in Breiman [17]. RF was modeled in Python 3.7 using the scikit-learn package [39]. The number of trees (k) and the number of predictive variables (n) are required tuning before modeling [40].

3.2.3. GBDT

GBDT forms weak classifiers (DT) iteratively based on Gradient Boosting [41]. The parameter of the weak classifier defaults to the direction of the. The GBDT was applied in Python 3.7 using the GBDT class library of scikit-learn.

3.2.4. AdaBoost-DT

AdaBoost (known as adaptive boosting) is another boosting algorithm, which was invented by Freund and Schapire [42]. Unlike gradient boosting, AdaBoost assigns incorrectly classified samples with modified weights after each iteration. The final classifier is constructed by combining all weak classifiers. AdaBoost-DT is also applied in Python 3.7 using the AdaBoost class library of scikit-learn.

3.2.5. Gini Index

The split method tree-based classifiers adopt is the minimum principle of Gini and thus Gini index is applied to calculate the relative importance of conditioning factors. The relevant formula is as follows:

$$\mathrm{Gini}\left(T\right)=1-{{\displaystyle \sum }}_{j=1}^N{P}_j^2$$

(9)

where T expresses the training set, N is the number of categories, and P is the probability of a sample that is classified into the kth class.

3.2.6. Stacking

The stacking ensemble consists of base-classifiers and meta-classifier. Stacking takes the results predicted by the base-classifiers as the input attributes and the meta-classifier merges the different predictions into the final prediction. It is believed that stacking performs better than any basic classifiers [43]. Figure 6 shows the structure of the Stacking. The basic classifiers of Stacking were three ensemble learning machines that have been showing great performance in statistical analysis: RF, GBDT, and AdaBoost-DT. LR model was used as the combiner. To avoid over-fitting of the meta-classifier, the dataset is divided into two disjoint subsets: one for training base-classifiers and the other for testing. To train the meta-level classifier, 5-fold cross-validation is applied to construct the meta-levels for all combining methods.

3.3. Evaluating Model Performance

Models need a reliable evaluation and/or validation process [44]. The capacity of a model to classify was evaluated by a 5-fold cross-validation procedure, where the data is divided into five independent groups, one at a time for testing and the remaining four groups for training [45].

Accuracy, sensitivity, and specificity were three statistical indexes evaluating the performance [13]:

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(10)

$$\mathrm{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(11)

$$\mathrm{Specificity}=\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$$

(12)

where True Positive (TP) refers to the number of landslide samples with correct classification, True Negative (TN) refers to the number of non-landslide samples with correct classification, False Positive (FP) refers to the number of landslide samples with incorrect classification and False Negative (FN) refers to the number of non-landslide samples with incorrect classification.

AUC is a metric commonly used to assess the quality of the model and it varies from 0.5 to 1. The higher the AUC value shows the stronger the predictive ability [46].

Non-parametric models need to be optimized by tuning related hyperparameters before application [47]. The involved parameters for modeling utilized in this study were shown in

Table 2. The optimized parameters of methods utilized in this study.

Methods	Parameters
DT	Criterion = ‘gini’; max_features = None; max_depth = 20; min_samples_split = 2; min_samples_leaf = 1; max_leaf_nodes = None; class_weight = None
RF	n_estimators = 500; criterion = ‘gini’; max_depth = None; max_features = ‘sqrt’;
GBDT	n_estimators = 100; learning_rate = 0.1; max_depth = 2; verbose = 1; subsample = 0.7; max_leaf_nodes = None
AdaBoost-DT	base_estimator = None; n_estimators = 100; learning_rate = 1.0; algorithm = ‘SAMME.R’; random_state = None

and the flowchart of methods involved was shown in Figure 7.

4. Results and Verification

4.1. Non-Landslide Samples Selected by FCM and K-Means

LSM generated based on cluster analysis does not need to identify the positive and negative labels of the samples in advance. Based on the curve of the clustering effectiveness index Vcs (Figure 8), the preferred value is five. Consequently, the study area was reclassified into five areas based on the FR values, which were very low, low, moderate, high, and very high. The proportions of each area are: very low (15.97%), low (23.25%), moderate (19.29%), high (33.5%) and very high (8%). Among them, the very-low area accounted for 15.97% of the whole study area with only 3.24% of landslide locations and an FR value of 0.2. Besides, the high or very-high area accounted for 41.5% of the study area with more than 55% of landslide locations and the FR values were both greater than 1.

Similarly, the results constructed by K-means were shown in

Table 3. Frequency ratios of five susceptibility classes assessed with FCM and K-means.

Method	Class	Landslide Ratio (%)	Area Ratio (%)	FR
FCM	Very low	3.24	15.97	0.20
	Low	19.73	23.25	0.85
	Moderate	21.35	19.29	1.11
	High	40.00	33.50	1.19
	Very high	15.68	8.00	1.96
k-means	Very low	1.62	11.66	0.14
	Low	15.41	22.30	0.69
	Moderate	15.57	18.71	0.83
	High	48.11	39.16	1.22
	Very high	17.30	8.17	2.11

. The proportions of each area are: very low (11.66%), low (22.30%), moderate (18.71%), high (39.16%), and very high (8.17%). The very-low area accounted for only 1.62% of landslide locations with an FR value of 0.14. The high or very-high area accounted for 47.33% of the study area with more than 55% of landslide locations.

Compared to the results obtained by FCM, the area with low or very low class predicted by K-means occupied a smaller area (5.26%) while a bigger area (5.83) with high or very high class. The zoning maps should follow two rules: (1) the recorded landslides should appear in high-susceptibility areas as many as possible and (2) the high-susceptibility area should occupy a small proportion (Bui et al., 2012). Therefore, the results obtained by FCM were more reasonable. Selecting the non-landslide samples in a more reliable area is the main purpose and it means that the bigger the very-low class area, the easier the sampling will be. Meanwhile, 370 non-landslides samples were collected from the area with very-low susceptibility predicted by FCM.

4.2. Evaluation and Comparison of Different Models

To highlight the performance of the Stacking model, three basic classifiers were also applied for modeling. Analyses of the statistical measures using the training set were shown in

Table 4. Models’ performance using training dataset.

Metrics	RF	GBDT	Ada-DT	Stacking
TP (%)	82.46	84.88	81.29	91.22
TN (%)	76.80	87.67	86.44	92.20
FP (%)	17.54	15.12	18.71	8.78
FN (%)	23.2	12.37	13.56	7.80
Sensitivity (%)	79.93	86.97	85.66	91.89
Specificity (%)	83.16	85.67	82.26	91.78
Accuracy (%)	81.56	86.29	83.87	91.84

. The Stacking showed the best performance in terms of classifying landslides (sensitivity = 91.89%), followed by the GBDT model (sensitivity = 86.97%), the Ada-DT model (sensitivity = 85.66%) and RF model (sensitivity = 79.93%). In terms of the classification of non-landslides zones, Stacking model also performed best (specificity = 91.84%), followed by the GBDT model (specificity = 85.67%), the Ada-DT model (specificity = 82.26%) and the CART model (specificity = 83.16%). Besides, the Stacking model also had the highest accuracy (91.84%). It was noticed that the Stacking model achieved an AUC of 0.963, while RF was 0.920, GBDT was 0.957 and Ada-DT was 0.959 (

Table 5. ROC analysis of the four models using training data.

Models	AUC	Standard Error	95% Confidence Interval
RF	0.920	0.011	0.899–0.941
GBDT	0.957	0.008	0.942–0.973
Ada-DT	0.959	0.009	0.942–0.976
Stacking	0.963	0.006	0.950–0.975

). The standard errors were less than 0.05 and the probability estimation was negligible.

The predictive capacity needs to be evaluated using validation data. The results confirmed that the Stacking model perform the best as the values of sensitivity, specificity, accuracy and AUC were highest (

Table 6. Models’ performance using verification dataset.

Metrics	RF	GBDT	Ada-DT	Stacking
TP (%)	77.22	86.30	83.54	90.54
TN (%)	79.71	83.78	86.96	91.78
FP (%)	22.78	13.70	16.46	9.46
FN (%)	20.29	16.22	13.04	8.22
Sensitivity (%)	81.33	86.11	86.96	91.78
Specificity (%)	75.34	84.00	82.19	90.54
Accuracy (%)	78.38	85.03	85.13	91.16

and

Table 7. ROC analysis of the models using validating data.

Models	AUC	Standard Error	95% Confidence Interval
RF	0.906	0.027	0.853–0.959
GBDT	0.910	0.026	0.859–0.962
Ada-DT	0.917	0.021	0.877–0.958
Stacking	0.944	0.018	0.908–0.980

), which was 91.78%, 90.54%, 91.16% and 0.944, respectively, followed by Ada-DT (sensitivity = 86.96%, specificity = 82.19%, accuracy = 85.13% and AUC = 0.917), GBDT (sensitivity = 86.11%, specificity = 84.00%, accuracy = 85.03% and AUC = 0.910), and RF (sensitivity = 81.33%, specificity = 75.34%, accuracy = 78.38 and AUC = 0.906) (Figure 9).

The Stacking model exhibited the best both in training and validation data compared to the other three ensemble learning methods, which indicated ideal goodness-of-fit to modeling and generalization capability. The performance of GBDT and Ada-DT was similar, and the RF model performed the worst but was still satisfactory. The gaps in performance between training and validation data were not obvious among the models. Compared to the RF model, the application of the Stacking model enhanced the performance significantly and was regarded as the most suitable model for LSM in this study.

4.3. Application of Stacking Method for LSM

The above analysis proves that the Stacking method has superior ability in LSM compared with the other three models. Therefore, the probability of landslides occurring was calculated for all mapping units in the whole study area. The LSM was also constructed with five susceptible classes, which were very low (0–0.2), low (0.2–0.4), moderate (0.4–0.6), high (0.6–0.8), and very high (0.8–1) (Figure 10). Table 3 showed the distribution ratio of each level. The very low susceptible level occupied 26.04% of the area while low, moderate, high, and very high susceptible levels represented 15.31%, 15.46%, 32.45%, and 10.74%, respectively (Figure 11). It was noticed that LSM has the smallest area percentage in very high susceptibility levels while the largest is in high. Landslide locations were mostly distributed in the red areas. Meanwhile, most of the non-landslide samples screened by FCM clustering appeared in blue areas.

The high or very-high susceptibility areas are mainly distributed closed to streams or provincial highway, which runs through three townships including Fanzipai Town, Sihetang Town, and Fengjiayu Town in the study area. These areas are densely populated.

The landslide susceptibility class ranged from very low to very high around the Miyun reservoir. It is noteworthy that once a landslide occurs in this area, a series of disaster chains may be induced.

4.4. Analysis of Major Conditioning Factors

The stacking method performed the best in terms of accuracy, but the results had a poor analysis of the occurrence of landslides, which was confusing. Understanding the major factors that have a significant contribution to landslides occurring helps in the prevention and treatment of landslides. Based on the Gini index, ten major parameters were selected and normalized as shown in

Table 8. Conditioning factors assigned by the Ada-DT.

Method	DTS	DTR	Elevation	Slope Angel	TWI	Maximum 24 h Rainfall	Lithology	MED	Maximum 7 Days Rainfall	Profile Curvature
GBDT	0.37	0.34	0.16	0.04	0.03	0.02	0.01	0.01	0.01	0.01

, including DTS, DTR, elevation, slope angle, TWI, maximum 24 h rainfall, lithology, MED, maximum seven days of rainfall, and profile curvature. Among them, DTS, DTR, and elevation have a significant impact on the occurrence of landslides (Figure 12), the weight values of which were 0.37, 0.34, and 0.16, respectively. While the weight values of lithology, MED, maximum seven days rainfall, and profile curvature were close to 0.01, which had a limited contribution. The weight values of slope angle, TWI, and maximum 24 h rainfall were close to 0.04, 0.03, and 0.02, respectively.

Therefore, three conditioning factors, namely DTS, DTR, and elevation, were considered the major factors responsible for the landslide. Rivers are an important factor affecting the occurrence of landslides. On the slopes closer to the river, the toe of the slope is easily soaked by the river water, which reduces the strength of the rock and makes landslides more likely. Road development and construction are important tasks in mountainous area construction. However, unreasonable road excavation is a common human factor that induces geological disasters. Road construction often produces a large number of slopes, which destroy the stability of the slope and finally, lead to the occurrence of landslides.

The relationship between the major factors and landslides was further explored by calculating the FRi of each parameter (

Table 9. Spatial relationship between landslide conditioning factors and landslides using frequency ratio.

Conditioning Factor	Zone	Landslide (%)	Non-Landslide (%)	FR
DTS(m)	<1000	46.99%	0.95%	49.30
	1000–2000	24.43%	0.14%	173.29
	2000–3000	14.33%	6.63%	2.16
	3000–4000	5.33%	15.72%	0.34
	>4000	8.91%	76.69%	0.12
DTR(m)	<1000	56.06%	7.13%	7.87
	1000–2000	23.02%	7.13%	3.23
	2000–3000	15.59%	9.29%	1.68
	3000–4000	3.95%	11.51%	0.34
	>4000	1.37%	66.79%	0.02
Elevation(m)	<200	4.36%	2.08%	2.09
	200–400	53.76%	12.29%	4.37
	300–600	30.36%	23.70%	1.28
	400–800	10.06%	34.52%	0.29
	>800	1.46%	27.41%	0.05

). As for DTS, the percentages of landslide area of the first two classes (<1000 m and 1000–2000 m) were 46.99% and 24.43% with the FR values of 49.3 and 173.29, accounting for more than 70% of the landslides area. Similarly, DTS showed a positive correlation in the first three classes (<1000 m, 1000–2000 m, and 2000–3000 m) with FR values greater than 1. Regarding elevation also a positive correlation in the first three classes (<200m, 200–400 m, and 400–600 m) with values of FR gradually decreasing with altitude and a negative relationship in the last two classes (>600 m).

The selection and analysis of major factors by combining basic machine learning and bivariate methods made up for the defects of stacking, thereby ensuring the integrity of geological hazard assessment.

5. Discussion

5.1. Ensuring the Reliability of Models

5.1.1. Internal and External Cross-Validation

The basic classifiers used in our work have several hyperparameters that control the behavior and performance. In some cases, reasonable “guesses” are available (e.g., n tree = 500 in RF), in other cases classifiers are very sensitive to the parameters, which means that default hyperparameter settings fail to guarantee optimal performance of machine-learning techniques. Therefore, hyperparameters need to be tuned before application and inner cross-validation should be used for this [48].

On the other hand, external cross-validation was also essential. One can find an “excellent model” using the method “Leave-One-Out” because of the randomness in the sampling scheme, the results of which are unconvincing. Only by implementing a more rigorous k-fold (or other types) cross-validation scheme can one infer the actual capacity of a model to learn the functional relationships between landslides and causative factors as well as the variability that the models and the susceptibility estimates exhibit [45,48].

While various machine-learning algorithms have been recognized in recent years due to their powerful capabilities of data processing and generalization, there are several practical challenges related to bias-reduced assessment of a model’s predictive power and some researchers often ignore them, which leads to an unreliable or uncertain result. Single hold-out model performance measures were popular [49]. However, statistically based landslide susceptibility models desire a more credible validation and assessment before generalization.

5.1.2. The Selection of Non-Landslide Samples

A complete disaster inventory map is emphasized in a multitude of studies, which consists of the locations and number of a certain disaster [9]. The quality of landslide presence samples is more convincing compared to that of landslide absence because non-landslide samples are selected randomly or subjectively although quite a few methods or principles will be adopted. Seldom do studies consider or discuss the noise and influence of the absence of data bring to data-driven models [50]. Non-landslide points need to be selected from low-prone areas as far as possible, which is arduous to implement by selecting randomly. Clustering analysis help solve the problem by combining with the bivariate methods. FR was calculated to judge the area with low susceptibility based on the results of FCM and K-means in this study and the non-landslide samples were generated from it, which improved the quality of non-landslide records and the performance of models logically.

5.2. Increasing the Accuracy of LSM

How to achieve an accurate landslide susceptibility zoning map is always a hot topic and the main concern of researchers. However, determining the most suitable model is challenging because the performance varies according to the study area and methods applied. Actually, related studies have applied various methods and compared their performance based on the value of AUC to obtain the best method for a given region [51,52]. Yet, it is controversial that we claim a model to be better than the other according to the decimal places down the line (AUC). Therefore, it is necessary to explore new methods for significant improvement and ensemble techniques are considered in our study, which have also been proven to be an excellent solution [53,54,55]. A detailed comparison among three ensemble techniques in LSM, namely bagging, boosting, and stacking was implemented. Bagging and boosting are two algorithms commonly used in LSM while stacking have rarely been applied. The results proved that the ensemble of the GBDT-Adaboost-DT-RF-LR had the ability to enhance the predictive performance and the improvement was obvious. This enhancement originates from reducing both bias and variance and avoiding over-fitting problems [56,57]. It is believed that the stacking technique and its comparison will guarantee a better result for further studies [58].

5.3. Maintain the Integrity of Geological Hazard Assessment

An optimal model should not only focus on accuracy, especially for geological hazard assessment [50]. An outstanding model should also require communication skills, that is, make it easy for researchers to understand, accept and apply, especially for natural disasters [59]. The capacity of communicating model behavior is another valuable quality for LSA, which is arduously achieved by machine learning methods because of the “black box” nature. Stacking performed the best in terms of accuracy while it had a low capacity for recognizing the importance of the variables. Gini index and FR were combined to determine the major conditioning factors and analyze the individual landslide-related factors in each interval and the relative importance among them, which improved the readability of the stacking model.

6. Conclusions

LSM is the basis of supplementary analyses, such as land use and hazard prevention. Meanwhile, this field of geomorphology has become an empty shell with no research question on whether a model can be evaluated in an unreliable sampling strategy and focus only on accuracy. Therefore, a more reliable and accurate landslide susceptibility map is urgently needed through further comparison and application of different methods. In the present study, three ensemble learning machines were compared in terms of the performance of LSM in Miyun County, Beijing, China. Non-landslide samples were determined in a more reliable way with the use of FCM and K-means clustering. Statistical indexes and AUC were combined to assess the accuracy performance of the models. The major conditioning factors were determined and analyzed based on the Gini index and FR. The following conclusions can be drawn from the present study:

• The performance of different ensemble techniques varies, but achieved satisfactory results as a whole. Stacking was considered the most suitable model with obvious improvement in terms of accuracy compared to the basic classifiers.
• The combination of the bivariate statistical method and Gini index helps better explore the major conditioning factors and improve the integrity of ensemble techniques.
• The non-landslide samples selected by FCM are more representative and improved the quality of samples. Overall, improvement of sample quality and selection of advanced methods help improve the practicability of LSM.