A Comparative Assessment of Machine Learning Models for Landslide Susceptibility Mapping in the Rugged Terrain of Northern Pakistan

Abstract

This study investigated the performances of different techniques, including random forest (RF), support vector machine (SVM), maximum entropy (maxENT), gradient-boosting machine (GBM), and logistic regression (LR), for landslide susceptibility mapping (LSM) in the rugged terrain of northern Pakistan. Initially, a landslide inventory of 200 samples was produced along with an additional 200 samples indicating nonlandslide areas and divided into training (70%) and validation (30%) groups using a stratified loop-based random sampling approach. Then, a geospatial database of 12 possible landslide influencing factors (LIFs) was generated, including elevation, slope, aspect, topographic wetness index (TWI), topographic position index (TPI), distance to drainage, distance to fault, distance to road, normalized difference vegetation index (NDVI), rainfall, land cover/land use (LCLU), and a geological map of the study area. None of the LIFs were redundant for the modeling, as indicated by the multicollinearity test (tolerance > 0.1) and information gain ratio (IGR > 0). We extended the evaluation measures of each algorithm from area-under-the-curve (AUC) analysis to the calculation of performance overall (POA) with the help of precision, recall, F1 score, accuracy (ACC), and Matthew’s correlation coefficient (MCC). The results showed that the SVM was the most promising model (AUC = 0.969, POA = 2669) for the LSM, followed by RF (AUC = 0.967, POA = 2656), GBM (AUC = 0.967, POA = 2623), maxENT (AUC = 0.872, POA = 1761), and LR (AUC = 0.836, POA = 1299). It is important to note that the SVM, RF, and GBM were the top performers, with almost similar accuracy. Thus, each of these could be equally effective for LSM and can be used for risk reduction and mitigation measures in the rugged terrain of Pakistan and other regions with similar topography.

1. Introduction

Landslides, in comparison to other natural disasters, are one of the most fatal geological hazards. Under the influence of gravity, these events occur when internal resistive forces of a mass of rock, debris, or earth weaken the slope stability [1]. Factors that influence landslides often include intensive rainfall, volcanic eruption, land use/land cover (LULC), earthquakes, geological evolution, irrigation, and climatic variations [2,3]. Brabb [4] suggested that losses due to landslides could be controlled and minimized by 90% if proper attention is given to investigating their trigger factors and identifying high-risk areas. Guzzetti et al. [5] found that only 1% of all the Earth’s continents have been evaluated for landslide hazards, out of which 17% are attributed to landslides [6].

Researchers have proposed various landslide susceptibility mapping (LSM) approaches using satellite remote sensing (SRS) and geographic information systems (GIS) to identify landslide-prone areas [7,8,9,10,11] by analyzing and evaluating the probability of landslide occurrences [5]. These mapping approaches are divided into two categories, i.e., qualitative and quantitative [1,12]. The qualitative approaches mostly rely on the level of an expert’s knowledge, whereas the quantitative methods are based on probabilistic/statistical theories or models [13]. Both approaches have their limitations, for instance, the misinterpretation of the expert may lower the accuracy in the qualitative methods, whereas the use of poor, low-precision, and inaccurate data may decrease the accuracy in the quantitative methods [14]. Based on historic events, these prediction and mapping approaches can identify and locate the probability of occurrences of such events in a particular region with similar environmental, geophysical, geological, and other related conditions [15]. Although research on LSM is not capable of stopping landslide hazards totally due to its complex nature, highly precise LSM can help in reducing the impacts of this hazard by developing effective mitigation policies and measures [16,17]. In order to generate high-precision susceptibility maps, machine learning (ML) and data-mining approaches have attracted a number of researchers, particularly since 2005, and the development of ML-based quantitative methods has proven to provide accurate and reliable results of landslide susceptibility [18]. Based on the assumption of certain geological, geotechnical, and climatic conditions exhibiting similar patterns of landslide [19], these methods are considered as most effective in solving the nonlinear nature of geospatial problems.

Numerous studies have discussed the advantages of different ML methods including logistic regression (LR) [20], artificial neural network (ANN) [21], support vector machine (SVM) [10,21,22], decision trees (DT) [22], maximum entropy (maxENT) [11], random forest (RF) [23,24,25,26,27], and boosted trees (BT) [28]. A comprehensive review of the architecture and working mechanisms of popular ML methods can be found in Merghadi et al. [29]. Several efforts have been made to implement, investigate and compare these ML methods in different geographical settings [10,15,22,23,24,25,30,31]. For example, Merghadi et al. [32] investigated and compared the performances of RF, gradient-boosting machine (GBM), SVM, neural network (NNET), and LR in Mila Basin, Algeria. The authors of this study found that the GBM and RF methods provided good results, with areas under the curve (AUCs) of 0.897 and 0.895, respectively. On the other hand, a similar assessment was conducted by Wang et al. [33] in Shexian County, China, using the RF, SVM, GBM, multilayer perceptron (MLP), and LR models and found that RF and SVM provided the results with the best AUCs of 0.821 and 0.803, respectively. Recently, Qing et al. [34] implemented multiple ML models including generalized linear models (GLM), SVM, Naïve Bayes (NB), and other tree-based models to study the susceptibility of a debris flow along the China–Pakistan Karakoram Highway. The authors adopted different modeling strategies based on the watershed and catchment boundaries in the peripheries of the highway and found that the SVM performed best (AUC = 0.96). Ali et al. [35] assessed the prediction accuracies and performances of NB and RF by comparing their results with a fuzzy decision-making trial and evaluation laboratory approach combined with the analytic network process approach (FDEMATEL–ANP) in Kysuca river basin, Slovakia. RF was found to be an optimal model with an AUC value of 0.954 in the region. As a conclusion to the studies published so far, the accuracies of ML algorithms vary under different conditions of topography, geological setting, and climate [20]. Regardless of the number of ML methods, there is ‘no rule of thumb’ and ‘no free lunch’ [36] regarding which method or technique is the most suitable for landslide susceptibility due to the high level of uncertainty behind the processes and the varying topographical and climatic conditions of areas [37]. Therefore, it is necessary to study landslide dynamics and proneness by testing these algorithms for different site conditions for effective risk management and planning.

In the current study, we adopted five ML algorithms, RF, SVM, maxENT, GBM, and LR, to study the landslide susceptibility at the regional level in deep canyons and rugged terrain of northern Pakistan, where slopes vary up to 89 degrees. The selection of these five ML models was based on several reasons. First, these ML models constitute limited studies at a regional level and two of them (i.e., GBM and maxENT) have never been explored for LSM at this scale. Second, unlike other ML models, these models do not need a large amount of distributed sample data. Third, considering the working mechanisms of these models, it is expected that these models are suitable to achieve a high accuracy of landslide modeling at this scale. The diverse and sudden variation in the slopes and the presence of active thrust and strike slip faults in the area gives rise to frequent landslide events, especially after the October 2005 earthquake [38], and no significant ML-based approaches have been conducted in this region for LSM. Therefore, this study is the first of its kind in the area with the main objectives of investigating and highlighting the important influencing factors and assessing the performances of ML algorithms. Therefore, it is believed that this research study could provide a significant contribution to the scientific community which can be further used to delineate, identify, and understand the landslide risk and hotspots for planning and decision making in this hazardous region and other regions with similar topography.

2. Materials and Methods

In this research study, five ML techniques (RF, SVM, maxENT, GBM, and LR) were implemented to generate landslide susceptibility maps and to evaluate their performances in the region. The overall adopted methodology in this research comprised five main steps: (i) preparation of landslide inventory; (ii) preparation of landslide influencing factors; (iii) evaluation of the selected landslide influencing factors; (iv) tuning, training, and modeling through the selected models using training data; and (v) performance evaluation of the modeled results using validation data. A workflow of this methodology is summarized in Figure 1 and a step-by-step discussion is elaborated in later sections. The data were prepared in ArcMap, whereas modeling and evaluation were performed in R v3.6.3 (The R Foundation for Statistical Computing, Vienna, Austria).

2.1. Description of the Study Area

The northern areas of Pakistan are known for various natural hazards such as earthquakes, landslides, and floods [39]. One of the worst earthquakes in October 2005 (Mw = 7.6) left behind more than 80,000 deaths and a huge loss to infrastructure [40] and gave rise to slope failure events. This single event destabilized a large number of slopes in this part of the country. The central part of the study area under investigation constitutes the Hazara Division of the Khyber Pakhtunkhwa province of Pakistan, including part of Azad Jammu and Kashmir (AJK) in the east, Gilgit in the north, and Islamabad (the capital city of Pakistan) in the south. Geographically, the study area extends between 72°30′30.16″ E, 33°42′48.92″ N and 74°11′46.64″ E, 35°51′28.29″ N with an elevation range from 211 to 8558 m (Figure 2).

With an area of more than 34,000 sq·km, the study area covers several mountain hills and tourist places with a diverse variety of land features including rivers, major faults, snow-covered peaks, and sparse to medium-dense forests [41]. Geologically, the formation of the region ranges from Precambrian to Quaternary, which includes metasediments and metamorphic and other consolidated materials. The area is comprised of two main zones, i.e., the Harman seismic zone (HSZ) and the Indus Kohistan seismic zone (IKSZ). The study area is comprised of four main units, i.e., (i) the Jijal complex, (ii) Kamila amphibolite with para and ortho amphibolite plutonic rocks, (iii) Kohistan batholith, and (iv) metavolcanic/metasedimentary groups [42]. The major thrust faults in the study area include the main mantle thrust (MMT) and the main Karakorum thrust (MKT). The minor faults such as the Kohistan fault are in depositional contact, i.e., contact separating the Higher Himalayan Crystalline (HHC) with two faults making cross-sectional lines [43]. The study area is part of the upper Indus basin and feeds two of the major dams in Pakistan, i.e., the Tarbela and Mangla Dams. The main highway in the study area, Karakoram Highway (KKH), which is part of the China–Pakistan Economic Corridor (CPEC), mostly follows the Indus River and has been facing a severe problem of a large number of unstable slopes, degradation, and landslides due to a variety of factors such as steep slopes, geology, and rainfall.

The study area can be broadly divided into two climate zones, i.e., the southern zone and the northern zone. Geographically, the southern zone extends from Islamabad/Haripur up to the Dasu area and exhibits a high amount (approximately 1000 mm/year) of rainfall, with maximum rainfall during monsoon season. The northern zone, which extends from Dasu to Chilas and Gilgit, experiences lower rainfall (approximately 220 mm/year) as compared to the southern zone. One of the reasons for this sharp change in the climatic condition is the sudden change in the topography and terrain orientation in the area where the Indus valley turns into sharp bends. The changing patterns in terms of average rainfall and temperature (minimum and maximum) are shown in Figure 3.

2.2. Data Acquisition and Data Preparation

A Landsat-8 OLI image on 27 October 2017 and a Shuttle Radar Topographic Mission (SRTM) Digital Elevation Model (DEM) were obtained from United States Geological Survey (USGS) Earth Resources Observation and Science (EROS) data center. The data were preprocessed to develop a land-cover/land-use (LCLU) map; normalized difference vegetation index (NDVI); and subsequent topographic parameters such as slope, aspect, topographic wetness index (TWI), and topographic position index (TPI). Geological layers and fault lines were extracted from the maps published by the Geological Survey of Pakistan in 1993 and 1982, which were further modified from [44]. Other toponyms, such as catchments, roads, and drainages, were extracted from topographic sheets developed by the Survey of Pakistan (SoP). Due to limited ground weather stations in the area, time-averaged monthly precipitation data product (GPM_3IMERGM) of Integrated MultiSatellite Retrievals for the Global Precipitation Measurement (GPM) mission (IMERG) for the year 2017 was processed to generate the annual precipitation layer.

2.3. Preparation of Landslide Inventory

In prediction and susceptibility modeling, it is typically assumed that an event that occurred in the past can occur again in the future at the same place or other places under similar conditions. Therefore, a landslide inventory is imperative for assessing the link between past events and landslide influencing factors. However, the records of such events (i.e., landslide inventories) are often limited to events that occurred at large scales in accessible areas, lacking the details of small-scale events in inaccessible regions [5,9]. To overcome this limitation of the inventory in the study area, a database (inventory) of past landslides, including rockfall, topples, slides, and debris flow events, was developed by integrating the historic records from Frontier Works Organization (FWO), Geological Survey of Pakistan, and from the databases published in [45,46,47]. Google Earth (GE) images were used to validate geospatial locations and hotspots of these events. Figure 4 represents an example of the landslide event that occurred near the Dasu area, which can be easily interpreted on the GE image shown in Figure 4b.

With an inventory of 200 landslide events, an equal number of hotspots for nonlandslides were also generated for modeling. Considering the geographic locations of the landslide events, the 200 nonlandslide locations were randomly selected, particularly in the areas with dense forest, water bodies, peaks with permanent snow, and settlements in relatively plainer areas (slope angle < 3°), as implemented by [35]. Thus, a total of 400 samples were used to train and validate the landslide susceptibility maps. These samples were divided into training (70%) and validation (30%) data sets using a stratified loop-based random sampling technique. Of the data, 70% (280 samples) were used for training and model fitting and the remaining 30% (120 samples) of the data were used to validate the results of the models. Figure 2 shows the spatial distribution of these training and validation samples.

2.4. Selection of Landslide Influencing Factors (LIFs)

The accuracy of susceptibility maps is highly affected by LIFs. There are no universal criteria for selecting conditional factors due to the complex nature of the geoenvironment in different areas [25]. The selection of these LIFs largely depends on the climatic, geological, and topographical condition of the area; therefore, considering the topography, LCLU, and geological conditions of the study area, a total of 12 LIFs (

Table 1. Characteristic details of LIFs.

S. No.	Influencing Factor	Input Data	Description
1	Elevation	SRTM DEM	Digital elevation of the terrain surface. Values vary between 279 and 5934 m (Figure 5a).
2	Slope	SRTM DEM	A most crucial parameter that has a direct influence on slope failure and susceptibility. Values are up to 89 degrees (Figure 5b).
3	Aspect	SRTM DEM	The exposure of the slope to conditions such as sunlight, temperature, and winds. An important causative factor to susceptibility (Figure 5c).
4	TWI	SRTM DEM	TWI [48] is an index used to quantify topographic control of hydrological processes. Values vary from −3.7 to 16.43 (Figure 5d).
5	TPI	SRTM DEM	TPI [48] is an index that reflects the morphology of the topography (Figure 5e).
6	Distance to drainage	Topographic Sheet	Rivers and natural streams play an important role in slope failure due to the accumulation of water in the surrounding surface and subsurface. The proximity layer was generated based on Euclidean distance [49]. Values vary from 0 to 50,000 m (Figure 5f).
7	Distance to fault	Geological Map	The strength of rocks/soil decreases with the presence of faults and lineaments. The fault lines were extracted from input datasets and the proximity layer was calculated based on Euclidean distance [49]. Values vary from 0 to 7250 m (Figure 5g).
8	Distance to road	Topographic Sheet	To evaluate the effects of road engineering, this proximity layer was calculated based on Euclidean distance [49]. Values vary from 0 to 30,000 m (Figure 5h).
9	NDVI	Landsat-8 OLI	NDVI [50] illustrates the density and spread of vegetation in contrast with the non-vegetated land. It is an important factor that exploits the relationship between landslide occurrence and vegetation cover density. Values range between −1 and +1 (Figure 5i).
10	Rainfall	Topographic Rainfall Mission (TRMM)/GPM	Monthly averaged data product for the year 2017 was downloaded and used for the assessment with other factors. As discussed in Section 2.1 on the study area, the two climatic zones (north and south) can easily be identified in this rainfall map (Figure 5j).
11	LCLU	Landsat-8 OLI	The object-based image analysis (OBIA) technique [51] was adopted to extract 8 LCLU classes with an accuracy of 86% (Figure 5k).
12	Geological layer	Geological Map	Fourteen different units of geology were identified. The distribution and details of these units can be found in Figure 5l.

, Figure 5) including elevation, slope, aspect, TWI, TPI, distance to drainage, distance to fault, distance to road, NDVI, rainfall, LCLU, and geological layers were prepared. All the influencing factors were resampled at a standard pixel size of 30 m. The details of the chosen influencing factors are discussed in Table 1 and their spatial patterns and distribution are presented in Figure 5.

2.5. Evaluation of Landslide Influencing Factors

The accuracy of the model output strongly depends on the quality of influencing factors, which require prior knowledge of the terrain. Adding a large number of influencing factors into the model does not always ensure the reliability and accuracy of the modeled results and can introduce abnormalities in modeling processes. Therefore, it is necessary to evaluate the incorporated factors before using them for susceptibility modeling. For this purpose, in this study, all 12 selected influencing factors were scanned through the two tests, i.e., information gain ratio (IGR) [52] and multicollinearity [53]. The IGR test, as provided in Equation (1), is a well-known feature selection method in ML, which is used to assess the contribution of factors, whereas the multicollinearity test was performed to evaluate the interassociation (the linear relationship between two or more factors) [54].

$$\mathrm{IGR}\left({\mathrm{E}}_{\mathrm{x},\mathrm{a}}\right)=\frac{\mathrm{IG}}{\mathrm{IV}}$$

(1)

where information gain is IG and the intrinsic information is IV. Higher values of IGR represent the higher contribution of a particular factor for modeling and vice versa. For the multicollinearity test, variance inflation factor (VIF) and tolerance (TOL) were calculated using Equation (2).

$${\mathrm{VIF}}_{\mathrm{i} }=\frac{1}{1-{\mathrm{R}}_{\mathrm{i}}^2}=\frac{1}{\mathrm{TOL}} \mathrm{for} \mathrm{i}=1,2,\dots ,\mathrm{k}$$

(2)

where ${\mathrm{R}}_{\mathrm{i}}^2$ is the coefficient of determination of the explanatory variables. A VIF value greater than 10 or a TOL value of less than 0.1 is considered to have a serious multicollinearity problem [55]. Figure 6 shows the obtained IGR, TOL, and VIF values of each influencing factor and it is found that none of the factors exhibit a collinearity problem. To calculate VIF and TOL, we used ‘car’ and ‘VIF’ packages, whereas for the IGR test, ‘FSelector’ package was used.

2.6. Machine Learning Algorithms and Susceptibility Modeling

ML has proven to be the most effective approach which incorporates the best use of past events to map and model susceptibility with higher accuracies. In this study, the selected ML models were carefully tuned and trained to generate landslide susceptibility maps of the area with higher accuracy. The processing mechanisms and the framework of these selected models are elaborated in detail in the following subsections.

2.6.1. Random Forest (RF)

RF is an ensemble learning method that works on the principle of generating a large number of random trees to minimize the risk of overfitting [56]. RF does not require rescaling and resampling the data, as this model can handle the missing values very easily by building a DT. A DT, being a binary model, is constructed by dividing input data into an increasing number of homogenous nodes. The number of trees (ntree) and number of variables used to split each node (mtry) are the two main parameters used to tune and train the RF models. Considering the size and type of the sample data, the optimal values of these parameters can be found with the help of different methods [57] to achieve higher performance in modeling. In this study, a grid-based search method [58], a widely accepted approach for parametrization, was utilized for tuning the models. The modeling was performed through the ‘caret’ and ‘randomForest’ packages in R.

2.6.2. Support Vector Machine (SVM)

The SVM is considered as one of the best-performing algorithms and uses different kernels to quantify the similarity of two observations in feature space. Typical examples of the kernels used in SVM are linear (LN), polynomial (PL), radial-based function (RBF), and sigmoid (SIG). The performances of using these kernels highly affect the modeling process and its results [7]. Considering the spread and complexity of the sample data, the RBF kernel was adopted in this study. The RBF kernel in SVM is influenced by hyperplane parameters responsible for the smoothness of the decision boundary in various ways [59]. To increase the efficiency of the SVM, these parameters were first evaluated using 10-fold repeated cross-validation for tuning and training process. We used the ‘caret’ and ‘e1071’ R packages for the SVM modeling.

2.6.3. Maximum Entropy (maxENT)

Maximum entropy has been successfully applied to study species distribution and related research [60]. The Bayesian approach of maxENT modeling analyzes the available data for predicting distribution based on responses from influencing factors without relying on the sample size. Maximum entropy is a statistical learning approach that works on the principle of entropy and attempts to fit the model based on given data. Therefore, it also requires parametrization like other ML algorithms to avoid overfitting problems. The parametrization is done before generating the final susceptibility map. Other than the number of background samples, the parameterization and model tuning mainly involve a tradeoff factor known as the regularization parameter [61]. After several iterations with the help of already defined training and validation datasets, the best fit of the model was found with the regularization parameter of 1.0 and background samples of 8000 in this study. We used the ‘dismo’ package for modeling through the maxENT.

2.6.4. Gradient-boosting Machine (GBM)

This powerful ensemble learning model uses a combination of different multiple trees [28]. In recent years, GBM has proven to be more successful in different domains and is being considered as a leading method for prediction modeling [24,29]. As compared to the RF technique, this boosted tree model builds successive trees in which each tree reduces errors from the previous tree to improve the performance of the model in an iterative manner [62]. Among the different parameters, number of trees, shrinkage, and interaction depth are the most important in this model. Number of trees is required to minimize the loss function in the model, learning rate (shrinkage) is required to control the complexity and contribution of trees in the model, whereas interaction depth defines the number of splits in each tree [63]. To find the best fit of the model, a grid-based search with 10-fold repeated cross-validation approach was adopted to find the optimal values of these parameters used in training and modeling processes. For modeling through GBM, we used the caret’ and ‘gbm’ R packages.

2.6.5. Logistic Regression (LR)

LR is one of the most widely used approaches and has been very famous for providing better results with limited computer resources [57]. This GLM works on the principle of linear regression theory and can be an effective tool to solve probabilistic problems with the help of the logitic function [64]. This logistic function defines a relationship between sample data, their probabilities, and their dependencies on the influencing factors, as provided in Equation (3).

$$\mathrm{P}\left(\mathrm{x}\right)=\frac{{\mathrm{e}}^{\mathrm{y}}}{1+{\mathrm{e}}^{\mathrm{y}}}$$

(3)

where ‘$\mathrm{P}\left(\mathrm{x}\right)$’ is the occurrence probability and ‘y’ corresponds to the fitting function with the multiple influencing factors, which can be understood with the help of the expression presented in Equation (4).

$$\mathrm{y}={\mathsf{\beta }}_0+{\mathsf{\beta }}_1{\mathrm{x}}_1+{\mathsf{\beta }}_2{\mathrm{x}}_2+\dots +{\mathsf{\beta }}_{\mathrm{n}}{\mathrm{x}}_{\mathrm{n}}$$

(4)

where ${\mathsf{\beta }}_0,{ \mathsf{\beta }}_1,$ and ${\mathrm{x}}_{\mathrm{n}}$ are model intercept, coefficient of regression, and influencing factors, respectively. The LR model is known for its ability to find the optimal fitting functions among training samples and LIFs; therefore, to achieve accuracy through this model, the incorporated LIFs must not attain any collinearity issue between them. To achieve LSM results through LR, we used the ‘caret ‘and ‘nnet’ R packages.

The details of the model tuning and parametrization processes for each ML algorithm are given in the Supplementary Materials (Figures S1–S5).

2.7. Performance Evaluation of Models

Model validation and performance evaluation are the main steps in ML [30]. Assessment using training data reflects the level of model fitness, whereas evaluation using validation data is an unbiased assessment of the model fit [21]. In this study, the modeling results were first evaluated using a receiver operating curve (ROC) followed by validation through five performance evaluation indices, i.e., precision, recall, F1 score, accuracy (ACC), and Matthew’s correlation coefficient (MCC), with the help of the ‘caret’ and ‘ModelMetrics’ packages. The statistical descriptions of these indices are shown in

Table 2. Characteristic details of index-based evaluation measures.

Index	Statistical Definition	Usage
Precision	$\frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FP}\right)}$	This evaluates the fraction of TP samples among all predicted positive samples.
Recall	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$	This quantifies the fraction of TP samples among all real positive samples in the data.
F1 score	$\frac{2\ast \mathrm{PPV}\ast \mathrm{SST}}{\mathrm{PPV}+\mathrm{SST}}$	This harmonic mean of precision and recall provides a value between 0 (worst) and 1 (best).
ACC	$\frac{\left(\mathrm{TP}+\mathrm{TN}\right)}{\left(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}\right)}$	Accuracy is the quantification of percentage samples for accurately predicted data in inventory/catalogue.
MCC	$\frac{\mathrm{TP}\ast \mathrm{TN}-\mathrm{FP}\ast \mathrm{FN}}{\sqrt{\left(\mathrm{TP}+\mathrm{FP}\right)\left(\mathrm{TP}+\mathrm{FN}\right)\left(\mathrm{TN}+\mathrm{FP}\right)\left(\mathrm{TN}+\mathrm{FN}\right)}}$	The most comprehensive index provides values between −1 (disagreement between sample and prediction) and 1 (as a perfect prediction with respect to samples).

. All of these indices depend upon the four parameters, i.e., true positive (TP), true negative (TN), false positive (FP), and false negative (FN) [23].

The results obtained from the evaluation measures helped rank the models for their performances. However, there was a chance that one model would perform well in some indices but give a poor performance in others. Therefore, in this study, we incorporated a combined index to quantify the overall performance of the models through the performance overall (POA) [65] by taking the sum of the calculated indices, i.e., accuracy, F1 score, and MCC (Equation (5)).

$${\mathrm{Performance}}_{\mathrm{overall}} \left(\mathrm{POA}\right)=\mathrm{ACC}+\mathrm{MCC}+\mathrm{F}1-\mathrm{score}$$

(5)

The model with the highest magnitude of POA represented the best model. Moreover, a decreasing trend in the POA values corresponded to the performance ranking of the models for LSM.

3. Results

This study presents a comparative assessment of five ML models for LSM in the rugged terrain of northern Pakistan. In order to gather the objectives of this study, the contribution and importance of the LIFs to the modeling were first assessed, providing modeling results and an evaluation of their performance, comparison, and accuracy based on the conventional assessment approach in the machine learning framework.

3.1. Importance and Contribution of Landslide Influencing Factors (LIFs)

The independent importance of an influencing factor is one of the most significant analyses in ML. The role of the influencing factors and their contribution to modeling plays a vital role in achieving reliable results. In this study, the factor importance in regard to all of the five models was first assessed using the varImp function of the ‘caret’ package. This function calculates and documents the mean decrease (scaled) of accuracy in ‘class-specific’ manners. The details of the contribution of these factors in each model are shown in Figure 7a. It can be observed in Figure 7a that during modeling through the RF and GBM approaches, LCLU was the most significant contributing factor. On the other hand, elevation took the lead in the SVM and LR models, and aspect was the most significant contributing factor in the maxENT model. The varying nature of the factor contribution in the different models reflects the working behavior of these models. Therefore, keeping in view the contribution of these factors, each model provided different results in terms of prediction (Figure 8).

However, considering this inconsistency of the factors contributing to each model, the need to assess the relative importance and overall contribution of these incorporated factors was important. Therefore, in this study, we also incorporated a technique presented by Breiman et al. [66], which works on the principle of RF to assess the importance of factors using MeanDecreaseAccuracy and MeanDecreaseGini in the ‘caret’ and ‘randomForest’ libraries in R. MeanDecreaseAccuracy rescales the class-specific importance scores using ‘standard error’, employing their mean. On the other hand, MeanDecreaseGini works on the principle of splitting a variable through the average gain, which works on a local decision function to select the best split. This importance analysis counts the number of observations against each factor in the data. The importance of each factor calculated through this approach is shown in Figure 7b. The results showed that LCLU, NDVI, elevation, slope and aspect, distance to drainage, distance to road, and distance to faults exhibited high importance for landslide susceptibility, compared with the other predictors such as rainfall, geology, TPI, and TWI in the area. This ranking and importance contribution analysis agrees with the performances of the models in terms of their accuracies, as described in the performance evaluation section.

3.2. Landslide Susceptibility Maps (LSMs)

The prediction results of the modeling are presented in Figure 8, which highlights the higher vulnerability of the study area for landslide susceptibility. For interpretation and analysis purposes, the value ranges of the modeled landslide susceptibility maps (0 to 1) were classified into five different classes. The division of the ranges was purely based on Jenke’s natural break approach [67] and has been adopted by a number of researchers in the field [35]. The value ranges for each of these five classes in each model is shown in Figure 8. The class with the lowest susceptibility range was classified as conditionally stable (very low susceptibility), whereas the other ranges were classified as low-, medium-, high-, and very-high-susceptibility classes in the map [15]. The selected LIFs were then analyzed for their contribution to the modeling to identify the major causative factor for this hazard, leading to a comprehensive discussion of the model performances, comparisons, and validations.

3.3. Performance Evaluation of Models and Analysis of the Results

The most important factor involved in the evaluation of the models was the analysis based on the area under the curve (AUC), which represents a measure of the separability with values ranges from 0 (lower) to 1 (higher) [68]. Higher values of AUC reflect the highest performance. A model is considered reliable if the AUC values are higher than 0.7 [69]. The AUC values of these models calculated using validation data are shown in Figure 9.

All the investigated ML models produced results with AUC values above 83%, which highlights the reliability of these models in providing good results in the study area. However, the variations in AUC values among all the models was relatively high, with AUCs of 0.969 in SVM, 0.967 in RF, 0.872 in maxENT, 0.967 in GBM, and 0.836 in LR (Figure 9). Keeping in view the assessment and validation of the models’ output, it is not recommended to investigate using only one parameter (i.e., AUC) for ML approaches, as this may not reflect the true prediction ability of models with varying and closer values. Therefore, the model results were further scanned through the other evaluation procedures based on the calculation of indices, and the results of these indices-based analyses are provided in Figure 10a. Figure 10a shows that across all five ML algorithms, the results of RF and SVM were found to be consistent in terms of providing higher performance accuracies than the rest of the models. The achieved values of these indices for ACC, precision, recall, F1 score, and MCC in RF were 0.87, 0.75, 0.98, 0.85, and 2655, respectively, whereas in SVM these values were 0.87, 0.77, 0.96, 0.85, and 2667.99, respectively (Figure 10a). However, in addition to SVM and RF as the top two models, GBM was also one of the leading models, providing results with higher accuracies, and it can be seen in Figure 10a that the difference between the performances of this model and the performances of SVM and RF was almost negligible, with ACC 0.86, precision 0.77, recall 0.94, F1 score 0.84, and MCC 2621.99. Therefore, it can be stated that no significant performance difference was found between these three models (RF, SVM, and GBM). LR and maxENT provided an averaged result with an ACC of above 0.65; a recall value of above 0.86; and the lowest values in precision, F1 score, and MCC (Figure 10a). To evaluate the best performer, the POA was calculated, which indicated that the SVM (POA = 2669) was the best model and placed RF (POA = 2656) and GBM (POA = 2623) second and third, respectively (Figure 10b). The models in this study were trained with the best possible configurations; therefore, the performance rankings of some of the models in this study followed the results of previous research [34]. However, the importance of the GBM model for LSM cannot be neglected, as it stands with SVM and RF in terms of the evaluation measures.

4. Discussions

LSM provides a basis for risk management and planning; therefore, highly accurate LSM is required for effective mitigation measures. Due to the complex nature of this hazard, it is quite challenging to produce high-quality and accurate maps at a regional scale in rugged mountains. In comparison to the conventional approaches, ML methods are efficient in solving geotechnical problems related to landslides. With a number of different modeling algorithms, the prediction results obtained by applying a single algorithm for mapping and modeling the susceptibility cannot be reliable, as most of the ML algorithms provide different performances in different areas at different scales, which further depends on the topography, climate, LCLU, and other natural and anthropogenic factors. Therefore, it is important to investigate, understand, and evaluate the differences in the outputs of various ML models to select and identify an optimal model [70]. Highlighting the various issues with the LSM during the modeling processes, Lee et al. [10] suggested putting a great effort into the selection and evaluation of the influencing factors. Therefore, in the present research study, as the very first stage, the twelve different landslide influencing factors selected (Figure 5a–l) were scanned through two different tests based on IGR and multicollinearity prior to modeling the susceptibility. Then, the contribution analysis was performed to assess the relative importance of the LIFs, ensuring the models were carefully tuned and trained by finding the optimal corresponding parameters in five different ML models: RF, SVM, maxENT, GBM, and LR. After this, a comparative assessment of the obtained modeled results was performed by calculating the AUC and confusion matrix, and they were ranked on the basis of their performances by calculating the POA.

4.1. Evaluation of LIFs and Modeling Criteria

The results of the IGR test indicated that with an IGR > 0, all of the selected LIFs were important and would not introduce any problem in the modeling process (Figure 6). The results of the multicollinearity test showed that none of the LIFs had a collinearity problem (Figure 6). In addition to the other factors affecting the accuracy of the models discussed above, the model results were also influenced by the training data in terms of the number of samples, which played a vital role in tuning the models. As a rule of thumb, Jain et al. [71] found that a minimum of 10 × d × C samples are required to tune models with ‘C’ classes in d-dimensional problems. Therefore, in this study, the number of training samples used (280) was above the minimum requirement, and these were further used to tune and train the models to achieve training accuracies of 0.866, 0.854, 0.760, 0.892, and 0.707 for RF, SVM, maxENT, GBM, and LR, respectively.

The modeling results of RF, SVM, maxENT, GBM, and LR, obtained using training data, were classified into five different classes. The spatial distributions of the landslide susceptibility classes shown in Figure 8 present the varying nature of the results in different models. Therefore, to check the similarities in the prediction results of the models in the area, a cross-correlation between the model outputs using validation data was performed, and it was found that RF–SVM, RF–GBM, and SVM–GBM had the most-correlated models in this study, with correlation values of 0.993, 0.981, and 0.985, respectively (Figure 11a). This verifies that the three models, RF, SVM, and GBM, performed very well, with approximately similar accuracy in the area.

4.2. Performances of Susceptibility Modeling

Figure 8 and Figure 12a–e show that most of the predicted landslide zones follow a drainage network in the region and correspond to the high and very-high susceptibility classes along with this network. Aspect-based analysis conducted using SVM results showed that these high and very-high classes are mostly attributed to the slopes facing NE to WSW through E and S. These aspects are considered as warmer aspects in the region and can be correlated with climatic conditions. This distribution of the susceptible classes modeled using the SVM approach with reference to slope aspect can be seen in Figure 11b. Similar results of the landslide abundance along such aspects were also produced by Koukis et al. [72] and Tsangaratos and Ilia [14]. The variations in climatic conditions along these aspects can be attributed to some other factors [73]. For example, these aspects mainly constitute relatively mid-level values of NDVI with sparse vegetation and experience a large and intense portion of direct rainfall, especially on the ESE–SW direction. The other slopes along the W to NE through the NW, normally known as cooler aspects, mainly constitute dense forest and snow at relatively higher altitudes. As shown in Figure 7b, the contribution of LCLU, NDVI, slope aspect, and elevation plays a vital role in the abundance of such a hazard; therefore, the prediction results of susceptibility and aspect-based analysis coincide mainly with the existence of these hazards. As the area is comprised of a diverse topography, with sudden changes in terrain, slopes, and aspect angles at some places (southern and northern zones), the contribution of other factors in the modeling cannot be neglected in this region. With this diverse topography, it can be highlighted that landslides in the area are mostly a cumulative effect of all the above mentioned factors.

The modeled results showed a very good agreement with validation data during the AUC analysis. In this study, the performance analysis of the modeled results showed that all five models were reliable in achieving good results in the area, with AUC values above 0.83. The better performances of RF, SVM, and GBM were observed in the scanning results of the models using indices-based evaluation measures (Figure 10a,b). Wang et al. [65] concluded that if the difference between the accuracies of the model using training and validation data is higher, then there is a possibility of overfitting the models during the tuning and training process. In this study, we used validation data to acquire ACC values of 0.87, 0.87, 0.73, 0.86, and 0.68 for the RF, SVM, maxENT, GBM, and LR models, respectively, which corresponds to minimal differences between the accuracies obtained using the training data; therefore, the results obtained in this study showed fine-tuning, training, and modeling with higher reliability. From POA values of 2656 in RF, 2669 in SVM, 2623 in GBM, 1761 in maxent, and 1299 in LR (Figure 10b), we concluded the higher performance of the SVM model in this study followed by RF, GBM, maxENT, and LR.

SVM models are considered to provide the best solution to a nonlinear process such as landslides, which makes this model best-suited to correctly predict the probability of landslide events by separating them with the help of a hyperplane [7]. Brenning [8] highlighted the ability of the SVM model to produce smooth prediction surfaces when compared with the tree-based models, with the presence of spatial artefacts in his results. Comparing results generated from LR and SVM, Yao et al. [74] found that SVM produced a higher prediction accuracy. Similar findings of outperformance and the production of smoothed results in the SVM model have been reported by Goetz et al. [75] and Yilmaz [76] when performing a comparison with NN, DT, and LR models. Similar to the findings of the SVM model’s outperformance in this study, Qing et al. [34] also reported that SVM performed the best among several other ML approaches for debris flow susceptibility along with the China–Pakistan Karakoram highway. However, the effectiveness of the RF model over SVM in different studies has also been discussed in [14,34]. For example, Youssef and Pourghasemi [77] compared SVM and RF with several other machine learning techniques (MLTs) in Saudi Arabia and found that the RF model was the best performer in the region for LSM. Similar to this finding, Tsangaratos and Ilia [14] observed a slight difference between the prediction rates of RF and SVM when applied to the north-west (NW) Peloponnese, Greece, and concluded the better performance of the RF model compared to SVM. Similar results with a minute difference in both of these models are observed by Wang et al. [33] in Shexian County, China. With the varying nature of the accuracy and performance of SVM and RF in different studies, we infer that comparative to other MLTs, the performance results of SVM and RF are quite close, with minimal differences (Figure 9 and Figure 10), and that both of these models can be considered as equivalent in providing reliable results in the area. The similarity in AUC values (0.967) of RF and GBM was due to the ensemble natures of these models, with decent interpretability. This finding is in agreement with the results from recent studies [24,78]. Considering the adhering similarity in prediction abilities, normally GBM offers a relatively higher number of sensitive parametrizations than RF, which makes it difficult to implement in a larger area [32]. Therefore, with the slight difference in POA values (about 1.24%) in this study between these two models, RF was found to be more reliable than GBM. Although maxENT has been successfully applied to species distribution modeling, a comparison of its performance with other ML models has not been fully investigated, particularly for LSM. However, some efforts have been made to compare the performance of this model [11,79,80,81]. For example, Park [11] compared the performance of the maxENT with the LR model by producing landslide susceptibility maps of Boeun, Korea and found the effectiveness of maxENT over LR. Similar to the findings of Park [11], Felicísimo et al. [81] found that this model performed better than LR in Deba Valley, Northern Spain. In contrast to the other outperforming models, LR yields the lowest accuracy, and our findings regarding the relatively poor performance of LR are also supported by other related studies [15,32].

4.3. Realtime Validation of Modeled Results

Some recent landslide events in the area also helped to validate the results of modeled landslide susceptibility. For example, eight landslide events occurred along Mansehra–Naran–Jalkhad (MNJ) Road on 26 July 2019, resulting in the destruction of several houses and a road blockage for days, with some causalities [82]. Another landslide event occurred on 15 July 2019 along the Muzaffarabad–Abbottabad road, resulting in the blockage of the road for two days. Figure 12f shows the Google Earth image of the landslide section (black polygon) along MNJ road near Jared town. The modeled outputs along this region from all five models are shown in Figure 12a–e, and it can be seen that this landslide section has been modeled and categorized as mostly in the high- and very-high-susceptibility classes in all the investigated models.

5. Conclusions

Landslides, being the most critical natural hazard, are a very serious threat to human life, infrastructure, and natural resources across the globe. The identification of an accurate and stable ML model for predicting landslide susceptibility is often a very difficult and demanding task, especially when applied to a region with rugged terrain. In this study, we implemented five state-of-the-art ML algorithms, i.e., RF, SVM, maxENT, GBM, and LR, to investigate their performances for LSM in part of northern Pakistan. The study indicated that:

• The 12 selected influencing factors (i.e., elevation, slope, aspect, TWI, TPI, distance to drainage, distance to fault, distance to road, NDVI, rainfall, LCLU, and geological layer) were useful when scanned through the IGR test, and none of the LIFs had a multicollinearity problem.
• The extracted 70% of the sample data used for the tuning and training of the models showed a good agreement with the tuning parameters, which helped in achieving higher accuracies during the training process.
• Apart from predicting large-scale landslides, a visual analysis (Figure 12) of modeled susceptibility maps indicated that all five models were also able to predict very-small-scale landslides, which can provide a better understanding to identify landslide risks in the region.
• The descriptive analysis of the LIF contribution showed that the area lying in the elevation range of 505 to 3895 m, with NE–WSW facing slopes, near the drainage networks, with LCLU other than snow, dense vegetation, cultivated land, and water bodies could be attributed directly to the slope failure manifestation in the study area, which can experience a relatively higher chance of landslide occurrences in the future.
• During the validation process, using 30% of the samples, the results showed that SVM (AUC = 0.969, POA = 2669), RF (AUC = 0.967, POA = 2656), and GBM (AUC = 0.967, POA = 2623) were the top three performers, leaving behind maxENT (AUC = 0.872, POA = 1761) and LR (AUC = 0.836, POA = 1299). The validation through the calculation of other parameters (precision, recall, ACC, F1 score, and MCC) showed a similar trend in terms of the performance sequence of the models.
• Exhibiting minimal differences in terms of all the evaluated parameters with SVM and RF, the study also found the better performance of the GBM model for LSM in the region.
• SVM > RF > GBM > maxENT > LR represents the overall performance ranking with decreasing POA values.
• Conclusively, these modeled results can be helpful in understanding and assessing the scope of landslide problems in this area and can also provide help in risk reduction and mitigation measures in this region as well as in other parts of the globe with similar topographical, climatic, and geological conditions.