Landslide Susceptibility Assessment by Machine Learning and Frequency Ratio Methods Using XRAIN Radar-Acquired Rainfall Data

Abstract

This study is an efficiency comparison between four methods for the production of landslide susceptibility maps (LSMs), which include random forest (RF), artificial neural network (ANN), and logistic regression (LR) as the machine learning (ML) techniques and frequency ratio (FR) as a statistical method. The study area is located in the Southern Hiroshima Prefecture in western Japan, a locality known to suffer from rainfall-induced landslide disasters, the most recent one in July 2018. The landslide conditioning factors (LCFs) considered in this study are lithology, land use, altitude, slope angle, slope aspect, distance to drainage, distance to lineament, soil class, and mean annual precipitation. The rainfall LCF data comprise XRAIN (eXtended RAdar Information Network) radar records, which are novel in the task of LSM production. The accuracy of the produced LSMs was calculated with the area under the receiver operating characteristic curve (AUROC), and an automatic hyperparameter tuning and result comparison system based on AUROC scores was utilized. The calculated AUROC scores of the resulting LSMs were 0.952 for the RF method, 0.9247 for the ANN method, 0.9016 for the LR method, and 0.8424 for the FR. It is also noteworthy that the ML methods are substantially swifter and more practical than the FR method and allow for multiple and automatic experimentations with different hyperparameter settings, providing fine and accurate outcomes with the given data. The results evidence that ML techniques are more efficient when dealing with hazard assessment problems such as the one exemplified in this study. Although the conclusion that the RF method is the most accurate for LSM production as found by other authors in the literature, ML method efficiency may vary depending on the specific study area, and thus the use of an automatic multi-method LSM production system with hyperparameter tuning such as the one utilized in this study is advised. It was also found that XRAIN radar-acquired mean annual precipitation data are effective when used as an LCF in LSM production.

1. Introduction

The United Nations Office for Disaster Risk Reduction reports that between 1998 and 2017, more than 4.8 million people were affected, and more than 18,000 lost their lives in landslide disasters [1], which are expected to become more frequent in the next few decades due to urbanization, deforestation and especially owing to climate change effects [2]. Landslide damage is particularly heavy in urban areas where the topographical settings force urbanization into or near mountainous areas. Although predisposed by local factors attributed to the slope, landslides primarily occur due to some kind of triggering factors. According to Osanai et al., 17,640 cases of slope failures were triggered by rainfall in Japan between 1972 and 2007 [3].

A recent case of rainfall-induced landslide disaster in Japan was recorded in July 2018. The landslides, together with massive flooding in a large part of Southwest Japan, were caused by heavy rains, an event officially referred to as “Heavy Rains of July, Heisei Year 30”. During the course of about 10 days, from 28 June until 8 July, rainfall records reached as high as 1800 mm on the island of Shikoku and 1200 mm in the Tokai region [4]. The most severe period of the disasters was between 5 and 7 July, when most of the rainfall intensity peaks were observed, and many landslides and floods occurred. Property loss caused by the July 2018 disasters was estimated to be ¥1.16 trillion (i.e., about US$ 10 billion), including damage to industries and public infrastructures [5]. Although emergency warnings were issued for eight prefectures, the death toll caused by the landslides and floods during the July 2018 disasters was above 225. The most affected prefecture was Hiroshima, with 113 fatalities, and one of the most affected areas was Kure City, with 24 deceased due to landslides. Additionally, most transportation lines into the city (except maritime ways) were cut off, and 760 houses were damaged.

A valuable source of data that has been used extensively in recent years for the analysis of rainfall-induced landslides in the context of Japan [6,7,8,9] is XRAIN (eXtended Radar Information Network). Sponsored by the Ministry of Education, Culture, Sports, Science, and Technology (MEXT) and available through the University of Tokyo’s Data Integration & Analysis System (DIAS) platform since 2014, XRAIN offers real-time rainfall measurements using Multi-factor (MP) radars [10]. These radars provide spatially accurate rainfall intensity data. In landslide susceptibility mapping, rainfall-related data are commonly based on point measurements from rain gauges located in or near the municipality’s center. However, rain gauge data are limited to specific points, requiring extrapolation for areas in between, potentially leading to inaccuracies, especially over large distances. In contrast, radar-based rainfall data samples a bi-dimensional grid over the measurement equipment, providing genuinely gauged and geographically accurate results when visualized in GIS for large areas. Given the variation in parameters over short distances in Landslide susceptibility mapping, utilizing radar-acquired rainfall data like XRAIN can offer significant benefits.

One of the strategies for minimizing the damage of landslide disasters is the production and use of landslide susceptibility maps (LSMs), which assess the probability of landslide occurrence in an area considering slope failure-related factors and the actual occurrence of past landslides in a GIS platform. An LSM is a vital tool for urban planning and risk management of urbanized areas prone to landslide activity. Currently, there are various methods of calculating spatial landslide probability and producing LSMs. Yilmaz et al. [11] defend that the frequency ratio (FR) method is one of the most practical and efficient methods for landslide susceptibility calculation in the GIS platform. However, advancements in programming and computation technology in recent years have brought about the extensive use of machine learning (ML) methods in myriads of areas of application, including the domain of natural hazards and landslide susceptibility assessment, as advocated by Goetz et al. [12]; Youssef and Pourghasemi [13]; Ado et al. [14], and others. Liu et al. [15], in a review study of the most recent advancements in LSM production, find that although different study areas may present different performances of specific methods, with sufficient and good quality data, ML-based methods efficiently produce reliable LSMs and argue that the application of hybrid or ensemble models further improves the map’s accuracy, a conclusion also verified by Fang et al. [16] and Ado et al. [14]. These authors also point out that in areas with insufficient landslide inventory, the introduction of generative adversarial networks (GAN) and transfer learning techniques allow the production of accurate maps even in areas with scarce data.

Many studies also point out that deep learning-based methods present excellent results in LSM production [17,18], but these models require a very large amount of data inventory to satisfactorily train their complex neural networks, which are usually not available in most situations, even in developed countries such as Japan [15]. Huang et al. [19] also attempted the production of LSMs with the approach of utilizing rainfall data by coupling susceptibility maps and rainfall threshold models. A similar approach was also proposed by Palau et al. [20] to integrate an LSM with real-time radar-acquired rainfall data.

Attempts at landslide susceptibility mapping of the area of Kure City have been made by Wu and Nakahita [7] with the logistic regression (LR) method using XRAIN radar-acquired rainfall data as a landslide conditioning factor (LCF) along with soil water index (SWI) and lithology type, as well as landslide data from the July 2018 disasters. Another attempt was made by Rodrigues Neto [21] with the FR method using a wider collection of LCFs.

Among many different ML methods, one of the most recommended for landslide susceptibility assessment is the random forest (RF) method, as also pointed out by Goetz et al. [12], Chen et al. [22], Youssef and Pourghasemi [13], Li et al. [23], Xia et al. [24], Imtiaz et al. [25], Huang et al. [19], and Daviran et al. [26].

As the main validation assessment of produced LSMs, the great majority of LSM and the majority of recent studies use the receiver operating characteristic (ROC) test’s area under curve (AUC) value, where 0.5 is considered the threshold value and values closer to 1 represent more accurate results [14,27,28,29,30].

Although much work has been performed in recent years on ML-based LSM production, most of them admit that the presented methods and conclusions do not present a universal application and that specific study area circumstances in LCFs, such as topography, lithology or rainfall characteristics may affect the susceptibility technique, and different methods may display different performances in different areas [14,15]. For that aspect, it is important to experiment with landslide susceptibility mapping methods for specific study areas. Although LSMs for the study area have been performed with the use of XRAIN radar data, no attempt has been made yet to use ML techniques and an extensive LCF collection.

This study aims at improving LSM production methods with the novel aspect of using XRAIN radar-acquired rainfall data as one of the LCFs and producing the maps using random forest (RF), artificial neural networks (ANN), and logistic regression (LR) as three different ML algorithms. For comparison purposes, the statistical method of frequency ratio (FR) was also employed to produce the LSM. An accuracy assessment is performed for the produced maps to find out the most efficient ML algorithm for LSM production in the proposed study area. A new and original computer-code-based method is also proposed to perform LSM production, performance assessment and comparison automatically, a technique which would significantly improve both practicability and accuracy of LSM production in virtually any study area where data are available. The research aims to use Kure City (Hiroshima Prefecture) as a study area and the July 2018 heavy rain-induced disasters as a specific case for the study since both landslide and rainfall records are abundant in the specific event for the area. It is expected that the investigations and experimentations concerning LSM production will not only lead to improvements in landslide disaster prevention strategies through the creation of better LSMs and the development of an original, more practical, and accurate landslide susceptibility mapping method but also provide insights into the characteristics of rainfall-induced rapid-moving landslides in the context of Japan, and in other regions with similar geological, geomorphological, meteorological, and social circumstances.

2. Materials and Methods

2.1. Study Area

For this study, a rectangular area of 390.5 km² (approximately 28 km × 14 km) covering Kure City, south of Hiroshima Prefecture, was used (Figure 1). The cartographical edges that limit the study area are Top: 34.325104 dd; Right: 132.800008 dd; Bottom: 34.197426 dd; Left: 132.500202 dd.

A geographically small port town adjacent to the Seto Inland Sea, Kure started as a shipbuilding facility at the end of the 19th century. Kure was soon made into a major dockyard and military base; the port and the city around it grew quickly due to the Imperial Japanese Navy’s and its facilities’ rapid development until the end of World War Two. However, the area’s flat terrains are cramped and limited by mountains (a common scenario in Japan), which forced the town’s rapid expansion into and near adjacent hills. As a result, a lot of the town’s habitations are located in rugged terrain. In fact, about 14% of the buildings framed in the study area are on slopes with steepness above 20°, a situation susceptible to landslides. This risky scenario was explicit in the heavy damages suffered by the city of Kure in the July 2018 heavy rain disasters.

Topographically, the area is mostly comprised of rugged terrain: from the 390.5 km² of land in the study area, 191 km² (about 49%) is composed of slopes above 20° steepness. The altitude varies between sea level around the dockyard areas and 839 m at the peak of Mt. Noro. Other predominant peaks in the area include Mt. Enofuji (664 m), Mt Yasumi (497 m), and Mt. Ege (568 m).

Geological mapping performed by Yamada et al. [32] in the area shows that the region’s lithographical setting is composed primarily of igneous rocks of the Late Cretaceous, predominantly biotite granite and hornblende–biotite granite of the Hiroshima Granitic Rocks group, commonly referred to as “Hiroshima Granite”. In the study area, these rocks are distributed in most of the western portion of the map (Mt. Eboshiwa, Mt. Ege, Mt. Myojin, Mt. Tengujo, Mt. Hachimaki, Mt. Mitsuishi, Mt. Yasumi, Mt. Mitsumine, and Mt. Hisago), as well as in a peninsula in the far east of the area (Mt. Mizunoura).

The other Late Cretaceous igneous rocks that occur secondarily but still dominantly in the area, particularly Kure City, are volcanic rocks from the Takada Rhyolites and Hikimi Group, namely dacite to rhyolite welded tuff, tuffaceous sandstone, mudstone conglomerate, found in the central–west (Mt. Ozumi Mt. Enofuji, Mt. Age and Mt. Tsuchi) and the far northeastern part of the area (Mt. Kanashioku, Mt. Mosuke, Mt. Nada), as well as rhyolite welded tuff with non-welded pyroclastic rock, dacite welded tuff, found in the central east part of the area (Mt. Noro).

The region is occasionally carved by granite porphyry and granophyre dikes of the late cretaceous (though posterior to the Hiroshima Granitic Rocks), cutting the granitic rocks in NE-SW orientation and the granodioritic and dacitic rocks in NW-SE orientation. The Quaternary deposits in the area, found along river vales, plains, and deltas, comprise sand, mud, and gravel from the Saijo Formation, as well as gravel, sand, and mud alluviums from the Holocene. Much of the port and dockyard terrain (sometimes directly adjacent to the granitic mountains) comprises artificial reclamation grounds.

Regarding the structural geology arrangement of the study area, the regional structures are primarily dominated by NE–SW lineaments (especially near Mt. Noro), which might be related to the Triassic Maizuru Belt [33]. This orientation governs many morphological aspects of the area, as exemplified by Mt. Yasumi’s peninsula or Kurose River and its valley. Though not so common in a regional aspect, the study area around Kure City also presents an NNW–SSE fault line along the northwestern granites of the area, near Mt. Ege and Mt. Myojin.

The most predominant bedrock lithology of the area, coarse biotite granite/granodiorite and hornblende-biotite granite/granodiorite of the Hiroshima Granitic Rocks group, or simply “Hiroshima Granites”, are very easily weathered, changing into a soil commonly referred to as Masado or “real sand”, in Japanese [34]. Masado granitic soil is known to have good permeability and be very brittle when wet, which causes it to be prone to lose its structure and stability when rainfall infiltration occurs and, thus, to be a very susceptible soil for landslide occurrence during heavy rainfall events.

Additionally, according to Chigira [35], granites in mountainous areas are microscopically sheeted around depths of 50 m to undergo the stress of the ridge morphology. According to the author, these micro-sheet configurations stimulate the formation of cracks in the granite due to stress release, temperature change, and water content near the surface, thus facilitating landslide activity in events of heavy rainfall. The Seto Inland Sea has little rainfall compared to the surrounding oceanic coastal areas in Japan, like the Sea of Japan and the Pacific Ocean. Although a fairly good part of the oceanic precipitation clouds is blocked either by the Chugoku Mountains northward or the Shikoku Mountains southward and the region is considered relatively dry [36], heavy rainfall is particularly concentrated in mountainous areas. In Kure, the average annual average precipitation ranges from 1000 to 1600 mm, characterizing a relatively mild rainy zone. Mountain areas around the Seto Inland Sea, however, reach annual average precipitation of 2000 mm to 3000 mm. The period of the year with the heaviest rainfall occurs between June and July every year when the average monthly precipitation reaches 227 mm [37].

2.2. Landslide Conditioning Factors (LCFs)

In this study, nine LCFs (i.e., slope failure-causing factors), as detailed in the following sub-sections, were used for landslide susceptibility assessment.

2.2.1. Lithology

The physical properties and weathering susceptibility of rocks depend on their lithological setting. Numerous studies (e.g., [11,35,38,39,40,41,42,43]) have explored this relationship. In the study area, granites with medium to coarse granulometry are highly prone to weathering [34], making them susceptible to landslides. Guzzetti et al. [38] also found that slope failure events correlate with sedimentary and tectonic discontinuities, particularly between hard and soft rocks and permeable and impermeable layers.

However, detailed and spatially localized weathering grading requires time-consuming fieldwork, making it impractical for large-scale analysis. Therefore, weathering levels should be considered in the context of the specific location’s climate and geological features. Sediments and deposits in our study area are mostly found in landscapes with low to neutral slope inclinations, leading to a low correlation with the probability of slope failure.

The lithology LCF in this study originates from the geological mapping performed by Yamada et al. [32] and is presented in a 1:200,000 Hiroshima geological map (NI-53-33).

2.2.2. Land Use

Landslides, though a natural process in landscape shaping, can be exacerbated by human influences [43]. Changing drainage patterns for habitation or agriculture is a common human cause of landslide activity. Destabilizing slopes through construction overload or poorly planned re-landscaping also reduces slope stability. Deforestation increases landslide probability due to the importance of vegetation cover. In Japan, the cramped setting of plain areas forces the population onto rugged terrain for habitation, agriculture, or industry, making land use a significant factor in landslide probability compared to more spacious areas. The land use LCF of this study originates from data provided by Japan’s National Land Information Division (NLID), Ministry of Land, Infrastructure, Transport and Tourism (MLIT) [44].

2.2.3. Elevation

The elevation attribute is often used in landslide risk assessment due to its influence on contextual geomorphology and geology [11,40]. Low elevations typically feature gentle terrains with thick soils, while high elevations consist of leveled mountain summits or cliffs with strong bedrock [40]. This leads to a concentration of landslide risk in areas with intermediate elevations. However, the impact of elevation on slope failure occurrence is relative to the regional geology of the research area, so it requires specific consideration and adjustment rather than adopting global characteristics. The altitude LCF of this study is derived from the digital elevation model (DEM) availed by the Geospatial Information Authority of Japan (GSI) [45].

2.2.4. Slope Angle

Slope angle significantly influences landslide susceptibility. In a uniform slope with isotropic material, failure probability increases with the slope gradient [39]. Thus, slopes with low angles have low to insignificant landslide probability. However, actual material stability varies along slopes based on geology and weathering levels. Consequently, slopes with higher gradients are often more resistant to slope failure [40]. In the study area, high-inclination slopes (usually above 50°) are composed of outcropping igneous bedrock sections that are comparatively resistant to slope failure due to limited weathering impact. The slope angle LCF in this study was derived from the DEM availed by the GSI [45].

2.2.5. Slope Aspect

The relation of slope aspect (diving direction) with landslide probability is not fully understood. Yet, some investigators use it in statistically calculating landslide failure, as patterns between failure occurrence and slope direction are commonly observed. Two widely accepted hypotheses for the slope aspect’s effect on landslides are: (a) it relates to the area’s general physiographic trend, and (b) depending on the slope direction in relation to the sun’s trajectory, humidity may be more or less prone to evaporation, affecting landslides differently based on the main precipitation direction [46]. The slope aspect LCF in this study was derived from the DEM availed by the GSI [45].

2.2.6. Distance to Drainage

Drainage systems significantly influence landscape morphology and slope settings, often associated with increased landslide activity [42]. Slope stability models heavily depend on subterranean water contents, like pore water pressure capacity, which is influenced by groundwater systems associated with surface-level drainage systems. In landslide risk assessments, the drainage system factor is usually considered in terms of relative distance to drainage channels or drainage density values [11,40,41]. This study uses distance to drainage as the relevant measure extracted from the DEM availed by the GSI [45].

2.2.7. Distance to Lineaments

Geological structures are strain marks or “wounds” caused by tectonic forces throughout a rock’s history [47]. They are crucial for understanding rock formation and have practical applications in economic exploration and geotechnics. In map view, these structures are represented as lineaments. Faults, folds, foliations, and joints are common structural defects that weaken the rock’s cohesion and contribute to slope instability. Therefore, geological structures, particularly lineaments, are essential in assessing landslide risk [11,38,39,43,48,49]. In GIS analysis, they are considered as the distance to lineament factor. The distance to lineament LCF in this study is derived from a 1:200,000 geological map (NI-53-33) by Yamada et al. [32].

2.2.8. Soil Classification

The topsoil characteristics of a slope affect its stability and landslide susceptibility in the sense that it greatly influences how rainwater infiltrates the slope material, being considered an important LCF [50]. The soil class LCF utilized in this study was prepared from a 1:50,000 Kure soil classification map (NI-53-33-7) by Tanimoto et al. [51].

2.2.9. Rainfall

Landslide disasters are often triggered by increased pore water density and saturation due to heavy rainfall [34,35,43,52,53,54,55,56]. The July 2018 Southwestern Japan disasters exemplify how landslides are commonly accompanied by flooding. However, the association is not solely due to precipitation and ground saturation. Flooding can cause landslides by cutting slopes’ basal section around stream banks. Additionally, debris flows, and mudflows can be mistaken for floods, as they occur in small drainage systems, and they might happen simultaneously. On the other hand, landslides may deposit debris into rivers, causing sudden water level changes and even dam collapses in severe cases [43].

Although rainfall is commonly regarded as a triggering factor of landslides more than a causative factor, it is found that long-term precipitation localization patterns are recurrent, and the rainfall localization patterns of new rainfall events are generally expected to follow mean annual precipitation localization patterns [31,57,58], a phenomenon mostly associated with topographical interactions, and thus particularly remarkable in areas with rugged terrain such as Kure. For that reason, it is expected that recorded mean annual precipitation may be utilized in the prediction of future rainfall localization, thus making it a relevant LCF.

The mean annual precipitation between the years of 2016 and 2021 data utilized in this study were acquired through XRAIN radar-acquired data, thoroughly described in the following section.

2.3. XRAIN Radar-Acquired Rainfall Data

One of the landslide-causing factors commonly used in the production of LSMs is precipitation volume. However, most studies use rainfall data acquired through rain gauge measurement stations, which are only representative of the measurement location and not detailed enough for spatial modeling problems. However, new strategies in precipitation measurement have been developed in a few countries, such as Japan. XRAIN (eXtended Radar Information Network) data are operated by the Ministry of Land, Infrastructure, Transport and Tourism (MEXT) and made available through the University of Tokyo’s Data Integration & Analysis System (DIAS) platform. Starting operation in 2014, XRAIN comprises a real-time rainfall measuring system based on Multi-factor (MP) radars, which allow for more spatially accurate measurements of rainfall volume. In contrast to more conventional point-based or extrapolated rain gauge measurement station data (commonly used in landslide susceptibility mapping methods), radar-acquired data allows for bidimensional sampling of rainfall over an area, which makes up for much more detailed localization aspects of rainfall data.

In this research, the XRAIN rainfall data were downloaded as .zip packed .csv files spaced in 5 min, 30 min, or 1 h intervals. The .csv files comprise tables with cells spatially organized so that each cell represents a 287 × 230 m pixel in a north-oriented grid representing the designated area, and each cell’s value expresses the rainfall intensity in mm/h at the time of measurement. For the study area, each of the files comprised a 97 × 67 grid with 4999 pixels. XRAIN data collection for the area of Chugoku (where Kure is located) started in 2016, so the range of collected data span from 2016 to 2021, summing up to more than 60,000 records.

2.4. LSM Production Methods

In this study, four different methods of LSM production were tested and compared for prediction accuracy. As a general statistical approach, the frequency ratio (FR) method was used, while machine learning (ML) techniques random forest (RF), artificial neural network (ANN), and logistic regression (LR) were adopted in order to find which one is the most efficient for the intended objectives. The target area for the LSM production contains 432,258 20 m pixels.

In order to make a comparison between the above four methods, an identical collection of eight LCFs was used in shapefile form: (a) lithology, (b) land use, (c) altitude, (d) slope angle, (e) slope aspect, (f) drainage density, (g) distance from lineaments, (h) soil class, and (i) mean annual precipitation. These LCFs are presented in a map view in Figure 2.

Multi-collinearity of the LCFs, that is, a phenomenon in which two or more factors are too closely correlated, would cause distortion and inaccuracy of the model estimation of the susceptibility assessment model [22,59]. In order to determine the independence of LCFs, many authors utilize the variance inflation factor (VIF) method, where it is generally accepted that VIF > 5 indicates potential multi-collinearity [22,60,61]. T. The VIF values of the 8 LCFs used in this study are shown in

Table 1. Variance inflation factor (VIF) for the utilized LCFs of this study. All LCFs display a VIF < 5, evidencing independence between variables and no multi-collinearity.

LCF	Variance Inflation Factor (VIF)
Lithology	2.147
Land use	1.037
Altitude	1.525
Slope angle	1.029
Slope aspect	1.002
Distance from drainage	1.072
Distance from lineaments	1.187
Mean annual precipitation	1.674

, ranging from 1.03 for the lowest value to 2.14 for the highest value, indicating no multi-collinearity and independence of the utilized 8 LCFs of this study.

The DEM used in this study was provided by the GSI of Japan [45]. The land use data were provided by the NLID, MLIT of Japan [44]. Lithology and lineament data were extracted from the Hiroshima Geological map by Yamada et al. [32]. The mean annual precipitation data comprises recorded rainfall between 2016 and 2021 with XRAIN technology. A summary of all the data used in the analysis is shown in

Table 2. Summary of the data used in the analysis of this research.

Data	Format	Source
Landslides from July 2018 disasters	Point-type shapefile	Geospatial Information Authority of Japan (GSI) [45].
Lithology	Polygon-type shapefile	Hiroshima 1:200,000 geological map (NI-53-33) by Yamada et al. [32].
Altitude	20 m Raster	DEM by GSI [45].
Slope angle	20 m Raster	(Extracted from) DEM by GSI [45].
Slope aspect	20 m Raster	(Extracted from) DEM by GSI [45].
Distance to drainage	Polygon-type shapefile	(Extracted from) DEM by GSI [45].
Distance to lineament	Polygon-type shapefile	Extracted from geological map (Yamada et al. [32]).
Land use	Polygon-type shapefile	NLID, MLIT of Japan [44].
Soil class	Polygon-type shapefile	1:50,000 Kure soil map (NI-53-33-7) by Tanimoto et al. [51].
Mean annual precipitation (2016–2021)	Polygon-type shapefile	XRAIN radar-acquired data, Ministry of Education, Culture, Sports, Science and Technology (MEXT), University of Tokyo’s Data Integration & Analysis System (DIAS) (2018).

.

The landslide inventory used in this study comprises 1177 landslide points referent to the July 2018 disasters, which were mapped and provided by the GSI of Japan [45] with aerial photography immediately after the disasters (Figure 1). The landslide points were randomly separated into training and test sets, with a ratio of 70% for the training set (824 landslide points) and 30% for the test set (353 landslide points). These points are distributed along the pixels of the study area, characterizing landslide pixels and non-landslide pixels along the study area, where the non-landslide pixels are more numerous than the former.

2.4.1. Frequency Ratio (FR) Method

The FR method uses the assumption that landslide occurrence is determined by factors that are related to the event and, thus, that new landslides will generally occur under the same conditions as past landslides [11,48,62]. In the FR method, FR values represent the ratio between the landslide occurrence and the area of the specified factor for a given factor class. In order to acquire values representative of FR analysis, the landslide occurrence and factor area values are normalized by calculating the percentage of the specific class to the total factor or landslide of the analysis [11]. For example, in a study area with 100 km² and 20 landslides, a factor class of 10 km² and 5 landslides represents 10% of the analyzed area and 25% of the landslides, resulting in a value of 2.5 for that class. In FR calculation, 1 is considered the medium value, and thus, values larger than 1 represent a higher correlation of landslide occurrence with the given attribute and, consequently, a higher risk [63]. In some FR method calculations, such as the one conducted by Yan et al. [64], FR values are normalized according to the other classes of FR values in that factor for the sake of a smooth analysis process. After the FR is calculated for each range or type of factor, the values are summed in each pixel of the susceptibility map to calculate the landslide susceptibility index (LSI) for that specific pixel [11,48]:

$$\mathrm{L}\mathrm{S}\mathrm{I}=\sum \mathrm{F}\mathrm{R}$$

(1)

Once the LSI values are established for each pixel of the map, a final LSM is produced, where higher LSI values represent a higher risk of landslide occurrence in that location.

In this study, the platforms used for FR calculation were ArcGIS Pro (for data manipulation) and Microsoft Excel (for calculations and graphical presentations). Each of the landslide-related factors was reclassified into 10 classes (for the quantitative factors) or was used in the original classification (for qualitative factors). The number of landslides in each class was found using the spatial join analysis tool in ArcGIS, and the class area was calculated using the calculate geometry tool. With the variables necessary for FR calculation attained, the FR value for each factor class was calculated in the field calculator. Finally, the LSI value for each 20 m pixel of the study area was obtained by summing up the respective FR values for the specific blending of factor classes of that pixel, resulting in the FR-based LSM.

2.4.2. ML Methods

ML is a form of artificial intelligence comprising methods where a system “learns” based on a set of data, looking for patterns in it and how they affect a certain result relative to a problem. It was found that this technology is successful in completing varied specialized tasks when set up with a sufficient dataset and adequate hyperparameter tuning. The past decade has seen a substantial rise in ML techniques in various areas of application, such as autonomous driving, health care, finance, manufacturing, and energy [65]. The utilization of ML methods in the domain of natural hazards (including landslide susceptibility assessment) is supported by Goetz et al. [12] and Youssef and Pourghasemi [13]. Some of the advantages encompassed by the use of MLs include adjusting its internal structure to the experimented data, the ability to extract knowledge automatically from a dataset and to build classification and regression to provide accurate models, as well as efficiency and practicability even in large areas [13,66,67].

ML includes various specific techniques with varied degrees of complexity. Although all these techniques employ the common concept in ML of identifying patterns in data and building a prediction model, the specific mechanisms and algorithms used differ.

Random Forest (RF) Method

Much of the bibliographical research suggests that the ML technique judged most effective for the specific case of landslide risk assessment mapping is the RF technique [11,13]. The RF technique is actually an expansion of another ML method, the decision tree. The decision tree technique is a supervised ML method; that is, a set of inputs (samples) and outputs (labels) are given so that the algorithm may figure out a function (model) to find outputs for “unlabeled” inputs. For this specific method, the machine observes a provided dataset and creates a “tree” (e.g., prediction model) that best represents it. In the “tree”, every time the data split according to a variance in a parameter, a new “branch” cleaves out of the tree in what is called a decision node. The splitting goes along for every other parameter variety, creating new “branches” in decision nodes until it finally ends in a “leaf”, representing a predicted outcome for that particular set of parameters in the data. Due to their simplicity and intelligibility, decision trees are a popular ML technique [68].

However, decision tree learners have the limitation of being prone to creating over-complex trees, which do not reflect an accurate representation of the data in a generalized way, which is known as overfitting. This was solved with the RF ML technique, which, as the name suggests, is an ensemble model comprising a “forest” with many decision trees (Figure 3). Each tree is an entirely random and independent experiment, which prevents overfitting by outputting a result comprising an ensemble-averaged prediction for all the decision trees in the RF, avoiding the result given by a singular (possibly unrepresentative) decision tree. Data used in the RF learning method have few necessities of transformation, as the algorithm deals with outliers and missing values [11].

In this study, the RF algorithm utilized was the one provided in the scikit-learn ML library (sklearn.ensemble.RandomForestClassifier), based on the Jupyter Notebook platform with Python 3.9 programming language, developed by Pedregosa et al. [69].

Artificial Neural Network (ANN) Method

Although all ML models are in some way inspired by the human brain, ANNs most directly resemble the brain structure, as is suggested in the method’s name itself. ANNs are composed of nodes called artificial neurons, which work by transmitting signals to the next node, much like synapses of neuron cells in a biological mammalian brain [70].

In its computational model, ANNs acquire, represent, and compute a mapping from a multivariate space of information to another based on a dataset that represents that mapping [71]. The most commonly utilized ANN method is the backpropagation learning algorithm, which consists of creating a network composed of one input layer, one output layer, and various hidden layers in between them [72]. Each layer is composed of processing unit nodes (neurons), and the nodes in one layer are connected to the ones in other layers by attributed weights and transfer functions. Each neuron receives “signals” (represented by real numbers) from the neurons of the previous layer, and its output to the subsequent layer is computed through a non-linear function of the sum of its inputs, and the “strength” of its signal is determined by the weight of the connection, attributed with training. The result is processed from the input layer through hidden layers until reaching the output layer, which represents the result calculated by the model. As in other ML methods, the ANN model goes through a training process equipped with a dataset of known input and results, adjusting the weights of its connections based on a trial-and-error process known as supervised learning. The process goes on until it gets enough “experience” to finely tune the communication between layers so as to minimize errors and achieve the best results [70]. An illustration of the ANN concept is shown in Figure 4.

Logistic Regression (LR) Method

Logistic regression is a well-known statistical model used for calculating the probability of an event with a binary (0 or 1, yes or no) outcome [73]. It is differentiated from linear regression in the sense that the latter predicts continuous outcomes; that is, the outcome may be any possible value between an interval. In this sense, LR prediction is more suitable to apply in cases where the aftermath may either go one way or the other, such as in landslide susceptibility analysis. The slope failure either occurs or does not, with no in-between.

Although the analyzed event in binary logistic regression only has two outcomes, the logistic function, represented by a sigmoid curve in a bidimensional graph where the vertical axis represents the dependent binary valuable and the horizontal axis represents the independent variables, calculates probabilistic values between 0 and 1, allowing for a usable result in landslide susceptibility assessment tasks. An example of a typical logistic regression with a sigmoid curve is shown in Figure 5.

2.5. Performance Assessment

In order to assess the performance of the produced LSMs, the receiver operating characteristic (ROC) analysis was employed in this study. Initially developed for radar accuracy tests, the ROC method is recommended for landslide zoning validation tasks due to its threshold-independent nature [27,28,29]; that is, it does not require a fixed value to determine that either negates or requires a landslide activation since LSI is a probability assessment, not a deterministic one. Thus, ROC analysis uses multiple thresholds with different interspaces, and the points in the ROC curve represent these possible cutoff thresholds given by the multiple cases in a model (i.e., LSM). Each threshold forms a confusion matrix with four types of outcomes for each pixel: true positive (TP), false positive (FP), true negative (TN), and false negative (FN). Each threshold point in the curve then corresponds to a pair of sensitivity and specificity values, where sensitivity is represented by the y-axis and represents the true positive rate (TPR), and the specificity is shown in the x-axis, representing the false positive rate (FPR). TPR and FPR are calculated as follows:

$$\mathrm{T}\mathrm{P}\mathrm{R}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(2)

$$\mathrm{F}\mathrm{P}\mathrm{R}=\frac{\mathrm{F}\mathrm{P}}{\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}}$$

(3)

After the ROC curve is plotted, its area under the curve (AUC) may be calculated derived from the constructed ROC curve as follows:

$$\mathrm{A}\mathrm{U}\mathrm{C}={\sum }_{i=1}^{n+1}\frac{1}{2}\sqrt{{(x_i-x_{i+1})}^2}\times (y_i+y_{i+1})$$

(4)

The AUC is used as a metric to assess the quality of the LSM, where a larger area (ranging from 0.5 to 1) represents better prediction performance, that is, how well the model separates the validations landslides throughout the susceptibility zones of the LSM. For that reason, the AUC value is used as the primary measure for LSM accuracy in this study [27,28,29].

As an additional method for LSM validation, landslide density is checked for each one of the 5 LSI zones attributed in the LSM. It is expected that in an efficient LSM, the landslide density distribution will follow a proportionally direct growth with each zone change from very low to very high.

2.6. Automated Hyperparameter Tuning Based on AUROC Analysis Validation

As illustrated in the previous topics, a varied selection of LSM production methods is available, each with its specific hyperparameters and configurations that may yield better or worse (more or less accurate results) in the final LSM. Although many studies have been made comparing LSM production methods based on their accuracy, it is a known fact that the optimal choice of method and its inbuilt parameters may vary depending on the specific context of the LSM area. Although LCFs generally show similar patterns concerning landslide susceptibility, the peculiar characteristics of a specific area’s topography, lithology, or climatological aspects may significantly change the dynamics between LCFs and landslide susceptibility, and different LSM production methods may be more or less proficient in dealing with such dynamics. For that reason, this study takes the novel approach of using an automated process based on programming code to experiment with different LSM production methods (namely ML ones), probing different parameters and configurations in each method allowing for automatic hyperparameter tuning. The accuracy of each experimented LSM is automatically tested through AUROC so that the most efficient method and resultant LSM is assigned, all through an automated process built in the Python programming language.

3. Results

3.1. LCF FR Values

The FR values calculated for each LCF class are exhibited in

Table 3. FR calculation table for all classes of each LCF. Classes with no landslide occurrence (and thus with an FR value of 0) are omitted.

Parameters and Classes		Landslides	Area (km2)	Landslides (%)	Area (%)	FR
Lithology	Argillite	5	3.55	0.47	2.10	0.22
	Porphyre granite	31	9.97	2.89	5.88	0.49
	Clastics	29	8.53	2.71	5.03	0.54
	Gravel	2	0.45	0.19	0.26	0.71
	Hiroshima granite	317	64.73	29.60	38.19	0.78
	Granitic rock	106	12.87	9.90	7.59	1.30
	Rhyolites	581	69.40	54.25	40.94	1.32
Land use	deforested area	200	32.58	18.67	19.20	0.97
	forest	751	106.49	70.12	62.76	1.12
	crop field	15	13.06	1.40	7.70	0.18
	rice field	1	4.91	0.09	2.90	0.03
	others	104	12.64	9.71	7.45	1.30
Altitude (m)	0–152	27	20.98	2.52	12.24	0.21
	153–236	70	36.01	6.54	21.01	0.31
	237–320	194	36.36	18.11	21.21	0.85
	321–404	378	33.82	35.29	19.73	1.79
	405–488	279	20.48	26.05	11.95	2.18
	489–572	103	10.88	9.62	6.35	1.52
	573–824	12	6.81	1.12	3.97	0.28
	>825	8	6.03	0.75	3.52	0.21
Slope angle (degrees)	20–24	3	4.51	0.28	2.63	0.11
	25–29	42	16.08	3.92	9.38	0.42
	30–34	230	38.76	21.48	22.61	0.95
	35–39	487	61.39	45.47	35.82	1.27
	40–44	261	39.52	24.37	23.06	1.06
	45–49	47	10.34	4.39	6.04	0.73
	≥50	1	0.76	0.09	0.45	0.21
Slope aspect	NNE	144	18.86	13.45	11.00	1.22
	ENE	127	19.19	11.86	11.20	1.06
	ESE	119	23.01	11.11	13.43	0.83
	SSE	102	24.38	9.52	14.22	0.67
	SSW	126	24.63	11.76	14.37	0.82
	WSW	141	21.50	13.17	12.55	1.05
	WNW	169	20.37	15.78	11.89	1.33
	NNW	143	19.41	13.35	11.33	1.18
Distance to drainage (m)	<50	368	59.77	34.36	34.87	0.99
	50–99	185	25.84	17.27	15.08	1.15
	100–149	178	20.54	16.62	11.98	1.39
	150–199	113	15.32	10.55	8.94	1.18
	200–249	79	11.42	7.38	6.66	1.11
	250–299	50	8.82	4.67	5.15	0.91
	300–349	32	6.61	2.99	3.86	0.77
	350–399	25	5.06	2.33	2.95	0.79
	400–449	15	4.01	1.40	2.34	0.60
	≥450	26	13.97	2.43	8.15	0.30
Distance to lineament (m)	<250	71	9.71	6.63	5.66	1.17
	240–399	96	11.21	8.96	6.54	1.37
	400–549	44	5.98	4.11	3.49	1.18
	550–699	90	11.78	8.40	6.87	1.22
	700–849	38	5.46	3.55	3.18	1.11
	850–999	67	10.67	6.26	6.22	1.00
	1000–1149	40	5.24	3.73	3.06	1.22
	1150–1299	75	9.79	7.00	5.71	1.23
	1300–1449	35	4.49	3.27	2.62	1.25
	≥1500	515	97.05	48.09	56.62	0.85
Soil class	Mih-1	35	1.37	3.16	0.12	2.56
	ZZ3	9	0.48	0.81	0.04	1.89
	Fut-1	15	1.01	1.36	0.09	1.48
	Har-1	518	39.07	46.84	3.53	1.33
	Ser-1	9	0.70	0.81	0.06	1.28
	Fjs	2	0.18	0.18	0.02	1.11
	Swa	7	0.79	0.63	0.07	0.89
	Tsm	4	0.55	0.36	0.05	0.73
	Gsa-1	344	50.44	31.10	4.56	0.68
	Isi-1	47	9.14	4.25	0.83	0.51
	Har-2	63	13.00	5.70	1.18	0.48
	Tuc	9	2.10	0.81	0.19	0.43
	Trg	2	0.59	0.18	0.05	0.34
	Gsa-2	24	10.03	2.17	0.91	0.24
	Kmi	1	0.50	0.09	0.05	0.20
	Urt	9	5.13	0.81	0.46	0.18
	Ngz	1	0.80	0.09	0.07	0.13
	Une-3	1	1.25	0.09	0.11	0.08
	Une-1	1	1.36	0.09	0.12	0.07
	ZZ2	5	8.88	0.45	0.80	0.06
	Ebe	0	0.65	0.00	0.06	0.00
	Gos	0	0.22	0.00	0.02	0.00
	Km	0	0.41	0.00	0.04	0.00
	Kri-1	0	0.48	0.00	0.04	0.00
	Kyt	0	0.09	0.00	0.01	0.00
	Kzs	0	0.18	0.00	0.02	0.00
	Okk	0	0.05	0.00	0.00	0.00
	Tdn	0	0.24	0.00	0.02	0.00
	Tns	0	0.07	0.00	0.01	0.00
	Ttr	0	0.02	0.00	0.00	0.00
	Yad	0	0.03	0.00	0.00	0.00
	Znt	0	0.05	0.00	0.00	0.00
	ZZ	0	0.08	0.00	0.01	0.00
Mean Annual Precipitation (mm)	<2100	14	6.28	1.31	3.66	0.36
	2100–2195	136	50.26	12.70	29.32	0.43
	2196–2291	381	65.62	35.57	38.28	0.93
	2292–2386	233	21.41	21.76	12.49	1.74
	2387–2482	133	10.84	12.42	6.33	1.96
	2483–2578	91	7.38	8.50	4.31	1.97
	2579–2674	53	5.82	4.95	3.40	1.46
	2675–2769	27	3.38	2.52	1.97	1.28
	2770–2865	3	0.37	0.28	0.22	1.29

, and graphs showing class dynamics in each LCF are shown in Figure 6.

The lithology LCF FR value calculation results are shown in Figure 6a. It was noted that non-alkaline felsic volcanic rocks (of the Takada Rhyolites/Hikimi Group) present the highest FR value of 1.34, followed by granites of the Hiroshima Granites with an FR value of 0.9. The interpretation that these lithologies, particularly the Hiroshima Granites, are highly susceptible to landslides is backed by much of the literature on the area, such as by Wang et al. [17] and Chigira [35].

The land use LCF FR values (shown in Figure 6b) show that the highest FR values in the area are found in the “golf course” class with 3.72, followed by “forest” with 1.10, and the lowest FR values are found in the “constructions” class, with 0.02. Many large landslides were recorded in golf courses during the July 2018 disasters around Hiroshima Prefecture. It is expected that such zones, when near slopes, are highly prone to landslides due to their lack of vegetation and susceptibility to rainwater saturation. Constructions, although also being areas altered by human activities, are usually cemented and thus do not represent slope failure risk.

FR calculation of classes in the altitude LCF (Figure 6c) shows that FR values peak at the 405 m class with a value of 2.18, bottoming with an FR of 0.21 on either extremity (0 m and 825 m). It is judged that this dynamic is explained by the fact that low-altitude classes are represented by zones with low slope angles, mostly related to valley bottoms. The highest altitudes, on the other hand, are represented by mountaintops with low slope angles or cliffs with steep slope angles but no soil material. Landslide activity peaks at intermediate altitudes in the flanks of mountains and hills.

The FR values for the slope angle LCF are shown in graph form in Figure 6d. It is noticeable that landslide activity peaks at intermediate slope angle values, particularly in the 35 to 40 degrees interval, with an FR value of 1.27, and bottoms in the extreme classes with 0.11 for the 20 to 25 degrees interval and 0.21 for the steeper than 50 degrees interval. Although it may be commonly assumed that steeper slope angles will be more susceptible to landslide because of gravity, that dynamic does not occur in nature due to the simple fact, in general, the steeper the slope, the lower its amount of soil, with the actual material of the slope becoming increasingly composed by hard rock material, not susceptible to common landslides.

The slope aspect FR values calculated in the Kure study area are shown in Figure 6e. Higher FR values are concentrated in northwestern-bound slopes, with FR peaking at the west at 1.33. Accordingly, lower FR values are found in southwestern-bound slopes, bottoming at 0.67. It is supposed that the specific topographical characteristics of hills in the study area, highly influenced by structural geology aspects, may lead to the scenario in the study area.

Distance to drainage FR values (shown in graph form in Figure 6f) start with a relatively high value of 0.99 in the smallest classes (closest to drainage), peaking in the 200 m class with 1.39. This behavior is expected since areas closer to drainage lines are more prone to be affected by water discharge from rainfall, making the soil saturated and prone to landslide activity.

The distance to lineament LCF values (shown in graph form in Figure 6g) display higher values closer, peaking at the 400 m class with a value of 1.37 and decreasing the farther it gets from lineaments, reaching the lowest value of 0.85. This is expected since areas closer to faults and lineaments are prone to exhibit more brittle geological features, which may present less slope stability.

The FR values for the soil class LCF, shown in 6H, peak in the “Mih-1” class, with a value of 2.56. “Mih-1”, according to the soil mapping performed by Tanimoto et al. [51], is referent to soil unit Mihara-1, classified as “residual immature soils”, a soil type that does not present consolidated structure, being highly unstable and thus prone to landslides, which explains the high FR value calculated.

Finally, the mean annual precipitation LCF presents FR values (shown in graph form in Figure 6i) starting with the lowest values in areas of low precipitation (0.36 for below 2100 mm), increasing until reaching peak FR value in the intermediate 2578 mm class. However, from that point FR values decrease, reaching 1.29 in the highest precipitation class of 2865 mm. This behavior is expected because the highest precipitation zones are usually found in high-altitude areas [31]. As shown in the altitude LCF FR values, high altitude areas are referent to mountaintops and hard rock hillsides, which are not prone to landslide activity.

It is noticed that all LCFs show a significant variation of FR values between classes, with the lowest FR variance being shown in the distance to lineaments LCF with a still relevant maximum difference of 0.52. This evidences the importance of all LCFs in the susceptibility assessment calculation, as there are considerable tendencies for specific classes to be associated with areas with lower or higher landslide densities.

3.2. FR Method LSM

With all the FR values calculated for each LCF class, LSI was summed for 20 m pixels in the study area in order to construct the final FR LSM for the Kure study area, shown in Figure 7. Visual analysis of the map shows a relatively good distribution of the LSI zones in landslide-affected areas, particularly in the flank of mountains, with very high LSI zones found particularly in the northeastern side of Mt. Noro, heavily affected by landslide activity. It is noticeable, however, that higher LSI zones seem too wide and not always associated with landslides of the July 2018 disasters, as seen near Peak Hayaga, where the FR-based LSM attributed large high and very high LSI zones even though landslide activity was not substantial, an aspect which may result in high false positive rates.

The AUROC analysis of the resultant LSM map constructed with the FR method for the Kure study area resulted in a score of 0.8424 (84.24% predictability potential) and is shown in graph form in Figure 8. The value is considered as “good” standing for landslide susceptibility assessment.

The analysis of landslide density per LSI zone for the FR map shows an increasing trend, though slightly exponential, in higher LSI zones. Pearson’s correlation coefficient analysis showed a final value of 0.88, configuring a good correlation between LSI increase and actual landslide density in each zone (Figure 9).

3.3. RF Method LSM

When applying the Kure study area dataset to the utilized RF algorithm, automatic parameter testing found that the best setting of the minimum number of samples required to be at a leaf node (“min_samples_leaf”) was 4560. Using the RF ML model produced the LSM seen in Figure 10. Visual analysis of the map shows excellent distribution of LSI zones, particularly with very high LSI in the northeast flank of Mt. Noro, where landslide activity was substantial during the July 2018 disasters. Although some false positives are still found in areas of Peak Hayaga, and some occasional false negatives can be seen particularly in some areas in the south of the study area, the map seems to sufficiently reflect the actual behavior of landsliding during the July 2018 disasters.

AUROC analysis of the RF LSM resulted in a score of 0.952 (95.2% predictability potential), considered excellent for landslide susceptibility assessment. The analysis is shown in graph form in Figure 11.

The distribution of landslide density for each of the LSI zones of the RF map in the Kure study area also shows a steady upward trend with increasing LSI zones (Figure 12), and Pearson’s correlation coefficient score was found to be 0.93. The steady correlation between LSI zones and landslide density suggests an LSM with good predictability.

3.4. ANN Method LSM

Applying automatic parameter testing of the ANN method in the Kure study area led to an optimal configuration of the hidden layer sizes parameter (“hidden_layer_sizes”) with three hidden layers with 20 nodes each. The resultant LSM produced with the ANN method for the study area is shown in Figure 13.

Visual analysis of the map shows the concentration of high LSI zones near mountain flanks in intermediate altitudes and near lineaments. The LSI distribution seems mostly accurate and related to landslide occurrences, especially in the northeastern flank of Mt. Noro. A few areas, like the flanks of Peak Hayaga and the region of Yanocho, however, present many false positives.

AUROC analysis of the map resulted in a score of 0.9247 (92.47% predictability potential), considered excellent for landslide susceptibility assessment. The AUROC graph is shown in Figure 14.

The correlation of landslide density for each of the LSI zones is shown in Figure 15. It is noticeable that the landslide density has a somewhat abrupt increase for each, especially in the intermediate zones. The increase is constant, though, and Pearson’s correlation coefficient was found to be 0.92, which conveys an excellent correlation.

3.5. LR Method LSM

Production of an LSM in the study area with the LR method was executed after automatic parameter testing, resulting in an optimal setting of using solver model (“solver”) “saga”, maximum iterations (“max_iter”) at 268, class weight (“class_weight”) as “balanced”, random state (“random_state”) at 101, and inverse of regularization strength (“C”) as 0.1. The produced LSM is shown in Figure 16.

AUROC analysis of the resultant LSM produced with the LR method in the Kure study area resulted in a score of 0.90.16 (90.16% predictability potential), considered excellent for landslide susceptibility assessment. The analysis graph is shown in Figure 17.

Landslide density per LSI zone in the map shows a steady increase until the high LSI zone and then a small decrease in the very high LSI zone. Although the trend is mostly steady upward in most of the LSI zones, the decrease between high and very high zones may convey a not-so-reliant LSM. Pearson’s correlation coefficient, though, resulted in a value of 0.83, conveying a good correlation coefficient (Figure 18).

3.6. LSM Accuracy Comparison

The production of LSMs for the Kure study area and subsequent validation with AUROC analysis for all the verified four methods (Figure 19) led to the finding that the RF method had the best results, with an AUC of 0.952, relative to a 95.2% predictability potential. The RF-produced LSM for the Kure study area is followed in AUC value by the ANN method (AUC = 0.9247), LR method (AUC = 0.9016), and the statistical FR method (0.8424).

The assessment that the RF method scored an AUROC of 95.2% in this study, the best among all the seven methods and produced an excellent LSM with finely tuned LSI zone distribution and significant landslide density increase with LSI. The assessment that the RF method is the most appropriate for landslide susceptibility mapping has also been indicated by other studies such as Goetz et al. [12]; Chen et al. [22]; Youssef and Pourghasemi [13]; Xia et al. [24]; Imtiaz et al. [25]; and Daviran et al. [26]. The RF’s decision trees are considered good models for tasks with large datasets and a high number of features (LCFs), as is the case of landslide susceptibility mapping. Although decision trees are easily subject to mistakes, being an ensemble model, the RF algorithm is highly unsusceptible to some problems in ML algorithms, such as under/overfitting, and can deal well with datasets such as the ones involved in the production of LSMs.

The ANN method, widely regarded as one of the most appropriate models for LSM production [15,23,74,75], scored an AUROC of 92.47% in this study, second in the list, and exhibiting an appropriate LSI zone distribution in the resultant LSM and steady increase in landslide density per LSI. Being able to work well with large datasets and adaptable to many different tasks, neural network models may still be regarded as one of the leading ML models for many kinds of assessment, as was demonstrated in this study.

Some studies rank the LR method as a high-ranking method for landslide susceptibility mapping [76,77]. In this study, the method performed relatively well with an AUROC score of 90.16%, scoring third among the utilized machine learning methods. The produced LSMs show a slightly exaggerated distribution of high LSI zones (and thus a relatively false positive rate), which may explain the maps’ lower performance when compared to the other produced LSMs.

Finally, the traditional statistical FR method ranked below all ML methods in this study, with an AUROC of 0.8424. Despite it being the primary method for LSM production before the advent of ML methods [11,78,79], more recent studies usually fall under the same conclusion that the method is usually outperformed by ML models [11,17,74,80]. It is also noteworthy that with the need for manual processing and manipulation of data in GIS platforms during the production of an FR LSM, this LSM took the most time and effort during production, which may be considered a significant blow in efficiency. Moreover, quick and automatic parameter tuning with code-based AUROC analysis is not yet possible.

3.7. Impact of XRAIN Radar-Acquired Rainfall Data

In order to assess whether XRAIN data provided a significant impact on the LSM accuracy and, if so, whether it was positive or negative, the RF algorithm (ranked best among the tested) was executed multiple times with the same parameter set as in the top ranking process, but with no XRAIN rainfall data as an LCF for comparison purposes. This test resulted in a mean AUROC score of 0.92, approximately 3.2% below the RF method executed with X.RAIN rainfall data included as an LCF.

4. Discussion

The AUROC test AUC value of 0.952 for the RF map, followed by 0.9247 for the ANN method, 0.9016 for the LR method, and 0.8424 for the FR method, evidences the superior efficiency of the ML technique RF in LSM production by a margin of about 11% in accuracy results compared to the lowest scoring LSM. The better efficiency of the RF-based LSM is also evidenced by the steady distribution of landslide density per each LSI zone. When seen in map form, it is noticeable that the RF-based map, when compared to the other 3 LSMs, seems to more finely define high LSI zones fitting with actual landslide points of the July 2018 disasters, characterizing a low false positive rate result. All maps point out high landslide susceptibility in the flanks of the mountains in the central-east region of the map, where there was, in fact, high landslide activity during the July 2018 disasters.

As explored by other authors that have assessed the efficiency of ML techniques in LSM production, it is judged that ML techniques, with their ability to deal with large sets of data and find specific patterns and relations in the intrinsic details of the dataset, perform a more accurate and fine assessment of landslide susceptibility when compared to simpler purely statistical methods such as the FR method. Moreover, as an ensemble model, the RF technique is also less subject to errors that may be present in ML techniques, such as over or underfitting [14,16], which may also explain the good performance of the model in this study’s attempt. ML techniques also have the advantage of allowing for fine parameter tuning rather easily for users familiarized with the method. Namely, this study’s automatic parameter tuning method found a large variance in the RF algorithm’s efficiency by tuning the “minimal samples leaf” parameter, which allowed for the optimal settings that delivered the final resultant LSM with an AUC of 0.952

One arguable advantage of the FR model, however, is that it evidences the relevance and importance of each LCF in the susceptibility analysis, as the variations in FR values demonstrate that each factor’s different class does relate with lower or higher landslide density, as demonstrated by the possibility to visualize the calculated FR values for each LCF class, allowing for a detailed understanding of how each LCF may dynamically affect slope stability by analysis of FR value for each LCF class. Although this information is not visualized in the final LSM, it may be useful for other landslide mitigation and prevention methods, as well as knowledge for decision-making in situations where an LSM is not readily available. In this study, for example, it was evidenced that both slope angle and altitude LCF landslide activity peaks in intermediate LCF values (i.e., 35–39-degree slopes, 405–488 m) instead of maximum LCF values, as would be normally assumed. This phenomenon is explained by the hard material with almost no soil content of high-degree slopes or the flat mountain peaks that usually dominate the high-altitude areas of Kure city. These aspects are not universal and may change in areas with different topographical and geological contexts, which evidences the importance of landslide susceptibility assessments for different areas.

The higher mean score of AUROC with LSM production with XRAIN rainfall data included as an LCF (approximately 3.2% above the same algorithm with no XRAIN data) evidences the advantage of utilizing this kind of spatially accurate radar-acquired rainfall data in landslide susceptibility analysis.

5. Conclusions

The production and subsequent validation through AUROC analysis of 14 LSMs for the study area using four different methods, one of them being a traditional statistical method (frequency ratio, or FR) and the other 3 being ML methods (random forest, or RF; artificial neural networks, or ANN; logistic regression, or LR) led to the final AUROC-assessed ranking from best to worst of RF; ANN; LR; and FR, with respective AUCs of 0.952; 0.9247; 0.9016; and 0.8424. Although all maps showed good to excellent results, the RF method was judged the best with a predictability potential of 95.2.

The assessment that the RF method is the most appropriate for landslide susceptibility mapping according to AUROC analysis is supported by the majority of the available literature [12,13,22,24,25,26], as well as that the FR method is outperformed by ML methods [11,17,74,80]. It is noteworthy, however, that the FR method provides the unique opportunity to examine all the LCF classes’ FR values (susceptibility to landslides), valuable knowledge to understand the deep dynamics of how environmental factors may affect the probability of slope failure in a particular area, which is not possible with ML methods.

The novelties of this LSM comparison study include the use of high-precision XRAIN radar-acquired rainfall data, which significantly differs from the commonly utilized rain gauge station measurement data in terms of spatial accuracy, which is significantly important in GIS-based susceptibility assessments, and the use of a code-based automated hyperparameter tuning based on iterated AUROC analysis validation, which allows for quick assessment of the ML method’s efficiency, each finely tuned with the best parameters for the specific context of the assessed study area. This method proposed a quick and practical process to achieve the optimal LSM given a dataset of LCFs and landslides. It is hoped that the LSM production process used in this study may provide more prompt and more practical LSM production with excellent results for a variety of areas, allowing for more efficient resilience strategies against landslide disasters.