A Systematic Literature Review and Bibliometric Analysis of Semantic Segmentation Models in Land Cover Mapping

Abstract

Recent advancements in deep learning have spurred the development of numerous novel semantic segmentation models for land cover mapping, showcasing exceptional performance in delineating precise boundaries and producing highly accurate land cover maps. However, to date, no systematic literature review has comprehensively examined semantic segmentation models in the context of land cover mapping. This paper addresses this gap by synthesizing recent advancements in semantic segmentation models for land cover mapping from 2017 to 2023, drawing insights on trends, data sources, model structures, and performance metrics based on a review of 106 articles. Our analysis identifies top journals in the field, including MDPI Remote Sensing, IEEE Journal of Selected Topics in Earth Science, and IEEE Transactions on Geoscience and Remote Sensing, IEEE Geoscience and Remote Sensing Letters, and ISPRS Journal Of Photogrammetry And Remote Sensing. We find that research predominantly focuses on land cover, urban areas, precision agriculture, environment, coastal areas, and forests. Geographically, 35.29% of the study areas are located in China, followed by the USA (11.76%), France (5.88%), Spain (4%), and others. Sentinel-2, Sentinel-1, and Landsat satellites emerge as the most used data sources. Benchmark datasets such as ISPRS Vaihingen and Potsdam, LandCover.ai, DeepGlobe, and GID datasets are frequently employed. Model architectures predominantly utilize encoder–decoder and hybrid convolutional neural network-based structures because of their impressive performances, with limited adoption of transformer-based architectures due to its computational complexity issue and slow convergence speed. Lastly, this paper highlights existing key research gaps in the field to guide future research directions.

1. Introduction

Semantic segmentation models are crucial in land cover mapping especially in generating precise Land Cover (LC) maps [1]. LC maps show various types of land cover, such as forests, grasslands, wetlands, urban areas, and bodies of water. These maps are typically created using Remote Sensing (RS) data like satellite imagery or aerial photography [2]. Land cover maps serve for different purposes, including land use management [3], disaster management, urban planning [4], precision agriculture [5], forestry [6], building infrastructure development [7], climate changes problems [8], and others.

Due to advancements in Deep Convolutional Neural Network (DCNN) models, the domain of land cover mapping has progressively evolved [9]. DCNN are potentially successful for extracting information from high-resolution RS data [10]. They possess deep layers and hierarchical architectures, aiming to automatically identify high-level patterns in data [11]. Although DCNN have shown impressive performance in the image classification task, the conventional models still struggle to capture comprehensive global information as well as long-range dependencies inherent to RS data [12]. Consequently, they may not achieve precise image segmentation because they could potentially overlook certain edge details of objects [13]. However, they have successfully contributed valuably to semantic segmentation methodologies.

The semantic segmentation model allocates every pixel in an image to a predefined class [14]. Its effectiveness in landcover segmentation has established it as a mainstream method [15]. It has demonstrated an enhanced performance leading to more accurate segmentation outcomes. A prominent example of a state-of-the-art (SOTA) semantic segmentation model is UNet [16]. Recently, many novel semantic segmentation models tailored for land cover mapping have been proposed, including DFFAN [12], MFANet [17], Sgformer [18], UNetFormer [19], and CSSwin-unet [20]. Accordingly, these models have demonstrated exceptional segmentation accuracy in this domain.

In recent years, literature reviews have predominantly explored Deep Learning (DL) semantic segmentation models. The review by [21] presented significant methods, their origins, and contributions, including insights into datasets, performance metrics, execution time, and memory consumption relevant to DL-based segmentation projects. Similar reviews by [14,22] categorized existing semantic segmentation with DL methods based on criteria like supervision degree during training and architectural design. In addition, ref. [23] summarized various semantic segmentation models for RS Imagery. These reviews offer comprehensive overviews of DL-based semantic segmentation models but have not specifically examined their application to land cover mapping. To address this gap, this literature review focuses on emerging semantic segmentation models in land cover mapping, aiming to answer predefined research questions quantitatively and qualitatively. Our objective is to identify knowledge gaps in semantic segmentation models applied to land cover mapping and understand the evolution of these models in relation to domain-specific studies, data sources, model structures, and performance metrics. Furthermore, this review offers insights for future research directions in land cover mapping.

The next sections of this review are structured as follows: Section 2 provides the research questions and method used in conducting the review. Section 3 delves into the results obtained from the performed systematic literature review, discusses the evolution and trends, the domain study, the data, and semantic segmentation methodologies. Section 4 challenges and provides future insights into land cover mapping. Finally, Section 5 summarizes the highlights of the review.

2. Materials and Methods

The methodology used in conducting this comprehensive literature review follows the PRISMA framework as outlined by [24] of identification, eligibility, screening, and data extraction. A search strategy was developed to identify the literature for this review (Figure 1). Peer-reviewed papers published in relevant journals between 2017 and 2023 are reviewed. In this section, the research questions are formulated, thereafter, the search strategy was defined using the Scopus database (https://scopus.com accessed on 23 February 2024). The selection of the inclusion and exclusion criteria was explained as well as the eligibility criteria defined to assess articles utilized as the final record for the bibliometric analysis.

2.1. Research Questions (RQs)

The objectives of the study are addressed based on 4 main research questions. These RQs are specifically selected to elicit trends, benchmark datasets, the state-of-the-art architecture, and performances of semantic segmentation in land cover mapping. This review is built around the following RQs:

• RQ1. What are the emerging patterns in land cover mapping?
• RQ2. What are the domain studies of semantic segmentation models in land cover mapping?
• RQ3. What are the data used in semantic segmentation models for land cover mapping?
• RQ4. What are the architecture and performances of semantic segmentation methodologies used in land cover mapping?

2.2. Search Strategy

The search strategy was carried out using Scopus (Elsevier) database. This database is renowned in the scientific community for its high-quality journals, extensive, multidisciplinary abstract collection, and it is an excellent fit for the purposes of review articles [25]. The search keywords used as search criteria to identify articles from the Scopus database are ‘semantic AND segmentation’, AND ‘land AND use AND land AND cover’, AND ‘deep AND learning’, AND ‘land AND cover AND classification’. The search string is used to find relevant papers that link deep learning semantic segmentation to land use and land cover.

2.3. Study Selection Criteria

After the defined search strings were entered on Scopus, a total of 218 articles were initially retrieved from the scientific database. To further process the data retrieved, the papers are filtered between 2017 and 2023 excluding conference papers, conference reviews, data paper, books, and book chapters, articles in the press, conference proceedings, book series and articles that are not in English language. The stated period 2017–2023 is selected to provide us with the recent developments in the field and during search, there were almost no notable articles published before 2017 on the subject matter. Consequently, the retrieved record was reduced to 121 articles.

2.4. Eligibility and Data Analysis

To determine eligibility and quality assessment of the extracted papers, we remove articles published in journals that are not Q1 or Q2. This is to ensure that the articles with a rigorous peer-review process and quality research output in this field are selected and synthesized for a top-quality review. Following this criteria, 9 articles are excluded, and 112 records are retrieved. For further assessment, titles and abstracts are assessed regarding their relevance to the study. The relevant study focused on articles that implemented various deep learning semantic segmentation models focused on land use/land cover classification, and/or various satellite datasets extracted through semantic segmentation in different applications. At the end of this step, 6 articles are further excluded bringing down the record to a total number of 106 articles between the period of 2017 to 2023, which are eligible for bibliometric analysis.

2.5. Data Synthesis

Data synthesis is an important way to the answer the RQs. The data are visualized in such a way that it presents findings and synthesizes through quantitative and qualitative analysis. Categorization and visualization were carried out to draw important trends and findings of deep learning semantic segmentation models in land cover mapping in relation to datasets, applications, architecture, and performance. The study’s synthesizing themes were developed by full-text content analysis of the 106 articles. Vosviewer software version 1.6.18 (https://vosviewer.com accessed on 24 February 2024) was used to provide graphical visualizations of occurrence of key terms, taking title, abstract, and keywords as input.

3. Results and Discussion

This section investigates the results derived from the performed systematic literature review. Using 106 extracted articles, the results are presented and discussed based on the RQs.

3.1. RQ1. What Are the Emerging Patterns in Land Cover Mapping?

• Annual distribution of research studies

Figure 2 shows the number of research articles published annually from 2017 to 2023. The year 2017 saw a modest output of merely two articles. There is a surge in the number of articles to 12 in 2020, 17 articles by the year 2021, 30 articles by 2022 and 40 articles by 2023. This observation aligns with the understanding that the adoption of deep learning semantic segmentation models on satellite imagery gained significant momentum in 2020 and subsequent years.

• Leading Journals

Figure 3 depicts the number of articles published in academic journals of this domain. The top 13 journals produced over 81% of the number of research studies of semantic segmentation in land cover mapping. MDPI Remote Sensing (30) has the highest number of published articles in this domain, followed by IEEE Journal of selected topics in Earth Science (15), IEEE Transactions on Geoscience And Remote Sensing (12), IEEE Geoscience and Remote Sensing Letters (5), ISPRS Journal Of Photogrammetry And Remote Sensing (4), and so on, while 20 other journals have one article each published and are grouped as the “other” category.

• Geographic distribution of studies

In terms of geographical distribution of studies extracted from the Scopus database, 35 countries contributed to the study domain. Almost all continents have contributions, except for the African continent. Figure 4 shows that China published 63 articles of the total 106 articles, the second is the United States with 10 articles, followed by India (6), Italy (5), South Korea (4), the United Kingdom(4), Canada, Finland, and Germany (3), then Austria, Australia, Brazil, France, Greece, Turkey, the Netherlands, and Japan (2) while the rest of the countries have one each.

• Leading Themes and Timelines

The significant keyword occurrences are obtained from the titles and abstracts of the extracted articles. Figure 5 shows relevant and leading keywords. A threshold of five was set, which means the minimum number of occurrences of a keyword. Only 68 out of 897 keywords met the threshold. Bibliometric analysis reveals that keywords such as "high-resolution RS images", "Remote Sensing", "satellite imagery" and "very high resolution" exhibit prominence, showing strong associations with neural network-related terms including "Semantic Segmentation", "Deep Learning,", "Machine Learning", and "Neural Network". The Semantic Segmentation has “attention mechanisms” and “transformer” as different model’s architectural component. These learning models are further linked to various application domains, evident in their connections to terms like "Land Cover Classification", "Image Classification", "Image Segmentation", "Land Cover", "Land Use “, “Change Detection”, and "Object Detection." In 2020, the research revolved around network architectures, object detection, and image processing. In the later part of 2021, there was a notable shift in research domains, predominantly towards image segmentation, image classification, and land cover segmentation. In 2022 and 2023, there was a pronounced shift in research focusing more on semantic segmentation employing satellite high-resolution images for change detection, land use, and land cover classification and segmentation.

3.2. RQ2. What Are Domain Studies of Semantic Segmentation Models in Land Cover Mapping?

In this section, each extracted paper is clustered based on similar study domain areas. Figure 6 shows the overall mind map of the domain studies. Land cover, urban, precision agriculture, environment, coastal areas, and forest are the mostly studied domain areas.

Figure 7 shows that among the 106 articles, 36 studies cover both land cover classification and segmentation, 24 specifically focus on land cover classification, 15 concentrate on urban applications, 9 address land cover segmentation alone, 8 addresses environment issues, 5 center around precision agriculture, 5 are oriented toward coastal applications, 3 articles addressing land cover change detection and 2 focus on forestry.

• Land Cover Studies

In the 72 out of 106 studies related to land cover, research activities encompass land cover classification (33.3%), land cover segmentation (12.5%), change detection (4.2%), and the combined application of land cover classification and segmentation (50%). These areas of study are extensively documented and represent the most widely researched applications within land cover studies. Land cover classification involves assigning an image to one of several classes of land use and land cover (LULC), while land cover segmentation entails assigning a semantic label to each pixel within an image [26]. Land cover refers to various classes of biophysical Earth cover, while land use describes how human activities modify land cover. On the other hand, change detection plays a crucial role in monitoring LULC changes by identifying changes over time periods, which can help predict future events or environmental impacts. Change detection methods employing DL have attained remarkable achievements [27] across various domains, including urban change detection [4], agriculture, forestry, wildfire management, and vegetation monitoring [28].

• Urban

Among the 15 publications related to urban studies, 38% focus on segmentation applications, including urban scene segmentation [19,29,30], while 31% address urban change detection for mapping, planning, and growth [31,32]. For instance, change detection techniques provide insights into urban dynamics by identifying changes from remote sensing imagery [7], including changes in settlement areas. Additionally, these studies involve predicting urban trends and growth over time, managing land use [3], and monitoring urban densification [33] as well as mapping built-up areas to assess human activities across large regions [34].

Publications in urban studies also cover 6% in land survey management [35] and 25% in urban classification and detection [36,37,38,39,40], particularly in building applications and for urban land-use Classification [40].

• Precision Agriculture

Out of the five publications concerning precision agriculture studies, the research focuses on various aspects such as crop mapping [41], identification [42,43], classification [44], and monitoring [45]. For example, mapping large-scale rice farms [5], monitoring crops to analyze different growth stages [42], and classifying Sentinel data for creating an oil palm land cover map [46].

• Environment

Among the eight environmental studies analyzed, the research spans various applications. These include 25% focusing on soil erosion applications [47,48], which involves rapid monitoring of ground covers to mitigate soil erosion risks. Additionally, 12.5% of the studies center around wildfire applications [49], encompassing burned area mapping, wildfire detection [50], and smoke monitoring [51], along with initiatives for preventing wildfires through sustainable land planning [8]. Another 12.5% of the studies involve haze classification [52], specifically cloud classification using Sentinel-2 imagery. Climate change research [53] accounts for another 12.5% of the studies, focusing on aspects such as the urban thermal environment. Furthermore, 12.5% of the studies are dedicated to vegetation classification [54] and an additional 25% address mining applications [55] including the detection of changes in mining areas [56].

• Forest

Research in this domain encompasses forest classification, including the classification of landscapes affected by deforestation [57]. More so, change detection in vegetation and forest areas enables decision-makers, conservationists, and policymakers to make informed decisions through forest monitoring initiatives [6] and mapping strategies tailored to tropical forests [58].

• Coastal Areas

Within this field, 5 studies out of the 106 articles focus on wetland mapping, classification, and segmentation [10,59,60,61,62]. However, the exploration and study of coastal area remote sensing image segmentation remains a relatively underexplored research area, as noted by [60]. This challenge is primarily attributed to the significant complexities associated with coastal land categories, including issues such as homogeneity, multiscale features, and class imbalance, as highlighted [63].

3.3. RQ3. What Are the Data Used in Semantic Segmentation Models for Land Cover Mapping?

In this section, this paper synthesizes extensively employed, particularly for the study location, data source and benchmark datasets used for land cover mapping.

• Study Locations

Figure 8 illustrates the countries where the study areas were located and where in-depth research was conducted among the extracted articles. Among the 22 countries represented in 51 studies, 35.29% of the study areas are located in China, with 11.76% in the USA, 5.88% in France, and 3.92% in Spain, Italy, Brazil, South Korea, and Finland each. Other countries in the chart each account for 1.96% of the study areas.

• Data Sources

Table 1. Data sources identified in review.

Data Sources	Number of Articles	References
RS Satellites
Sentinel-2	7	[45,64,65,66,67]
Landsat	5	[33,68,69,70]
Worldview-03	2	[71,72]
Rapid eye	1	[73]
Worldview-02	1	[74]
Quickbird	1	[74]
ZY-3	1	[48]
PlanetScope	1	[49]
GF-2	2	[48,75]
Aerial images
Phantom m multi-rotor AUS	1	[59]
Quadcopter drone	1	[61]
Vexcel Ultracam Eagle Camera	1	[76]
DJI-Phantom 4 pro UAV	1	[47]
SAR SAT
RADARSAT-2	1	[77]
Sentinel-1	6	[10,41,43,64,65,78]
GF-3	1	[79]
ALOS-2	1	[80]
Others
Earth digitalglobe	2	[44,60]
Mobile phone	1	[35]
Lidar Sources	1	[37]

presents the identified data sources along with the number of articles and their corresponding references. The data sources identified in the literature include RS satellites, RS Unmanned Aerial Vehicles (UAVs) and Unmanned Aircraft Systems (UAS), mobile phones, Google Earth, Synthetic Aperture Radar (SAR), and Light Detection and Ranging (LiDAR) sources. Among these, Sentinel-2, Sentinel-1, and Landsat satellites are the most frequently utilized data sources. It is important to recognize that the primary RS technologies include RS satellite imagery, SAR, and LiDAR.

RS Imagery data are the most extensively utilized in land cover mapping. In RS, data are collected from satellite sources such as Sentinel-1, Landsat, Sentinel-2, WorldView-2, and QuickBird at certain time step intervals for a period. The data capture products of these satellite include Panchromatic (1 Channel – 2D), Multispectral, or Hyperspectral images [1]. RSI can be represented as aerial images [2]; these are taken using Drone or UAVs. These data usually possess spatial resolution and spectral resolution of certain image sizes.

SAR is among the radar systems used in Land Cover Mapping; SAR stands out as a notable data source [81]. SAR utilizes radio detection technology and constitutes an essential tool in this field. SAR data carry distinct advantages, especially in scenarios where optical imagery faces limitations such as cloud cover or limited visibility. SAR can penetrate through cloud cover and offer Earth surface imaging even in the presence of clouds or unfavorable weather conditions. This is one of the key advantages of SAR technology, unlike electromagnetic spectrum which is obstructed by clouds [82].

There are different types of SAR identified in the literature based on different parameters harnessed for the purpose of land cover mapping, such as polarimetric synthetic aperture radar (PolSAR) [80], E-SAR, AIRSAR, and Interferometric Synthetic Aperture Radar [6]. Notable benchmark SAR datasets include Gaofen-3 and RADARSAT-2 datasets [83]. At present, semantic segmentation of PolSAR images holds significant utility in the interpretation of SAR imagery, particularly within agricultural contexts [84]. Similarly, the High-Resolution GaoFen-3 SAR Dataset is useful for the Semantic Segmentation of Building [34,79,85]. The benchmark dataset Gaofen-3 (GF-3), comprised of single-polarization SAR images, holds significant importance [86]. This dataset is derived from China’s pioneering civilian C-band polarimetric SAR satellite, designed for high-resolution RS. Notably, FUSAR-Maps are generated from extensive semantic segmentation efforts utilizing high-resolution GF-3 single-polarization SAR images [87], while the GID dataset is collected from the Gaofen-2 satellite.

LiDAR holds a significant role within the sphere of land cover mapping and climate change [88]. LiDAR involves the emission of laser pulses and the measurement of their return times to precisely gauge distances, creating highly accurate and detailed elevation models of the Earth’s surface. It provides detailed information, including topographic features, terrain variations, and the vertical structure of vegetation. It stands as an indispensable data source for land cover mapping endeavors. Notable examples include the utilization of multispectral LiDAR [54], an advanced RS technology merging conventional LiDAR principles with the capacity to concurrently capture multiple spectral bands. There’s the Follo 2014 LiDAR data, a dataset that specifically captures Light Detection and Ranging (LiDAR) data in the Follo region during 2014. Additionally, the NIBIO AR5 (Norwegian Institute of Bioeconomy Research—Assessment Report 5) Land Resources dataset, developed by the Norwegian Institute of Bioeconomy Research, represents a comprehensive evaluation of land resources. This dataset encompasses a range of attributes including land cover, land use, and pertinent environmental factors [89].

• Benchmark datasets

The benchmark datasets used for evaluation in this domain as identified in the review are shown in Figure 9. ISPRS Vaihingen and Potsdam are the widely used benchmark datasets, followed by GID, Landcover.ai, DeepGlobe, and WHDLD.

The ISPRS Vaihingen comprises 33 aerial image patches in IRRGB format along with their associated Digital Surface Model (DSM) data, each with a size of around 2500 × 2500 pixels at 9 cm spatial resolution [90,91]. Similarly, the publicly accessible ISPRS Potsdam dataset encompasses Potsdam city, Germany. It is composed of 38 aerial IRRGB images measuring 6000 × 6000 pixels each at a spatial resolution of 5 centimeters [92].

The Gaofen Image Dataset (GID) dataset [93] contains 150 images of GaoFen-2 data (GF-2 images) [94,95] collected from different cities in China. The GaoFen-2 (GF-2) is high-resolution images acquired from the GaoFen-2 satellite, which captures Panchromatic (PAN) and Multispectral (MS) Spectral Bands. It offers valuable insights into land cover characteristics and patterns. Every image is composed of pixels measuring 6908 × 7300 and R, G, B, and NIR bands, each with a spatial resolution of 4 meters. The GID dataset consists of five land-use categories: farmland, meadow, forest, water, and built-up [96].

The LandCover.ai dataset [97] comprises images chosen from aerial photographs encompassing 216.27 square kilometers of Poland. The dataset includes 41 RS images, with 33 images having an approximate resolution of 25 cm, measuring around 9000 × 9000 pixels, and 8 images with a resolution of approximately 50 cm, spanning about 4200 × 4700 pixels [12,17,98]. The dataset was manually categorized into four types of objects such as buildings, woodland, and water, as well as the background.

DeepGlobe Data [99] is another important dataset in land cover mapping. The dataset stands out as the inaugural publicly available collection of high-resolution satellite imagery primarily emphasizing urban and rural regions. This dataset comprises a total of 1146 satellite images, each with dimensions of 20448 × 20448 [100]. It is of great importance to the Land Cover Classification Challenge. Likewise, the Inria dataset [101] consists of aerial visual images encompassing 10 regions in the United States and Austria, collected at a 30 cm resolution, with RGB bands [102]. It is organized by five cities in both in training and test data. Every city includes 36 image tiles, each sized at 5000 × 5000 pixels, and these tiles are divided into two semantic categories: buildings and non-building classes.

In addition, the Disaster Reduction and Emergency Management Building dataset exhibits a notable similarity to the Inria dataset. It has image tiles size of 5000 × 5000 with a spatial resolution of 30 cm, all the tiles contain R, G, and B bands [96]. The building dataset from Wuhan University comprises an aerial dataset encompassing 8189 image patches captured at a 30 cm resolution. These images are in RGB format, with each patch measuring 512 × 512 pixels [103]. The Aerial Imagery for Roof Segmentation dataset [104] is composed of aerial images that encompass the Christchurch city area in New Zealand. These images are captured at a resolution of 7.5 cm and include RGB bands [102]. It was captured following the seismic event that impacted the town of Christchurch in New Zealand. Four images, each with dimensions of 5000 × 4000 pixels, were labeled to include the following categories: buildings, cars, and vegetation [105]. Other benchmark datasets include the Massachusetts building and road datasets [106], Dense labeling RS dataset [107], VEhicle Detection in Aerial Imagery (VEDAI) dataset, and LoveDA dataset [108].

3.4. RQ4. What Are the Architecture and Performances of Semantic Segmentation Methodologies Used in Land Cover Mapping?

This section investigates the design and effectiveness of recent advancements in novel semantic segmentation methodologies applied to land cover mapping. In this paper, the methodologies employed in land cover mapping are classified based on similarities in their structural components. We have identified three primary architectural structures: (i) encoder-decoder structures; (ii) transformer structures; and (iii) hybrid structures. Figure 10 shows that among 80 articles employing different model structures, 59% utilized hybrid structures, 36% utilized encoder-decoder structures, and 5% utilized transformer-based structures.

• Encoder-Decoder based structure

Encoder-decoder structures consist of two main parts: an encoder that processes the input data and extracts high-level features, and a decoder that generates the output (e.g., segmentation map) based on the encoder’s representations [16]. The authors [109] suggested an innovative encoding-to-decoding technique known as the Full Receptive Field network, which utilizes two varieties of attention mechanisms, with ResNet-101 serving as the fundamental backbone. Similarly, a different DL segmentation framework known as the DGFNET Dual-gate fusion network. Paper [110] adopts an encoder-decoder architecture design. Typically, encoder-decoder architectures encounter difficulties with the semantic gap. To address this, the DGFNET framework comprises two modules: the Feature Enhancement Module as well as the Dual Gate Fusion Module to mitigate the impact of semantic gaps in deep convolutional neural networks, leading to improved performance in land cover classification. The model underwent evaluation using both the landcover dataset and the Potsdam dataset, achieving MIoU scores of 88.87% and 72.25%, respectively.

The article [111] proposed U-Net incorporating asymmetry and fusion coordination. It is an encoder and decoder architecture with an integrated coordinated attention mechanism, a non-symmetric convolution block refinement fusion block that gets long term dependencies and intricate information from RS data. It was reported that the method was evaluated on DeepGlobe datasets and performed best with an MIoU of 85.54% as reported compared to other models like UNet, MAResU-Net, PSPNet, DeepLab v3+, etc. However, the model has low network efficiency and is not recommended for mobile applications. Also, ref. [112] suggested the Attention dilation-LinkNet neural network, which contains an encoder-decoder structure. It takes advantage of serial-parallel combination dilated convolution and two channel-wise attention mechanisms, as well as a pretrained encoder to be useful for satellite image segmentation particularly road extraction. The best performance of an ensemble of the model achieved an IoU of 64.49% on DeepGlobe road extraction dataset.

Table 2. Encoder-decoder-based semantic segmentation models for land cover segmentation.

Models	Datasets	Performance Metrics	Limitation/Future Work
RAANet [108]	LoveDA, ISPRS Vaihingen	MIoU = 77.28, MIoU = 73.47	Accuracy can be improved with optimization.
PSE-UNet Model [113]	Salinas Dataset	MIoU = 88.50	Inaccurate segmentation of land cover features with low frequencies, superfluous parameter redundancy, and unvalidated generalization capabilities.
SEG-ESRGAN [114]	Sentinel-2 and WorldView-2 image pairs.	MIoU = 62.78	The assessment of utilizing medium-resolution images has not been tested
Class-wise FCN [26]	Vaihingen, Potsdam	MIoU = 72.35, MIoU = 76.88	Enhancements in performance can be achieved through class-wise considerations for multiple classes, along with improved and more efficient implementations.
MARE [115]	Vaihingen	MIoU = 81.76	Improve performance through parameter optimization and extend approach incorporating other self-supervised algorithms.
Feature fusion with dual attention and flexible contextual adaptation [94]	Vaihingen, GaoFen-2	MIoU = 70.51, MIoU = 56.98	Computational complexity issue.
Deanet [100]	LandCover.ai, DSTL dataset, DeepGlobe	MIoU = 90.28, MIoU = 52.70, MIoU = 71.80	Suboptimal performance. Future efforts involve incorporating the spatial attention module into a single unified backbone network.
An encoder-decoder framework featuring attention-guided multi-scale context integration [116]	GF-2 images	MIoU = 62.3%	Reduced accuracy on imbalance data.

tabulates some semantic segmentation models using the encoder-decoder structure. It shows that these models have relatively impressive generalization performances on different data; however, accuracy can be further enhanced through parameter optimization.

• Transformer-based structure

Transformer-based architectures are neural network structures originally designed for natural language processing (NLP), utilizing transformer modules as their fundamental building components. In the context of land cover mapping tasks, transformer-based architectures such as the Swin-S-GF [117], BANet [30], DWin-HRFormer [29], spectral spatial transformer [118], Sgformer [18], and Parallel Swin Transformer have been developed.

Table 3. Transformer-based semantic segmentation models for land cover segmentation.

Models	Data	Performance	Limitation
Swin-S-GF [117],	GID	OA = 89.15 MIoU = 80.14	Computational complexity issue and slow convergence speed.
CG-Swin [119]	Vaihingen, Potsdam	OA = 91.68 MIoU = 83.39, OA = 91.93 MIoU = 87.61	Extending CG-Swin to accommodate multi-modal data sources for more comprehensive and robust classification.
BANet [30]	Vaihingen, Potsdam, UAVid dataset	MIoU = 81.35, MIoU = 86.25, MIoU = 64.6	Combine convolution and Transformer as a hybrid structure to improve performance.
Spectral spatial transformer [118]	Indian dataset	OA = 0.94	Computational complexity issue
Sgformer [18]	Landcover dataset	MIOU = 0.85	Computational complexity issue and slow convergence speed.
Parallel Swin Transformer [120]	Postdam, GID WHDLD	OA = 89.44, OA = 84.67, OA = 84.86	Performance can be improved.

presents various transformer-based structures alongside their performance metrics and limitations. Researchers have noted that while these architectures achieve effective segmentation accuracy with an average OA of approximately 89%, transformers can exhibit slow convergence and be computationally expensive, particularly in land cover mapping tasks. This limitation contributes to their relatively low adoption in land cover segmentation applications.

• Hybrid-based structure

A hybrid-based structure combines elements from different neural network architectures or techniques to create a unified model for semantic segmentation. Traditional convolutional neural network methods face limitations in accurately capturing boundary details and small ground objects, potentially leading to the loss of crucial information. While deep convolutional neural networks are applied for classifying land use covers, results often show suboptimal performance in the land cover segmentation task [73]. However, this result can be tackled by a hybrid through introduction of encoder-decoder style semantic segmentation models, leverage existing deep learning backbone [68], and explore diverse data settings and parameters in their experimentation [121]. Other methods of the structure’s enhancement include architectural modifications through the integration of attention mechanisms, transformer architecture, module fusion, and multi-scale feature fusion [122,123]. An example is the SOCNN framework [124], which addresses the limitation faced by CNN through module integration: (i) a module for semantic segmentation; (ii) a module for superpixel optimization; and (iii) a module for fusion. While the evaluated performance of the framework demonstrated improvement, further enhancement in boundary retrieval can be achieved by incorporating superior boundary adhesion and integrating it into the boundary optimization module.

Moreso, paper [125] proposed multi-level context-guided classification method object-based CNN. It involves high-level feature-fusing and employed a Conditional Random Field for better classification performance. The model attained a comparable overall accuracy with DeepLabV3+ at various segmentation scale parameters on the Vaihingen dataset and suboptimal overall accuracy compared to DeepLabV3+ on the Potsdam dataset. Another approach identified is utilizing a Generative Adversarial Network-based approach for domain adaptation, such as the Full Space Domain Adaptation Network [103], as well as leveraging domain adaptation and transfer learning [126]. It has been proven to enhance accuracy in scenarios where source and target images originate from distinct domains. However, the domain adaption segmentation using RS images remains largely underexplored [127]. The authors [92] presented a CNN-based SegNet model that classifies terrain features using 3D geospatial data; the model did better on building classification than other natural objects. The model was validated on the Vaihingen dataset and tested on the Potsdam dataset, achieving an IoU of 84.90%.

In addition, ref. [128] proposed SBANet, which stands for Semantic Boundary Awareness Network used to extract sharp boundaries, and ResNet was employed as the backbone. Subsequently, it was enhanced by introducing a boundary attention module and applying adaptive weights in multitask learning to incorporate both low- and high-level features, with the goals of improving land-cover boundary precision and expediting network convergence. The method was evaluated on the Potsdam and Vaihingen semantic labelling datasets; they reported that SBANet performed best compared to models like UNet, FCN, SegNet, PSPNet, Deeplab3+, and others. The DenseNet-based model [129], a proposed method, modified one of the DL backbones DenseNet by adding two novel fusions that is the unit fusion and cross-level fusion. The unit fusion is detailed-oriented fusion, and the other integrates different information levels. This model with both fusions performed best on the DeepGlobe dataset.

Furthermore, paper [93] suggested a bidirectional grid fusion network, a two-way fusion architecture for classifying land in very high-resolution RS data. It encourages bidirectional information flow with mutual benefits of feature propagation, and a grid fusion architecture is attached for further improvement. The best refined model was tested on ISPRS and GID datasets, achieving MIoU performances of 68.88% and 64.01%, respectively.

Table 4. Hybrid-based semantic segmentation models for land cover segmentation.

Models	Datasets	Performance Metrics	Limitation
RSI-Net [95]	Vaihingen, Potsdam, GID	OA = 91.83, OA = 93.31, OA = 93.67	Limitation in segmentation of pixel-wise semantics. Enhanced feature map fusion decoders can lead to performance improvements.
HMRT [32]	Potsdam	OA = 85.99 MIoU = 74.14	Parameter complexity issue, decrease in segmentation accuracy due to a lot of noise. Optimization is required.
UNetFormer [19]	UAVid, Vaihingen, Potsdam, LoveDA	MIoU = 67.8, OA = 91.0 MIoU = 82.7, OA = 91.3 MIoU = 86.8, MIoU = 52.4	Investigate the Transformer’s potential and practicality in addressing geospatial vision tasks is open for research.
(TL-ResUNet) model [130]	DeepGlobe	IoU = 0.81	Improve classification performance is open for research, and developing real time and automated solution for land use land cover.
CNN-enhanced heterogeneous GCN [131]	Beijing dataset, Shenzhen dataset.	MIoU = 70.48, MIoU = 62.45	Future endeavor is to optimize the utilization of pretrained deep CNN features and GCN features across various segmentation scales.
HFENet [132]	MZData, LandCover Dataset, WHU Building Dataset	MIoU = 87.19, MIoU = 89.69, MIoU = 92.12	Time and space complexity issues. Future work can be to automatically fine-tune the parameters to attain the optimal performance of the model.

shows some identified hybrid semantic segmentation models and performance metric in land cover mapping. These models have demonstrated effective performances, with an average overall accuracy of 91.3% across presented datasets.

• Performance analysis of semantic segmentation model structures on ISPRS 2-D labelling Potsdam and Vaihingen datasets

Our study examines the performance of various semantic segmentation model structures using the most widely utilized datasets in land cover mapping. The models are categorized based on the three primary architectural structures, and their performances are averaged across ISPRS benchmark datasets.

Figure 11 and Figure 12 show that semantic segmentation models demonstrate impressive generalization capabilities across all ISPRS land cover categories. Among the different structures, the encoder-decoder-based models performed the least well across ISPRS land cover categories. On the Potsdam dataset, hybrid models outperformed others across all landcover categories. For the Vaihingen data, Transformer-based models slightly outperformed other structures in classifying buildings, low vegetation, and trees. Notably, all three model structures showed the highest effectiveness in the building category, with the least generalization performance observed in either the cars or low vegetation categories. Semantic segmentation models perform better in categorizing buildings compared to natural objects such as low vegetation and forests in remote sensing (RS) applications. This supports the authors’ assertion [92] that their model is more effective in man-made features due to its geometric properties than in natural features.

• Common experimental training settings

The common experimental training settings found in studies are detailed in

Table 5. Frequently used and common hyper-parameter values for segmentation models structures.

Model’s Structures	Batch Size	Epochs	Learning Rate	Data Augmentation	Backbone	Popular Optimizer	Parameters	Evaluation Metrics
Encoder/decoder-based	4, 8, 16, 64	100–500	0.01	Yes	ResNet	SGD	Low–High	MIoU, OA, F1
Transformer-based	6, 8	100–200	0.0006	Yes	ResNet/Swintiny	Adam	High	MIoU, OA, F1
Hybrid models	8, 16	40–100	0.0006	Yes	ResNet	Adam	Low–High	MIoU, OA, F1

.

Table 5 shows that all three model structures typically utilize data augmentation during pre-processing. Both the encoder-decoder and hybrid structures use the ResNet model as their backbone, while the Transformer-based structure employs either the ResNet or Swintiny model. For evaluation, commonly used metrics include Mean Intersection over Union (MIoU), Overall Accuracy (OA), and F1 score, among others. It is notable that the Adam optimizer is frequently used for both Transformer-based and hybrid structures, as it helps to improve convergence speed. This is particularly important for Transformer-based structures, which have a high number of parameters that tend to slow down convergence.

4. Challenges, Future Insights and Directions

In this section, we highlight the challenges in land cover mapping and semantic segmentation methodologies in the context of land cover mapping, as revealed through an extensive review of the existing literature. We found persistent gaps requiring targeted attention and innovative solutions in forthcoming research efforts. This section examines these challenges and outlines potential avenues for future investigation.

4.1. Land Cover Mapping

In land cover mapping, the key challenges include extracting land cover boundary information and generating precise land cover maps.

• Extracting boundary information

The precise delineation of sharp and well-defined boundaries [133], the refinement of object edges [134], the extraction of boundary details [128], and the acquisition of content details [135] from RS Imagery, all aimed at achieving accurate land cover segmentation. This research gap remains an area that needs further exploration.

Defining clear land cover boundaries is a crucial aspect of RS and geospatial analysis. It involves precisely delineating the borders that divide distinct land cover categories on the Earth’s surface. Semantic segmentation encounters performance degradation due to the loss of crucial boundary information [124]. The challenge of delineating precise boundaries in land cover maps is exacerbated by the heightened probability of prediction errors occurring at borders and within smaller segments [19,136]. This holds particularly true in scenarios like the segmentation and classification of vegetation land covers [54]. An illustrative example of this complexity is encountered when attempting to segment the boundaries among ecosystems characterized by a combination of both forest and grassland in regions with semi-arid to semi-humid climates [137].

Unclear boundaries, and loss of essential and detailed information at boundaries decreases the chance of producing fine segmentation results [128]. The task of effectively capturing entire and well-defined boundaries in intricate RS images of very high resolution remains open for further research and improvement.

• Generating Precise Land Cover Maps

The demand for accurate and timely high-resolution land cover maps is high and are of immediate importance to various sectors and communities [125]. The creation of precise land cover maps holds significant value for subsequent applications. These applications encompass a diverse range of tasks, including vehicle detection [105], the extraction of building footprints [138], building segmentation [32], road extraction, surface classification [112], determining optimal seamlines for orthoimage mosaicking within settlements [139], and land consumption [140]. This offers valuable assistance in the monitoring and reporting of data within rapidly changing urban regions. However, it has been reported that generating and automating accurate land cover maps still present a formidable challenge [141,142].

4.2. Semantic Segmentation Methodologies

The key challenges of semantic segmentation methodologies in land cover mapping, as identified in reviewed articles, includes enhancing semantic segmentation models performances, improving RS Images analysis using semantic segmentation models, addressing imbalance and unlabeled RS data problem.

• Enhancing deep learning model performance

Improving deep learning semantic segmentation architecture is a notable research gap in land cover mapping. This gap is characterized by the ongoing need to advance the capabilities of these models to address complex challenges emanating in land cover mapping such as improved performance on natural images and convergence speed. Various studies have critically compared the accuracy assessment and effectiveness of models, both on natural images and RS imagery, aiming to enhance their generalization performance [10] and convergence speed. Moreover, multiple studies such as [143] have modified the semantic segmentation model to improve generalization performance for land cover mapping.

• Analysis of RS images

In terms of RS data, extracting information from RS imagery data remains a challenge. The following factors contribute to inaccurate RS classification, namely the complexities inherent in deciphering intricate spatial and spectral patterns within RS Imagery [13,125], the challenge of handling diverse distributions of ground objects with variations within the same class [26] and significant intra-class and limited inter-class pixel differences [144,145,146]. Other contributing factors include data complexity, geographical time difference [147], foggy conditions [148], and data acquisition errors [141]. More so, several studies have implemented works for very high RS resolution images. However, only few studies have focused on low and medium-resolution images [68,75,114,149]. As a future insight, it is recommended to conduct more research using fast and efficient DL methods for low and medium resolution RS.

Another area is the analysis of SAR Images. SAR images is extremely important for many applications especially in Agriculture. Researchers find it very challenging classifying SAR data and the segmentation is poorly understood [84]. Some studies have undertaken land cover classification and segmentation tasks across diverse categories of SAR data [86], including polarimetric SAR imagery [77,84,85], single-polarization SAR images [87], and multi-temporal SAR data [81]. As a consideration, we recommend that a roadmap for simplified and automated semantic segmentation of SAR images should be investigated.

In LiDAR data analysis, ref. [89] pioneered novel deep learning architectures designed specifically for land cover classification and segmentation, which were extensively validated using airborne LiDAR data. Additionally, ref. [150] developed DL models that synergistically leverage airborne LiDAR data and high-resolution RS Imagery to achieve enhanced generalization performance. Beyond RS Imagery and LiDAR, semantic segmentation is applied to high-resolution SAR images [79,83] and aerial images of high resolution [2,151].

• Unlabeled and Imbalance RS data

A large majority of RS images lack high-quality or are largely unlabeled [152], with weak and missing annotations that reduce the model’s performances. The efforts to obtain well-annotated RS data are expensive, laborious, and highly time-consuming. This affects generating accurate land use maps, thereby impacting the generation of precise land cover maps negatively. To mitigate this challenge, exploration into study areas like domain adaptation techniques [153] and the application of DL models can be considered to facilitate the creation of accurate land cover maps [154]. Another suggested approach for addressing the shortage of well-annotated data is to utilize networks capable of utilizing training labels derived from lower resolution land cover data [155]. Furthermore, it is advisable to harness the benefits of multi-modal RS data, as its potential to enhance model performance particularly in situations with limited training samples has not been fully realized [156]. To overcome the laborious and time-consuming process of manually labeling data, certain studies such as [148,157] introduced DL models that addressed this issue. The authors [135] proposed the effectiveness of semi-supervised adversarial learning methods for handling limited and unannotated high-resolution satellite images. Furthermore, the investigations by [39,149,158,159] sought remedies for challenges stemming from the scarcity of well-annotated pixel-level data and as well as other studies proposed steps on how to tackle instances of class imbalance [64,154,160].

Class and sample imbalance of RS images, and data with limited annotation cause performance degradation, which can lead to poor performance especially in minority classes [161,162].

5. Conclusions

This study conducted an analysis of emerging patterns, performance, applications, and data sources related to semantic segmentation in land cover mapping. The objective is to identify knowledge gaps within this domain and offer readers a roadmap and detailed insights into semantic segmentation for land use/land cover mapping. Employing the PRISMA methodology, a comprehensive review is undertaken to address predefined research questions.

The results reveal a substantial increase in publications between 2020 and 2023, with 81% appearing in the top 13 journals. These studies originate from diverse global institutions, with over 59% attributed to Chinese institutions, followed by the USA and India. Research focuses primarily on land cover, urban areas, precision agriculture, environment, coastal areas, and forests, particularly in tasks such as land use change detection, land cover classification, and segmentation of forests, buildings, roads, agriculture, and urban areas. RS satellites, UAV and UAS, mobile phones, Google Earth, SAR, and LiDAR sources are the major data sources, with Sentinel-2, Sentinel-1, and Landsat satellites being the most utilized. Many studies use publicly available benchmark datasets for semantic segmentation model evaluation. ISPRS Vaihingen and Potsdam being widely employed, followed by GID, Landcover.ai, DeepGlobe, and WHDLD datasets.

In terms of semantic segmentation models, three primary architectural structures are identified and grouped as encoder-decoder structures, transformer structures, and hybrid structures. Based on our research, all semantic segmentation model’s structures demonstrate effective performance in land cover mapping. Hybrid and encoder-decoder structures are most popular due to their impressive generalization capabilities and speed. Transformer-based structures show good generalization but slower convergence. Current research directions and expanding frontiers in land cover mapping emphasize the introduction and implementation of innovative semantic segmentation techniques for satellite imagery in RS. Furthermore, key research gaps identified include the need to enhance model accuracy on RS data, improve existing model architectures, extract precise boundaries in land cover maps, address scarcity of well-labeled datasets, and tackle challenges associated with low and medium-resolution RS data. This study provides useful domain specific information. However, there are some threats to review validity, which include database choice, searched keywords, and classification bias.