SPT-UNet: A Superpixel-Level Feature Fusion Network for Water Extraction from SAR Imagery

Abstract

Extracting water bodies from synthetic aperture radar (SAR) images plays a crucial role in the management of water resources, flood monitoring, and other applications. Recently, transformer-based models have been extensively utilized in the remote sensing domain. However, due to regular patch-partition and weak inductive bias, transformer-based models face challenges such as edge serration and high data dependency when used for water body extraction from SAR images. To address these challenges, we introduce a new model, the Superpixel-based Transformer (SPT), based on the adaptive characteristic of superpixels and knowledge constraints of the adjacency matrix. (1) To mitigate edge serration, the SPT replaces regular patch partition with superpixel segmentation to fully utilize the internal homogeneity of superpixels. (2) To reduce data dependency, the SPT incorporates a normalized adjacency matrix between superpixels into the Multi-Layer Perceptron (MLP) to impose knowledge constraints. (3) Additionally, to integrate superpixel-level learning from the SPT with pixel-level learning from the CNN, we combine these two deep networks to form SPT-UNet for water body extraction. The results show that our SPT-UNet is competitive compared with other state-of-the-art extraction models, both in terms of quantitative metrics and visual effects.

1. Introduction

Water is an indispensable and invaluable resource for human survival and social development. Water bodies extracted from remote sensing images have gained substantial popularity and usage in diverse fields such as surface water management and flood monitoring [1,2,3]. Optical satellites typically provide high spatial resolution images for water extraction, but they are often affected by thick clouds and rainfall, resulting in poor image quality. Recent studies [4,5] have highlighted the potential of GNSS-R for mapping inland water bodies. However, this technology is still in its early stages of development. In contrast, synthetic aperture radar (SAR) images are an ideal data source for water extraction due to their all-day, all-weather imaging capabilities and established technology [6,7,8,9]. However, the explosive growth of SAR images has presented challenges such as complex surface environments (e.g., speckle noise, shadow, and water-like surfaces). Consequently, in the era of big data [10,11], building a robust and accurate SAR image water extraction network has substantial practical importance.

In the previous decade, threshold-based methods have been widely used for water body extraction because water has weak scattering intensity in SAR images [12,13,14,15]. The underlying assumption of threshold-based methods is that the histogram of an SAR image comprises water and non-water pixels, represented by two partially overlapped distributions. The point of intersection between these distributions indicates the optimal threshold for minimizing errors. However, for SAR images with small water areas, the bimodal assumption is not satisfied, which makes it challenging to select the threshold. To alleviate this issue, local thresholding approaches have been introduced [16,17]. Despite the substantial enhancements in the precision of water body extraction achieved through the aforementioned threshold-based approaches, it is still challenging to find a threshold that can effectively distinguish water bodies from shadow interfered areas and water-like surfaces in SAR images [18]. Additionally, machine learning-based methods are also limited in extracting water bodies due to the reliance on manually selected features and empirical parameters [19,20].

Deep learning methods have been utilized for many applications of remote sensing imagery [21,22,23,24], owing to their ability for end-to-end learning of shallow detailed features and deep semantic features, as well as their characteristics of local connectivity and weight sharing. Many well-established deep-learning networks have been successfully utilized for water extraction from SAR imagery, yielding promising results. For instance, Dai et al. [25] designed an enhanced water extraction model based on a bilateral segmentation network for water body extraction. Similarly, Zhang et al. [26] introduced a fully convolutional up-sampling pyramid network to address the resolution loss caused by high stride convolution in conventional convolutional neural network (CNN) while utilizing a fully convolutional conditional random field to tackle the issue of water boundary blurring. Despite the substantial advancements afforded by these networks in SAR image water extraction, their capability to capture target features remains limited to local ranges, as illustrated in Figure 1a. Consequently, they fail to ensure the integrity and continuity of the targets.

Recently, the Transformer [27] has found extensive application in natural language processing (NLP) due to its capacity to capture global sequence semantic features. Inspired by the successes of NLP, many works have attempted to apply the transformer architecture in computer vision (CV) to capture the global contextual information of images [28,29,30,31]. For instance, the Vision Transformer (ViT) [28] has achieved state-of-the-art results in image classification tasks by applying the transformer architecture to images via regular patch partition. Recently, transformer-based models in the computer vision domain have also been utilized to extract features from the global range of SAR images and have obtained impressive classification performance [32,33]. However, when transformer-based models in the computer vision domain are used to extract water bodies from SAR images, there are two main challenges to be considered:

(1)

Predefined patch sizes (e.g., 16 × 16) are used for image partition, resulting in the issue of edge serration in water bodies extraction due to the heterogeneity within each patch;

(2)

The transformer-based models lack specific inductive biases, such as translation invariance and local correlation, which makes the model heavily rely on the dataset for accurate extraction of water bodies.

Therefore, we propose a Superpixel-based Transformer (SPT) to alleviate the above issues. Notably, superpixels are locally homogeneous regions composed of a collection of pixels with similar characteristics. Firstly, to overcome the problem of edge serration, the SPT substitutes the regular patch partition of the transformer-based models with irregular superpixel segmentation, as shown in Figure 1c. Additionally, to reduce the need for a large-scale water training dataset, the SPT implements a constrained optimization of the Multi-Layer Perceptron (MLP), putting forward a constrained MLP (C-MLP). Furthermore, inspired by heterogeneous networks [34,35], we incorporate a CNN and SPT into a unified network, termed SPT-UNet, where the CNN branch acquires pixel-level semantic features in the image to complement the features at the superpixel-level of the SPT branch.

Specially, the SPT-UNet mainly comprises four modules: the Superpixel Segmentation Module (SPSM), the Superpixel-level Transformer Sub-network (SPTsN), the Pixel-level Convolution Sub-network (PCsN), and the Feature Fusion Module (FFM). The SPSM mainly performs superpixel segmentation on SAR images. Subsequently, the SPTsN models the global spatial structure in images and forms semantic features of water bodies at the superpixel level. Meanwhile, the PCsN is utilized to extract pixel-level semantic features. Finally, the FFM incorporates both pixel- and superpixel-level features to enhance the performance of water extraction.

The primary contributions of the study are as follows:

(1)

To mitigate the issue of edge serration in water body extraction based on transformer-based models, we propose to use adaptive superpixels instead of regular patches;

(2)

To alleviate the dependence of transformer-based models on data, we introduce the C-MLP module, which imposes prior constraints on the SPT by utilizing a normalized adjacency matrix between superpixels;

(3)

Combining a CNN and SPT, we propose the SPT-UNet, which facilitates simultaneous learning on both the pixel and superpixel-level semantic features of water bodies. The experiments on the public dataset demonstrate that SPT-UNet performs competitively compared to other state-of-the-art extraction networks, which can partially alleviate the interference caused by terrain-induced shadows and water-like surfaces.

The rest of the paper is organized as follows. Section 2 introduces the SPT-UNet and provides a comprehensive overview of each part. Section 3 introduces the public water dataset involved in the experiments and the relevant experimental settings. In Section 4, we conducted a series of experiments and discussions. Finally, Section 5 provides the conclusions.

2. Proposed Method

The extraction task of water bodies is designed to identify whether each pixel is water. We define the SAR image as $I\in {\mathbb{R}}^{3\times H\times W}$, where 3 corresponds to the bands (VV, VH, and SDWI) and H, W represents the height and width of the image, respectively. Notably, the VV and VH symbolize the polarization modes of the SAR image, and the SDWI symbolizes the Sentinel-1 Dual-Polarized Water Index to strengthen the features of water bodies [36]. The main aim of this study is to extract water bodies from I by training an extraction network of water bodies. To achieve this goal, we propose the SPT-UNet, as depicted in Figure 2, which comprises four modules: (1) the SPSM that generates the normalized adjacency matrix $\tilde{A}$ and the mapping matrix M; (2) the SPTsN that performs global modeling of superpixels in the SAR image and forms water body features at the superpixel level; (3) the PCsN that performs the local modeling of pixels in the SAR image and forms water body features at the pixel level; (4) the FFM that integrates the pixel- and superpixel-level semantic features of water bodies. The training process of SPT-UNet is shown in Algorithm 1. The four modules are detailed in the following subsections. Additionally, the hybrid weighted loss is also presented.

Algorithm 1: Training procedure for the SPT-UNet framework

Input: SAR dataset $Z_s={\left\{I^i\right\}}_{i=1}^N$, DEM dataset $Z_d={\left\{D^i\right\}}_{i=1}^N$, JRC-gsw dataset $Z_j={\left\{J^i\right\}}_{i=1}^N$, Ground Label $\mathrm{L}={\left\{L^i\right\}}_{i=1}^N$, The number of Superpixels Z, the hyperparameter a. Output: Water bodies extraction results E $={\left\{E^i\right\}}_{i=1}^N$.

for  i = 1; i < N;  i++ do Randomly batch sample Ii from Zs, Batch sample Di consistent with Ii from Zd, Batch sample Ji consistent with Ii from Zj, Batch sample Li consistent with Ii from L. // Stage 1 SPSM                               $\tilde{A},M=SPSM\left(I^i\right)$ // Stage 2 SPTsN                            $V^{\mathrm{*}}=SPTsN\left(I^i,D^i,J^i,\tilde{A},M\right)$ // Stage 3 PCsN                                  F = PCsN (Ii) // Stage 4 FFM                               Ei = FFM (F, V*) // Calculate the segmentation loss       $Los{s}_{all}=a\cdot \mathrm{BCELoss}({E^i,L}^i)+(1-a)\cdot \mathrm{DiceLoss}(E^i, L^i)$ end for

2.1. SPSM

In the SPT, the internal homogeneity of superpixels significantly impacts the subsequent water extraction performance. Therefore, we incorporate the Adaptive Superpixel Segmentation Algorithm (ASSA) [37] into the SPSM. The ASSA consists of three steps: (1) The initialization stage sets the required algorithm parameters. (2) The priority queue traversal stage adjusts or generates seed points based on the homogeneity formula defined in (1). (3) Finally, the post-processing stage merges superpixels smaller than a predefined threshold to alleviate the interference from speckle noise.

$$\left|G\left(I\left(p_i\right)\right)-{IM}_j\right|<2\cdot min\left({\sigma }_m,m=\mathrm{1,2},\dots ,M\right)$$

(1)

where G (·) represents the standard Gaussian kernel function, $\left|·\right|$ represents absolute value, min (·) represents the minimum standard deviation $\sigma$ of all Gaussian distributions in the Gaussian Mixture Model (GMM) [38], p_i = (x, y) represents the coordinates of the ith pixel in the SAR image, and IM_j represents the average intensity of the jth superpixel.

Integrating a CNN and SPT poses a significant challenge, primarily because of the inherent differences in data representation structures between pixel- and superpixel-level features. Pixel-level features are traditionally extracted utilizing uniform grids, while superpixel-level features are derived based on irregular, internally homogeneous superpixels. The SPSM transforms features between the pixel- and superpixel-level to address the above issue by utilizing the M [39]. This transformation achieves effective feature interaction and fusion between the pixel- and superpixel-level.

The primary process of the SPSM is as follows: Firstly, the ASSA is applied to segment the I into superpixels, obtaining a superpixel set $S={\left[S_i\right]}_{i=1}^Z$, where $S_i={[{I(p}_i^j)]}_{j=1}^{N_i}$ denotes the ith superpixel, and $p_i^j$ denotes the jth pixel within the ith superpixel. Z and N_i denote the number of superpixel objects and the number of pixels within the ith superpixel object, respectively.

Then, to achieve the conversion between pixel- and superpixel-level features, the Encoder and Decoder are defined [40], which can be written as follows:

$$V=Encoder\left(I;M\right)={\widehat{M}}^{T}Flatten\left(I\right)$$

(2)

$$I^{\mathrm{*}}=Decoder\left(V^{\mathrm{*}};M\right)=Reshape\left(M{V}^{\mathrm{*}}\right)$$

(3)

where $V\in {\mathbb{R}}^{Z\times 5}$ denotes the initial superpixel-level features, V* denotes the superpixel-level water semantic features after SPTsN, I* denotes the pixel-level features after mapping from superpixel-level features, Reshape (·) indicates the restoration of spatial dimensions of the image, and $\widehat{M}$ denotes the column-normalized $M\in {\mathbb{R}}^{HW\times Z}$ which is defined as follows:

$$M_{i,j}=\left\{\begin{array}{cc}1,& \mathrm{if} {\widehat{I}}_i\in S_j\\ 0,& \mathrm{otherwise}\end{array}\right.\widehat{I}=Flatten\left(I\right)$$

(4)

where Flatten (·) denotes flattening SAR images by the spatial dimension, ${\widehat{I}}_i$ denotes the ith pixel of $\widehat{I}$, and S_j denotes the jth superpixel.

Furthermore, to enhance the generalization ability of the SPTsN, we define the $\tilde{A}\in {\mathbb{R}}^{Z\times Z}$ [41] for use within the SPTsN. Its definition is as follows:

$$\tilde{A}={\tilde{D}}^{-{\displaystyle \frac{1}{2}}}\widehat{A}{\tilde{D}}^{-{\displaystyle \frac{1}{2}}}$$

(5)

$${\tilde{D}}_{ii}=\sum _j{\widehat{A}}_{ij}$$

(6)

$$\widehat{A}=A+I_N$$

(7)

$$A_{i,j}=\{\begin{array}{c}1{,}^{ }i{f}_{ }{S}_i{ }^{ }an{d}_{ }{S}_j{ }_{ }ar{e}_{ }adjancent\\ 0{,}_{ }otherwise{{\displaystyle }}_{ }{{\displaystyle }}_{ }{{\displaystyle }}_{ }{{\displaystyle }}_{ }{{\displaystyle }}_{ }{{\displaystyle }}_{ }{{\displaystyle }}_{ }\end{array}$$

(8)

where I_N denotes the identity matrix, $\widehat{A}$ denotes the adjacency matrix considering the superpixel itself, $\tilde{D}$ denotes the degree matrix of $\widehat{A}$, and A denotes the adjacency matrix.

In summary, leveraging the ASSA, we construct superpixel-level initial features. Additionally, we define the Encoder and Decoder to promote the transformation between pixel- and superpixel-level features. Further, the $\tilde{A}$ is defined for the subsequent training of the SPTsN.

2.2. SPTsN

The transformer-based models are widely used in the CV domain due to their powerful global modeling capability. However, directly applying it to water body extraction from SAR images faces two main challenges: the issue of edge serration caused by regular patch partition and the dependence on data caused by weak generalization ability. We define the SPT in SPTsN to alleviate the above issues, as depicted in Figure 3.

In the SPT, we first input the V and $\tilde{A}$, which are obtained from the SPSM. Here, the V incorporates JRC Global Surface Water (JRC-gsw) [42] and DEM [43] to reduce the probability of water misclassification.

In the initial stage of the SPT, a fully connected layer (FC) is used to map V to a D-dimensional space. To capture spatial positional information among the superpixels, we introduce a learnable one-dimensional position encoding denoted as ${P\in \mathbb{R}}^{Z\times D}$. The formula for feature mapping and incorporating the position encoding into $V_0{\in \mathbb{R}}^{Z\times D}$ is as follows:

$$V_0=VW+P$$

(9)

where W denotes the parameters of the FC.

In the middle stage of the SPT, we perform 8 layers to learn the superpixel-level semantic features of water bodies in SAR images. As shown in Figure 3, the middle stage mainly consists of the multi-head self-attention (MSA) and the C-MLP. The MSA is used to model the spatial relationships between superpixels globally. Then, the C-MLP ensures that all superpixels in the image propagate and learn features solely through themselves and their neighbors. The formula of C-MLP is as follows:

$$\mathrm{C}-\mathrm{MLP}\left(V\right)=Dropout\left(\tilde{A}Relu\right(\tilde{A}V{W}^{\left(0\right)}\left)W^{\left(1\right)}\right)$$

(10)

where W⁽⁰⁾ and W⁽¹⁾ denote the parameters of the FC, Relu (·) denotes the activation layer, and Dropout (·) denotes the dropout layer that randomly discards neurons with a certain probability. $\tilde{A}$ essentially reflects the prior correlation weights among any superpixels in the image. Incorporating $\tilde{A}$ into the MLP can be seen as introducing an inductive bias capability into the SPT. Consequently, it enhances the ability of the SPT to generalize in small-scale datasets. The formula for the output in the middle stage during the mth layer is as follows:

$$V_m^{\prime }=\mathrm{MSA}\left(\mathrm{LN}\left(V_{m-1}\right)\right)+V_{m-1}\left(m=1,\dots ,M\right)$$

(11)

$$V_m=\mathrm{C}-\mathrm{MLP}\left(\mathrm{LN}\left(V_m^{\prime }\right)\right)+V_m^{\prime }\left(m=1,\dots ,M\right)$$

(12)

where V_m denotes the output of the mth middle stage and $V_m^{\prime }$ denotes the output of the MSA module in the mth middle stage.

In the final stage of the SPT, the FC is used to compress the superpixel-level features, which capture abundant semantic information related to water bodies. These features are mapped to a D’-dimensional space, resulting in the output superpixel-level features denoted as $V^{\mathrm{*}}{\in \mathbb{R}}^{Z\times D^{\prime }}$.

The SPT within the SPTsN is designed to capture spatial relationships between different superpixels at the global scale of SAR images and establish superpixel-level features. We represent the SPT as a function denoted by $V^{\mathrm{*}}=SPT\left(V;\tilde{A}\right)$.

2.3. PCsN

To enable the model to possess pixel-level local feature learning capability, we propose Efficient-UNet in the PCsN, as illustrated in Figure 4.

The Efficient-UNet follows a standard encoder–decoder architecture. EfficientNet [44], a well-known convolutional neural network model introduced by Google, balances parameter efficiency and accuracy by uniformly scaling the width, depth, and image resolution of the network using a new compound scaling method. For the encoder part, we select the Efficient-B4 network to extract water bodies’ semantic features from the I, taking into account the model’s parameter count. The upsampling operation is used in the decoder part to gradually restore the low-resolution features obtained from the encoder to the original image size. Additionally, a skip connection is utilized to concatenate the low-level features from the encoder with the high-level features from the decoder, thus recovering the detailed characteristics of the image.

Generally, the Efficient-UNet within the PCsN captures spatial relationships between pixels at the local scale of SAR images and establishes pixel-level features. We represent the Efficient-UNet as a function, denoted by F = Efficient-UNet (I).

2.4. FFM

To fully integrate the pixel- and superpixel-level semantic features of water bodies, we construct the FFM, as shown in Figure 5.

In the FFM, the pixel-level feature F and the superpixel-level feature V* are first inputted. Then, the BN [45] is applied to both F and V* to address the issue of inconsistent scales between the pixel- and superpixel-level features and reduce the influence of internal covariate shifts for subsequent steps of the FFM. Next, the Add layer is used to sum up the pixel- and the superpixel-level features, obtaining the fused features. Afterward, the fused feature is further processed through a 3 × 3 convolution layer, a BN layer, and an activation layer to extract spatial and semantic features about the water bodies. Finally, the 1 × 1 convolutional and Sigmoid layers process the refined features and output the water body extraction result.

Overall, the FFM combines the pixel- and superpixel-level semantic features of the water bodies, leveraging their respective advantages.

2.5. Hybrid Weighted Loss

The SPT-UNet, which has been proposed in this study, is an end-to-end pixel-wise classification model for which we use a hybrid weighted loss, denoted as Loss_all. It combines the binary cross-entropy loss (BCELoss) and the Dice loss (DiceLoss). This combination leverages the stable training process of the BCELoss and the advantage of the DiceLoss in mitigating the imbalance in the dataset distribution. The formula is presented below:

$$\mathrm{BCELoss}(E,L)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[L\left(p_i\right)\mathit{log}(E\left(p_i\right))+(1-L\left(p_i\right)\left)\log \right(1-E\left(p_i\right))\right]$$

(13)

$$\mathrm{DiceLoss}\left(E,L\right)=1-{\displaystyle \frac{2\mid E\cap L\mid }{\mid E\mid +\mid L\mid }}$$

(14)

$$Los{s}_{all}=a\cdot \mathrm{BCELoss}+(1-a)\cdot \mathrm{DiceLoss}$$

(15)

where E (p_i) denotes the probability of the ith pixel, which is water in the predicted image, and a denotes the weight factor.

3. Dataset and Experimental Setup

3.1. Data Description

Our experiments used the C2S-MS dataset [46], a benchmark dataset for water extraction in SAR images. The C2S-MS dataset was obtained globally through sampling from different complex surface environments (e.g., the interference of the shadow and water-like surfaces). Consequently, evaluating the superiority and effectiveness of the model in extracting water bodies from this dataset presents a more significant challenge.

The C2S-MS dataset consists of 8100 images from Sentinel-1 with a size of 256 × 256. Four typical samples are shown in Figure 6. The images in the dataset have a spatial resolution of 10 m. To facilitate more effective extraction of water features and assess model generalization ability, the training, validation, and testing data sets were divided in a ratio of 7:1:1, as shown in

Table 1. The dataset parameters.

	C2S-MS
band	VV, VH
Train.	6300
Val.	900
Test.	900

.

3.2. Experimental Settings

The proposed network is implemented using PyTorch and trained on an NVIDIA TESLA P100 GPU platform. Adam was chosen to speed up the training process. The training process is conducted for 300 epochs, with an initial learning rate of 3 × 10⁻⁴. A cosine annealing optimization method is utilized to adjust the learning rate dynamically to enhance model performance.

3.3. Evaluation Indices

To measure the water extraction performance of the model, we chose the intersection over union (IoU), Precision, Recall, and F1 score as evaluation indices:

$$\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{N}+\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(16)

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{P}}}$$

(17)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(18)

$$\mathrm{F}1={\displaystyle \frac{2·\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}·\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(19)

where TP denotes the total number of correctly classified water pixels, TN denotes the total number of correctly classified background pixels, FP denotes the total number of background pixels wrongly classified as water pixels, and FN denotes the total number of water pixels wrongly classified as background pixels. The values of these four indices range from 0 to 1, and when the values are close to 1, it indicates higher accuracy in water segmentation and better algorithm robustness.

4. Experiment Results

4.1. Comparison of Extraction Performance

To evaluate the superiority and effectiveness of the proposed SPT-UNet, we compared it with various well-known segmentation algorithms, including U-Net [47], DeepLabv3+ [48], GCN [41], Swin-UNet [49], FWENet [50], and DWENet [51]. All baseline networks were performed under the same conditions to ensure a fair comparison, utilizing the hyperparameters specified in the original paper. The quantitative and visualized results of water body extraction with different networks are depicted in

Table 2. The accuracies of different networks. The best result is highlighted in bold.

Network	IoU	Recall	Precision	F1
U-Net	72.8	82.81	77.42	80.02
Deeplabv3+	72.48	82.11	84.12	83.10
FWENet	72.42	86.80	79.79	83.15
DWENet	73.00	78.94	89.03	83.68
GCN	66.22	74.87	82.87	78.67
Swin-UNet	66.35	78.77	78.63	78.70
SPT-UNet	80.10	86.20	91.07	88.57

and Figure 7.

Table 2 shows that the SPT-UNet displays the best value of IoU, Precision, and F1 score, outperforming other networks in extracting water bodies. Experimental results illustrate that our networks can capture the complementary pixel- and superpixel-level features of water bodies to reduce misjudgments and improve details of water bodies.

It is worth mentioning that the proposed SPT-UNet provides the PCsN branch to capture the pixel-level features of water bodies to supplement the SPTsN, thus surpassing the patch-based Swin_UNet and the superpixel-based GCN. Similarly, due to the provision of the SPTsN branch to capture superpixel-level water bodies features to supplement the PCsN, it surpasses pixel-based networks like U-Net and DeepLabv3+. Additionally, the mainstream extraction networks of water bodies, such as FWENet and DWENet, which consider the relevant characteristics of water bodies, are primarily pixel-level networks. As a result, they cannot model global structures, limiting their extraction accuracy.

Figure 7 shows that the water bodies extracted by the proposed SPT-UNet had more fitting boundaries. In contrast to the Swin-UNet, the proposed SPT-UNet significantly reduced edge serration, which validates the effectiveness of superpixel segmentation over patch partitioning. Additionally, the water bodies had fewer misjudgments. The above phenomena further prove the complementary effect of the PCsN and SPTsN in the proposed SPT-UNet.

4.2. Ablation Study

In this section, we performed a series of ablation experiments to assess the performance of various modules. Considering the essentiality of combining SPSM with SPTsN, we merge them by default. The performance of water body extraction under different module combinations is shown in Figure 8.

Compared to using either SPTsN or PCsN modules individually, combining both achieved higher extraction performance in water body extraction, indicating the complementarity between the pixel- and superpixel-level features. Furthermore, the extraction performance of the SPTsN module, with the inclusion of the normalized adjacency matrix $\tilde{A}$, outperformed that of the SPTsN module without $\tilde{A}$. This finding indicates that using the $\tilde{A}$ to impose knowledge constraints on superpixels is beneficial for enhancing the generalization ability of the SPTsN module in small-scale datasets.

Furthermore, to analyze the impact of combining the pixel- and superpixel-level features, we employed the Grad-CAM [52] on different networks, a visualization method that utilizes gradients to calculate the spatial importance of positions within a layer. We compared the visualization results of the last convolution layer in Efficient-UNet based on pixels, SPT based on superpixels, and SPT-UNet, which combines pixel and superpixel features. The results of Figure 9 indicate that SPT, relying on superpixels, plays a crucial role in maintaining accurate water body boundaries and minimizing misclassifications. Conversely, Efficient-UNet, operating at the pixel level, excels in extracting the detailed features of water bodies. In essence, the pixel- and superpixel-level features within the SPT-UNet complement each other effectively.

5. Discussion

5.1. Influence of Segmentation Scale

The number of superpixels, Z, could affect the performance of the SPT-UNet. To analyze its influence on our network, we set the number of superpixels, Z, to 256, 512, 800, 1024, and 1200, respectively, and tested their extraction accuracies, as shown in

Table 3. The accuracies of different Z. The best result is highlighted in bold.

Z	IoU	Recall	Precision	F1
256	77.36	87.27	86.21	86.74
512	80.10	86.20	91.07	88.57
800	75.40	83.03	87.22	85.07
1024	79.80	87.20	89.57	88.37
1200	76.84	82.45	91.40	86.69

.

The segmentation scale, Z, decides the smallest size of the superpixels in the image. As can be seen, the values of Recall mainly decreased with the increase in scale. One probable explanation is that a larger scale leads to small superpixel units that may contain a relatively large number of noise points, making it challenging to learn features at the superpixel level. In contrast, the values of Precision primarily increased with the increase in scale. One probable explanation is that a smaller scale results in large superpixel units that can effectively resist the interference of noise points but lack the feature learning of small objects. Considering the balance between noise point interference and the extraction effect of small water bodies, we fixed Z to 512 in all experiments.

5.2. Impact of a

The hyperparameter a in Loss_all is used to balance the benefits and drawbacks of BCELoss and DiceLoss. To validate the appropriateness of the set, we compared the extraction indices for different a values, as depicted in

Table 4. The accuracies of different a. The best result is highlighted in bold.

a	IoU	Recall	Precision	F1
0.3	77.11	86.57	86.20	86.38
0.4	78.96	83.86	92.19	87.83
0.5	78.82	85.87	90.03	87.90
0.6	80.10	86.20	91.07	88.57
0.7	78.03	86.54	88.06	87.29

.

The hyperparameter a can affect the training process of the networks. The accuracies of water extraction mainly decreased with the too-small a. This phenomenon may be attributed to the instability in model training caused by the increased weight of the DiceLoss, affecting the degree to which the parameters of the SPT-UNet fit the samples. Similarly, the accuracies of water extraction also primarily decrease with the too-large a, which may be caused by the imbalance between water bodies and the background in the dataset. Therefore, considering all factors, we set the a value to 0.6.

5.3. Impact of Additional Data

Additional data, such as SDWI image, JRC-gsw image, and DEM image, are introduced to enhance the performance of the SPT-UNet in water extraction. To investigate the effectiveness of the additional data, we carried out a series of experiments for different combination methods, the details of which can be found in

Table 5. The accuracies of different combination methods. The best result is highlighted in bold.

Combination Method	IoU	Recall	Precision	F1
VV + VH	72.98	81.58	86.20	83.83
VV + VH + SDWI	74.55	81.20	88.09	84.50
VV + VH + SDWI + DEM	75.11	87.14	83.37	85.21
VV + VH + SDWI + JRC-gsw	77.66	82.40	92.26	87.05
VV + VH + SDWI + DEM + JRC-gsw	80.10	86.20	91.07	88.57

. Compared to the SPT-UNet, which is composed only of VV and VH inputs, the water extraction indicators of the network generally improved after adding SDWI images. This phenomenon may be attributed to the enhancement of water features in the input section, making the network more inclined to learn such features. Adding DEM to the inputs of SPT-UNet composed of VV, VH, and SDWI resulted in improvements in the values of IoU, Recall, and F1 indicators. This phenomenon may be explained by alleviating the interference caused by terrain-induced shadow areas. Adding JRC-gsw to the inputs of SPT-UNet, which consists of VV, VH, and SDWI, leads to improvements in all four indicators. This may be caused by alleviating the impact of interfering objects in SAR images on water extraction. Simultaneously adding DEM and JRC-gsw to the inputs of SPT-UNet, which consists of VV, VH, and SDWI, achieves a balance between Recall and Precision, achieving optimal values in F1 and IoU.

5.4. Impact of Depth in SPT

In the SPT middle stage, we performed eight layers to extract and learn the water features in SAR images at the superpixel level. To validate the rationality of the aforementioned configuration, we performed the following experiments: training the SPT-UNet with 6, 8, 10, and 12 layers, respectively. A comparison of extraction indicators is shown in Figure 10.

The SPT-UNet achieved the highest segmentation IoU, Precision, and F1 score when the layer number of the SPT middle stage was eight. When depth sets were too small or large, the value of the four evaluation indicators all decreased. One probable explanation is that a small depth reduces the ability of SPT to model global semantic dependencies. In contrast, a large depth makes modeling difficult due to the small dataset scale, hindering network optimization. Thus, an eight-layer SPT middle stage was incorporated into the SPT-UNet, offering superior segmentation results.

6. Conclusions

Extraction of water bodies from SAR images has been crucial in many applications. In this study, we introduce an innovative network architecture, SPT. (1) The network leverages the ASSA for irregular superpixel segmentation instead of regular patch partition, which can mitigate edge serration in water body extraction. (2) The network uses the C-MLP to alleviate the problem of the weak generalization ability in small-scale datasets. The C-MLP imposes knowledge constraints on superpixels, which limits the learning range of the superpixels. (3) Additionally, to fuse the superpixel-level features extracted by SPT with pixel-level features, we proposed the SPT-UNet—a unified network based on CNN and SPT. Experimental results of the SPT-UNet showed that the superpixel-level water bodies’ semantic features complement pixel features, further improving the performance of water bodies extraction. Further work will improve the computational efficiency of the model using parallel strategies and other optimization methods and refine the SPT-UNet for the multi-scale features fusion at pixel- and superpixel-level in an adaptive manner to implement water bodies extraction.