Hierarchical Prototype-Aligned Graph Neural Network for Cross-Scene Hyperspectral Image Classification

Abstract

The objective of cross-scene hyperspectral image (HSI) classification is to develop models capable of adapting to the “domain gap” that exists between different scenes, enabling accurate object classification in previously unseen scenes. Many researchers have devised various domain adaptation techniques aimed at aligning the statistical or spectral distributions of data from diverse scenes. However, many previous studies have overlooked the potential benefits of incorporating spatial topological information from hyperspectral imagery, which could provide a more accurate representation of the inherent data structure in HSIs. To overcome this issue, we introduce an innovative approach for cross-scene HSI classification, founded on hierarchical prototype graph alignment. Specifically, this method leverages prototypes as representative embedded representations of all samples within the same class. By employing multiple graph convolution and pooling operations, multi-scale domain alignment is attained. Beyond statistical distribution alignment, we integrate graph matching to effectively reconcile semantic and topological information. Experimental results on several datasets achieve significantly improved accuracy and generalization capabilities for cross-scene HSI classification tasks.

1. Introduction

Hyperspectral image classification stands as a prominent research domain within hyperspectral image data processing. Currently, numerous methodologies have been devised for hyperspectral image classification, including traditional machine learning algorithms [1,2,3,4] and deep learning methods [5,6,7,8]. However, it is crucial to note that the efficacy of these techniques heavily relies on the availability of a substantial number of labeled samples. However, in the domain of hyperspectral imagery, acquiring a substantial amount of labeled samples is extremely challenging due to the significant time and manpower required [9]. In many practical hyperspectral image classification scenarios, it may even be impossible to obtain any labeled samples. Previous hyperspectral classification methods were typically applicable only to fixed scenarios, where the training and testing samples are independent and drawn from identical distribution. Consequently, these methods cannot be transferred to other scenarios and cannot be tested on real-time acquired data without labels [10]. In such cases, the task of transferring the knowledge of a well-trained model to unlabeled scenarios becomes a common challenge, known as the cross-scene classification of hyperspectral imagery.

In cross-scene hyperspectral classification tasks, the image data with known labels are typically denoted as the source domain (SD), while the image data with unknown labels are termed the target domain (TD). The primary goal is to leverage the labeled SD to train a model and transfer its knowledge to accurately classify land cover in the TD. However, practical hyperspectral image data acquisition is subject to various influencing factors, such as sensor noise variations, seasonal changes, and diverse weather conditions [11]. These factors lead to spectral reflectance disparities between the same land cover classes in the source and target domains. Consequently, direct application of the SD for classification in the TD often encounters spectral shift challenges. Therefore, the models trained within a single scene cannot be directly used in other scenes with unknown labels.

To address this challenge, many researchers use domain adaptation techniques targeting the reduction in spectral shift at the feature level, aiming to learn models invariant to domain variations across different scenes. This enables the training of a classifier on a source scene and its effective application to a target scene. For instance, Sun et al. proposed discriminative cross-view subspaces and utilized subspace alignment methods for unsupervised cross-scene remote sensing image classification [12]. Qin et al. introduced a tensor alignment approach for HSI classification, achieving subspace alignment across tensor domains [13]. Yang et al. introduced an ideal regularization discriminative multi-kernel subspace alignment method for HSI domain adaptation [14]. Furthermore, various deep learning methods enhance model classification performance in the target domain by employing adaptive layers to facilitate alignment between the source and target domains. Zhu et al. proposed a deep subdomain adaptation network by defining the subdomain concept and aligning relevant subdomains using local maximum mean discrepancy [15]. Long et al. introduced a deep adaptation network that simultaneously added three adaptive network layers and employed multi-kernel maximum mean discrepancy in each network layer to achieve domain alignment [16]. Qu et al. introduced a physically constrained transfer learning method that significantly improved results in real remote sensing scenarios by sharing rich spatial information in hyperspectral imagery [17]. Moreover, the concepts of contrastive learning and adversarial networks are currently widely utilized in cross-scene hyperspectral image classification. Ning et al. designed an instance-to-instance contrastive learning framework based on category matching, which helps extract domain-invariant and class-discriminative features simultaneously [18].

Wang et al. proposed a bi-classifier adversarial network that incorporates self-training strategies, effectively enhancing feature discriminative performance [19].

Liu et al. combined the adversarial learning strategy and the contrastive learning strategy to achieve cross-domain few-shot HSI classification [20].

Domain adaptation serves as an effective means of transferring knowledge from a labeled dataset (source domain) to an unlabeled dataset (target domain), finding extensive utility in cross-scene classification tasks for hyperspectral imagery [21,22]. However, most of the previous methods overlook the topological characteristics of hyperspectral imagery. These topological relationships that reflect global characteristics are not constrained by spatial coordinates alone and maintain their original properties even in the presence of deformations [10]. Such invariant features contribute to improving the classification performance, particularly in cross-scene applications with domain shift. Recently, graph neural networks have become increasingly popular for their effectiveness in representing and analyzing data within non-Euclidean spaces. They are particularly well-suited for graph-structured data, as they can model the intricate relationships between samples (or vertices). Given the absence of strict correspondence between local spatial relationships in two hyperspectral image scenes, exploiting the intrinsic topological relationships among land cover categories becomes paramount to better align the two domains based on non-local spatial relationships. As shown in Figure 1, even with distribution alignment alone, target domain samples may still be incorrectly aligned with different categories of source domain samples. While the classification boundaries in the source domain might suit its own data adequately, they could prove inadequate for discerning patterns in the target domain. However, aligning the topological structural information ensures the preservation of feature consistency and relationships within the scene, ultimately achieving category-level domain alignment.

In light of the challenge of leveraging spatial topological information in hyperspectral imagery for cross-scene classification tasks, an innovative approach grounded in hierarchical prototype graph alignment (HPGA) is proposed. The main contributions of our work are summarized as follows:

• A generic, end-to-end differentiable model framework for conducting domain alignment on prototype graph structure data derived from both source and target domains in a hierarchical manner is proposed. By aligning topological and semantic information at different scales, the accuracy and generalization capability of hyperspectral image classification are improved.
• Different scales of prototype graph structure data are obtained using differentiable graph pooling. This allows the model to analyze data at multiple levels, capturing richer semantic information at different hierarchies.
• The problem of the cross-domain alignment of graph structures is transformed into a graph-matching problem. During network optimization, the graph optimal transport (GOT) distance is minimized in order to align the source and target domains. This approach leverages graph structure information to better address the spectral shift problem.
• Experimental results demonstrate that the proposed hyperspectral cross-scene classification method based on hierarchical prototype graph alignment achieves excellent performance on several datasets. This indicates that the approach has strong generalization capabilities when dealing with hyperspectral cross-scene classification tasks.

2. Methodology

In this section, we begin by offering foundational insights into our approach through a discussion of essential definitions and notation. This includes an exploration of domain adaptation concepts and an introduction to graph-matching principles. Following this, we describe the proposed HPGA method in detail.

2.1. Preliminaries

2.1.1. Domain Adaptation

Assuming the training and testing data follow the same distribution, machine learning methods can be generalized. This assumption, however, is challenged when training a model in one domain (the source domain) and testing it in another (the target domain), where data distributions may diverge. Domain adaptation (DA) addresses this challenge by striving to enhance performance, aiming to identify features invariant across domains, thus facilitating the improved generalization of models trained in the source domain to the target domain [16,23].

One classic approach to domain adaptation is maximum mean discrepancy (MMD) [24]. MMD, a kernel-based measure, seeks to minimize the distance between the feature distributions of the source and target domains. By reducing MMD, the feature distributions of the source and target domains converge, thereby enhancing the model’s generalization performance. Let ${\mathit{X}}_s={\left\{{\mathit{x}}_i^s\right\}}_{i=1}^{n_s}\in {\mathbb{R}}^d$ and ${\mathit{X}}_t={\left\{{\mathit{x}}_i^t\right\}}_{i=1}^{n_t}\in {\mathbb{R}}^d$ denote the data in the source and target domains, respectively, where d represents the dimensionality of the data and $n_s$ and $n_t$ are the number of samples in the source and target domains, respectively. ${\mathit{Y}}_s$ and ${\mathit{Y}}_t$ represent the corresponding class labels. The computation of MMD involves mapping the samples from the source and target domains to a feature space using a mapping function $\varphi$ and then calculating the distance between the two distributions in the feature space. The MMD distance between two different domains can be described as follows:

$$\begin{array}{c}\hfill MMD\left({\mathit{X}}_s,{\mathit{X}}_t\right)={∥\frac{1}{n_s}\sum _{i=1}^{n_s}\varphi \left({\mathit{x}}_i^s\right)-\frac{1}{n_t}\sum _{j=1}^{n_t}\varphi \left({\mathit{x}}_j^t\right)∥}_H^2\end{array}$$

(1)

where $\varphi$ is a feature mapping function. The physical interpretation of MMD entails determining the distance between the means of the source domain (SD) and target domain (TD) within the Reproducing Kernel Hilbert Space (RKHS), which is presented by H. RKHS constitutes a complete function inner product space, and after squaring and expanding, the inner product in RKHS can be transformed into a kernel function. Consequently, MMD computation can be directly conducted using this kernel function. If the two distributions are identical, the MMD is 0; otherwise, the MMD is greater than 0.

2.1.2. Graph Matching

For graph-structured data, addressing cross-domain alignment involves formulating a graph-matching problem. Here, we employ graph optimal transport (GOT) [25] for feature matching and graph topological structure matching. GOT employs two optimal transport distances: the Wasserstein distance (WD) [26] for node matching and the Gromov–Wasserstein distance (GWD) [27] for edge matching. The WD quantifies the distance between node embeddings across domains, disregarding the encoded topological information in the graph. Conversely, the GWD compares graph structures by assessing distances between pairs of nodes in each graph. By combining these two distances, the GOT framework effectively accounts for both node and edge information, thereby achieving enhanced graph matching. The formula for the WD is as follows:

$$\begin{array}{c}\hfill {\mathcal{D}}_{\mathrm{WD}}\left[P\left({\mathit{X}}_s\right),P\left({\mathit{X}}_t\right)\right]=\underset{\gamma \in \Pi \left[P\left({\mathit{X}}_s\right),P\left({\mathit{X}}_t\right)\right]}{inf}{E}_{\left({\mathit{x}}^s,{\mathit{x}}^t\right)\sim \gamma }\left[c\left({\mathit{x}}^s,{\mathit{x}}^t\right)\right]\end{array}$$

(2)

$$\begin{array}{c}\hfill c\left({\mathit{x}}_i^s,{\mathit{x}}_j^t\right)=1-\frac{{\left({\mathit{x}}_i^s\right)}^{\mathrm{T}}{\mathit{x}}_j^t}{\parallel {\mathit{x}}_i^s{\parallel }_2{\parallel {\mathit{x}}_j^t\parallel }_2}\end{array}$$

(3)

where $c({\mathit{x}}^s,{\mathit{x}}^t)$ represents the cross-domain cost matrix obtained through the cosine distance, and $\Pi [P\left({\mathit{X}}_s\right),P\left({\mathit{X}}_t\right)]$ represents the joint distribution $\gamma ({\mathit{x}}^s,{\mathit{x}}^t)$ over all pairs. The Gromov–Wasserstein distance (GWD) can be expressed as follows:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{GWD}}\left[P\left({\mathit{X}}_s\right),P\left({\mathit{X}}_t\right)\right]=& \underset{\gamma \in \Pi \left[P\left({\mathit{X}}_s\right),P\left({\mathit{X}}_t\right)\right]}{inf}{E}_{\left({\mathit{x}}^s,{\mathit{x}}^t\right)\sim \gamma ,\left({\widehat{\mathit{x}}}^s,{\widehat{\mathit{x}}}^t\right)\sim \gamma }\left[L\left({\mathit{x}}_i^s,{\mathit{x}}_j^t,{\widehat{\mathit{x}}}_i^s,{\widehat{\mathit{x}}}_j^t\right)\right]\hfill \end{array}$$

(4)

$$\begin{array}{c}\hfill L\left({\mathit{x}}_i^s,{\mathit{x}}_j^t,{\widehat{\mathit{x}}}_i^s,{\widehat{\mathit{x}}}_j^t\right)=∥c_1\left({\mathit{x}}_i^s,{\widehat{\mathit{x}}}_i^s\right)-c_2\left({\mathit{x}}_j^t,{\widehat{\mathit{x}}}_j^t\right)∥\end{array}$$

(5)

where $L(x_i^s,x_j^t,{\widehat{x}}_i^s,{\widehat{x}}_j^t)$ is the cost function that evaluates the intra-graph structural similarity between the two pairs of nodes $(x_i^s,x_i^s)$ and $(x_j^t,x_j^t)$. During the computation of these two distances, GOT learns an optimal transport matrix $\mathit{T}\in {\mathbb{R}}^{n\times m}$, where $\mathit{T}$ is shared between the WD and GWD. This matrix is used to integrate the WD and GWD to obtain the GOT distance and optimize the alignment between the source and target domains. The optimal transport matrix $\mathit{T}$ possesses several unique characteristics that make it a favorable choice for addressing cross-domain alignment problems: (1) Self-normalization: the sum of all elements in the optimal transport matrix $\mathit{T}$ equals 1. (2) Sparsity: When solved exactly, the optimal solution $\mathit{T}$ contains at most $(2r-1)$ non-zero elements, where $r=max(n,m)$. This achieves alignment that is more interpretable and robust. (3) Efficiency: Compared to traditional linear programming solvers, the optimal solution can be easily obtained through an iterative process of matrix–vector multiplication. This allows for the application of GOT to large-scale deep neural networks. The computation process of the GOT distance is illustrated in Figure 2.

2.2. Overall Framework

This paper introduces HPGA, a novel method for cross-scene HSI classification that addresses the disparities between source and target domains through the alignment of hierarchical prototype graphs. Prototype representation is a method used to describe graph structure features, where representative vertices can be identified using graph pooling algorithms [28]. Hierarchical prototype representation refers to clustering at different levels to obtain prototype representations at different granularities. The purpose of this approach is to capture general structural information across all graphs and enable reliable vertex partitioning at different levels. Different from methods that apply multi-scale convolution on a single-layer graph, the proposed method imports the HSI into multiple layers of graphs, allowing for the learning of features at various scales. As depicted in Figure 3, our proposed algorithm centers around two key operations: domain alignment (including distribution alignment and graph alignment) and differentiable graph pooling.

2.3. Hierarchical Prototype Representation

In the target domain, the lack of supervision leads to initial nodes in graph construction frequently deviating from actual instances. Additionally, sample representations within the same category often display multi-modality [29], causing differences between the embedding space and the sample instances. As the scale of graph construction directly reflects the topological structure of the images at different scales, graph alignment at different scales allows for a more thorough comprehension of the scene, enabling the model to accurately capture the consistency of features and relationships (domain-invariant features) within the scene. Building on the approach of Snell et al. [30], which introduced a latent space where each class is represented by a prototype and data points of the corresponding class cluster around it, this paper adopts graph pooling algorithms to identify representative vertices as prototypes and construct graph topologies with larger receptive fields. Hierarchical prototype representation refers to pooling operations at different levels to obtain prototype representations at different granularities, gradually reducing the discrepancy between the node representations and the prototypes of the corresponding class across domains, ultimately achieving category-level prototype alignment.

Prototype representation learning involves clustering or pooling nodes based on the output of a graph neural network (GNN) to obtain a coarsened graph as input to another GNN layer. However, the challenge in designing such a pooling layer for GNNs lies in providing a general method for hierarchical pool nodes across a series of input graphs, not just within a single graph. Specifically, given the output $\mathit{Z}=\mathrm{GNN}(\mathit{A},\mathit{X})$ of a GNN module and the graph adjacency matrix $\mathit{A}\in {\mathbb{R}}^{n\times n}$, a graph pooling operation can be performed to obtain a new coarsened graph with $m<n$ nodes, represented by a weighted adjacency matrix ${\mathit{A}}^{\prime }\in {\mathbb{R}}^{m\times m}$ and node embeddings ${\mathit{Z}}^{\prime }\in {\mathbb{R}}^{m\times d}$. This new coarsened graph can serve as input to another GNN layer, and this process can be repeated for L times, resulting in a model with L GNN layers. To achieve this, a clustering assignment matrix is defined that determines which prototype category each node belongs to based on the output of the GNN module. This matrix can be used to generate new graph features and adjacency matrices for the next GNN module. Let ${\mathit{S}}^{\left(l\right)}\in {\mathbb{R}}^{n_l\times n_{l+1}}$ denote the clustering assignment matrix learned at the l-th layer. Each row of ${\mathit{S}}^{\left(l\right)}$ corresponds to a node (or cluster) at the l-th layer, and each column corresponds to a cluster at the $(l+1)$-th layer. Intuitively, ${\mathit{S}}^{\left(l\right)}$ provides a soft assignment of each node at the l-th layer to a cluster in the coarsened graph at the next layer.

Assuming that the assignment matrix ${\mathit{S}}^{\left(l\right)}$ for the l-th layer has been computed, let ${\mathit{A}}^{\left(l\right)}$ denote the input adjacency matrix and ${\mathit{Z}}^{\left(l\right)}$ denote the input node embedding matrix for this layer. Based on these inputs, the pooling process can be computed using the following two formulas to obtain a new coarsened adjacency matrix ${\mathit{A}}^{(l+1)}$ and a new embedding matrix ${\mathit{X}}^{(l+1)}$:

$$\begin{array}{c}\hfill {\mathit{X}}^{(l+1)}={\mathit{S}}^{{\left(l\right)}^T}{\mathit{Z}}^{\left(l\right)}\in {\mathbb{R}}^{n_{l+1}\times d}\end{array}$$

(6)

$$\begin{array}{c}\hfill {\mathit{A}}^{(l+1)}={\mathit{S}}^{{\left(l\right)}^T}{\mathit{A}}^{\left(l\right)}{\mathit{S}}^{\left(l\right)}\in {\mathbb{R}}^{n_{l+1}\times n_{l+1}}\end{array}$$

(7)

Equation (6) aggregates the node embeddings ${\mathit{Z}}^{\left(l\right)}$ based on the clustering assignment matrix ${\mathit{S}}^{\left(l\right)}$, resulting in embedded representations for $n_{l+1}$ prototypes. Similarly, Equation (7) transforms the adjacency matrix ${\mathit{A}}^{\left(l\right)}$ into a coarsened adjacency matrix that represents the connection strength between each pair of prototypes. The pooling layer achieves graph coarsening through Equations (6) and (7). It is important to note that ${\mathit{A}}^{(l+1)}$ is a real-valued matrix representing a fully connected adjacency graph with weights, where each element ${\mathit{A}}_{ij}^{(l+1)}$ can be considered as the connection strength between prototype i and prototype j. Likewise, each row of ${\mathit{X}}^{(l+1)}$ corresponds to the embedded representation of a prototype. The coarsened adjacency matrix ${\mathit{A}}^{(l+1)}$ and embedding representation ${\mathit{X}}^{(l+1)}$ can be used together as input for another GNN layer.

Next, we will discuss how to generate the assignment matrix ${\mathit{S}}^{\left(l\right)}$ and embedding matrix ${\mathit{Z}}^{\left(l\right)}$ used in Equations (6) and (7). In this process, two separate GNNs are used to generate these two matrices, both taking the prototype representation ${\mathit{X}}^{\left(l\right)}$ and the coarsened adjacency matrix ${\mathit{A}}^{\left(l\right)}$ as network inputs. The generation of the embedding matrix ${\mathit{Z}}^{\left(l\right)}$ can be expressed as

$$\begin{array}{c}\hfill {\mathit{Z}}^{\left(l\right)}={\mathrm{GNN}}_{l,embed}({\mathit{A}}^{\left(l\right)},{\mathit{X}}^{\left(l\right)})\end{array}$$

(8)

The above equations utilize the adjacency matrix between prototypes at the l-th layer (from Equation (7)) and the prototype feature representation (from Equation (6)) and employ a standard GNN to obtain the new cluster node embeddings $Z^{\left(l\right)}$. In contrast, the pooling GNN at the l-th layer utilizes the input prototype feature representation $X^{\left(l\right)}$ and the adjacency matrix $A^{\left(l\right)}$ to generate an assignment matrix:

$$\begin{array}{c}\hfill {\mathit{S}}^{\left(l\right)}=\mathrm{Softmax}\left({\mathrm{GNN}}_{l,pool}({\mathit{A}}^{\left(l\right)},{\mathit{X}}^{\left(1\right)})\right)\end{array}$$

(9)

Here, the Softmax function is computed along the row dimension. The output dimension of ${\mathrm{GNN}}_{l,pool}$ corresponds to the maximum predefined number of prototypes at the l-th layer, which is a hyperparameter of the model. It is important to note that these two GNNs share the same input data but have different network parameters and serve different purposes: the embedding GNN generates new embedding representations for the input nodes, while the pooling GNN generates a probabilistic assignment matrix that assigns the input nodes to $n_{l+1}$ clusters.

2.4. Domain Alignment

After obtaining graph structures at different levels, domain alignment between the source and target domains needs to be performed. Domain alignment in this context includes distribution alignment and graph alignment (graph matching). The maximum mean discrepancy (MMD) distance is a commonly used method for aligning marginal distributions. In practice, the unbiased estimate of the MMD distance compares the squared distance between empirical kernel mean embeddings. The formula is expressed as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{MMD}}\left({\mathit{Z}}_s,{\mathit{Z}}_t\right)=& \frac{1}{n_s^2}\sum _{i=1}^{n_s}\sum _{j=1}^{n_s}r\left({\mathit{z}}_i^s,{\mathit{z}}_j^s\right)+\frac{1}{n_t^2}\sum _{i=1}^{n_t}\sum _{j=1}^{n_t}r\left({\mathit{z}}_i^t,{\mathit{z}}_j^t\right)\hfill \\ \hfill & -\frac{2}{n_s{n}_t}\sum _{i=1}^{n_s}\sum _{j=1}^{n_t}r\left({\mathit{z}}_i^s,{\mathit{z}}_j^t\right)\hfill \end{array}$$

(10)

where the kernel function $r(·,·)$ represents the inner product of features in RKHS. The variable $\mathit{z}$ represents the feature vectors extracted from the source domain ${\mathit{Z}}_s$ and the target domain ${\mathit{Z}}_t$. Specifically, ${\mathit{z}}_i^s$ denotes the i-th feature vector in the source domain, while ${\mathit{z}}_j^t$ denotes the j-th feature vector in the target domain. In theory, to detect small distribution differences, commonly used kernel functions are typically chosen, such as the Gaussian Radial Basis Function (GRBF) or Laplacian kernel function [31]. In practice, they are both positive semi-definite functions that satisfy the Mercer condition [32]. Considering the linear function $r({\mathit{x}}_i,{\mathit{x}}_j)={\mathit{x}}_i^T{\mathit{x}}_j$, the MMD distance simplifies to the mean difference between two distributions in the input space. Therefore, selecting an appropriate non-linear kernel function becomes crucial. In the experiments of this paper, the GRBF is used as the kernel function:

$$\begin{array}{c}\hfill r({\mathit{x}}_i,{\mathit{x}}_j)=\exp (-\frac{\parallel {\mathit{x}}_i-{\mathit{x}}_j{\parallel }^2}{2{\sigma }^2})\end{array}$$

(11)

This kernel function ensures that MMD is an unbiased estimator for differentiating between two probability distributions, where $\sigma$ is the Gaussian kernel parameter.

The WD evaluates the distance between node features from the source and target domains, excluding topological details. Conversely, the GWD assesses the graph structures by measuring the distances between pairs of nodes within each graph. By integrating both metrics, the alignment process effectively accounts for information from both nodes and edges, leading to improved graph matching.

Graph alignment is achieved through the computation of the graph optimal transport (GOT) distance, which involves two metrics: the Wasserstein distance (WD) and the Graph Wasserstein distance (GWD). While the WD quantifies the dissimilarity between the node features of the two domains without factoring in topological information, the GWD assesses the divergence in graph structures by gauging the separation between node pairs within each respective graph. By combining these two distances, the information of both nodes and edges can be effectively considered to achieve better graph matching. As shown in Figure 2, the computation of the GOT distance can be represented as

$$\begin{array}{c}\hfill GOT\left({\mathbf{G}}_s,{\mathbf{G}}_t\right)=\sum _{i,\widehat{i},j,\widehat{j}}\left(\gamma {\mathbf{T}}_{ij}c\left({\mathbf{g}}_i^s,{\mathbf{g}}_j^t\right)\right.\left.+(1-\gamma ){\mathbf{T}}_{\widehat{i}\widehat{j}}L\left({\mathbf{g}}_i^s,{\mathbf{g}}_j^t,{\widehat{\mathbf{g}}}_i^s,{\widehat{\mathbf{g}}}_j^t\right)\right)\end{array}$$

(12)

where ${\mathbf{T}}_{ij}$ and ${\mathbf{T}}_{\widehat{i}\widehat{j}}$ are the transformation matrices for the two pairs of nodes, namely (${\mathbf{g}}_i^s,{\mathbf{g}}_j^t$) and (${\widehat{\mathbf{g}}}_i^s,{\widehat{\mathbf{g}}}_j^t$), and $\gamma$ is set to 0.5 in the experiments. Therefore, the GOT loss can be represented as:

$$\begin{array}{c}\hfill {\mathcal{L}}_{GOT}({\mathit{Z}}_s,{\mathit{Z}}_t)=\sum _{l=0}^{L}GOT({\mathbf{G}}_s^l,{\mathbf{G}}_t^l)\end{array}$$

(13)

Here, ${\mathbf{G}}^l$ represents the subgraph obtained after processing through l layers of GNN, where $l=0$ denotes the original subgraph features generated from ${\mathbf{Z}}_s$ and ${\mathbf{Z}}_t$.

2.5. Loss Function

The cross-entropy loss function is used to predict the probabilities of the samples in the source and target domains and calculate the error based on the labels. Specifically, for a labeled source domain sample ${\mathit{x}}_i^s$, the cross-entropy loss is defined as

$$\begin{array}{c}\hfill {\mathcal{L}}_y({\mathit{p}}_i,{\mathit{y}}_i)=-\sum _{c=1}^C{y}_i^{c}log{p}_i^c\end{array}$$

(14)

where ${\mathit{y}}_i$ is the one-hot encoded label of the source domain sample and ${\mathit{p}}_i$ is the predicted probability distribution obtained through the Softmax function. Therefore, the classification loss of the GNN on the source domain dataset is defined as the average of the cross-entropy losses for all labeled source domain samples:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{GNN}}\left({\mathbf{Z}}_s,{\mathbf{Y}}_s\right)=\frac{1}{n_s}\sum _{i=1}^{n_s}{\mathcal{L}}_y\left(S\left({\mathbf{g}}_i^s\right),{\mathbf{y}}_i\right)\end{array}$$

(15)

where $S(·)$ represents the Softmax function.

In practice, training the entire GNN network solely using the gradient signal from the graph classification task is challenging. Therefore, an auxiliary link prediction (LP) objective is used to train the entire GNN network. Specifically, in each layer, this objective minimizes a link prediction loss, which is defined as

$$\begin{array}{c}\hfill {\mathcal{L}}_{LP}=\sum _{l=0}^L{\parallel {\mathit{A}}^{\left(l\right)}-{\mathit{S}}^{\left(l\right)}{\mathit{S}}^{{\left(l\right)}^T}\parallel }_F\end{array}$$

(16)

where $|·|$ represents the Frobenius norm.

During the training process, the losses ${\mathcal{L}}_{LP}$, ${\mathcal{L}}_{MMD}$, and ${\mathcal{L}}_{GOT}$ are incorporated into the final classification loss function, defined as

$$\begin{array}{cc}\hfill \mathcal{L}({\mathit{X}}_s,{\mathit{X}}_t,{\mathit{Y}}_s)=& {\mathcal{L}}_{\mathrm{GNN}}\left({\mathit{Z}}_s,{\mathit{Y}}_s\right)+{\lambda }_1{\mathcal{L}}_{MMD}({\mathit{Z}}_s,{\mathit{Z}}_t)+{\lambda }_2{\mathcal{L}}_{GOT}({\mathit{Z}}_s,{\mathit{Z}}_t)\hfill \\ \hfill & +{\lambda }_3{\mathcal{L}}_{LP}({\mathit{Z}}_s,{\mathit{Z}}_t)\hfill \end{array}$$

(17)

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are regularization parameters. ${\lambda }_1$ and ${\lambda }_2$ control the contributions of data distribution alignment and graph alignment, respectively, while ${\lambda }_3$ ensures that neighboring nodes are assigned together. During the training of the entire GNN network, the weights are adjusted based on the losses related to sample probability prediction, data distribution alignment, graph alignment, and link prediction. This adjustment aims to achieve more accurate cross-domain HSI classification.

3. Experiments

3.1. Dataset Description

Two commonly used hyperspectral image (HSI) datasets were utilized to assess the performance of our proposed algorithm. The specifics of each dataset are outlined below.

3.1.1. Houston

This dataset comprises scene data obtained using different sensors in and around the University of Houston for two different years (2013 and 2018). The Houston 2013 dataset consists of 144 spectral bands covering a wavelength range of 380–1050 nm with a spatial resolution of 2.5 m, resulting in a total of 349 × 1905 pixels. The Houston 2018 dataset shares the same wavelength range, consisting of 48 spectral bands, and has a spatial resolution of 1 m. Both datasets consist of seven consistent categories.

In the experiments, the 48 spectral bands (wavelength range: 0.38–1.05 μm) corresponding to the Houston 2018 scenes were extracted from the Houston 2013 scenes. An overlapping region of size 209 × 955 was selected. The names of land cover categories along with their sample counts are listed in

Table 1. Number of source and target samples for the Houston dataset.

Class	Class Name	Number of Classes
		Houston 2013 (Source)	Houston 2018 (Target)
1	Grass healthy	345	1353
2	Grass stressed	365	4888
3	Trees	365	2766
4	Water	285	22
5	Residential buildings	319	5347
6	Non-residential buildings	408	32,459
7	Road	443	6365
Total		2530	53,200

, while Figure 4 illustrates pseudocolored images and ground truth maps.

3.1.2. HyRANK

The HyRANK dataset was established within the framework of the International Society for Photogrammetry and Remote Sensing (ISPRS) scientific program. This dataset consists of satellite hyperspectral images collected using the Hyperion sensor (EO-1, USGS) and contains 176 spectral bands. The dataset comprises two labeled scenes, Dioni and Loukia, with dimensions of $250\times 1376$ and $249\times 945$ pixels, respectively. The dataset includes 12 consistent categories, and these categories along with their quantities are listed in

Table 2. Number of source and target samples for the HyRank dataset.

Class	Class Name	Number of Classes
		Dioni (Source)	Loukia (Target)
1	Dense Urban Fabric	1262	206
2	Mineral Extraction Sites	204	54
3	Non-Irrigated Arable Land	614	426
4	Fruit Trees	150	79
5	Olive Groves	1768	1107
6	Coniferous Forest	361	422
7	Dense Sderophyllous Vegetation	5035	2996
8	Sparse Sderophyllous Vegetation	6374	2361
9	Sparsely Vegetated Areas	1754	399
10	Rocks and Sand	492	453
11	Water	1612	1393
12	Coastal Water	398	421
Total		20,024	10,317

. Pseudocolored images and ground truth maps for the two scenes are presented in Figure 5.

3.2. Experimental Settings

To comprehensively evaluate the performance of the proposed method, a comparison was made with some classical unsupervised deep domain adaptation algorithms, including DAN [16], DAAN [33], Multi-Representation Adaptation Network (MRAN) [34], Deep Subspace Alignment Network (DSAN) [15], Heterogeneous Transfer Convolutional Neural Network (HTCNN) [35], and bi-classifier adversarial Network (BCAN) [19]. Regarding training details, 5% of the samples were randomly selected from the source domain for training from both datasets. Notably, only data from the source domain (SD) and target domain (TD), along with labels from the source domain, were utilized during model training for all comparative methods, without leveraging label information from the target domain. Network optimization was performed using the Nesterov Adam algorithm. To comprehensively assess the classification performance of various methods, evaluation metrics including per-class accuracy (PA), overall accuracy (OA), average accuracy (AA), and the Kappa coefficient were employed. Specifically, OA represents the fraction of correctly differentiated samples, PA indicates accuracy for each class, AA is the average of all per-class accuracies, and the Kappa coefficient measures the robustness of the agreement’s degree.

All experiments were conducted using the PyTorch framework on a computer equipped with an Intel Core i7-9700 processor (32 GB RAM) and Nvidia GTX 3090 GPU (24 GB memory). Graph convolutions were implemented using the torch geometric library. Based on experimentation, a three-layer graph network yielded optimal results. Following the graph convolutions, feature construction was enhanced using a CNN layer and a Batch Normalization (BN) layer with a kernel size, stride, and padding all set to one. Optimization utilized batch stochastic gradient descent with a learning rate of 0.01, a momentum of 0.5, and L2 norm decay set to $1\times {10}^{-4}$. Training iterations totaled 2000 for the Houston dataset and 400 for the HyRANK dataset.

3.3. Experimental Results

Table 3. Classification results of different methods on the Houston dataset.

Class	DAN	DAAN	MRAN	DSAN	HTCNN	BCAN	HPGA
1	55.95 ± 4.15	61.57 ± 3.42	56.39 ± 5.24	57.95 ± 6.22	4.85 ± 3.78	69.56 ± 7.56	45.08 ± 3.72
2	72.18 ± 6.85	76.94 ± 2.89	75.57 ± 6.48	67.90 ± 9.13	71.57 ± 7.35	82.82 ± 3.27	89.13 ± 1.46
3	62.87 ± 4.93	66.67 ± 3.46	68.00 ± 4.58	71.69 ± 5.77	35.75 ± 5.34	57.40 ± 8.28	72.60 ± 4.74
4	100.00 ± 0.00	72.73 ± 2.78	63.64 ± 4.81	81.82 ± 3.67	54.64 ± 6.98	63.18 ± 5.27	100.00 ± 0.00
5	56.33 ± 4.78	52.76 ± 7.44	66.50 ± 3.65	61.79 ± 8.23	54.40 ± 5.76	80.49 ± 3.66	80.14 ± 7.55
6	74.11 ± 3.57	69.64 ± 4.88	68.54 ± 2.71	70.26 ± 11.85	90.80 ± 4.98	76.07 ± 3.90	72.82 ± 4.71
7	31.80 ± 7.33	54.23 ± 8.23	54.36 ± 4.76	54.53 ± 7.54	44.05 ± 9.77	53.14 ± 3.19	55.48 ± 6.19
OA (%)	66.05 ± 5.87	66.59 ± 6.72	66.95 ± 4.98	67.08 ± 2.48	74.72 ± 4.35	60.81 ± 6.99	72.27 ± 5.28
AA (%)	64.74 ± 3.67	66.77 ± 4.87	64.71 ± 3.85	66.56 ± 3.62	50.72 ± 5.15	73.25 ± 5.84	78.36 ± 5.18
Kappa (%)	47.82 ± 3.72	50.83 ± 6.77	51.51 ± 4.71	52.05 ± 1.20	55.24 ± 4.11	58.16 ± 6.16	58.29 ± 1.76

and

Table 4. Classification results of different methods on the HyRANK dataset.

Class	DAN	DAAN	MRAN	DSAN	HTCNN	BCAN	HPGA
1	10.68 ± 3.58	18.45 ± 6.41	11.65 ± 7.23	26.21 ± 4.37	5.34 ± 1.89	12.14 ± 8.32	10.22 ± 5.69
2	40.74 ± 7.12	14.81 ± 3.94	0.00 ± 0.00	18.52 ± 9.01	0.00 ± 0.00	31.85 ± 2.18	12.02 ± 3.57
3	23.47 ± 4.91	7.51 ± 1.47	3.52 ± 9.27	23.71 ± 6.81	45.07 ± 5.63	8.54 ± 2.89	69.48 ± 1.54
4	3.80 ± 2.22	10.13 ± 9.34	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	7.09 ± 1.93	0.00 ± 0.00
5	17.71 ± 9.84	10.12 ± 3.45	29.09 ± 8.57	48.69 ± 2.67	18.61 ± 4.55	17.52 ± 3.78	20.23 ± 5.41
6	4.98 ± 3.29	5.21 ± 2.43	40.52 ± 1.12	45.97 ± 6.54	2.61 ± 7.92	20.62 ± 4.78	13.88 ± 9.87
7	62.05 ± 4.67	81.91 ± 3.23	64.12 ± 6.98	60.58 ± 5.33	77.77 ± 8.45	85.34 ± 7.56	77.24 ± 2.88
8	65.14 ± 8.69	72.91 ± 5.43	57.65 ± 3.97	67.26 ± 9.15	62.22 ± 4.36	71.01 ± 7.83	69.72 ± 1.62
9	63.16 ± 7.19	7.02 ± 2.11	71.18 ± 9.45	37.09 ± 6.54	6.27 ± 8.22	57.84 ± 3.48	7.52 ± 5.17
10	0.00 ± 0.00	0.22 ± 0.15	23.18 ± 8.44	4.86 ± 1.75	0.00 ± 0.00	0.00 ± 0.00	58.28 ± 9.34
11	100 ± 0.00	100 ± 0.00	100 ± 0.00	100 ± 0.00	100 ± 0.00	100 ± 0.00	100 ± 0.00
12	95.96 ± 8.73	100 ± 0.00	100 ± 0.00	100 ± 0.00	97.86 ± 3.24	97.96 ± 7.89	57.72 ± 4.53
OA (%)	56.31 ± 5.87	60.47 ± 3.21	58.32 ± 9.48	60.92 ± 7.63	58.63 ± 1.22	42.49 ± 2.91	62.16 ± 8.36
AA (%)	40.64 ± 2.34	35.69 ± 9.12	41.74 ± 3.87	44.41 ± 6.58	34.65 ± 5.14	64.31 ± 1.82	44.84 ± 2.47
Kapp (%)	45.94 ± 8.54	49.71 ± 3.98	49.14 ± 7.12	52.44 ± 1.35	47.47 ± 6.47	54.99 ± 2.44	53.03 ± 4.78

, respectively, present the classification performance of all compared methods on the two sets of cross-scene datasets, including the the mean and standard deviation of ten experiments. Results indicate that HPGA consistently exhibits the best performance across all target domains. Particularly on the Houston 2018 dataset, HPGA achieves an average accuracy (AA) that is nearly 12% higher than the runner-up method. Although HPGA’s overall accuracy (OA) on the Houston dataset is slightly lower than that of HTCNN, it shows a more significant improvement in average accuracy (AA) compared to overall accuracy. This indicates that HPGA performs well across all categories, not just having high accuracy overall. This demonstrates that the algorithm proposed in this chapter has better balance and robustness when dealing with different categories, which is particularly important for practical applications in high-spectral-image classification where various complex land cover categories exist, necessitating good performance across all categories. For the HyRANK dataset, HPGA achieves the best performance in OA, AA, and the Kappa metric. However, classification performance is weaker on some categories, such as the fourth class (orchards), the fifth class (Olive Groves), and the sixth class (Coniferous Forests). This may be due to the fact that the topological relationships among land cover categories in forest scenes are significantly weaker than those in urban scenes. The experimental results on the HyRANK and Pavia cross-scene datasets show that our proposed method, which incorporates topological relationships among land cover categories in the training process, significantly surpasses domain adaptation methods that focus solely on statistical feature alignment.

Classification maps are further shown in Figure 6 and Figure 7. The results in Figure 6 illustrate HPGA’s superiority, showing reduced noise and better alignment with actual land cover distribution. In the marked area (Figure 6), HPGA accurately classifies trees and avoids misclassifying non-residential buildings as roads. In the HyRANK dataset (Figure 7), HPGA utilizes topological structure information to achieve a more accurate classification with lower noise levels in marked regions. Experimental results confirm HPGA’s excellent performance and strong generalization capabilities across several datasets.

In cross-scene hyperspectral image classification, a significant spectral shift occurs where the spectral reflectance of the same land cover category differs between the source domain (SD) and the target domain (TD).

To further demonstrate the cross-domain learning capability and alignment performance of our method HPGA, we visualized the domain-invariant features learned by HPGA. Figure 8 presents the T-SNE visualization using the original features from the Houston dataset and the features extracted by HPGA. In the figure, red hollow circles and blue hollow triangles represent samples from the source domain and target domain, respectively. The first row in each subplot displays the feature distribution of the source and target domains after dimensionality reduction of the original features, while the second row shows the feature distribution after domain adaptation. It is evident from the figure that the original samples from the source and target domains exhibit inconsistent distributions for the same categories. However, after applying the proposed domain adaptation method HPGA, the feature distribution distances are effectively reduced. This allows a model trained on the source domain to be directly applied to the target domain without the need for additional data labeling or model retraining. This phenomenon demonstrates that through distribution alignment and topological structure alignment, the proposed HPGA can project data from both domains into a domain-invariant subspace, thereby alleviating the spectral shift phenomenon to a large extent. This further confirms the effectiveness and practicality of HPGA in the problem of cross-scene hyperspectral image classification.

4. Discussion

In this section, we examine how various modules impact the proposed method’s performance and conduct experiments to assess the sensitivity of hyperparameters.

4.1. Ablation Experiments

To assess the influence of individual loss terms on the model’s performance, we conducted ablation studies on all components of the loss function. The overall accuracy (OA), average accuracy (AA), and Kappa coefficient results for the two datasets are presented in

Table 5. Ablation experiments of HPGA on two datasets.

Module				Dataset
${\mathcal{L}}_{\mathit{GNN}}$	${\mathcal{L}}_{\mathit{MMD}}$	${\mathcal{L}}_{\mathit{GOT}}$	${\mathcal{L}}_{\mathit{LP}}$	Houston			HyRANK
				OA (%)	AA (%)	Kappa (%)	OA (%)	AA (%)	Kappa (%)
✓	-	-	-	61.22 ± 3.45	64.29 ± 2.31	50.85 ± 4.67	45.80 ± 5.24	32.16 ± 1.78	40.23 ± 2.59
✓	✓	-	-	68.42 ± 2.54	70.17 ± 4.13	54.76 ± 3.11	56.91 ± 3.98	37.22 ± 1.67	46.34 ± 2.46
✓	✓	✓	-	69.25 ± 5.47	75.84 ± 2.36	56.89 ± 4.89	61.19 ± 1.24	41.75 ± 3.21	52.19 ± 2.88
✓	✓	✓	✓	72.27 ± 5.28	78.36 ± 5.18	58.29 ± 1.76	62.16 ± 3.72	44.84 ± 2.47	53.03 ± 4.78

. These results reveal that the model exhibits the poorest performance when solely utilizing the cross-entropy classification loss. Incorporating the MMD loss term results in a notable improvement in model accuracy on both datasets, with an increase of approximately 10%. This suggests that leveraging the MMD loss contributes significantly to enhancing the model’s generalization capability. Furthermore, with the introduction of the GOT loss, the model’s accuracy on the Houston dataset increases by 0.83%, while on the HyRANK dataset, it improves by 4.28%. This suggests that topological structure alignment provides different degrees of improvement for different datasets. Simultaneously, when the link prediction loss term is applied, the model’s accuracy on both datasets increases by 1% to 2%, indicating that the link prediction loss also contributes to the model’s performance improvement. The ablation experiment results indicate that each loss term contributes positively to enhancing the model’s performance and generalization capability. Notably, when all loss terms are combined, the model achieves an optimal classification performance on the target domain.

4.2. Parameter Sensitivity Analysis

A parameter sensitivity analysis was carried out to assess the impact of parameter variations on HPGA’s performance across the two target domains. The regularization parameters, ${\lambda }_1$ and ${\lambda }_2$, are crucial hyperparameters in the HPGA algorithm. These parameters individually regulate the influences of distribution alignment and graph alignment on domain generalization. The values of parameters ${\lambda }_1$ and ${\lambda }_2$ were varied within the set $0.001,0.01,0.1,1,10$. Figure 9 illustrates the change in HPGA’s classification results (OA) on the two experimental datasets under different combinations of ${\lambda }_1$ and ${\lambda }_2$. The results indicate that for the Houston dataset, the optimal parameters are ${\lambda }_1=1$ and ${\lambda }_2=0.1$, while for the HyRANK dataset, the optimal parameters are ${\lambda }_1={\lambda }_2=0.01$. This parameter sensitivity analysis underscores the robustness and stability of the HPGA algorithm across diverse datasets. Depending on specific application contexts and dataset characteristics, these hyperparameters can be fine-tuned to enhance performance accordingly.

In addition to ${\lambda }_1$ and ${\lambda }_2$, the parameter ${\lambda }_3$ is also an important parameter in the HPGA model. To find the optimal regularization parameter ${\lambda }_3$, a parameter sensitivity analysis was conducted on both sets of experimental data. ${\lambda }_1$ and ${\lambda }_2$ were both set to 0.1, and then the HPGA model was separately trained on each set of experimental data. The classification performance scores on each set of experimental data were recorded, and the regularization parameter that yielded the best performance was selected. The experimental outcomes are presented in

Table 6. Effect of parameter ${\lambda }_3$ on classification results on two datasets.

OA (%)	${\mathit{\lambda }}_3$
	0.001	0.01	0.1	1	10
Houston 2018	68.85 ± 2.34	71.15 ± 3.45	71.06 ± 1.78	67.35 ± 4.56	67.04 ± 2.67
Louki	60.17 ± 3.12	59.82 ± 1.54	60.84 ± 2.67	59.77 ± 4.89	60.04 ± 2.45

. It is evident that the regularization parameter ${\lambda }_3$ demonstrates greater robustness in comparison to ${\lambda }_1$ and ${\lambda }_2$ with respect to the results.

To validate the effectiveness of the hierarchical prototype alignment method, we experimented with the number of layers L involved in the alignment loss, as shown in

Table 7. Effect of parameter L on classification results on two datasets.

OA (%)	L
	1	2	3	4
Houston 2018	69.74 ± 3.14	72.03 ± 4.23	72.27 ± 2.67	68.64 ± 3.21
Louki	57.26 ± 4.12	59.97 ± 2.45	62.16 ± 3.36	59.77 ± 1.78

. The results indicate that increasing the number of layers generally improves classification performance. For the Houston 2018 dataset, overall accuracy (OA) rises from 69.4% with $L=1$ to 72.03% with $L=3$, then drops to 68.64% with $L=4$. Similarly, for the Louki dataset, OA increases from 57.26% with $L=1$ to 60.84% with $L=3$, then slightly decreases to 59.77% with $L=4$. These findings suggest that while more layers initially enhance accuracy, an optimal number $(L=3)$ exists beyond which performance declines due to increased complexity and potential overfitting. Therefore, the hierarchical prototype structure can improve classification effectiveness.

5. Conclusions

Conventional domain adaptation techniques in hyperspectral imagery mainly tackle the spectral shift between source and target domains by aligning statistical feature distributions, often overlooking the intricate relationships among different land cover classes. In response, this paper introduces a novel approach for cross-scene hyperspectral classification, employing hierarchical prototype graph alignment. This method harnesses prototypes as representative embedded representations of all samples within the same category. It achieves domain alignment across multiple scales through a series of graph convolution and pooling operations. In addition to distribution alignment, graph matching is introduced, effectively harmonizing semantic and topological information. The proposed method significantly enhances the accuracy and generalization capacity of cross-scene hyperspectral image classification.