Leveraging Self-Distillation and Disentanglement Network to Enhance Visual–Semantic Feature Consistency in Generalized Zero-Shot Learning

Abstract

Generalized zero-shot learning (GZSL) aims to simultaneously recognize both seen classes and unseen classes by training only on seen class samples and auxiliary semantic descriptions. Recent state-of-the-art methods infer unseen classes based on semantic information or synthesize unseen classes using generative models based on semantic information, all of which rely on the correct alignment of visual–semantic features. However, they often overlook the inconsistency between original visual features and semantic attributes. Additionally, due to the existence of cross-modal dataset biases, the visual features extracted and synthesized by the model may also mismatch with some semantic features, which could hinder the model from properly aligning visual–semantic features. To address this issue, this paper proposes a GZSL framework that enhances the consistency of visual–semantic features using a self-distillation and disentanglement network (SDDN). The aim is to utilize the self-distillation and disentanglement network to obtain semantically consistent refined visual features and non-redundant semantic features to enhance the consistency of visual–semantic features. Firstly, SDDN utilizes self-distillation technology to refine the extracted and synthesized visual features of the model. Subsequently, the visual–semantic features are then disentangled and aligned using a disentanglement network to enhance the consistency of the visual–semantic features. Finally, the consistent visual–semantic features are fused to jointly train a GZSL classifier. Extensive experiments demonstrate that the proposed method achieves more competitive results on four challenging benchmark datasets (AWA2, CUB, FLO, and SUN).

1. Introduction

Deep learning models typically necessitate extensive, heavily labeled data during training, incurring significant human and resource costs. The introduction of Zero-Shot Learning (ZSL) effectively mitigates this constraint of deep learning models by learning the mapping relationship from auxiliary (e.g., semantic) to visual space, facilitating the classification and recognition of unseen classes [1]. However, traditional ZSL settings are somewhat idealized as they assume that the test set solely comprises samples from seen classes, which is not reflective of real-world scenarios. Generalized Zero-Shot Learning (GZSL) introduces a more rigorous task where the test set can encompass samples from both seen and unseen classes, better aligning with practical needs.

Presently, research on GZSL primarily centers on two distinct strategies. Firstly, some researchers focus on methods grounded in Generative Adversarial Networks (GANs) [2,3,4,5,6,7], which employ generative models to learn the mapping relationship from semantic attributes to visual features and subsequently synthesize visual samples of unseen classes based on semantic information. Secondly, other researchers concentrate on embedding-based methods [8,9,10,11,12,13,14], striving to embed visual samples into a shared feature space to accurately reflect the semantic similarity between different classes. Through this approach, models can conduct classification reasoning using structural information in the embedding space with minimal or zero samples. Both of these strategies align visual–semantic features through either generative or embedding methods, tackling the challenges inherent in ZSL.

However, they introduce a new challenge: they often overlook the potential inconsistency in the visual–semantic features to be aligned. As illustrated in Figure 1, certain visual features, such as “fish fin”, demonstrate distinctiveness in discerning image samples within the visual modality. However, these features are not encompassed within the manually annotated semantic attributes, thus termed as semantically inconsistent visual features. Moreover, within the semantic modality, there may also be redundant semantic attributes that are inconsistent with visual features. For instance, classes like “dolphin”, “seal”, and “killer whales” in Figure 1 share multiple redundant semantic attributes unrelated to their visual features, such as “stripe:no”, “tree:no”, “vegetation:no”, “long legs:no”, and “long neck:no”. Most GZSL methods overlook these inconsistent visual–semantic features and forcibly align them, potentially introducing biases in visual–semantic feature alignment and undermining the recognition of unseen classes. Moreover, current GZSL approaches frequently employ pre-trained ImageNet models for extracting GZSL visual features and training generative models to synthesize visual features of unseen classes. However, the presence of cross-modal dataset bias [3] implies that the extracted and synthesized visual features might lack refinement and could stray from the visual features necessary for ZSL tasks, thereby worsening the problem of visual–semantic feature inconsistency.

We think that extracting and synthesizing refined visual features to enhance the semantic consistency of visual features, and segregating semantic-consistent visual features and visually consistent non-redundant semantic features from raw visual–semantic features to bolster the consistency of visual–semantic features, can alleviate the aforementioned issues. Hence, this paper proposes a GZSL framework that enhances the consistency of visual–semantic features using self-distillation and disentanglement network. Specifically, we first devise a self-distillation module that leverages self-distillation technology to augment both the feature extraction model and the generative model in the context of generative GZSL. This enables them to concurrently acquire refined mid-layer features and soft label knowledge from the auxiliary self-teacher network, thereby stimulating the model to extract and synthesize refined visual features. Additionally, we devise a disentanglement network applied to the visual–semantic modality. For instance, in the visual modality, the visual disentanglement encoder projects visual features into $z_r$ and $z_d$. To ensure the consistency of $z_r$ with semantic features, visual–semantic features are cross-reconstructed, and a semantic relationship-matching method is employed to calculate the compatibility score between $z_r$ and semantic information to guide the learning of $z_r$. Furthermore, a latent representation-independent method is applied to enforce the independence between $z_r$ and $z_d$. Ultimately, the disentanglement network attains consistent visual–semantic features, which are amalgamated to jointly train a GZSL classifier.

In summary, the contributions of this paper are as follows:

• We identified that most models typically do not handle visual–semantic inconsistent features and directly align them, which may lead to alignment bias. We propose an approach to enhance the consistency of visual–semantic features by refining visual features and disentangling original visual–semantic features.
• We designed a self-distillation embedding module, which generates soft labels through an auxiliary self-teacher network and employs soft label distillation and feature map distillation methods to refine the original visual features of seen classes and synthesized visual features of unseen classes from the generator, thereby enhancing the semantic consistency of visual features.
• We proposed a disentanglement network, which encodes visual–semantic features into latent representations and promotes visual–semantic consistent features to be separated from original features through semantic relation matching and latent representation independence methods, significantly enhancing the consistency of visual–semantic features.
• Extensive experiments on four GZSL benchmark datasets demonstrate that our model can separate refined visual–semantic features with consistency from original visual–semantic features, thereby alleviating alignment bias caused by visual–semantic inconsistency and improving the performance of GZSL models.

2. Related Work

2.1. Generative-Based Generalized Zero-Shot Learning

In recent years, numerous studies have employed generative models to bolster the efficacy of GZSL tasks. GANs or VAEs are commonly utilized in generative GZSL to synthesize visual features for unseen classes. These synthesized visual features for unseen classes are subsequently integrated with original visual features for seen classes to train classifiers. For example, Narayan et al. [15] employed VAEs and GANs to refine the quality of synthesized visual features for unseen classes. They introduced a feedback module to regulate the generator’s output, effectively diminishing ambiguity between classes. Zhang et al. [16] combined generative and embedding-based models by projecting real and synthesized samples onto an embedding space for classification, establishing a hybrid ZSL framework that effectively addresses data imbalance issues. Li et al. [17] proposed an innovative approach that integrates a Transformer model with VAE and GAN, capitalizing on the rich data representation from VAE and the diversity of data generated by GAN to mitigate dataset diversity bias, while utilizing Transformer to enhance semantic consistency. DGCNet [18] introduced a Dual Uncertainty Guided Cycle-Consistent Network, which examines the relationship between visual and semantic features through a cycle-consistent embedding framework and dual uncertainty-aware modules, effectively addressing alignment shift problems and enhancing model discriminability and adaptability. However, these methods often ignore the existence of semantically inconsistent visual features and redundant semantic attributes in the original visual–semantic features, which may affect the correct alignment of visual–semantic features. Instead, by first decoupling visual–semantic consistent features before alignment, we have improved the model’s accuracy.

2.2. Knowledge Distillation

Knowledge distillation [19] serves as a model compression technique, aiming to reduce the size and computational complexity of a model by transferring knowledge from a complex neural network (referred to as the teacher network) to a smaller neural network (referred to as the student network). Initially, the concept of knowledge distillation emerged by encouraging the student network to imitate the output log-likelihood of the teacher network [20]. Subsequent research introduced intermediate layer distillation methods, enabling the student network to acquire knowledge from the convolutional layers of the teacher network with feature map-level locality [21,22,23,24], or from the penultimate layer of the teacher network [25,26,27,28,29]. However, these methods necessitate pre-training a complex model as the teacher network, a process consuming substantial time and resources. Some recent studies have proposed self-knowledge distillation [30,31], enhancing the training of the student network by leveraging its own knowledge without requiring an additional teacher network. For instance, Zhang et al. [32] segmented the network into several parts and compressed deep-layer knowledge into shallow layers. DLB [33] utilizes instant soft targets generated in the training process of the previous iteration for distillation, achieving performance improvement without altering the model structure. FRSKD [34] introduces an auxiliary self-teacher network to refine knowledge transfer to the student’s classifier network, capable of performing self-knowledge distillation using both soft labels and feature map distillation. This paper adopts the concept proposed by FRSKD to construct a self-distillation embedding module, aiming to refine the original seen visual features and the unseen visual features synthesized by the generator.

3. Materials and Methods

3.1. Problem Definition

In GZSL, the dataset comprises visual features X, semantic attributes C, and labels Y, which can be divided into seen classes S and unseen classes U. Specifically, the visual feature set is defined as $X=\left\{X_S,X_U\right\}$, and the corresponding label set is represented as $Y=\left\{Y_S,Y_U\right\}$, where $Y_S$ and $Y_U$ are disjoint sets. Semantic attributes are defined as $C=\left\{C_S,C_U\right\}$. Visual features $x_s^i$ and $x_u^i$ are defined as the ith visual feature, where $x_s^i\in X_S$ and $x_u^i\in X_U$. The corresponding labels for seen and unseen classes are denoted as $y_s^i$ and $y_u^i$, while $c_s^i$ and $c_u^i$ represent the ith semantic feature, where $c_s^i\in C_S$ and $c_u^i\in C_U$. Thus, the training dataset is defined as $D_s={\left\{x_s^i,c_s^i,y_s^i\right\}}_{i=1}^{N^s}$, and the testing dataset is defined as $D_u={\left\{x_u^i,c_u^i,y_u^i\right\}}_{i=1}^{N^u}$. The objective of GZSL is to learn a classifier $F_{GZSL}:X\to Y_S\cup Y_U$.

3.2. Overall Framework

The SDDN architecture primarily comprises two key modules, as depicted in Figure 2. The first module, known as the self-distillation embedding module, utilizes feature fusion techniques and an auxiliary self-teacher network to transfer refined visual features to the student network. It then employs both soft label distillation and feature map distillation to facilitate the generation of refined features by the generative model, thereby enhancing the consistency of visual–semantic features. The second module, referred to as the disentanglement network, employs semantic relationship matching (SRM) method and latent representation independent (IND) method to guide the visual–semantic disentanglement autoencoder in decoupling semantic-consistent visual features and non-redundant semantic features, further strengthening the consistency of visual–semantic features.

3.3. Self-Distillation Embedding Module

In order to refine the seen visual features extracted by the pre-trained ResNet101 [35] and the unseen visual features synthesized by the generator, thereby improving their semantic consistency. We designed a self-distillation embedded (SDE) module, as shown in Figure 3. The SDE module, comprised of a self-distillation feature refinement (SDFR) module and a self-distillation conditional generation (SDCG) module, was designed for this purpose. The SDFR module integrates top-down and bottom-up feature fusion methods [36] to direct the auxiliary self-teacher network in generating refined intermediate feature maps and soft labels. Following this, feature distillation and soft label distillation methods refine the visual features extracted by the student network. The SDCG module encompasses a generator and a discriminator. It trains the generator to synthesize visual features through a generative adversarial approach and employs feature map distillation and soft label distillation methods for refining the generated visual features.

3.3.1. Auxiliary Self-Teacher Network

To refine the visual features extracted by the pre-trained ResNet101 and pass them to the generator of the SDCG module, we devised an auxiliary self-teacher network T based on the architecture proposed by Ji et al. [34]. This auxiliary self-teacher network, depicted in green in Figure 3, also uses the pre-trained ResNet101. Subsequently, we utilized top-down and bottom-up feature fusion methods [36] to guide the auxiliary self-teacher network in generating refined intermediate feature maps $x_{T_i}^t$. At the layer preceding the classification layer $f_t$, this network outputs the final extracted visual features $x^t$, while the last classification layer $f_t$ outputs soft labels $P_t$. Deep neural networks excel at learning representations at various levels; hence, outputs from intermediate and output layers can both contribute to training the student network. In our methodology, we employ ResNet101 (depicted in blue in Figure 3) as the student network A, enriching the visual features extracted by ResNet101 with soft labels $P_t$ from the output of the self-teacher network and refined feature maps $x_{T_i}^t$ from the intermediate layers. The formula for generating $P_t$ in the auxiliary self-teacher network is defined as follows:

$$\begin{array}{c}\hfill P_t=\frac{\exp (f_t\left(x^t\right)/T)}{{\sum }_{j=1}^n\exp (f_t\left(x_j^t\right)/T)}\end{array}$$

(1)

Here, T is the temperature parameter [20], typically set to 1. Higher values of T result in softer class probability distributions. $f_t$ denotes the classifier of the auxiliary self-teacher network. The student network learns from $P_t$ through KL divergence, expressed as

$$\begin{array}{c}\hfill {\mathcal{L}}_{KD}(x^a,P_t,T)=D_{KL}\left(\frac{\exp (f_a\left(x^a\right)/T)}{{\sum }_{j=1}^n\exp (f_a\left(x_j^a\right)/T)}\left|\right|P_t\right)\end{array}$$

(2)

The intermediate layer feature outputted by the ith student network block is denoted as $x_{A_i}^a$. The student network learns the refined intermediate layer features $x_{T_i}^t$ produced by the auxiliary self-teacher network through the feature distillation method, which is implemented by the loss function ${\mathcal{L}}_F$. The definition of ${\mathcal{L}}_F$ is as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_F(x_{T_i}^t,x_{A_i}^a)=\sum _{i=1}^n\left|\right|\varphi \left(x_{T_i}^t\right)-\varphi \left(x_{A_i}^a\right){\left|\right|}_2\end{array}$$

(3)

where $\varphi$ represents the channel pooling function. Additionally, class predictions are made on the enhanced visual features, and the cross-entropy loss ${\mathcal{L}}_{CE}$ between the prediction results and the true labels is minimized to ensure the accuracy of the enhanced visual features. Finally, the loss function for refining visual features by the auxiliary self-teacher network is denoted as ${\mathcal{L}}_{STN}$, which is defined as

$$\begin{array}{c}\hfill {\mathcal{L}}_{STN}={\mathcal{L}}_{CE}(x^t,y)+{\mathcal{L}}_{CE}(x^a,y)+\alpha {\mathcal{L}}_{KD}(x^a,P_t,T)+\beta {\mathcal{L}}_F(x_{T_i}^t,x_{A_i}^a)\end{array}$$

(4)

3.3.2. Self-Distillation Conditional Generation Module

In order to synthesize refined visual features for unseen classes, thereby addressing the issue of lacking unseen visual features in ZSL, we devised a self-distillation conditional generation module. This module employs a conditional generator G and a discriminator D to form a generative adversarial network. During the training of this module, Gaussian noise $N(0,1)$ and semantic descriptors of seen classes $c_s$ are firstly utilized as conditional inputs to G, to synthesize visual features $x_s^{\prime }=G(c_s,N)$. Subsequently, the synthesized $x_s^{\prime }$ is fed into the trained student network classifier $f_a$ to obtain soft label $f_a\left(x_s^{\prime }\right)$. To create a clear contrast with the synthesized fake seen class visual features $x_s^{\prime }$, we define the real seen visual features $x^a$ obtained by the student network in Figure 3 as $x^s$, minimizing the loss between $f_a\left(x_s^{\prime }\right)$ and soft labels $P_t$ to ensure the consistency between $x_s^{\prime }$ and real visual features $x_s$. The loss function is defined as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{KD}(x_s^{\prime },P_t,T)=D_{KL}\left(\frac{exp(f_a\left(x_s^{\prime }\right)/T)}{{\sum }_{j=1}^{n}exp(f_a\left(x_{s_j}^{\prime }\right)/T)}\left|\right|P_t\right)\end{array}$$

(5)

Simultaneously, to ensure the accuracy of the synthesized visual features, we compute the cross-entropy loss between $f_a\left(x_s^{\prime }\right)$ and the real labels $Y_S$. Additionally, the discriminator D is employed to distinguish between real seen class samples $(x_s,c_s)$ and synthesized seen class samples $(x_s^{\prime },c_s)$, minimizing their loss ${\mathcal{L}}_{wgan}$ to ensure that the visual features synthesized by the generator are close to the real visual features.

$$\begin{array}{c}\hfill {\mathcal{L}}_{wgan}(x_s,x_s^{\prime },c_s)=\mathbb{E}\left[D(x_s,c_s)\right]-\mathbb{E}\left[D(x_s^{\prime },c_s)\right]-\lambda \mathbb{E}\left[\right(\left|\right|{\nabla }_{{\widehat{x}}_s}D({\widehat{x}}_s,c_s){\left|\right|}_2-1{)}^2]\end{array}$$

(6)

Here, ${\widehat{x}}_s=\alpha x_s+(1-\alpha )x_s^{\prime },\alpha \sim U(0,1)$, and $\lambda$ represents the penalty coefficient. The loss function for the SDCG module is as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{SDCG}={\mathcal{L}}_{KD}(x_s^{\prime },P_t,T)+{\mathcal{L}}_{wgan}(x_s,x_s^{\prime },c_s)+\lambda {\mathcal{L}}_{CE}(x_s^{\prime },Y_S)\end{array}$$

(7)

Finally, the overall loss of the SDE module is ${\mathcal{L}}_{SDE}=L_{STN}+L_{SDCG}.$

3.4. Disentanglement Network

To further bolster the consistency of visual–semantic features, we propose a disentanglement network. This network utilizes a semantic relation matching method and an independent latent representation method to guide the visual–semantic disentangled autoencoder in separating visually consistent features and non-redundant semantic features from the original data. Furthermore, it aligns these features using a cross-reconstruction method to further strengthen their consistency.

3.4.1. Visual–Semantic Disentangled Autoencoder

The visual–semantic disentangled autoencoder (VSDA) comprises two parallel variational autoencoders dedicated to processing visual and semantic modalities separately. Each variational autoencoder includes a disentangled encoder and decoder. The disentangled encoder maps the feature space to the latent space, while the decoder maps the latent space back to the feature space. The primary role of the VSDA is to acquire effective latent representations, $z_1$ and $z_2$, for visual–semantic features and to separate $z_1$ along the column dimension into $z_r$ and $z_d$, and $z_2$ into $c_r$ and $c_d$. Specifically, the visual disentanglement encoder $E_V$ and the semantic disentanglement encoder $E_S$ encode visual features $x=\{x_s,x_u^{\prime }\}$ and semantic features c into $z_1$ and $z_2$, respectively. Taking $z_1$ as an example, its row dimension represents the number of samples in a batch n, while the column dimension represents m. Let l be a value along the column dimension belonging to the interval $(0,m)$. The elements of $z_1$ from column dimension 0 to l are defined as $z_r$, and the remaining columns from l to m are defined as $z_d$. $z_2$ follows the same process. This procedure can be mathematically expressed as

$$\begin{array}{c}\hfill E_V\left(x\right)=z_1=[z_r,z_d]\\ \hfill E_S\left(c\right)=z_2=[c_r,c_d]\end{array}$$

(8)

In the case of the visual modality, it is important to note that $z_r$ and $z_d$ at this stage merely represent a simple partitioning of $z_1$ along its dimensions. It is necessary to subsequently use the semantic relationship matching method in Section 3.4.2 and the latent representation independent method in Section 3.4.3 to make the latent representation represented by $z_r$ semantically consistent, and at the same time change the latent representation represented by $z_d$ into semantically irrelevant.

By minimizing the KL divergence loss between the latent variable distribution and the predefined prior distribution, the VSDA can effectively learn representations in latent space. Consequently, to guarantee the validity of the latent representations $z_1$ and $z_2$ obtained by the visual–semantic disentangled autoencoder, we optimize the KL divergence loss between the latent variable distribution and the predefined prior distribution. This process can be formulated as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{KLD}=& \hfill {\mathbb{E}}_{q_{\varphi }\left(z_1\right|x)}[logp\left(x\right|z_1)]-D_{KL}(q_{\varphi }\left(z_1\right|x)\parallel p_{\theta }\left(z_1\right))\hfill \\ \hfill & \hfill +{\mathbb{E}}_{q_{\varphi }\left(z_2\right|c)}[logp\left(c\right|z_2)]-D_{KL}(q_{\varphi }\left(z_2\right|c)\parallel p_{\theta }\left(z_2\right))\hfill \end{array}$$

(9)

To reduce information loss between the visual and semantic modalities, we employ disentangled decoders to reconstruct the latent representations $(z_r,z_d)$ and $(c_r,c_d)$, and minimize the loss between the reconstructed visual–semantic features and the real features. Specifically, we use the visual disentangled decoder to reconstruct $(z_r,z_d)$ into visual features $\overline{x}$, and the semantic disentangled decoder to reconstruct $(z_r,z_d)$ into semantic features $\overline{c}$. Subsequently, we compute the loss between the reconstructed visual features $\overline{x}$ and the real visual features x, and the loss between the reconstructed semantic features $\overline{c}$ and the real semantic features c. Summing these losses yields the total reconstruction loss ${\mathcal{L}}_{REC}$. ${\mathcal{L}}_{REC}$ can be expressed as

$$\begin{array}{c}\hfill {\mathcal{L}}_{REC}=\sum _{x\in X_S}\sum _{c\in C_S}\parallel x-\overline{x}{\parallel }^2+{\parallel c-\overline{c}\parallel }^2\end{array}$$

(10)

Simultaneously, to enable the model to learn the association between visual and semantic features and reduce the deviation between modalities, cross-modal cross-reconstruction is designed. Specifically, we employ a semantic disentangling decoder to reconstruct the visual latent representation $(z_r,z_d)$ into semantic features ${\overline{c}}_x$, and a visual disentangling decoder to reconstruct the semantic latent representation $(c_r,c_d)$ into visual features ${\overline{x}}_c$. Subsequently, we minimize the reconstruction loss ${\mathcal{L}}_{CROSS\_REC}$ between ${\overline{c}}_x$, ${\overline{x}}_c$, and the original semantic and visual features c and x. This process can be mathematically formulated as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{CROSS\_REC}=\sum _{x\in X_S}\sum _{c\in C_S}|x-{\overline{x}}_c|+|c-{\overline{c}}_x|\end{array}$$

(11)

Here, the mean square error (MSE) is utilized to calculate the reconstruction loss between the original visual features and the reconstructed features. Finally, the overall loss of the VSDA is

$$\begin{array}{c}\hfill {\mathcal{L}}_{VSDA}={\mathcal{L}}_{KLD}+{\mathcal{L}}_{REC}+{\mathcal{L}}_{CROSS\_REC}\end{array}$$

(12)

3.4.2. Semantic Relation Matching

To guide the visual disentangled encoder in separating semantically consistent latent representations into $z_r$, we designed the semantic relation matching (SRM) method. This method introduces a relation network (RN) [37] to assess the matching relationship between visual and semantic features, as illustrated in Figure 4. The RN evaluates the distance between two samples by constructing a neural network, thereby measuring their matching degree. Thus, we can constrain $z_r$ represented latent representations to match with c using RN. This means that RN will encourage the visual disentangled encoder to encode semantically consistent visual features from the original features into $z_r$, while semantically inconsistent visual features will be encoded as $z_d$. In the SRM method, we first concatenate $z_r$ with its uniquely corresponding semantic feature c, and compute their compatibility score CS. Specifically, when the labels of $z_r$ and c are the same, the match is successful, and CS is set to 1. When the labels of $z_r$ and c are different, the match fails, and CS is set to 0. This process can be formulated as

$$\begin{array}{c}\hfill \begin{array}{c}CS\left(z_{r\left(t\right)},c_{\left(b\right)}\right)=\left\{\begin{array}{cc}\hfill 0,\hfill & \hfill y_{\left(t\right)}\ne y_{\left(b\right)}\hfill \\ \hfill 1,\hfill & \hfill y_{\left(t\right)}=y_{\left(b\right)}\hfill \end{array}\right.\end{array}\end{array}$$

(13)

Here, t and b represent the tth semantically consistent representation and the bth unique corresponding semantic feature in the training batch, and $y_{\left(t\right)}$ and $y_{\left(b\right)}$ represent the class labels of $z_{r\left(t\right)}$ and $c_{\left(b\right)}$, respectively. Utilizing CS defined in Equation (13), a RN with a sigmoid activation function learns a compatibility score ranging from 0 to 1 for each pair $(z_r,c)$. Then, the following loss function is used to optimize $z_r$:

$$\begin{array}{c}\hfill \begin{array}{c}{\mathcal{L}}_{SRM}={\displaystyle \sum _{t=1}^B}{\displaystyle \sum _{b=1}^n}{∥RN\left(z_{r\left(t\right)},c_{\left(b\right)}\right)-CS\left(z_{r\left(t\right)},c_{\left(b\right)}\right)∥}^2\end{array}\end{array}$$

(14)

Here, B denotes the size of the training batch, and n denotes the number of unique semantic features corresponding to the training batch. In each training batch, calculate the mean square error between the output of the relation score of each pair $z_{r\left(t\right)}$ and $c_{\left(b\right)}$ and the ground truth CS, optimized by mean square error. This loss ensures that $z_r$ is a semantically consistent latent representation.

3.4.3. Independence between Latent Representations

In Section 3.4.3, we utilized the SRM method to guide the visual disentangled encoder to preliminarily transform $z_r$ into semantically consistent visual feature latent representations, while transforming $z_d$ into semantically irrelevant latent representations. To further enhance the decoupling between visually consistent latent representations $z_r$ and visually irrelevant ones $z_d$, while also encouraging the semantic disentangled encoder to encode visually consistent and visually irrelevant semantic features into latent representations $c_r$ and $c_d$, respectively, we devised the latent representation independence (IND) method. Specifically, from a probabilistic perspective, $z_r$ and $z_d$ can be considered to come from different conditional distributions in the visual modality, while $c_r$ and $c_d$ can be considered to come from different conditional distributions in the semantic modality:

$$\begin{array}{c}\hfill \begin{array}{c}\begin{array}{cc}\hfill z_r& \hfill \sim {\psi }_1\left(z_r\right|x),z_d\sim {\psi }_2\left(z_d\right|x)\hfill \\ \hfill c_r& \hfill \sim {\psi }_3\left(c_r\right|x),c_d\sim {\psi }_4\left(c_d\right|x)\hfill \end{array}\end{array}\end{array}$$

(15)

where ${\psi }_1$ and ${\psi }_2$ are distributions for $z_r$ and $z_d$, respectively, and ${\psi }_3$ and ${\psi }_4$ are distributions for $c_r$ and $c_d$, respectively. Therefore, the independence between $z_r$ and $z_d$ denoted as $IN{D}_v$, the independence between $c_r$ and $c_d$ denoted as $IN{D}_s$, and their overall independence denoted as $IND$ can be expressed as

$$\begin{array}{c}\hfill \begin{array}{c}IN{D}_v=KL(\psi \parallel {\psi }_1·{\psi }_2)={\mathbb{E}}_{\psi }(log\frac{\psi }{{\psi }_1·{\psi }_2})\\ IN{D}_s=KL(\psi \parallel {\psi }_3·{\psi }_4)={\mathbb{E}}_{\psi }(log\frac{\psi }{{\psi }_3·{\psi }_4})\\ IND=IN{D}_v+IN{D}_s\end{array}\end{array}$$

(16)

where $\psi :=\psi (z_r,z_d|x)$ is the joint conditional probability of $z_r$ and $z_d$, and similarly for the semantic modality. Taking the visual modality as an example, when $y=1$, $z_r$ and $z_d$ are dependent, they are denoted as $\tau \left(z_1\right|y=1)$, while when $y=0$, $z_r$ and $z_d$ are independent, they are denoted as $\tau \left(z_1\right|y=0)$. Therefore, $IN{D}_v$ can be represented as

$$\begin{array}{c}\hfill \begin{array}{c}IN{D}_v={\mathbb{E}}_{\psi }(log\frac{\tau \left(z_1\right|y=1)}{\tau \left(z_1\right|y=0)})={\mathbb{E}}_{\psi }(log\frac{\tau (y=1|z_1)\tau \left(z_1\right)/\tau (y=1)}{\tau (y=0|z_1)\tau \left(z_1\right)/\tau (y=0)})={\mathbb{E}}_{\psi }\frac{\tau (y=1|z_1)}{1-\tau (y=1|z_1)})\end{array}\end{array}$$

(17)

We introduce a discriminator $DI{S}_v$ to approximate $\tau (y=1|z_1)$; thus, $IN{D}_v$ can be approximated by the following formula:

$$\begin{array}{c}\hfill \begin{array}{c}\begin{array}{c}\hfill IN{D}_v={\mathbb{E}}_{\varphi }(log\frac{DI{S}_v\left(z_1\right)}{1-DI{S}_v\left(z_1\right)})\end{array}\end{array}\end{array}$$

(18)

During the training of the discriminator, we randomly shuffle $z_r$ and $z_d$ in each training batch, then concatenate them to obtain $\widehat{z_1}$. Finally, the loss of the discriminator on the visual modality and the semantic modality is given by

$$\begin{array}{c}\hfill \begin{array}{c}{\mathcal{L}}_{DIS}=DI{S}_v+DI{S}_s=logDI{S}_v\left(z_1\right)+log(1-DI{S}_v\left(\widehat{z_1}\right))\\ +logDI{S}_s\left(z_2\right)+log(1-DI{S}_s\left(\widehat{z_2}\right))\end{array}\end{array}$$

(19)

In summary, the total loss of our SDDN framework is formulated as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{total}& \hfill ={\mathcal{L}}_{SDE}+{\mathcal{L}}_{VSDA}+{\lambda }_1\ast {\mathcal{L}}_{SRM}+{\lambda }_2\ast IND+{\lambda }_3\ast {\mathcal{L}}_{DIS}\hfill \end{array}$$

(20)

3.5. Classification

When training the classifier, the seen visual features $x_s$, refined by the auxiliary self-teacher network, and the synthesized unseen visual features $x_u^{\prime }$ from the SDCG module are combined into $x=\{x_s,x_u^{\prime }\}$. Next, x is processed by the trained visual disentangling encoder to yield both semantically consistent latent representation $z_r$ and semantically irrelevant latent representation $z_d$. Similarly, the semantic attribute set c for both seen and unseen classes is fed into the trained semantic disentangling encoder, producing visually consistent latent representation $c_r$ and visually irrelevant latent representation $z_d$. Subsequently, these representations $z_r$ and $z_d$ are fused and input into a softmax-based GZSL classifier for category prediction. Finally, the training concludes by minimizing the loss between the real and predicted labels. The whole process is shown in Figure 5.

4. Experiments

4.1. Datasets

We conducted comprehensive tests on four publicly available benchmark datasets: Caltech-UCSD Birds-200-2011 (CUB) [38], Animals with Attributes2 (AWA2) [39], SUN Attribute Dataset (SUN) [40], and Oxford Flowers (FLO) [41]. All datasets and their statistics are summarized in

Table 1. Statistics of the AWA2, CUB, FLO and SUN datasets, including visual feature dimension $D^x$, semantic feature dimension $D^s$, number of seen classes $N^s$, number of unseen classes $N^u$ and number of all instances $N^i$.

Dataset	${\mathit{D}}^{\mathit{x}}$	${\mathit{D}}^{\mathit{s}}$	${\mathit{N}}^{\mathit{s}}$	${\mathit{N}}^{\mathit{u}}$	${\mathit{N}}^{\mathit{i}}$
AWA2	2048	85	40	10	37,322
CUB	2048	312	150	50	11,788
FLO	2048	1024	82	20	8189
SUN	2048	102	645	72	14,340

. CUB is a fine-grained dataset comprising 11,788 images from 200 different bird species, with 150 seen classes and 50 unseen classes. Each image in CUB is annotated with 312 dimensions of attributes. FLO is another fine-grained dataset consisting of flower images, containing 8189 images across 102 classes, including 82 seen classes and 20 unseen classes. The annotation attributes in FLO have 1024 dimensions. SUN is a fine-grained image dataset featuring various scenes, with 14,340 images covering 717 classes (645 seen classes and 72 unseen classes). Each scene in SUN is associated with 102-dimensional attributes describing its characteristics, such as lighting conditions, weather conditions, and terrain. AWA2 is a coarse-grained dataset with 37,322 animal images across 50 classes (10 seen classes and 40 unseen classes), covering a wide range of animals, including mammals, birds, and reptiles. Each image in AWA2 is labeled with attributes of 85 dimensions.

4.2. Evaluation Protocol

During testing, the accuracy is assessed on the test sets for both seen classes (S) and unseen classes (U). Here, U represents the average accuracy for each class on test images of unseen classes, indicating the model’s ability to classify samples from previously unseen classes. S represents the average accuracy for each class on test images of seen classes, reflecting the model’s ability to classify samples from seen classes. H (defined as $(H=(2\times S\times U)/(S+U)$) represents the harmonic mean of S and U, serving as an evaluation metric for the performance of GZSL classification.

4.3. Implementation Details

SDDN mainly consists of a SDE module and a disentanglement network. The SDE module mainly consists of a student network, a self-teacher network, a generator and a discriminator. The student network is a ResNet101 model pre-trained on ImageNet and is used to extract visual features with a dimension of 2048. The self-teacher network is composed of the student network itself and the feature fusion method. In addition, the generator is implemented using a multi-layer perceptron with a hidden layer dimension of 2048, and the discriminator is implemented using a fully connected layer and activation function. The disentanglement network consists of an encoder, a decoder, a discriminator and a semantic relationship matching model. Both the encoder and decoder are multi-layer perceptrons with a single hidden layer and 2048 hidden units. The semantic relationship matching model consists of two fully connected layers activated with Smooth Maximum Unit (SMU) [42] activation function and Sigmiod function, respectively. The discriminator is implemented using a fully connected layer and SMU activation function.

The hardware environment used by SDDN is an Intel i7-10700K CPU, RTX A5000 32GB GPU; the software environment is Ubuntu 20.04 LTS operating system, cuda 11.4.0, and cudnn 8.2.4. SDDN is implemented in PyTorch 1.10.1. The Adam optimizer is used to optimize the parameters of each module. The learning rate of the Adam optimizer is set to $lr=0.0001$, ${\beta }_1=0.9$ and ${\beta }_2=0.999$. The batch size is 64. The loss weight ${\lambda }_1$ of the semantic relation matching method, the loss weight ${\lambda }_2$ of the latent representation independence method, and the weight ${\lambda }_3$ of the visual–semantic discriminator are set between 0.1–25.

Comparing with the State of the Art

To validate the effectiveness of our proposed SDDN model, we computed the seen class accuracy rate S, unseen class accuracy rate U, and their harmonic mean H on the aforementioned four datasets. We compared them with 15 state-of-the-art models, and the comparison results are shown in

Table 2. Performance comparison in accuracy (%) on four datasets. Displaying the accuracies of seen and unseen classes in GZSL, denoted as U, S, and H for the harmonic mean. The methods above and below the horizontal line correspond to non-generative and generative approaches, respectively. Results in bold font indicate the highest performance.

Method	FLO			CUB			AWA2			SUN
	$\mathit{U}$	$\mathit{S}$	$\mathit{H}$	$\mathit{U}$	$\mathit{S}$	$\mathit{H}$	$\mathit{U}$	$\mathit{S}$	$\mathit{H}$	$\mathit{U}$	$\mathit{S}$	$\mathit{H}$
TCN [43]	-	-	-	52.6	52.0	52.3	61.2	65.8	63.4	31.2	37.3	34.0
DVBE [9]	-	-	-	53.2	60.2	56.5	63.6	70.8	67.0	45.0	37.2	40.7
RGEN [10]	-	-	-	60.0	73.5	66.1	67.1	76.5	71.5	44.0	31.7	36.8
TDCSS [44]	54.1	85.1	66.2	44.2	62.8	51.9	59.2	74.9	66.1	-	-	-
f-VAEGAN-D2 [45]	56.8	74.9	64.6	48.4	60.1	53.6	57.6	70.6	63.5	45.1	38.0	41.3
LisGAN [46]	57.7	83.8	68.3	46.5	57.9	51.6	52.6	76.3	62.3	42.9	37.8	40.2
CANZSL [5]	58.2	77.6	66.5	47.9	58.1	52.5	49.7	70.2	58.2	46.8	35.0	40.0
SE-GZSL [6]	-	-	-	41.5	53.3	46.7	58.3	68.1	62.8	30.5	40.9	34.9
TIZSL [7]	70.4	68.7	69.5	52.1	53.3	52.7	76.8	66.9	71.5	32.3	24.6	27.9
FREE [3]	67.4	84.5	75.0	55.7	59.9	57.7	60.4	75.4	67.1	47.4	37.2	41.7
SDGZSL [2]	83.3	90.2	86.6	59.9	66.4	63.0	64.6	73.6	68.8	48.2	36.1	41.3
JG-ZSL [16]	-	-	-	60.8	63.9	62.3	63.1	68.3	65.6	50.2	37.9	43.2
ICCE [47]	66.1	86.5	74.9	67.3	65.5	66.4	65.3	82.3	72.8	-	-	-
DGCNet-db [18]	-	-	-	51.5	57.5	54.4	50.4	72.8	59.6	26.8	39.6	32.0
DVAGAN [17]	-	-	-	52.5	57.3	54.8	65.9	82.0	73.1	44.7	37.9	41.0
Our SDDN	87.3	90.5	88.9	66.8	68.3	67.5	65.6	74.3	69.7	48.6	42.3	45.2

. These 15 models are categorized into methods based on generative models and methods not based on generative models. Generative-based methods typically utilize techniques such as GANs or VAEs to generate synthetic unseen class data to augment the training dataset. These synthetic data can be used to train models in ZSL to improve their generalization capability to unseen classes. Non-generative-based methods, on the other hand, do not rely on generating synthetic data but achieve generalization to unseen classes through techniques such as feature embedding and alignment of existing data. Our method belongs to the generative-based methods.

From the comparison results in the table, firstly, our SDDN achieved the highest accuracy on U, S, and H on the FLO dataset, surpassing all compared models. Specifically, we outperformed the second-best model by 2.3% in the H metric. There was a significant improvement in the U metric, where we led the second-best by 4%. In the S metric, we were ahead of the second-best by 0.3%. On the CUB dataset, we achieved the highest accuracy on the U metric, leading the second-best by 1.1%. Additionally, we obtained the second-best accuracy on both U and S metrics, leading the third-best by 6% and 1.9%, respectively. On the SUN dataset, we attained the highest accuracy on both S and H metrics, leading the second-best by 2% in the H metric and 1.4% in the S metric.

Overall, our performance was the best in the H metric on these three fine-grained datasets, FLO, CUB, and SUN; the best on U in FLO; and the best on S in both FLO and SUN. This indicates that the richer the information in the dataset, the more effectively our proposed method can capture it through self-distillation and disentanglement techniques, separating visual–semantic consistent features and aligning them effectively.

5. Model Analysis

5.1. Ablation Study

In our ablation study, we aim to isolate the key components of SDDN and assess their impact on GZSL. We remove the semantic relation matching loss (LSRM) to evaluate the contribution of the Visual–Semantic Matching module to extracting semantically consistent visual features. Omitting the Independence score (IND) allows us to evaluate its contribution to further separating visual–semantic consistent features. Additionally, we exclude the loss of the self-distillation embedding module (LSDE) and then used the pre-trained ResNet101 and regular generator without employing the self-distillation technique. This evaluation helps assess the refinement effect of SDE on extracting and synthesizing visual features, while validating the effectiveness of the disentanglement network. Our ablation experiments were conducted on the FLO and CUB datasets, with the experimental results presented in

Table 3. Ablation study of different component combinations on FLO and CUB datasets. Results are reported in %, with the best results highlighted in bold.

Method	FLO			CUB
	U	S	H	U	S	H
SDDN w/o LSRM and IND and LSDE	39.6	50.2	44.2	35.3	40.1	37.5
SDDN w/o LSRM and IND	48.7	54.5	51.4	38.6	43.2	40.8
SDDN w/o LSRM	60.3	71.6	65.5	48.1	57.8	52.5
SDDN w/o IND	79.6	81.2	80.4	56.7	60.9	58.7
SDDN w/o LSDE	86.9	88.7	87.8	65.5	67.9	66.7
SDDN	87.3	90.5	88.9	66.8	68.2	67.5

and Figure 6.

The results underscore the critical importance of the semantic relation matching module (LSRM), independence score (IND), and self-distillation embedding module (LSDE) for the performance of SDDN. Firstly, LSRM is particularly crucial for visual–semantic feature alignment, as its removal leads to a significant decrease in the accuracy of seen classes (S), unseen classes (U), and the harmonic mean (H). Secondly, IND is essential for further separating visual–semantic consistent features from the original features, as its removal results in lower U, S, and H values. Additionally, LSDE helps refine the original visual features of seen classes and the synthesized features of unseen classes, as the model without LSDE performs lower in U, S, and H compared to the complete SDDN. Furthermore, when comparing Table 1, it is found that the model without SDE still achieves the highest H score on FLO and CUB, indicating the effectiveness of the disentanglement network. Finally, the complete SDDN model demonstrates superior performance across all metrics, proving its effectiveness in GZSL.

5.2. Hyper-Parameter Analysis

In this study, the optimization objective of SDDN is determined by three critical hyperparameters: the coefficient of semantic relationship matching loss (${\lambda }_1$), the coefficient of independence of latent representations (${\lambda }_2$), and the coefficient of discriminator loss (${\lambda }_3$). To elucidate the influence of each hyperparameter on model performance, sensitivity analysis was conducted by varying the hyperparameter values in the experiments. Specifically, ${\lambda }_1$ was varied within the range of 0.3–20.0, while ${\lambda }_2$ and ${\lambda }_3$ were varied within the range of 0.1–3.0. Figure 7 illustrates the significant impact of hyperparameter values ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ on the experimental outcomes. Notably, when ${\lambda }_1$ is set to 18, ${\lambda }_2$ is set to 0.5, and ${\lambda }_3$ is set to 2, the model achieves its highest accuracy on the FLO dataset, whereas when ${\lambda }_1$ is set to 1, ${\lambda }_2$ is set to 0.6, and ${\lambda }_3$ is set to 0.3, the model achieves its highest accuracy on the CUB dataset. These observations underscore the substantial influence of hyperparameter weights on model accuracy, indicating the model’s high sensitivity to these hyperparameters. Based on these findings, we advocate for future experiments to focus on exploring the specific impact of minor fluctuations in these three hyperparameter values on accuracy. This systematic analysis of hyperparameters will contribute to a deeper comprehension of model behavior and offer valuable insights for optimizing model performance across diverse datasets.

5.3. Zero-Shot Retrieval Performance

To assess the practical application performance of our SDDN framework, we conducted zero-shot retrieval experiments comparing SDDN with two other state-of-the-art generative-based GZSL frameworks: DGGNet-db and DVAGAN. The experiment follows the zero-shot retrieval protocol in SDGZSL [2]. In zero-shot retrieval experiments, we initially provide semantic features of unseen classes, followed by employing the generation modules of SDDN, DGGNet-db, and DVAGAN to synthesize a certain number of visual features for these unseen classes. Throughout this process, the average of the synthesized visual features for each category is computed as the retrieval feature. Subsequently, the cosine similarity between the retrieval features and the true features is calculated, and the true features are ranked in descending order based on this similarity. The performance of zero-shot image retrieval is evaluated using mean Average Precision (mAP). Experimental analyses are performed on three datasets: CUB, AWA2, and SUN. The results, illustrated in Figure 8, compare SDDN, DGGNet-db, and DVAGAN in terms of zero-shot retrieval performance. The horizontal coordinates 100, 50, and 25 represent the proportions of unseen category images in the test dataset, being 100%, 50%, and 25%, respectively, while the vertical coordinate represents the average retrieval accuracy. Results indicate significantly higher zero-shot retrieval performance of the SDDN framework on the CUB and SUN datasets compared to DGGNet-db and DVAGAN. In the AWA2 dataset, we achieve the best performance when the proportion of unseen category images reaches 50%, and it remains close to the best performance when the proportion reaches 100%. These zero-shot retrieval performance tests on the three datasets further validate the effectiveness of the model.

6. Conclusions

In this paper, we propose a generalized zero-shot learning framework that utilizes a self-distillation and disentanglement network to enhance visual–semantic feature consistency. Initially, for improving the semantic consistency of visual features, we develop a self-distillation embedding framework integrating self-distillation techniques with a conditional generator to prompt the synthesis of refined visual features. Subsequently, to further promote visual–semantic feature consistency, we design a disentanglement network. We use semantic relation matching networks and latent representation independence methods to facilitate the separation of visually semantically consistent features from inconsistent features. Additionally, we devise a cross-reconstruction method to align visual and semantic features within a visual–semantic common space, thereby enhancing the semantic consistency of visual–semantic features. Extensive experiments are conducted on four widely used benchmark datasets in GZSL. We compare SDNN with current state-of-the-art methods, thereby demonstrating the superiority of the proposed SDNN framework. In future work, we intend to optimize the model further and apply it in the field of medical diagnostics to assist in identifying new disease patterns.