Discrepant Semantic Diffusion Boosts Transfer Learning Robustness

Abstract

Transfer learning could improve the robustness and generalization of the model, reducing potential privacy and security risks. It operates by fine-tuning a pre-trained model on downstream datasets. This process not only enhances the model’s capacity to acquire generalizable features but also ensures an effective alignment between upstream and downstream knowledge domains. Transfer learning can effectively speed up the model convergence when adapting to novel tasks, thereby leading to the efficient conservation of both data and computational resources. However, existing methods often neglect the discrepant downstream–upstream connections. Instead, they rigidly preserve the upstream information without an adequate regularization of the downstream semantic discrepancy. Consequently, this results in weak generalization, issues with collapsed classification, and an overall inferior performance. The main reason lies in the collapsed downstream–upstream connection due to the mismatched semantic granularity. Therefore, we propose a discrepant semantic diffusion method for transfer learning, which could adjust the mismatched semantic granularity and alleviate the collapsed classification problem to improve the transfer learning performance. Specifically, the proposed framework consists of a Prior-Guided Diffusion for pre-training and a discrepant diffusion for fine-tuning. Firstly, the Prior-Guided Diffusion aims to empower the pre-trained model with the semantic-diffusion ability. This is achieved through a semantic prior, which consequently provides a more robust pre-trained model for downstream classification. Secondly, the discrepant diffusion focuses on encouraging semantic diffusion. Its design intends to avoid the unwanted semantic centralization, which often causes the collapsed classification. Furthermore, it is constrained by the semantic discrepancy, serving to elevate the downstream discrimination capabilities. Extensive experiments on eight prevalent downstream classification datasets confirm that our method can outperform a number of state-of-the-art approaches, especially for fine-grained datasets or datasets dissimilar to upstream data (e.g., 3.75% improvement for Cars dataset and 1.79% improvement for SUN dataset under the few-shot setting with 15% data). Furthermore, the experiments of data sparsity caused by privacy protection successfully validate our proposed method’s effectiveness in the field of artificial intelligence security.

1. Introduction

Transfer learning could mitigate the potential privacy and security risks through effectively re-utilizing the pre-trained model, which is initially trained on a voluminous and information-dense dataset, commonly referred to as the upstream dataset. The pre-trained model is adaptively fine-tuned on a novel and less-data-intensive dataset, denominated as the downstream dataset. For example, in special critical scenarios where data are seldom publicly available, transfer learning can reduce the training data requirement for models on new tasks, thereby lowering the potential security risks [1,2,3,4]. It is capable of accelerating the learning of downstream models, while reducing the labor costs and computational consumption. Moreover, pre-trained models, which are typically public and thus more vulnerable to attacks, have been shown to benefit from the security enhancements provided by transfer learning. For example, studies [5,6,7] examining the susceptibility of downstream models to attacks have confirmed that transfer learning can protect these downstream models from being easily attacked, thereby enhancing their robustness. Due to its practicality, transfer learning has attracted extensive attention in the field of computer vision [8,9,10,11,12], and has been applied in many task scenarios such as transportation, medical treatment [13], social media [14] and art [15,16,17,18]. Various works [19] have been proposed to explore the problem of what and how to transfer from the pre-trained models.

Transfer learning allows the use of a pre-trained model’s comprehensive upstream knowledge to better accomplish the downstream task, even with limited labeled data. By aligning the knowledge domain, model convergence in downstream tasks is accelerated. Some attempts have been made to explore and apply the shared downstream–upstream knowledge alignment through the well-designed regularization in the fine-tuning stage. Specifically, Refs. [20,21] tried to add the regularization of parameters or features in the fine-tuning stage to preserve the diversity of pre-training information, and thus to selectively transfer the upstream knowledge of the pre-trained network. Ref. [22] adaptively selected layers to freeze or fine-tune according to each training sample. To preserve as much upstream information as possible, Ref. [23] placed constraints on the weights of the fine-tuned model to approximate those of the pre-trained model. Similarly, Ref. [24] supervised the fine-tuning process with both upstream probabilistic labels and downstream labels. Meanwhile, there are also a few works [25,26,27] aiming to boost the generalization ability of pre-trained feature representations.

Although previous transferring methods have achieved satisfactory performance, they neglected the discrepant downstream–upstream connections and rigidly preserved the upstream information without an adequate regularization of downstream semantic discrepancy, leading to weak generalization, the collapsed classification problem, and inferior transferring performance. Discrepant downstream–upstream connections pertain to the mapping of semantics from downstream classes to upstream classes, aiming to preserve the semantic diversity and discrimination in downstream data. As shown in Figure 1, the connection marked with a check symbol after ’discrepant’ is key to maintaining feature distinctiveness in the downstream embedding space, a process we also describe as ensuring downstream semantic discrepancy. As shown in Figure 1a, previous methods assumed that the downstream classes and upstream classes share similar semantic granularity (a one–one downstream–upstream mapping), where the downstream–upstream connection of each downstream category is discrepant from the others, and thus concluded that simply preserving the upstream information will benefit the downstream discrimination. However, for the fine-grained dataset or the dataset of low similarity with the upstream dataset as shown in Figure 1b, if a downstream category is only allowed to connect a single upstream class, it is very likely that an upstream class simultaneously contains the representative characteristics of different downstream categories, due to the mismatched downstream–upstream granularity (a many–one downstream–upstream mapping). In this condition, the fine-tuned model, which is rigidly restricted to preserve the upstream information, forces the representations of those downstream classes close to each other and thus counters to the downstream discriminative classification task, resulting in performance degradation.

To address the above problems as shown in the right column in Figure 1c, we should encourage that a downstream class diffusively relates to k ($k>1$) upstream classes rather than a single one during transferring (for example, $k=2$ in the figure), introducing more semantic diversity and discrimination to adjust the mismatched downstream–upstream connection, meanwhile ensuring the discrepancy of these downstream–upstream connections for different downstream classes to achieve robust classification. Therefore, we propose a discrepant semantic diffusion method for transfer learning, which could ensure the downstream semantic discrepancy to effectively accomplish downstream tasks. This method encourages semantic diffusion, a process where knowledge from the upstream task is diffused in a controlled and differentiated manner to the downstream task to avoid collapsed classification.

Specifically, the proposed discrepant semantic diffusion contains Prior-Guided Diffusion (PGD) in the pre-training stage and Discrepant Diffusion (DI2) in the fine-tuning stage. PGD relaxes the centralized one-hot classification regularization during the pre-training stages through a semantic prior, which is a predefined knowledge structure that enables the model to understand potential semantic associations in the data. PGD could encourage the following diffusive one-k downstream–upstream mapping during the transferring and improve the generalization ability of the pre-trained model. The semantic prior indicates the semantic similarity between two classes, according to which we could reduce the penalty for classes similar to the ground-truth class during the gradient back-propagation, and thus sets the foundation for top-k mapping during the fine-tuning stage. Moreover, DI2 adjusts the semantic granularity by encouraging diffusive downstream–upstream connection during the fine-tuning stage, and regularizes the diffusive connection via the downstream semantic discrepancy, effectively and adaptively transferring the upstream information to accomplish the downstream task. This approach introduces more semantic diversity with the one–k downstream–upstream mapping, maintaining a different granularity diffusion sequence for each downstream class, rather than collapsing into the same category. By discreetly determining the value of k, downstream discrimination is ensured from the downstream–upstream connection perspective. It is noteworthy that for downstream datasets with varying semantic granularities, the value of k differs, endowing the model with a continuous adaptability capability. Consequently, DI2 could guarantee effective transferring specific to the downstream task.

To summarize, we build the diffusive downstream–upstream connection with the semantic discrepancy in transfer learning. With the discrepant semantic diffusion, the proposed model could effectively avoid the collapsed classification problem to boost transfer learning robustness and improve performance on downstream tasks. The comprehensive experiments on eight downstream classification datasets demonstrate the superiority and generalization of our method compared to the state-of-the-art transfer learning models, especially for the fine-grained datasets or datasets, unlike upstream data (e.g., 3.75% improvement for the Cars dataset and 1.79% improvement for the SUN dataset). Furthermore, a key aspect of our method is its impact on privacy and security risks, especially during the fine-tuning process on downstream datasets. As the model adapts to new tasks, there is a potential risk of exposing sensitive information embedded in the training data [28,29]. This risk is particularly significant in scenarios involving downstream datasets that contain private or confidential information. Therefore, it is necessary to study the performance of our method in scenarios of data sparsity caused by privacy protection. We explore the data sparsity issue caused by privacy protection in the field of artificial intelligence (AI) security, successfully validating our proposed method’s effectiveness through experiments. We make the following contributions:

• We propose a transfer learning method with discrepant semantic diffusion to better adapt to various downstream datasets, diffusing more upstream in-depth semantic discrepancy information into the downstream dataset categories.
• To empower the pre-trained model with the semantic diffusion ability, Prior-Guided Diffusion (PGD) is introduced into the pre-training stage, which relaxes the centralized one-hot classification regularization through the semantic prior.
• To further promote the downstream discrimination, Discrepant Diffusion (DI2) is designed to maintain a different granularity diffusion sequence for each downstream category, which guides the model to focus on discriminative information among the downstream categories.
• Experimental results on eight prevalent downstream classification datasets and various networks verify the effectiveness of the proposed method. The additional experiments demonstrate that our method significantly reduces the training data requirements for models on new tasks and lowers potential security risks.

2. Related Work

ImageNet pre-trained deep neural networks have shown remarkable transferability to various tasks, such as image classification [30,31,32], image segmentation [33,34,35,36], object detection [37,38,39,40], image retrieval [41], action recognition [42,43], etc. Even in cases where upstream and downstream tasks have significant differences, such as transfer for depth estimation [44], medical imaging [45,46,47] and other downstream tasks, ImageNet pre-trained deep neural networks are also effective and could speed up the generalization proved by [25].

Following the typology of [48], from the perspective of transferring and aligning robust knowledge, the current transfer learning methods are mainly twofold, knowledge of feature representations transfer and knowledge of parameters (i.e., inductive bias) transfer. For feature representation transfer learning, researchers aim to improve the generalization and transferability of pre-trained feature representations. Through a thorough study of transfer learning, Refs. [25,26] pointed out it is mainly low-level and mid-level general representations that are transferred instead of specialized high-level representations. Follow the above findings, Ref. [27] increases the transferability of high-level feature using adversarial robust pre-trained networks, while [49] obtains representations with higher generality by pre-training on large-scale upstream datasets. Some methods use contrast learning or self-supervised learning during model pre-training to achieve better generalization ability. Using nearest-neighbor contrastive learning, simple-architecture contrastive learning and cross-view alignment contrastive learning, respectively, Refs. [50,51,52] enhances the visual representations to improve the transfer learning performance. Ref. [53] enables the control of upstream performance and transferability through a self-supervised learning setup.

Inductive transfer tends to regularize the weights of the fine-tuning model with the pre-trained weights, which allows the model to leverage previously learned knowledge, aligning knowledge domains to enhance robustness. Since the parameters preserve the inductive bias gained from the upstream tasks, fine-tuned parameter values ending up far from the pre-trained ones are regarded as a sign of forgetting [54]. Therefore, inductive transfer methods mostly focus on retaining the information mined from upstream datasets throughout the fine-tune process. For example, to transfer more upstream information, Ref. [23] regularizes the weights of the fine-tuned model with L2 constraints to make it approximate to the weights of the pre-trained model. To selectively preserve the upstream information, Ref. [20] penalizes deviations of network activation. For similar purposes, Ref. [21] penalizes small eigenvalues in representations that cause negative transfer.

Though ImageNet is still the de facto pre-train dataset [55], not all knowledge gained from its domain and task is as informative and universal for downstream tasks. Forcing the model to simply retain as much knowledge as possible may sometimes degrade the performance [56]. On the representation side, the fixed-feature setting where a linear classifier is trained on top of the pre-trained extractor is usually outperformed by fine-tuning the full model. Even when the robustness of pre-trained representations is improved, Ref. [27] suggests that ImageNet pre-trained networks still work better as the weight initialization instead of feature extractors. Ref. [55] found out that higher accuracy on ImageNet does not guarantee better transfer performance, which indicates there might be some disagreement between the discrimination among upstream and downstream categories. And on the inductive side, Ref. [56] revealed that weight preservation would sometimes hinder the transfer, especially when downstream domains diverge from ImageNet. Besides the domain differences, analysis from [26] also shows that the task misalignment between ImageNet and downstream datasets weakens the transferability. Therefore, to improve transfer learning robustness and better adapt to various downstream datasets, it is crucial to distinguish and align the useful part of the knowledge and discard the disadvantaging ones.

3. The Approach

Previous transfer learning approaches neglect the valuable semantic discrepancy of the downstream–upstream connection during the transferring process, leading to weak generalization and collapsed classification, especially on downstream datasets, unlike the upstream dataset, as we mentioned in Section 1. To handle this problem, in this paper, we propose a robust transfer learning method with discrepant semantic diffusion, which could introduce the semantic discrepancy during the transferring process through the semantic diffusion, building the discrepant one–many downstream–upstream connection for different downstream classes. Therefore, we first propose Prior-Guided Diffusion (PGD) for pre-training to encourage the semantic diffusion, and then introduce Discrepant Diffusion (DI2) for fine-tuning to regularize the transferring with a diffusive downstream–upstream connection constrained by the semantic discrepancy. In this way, we could generalize and transfer the effective upstream information, especially for downstream tasks, improving model generalization and performance. Figure 2 illustrates the architecture of the proposed method.

In the following sections, we will first introduce the preliminary and discuss the possible reason for the inferior performance of previous methods. Then, we will elaborate the details of the proposed PGD for pre-training and DI2 for fine-tuning. Finally, we will present the whole learning pipeline.

3.1. Preliminary

In this section, we will give a formal description of the transfer learning pipeline. The standard transfer learning process usually contains a pre-training stage, where a network is trained on a large-scale upstream dataset and a fine-tuning stage, where the pre-trained network will be fine-tuned adapted to the downstream tasks. We first denote $\mathcal{D}=\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ as the downstream dataset, where ${\mathbf{x}}_i$ is the i-th data sample, while ${\mathbf{y}}_i\in \mathcal{C}$ is its corresponding class label. $\mathcal{C}$ is the class set containing M classes $\mathcal{C}=\{c_1,c_2,...,c_M\}$. Similarly, $\overline{\mathcal{D}}=\{{\overline{\mathbf{x}}}_i,{\overline{\mathbf{y}}}_i\}$ is the upstream dataset, where ${\overline{\mathbf{x}}}_i$ is the i-th data sample, while ${\overline{\mathbf{y}}}_i\in \overline{\mathcal{C}}$ is its corresponding class label. $\overline{\mathcal{C}}$ is the class set containing N classes $\overline{\mathcal{C}}=\{{\overline{c}}_1,{\overline{c}}_2,\dots ,{\overline{c}}_N\}$. Normally, $N\gg M$, meaning that the upstream dataset is more informative and the model trained on such a dataset is empowered with strong generalizable representation learning capability. Therefore, we would like to take advantage of a model with strong robustness and generalization and apply it to the downstream task, avoiding the time-consuming training on the downstream dataset from scratch.

To transfer the strong representation ability of the model $\overline{\mathbb{G}}=\{\overline{\mathbb{E}},\overline{\mathbb{FC}}\}$ pre-trained on $\overline{\mathcal{D}}$, where $\overline{\mathbb{E}}$ is the feature extractor and $\overline{\mathbb{FC}}$ is the task-specific classifier, the most intuitive practice is replacing the original classifier with the new classifier adapted to the downstream task and fine-tuning the whole network $\mathbb{G}=\{\mathbb{E},\mathbb{FC}\}$:

$$min\frac{1}{\left|\mathcal{D}\right|}\sum _{i=0}^{\left|\mathcal{D}\right|}{\ell }_c(\sigma \left(\mathbb{G}\left({\mathbf{x}}_i\right)\right),{\mathbf{y}}_i)+\lambda {\ell }_r(·),$$

(1)

where ${\ell }_c$ is the multi-classification loss and $\sigma$ is the sigmoid activation function. $\lambda$ is the weighting hyper-parameter. ${\ell }_r(·)$ is usually the regularization term forcing the fine-tuned network to preserve the appropriate upstream information. For instance, Ref. [24] models the downstream–upstream matching $p(\overline{C}|c_m)$ through averaging the predictions of the pre-trained model over all samples of each downstream category $c_m$, and thus encourages the output of fine-tuned model $\mathbb{E}$ for each sample of $c_m$ closed to the $p(\overline{C}|c_m)$:

$${\ell }_r=\sum _m\sum _{\{({\mathbf{x}}_i,{\mathbf{y}}_i)\in \mathcal{D}|{\mathbf{y}}_i=m\}}{\ell }_c(\overline{\mathbb{FC}}\left(\mathbb{E}\left({\mathbf{x}}_i\right)\right),p\left(\overline{C}\right|c_m)).$$

(2)

Although these methods transfer upstream information rapidly and achieve satisfactory performance on the downstream task, without sufficiently considering the discrepant downstream–upstream connection during transferring on different downstream dataset, it leads to insufficient robustness and poor generalization of the model.

3.2. Prior-Guided Diffusion for Pre-Training

The pre-training model of traditional transfer learning methods is usually supervised by the centralized one-hot labels, posing a many–one downstream–upstream connection risk for collapsed classification when generalizing and transferring to other datasets of dissimilar distribution. The semantic diffusion could encourage a sample to relate to different classes rather than its ground-truth class, and thus will introduce more semantic diversity when transferring. Therefore, we present a Prior-Guided Diffusion approach for pre-training, introducing a semantic prior which indicates the precious class relations and potential semantic associations, thereby guiding diffusive one–k downstream–upstream mapping. According to the precious semantic prior, we could select the neighboring classes of the ground-truth class and reduce the penalty for the misclassification on them. Moreover, we also maintain the discrimination ability of the pre-trained model, assigning the largest prediction score for the ground-truth class. The PGD could enhance the generalization and diffusion ability of the pre-trained model, exploring common semantic patterns rather simply concentrating on discriminative ones, and thus could benefit the transferring and promote the downstream semantic discrepancy in the following fine-tuning stage.

The semantic prior, such as additional teacher work (such as the prevalent transformer [57,58]), or an artificially defined WordNet synonym structure tree, is introduced to better model the category connection. It reflects the similarity degree between different classes from the semantic perspective. For an upstream sample ${\overline{\mathbf{x}}}_i$, we denote the function $\mathcal{N}\left({\overline{\mathbf{y}}}_i\right)$ to return the similar upstream category sequences for the ground-truth upstream category ${\overline{\mathbf{y}}}_i$ according to the prior. Specifically, we introduce a hyper-parameter $\tau$ to the n-th element of the logits ${\mathbf{l}}_i=\mathbb{G}\left({\mathbf{x}}_i\right)$, diminish the penalty for this class, and thus encourage a similar connection:

$$\begin{array}{cc}\hfill {\ell }_u& =-\sum _{i=0}^{|\overline{\mathcal{D}}|}\sum _{n=0}^{N-1}({\overline{\mathbf{y}}}_i\left[n\right]log\left(\sigma ({\mathbf{l}}_i\left[n\right]-\mathcal{I}({\overline{c}}_n\in \mathcal{N}\left({\overline{\mathbf{y}}}_i\right))log\tau )\right)\hfill \\ \hfill & +(1-{\overline{\mathbf{y}}}_i\left[n\right])log(1-\sigma ({\mathbf{l}}_i\left[n\right]-\mathcal{I}({\overline{c}}_n\in \mathcal{N}\left({\overline{\mathbf{y}}}_i\right))log\tau )))\hfill \end{array}$$

(3)

where $\mathcal{I}(·)$ is the Kronecker delta function that is equal to 1 when the input condition holds, and 0 otherwise. We adopt Binary Cross Entropy (BCE) loss to guide the learning.

Theoretically speaking, the absolute value of the loss gradient could be divided into the following conditions:

$$\left\{\begin{array}{cc}\hfill \frac{\tau }{\tau +e^{{\mathbf{l}}_i\left[n\right]}},& {\overline{\mathbf{y}}}_i\left[n\right]=1\hfill \\ \hfill \frac{e^{{\mathbf{l}}_i\left[n\right]}}{\tau +e^{{\mathbf{l}}_i\left[n\right]}},& {\overline{\mathbf{y}}}_i\left[n\right]=0,c_n\in \mathcal{N}\left({\overline{\mathbf{y}}}_i\right)\hfill \\ \hfill \frac{e^{{\mathbf{l}}_i\left[n\right]}}{1+e^{{\mathbf{l}}_i\left[n\right]}},& {\overline{\mathbf{y}}}_i\left[n\right]=0,c_n\notin \mathcal{N}\left({\overline{\mathbf{y}}}_i\right)\hfill \end{array}\right.$$

(4)

Since $\tau \gg 1$, this gradient increases the penalty of the ground-truth class, and reduces the network’s suppression of the wrong but similar classes while maintaining the penalty of the others. Hence, it could explore the valuable category correlation through the semantic prior, encouraging the semantic diffusion and the robustness of the pre-trained model.

PGD could assist the semantic diffusion in DI2, providing an advanced initialization for DI2 optimization. Moreover, building valuable class relations could help the pre-trained model explore the general semantic attributes rather than the discriminative ones for classification, improving the generalization ability cost effectively. Consequently, simply adopting our PGD without the semantic discrepancy constraint in DI2 will also achieve a good performance.

3.3. Discrepant Diffusion for Fine-Tuning

We further introduce Discrepant Diffusion in the fine-tuning stage, and build the diffusive downstream–upstream connection with semantic discrepancy, which could diffusively and discriminatively transfer the appropriate upstream information and enhance the downstream discrimination. We will first illustrate the semantic diffusion between upstream and downstream classes, and then demonstrate how the downstream semantic discrepancy regularizes the diffusion.

Semantic diffusion: The semantic diffusion could effectively adjust the mismatched downstream–upstream granularity through a diffusive downstream–upstream connection, alleviating the collapsed classification problem. To explicitly model the downstream–upstream connection, we refer to the work [24] to calculate the average logits $p(\overline{C}|c_m)\in {\mathcal{R}}^{1\times N}$ on N upstream classes for each downstream category $c_m$. Instead of connecting a downstream class to the most similar upstream class to transfer the upstream information, we select top-k similar upstream categories for each downstream category according to $p(\overline{C}|c_m)\in {\mathcal{R}}^{1\times N}$, forming granularity diffusion sequence ${\mathcal{S}}^m$, where $|{\mathcal{S}}^m|=k$. Diffusing to multiple upstream classes could introduce more semantic diversity through the one–k mapping, and thus provide the possibility of discrepancy for downstream samples of different classes, rather than collapsing into the same category.

Semantic discrepancy: Although semantic diffusion could alleviate the collapsed classification problem, it cannot ensure downstream discrimination, which is the objective of transfer learning. Hence, we should further constrain the diffusion process towards the downstream discrimination. The semantic discrepancy is introduced to the diffusion through tuning the k-value. This adjustment, maintaining consistency with our earlier definition of k, ensures that the granularity diffusion sequence of an arbitrary downstream category is different from the counterparts of the others. In other words, for each downstream dataset, the k-value is adaptively determined according to the inherent discrepancy of the downstream dataset itself, which is totally consistent with our intuition. For example, the “banded" and “lined" categories in the DTD dataset are significantly different from the upstream ImageNet dataset, and even the top four adjacent upstream categories are the same, which makes it hard to ensure the discrimination. However, the fifth adjacent upstream categories are different, and $k=5$ for other downstream category pairs also satisfies the condition. Therefore, $k=5$ is appropriate for the DTD dataset. We list the different values of k for the downstream datasets in Figure 3, which also reflects the downstream–upstream semantic granularity differences.

As a result, the regularization term, which forces the fine-tuned network to preserve the appropriate upstream information and considers the discrepant semantic diffusion, could be written as:

$$\begin{array}{cc}\hfill {\tilde{\ell }}_r& \hfill =-\sum _{m=0}^M\sum _{\{({\mathbf{x}}_i,{\mathbf{y}}_i)\in \mathcal{D}|{\mathbf{y}}_i=m\}}\sum _{n=0}^{N-1}(\mathcal{I}(c_n\in {\mathcal{S}}^m)log{\mathbf{h}}_i^m\left[n\right]+\\ & \hfill (1-\mathcal{I}(c_n\in {\mathcal{S}}^m))log(1-{\mathbf{h}}_i^m\left[n\right]))\\ \hfill & s.t.({\mathcal{S}}^u-{\mathcal{S}}^u\cap {\mathcal{S}}^v)\ne \varphi ,0\le u,v\le M,u\ne v\hfill \end{array}$$

(5)

where ${\mathbf{h}}_i^m=\sigma \left(\overline{\mathbb{FC}}\left(\mathbb{E}\left({\mathbf{x}}_i\right)\right)\right)$, and ${\mathbf{h}}_i^m\left[n\right]$ means the n-th elements of the prediction ${\mathbf{h}}_i^m$.

The semantic discrepancy could restrict the semantic diffusion towards a reasonable direction, selectively transferring the upstream information for different downstream categories of different downstream datasets, improving the transferring ability and the downstream performance.

Combined with the semantic diffusion and semantic discrepancy, DI2 could construct a diffusive downstream–upstream connection and thus adjust the mismatched semantic granularity. Meanwhile, it ensures the semantic discrepancy of the downstream tasks, further improving the transfer learning robustness. DI2 could effectively take advantage of the abundant upstream information and focus more on the discriminative characteristics of the downstream categories to avoid the collapsed classification. However, DI2 could also lead to a limitation in that our method is more suited for handling downstream tasks with fine-grained datasets, and tends to be less effective on coarser-grained downstream datasets.

3.4. Learning Pipeline

The proposed transfer learning method first conducts the pre-training with the semantic prior, preparing for the further diffusion in the fine-tuning stage. We determine the output of $\mathcal{N}$ for each upstream category through the prior, and thus utilize the prior to guide the pre-training through ${\ell }_u$. After that, we obtain a more generalized pre-trained model empowered with the semantic diffusion ability.

Then, the discrepant diffusion for fine-tuning trains the model adaptive to the downstream classification tasks through the binary cross-entropy loss ${\ell }_d$:

$$\begin{array}{cc}\hfill {\ell }_d=-\sum _i^{\left|\mathcal{D}\right|}\sum _m^M& ({\mathbf{y}}_i\left[m\right]log\left(\sigma \left(\mathbb{G}\left({\mathbf{x}}_i\right)\left[m\right]\right)\right)+\hfill \\ \hfill & (1-{\mathbf{y}}_i\left[m\right])log(1-\sigma \left(\mathbb{G}\left({\mathbf{x}}_i\right)\left[m\right]\right))),\hfill \end{array}$$

(6)

and regularizes the upstream information, preserving it as ${\tilde{\ell }}_r$ in Equation (5). The overall loss for the fine-tuned model is:

$${\ell }_o={\ell }_d+\lambda {\tilde{\ell }}_r,$$

(7)

where $\lambda$ is the weighting hyper-parameter.

In the end, we could obtain the fine-tuned model adaptive to each downstream task, speeding up the convergence of the network with limited labeled data and enhancing the downstream performance. The computation of our algorithm is not complicated, so it can be implemented at a relatively low cost on the backbone network and does not require expensive computing resources.

4. Experiments

In this section, we evaluate the proposed model on widely used benchmarks compared with the state-of-the-art transfer learning approaches. We will first introduce the experimental setting. Then, we will illustrate the comparison results with the state of the art under the full-dataset and few-shot settings, respectively. After that, we will investigate each component of our model to verify their effectiveness. We will also examine the transfer performance on different model architectures and different pre-training methods. Finally, we will present rich experimental results for hyper-parameter analysis.

4.1. Experimental Setting

Dataset: The upstream dataset is the most commonly used ImageNet-1k [59], consisting of 1000 classes with a total of 1.28 million images. As for the downstream datasets, we utilize the widely adopted downstream dataset Stanford Cars [60], FGVC Aircraft [61], Oxford Flowers 102 [62], Birdsnap [63], and Oxford-IIIT Pets [64], which are fine-grained. The texture classification dataset (DTD [65]) and scene classification dataset (SUN [66]), which are significantly dissimilar to the upstream dataset, are also involved. Meanwhile, to prove the generality of our method, we also conduct experiments on Caltech-256 [67], which is similar to ImageNet-1k.

Implementation details: We choose the commonly used ResNet-50 [68] as the baseline model for all methods and follow the same pre-training procedure across all CNN-based model architectures from scratch with a batch size of 1024. We first train it for 100 epochs with an initial learning rate of 0.4, which is dropped by a factor of 10 every 30 epochs. The optimizer is SGD, and the momentum and weight decay are set as 0.9 and $1\times {10}^{-4}$, respectively. In the fine-tuning stage, we fine-tune the pre-trained network for 150 epochs with an initial learning rate of 0.01, which dropped by a factor of 10 every 50 epochs. The $\lambda$ in Equation (7) is set to 0.01. All experiments are conducted on a single NVIDIA GTX 1080 Ti 11 GB GPU. In order to ensure the fairness of the comparison, for all the comparison methods, we conduct the experiments under the same setting with their published codes, and the experimental results are run three times and averaged.

4.2. Comparison with State-of-the-Arts

We compare our method on both the full-dataset setting and few-shot setting with the following state of the art: the vanilla fine-tuning model (Baseline), L2-SP [23], BSS [21], Co-Tuning [24], BYOL [50], SimCLR [51], NNCLR [52], SD [69] and NCTI [70]. We carefully retrain the first four methods under the same setting to ensure sufficient convergence of the network on the training set.

Comparison under the full-dataset setting:

Table 1. Transfer performance on eight prevalent downstream datasets under the full-dataset setting. Our model achieves almost the best performance compared to others. Bold indicates the best result.

Dataset	Method
	Baseline	L2-SP	BSS	Co-Tuning	BYOL	SimCLR	NNCLR	SD	NCTI	Ours
DTD	75.18 ± 0.24	74.95 ± 0.38	75.28 ± 0.28	74.46 ± 0.24	75.50	75.70	76.70	75.90	70.40	76.91 ± 0.23
SUN	62.83 ± 0.08	62.33 ± 0.06	62.87 ± 0.08	62.21 ± 0.16	62.20	60.60	62.50	62.50	75.60	64.26 ± 0.14
Caltech-256	84.28 ± 0.09	84.35 ± 0.12	84.26 ± 0.14	84.43 ± 0.11	-	-	-	-	-	86.32 ± 0.06
Cars	90.47 ± 0.05	90.78 ± 0.06	90.73 ± 0.02	90.99 ± 0.22	67.80	49.30	67.10	57.20	64.70	91.62 ± 0.19
Aircraft	86.65 ± 0.36	87.06 ± 0.12	87.04 ± 0.17	88.05 ± 0.23	60.60	49.80	64.10	55.30	49.60	88.81 ± 0.21
Flowers 102	96.60 ± 0.09	96.64 ± 0.06	96.87 ± 0.18	96.82 ± 0.20	96.10	92.60	95.10	92.90	54.10	97.44 ± 0.14
Birdsnap	72.58 ± 0.33	72.58 ± 0.18	73.32 ± 0.30	74.19 ± 0.23	57.20	42.4	61.40	-	-	75.32 ± 0.19
Pets	93.58 ± 0.14	93.59 ± 0.16	93.54 ± 0.10	93.18 ± 0.09	90.40	84.6	91.80	88.70	92.40	94.01 ± 0.13

lists all the quantitative results of all methods under the full-dataset setting. Owing to the effective discrepant semantic diffusion, our method achieves peak performance in almost all cases, such as 2.04% improvement on the Caltech-256 dataset. While our method records a 1.43% improvement over the baseline on the SUN dataset, it does not surpass the NCTI method. It is noteworthy that the results of NCTI are obtained from an average over a diverse array of eleven models, such as ResNet-50, ResNet-151, DenseNet-121, DenseNet-169, GoogleNet, and Inception-v3, among others. Given that our approach exclusively uses a ResNet-50 backbone, a fair comparison cannot be made. Moreover, we can find that previous methods which preserve the upstream information while under-utilizing the semantic discrepancy struggle to bring obvious improvement on the downstream datasets of mismatched semantic granularity from the upstream dataset.

For example, co-tuning considers the upstream and downstream category relationships of different datasets but neglects their discrepancies, and thus, it is difficult to yield better performance on datasets such as DTD, which are quite dissimilar to the upstream dataset. BYOL, SimCLR, and NNCLR employ the contrastive learning framework, whereas SD utilizes the synthetic images to pretrain strong general-purpose visual encoders. However, the performance of both approaches is relatively poor. In contrast, with the discrepancy information in the form of the granularity diffusion sequence, the proposed method could achieve superior performance on both fine-grained datasets and datasets of low similarity to the upstream, such as 1.63% improvement on the DTD dataset and 1.13% improvement on the Birdsnap dataset. It is worth noting that our improvement on some datasets is less than 1%, such as Flowers 102 and Pets, which is also an obvious gain in the relatively mature transfer learning field compared to other state of the art.

Comparison under the few-shot setting: One practical application of transfer learning is to quickly adapt pre-trained information to new tasks with limited data, addressing the issue of data sparsity caused by privacy protection. Therefore, the transferring performance under the few-shot setting is a key factor in measuring the effectiveness of the transfer learning algorithm, especially in the field of AI security.

Table 2. Transfer performance on Aircraft, Cars, DTD, and SUN datasets under the few-shot setting. Our method outperforms the others almost in all cases, and brings obvious gains especially with limited downstream data (15%). Bold indicates the best result.

Sampling Rates		Method
		Baseline	L2-SP	BSS	Co-Tuning	Ours
Aircraft	15%	43.65 ± 0.25	43.02 ± 0.14	45.74 ± 0.22	43.52 ± 0.35	48.52 ± 0.29
	30%	65.97 ± 0.29	66.08 ± 0.20	66.21 ± 0.21	66.60 ± 0.37	67.71 ± 0.17
	50%	77.41 ± 0.34	77.39 ± 0.24	78.06 ± 0.11	77.78 ± 0.26	79.59 ± 0.34
Cars	15%	42.27 ± 0.27	42.87 ± 0.33	43.03 ± 0.35	43.42 ± 0.37	47.17 ± 0.18
	30%	71.55 ± 0.24	71.87 ± 0.31	71.97 ± 0.19	72.08 ± 0.21	74.20 ± 0.16
	50%	83.44 ± 0.19	84.16 ± 0.25	84.34 ± 0.09	84.91 ± 0.21	85.59 ± 0.22
DTD	15%	60.21 ± 0.32	61.38 ± 0.38	61.06 ± 0.39	62.03 ± 0.24	60.44 ± 0.14
	30%	66.70 ± 0.35	66.96 ± 0.09	66.75 ± 0.26	66.62 ± 0.18	67.16 ± 0.21
	50%	68.22 ± 0.34	69.73 ± 0.17	69.62 ± 0.31	70.15 ± 0.19	71.73 ± 0.18
SUN	15%	45.35 ± 0.22	45.75 ± 0.15	45.02 ± 0.18	45.60 ± 0.16	47.54 ± 0.11
	30%	53.68 ± 0.14	53.40 ± 0.14	52.86 ± 0.08	53.73 ± 0.25	54.83 ± 0.16
	50%	57.72 ± 0.15	57.87 ± 0.15	57.53 ± 0.06	57.65 ± 0.07	59.34 ± 0.10

lists the results of different methods under few-shot settings. We conduct experiments on Aircraft, Cars, DTD, and SUN datasets, with 15%, 30%, and 50% of the whole data, respectively. Our method outperforms the others almost in all cases, and brings obvious gains especially with limited downstream data (15%), such as 3.75% improvement on the Cars (fine-grained) dataset and 1.79% improvement on the SUN dataset (dissimilar to upstream data), compared with suboptimal results. This proves that the proposed discrepant semantic diffusion could effectively improve the generalization of the model and is more practical, applicable and secure than the existing methods. Our method significantly reduces the training data requirements for models on new tasks, thereby lowering the potential security risks.

4.3. Ablation Study

In this section, we verify the efficacy of each component in our method on the Aircraft dataset and report the top-1 accuracy in Figure 4. Figure 4 shows the evaluation of (I) the baseline model, (II) the baseline model with PGD, (III) the baseline model with DI2, (IV) the full model under both the full-dataset setting (100%) and few-shot setting (50%, 30%, and 15%). From Figure 4, we can observe that each contribution brings a performance gain in most cases under the full-dataset setting. On the contrary, under the few-shot setting, (II) the baseline model with PGD shows a negative performance gap. This is because without sufficient downstream data, simply adopting the semantic prior would blur the boundaries of similar categories and do harm to the downstream performance. However, when further combined with our DI2, (IV) the full model, it can achieve the best results with sufficient semantic discrepancy.

Moreover, as shown in the results of (III) the baseline model with DI2, we can find that the proposed DI2 can achieve good performance when used alone, demonstrating its effectiveness and robustness. The baseline model performance with DI2 approaches the full model under the full-dataset setting but not when the data are insufficient. For example, in the experiment of Aircrafts with 15% of the whole data, the full model outperforms the baseline model with DI2 by about 2% as shown in Figure 4. In summary, the performance of the baseline model with DI2 is significantly inferior to that of the full model under the few-shot setting, which further proves the effectiveness of the semantic prior in PGD with sufficient semantic discrepancy involved.

4.4. Verification on Various Networks

In this section, we investigate the performance of the proposed discrepant semantic diffusion method with different backbones. We examine the transfer performance on three common networks (ResNet-{50, 101} [68] and MobileNetV3-Large 1.0 [71]). As shown in

Table 3. Transfer performance of the proposed method with multiple backbones, which proves the versatility of the proposed discrepant semantic diffusion method with different model structures. Bold indicates the best result.

Method	Dataset
	DTD	SUN	Caltech-256	Cars
Baseline (ResNet50)	75.18 ± 0.24	62.83 ± 0.08	84.28 ± 0.09	90.47 ± 0.05
Ours (ResNet50)	76.91 ± 0.23	64.26 ± 0.14	86.32 ± 0.06	91.62 ± 0.19
Baseline (ResNet101)	75.53 ± 0.10	63.23 ± 0.19	85.50 ± 0.19	91.23 ± 0.09
ours (ResNet101)	76.59 ± 0.11	64.73 ± 0.18	87.17 ± 0.06	91.65 ± 0.17
Baseline (MobileNetV3-Large)	71.64 ± 0.28	60.93 ± 0.05	82.50 ± 0.05	85.99 ± 0.10
Ours (MobileNetV3-Large)	72.25 ± 0.06	61.26 ± 0.06	82.95 ± 0.12	87.76 ± 0.13
Method	Dataset
	Aircraft	Flowers 102	Birdsnap	Pets
Baseline (ResNet50)	86.65 ± 0.36	96.60 ± 0.09	72.58 ± 0.33	93.58 ± 0.14
Ours (ResNet50)	88.81 ± 0.21	97.44 ± 0.14	75.32 ± 0.19	94.01 ± 0.13
Baseline (ResNet101)	87.80 ± 0.31	96.79 ± 0.38	73.71 ± 0.23	93.90 ± 0.11
ours (ResNet101)	88.86 ± 0.08	97.43 ± 0.16	75.41 ± 0.03	94.22 ± 0.02
Baseline (MobileNetV3-Large)	79.83 ± 0.10	96.19 ± 0.15	68.70 ± 0.23	91.73 ± 0.15
Ours (MobileNetV3-Large)	81.32 ± 0.26	96.68 ± 0.08	69.80 ± 0.03	92.14 ± 0.08

, the proposed discrepant semantic diffusion method is effective on both a large network (ResNet101) and a light network MobileNet, indicating it could be applied with multiple network structures. This further demonstrates the generality and versatility of our method across different model architectures in different scenarios.

4.5. Verification on Various Pre-Training Methods

We further prove the robustness of our DI2 method with different pre-training methods, including contrastive unsupervised pre-training MoCov2 [72], naive supervised pre-training, and our PGD. As shown in Figure 5, we can easily find that the proposed DI2 can be applied to various pre-training methods and achieve better performance. As for all the pre-training method, it is intuitive that naive supervised pre-training performs better than the unsupervised MoCov2, ranking the second. But our PGD pre-training yields the competitive results on both fine-tuning methods.

4.6. Hyper-Parameter Analysis

Selection of k: In this section, we investigate the effect of the hyper-parameter k in the semantic discrepancy of Section 3.3. As we mentioned, we determine k by adding 1 dissimilar class to guarantee that the granularity diffusion sequence of an arbitrary downstream category is different from the counterparts of the others. As shown in Figure 6a, we list the results of ranging the k from adding 0 (“+ 0”) to adding 5 (“+ 5”) dissimilar classes compared to the baseline model. We can easily find that the semantic diffusion operation can always help the model achieve better performance than the baseline model. But the “+ 0” setting, which cannot guarantee the semantic discrepancy of each of the two categories, performs worse than the other k settings. It proves that the semantic discrepancy is important for the transferring process. As k changes, our method can always achieve better and more stable performance than the baseline model. It proves that as long as the discrepant semantic diffusion can be guaranteed, the model can always achieve better transfer performance, which proves the rationality and effectiveness of our discrepancy diffusion method. Therefore, considering the training costs, we choose to add one dissimilar upstream class when determining k.

Selection of $\lambda$: In this section, we investigate the sensitivity of hyper-parameter $\lambda$, which are depicted in Equation (7). As mentioned in Section 3.4, $\lambda$ is introduced as a balancing factor to balance the original multi-class cross entropy loss and the proposed discrepant diffusion loss. We vary the $\lambda$ to [0.001, 0.01]. The performance of the Birdsnap and DTD datasets is reported in Figure 6b. It can be easily included that with the change of the introduced hyper-parameters, the proposed methods could achieve stable performance that is better than the baseline model on a large parameter range [0.001, 0.8]. Given $\lambda$, the model can achieve similar performance on multiple datasets. Therefore, it can be proved that our method has better robustness to the introduced hyper-parameters. It is worth noting that, for the DTD dataset, a bigger $\lambda$, such as $\lambda =1$, leads to poor performance. It is because the DTD dataset has less similarity to the upstream dataset. Therefore, a particularly large $\lambda$ can affect the original classification loss, resulting in poor classification performance.

Selection of $\tau$: In this section, we investigate the effect of the hyper-parameter $\tau$ in Equation (3) on the Birdsnap and DTD datasets. The hyper-parameter could control the penalty degree for the ground-truth class, the similar classes of the ground-truth classes, and the others. From Figure 6c, we can observe that, with the increasing of $\tau$, the top-1 accuracy first rises, reaching 76.91% when $\tau =100$, and then drops. It can be concluded that a larger $\tau$ would yield competitive performance since it could model the reasonable class relation and enable the semantic diffusion ability. However, when $\tau$ becomes too large, it will affect the semantic discrimination. Similar trends could be seen on both datasets. Therefore, we set $\tau =100$ under all the settings.

4.7. Visualization

Figure 7 shows the t-SNE visualizations of embeddings of “icefloe" and “snowfield" in the SUN dataset. We show the embeddings extracted from (a) the pretrained model, (b) the baseline fine-tuned model and (c) the model with our discrepant semantic diffusion, respectively. As Figure 7 depicts, the pretrained model shows collapsed classification due to the mismatched downstream–upstream semantic granularity, and thus two categories are mixed in the embedding space and cannot be distinguished. And rigidly preserving the upstream information while fine-tuning with the downstream data cannot well address this problem, without considering the discrepant downstream–upstream mapping to ensure the downstream classification. In contrast, our method could obviously alleviate the collapsed classification problem, benefiting from the proposed discrepant semantic diffusion and thus achieve superior performance.

5. Conclusions

In this paper, we propose a discrepant semantic diffusion method for transfer learning to handle the downstream collapsed classification problem and further boost the transfer learning robustness. The proposed framework consists of Prior-Guided Diffusion (PGD) for pre-training and Discrepant Diffusion (DI2) for fine-tuning. The PGD utilizes a semantic prior to model the valuable semantic connection, empowering the pre-trained model with the semantic diffusion ability. As a result, the pre-trained model becomes more generalized, paving the way for enhanced downstream discrimination. The DI2 in the fine-tuning stage models the diffusive downstream–upstream connection constrained by the semantic discrepancy, effectively aligning the shared downstream–upstream knowledge. Extensive experiments prove the superiority of our method for various downstream tasks, especially for the fine-grained datasets or datasets dissimilar to the upstream data (e.g., 3.75% and 1.79% improvements for the Cars and SUN datasets, respectively).