Transfer Learning for Logistic Regression with Differential Privacy

Abstract

Transfer learning, as a machine learning approach enhancing model generalization across different domains, has extensive applications in various fields. However, the risk of privacy leakage remains a crucial consideration during the transfer learning process. Differential privacy, with its rigorous mathematical foundation, has been proven to offer consistent and robust privacy protection. This study delves into the logistic regression transfer learning problem supported by differential privacy. In cases where transferable sources are known, we propose a two-step transfer learning algorithm. For scenarios with unknown transferable sources, a non-algorithmic, cross-validation-based transferable source detection method is introduced, to mitigate adverse effects from non-informative sources. The effectiveness of the proposed algorithm is validated through simulations and experiments with real-world data.

1. Introduction

Machine learning, as a critical tool for data utilization, has become the engine driving transformation across various industries. Through data modeling, it empowers computers with the ability to learn autonomously and to predict, finding wide applications in fields like financial risk assessment [1], autonomous driving [2], AI-driven healthcare [3], smart manufacturing [4], and more. In today’s digital era, safeguarding personal privacy has emerged as a crucial challenge in the information technology domain. Despite the rapid advancements in big data analysis and machine learning technologies, it is still a significant challenge to safeguard sensitive user information. It is vital to prioritize the protection of personal information while extracting valuable insights and knowledge from massive datasets.

Traditional methods of data privacy protection, such as anonymization [5] and de-identification, have become increasingly vulnerable and prone to exploitation by advanced data analysis techniques. In this context, differential privacy [6] has garnered extensive attention as a rigorous privacy protection framework. Differential privacy offers a mathematically rigorous approach that ensures that no specific individual’s information can be inferred during data analysis, even with knowledge of all other people’s data. This robust privacy-preserving property has facilitated the widespread application of differential privacy in sensitive environments.

In the realm of regression analysis, numerous studies have been conducted based on differential privacy analysis. Chaudhurietal et al. [7] proposed several differential privacy regression analyses, but they are limited by the requirement that the objective function must be convex and twice differentiable. Lei [8] introduced m-estimators differential privacy methods based on a maximum likelihood estimation framework, generating noise multidimensional histograms using the Laplace mechanism and utilizing synthetic data to compute regression results. However, this method is only applicable to lower-dimensional data. To handle high-dimensional data, Zhang et al. [9] proposed a noise addition method based on a functional mechanism, perturbing the objective function of the optimization problem instead of the results, to enforce $ϵ$-differential privacy. Kifer et al. [10] focused on high-dimensional sparse regression problems, presenting a differential privacy-based convex empirical risk minimization (ERM) method. Smith [11] combined the Laplace mechanism and the exponential mechanism, proposing a general differential privacy framework, but with limitations concerning the bounded output space, studying the asymptotic properties of differentially private algorithms for statistical inference. Barrientos et al. [12]. designed algorithms for differential private estimators, regarding significance levels and symbols. Cai et al. [13] studied the trade-off between statistical accuracy and privacy in mean estimation and linear regression in both classical low-dimensional and modern high-dimensional settings, proposing a novel private iterative hard threshold algorithm for high-dimensional linear regression. In the field of logistic regression, there is existing literature discussing the relevance of differential privacy. Chaudhuri and Monteleoni [14] discussed the trade-off between privacy and learnability in designing algorithms for learning from private databases. They proposed a privacy-preserving logistic regression algorithm that solves the problem of sensitivity in regularized logistic regression. The approach involves adding noise to the learned classifier, which is proportional to the sensitivity. This is a method that is independent of the function’s sensitivity and can be used for a class of convex loss functions. Khanna et al. [15] introduced a differentially private method for sparse logistic regression that maintains hard zero coefficients. They achieved this by first training a non-private LASSO logistic regression model to determine the appropriate number of non-zero coefficients for the final model selection. Xu et al. [16] took a unique approach by combining functional mechanism and decision boundary fairness, to develop a differentially private and fair logistic regression model. This innovative method ensures both privacy preservation and fairness while maintaining good utility. Fan et al. [17] proposed a privacy-preserving logistic regression algorithm (PPLRA) that uses homomorphic encryption to prevent data privacy leakage. They shifted the majority of computational tasks to the cloud, to enhance efficiency while safeguarding data privacy. Ji et al. [18] modified the update steps in the Newton–Raphson method and presented a differentially private distributed logistic regression model based on both public and private data. This model improves practicality by leveraging a public dataset while simultaneously protecting the private dataset, all without compromising stringent privacy guarantees.

Simultaneously, many practical applications face a common challenge: the scarcity or expense of data in the target domain (the domain of the target task) while there may be ample labeled data available in the source domain (often a related but not entirely identical task domain). To overcome this data scarcity issue, transfer learning has emerged. The core concept of transfer learning lies in transferring knowledge learned from the source domain, to enhance the learning performance in the target domain.

The concept of transfer learning initially originated in the field of machine learning. In 1993, Pratt [19] introduced a neural network-based transfer learning method, using weights obtained from a network trained for related source tasks to expedite learning for the target problem, exploring how to transfer knowledge across different tasks to improve learning effectiveness. Subsequently, in 1995, Thrun [20] first proposed the concept of “knowledge transfer” and investigated methods for sharing knowledge between different tasks. This paper is considered one of the pioneering works in the field of transfer learning, laying the foundation for subsequent research. Building upon these pioneering papers, many researchers conducted in-depth studies. However, currently, transfer learning is mostly applied in the classification domain, including tasks like image classification [21], text classification [22], and time series classification [23]. In the regression analysis domain, there are also some related literatures. Li et al. [24] proposed a two-step method for estimating and predicting transfer learning on high-dimensional linear regression models, called Trans-Lasso, including a detection algorithm for consistently identifying transferable but unknown source datasets and a contrastive regularized estimator anchored to the target task. They demonstrated the robustness of Trans-Lasso in dealing with information-free auxiliary samples and its efficiency in knowledge transfer even when the auxiliary sample set is unknown. This paper marked an important step in the application of transfer learning in the regression domain. Bastani [25] proposed a two-step procedure for studying transfer learning methods in high-dimensional linear regression models using a single source dataset without needing to detect transferable source datasets from candidate datasets. Tian and Feng [26] extended this method to generalized linear models, introducing an algorithm to construct confidence intervals for each coefficient component and accompanying theoretical results. Yang et al. [27] proposed a method that combines source and target datasets using a two-layer linear neural network, accurately calculating the asymptotic limits of transfer learning prediction risk on high-dimensional linear models. Zhou et al. [28] proposed double robust transfer learning, to adapt to the challenges of label scarcity and covariate shift in the target task. Additionally, Lin and Reimherr [29] conducted transfer learning research on functional linear regression models and established the optimal convergence rate for excess risk. Takada and Fujisawa [30] proposed a method to transfer knowledge from the source domain to the target domain through high-dimensional ${\ell }_1$ regularization, which, in addition to ordinary ${\ell }_1$ regularization, also incorporates ${\ell }_1$ regularization of the difference between source parameters and target parameters. This approach produces sparsity both for the estimate itself and for changes in the estimate, has tight estimation error bounds in stationary environments, and the estimate remains invariant to the source estimate under small residuals; even if the source estimate is wrong due to non-stationarity, the estimates are also consistent with the basis functions.

While differential privacy and transfer learning have each demonstrated effectiveness in real-world applications, combining these two paradigms and applying them to logistic regression models is a challenging and compelling area of research. Combining transfer learning and differential privacy solves the fundamental problem of how to leverage knowledge in one domain to enhance learning capabilities in another domain while ensuring the privacy of sensitive information within the model.

This paper proposes a method that combines differential privacy with transfer learning and applies it to logistic regression models. Our approach not only protects individual privacy but also facilitates knowledge transfer between the source and target domains. Specifically, (1) we protect individual privacy by introducing differential privacy noise into input data, and (2) we design a transfer learning strategy to achieve knowledge transfer by sharing feature representations between the source and target domains. Our experimental results demonstrate that our method achieves good performance in the target domain while preserving individual privacy, proving its effectiveness and feasibility in practical applications.

Figure 1 provides a general overview of the proposed algorithm’s conceptual framework in this paper.

The structure of this paper is as follows. In Section 2, we present a setup for transfer learning using logistic regression with a focus on differential privacy. We also propose a transferable source domain detection algorithm based on cross-validation. In Section 3, we conduct various numerical simulations and empirical analyses. Finally, in Section 4, we provide a summary of our findings and suggest future research directions.

2. Method

2.1. Function Mechanism Differential Privacy Method Based on Logistic Regression Model

We first consider the function mechanism (FM) based on the logistic regression model, which is an extension of the Laplace mechanism in differential privacy. This privacy-preserving method does not directly inject noise into the regression results; instead, it ensures privacy by perturbing the optimization objective of the regression analysis.

The Laplace mechanism is a randomization technique in differential privacy that adds noise to query results using samples from the Laplace distribution. The probability density function of the Laplace distribution is given by $f(x\mid b)={\displaystyle \frac{1}{2b}}{e}^{{\displaystyle \frac{-\left|\mathrm{x}\right|}{b}}}$, where b is the scale parameter. The Laplace distribution, being centered around zero with heavy tails, is suitable for introducing random noise. The fundamental idea of the Laplace mechanism is to maintain the usability of data analysis results while introducing moderate noise to make individual data changes difficult to trace. This mechanism aims to balance privacy protection and data utility. Its mathematical representation is $q\left(D\right)+$$Laplace(\Delta q/\epsilon )$, where $q\left(D\right)$ is the query result, $\Delta q$ is the sensitivity of the query result, and $\epsilon$ is the privacy parameter.

Let $\mathcal{D}$ be a dataset containing n tuples $t_1,t_2,\dots ,t_n$ with $d+1$ attributes $X_1,X_2,\dots ,X_d,Y$. For each tuple $t_i=\left(x_{i1},x_{i2},\dots ,x_{id},y_i\right)$, we assume that $\sqrt{{\sum }_{i=1}^d{x}_{id}^2}\le 1$. Our objective is to construct a regression model from $\mathcal{D}$ that allows us to predict the value of any tuple on Y based on the values of $X_1,X_2,\dots ,X_d$. In other words, our goal is to obtain a function $\rho$ that takes $\left(x_{i1},x_{i2},\dots ,x_{id}\right)$ as input and outputs a prediction for $y_i$ as accurately as possible.

For logistic regression, assuming that the attribute Y in $\mathcal{D}$ takes values in the Boolean domain $\{0,1\}$, logistic regression on $\mathcal{D}$ returns a predictive function with the probability

$$\rho \left(x_i\right)=exp\left(x_i^T{\omega }^{*}\right)/\left(1+exp\left(x_i^T{\omega }^{*}\right)\right)$$

(1)

predicting $y_i=1$, where ${\omega }^{*}$ is a vector of d real numbers. This can be achieved by minimizing the cost function

$$f\left(t_i,\omega \right)=log\left(1+exp\left(x_i^T{\omega }^{*}\right)\right)-y_i{x}_i^T\omega$$

(2)

Namely,

$${\omega }^{*}=arg\underset{\omega }{min}\sum _{i=1}^n\left(log\left(1+exp\left(x_i^T{\omega }^{*}\right)\right)-y_i{x}_i^T\omega \right)$$

(3)

To ensure privacy, we require that the regression analysis should be performed using an algorithm that satisfies $\epsilon$-differential privacy.

A random algorithm $\mathcal{M}$ satisfies $\epsilon$-differential privacy if and only if for any output $\mathcal{O}$ of M and for any two neighboring databases $D_1$ and $D_2$ we have

$$Pr\left[\mathcal{M}\left(D_1\right)=\mathcal{O}\right]\le e^{\epsilon }Pr\left[\mathcal{M}\left(D_2\right)=\mathcal{O}\right]$$

(4)

If $\mathcal{M}$ satisfies $\epsilon$-differential privacy, then the probability distribution of its output remains almost the same for any two input databases that differ by only one tuple.

The FM method we use does not directly inject noise into ${\omega }^{*}$. Instead, it perturbs the objective function $f_{\mathcal{D}}\left(\omega \right)={\sum }_{t_{i\in \mathcal{D}}}f\left(t_i,\omega \right)$, then releases the model parameters $\overline{\omega }$ that minimize the perturbed objective function $\overline{f_{\mathcal{D}}\left(\omega \right)}$. Here, we address this by utilizing a polynomial representation of $f_{\mathcal{D}}\left(\omega \right)$. As $\omega$ is a vector containing values ${\omega }_1,{\omega }_2,\dots \dots \dots ,{\omega }_d$, let $\varphi \left(\omega \right)$ represent the product of ${\omega }_1,{\omega }_2,\dots \dots \dots ,{\omega }_d$, defined as $\varphi \left(\omega \right)={\omega }_1^{c1}·{\omega }_2^{c2}·\dots \dots \dots ·{\omega }_d^{cd}$, where $c_1,\dots \dots \dots ,c_d\in \mathbb{N}$. Let ${\Phi }_j(j\in \mathbb{N})$ represent the powers of ${\omega }_1,{\omega }_2,\dots \dots \dots ,{\omega }_d$ up to j, i.e., ${\Phi }_j=$$\left\{{\omega }_1^{c1}·{\omega }_2^{c2}·\dots \dots \dots ·{\omega }_d^{cd}\mid {\sum }_{l=1}^d{c}_l=j\right\}$. According to the Stone–Weierstrass theorem, any continuous differentiable function $f\left(t_i,\omega \right)$ can always be written as a (potentially infinite) polynomial in ${\omega }_1,{\omega }_2,\dots \dots \dots ,{\omega }_d$, i.e., for some $J\in [0,+\infty ]$ we have $f_{\mathcal{D}}\left(\omega \right)=$${\sum }_{j=0}^J{\sum }_{\varphi \in {\Phi }_j}{\lambda }_{\varphi t_i}\varphi \left(\omega \right)$, where ${\lambda }_{\varphi t_i}\in \mathbb{R}$ represents the coefficients of $\varphi \left(\omega \right)$ in the polynomial.

Let $\mathcal{D}$ and ${\mathcal{D}}^{\prime }$ be any two arbitrarily neighboring databases, and let $f_{\mathcal{D}}\left(\omega \right)$ and $f_{{\mathcal{D}}^{\prime }}\left(\omega \right)$ represent the objective functions of regression analysis on $\mathcal{D}$ and ${\mathcal{D}}^{\prime }$, respectively, with polynomial representations as

$$f_{\mathcal{D}}\left(\omega \right)=\sum _{j=1}^J\sum _{\varphi \in {\Phi }_j}\sum _{t_i\in \mathcal{D}}{\lambda }_{\varphi t_i}\varphi \left(\omega \right)$$

(5)

$$f_{{\mathcal{D}}^{\prime }}\left(\omega \right)=\sum _{j=1}^J\sum _{\varphi \in {\Phi }_j}\sum _{t_i^{\prime }\in {\mathcal{D}}^{\prime }}{\lambda }_{\varphi t_i^{\prime }}\varphi \left(\omega \right)$$

(6)

Thus,

$$\sum _{j=1}^J\sum _{\varphi \in {\Phi }_j}∥\sum _{t_i\in \mathcal{D}}{\lambda }_{\varphi t_i}-\sum _{t_i^{\prime }\in {\mathcal{D}}^{\prime }}{\lambda }_{\varphi t_i^{\prime }}∥\le 2\underset{t}{max}\sum _{j=1}^J\sum _{\varphi \in {\Phi }_j}{∥{\lambda }_t∥}_1.$$

(7)

Let

$$\Delta =2\underset{t}{max}\sum _{j=1}^J\sum _{\varphi \in {\Phi }_j}{∥{\lambda }_t∥}_1.$$

(8)

For $\varphi \in {\Phi }_j$, let

$${\lambda }_{\varphi }=\sum _{t_i\in \mathcal{D}}{\lambda }_{\varphi t_i}+Lap\left({\displaystyle \frac{\Delta }{\epsilon }}\right).$$

(9)

This yields the perturbed objective function, and the minimum solution can be computed from it.

Due to the requirement in differentially private methods based on the function mechanism that the polynomial form of the objective function should only contain bounded-degree terms, the logistic regression model fails to meet this condition. Hence, Zhang et al. [9] proposed an approach based on Taylor expansion, to derive an approximate polynomial form for the objective function, demonstrating its effectiveness in achieving differential privacy. Therefore, in this paper, we adopt this validated approach. The form of the objective function is as follows:

$$\widehat{f_{\mathcal{D}}}\left(\omega \right)=\sum _{i=1}^n\left(0.5x_i^T·\omega +1/8{\left(x_i^T·\omega \right)}^2-y_i·x_i^T·\omega +log2\right)$$

(10)

2.2. Regression Transfer Learning Based on Differential Privacy

In this paper, we address the issue of multi-source domain transfer learning. Consider a target dataset $\left(X^{\left(0\right)},y^{\left(0\right)}\right)$ and K source domain datasets $\left(X^{\left(K\right)},y^{\left(K\right)}\right)$, where $X^{\left(k\right)}\in {\mathbb{R}}^{n_k\times p}$ and $y^{\left(k\right)}\in {\mathbb{R}}^{n_k\times 1}$. Here, $X_i^{\left(k\right)}$ and $y_i^{\left(k\right)}$ represent the i-th row of $X^{\left(k\right)}$ and the i-th element of $y^{\left(k\right)}$, respectively. The objective is to transfer useful information from the source data, to enhance the model’s performance for the target data. We assume that the relationship between independent variables and dependent variables in the target data and source data follows the logical model

$$P\left(y_i^{\left(k\right)}=1\mid x_i^{\left(k\right)}\right)={\displaystyle \frac{1}{1+e^{-\left(x_i^{\left(k\right)}·{\omega }^{\left(k\right)}\right)}}},i=1,\dots ,n_k$$

(11)

As $k=0,\dots \dots \dots ,K$, distinct coefficients ${\omega }^{\left(k\right)}$ are present. We refer to the target parameter as $\beta ={\omega }^{\left(0\right)}$ and assume that it is sparse in the zero norm. This means that among p variables, only s variables contribute to predicting the response variable, where $s<p$. The similarity between the coefficients ${\omega }^{\left(k\right)}$ of the source domain data and $\beta$ determines the usefulness of the k-th source domain in predicting the target domain. The k-th source domain is more helpful in predicting the target domain if its coefficients ${\omega }^{\left(k\right)}$ are closer to $\beta$. We measure the difference between the k-th source domain and the target domain as

$${\delta }^{\left(k\right)}=\beta -{\omega }^{\left(k\right)}$$

(12)

Consequently, we can derive the information set $\mathcal{A}=\left\{1\le k\le K:{∥\beta -{\omega }^{\left(k\right)}∥}_1\le h\right\}$. In terms of the ${\mathrm{L}}_1$ norm, if ${∥{\delta }^{\left(k\right)}∥}_1\le h$, we refer to the k-th source as h-transferable; if ${∥{\delta }^{\left(k\right)}∥}_1>$h then we refer to the k-th source as h-non-transferable. It is evident that a smaller h value implies greater benefits obtained from these source domains in the context of transfer learning.

In the field of differential privacy-based logistic regression transfer learning, there is a method known as the Oracle algorithm when $\mathcal{A}$ is already known. Our proposed algorithm follows the principles used by Bastani [25], Li et al. [24], Zhang et al. [9], and others, which is called the differential privacy transfer learning algorithm. The main idea behind this algorithm is to transfer information from transferable sources, to obtain rough estimators in the first step. Then, in the second step, the target data are used to correct biases. To ensure privacy protection, Laplace noise is added to the data in both steps.

In the first step of transfer learning, parameter estimation is required:

$${\widehat{\omega }}^{\mathcal{A}}=arg\underset{\beta }{min}\left\{{\displaystyle \frac{1}{n_{\mathcal{A}}+n_0}}\sum _{k\in \left\{0\right\}\cup \mathcal{A}}\sum _{i=1}^{n_k}\left(0.5·x_i^{\left(k\right)}\beta +1/8{\left(x_i^{\left(k\right)}\beta \right)}^2-y_i^{\left(k\right)}{x}_i^{\left(k\right)}\beta +log2\right)+{\lambda }_{\omega }{\parallel \beta \parallel }_1\right\}$$

(13)

We simplify $0.5·x_i^{\left(k\right)}\beta +1/8{\left(x_i^{\left(k\right)}\beta \right)}^2-y_i^{\left(k\right)}{x}_i^{\left(k\right)}\beta$, adding noise to coefficients before $\beta$:

$$0.5·x_i^{\left(k\right)}\beta +1/8{\left(x_i^{\left(k\right)}\beta \right)}^2-y_i^{\left(k\right)}{x}_i^{\left(k\right)}\beta =x_i^{\left(k\right)}\left(0.5-y_i^{\left(k\right)}\right)\beta +1/8{\left(x_i^{\left(k\right)}\right)}^2{\beta }^2,$$

adding Laplace noise to $x_i^{\left(k\right)}\left(0.5-y_i^{\left(k\right)}\right)$ and ${\left(x_i^{\left(k\right)}\right)}^2$ to obtain the privacy-protected ${\left(x_i^{\left(k\right)}\left(0.5-y_i^{\left(k\right)}\right)\right)}^{\prime }$ and ${\left(x_i^{\left(k\right)}\right)}^{2\prime }$. As the value of ${\widehat{\omega }}^{\mathcal{A}}$ depends solely on $\beta$, the expression is modified to

$${\widehat{\omega }}^{\mathcal{A}}=arg\underset{\beta }{min}\left\{{\displaystyle \frac{1}{n_{\mathcal{A}}+n_0}}\sum _{k\in \left\{0\right\}\cup \mathcal{A}}\sum _{i=1}^{n_k}\left({\left(x_i^{\left(k\right)}\left(0.5-y_i^{\left(k\right)}\right)\right)}^{\prime }\beta +1/8{\left(x_i^{\left(k\right)}\right)}^{2\prime }{\beta }^2\right)+{\lambda }_{\omega }{\parallel \beta \parallel }_1\right\}$$

(14)

We are performing both differential privacy and feature selection at the same time. To achieve this, we obtain the value of ${\lambda }_{\omega }$ through cross-validation.

Likewise, in the second step of transfer learning, parameter correction is required:

$$\widehat{\beta }={\widehat{\omega }}^{\mathcal{A}}+{\widehat{\delta }}^{\mathcal{A}}$$

(15)

$${\widehat{\delta }}^{\mathcal{A}}=arg\underset{\delta }{min}\left\{{\displaystyle \frac{1}{n_0}}\sum _{i=1}^{n_0}\left(0.5·x_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)+1/8{\left(x_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)\right)}^2-y_i^{\left(0\right)}{x}_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)+log2\right)+{\lambda }_{\delta }{\parallel \delta \parallel }_1\right\}$$

(16)

We simplify $\left(0.5·x_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)+1/8{\left(x_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)\right)}^2-y_i^{\left(0\right)}{x}_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)\right)$, adding noise to the coefficients before $\delta$:

$$\left(0.5·x_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)+1/8{\left(x_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)\right)}^2-y_i^{\left(0\right)}{x}_i^{\left(0\right)}\left({\widehat{\omega }}^{\mathcal{A}}+\delta \right)\right)=x_i^{\left(0\right)}\left(0.5-y_i^{\left(0\right)}+0.25x_i^{\left(0\right)}{\widehat{\omega }}^{\mathcal{A}}\right)\delta +1/8{\left(x_i^{\left(0\right)}\right)}^2{\delta }^2$$

We add Laplace noise to $x_i^{\left(0\right)}\left(0.5-y_i^{\left(0\right)}+0.25x_i^{\left(0\right)}{\widehat{\omega }}^{\mathcal{A}}\right)$ and ${\left(x_i^{\left(0\right)}\right)}^2$, to obtain the privacy-protected $x_i^{\left(0\right)}\left(0.5-y_i^{\left(0\right)}\right.{\left.+0.25x_i^{\left(0\right)}{\widehat{\omega }}^{\mathcal{A}}\right)}^{\prime }$ and ${\left(x_i^{\left(0\right)}\right)}^{2\prime }$. As the value of ${\widehat{\delta }}^{\mathcal{A}}$ depends solely on $\delta$, the expression is modified to

$${\widehat{\delta }}^{\mathcal{A}}=arg\underset{\delta }{min}\left\{{\displaystyle \frac{1}{n_0}}\sum _{i=1}^{n_0}\left(x_i^{\left(0\right)}{\left(0.5-y_i^{\left(0\right)}+0.25x_i^{\left(0\right)}{\widehat{\omega }}^{\mathcal{A}}\right)}^{\prime }\delta +1/8{\left(x_i^{\left(0\right)}\right)}^{2\prime }{\delta }^2\right)+{\lambda }_{\delta }{\parallel \delta \parallel }_1\right\}$$

(17)

We simultaneously conduct differential privacy and feature selection, where ${\lambda }_{\delta }$ is obtained through cross-validation.

The aforementioned approach is denoted as the Oracle trans DPLR algorithm.

2.3. Transferable Source Detection

During our previous discussions, we assumed that the transferable set $\mathcal{A}$ was known. However, in practical applications, it can be challenging to fulfill this assumption. Simply selecting all sources based on the target data might not improve model performance; instead, it could lead to negative transfer and a decline in learning performance for the target task. To avoid negative transfer, this paper utilizes a simple, non-data-driven method, proposed by Tian et al., to detect transferable sources for information transfer. Initially, the target data are separated into three subsets: ${\left\{\left(x^{\left(0\right)\left[r\right]},y^{\left(0\right)\left[r\right]}\right)\right\}}_{r=1}^3$. The average loss ${\widehat{L}}_0^{\left(0\right)}$ is then computed through cross-validation of the target data. Next, Algorithm 1 is applied to various data subsets from each source domain and the target domain, and estimated coefficients are obtained. The average loss ${\widehat{L}}_0^{\left(k\right)}$ is then computed on the test set. Finally, the difference between these two losses is compared against a predefined threshold. If the difference falls below the threshold, the source domains are added to set $\mathcal{A}$.

Algorithm 1: Oracle Trans DPLR

Input: target data $\left(X^{\left(0\right)},y^{\left(0\right)}\right)$, all h-transferable source data ${\left\{\left(X^{\left(k\right)},y^{\left(k\right)}\right)\right\}}_{k\in \mathcal{A}}$, penalty parameters ${\lambda }_{\omega }$ and ${\lambda }_{\delta }$, Laplace noise $Lap\left({\displaystyle \frac{\Delta }{\epsilon }}\right)$ Output: the estimated coefficient vector $\widehat{\beta }$ Transferring Step: $${\widehat{\omega }}^{\mathcal{A}}=arg\underset{\beta }{min}\left\{{\displaystyle \frac{1}{n_{\mathcal{A}}+n_0}}\sum _{k\in \left\{0\right\}\cup \mathcal{A}}\sum _{i=1}^{n_k}\left(\left(x_i^{\left(k\right)}(0.5-y_i^{\left(k\right)})+Lap\left({\displaystyle \frac{\Delta }{\epsilon }}\right)\right)\beta +\left({\displaystyle \frac{1}{8}}{\left(x_i^{\left(k\right)}\right)}^2+Lap\left({\displaystyle \frac{\Delta }{\epsilon }}\right)\right){\beta }^2\right)+{\lambda }_{\omega }{\parallel \beta \parallel }_1\right\}$$ Debiasing Step: Compute $${\widehat{\delta }}^{\mathcal{A}}=arg\underset{\delta }{min}\left\{{\displaystyle \frac{1}{n_0}}\sum _{i=1}^{n_0}\left(\left(x_i^{\left(0\right)}\left(0.5-y_i^{\left(0\right)}+0.25x_i^{\left(0\right)}{\widehat{\omega }}^{\mathcal{A}}\right)+Lap\left({\displaystyle \frac{\Delta }{\epsilon }}\right)\right)\delta +\left({\displaystyle \frac{1}{8}}{\left(x_i^{\left(0\right)}\right)}^2+Lap\left({\displaystyle \frac{\Delta }{\epsilon }}\right)\right){\delta }^2\right)+{\lambda }_{\delta }{\parallel \delta \parallel }_1\right\}$$ Step3: Let $\widehat{\beta }\leftarrow {\widehat{\mathit{w}}}^{\mathcal{A}}+{\widehat{\delta }}^{\mathcal{A}}$ Step4: Output $\widehat{\beta }$

To facilitate notation, assuming $n_0$ is divisible by 3, the average loss over r iterations on the target domain dataset $\left(x^{\left(0\right)\left[r\right]},y^{\left(0\right)\left[r\right]}\right)$ for any estimated parameter $\omega$ can be formally defined as

$${\widehat{L}}_0^{\left[r\right]}\left(\omega \right)={\displaystyle \frac{1}{n_0/3}}\sum _{i=1}^{n_0/3}\left(0.5x_i^{\left(0\right)\left[r\right]}\omega +1/8{\left(x_i^{\left(0\right)\left[r\right]}\omega \right)}^2-y_i^{\left(0\right)\left[r\right]}{x}_i^{\left(0\right)\left[r\right]}\omega +log2\right)$$

(18)

Algorithm 2 illustrates the specific details.

Algorithm 2: Transferable Source Detection

3. Simulation Study

For this section, we validated the effectiveness of the proposed algorithm through multiple simulation experiments and real data verification. Some parameters and formula configurations in the data-generation phase were inspired by Tian and Feng [26]. In the simulation segment, we initially compared the fitting results under different values of h (the maximum deviation between the source and the target coefficients in $\mathcal{A}$) and |$\mathcal{A}$| (the cardinality of $\mathcal{A}$) for naive DPLR using only target data and the Oracle trans DPLR proposed in Algorithm 1, involving three error settings. Subsequently, introducing h-non-transferable sources, we compared naive DPLR using only target data, the Oracle trans DPLR proposed in Algorithm 1, all_trans DPLR using transfer learning on all source and target data, and trans DPLR involving transferable source detection. In the empirical study, we explored different approaches of naive DPLR, trans DPLR, and all_trans DPLR. All subsequent experiments were implemented using Python 3.12 code and solved optimization problems using the cvxpy library.

3.1. Known Transferable Source Domain

This article considers the following sparse logistic regression model, where the target and source datasets are independently generated from Equation (19):

$$Y=X\beta +ϵ.\left(19\right)$$

(19)

Here, $X\sim N\left(0,{\left(0.5^{|i-j|}\right)}_{i,j=1}^p\right)$, and we transform the target variable Y into a binary attribute by mapping values above a predefined threshold to 1 and values below or equal to the threshold to 0. Therefore, when employing the logistic regression model for the classification of $(X,Y)$, if ${\displaystyle \frac{exp\left(x_i^T\omega \right)}{1+exp\left(x_i^T\omega \right)}}>0.5$ then we predict Y as 1; otherwise, it is predicted as 0. The parameters are set as follows: $n_0=300,n_k=400,K=\{3,6,9,12,15,18\},p=50,s=10$. The coefficient vector for the target domain is ${\beta }_0={\left(1_s,0_{p-s}\right)}^T$. For simplicity, let $d_k$ be a p dimensional vector with each component generated as 1 or −1 with a probability of 0.5. For the transferable source dataset ${\mathcal{A}}_h$, we consider $h=\{2,4,6,8,10,12,14,16,18,20\}$, and the coefficients are given by ${\beta }_k={\beta }_0+{\displaystyle \frac{h}{p}}{d}_k$, ensuring $\left|{\beta }_k-{\beta }_0\right|\le h$. Three error distributions are considered:

• Standard normal distribution: $ϵ\sim N(0,1)$
• $\mathrm{t}$ distribution: $ϵ\sim t\left(3\right)$
• Mixture of normal distributions: $ϵ\sim 0.9N(0,1)+0.1N\left(0,4^2\right)$

To balance classification accuracy and privacy protection, we compared the changes in classification accuracy of logistic regression models before and after applying transfer learning under two different conditions: ${\displaystyle \frac{\Delta }{\epsilon }}=0.1$ and ${\displaystyle \frac{\Delta }{\epsilon }}=0.01$ in the scenario of $h=3$. The results are shown in

Table 1. Classification accuracy under different ${\displaystyle \frac{\Delta }{\epsilon }}$ conditions.

		$\frac{\mathbf{\Delta }}{\mathit{\epsilon }}=0.1$	$\frac{\mathbf{\Delta }}{\mathit{\epsilon }}=0.01$
N	naive DPLR	87.67%	90.00%
	Oracle trans DPLR	92.33%	96.67%
Mix	naive DPLR	82.33%	83.67%
	Oracle trans DPLR	84.33%	89.00%
T	naive DPLR	80.00%	82.33%
	Oracle trans DPLR	82.67%	85.67%

:

As can be seen from Table 1, with the decrease of the noise addition amount ${\displaystyle \frac{\Delta }{\epsilon }}$, the accuracy of the model before and after transfer learning gradually improved. When ${\displaystyle \frac{\Delta }{\epsilon }}=0.1$ the accuracy of the model before and after transfer learning decreased compared to ${\displaystyle \frac{\Delta }{\epsilon }}=0.01$, but this decrease was still within an acceptable range, and the model’s performance could meet the needs of practical applications. Considering the need for privacy protection, we set ${\displaystyle \frac{\Delta }{\epsilon }}=0.1$ in subsequent experiments.

For each scenario, the experiment was repeated 100 times and the average $l_2$ error of the estimated parameters $\widehat{\beta }$ was used as the model evaluation metric. Each $\lambda$ during the experiment was chosen using cross-validation criteria. The average relative estimation error of the Oracle algorithm under three error settings is illustrated in Figure 2.

Here we define the relative error of the oracle algorithm as

$$\mathrm{relative}\mathrm{error}=\log (\mathrm{error}\mathrm{of}\mathrm{the}\mathrm{Oracle}\mathrm{trans}\mathrm{DPLR}/\mathrm{error}\mathrm{of}\mathrm{naive}\mathrm{DPLR})$$

Therefore, negative values indicate that the Oracle trans DPLR algorithm outperformed the naive DPLR algorithm, with smaller values (represented by lighter colors in Figure 2) indicating better performance.

Based on the detailed analysis of Figure 2, the Oracle trans DPLR algorithm demonstrated significant performance advantages over the naive DPLR algorithm across a wide range of combinations of h and dataset sizes $\left|\mathcal{A}\right|$. This finding profoundly reveals the efficiency of the Oracle trans DPLR algorithm in handling complex data scenarios. In particular, as the number of data sources increased, the performance of the Oracle trans DPLR algorithm showed a marked improvement. Simultaneously, as the parameter h increased, the complexity of the problem also rose, leading to an increase in the estimation error of the Oracle trans DPLR algorithm. However, it is noteworthy that the algorithm maintained stable prediction performance even when faced with different types of error distributions, which fully validates the robustness and generalization capability of its design.

3.2. Unknown Transferable Source Domain

First, we set the total number of source domains to 20, i.e., $K=20$. Further, we construct transferable source domains and non-transferable source domains in scenarios where $\left|{\mathcal{A}}_h\right|=\{0,2,4,6,8,10,12,14,16,18,20\}$. For transferable source domains, we keep the logistic regression parameters consistent with the previous context. For non-transferable source domains, we randomly select a subset $I_k$ of size s from the set $\{2s+1,\dots ,p\}$, and we set the j-th component of the logistic regression parameter ${\beta }_k$, as follows:

$${\beta }_{kj}=\left\{\begin{array}{cc}1+{\displaystyle \frac{h}{p}}{d}_{kj}& j\in \{s+1,\dots 2s\}\cup I_k\\ {\displaystyle \frac{h}{p}}{d}_{kj}& \mathrm{otherwise}\end{array}\right.$$

(20)

All the other parameters remain the same as in the previous section. This paper compares the estimated values of the following algorithms:

• Naive DPLR: Conducting differentially private logistic regression using only target domain data.
• Trans DPLR: Performing differentially private transfer learning using Algorithm 1 and Algorithm 2.
• All_trans DPLR: Conducting differentially private transfer learning using all available target domains.
• Oracle trans DPLR: Conducting differentially private transfer learning based on Algorithm 1, given known transferable source domains.
• Naive LR: Conducting logistic regression using only target domain data.
• Trans LR: Performing transfer learning using transfer learning algorithm and transferable source detection algorithm.

For the six algorithms, we repeated the experiment 100 times and calculated the average error. Simultaneously, we treated the transferable source domain detection as a binary classification problem, and we computed model evaluation metrics for various scenarios.

Based on Figure 3, it is evident that when the differential privacy mechanism was not introduced into the transfer learning framework, the prediction errors of the two algorithms were significantly lower than their differential privacy-enhanced versions. This stark difference highlights the accuracy loss caused by data perturbation (i.e., adding noise) as a means of privacy protection. However, this trade-off is a necessary sacrifice, to ensure privacy, aligning closely with the focus of our study. Within the differential privacy framework, the Oracle trans DPLR algorithm consistently demonstrated optimal performance. This was because the algorithm not only fully leveraged all transferable source data but also excluded potential noise sources, i.e., non-transferable source data, thereby ensuring efficient and accurate knowledge transfer. The trans DPLR algorithm also performed excellently, almost perfectly replicating the performance of the Oracle trans DPLR. This outcome reflects the high effectiveness and accuracy of our proposed transferable source detection algorithm. Further analysis revealed that when the scale of the transferable source set |$\mathcal{A}$| was small, the performance of the all trans DPLR algorithm faced challenges, even performing worse than the naïve DPLR. This phenomenon proves the existence and significant negative impact of negative transfer, further validating the importance and effectiveness of our transferable source detection strategy. As |$\mathcal{A}$| increased, the performance of all trans DPLR gradually improved, eventually reaching a level comparable to Oracle trans DPLR and trans DPLR at |$\mathcal{A}$|= K = 20. Additionally, as the problem complexity h increased, the errors of all four algorithms showed a significant upward trend.

3.3. A Real-Data Study

Our study focuses on the second-hand car market, aiming to accurately predict used car prices through an in-depth analysis of a real dataset containing over 50,000 records and encompassing 31 initial variables. The data preprocessing steps included handling missing values, addressing outliers, performing feature engineering, and conducting feature selection, which expanded the original dataset’s feature variables to 70, significantly enhancing the data foundation for the model.

To optimize the efficiency of applying differential privacy techniques, we normalized the predictor variables and transformed the response variable into a binary form, mapping values exceeding a predefined threshold to 1 and other values to 0, thereby simplifying the complexity of the analysis and meeting specific needs. Descriptive statistical analysis of the data showed a class imbalance in the “bodyType” feature, with the number of luxury sedan samples being the highest (32,291) and the number of concrete mixer samples being the lowest (799). Consequently, the concrete mixer samples were designated as target domain data, while samples of other body types were considered as source domain data.

To evaluate the performance of the transfer learning algorithms, 5-fold cross-validation was used in the experiments. After applying Algorithm 2 to source domain detection across all source domains, it was observed that vehicles with body types 2, 3, 4, and 6 exhibited the least error. Therefore, these body types were selected as transferable source domains for further analysis in subsequent experiments. The evaluation metric used was the misclassification rate of logistic regression, providing an objective measure of model performance (Figure 4).

We used the following algorithm for the subsequent experiment:

• Naive DPLR: Conducting differentially private logistic regression using only target domain data.
• Trans DPLR: Performing differentially private transfer learning using Algorithms 1 and 2.
• All_trans DPLR: Conducting differentially private transfer learning using all available target domains.
• Naive LR: Conducting logistic regression using only target domain data.
• Trans LR: Performing transfer learning using a transfer learning algorithm and a transferable source detection algorithm.

The experimental results indicate that although the introduction of differential privacy had some impact on the accuracy of transfer learning it effectively safeguarded data privacy and the accuracy loss remained within an acceptable range. Notably, the trans DPLR algorithm significantly outperformed the other algorithms in predictive capability for specific body types, highlighting its unique advantage in integrating diverse data sources. However, the overall performance of transfer learning using all source domains was suboptimal, primarily due to the negative effects of non-transferable sources. Nonetheless, the integration of data from different vehicle types with the trans DPLR algorithm significantly enhanced predictive accuracy for specific body types, validating the effectiveness of our transfer learning algorithm in cross-domain data utilization. From a practical perspective, the findings of this study have profound positive implications for the second-hand car market. Accurate prediction of used car prices boosts market participants’ confidence in transactions, fostering healthy and orderly market development; also, the application of differential privacy technology provides a robust protection for user data, meeting international data protection standards such as GDPR. Additionally, this study emphasizes the importance of identifying and eliminating negative transfer, further enhancing the practical effectiveness of the model by precisely selecting transferable source domains. In summary, our experimental results fully demonstrate the superiority of our transfer learning algorithm. Rooted in a function-based differential privacy mechanism and utilizing transferable source domains, this algorithm not only mitigates privacy risks during the transfer process but also simultaneously improves predictive capability within the target domain.

4. Conclusions

We have integrated differential privacy and transfer learning to propose an innovative logistic regression transfer learning method based on differential privacy. In comparison to traditional linear regression, our approach exhibits further extension into the realm of classification. To mitigate potential negative transfer effects, we have introduced a transferable source detection method that operates without the need for algorithms, employing cross-validation for transferable source detection. Our extensive simulation experiments and validation with real-world data robustly confirmed the effectiveness of our algorithm. In summary, our method effectively addresses key challenges in transfer learning, yielding satisfactory results. Future research could delve more deeply into the theoretical foundations, extend this method to other models, explore alternative differential privacy methods combined with transfer learning, and further investigate practical application areas.