Vertically Federated Learning with Correlated Differential Privacy

Abstract

Federated learning (FL) aims to address the challenges of data silos and privacy protection in artificial intelligence. Vertically federated learning (VFL) with independent feature spaces and overlapping ID spaces can capture more knowledge and facilitate model learning. However, VFL has both privacy and utility problems in framework construction. On the one hand, sharing gradients may cause privacy leakage. On the other hand, the increase in participants brings a surge in the feature dimension of the global model, which results in higher computation costs and lower model accuracy. To address these issues, we propose a vertically federated learning algorithm with correlated differential privacy (CRDP-FL) to meet FL systems’ privacy and utility requirements. A privacy-preserved VFL framework is designed based on differential privacy (DP) between organizations with many network edge devices. Meanwhile, feature selection is performed to improve the algorithm’s efficiency and model performance to solve the problem of dimensionality explosion. We also propose a quantitative correlation analysis technique for VFL to reduce the correlated sensitivity and noise injection, balancing the utility decline due to DP protection. We theoretically analyze the privacy level and utility of CRDP-FL. A real vertically federated learning scenario is simulated with personalized settings based on the ISOLET and Breast Cancer datasets to verify the method’s effectiveness in model accuracy, privacy budget, and data correlation.

1. Introduction

Machine learning (ML) has developed rapidly in artificial intelligence (AI) applications, such as computer vision, automatic speech recognition, natural language processing, and recommendation system. Big data plays a particularly prominent role in machine learning, especially deep learning [1,2,3,4,5]. In this context, various countries and institutions are gradually becoming aware of the importance of data ownership and security. To ensure the safe management and use of data, China promulgated the network security law in 2017. The United States and the European Union passed the general data protection regulations, the California consumer privacy law, and other laws to strictly control data collection and transactions [6,7]. In this new legislative environment, it becomes more and more difficult for different organizations to collect and share data. Due to competition, user privacy, data security, complex management procedures, and other reasons, data islands are formed. At present, data islands and privacy protection have become two significant challenges for the development of data-driven artificial intelligence [8,9,10].

Federated learning (FL) is a framework that can effectively help multiple organizations model data use and machine learning while meeting the requirements of user privacy protection, data security, and government regulations [11]. Different organizations or institutions span the fields of health care, banking, finance, and retail and face legal, administrative, or moral constraints, requiring keeping data locally and forming an independent data shaft. At the same time, due to the popularity of Internet of Things technology and wearable edge devices, datasets of different organizations may be obtained by different network edge devices. Because the ID spaces between different data silos may coincide, while the feature spaces are independent, vertically federated learning (VFL) can acquire more modeling knowledge, thus promoting machine learning. At present, this vertically federated learning scenario has attracted the attention of academia and industry [12,13,14]. However, the increase in vertically federated learning participants will lead to a sharp increase in global model dimensions. The vast feature space will lead to an increase in computational complexity and a decrease in model accuracy. In traditional machine learning, feature selection is an essential means to improve the model’s accuracy. However, the existing vertically federated learning algorithm needs more research on the feature selection of the global model.

Moreover, the research shows that parameter transmission may cause privacy disclosure. The client’s private data faces a single attack or collusion attack from server attackers and other malicious clients during federated learning [15]. In the federated framework, the client holds local data for training, while the server only shares local model parameters to protect privacy to a certain extent. However, it has been shown that sharing original local updates damages users’ privacy [11,16]. Much research concerns privacy issues in federated learning [8,17,18,19,20]. Among them, differential privacy (DP) has attracted wide attention because of its solid theoretical guarantee and small communication cost. It provides a privacy guarantee for ML by adding a small amount of noise to the model parameters. Traditional differential privacy techniques rely on a trusted analyzer to collect local updates. However, a trusted third party may not exist. Since it does not rely on a trusted analyzer, local differential privacy (LDP) has been widely concerned [21]. However, research shows that correlated data increases the risk of privacy disclosure. The correlated data in different data sets can be divided into direct and indirect correlations. The direct correlation is defined as two identical records. Compared with direct correlation, indirect correlation is more complex, and the degree of correlation between records needs to be considered. To address the problem, correlated differential privacy (CRDP) has been widely studied [22,23]. Some correlation analysis technologies such as group DP [24], Correlated DP [25], and Bayesian DP [26] have improved the effectiveness of the algorithm at the same privacy level by reducing sensitivity. However, the existing correlated differential privacy technologies only aim at the scenario of centralized data training. There needs to be more research on the impact of correlated data in differentially private federated learning. In particular, in the vertically federated learning scenario, the personalized research on the correlated differential privacy technology, such as selecting the optimal feature set of the global model to balance the impact of data correlation while ensuring accuracy, is noteworthy.

To solve the above problems, we proposed a correlated differentially private federated learning (CRDP-FL) algorithm based on the scenario of vertically federated learning and considering the utility requirements of privacy-preserving federated learning. The vertically federated learning framework is designed for the ubiquitous scenario with many network edge devices and the proposed vertically federated learning through organizations. That is, the data of different network edge devices are uploaded to the organization, and the organizations jointly train a global federated model to ensure privacy. Focusing on privacy disclosure issues caused by parameter transmission and untrusted analyzers, we introduce differential privacy technology into a vertically federated learning framework to protect privacy. We first design the CRDP-FL framework, which contains two layers. The top layer is designed as a decentralized differential privacy-preserving vertically federated learning with N data centers (organizations), each having at least one client (edge device) in the bottom layer. Laplace noise is added to local parameters before communicating to meet the differential privacy protection requirements. The data organizations form a complete graph structure, which can provide bilateral communication. Furthermore, they are decentralized, without a trusted central server in the top layer, and also trustless with each other. Therefore, CRDP-FL further satisfies local differential privacy. The bottom layer of CRDP-FL is client-side data aggregation for VFL between data organizations and injecting DP noise for privacy protection. There is no communication between the bottom-layer clients. A correlated differential privacy utility optimization strategy is proposed to achieve utility optimization of CRDP-FL, including privacy-preserved feature selection and privacy-preserved feature optimization based on correlated analysis for VFL. Aiming at the problem of the global model dimension mentioned above, we perform feature selection to improve the global models’ efficiency performance. In the process, we also introduce DP to protect privacy. A correlated differential privacy utility optimization strategy under the federated framework is also proposed, which provides a quantitative analysis technology between data correlation and dimensions, reduces the correlated sensitivity, and effectively balances the utility decline caused by privacy protection operations. For privacy protection in feature optimization, the DP exponential mechanism is used to obtain the minimum correlated sensitivity with the highest score.

Therefore, the proposed CRDP-FL algorithm can effectively improve the model’s performance in protecting privacy. The contributions of our work are as follows:

• We propose a vertically federated learning algorithm with correlated differential privacy. A general vertically federated learning framework is constructed, collecting data from the client and jointly training the global model between organizations. Furthermore, by injecting DP noise into the VFL framework, CRDL-FL achieves local differential privacy and provides a strong privacy guarantee for VFL.
• To balance the utility and privacy of CRDP-FL, a utility optimization strategy is proposed, including feature selection and optimization for vertically federated learning, thereby improving the algorithm’s efficiency and the global model’s performance. A quantitatively correlated analysis method and correlated sensitivity in VFL (CS-VFL) are proposed to reduce additional noise injecting from DP operation due to data correlation.
• We analyze and verify the algorithm from theoretical and practical aspects. The privacy and utility analysis is performed theoretically. Comperhanserve experiments based on the ISOLET and Breast Cancer datasets demonstrate that CRDP-FL is superior to existing methods in model accuracy, privacy budget, and data correlation.

2. Related Works

2.1. Federated Learning and Privacy Concern

Federated learning addresses the issue of privacy or confidentiality by providing a distributed machine learning model to reduce the risk of exposing training data. From a technical classification perspective, existing federated learning research can be classified by network topology and data distribution features. Based on the network topology, federated learning can be classified into centralized and decentralized federated learning [27]. Although federated learning is fundamentally based on the decentralized data approach, quite a few algorithms still require a central server to collect the trained models from participants in the federated learning framework, build a global model, and share it with all participants. This strategy is mainly implemented by establishing a trusted third party as the server and building trust between the server and the participants. In contrast, decentralized federated learning uses reliability algorithms to replace the reliance on the central server for the model aggregation process. A typical algorithm is an adaptive averaging algorithm based on the Byzantine, which assumes that more than 2/3 of the systems in FL are honest. Through this approach, a group of clients from different domains with a common goal can collaborate, share data, build machine learning models, and take advantage of high accuracy without relying on a third-party centralized server [28,29,30]. According to the data distribution features, federated learning can be divided into three types: federated transfer learning, horizontal federated learning, and vertically federated learning. Federated transfer learning applies to the case where the participants slightly overlap the data samples and data features, such as training a new requirement on a model that has been trained on a similar dataset and used to solve an entirely different problem. Horizontal federated learning applies when the participants’ data have overlapping data features but different data samples. A representative of such algorithms for sample partitioning is federated averaging (FedAvg) [31,32], which derives from the stochastic gradient descent algorithm, also called stochastic parallel gradient descent. Vertically federated learning applies when the participants have overlapping data samples but different data features, and its algorithm has the characteristic of feature partitioning. A representative algorithm of vertically federated learning is distributed block coordinate descent [18]. Assuming that the feature partitioning is given and performs synchronous block updates of variables, it is an algorithm in which participants exchange their respective intermediate values frequently during the training process to train a global model jointly. Due to its high communication cost, many practical studies have reduced its communication cost, such as federated stochastic block coordinate descent (FedBCD) [14]. In addition, Das et al. proposed a new, decentralized stochastic coordinate descent algorithm [33] based on FedBCD. This algorithm is a decentralized two-tier federated learning algorithm and constructs a two-tier federated learning framework. However, as mentioned above, vertically federated learning still suffers from utility problems, such as the vast feature space of the overall model leading to an increase in computational complexity and a decrease in model accuracy. There still needs to be more related research.

Although, each participant in federated learning uses its local data to train the machine learning model, sharing only the model’s updated weight and gradient information with other participants. However, if the data structure is known, then the gradient information may also be exploited, leading to the disclosure of additional information about the training data [16]. Since federated learning requires different participants to upload and aggregate parameters to train the global model repeatedly, this process leads to the leakage of more private information, and personal privacy can still be obtained through model inversion. Therefore, reconstruction attacks are the main privacy concern of federated learning. In addition, during the model inference phase, an adversary may use reverse engineering to obtain additional information about the model to perform model inversion attacks or membership-inference attacks. The adversary in model inversion attacks aims to extract training data or feature vectors of training data from the model, while in membership-inference attacks, the adversary’s goal is to determine whether the training set of the model contains specific samples. Currently, privacy-preserving techniques in federated learning mainly include homomorphic encryption [17], secure multi-party computation [18], and differential privacy [19]. Differential privacy provides a rigorous, quantifiable privacy-preserving method for machine learning, independent of the attack’s background knowledge, and it has received much attention.

In summary, decentralized federated learning is currently more promising because it does not require a trusted third-party institution. Additionally, there are more research results for horizontal federated learning frameworks, while there are still gaps in the research of vertically federated learning frameworks. However, it is more realistic that the feature spaces are independent in data silos of unrelated industries. Therefore, this paper investigates the decentralized vertically federated learning framework and proposes a differential privacy protection solution to address the privacy leakage problem.

2.2. Differential Privacy in Machine Learning

Differential privacy is widely used in machine learning algorithms to counter various attacks and provide privacy by adding noise to the model parameters, all of which have achieved good results. Ref. [34] proposed a privacy-preserving logistic regression algorithm based on differential privacy. They optimized random data, which makes it possible to balance model performance and privacy preservation. They proved that their result satisfies $ϵ$-differential privacy according to the differential privacy definition. Mangasarian et al. [35], respectively, researched privacy-preserving support vector machine models on horizontally and vertically divided datasets. They used randomly generated kernel functions to hide the original learning kernel functions and achieve similar performance to the SVM model without privacy preservation. In addition, differential privacy is widely used for privacy preservation in deep learning and federated learning. Song et al. [36] proposed a differentially private stochastic gradient descent algorithm (DP-SGD), which slices the gradient and adds noise to the gradient during the training process, enabling the trained deep model to have ($ϵ,\delta$)-differential privacy. Many federated learning algorithms based on differential privacy are created and achieve good results [19]. Some personalized differentially private federated learning algorithms have been proposed for complex models [37,38] and non-uniform sampling processing [39].

In conclusion, differential privacy is commonly used in distributed machine learning and federated learning systems due to its computational efficiency and ease of implementation. However, the intake of noise will affect data accuracy and model performance. So balancing privacy and the utility of differential privacy preservation is always an open problem. Therefore, this paper investigates vertically federated learning algorithms of differential privacy preservation and works on improving the algorithm’s utility.

2.3. Correlated Differential Privacy

Differential privacy provides a rigorous mathematical approach to protect privacy by defining indistinguishability, ensuring that the addition or deletion of any individual record does not affect the analysis results. However, studies have shown that when multiple datasets are correlated, it increases the risk of privacy disclosure. At the same level of privacy preservation, correlated data will increase sensitivity, which adds an additional cost of noise and reduces the utility of the data [40,41]. Balancing the privacy and utility of correlated datasets has emerged as a new research hotspot in correlated data privacy techniques. Some studies have focused on correlated measurements to reduce sensitivity. When some users of different datasets have identical records, these datasets are considered as directly correlated. Correlation is strictly defined as two identical records. Unlike direct correlation, indirect correlation is more complex and is defined as two rented records about a user or its correlated users. For example, some information flows about user activities, such as GPS and social network records, may correlate with each other. It is not easy to identify and measure the different degrees of correlation. Zhu et al. [42,43] used the correlation matrix to express the correlation between correlated datasets. By converting global sensitivity to correlated sensitivity and using it as the upper sensitivity limit, it can limit the impact of the correlation. For correlation measurement of data, some uncertain correlation models are proposed, such as the Gaussian correlation model [26,44] and the Markov chain model [45]. Song et al. [46] proposed two mechanisms based on Pufferfish privacy mechanisms using the Markov chain to measure data correlation between adjacent states, which reduced global sensitivity. Zhang et al. [42] reduced global sensitivity by performing feature selection based on correlated sensitivity for datasets with correlated records. Recently, some studies have achieved results on differential privacy preservation for correlated data with unique formats, such as sequential data, tuple data, and trajectory data [47,48,49,50]. In order to balance privacy and utility in the algorithm, some scholars have also proposed correlated differential privacy preservation techniques, which focus on the privacy parameters in the correlated dataset. For example, Wu et al. [51] proposed a definition of correlated differential privacy and provided a fuzzy measurement method of correlation. They dynamically adjusted the level of privacy preservation in the correlated datasets based on Nash equilibrium theory and experimentally verified the effect of privacy parameters for multiple datasets on global data utility. Zhao et al. [52] extended correlated differential privacy to multi-party data release scenarios and performed feature selection through stability algorithms and exponential mechanisms, which reduced data sensitivity and noise intake to improve data utility.

However, the current research only applies to centralized learning scenarios, and there needs to be more research on data correlation in the scenario of joint training of multiple organizations in federated learning. In this paper, we investigate data correlation in the vertically federated learning framework with differential privacy preservation. We balance privacy and utility by reducing correlated sensitivity and noise intake.

3. Preliminary

3.1. Vertically Federated Learning

Datasets owned by different organizations often have different feature spaces for different purposes, but these organizations may share a large user population. We can build a better machine learning model from the heterogeneous data scattered across organizations without exchanging and disclosing private data through vertically federated learning. Each organization has the same identity and status as a participant in this system. Data used in vertically federated learning can be defined as the following equation.

$$F_i\ne F_j,I_i=I_j,\forall D_i,D_j,i\ne j$$

(1)

Here, F represents the feature space of a dataset. I represents the sample ID space. D represents datasets owned by different participants. For dataset D, define M represents the dataset sample space. Then we have the following equation.

$$D\in {\mathbb{R}}^{M\times (I,F)}$$

(2)

For the dataset label space y, different vertically federated learning frameworks will be slightly different. In one situation, only one participant has the label space, and in the other, all participants have the label space. The vertically federated learning framework described in this work is the latter case, where all participants have their own label space. In vertically federated learning, each dataset owns its unique features. Such a dataset can be called feature-partitioned distributed data.

Definition 1 

(Vertically Federated Learning, VFL). A task with N participants training a vertical federated learning model should be defined as the following equation.

$$\underset{\mathrm{\Theta }}{min}\mathcal{L}(\mathrm{\Theta },D;y)\stackrel{\mathrm{\Delta }}{=}\frac{1}{M}\sum _{i=1}^{M}f({\theta }_1,...,{\theta }_N;D^i,y^i)+\lambda \sum _{j=1}^N\omega \left({\theta }_j\right)$$

(3)

where the global model Θ consists of local models of each participant and is defined as the following equation.

$$\mathrm{\Theta }={\{{\theta }_1^T,...,{\theta }_N^T\}}^T$$

(4)

In addition, $f(·)$ represents the loss function, $\omega (·)$ represents the regularization term, and $\lambda$ is the hyperparameter. For general ML models, such as logistic regression (LR), support vector machine (SVM), and neural network (NN), the loss function has the following forms:

$$f({\theta }_1,...,{\theta }_N;X^i,y^i)=f(\sum _{j=1}^N{\theta }_j{X}_j^i,y^i)$$

(5)

The federated random coordinate descent method [14] (FedBCD) is a classical vertically federated learning method in which minibatch based on training extraction is defined as S. For participant k, this part of the data can be defined as $D_k^S$. For formula $\forall i\in S$, we can call the eigenvectors $d_k^i$ and the label vectors $y_k^i$. The intermediate information obtained from the calculation of data and model parameters is as the following equation.

$${\mathrm{\Phi }}_k^S=D_k^S{\theta }_k$$

(6)

For the whole model $\mathrm{\Theta }$, the data used for random gradient descent should be the sum of the intermediate information of all participants, denoted as the following equation.

$${\mathrm{\Phi }}^S=\sum _{k=1}^N{\mathrm{\Phi }}_k^S$$

(7)

Then, for formula $\forall i\in S$ and ${\varphi }^i\in {\mathrm{\Phi }}^S$, we have

$${\varphi }^i=\sum _{k=1}^N{\varphi }_k^i=d_k^i{\theta }_k$$

(8)

The label vector is called $y_k^i$. Let us call $n_S$ the number of samples of S. Then, a part of the random gradient of ${\theta }_k$ can be defined as the following equation.

$$\begin{array}{cc}\hfill g_k(\mathrm{\Theta };S)& ={\nabla }_{k}f(\mathrm{\Theta };S)+\lambda {\nabla }_{\omega }\left({\theta }_k\right)\hfill \\ \hfill & =\frac{1}{n_s}g({\mathrm{\Phi }}^S,y_k^S)+\lambda {\nabla }_{\omega }\left({\theta }_k\right)\hfill \\ \hfill & =\frac{1}{n_s}\sum _{i\in S}\frac{\partial f({\varphi }^i,y_k^i)}{\partial {\varphi }^i}{\left(d_k^i\right)}^T+\lambda {\nabla }_{\omega }\left({\theta }_k\right)\hfill \end{array}$$

(9)

For a participant model, the calculation of ${\nabla }_{k}f(\mathrm{\Theta };S)$ requires using intermediate information computed by other participants. In other words, participant k should broadcast calculated ${\mathrm{\Phi }}_k$ to all models within the framework during each training round and receive intermediate information from other participants simultaneously, which can be defined as the following equation.

$${\mathrm{\Phi }}_{-k}^S={\left\{{\mathrm{\Phi }}_q^S\right\}}_{q\in N,q\ne k}$$

(10)

To highlight the communication process, the part of the random gradient can be further defined as the following equation.

$$\begin{array}{cc}\hfill g_k(\mathrm{\Theta };S)& ={\nabla }_{k}f({\mathrm{\Phi }}_{-k},{\theta }_k;S)+\lambda {\nabla }_{\omega }\left({\theta }_k\right)\hfill \\ \hfill & =g_k({\mathrm{\Phi }}_{-k},{\theta }_k;S)\hfill \end{array}$$

(11)

The random global gradient can be expressed as:

$$g(\mathrm{\Theta };S)={\left\{g_k({\mathrm{\Phi }}_{-k},{\theta }_k;S)\right\}}_{k\in N}$$

(12)

Then, according to Equation (3) and the definition of the random gradient descent method, the learning rate $\eta$ is given. For S agreed upon by each participant, the parameter is updated according to the following equation.

$${\theta }_k\leftarrow {\theta }_k-\eta g_k({\mathrm{\Phi }}_{-k},{\theta }_k;S),\forall k\in N$$

(13)

On this basis, FedBCD sets a local update count R. Each participant can transfer information only once per R round. Each message pass determines the minibatch of R groups, and then the intermediate value $\mathrm{\Phi }$ is uniformly calculated and packaged for broadcast. The iteration of information transfer is called the communication round. Each time the local training takes place, the participant computes a set of intermediate values from the other participants. Suppose that when the training arrives at the t round, the latest communication round is ${\tau }_o$, the parameter update can be determined as the following equation.

$${\theta }_k\leftarrow {\theta }_k-\eta g_k({\mathrm{\Phi }}_{-k},{\theta }_k;S),\forall k\in N$$

(14)

The communication cost savings of FedBCD is evident; however, it still has the risk of privacy disclosure in communication. Introducing the differential privacy method is a reasonable scheme to ensure privacy protection while saving communication costs.

3.2. Differential Privacy

Differential privacy is a concept of privacy proposed by Dwork et al. in 2006 for the privacy disclosure of statistical databases [53,54]. The technique based on differential privacy protection designs a mechanism to add noise to the target database to minimize the loss of statistical information between the published dataset and the original dataset, ensuring that the modification of an individual record in the dataset will not significantly affect the statistical results.

Definition 2 

($ϵ$-Differential Privacy). For any dataset $D_1$ and $D_2$ with at most one different record and any processed dataset $r\in Range\left(M\right)$, the random mechanism satisfies differential privacy if and only if:

$$DP\left(M\right)=\underset{D_1,D_2,S}{sup}log\frac{Pr[r\in S|D_1]}{Pr[r\in S|D_2]}\le ϵ,$$

(15)

where $ϵ$ is called a privacy budget, it is the probability ratio of the random mechanism algorithm M to obtain the same return value on two adjacent datasets, reflecting the privacy protection level that M can provide. The smaller $ϵ$ is, the higher the level of privacy protection is. The value of $ϵ$ must be based on the requirements of specific scenarios to achieve a balance between privacy protection security and data availability.

Definition 3 

(Global Sensitivity). Suppose there is a function $Q:D_i\to {\mathbb{R}}^d$, whose input is a dataset and output is a d-dimensional real vector. For any adjacent datasets $D_1$ and $D_2$, its sensitivity is defined as follows.

$$\mathrm{\Delta }GS=\underset{D_1,D_2}{max}{∥Q\left(D_1\right)-Q\left(D_2\right)∥}_1$$

(16)

Definition 4 

(Random Mechanism). For dataset D, if the output satisfies the distribution of

$$Pr(r\in S|D),$$

(17)

where the random function $M\left(D\right)$ is a random perturbation mechanism on D.

Definition 5 

(Laplacian Mechanism). The Laplace mechanism adds Laplacian noise to the actual output of the function to achieve differential privacy. The mechanism takes dataset D, function Q, and privacy budget ϵ as inputs; it is de-symbolized for functions whose output is actual. For any function $Q:D_i\to {\mathbb{R}}^d$, the following mechanisms provide ϵ-differential privacy.

$$M\left(D\right)=Q\left(D\right)+Laplace\left(\frac{\mathrm{\Delta }GS}{ϵ}\right)$$

(18)

McSherry and Talwar [55] proposed an exponential mechanism that satisfies differential privacy while outputting a near-optimal $t\in T$ according to the utility function.

Definition 6 

(Exponential Mechanism). The indexing mechanism takes dataset D, privacy budget ϵ, and utility function $u:(D\times T)\to {\mathbb{R}}_d$ as inputs. The utility function assigns a score to each output $t\in T$, with a higher score indicating better utility. For any utility function $u:(D\times T)\to {\mathbb{R}}_d$, the algorithm that chooses output t with probability proportional to $exp\left(\frac{ϵu(D,t)}{2\mathrm{\Delta }u}\right)$ satisfies ϵ-differential privacy.

According to the mechanism’s scope, the accumulative privacy budget satisfies the sequential and parallel composition [53,54].

Theorem 1 

(Serial Composition). Assuming that a set of privacy steps $\{M_1,...,M_m\}$ operate sequentially on a dataset, and each $M_i$ provides a ${ϵ}_i$ privacy guarantee, then M satisfies ${\sum }_{i=1}^m{ϵ}_i$-differential privacy.

Theorem 2 

(Parallel Composition). Parallel synthesis. Assuming that a set of privacy steps $\{M_1,...,M_m\}$ operate sequentially over multiple disjoint data subsets, and each $M_i$ provides a ${ϵ}_i$ privacy guarantee, then M satisfies $max\left({ϵ}_i\right)$-differential privacy.

3.3. Correlated Differential Privacy

In single-party data scenarios, the correlated sensitivity has worked well. Under the differential privacy mechanism, as records are only partially correlated, deleting one record may impact other records differently. In the framework of correlated sensitivity, the influence of different intensities is defined as the relevance of records. Different methods can be used to calculate the correlation between records from the correlation data analysis [43]. The standard method calculates the Pearson’s correlation coefficient of records i and j. This coefficient is used to measure the linear dependence between variables, and $\left|w_{ij}\right|\in [0,1]$ represents the degree of correlation between records i and j. When $\left|w_{ij}\right|>0$, there is a specific correlation between record i and j. $\left|w_{ij}\right|=1$ indicates that records i and j are entirely correlated. $\left|w_{ij}\right|=0$ indicates that record i is entirely unrelated to record j. The higher the correlation, the stronger the correlation. At the same time, the correlation matrix $\mathrm{\Delta }$ is used to describe the correlation of a set of data.

$$\mathrm{\Delta }=\left[\begin{array}{cccc}{w}_{11}& w_{21}& ...& w_{n1}\\ w_{21}& w_{22}& ...& w_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ w_{n1}& w_{n2}& ...& w_{nn}\end{array}\right]$$

(19)

To meet the requirements of correlation analysis, the correlation threshold is defined to screen the correlation. For the given correlation threshold $w_0$, the following formula filters the correlation that meets a certain intensity.

$$w_{ij}=\left\{\begin{array}{cc}{w}_{ij}& w_{ij}\ge w_0\\ 0& w_{ij}<w_0\end{array}\right.$$

(20)

The filtered correlation matrix can be used to describe the correlation between the records of a dataset that meet the correlation threshold $w_0$, through which the correlated sensitivity of the dataset can be calculated. The sensitivity of record i can be denoted as $\mathrm{\Delta }C{S}_i$, which the following formula can calculate as the following equation.

$$\mathrm{\Delta }C{S}_i=\sum _{j=0}^l\left|w_{ij}\right|\left({∥Q\left(D^j\right)-Q\left(D^{-j}\right)∥}_1\right)$$

(21)

where $D^j$ represents the dataset with $r_j$ and $D^{-j}$ represents the data set with $r_j$ deleted from D. Then, the correlated sensitivity of a single-party dataset containing q data records is denoted as the following equation.

$$\mathrm{\Delta }C{S}_q=\underset{i\in q}{max}\mathrm{\Delta }C{S}_i$$

(22)

Compared with the correlated sensitivity, the global sensitivity $\mathrm{\Delta }GS$ only measures the maximum number of correlated records without considering the degree of data correlation. Since the correlated sensitivity measures the degree of correlation, it reduces the global sensitivity and the noise for private data.

3.4. Problem Statement

To meet the actual vertically federated learning requirement, we summarize the current issues of vertically federated learning from the framework design, privacy protection, and utility optimization as follows:

• According to the actual scenarios, there are problems such as network transmission performance differences and insufficient computing power due to network edge devices’ performance differences. Therefore, taking small data nodes (devices) as participants of federated learning significantly impacts global model training, especially for deep learning models with big data volumes. Therefore, different organizations should collect clients’ data, and this method is a feasible utility scheme. Usually, various data organizations, such as large medical service institutions, may collect many downstream units or customer data for model training. Data collected by different organizations are limited in application due to data barriers. So it is necessary to federated train a global model for various organizations. In this paper, we design a two-layer vertically federated learning framework to meet the requirements of federated learning from clients to organizations and between organizations.
• As discussed above, federated learning suffered from privacy issues during training, and differential privacy realizes privacy protection by disturbing the parameters in the training process. Especially in the entirely decentralized federated learning framework, local differential privacy technology further satisfies the privacy protection of third-party untrusted institutions. Therefore, we perform a differential private operation for a decentralized vertically federated learning framework by injecting noise into the information transmission and feature selection process.
• Utility optimization is the most critical for privacy-preserving vertically federated learning. On the one hand, increasing participants in vertically federated learning bring a massive surge in global model dimensions. The vast feature space may cause an increase in computational complexity and cost. Moreover, unrelated features result in the utility decreasing of the global model’s accuracy. Therefore, feature selection is necessary for utility optimization in client data aggregation. On the other hand, introducing differential privacy causes utility decline while protecting privacy. In particular, the data correlation will cause additional utility loss. To address these issues, we propose a correlated differential privacy utility optimization strategy, including feature selection and correlation analysis for VFL. Correlated sensitivity in vertically federated learning, i.e., CS-VFL, is defined to reduce sensitivity to improve the algorithm’s effectiveness.

For the given problem, we design vertically federated learning with correlated differential privacy, and the problem should be formalized as follows. We assume that there are N data centers in the vertically federated system, where the data organization j has $K_j$ devices. Then, N data centers jointly train a global model $X\in {\mathbb{R}}^{M\times \left(I,F^{\prime }\right)}$ on data space $X\in {\mathbb{R}}^{M\times (I,F^{\prime })}$ based on the FedBCD algorithm. In intermediate information communication, the privacy protection requirements are satisfied by adding noise, and the process satisfies $ϵ$-differential privacy. In addition, achieving utility by utility optimization mainly refers to optimizing the precision of the global model.

4. CRDP-FL

4.1. Outline

In this paper, we design a general vertically federated learning framework and provide a privacy guarantee. As shown in Figure 1, the proposed CRDP-FL framework contains two layers. The top layer is designed as a decentralized differential privacy-preserving vertically federated learning with N data centers (organizations), each having at least one client (edge device) in the bottom layer. Laplace noise is added to local parameters before communicating to meet the differential privacy protection requirements. The data organizations form a complete graph structure, which can provide bilateral communication. Furthermore, they are decentralized, without a trusted central server in the top layer, and also trustless with each other. Therefore, CRDP-FL further satisfies local differential privacy. The bottom layer of CRDP-FL is client-side data aggregation for VFL between data organizations and also injecting DP noise for privacy protection. There is no communication between the bottom-layer clients. A correlated differential privacy utility optimization strategy is performed to achieve utility optimization of CRDP-FL, including privacy-preserved feature selection and feature optimization based on correlated analysis for VFL.

The core of CRDP-FL, i.e., client-side data aggregation and differentially private vertically federated learning, is described in detail in Section 4.2 and Section 4.3, respectively.

4.2. Client-Side Data Aggregation

Client-side data aggregation in CRDP-FL is the process of aggregating data from the clients to the data organization in the bottom-layer model. In particular, the correlated differential privacy utility optimization strategy is proposed. We perform feature selection for the vertically federated framework to reduce dimensionality and make the algorithm efficient. The features most relent to learning tasks are selected to improve data training accuracy. We also propose correlated sensitivity for vertically federated learning to analyze the quantitative relationship between feature sets and data correlation and achieve utility optimization by reducing noise intake through sensitivity reduction. On this basis, we propose feature optimization to vertically federated learning to determine the optimal feature set. Client-side data aggregation consists of the following steps. Algorithm 1 presents the implementation process of these processes.

• Feature selection. For selecting the most relevant to the learning task, in CRDP-FL, the stability feature selection is performed. The features are divided into the optimal feature set and the adjusted feature set. We also use Pearson’s correlation coefficient to filter out highly correlated features and move one of the correlated features from the optimal feature set to the adjusted feature set. In the process, DP noise is injected for privacy-preserving.
• Correlation analysis. We proposed correlated sensitivity in VFL to quantitative analysis of the relationship between features and data correlations for performing correlation analysis in VFL. To improve the classic correlated sensitivity, we propose feature-oriented correlation to measure the correlation in VFL and mean correlated degree as the standard of the correlated records.
• Feature optimization. Based on the feature selection results, we generate candidate feature sets from the adjusted feature set. The DP exponential mechanism is used for obtaining the minimum correlated sensitivity with the highest score. Then, we add the features from the best candidate feature solutions to the optimal feature set. The final obtained optimal feature set is relaxed from the initial one since reducing the DP noise is essential for balancing privacy and utility in CRDP-FL.
• Data aggregation. Aggregate data into data organizations based on the final optimal feature set.

Algorithm 1: DataAggregation

Input: distributed dataset $D_j$; private budget for Pearson’s correlation ${ϵ}_1$; private budget for feature optimization ${ϵ}_2$; important feature threshold $\theta$; Pearson’s correlation threshold ${\theta }_{per}$; initial $w_0$; adjusting coefficient b Output: optimal $X_j$ 1 FeatureSelection ($D_j$, ${ϵ}_1$, $\theta$, ${\theta }_{per}$), obtain best features set $\alpha$ and adjusted features set $\beta$; 2 CalculateCorrelationThreshold ($D_j$, $w_0$) update $w_0$; 3 FeatureOptimization ($D_j$, ${ϵ}_2$, b, $w_0$, $\beta$, $\alpha$) update $\beta$; 4 Aggregate optimal $X_j$ according to $\beta$; 5 return $X_j$

4.2.1. Feature Selection

In traditional machine learning algorithms, feature selection before training models significantly reduces data dimensionality and improves model accuracy. We propose a differentially private feature selection algorithm shown in Algorithm 2 based on the stability feature selection method [42] and combing Pearson’s correlation coefficient to merge feature sets on feature-partitioned distributed data. The DP noise is injected for privacy protection. Stability feature selection traverses the distributed data feature set and filters out the less correlated features to the prediction. As shown in Equation (23), the important feature score of feature $f_i$ in a distributed dataset is the ratio of the frequency of important features to the number of data subsets. The feature that exceeds the threshold is considered an important feature.

$$S_i=\frac{T_{freq}}{N}$$

(23)

The stability selection algorithm is helpful for overcoming overfitting and understanding the data and for the scenario of client-side data aggregation for utility optimization in this paper. This method is one of the best-performing methods in a multi-party dataset environment. However, the method does not make the important features score 0 because of having similar features and associated features, so more similar and associated important features are retained. To overcome this weakness of stability selection, we combine Pearson’s correlation coefficient calculation in our feature selection algorithm. Equation (24) can calculate the Pearson’s correlation coefficients of features $f_m$ and $f_n$, where ${\mu }_m$ and ${\mu }_n$ are the means of $f_m$ and $f_n$, respectively, and ${\sigma }_m$ and ${\sigma }_n$ are the standard deviations of $f_m$ and $f_n$.

$$P_{m,n}=\frac{E\left[(f_m-{\mu }_m)(f_n-{\mu }_n)\right]}{{\sigma }_m{\sigma }_n}$$

(24)

Definition 7 

(Correlation Coefficient Record Sensitivity). For a query Q, the record sensitivity of Pearson’s correlation coefficient for record i can be defined as follows.

$$\mathrm{\Delta }C{S}_i=\underset{\forall m,n\in p}{max}{∥p_{m,n}-p_{m,n}^{\prime }∥}_1$$

(25)

Definition 8 

(Correlation Coefficient Sensitivity). For a query Q, the sensitivity of the correlation coefficient is determined by the maximum record sensitivity of the correlation coefficient.

$$\mathrm{\Delta }C{S}_{per}=\underset{\forall l\in D_{\beta }}{max}\left(\mathrm{\Delta }C{S}_i\right)$$

(26)

where Q is a query for a set of records Pearson’s correlation coefficients, it is easy to know the correlation coefficient sensitivity $\mathrm{\Delta }C{S}_{per}\le 1$, since the correlation coefficient takes values from 0 to 1.

Algorithm 2: FeatureSelection

4.2.2. CS-VFL

Our proposed CS-VFL is a correlation analysis technique for distributed datasets such as vertically federated frameworks. In CS-VFL, we define feature-oriented correlation, which measures the correlation between records by the coincidence of feature values, and thus provides an intuitive quantitative relationship between feature sets and data correlation. Moreover, by defining the mean correlated degree in VFL as the correlation standard, our method utilizes the a priori knowledge in VFL and thus provides a more objective measure. On these bases, the correlated sensitivity in VFL can be calculated, which effectively reduces the amount of noise due to a tighter correlation threshold than the global sensitivity (GS) and the traditional correlated sensitivity (CS) in the single-party dataset. Since calculated correlation is feature-oriented, it provides a solid foundation for the subsequent feature optimization.

Definition 9 

(Feature-oriented Correlation). Feature-oriented correlation is the record correlation that measures the degree of matching between record i and record j for all features taken.

$$w_{ij}=\frac{match(i,j)}{l}$$

(27)

$$match(i,j)=\left\{\begin{array}{cc}1,& if{v}_i^m=v_j^{m}for\forall m\in l,\\ 0,& otherwise.\end{array}\right.$$

(28)

where $v_i^m$ and $v_j^m$ are the $m_{th}$ feature values of record i and record j.

Definition 10 

(Mean Correlated degree in VFL). For a distributed dataset with n-sided data with feature partitioning, the correlated sensitivity of one of the data subsets is denoted as $\mathrm{\Delta }C{S}_n$, which is the sum of the correlations of the k correlated records in the data subsets, so that the mean correlated degree in the data subsets can be found as $\overline{\mathrm{\Delta }C{S}_n}$, then the distributed mean correlation of the n-party dataset is defined as the following equation.

$$MCD=\frac{1}{N}\sum _{k=1}^N\overline{\mathrm{\Delta }C{S}_n}$$

(29)

$MCD$ is used to measure the average level of correlation strength in a vertically federated framework with feature-partitioned distributed data. Since $w_{ij}\in [0,1]$ and $\overline{\mathrm{\Delta }C{S}_n}\in [0,1]$, the value of $MCD$ is also between 0 and 1. The detailed procedure for computing $MCD$ is shown in Algorithm 3.

Algorithm 3: CalculateCorrelationThreshold

$MCD$ reflects the trend of correlation in the merged dataset. Using the value of $MCD$ as the correlation threshold $w_0$, the correlation between records is marked according to Equation (19), otherwise, the correlation is zero. The distributed correlated sensitivity $\mathrm{\Delta }C{S}_p$ of the merged subset according to Equation (19) and Equation (20) is as the following equation.

$$\mathrm{\Delta }C{S}_p=\underset{i\in p}{max}\sum _{j=0}^i\left|w_{ij}\left({∥Q\left(D_p^j\right)-Q\left(D_p^{-j}\right)∥}_1\right)\right|$$

(30)

where $D_p$ denotes the merged dataset, $w_{ij}$ is the degree of correlation between record i and record j when the dataset differs between only record j, and $\mathrm{\Delta }C{S}_p$ is used to measure the maximum impact of merging data and all records due to changing one record for any query Q. For any query Q, the output of the differential privacy technique perturbation based on the Laplace mechanism can be calculated using Equation (31).

$$\widehat{Q}\left(D_p\right)=Q\left(D_p\right)+Lap\left(\frac{\mathrm{\Delta }C{S}_p}{ϵ}\right)$$

(31)

In summary, our proposed CS-VFL provides a basis for utility optimization in client-side data aggregation. We perform correlation analysis in VFL balancing accuracy changes due to dimensional changes and effectively reduce distributed correlated sensitivity, thereby reducing noise ingestion because the mean correlated degree provides a more stringent correlated sensitivity threshold.

4.2.3. Feature Optimization

Through feature selection, we retain a number of features that are most relevant to the training task. However, considering the inevitable existing data correlation due to data aggregation, removing more features usually leads to a higher degree of data correlation. In the DP mechanism, under the same privacy level, it will introduce additional noise, thereby reducing the utility. Therefore, feature optimization aims at relaxing the optimal feature set by adding a certain number of features. To examine the impact of adding noise and features on model training accuracy, we define the utility function based on correlated sensitivity to verify the effect of reducing data correlation to reduce noise on the accuracy of machine learning algorithms. The utility function is defined based on CS-VFL; therefore, it can minimize the correlated sensitivity. The distributed correlated sensitivity $\mathrm{\Delta }C{S}_p^{c_i}$ of the division scheme $c_i$ can be obtained according to CS-VFL as shown in Equation (30) so that the candidate feature set with the minimum correlated sensitivity can be selected with a high probability, as shown in Equation (32).

$$u_{(D_p,c_i)}=\frac{w_0}{\mathrm{\Delta }C{S}_p^{c_i}}$$

(32)

The exponential mechanism is utilized to select relative feature sets with low data correlation levels based on the utility function, maintaining good utility for VFL. The utility scores of all the candidate sets are known, and the probability of selecting candidate $c_i$ by the exponential mechanism is as follows.

$$\frac{exp\left(\frac{ϵu(D_p,c_i)}{2\mathrm{\Delta }u}\right)}{{\sum }_{c_i\in c}exp\left(\frac{ϵu(D_p,c)}{2\mathrm{\Delta }u}\right)}$$

(33)

The process of feature optimization in CRDP-FL is to generate a certain number of candidate feature sets according to the given adjustment factor b from the adjusting feature set. Then, we calculate the utility score for all candidate feature sets. The exponential mechanism is helpful for selecting the best candidate feature set with the highest utility score. This reduces the correlated sensitivity features and adds to the optimal feature set to improve the utility. The process of feature optimization implementation is shown in Algorithm 4.

Algorithm 4: FeatureOptimization

4.3. Differentially Private VFL

The top layer of our proposed CRDP-FL is a differentially private VFL. FedBCD is chosen as a fundamental training framework, in which each participant jointly trains their local model using coordinate descent through other participants. Eventually, each data organization maintains its local model for subsequent predictions. We introduce differential privacy into the framework to protect privacy.

• Data aggregation. The bottom layer is client-side data aggregation and has been shown in Section 4.2. Since the data organization is semi-trustworthy to the clients it owns, each client will only send the data that is determined to be the optimal set of features and formed after utility optimization to the data organization. The data organization does not know the feature space of the training data until it gets the optimal dataset resulting from the data aggregation process.
• Model initialization. Each organization forms its local data after data aggregation. Then, the data organization initializes the model parameters according to the feature space of local data. Overall, the data owned by each data organization is feature partitioned. Additionally, in federated learning, no exchange of local data occurs. All data organizations maintain a part of the model, so the value calculated by the local model with the local data is also not the predicted value of the whole model but a part of the predicted value, which is called intermediate information. For a sample to be inferred, only the intermediate information computed by all data organizations is accumulated to be the actual predicted value.
• Intermediate information synchronization. The FedBCD algorithm sets a local training round R to reduce communication costs. When the number of iterations is $n\ast R$ (e.g., $0,R,2R...$), information can be passed between data organizations. These rounds are collectively referred to as communication rounds, while other rounds are not available for information transfer. At the beginning of the communication round, each data organization agrees that the next R non-communication rounds are used to update the R-group minibatch, compute the R-group intermediate information and broadcast it, and receive the intermediate information from all other data organizations. These messages are used for the next R rounds of local updates. In addition, since data organizations do not trust each other, differential privacy protection of the intermediate information is required before data organizations broadcast their own intermediate information by perturbing the intermediate information to avoid privacy leakage due to inference attacks.
• Local model updates. Data organizations update the local model according to Equation (14) whether it is an exchange round or not. When performing local updates, the intermediate information from other data organizations is calculated in the most recent synchronization round, except for the latest intermediate information calculated from the part of the model owned by itself. Since the data organization only updates a portion of the parameters belonging to itself by using coordinate descent, it is a biased estimation, which may impact the training accuracy. However, the study by Liu [14] et al. demonstrates that even biased estimates can converge with sufficient training rounds.

Algorithm 5 shows the implementation of the above steps, where line 9 achieves privacy protection by adding Laplace noise. At the same time, the correlated sensitivity is calculated by CS-VFL, thus introducing less noise to achieve utility improvement.

Algorithm 5: CRDP-FL

5. Utility Analysis

5.1. Privacy Analysis

We first analyze which steps of the CRDP-FL proposed in this paper consume the privacy budget and then analyze the privacy level of CRDP-FL. During the client-side data aggregation, two parts consume the privacy budget. One is the differential privacy operation on the merged dataset during the feature selection, and the other is the candidate feature set selection using the exponential mechanism during the feature optimization. In the differentially private VFL, the exchange of intermediate information by data organizations requires differential privacy perturbation and consumes the privacy budget. Therefore, the privacy level of CRDP-FL can be analyzed as follows.

• In the feature selection process, assuming that D and $D^{\prime }$ differ by one data record and $Q(\cdot )$ is a query about the Pearson’s correlation coefficient between any two features of the proximity database, we have the following equation. $M_1(x,Q_1(\cdot ),{ϵ}_1)=Q_1\left(x\right)+lap\left(\frac{\mathrm{\Delta }C{S}_{per}}{{ϵ}_1}\right)$. When x, y denote the adjacent dataset variables, respectively, and the random variable $x\in R$, the adjacent dataset probability density ratio is shown as the following equation.

  $$\frac{p_x\left(z\right)}{p_y\left(z\right)}=\begin{array}{c}\prod _{i=1}^N\frac{exp\left(-\frac{{\epsilon }_1|Q_1{\left(x\right)}_i-z_i|}{\mathrm{\Delta }C{S}_{per}}\right)}{exp\left(-\frac{{\epsilon }_1|Q_1{\left(y\right)}_i-z_i|}{\mathrm{\Delta }C{S}_{per}}\right)}\end{array}\le exp\left({ϵ}_1\right)$$

  (34)
  Thus, the process satisfies ${ϵ}_1$-differential privacy, and the algorithm requires only a small injection of noise since the sensitivity $\mathrm{\Delta }C{S}_{per}\in [0,1]$.
• In the feature optimization process, the candidate feature set is selected by the exponential mechanism. For D and $D^{\prime }$, the privacy level is analyzed as follows.

  $$\frac{\frac{exp\left(\frac{{ϵ}_{2}u(D_p,c_i)}{2\mathrm{\Delta }u}\right)}{{\sum }_{c_i\in c}exp\left(\frac{{ϵ}_{2}u(D_p,c)}{2\mathrm{\Delta }u}\right)}}{\frac{exp\left(\frac{{ϵ}_{2}u(D_p^{\prime },c_i)}{2\mathrm{\Delta }u}\right)}{{\sum }_{c_i\in c}exp\left(\frac{{ϵ}_{2}u(D_p^{\prime },c)}{2\mathrm{\Delta }u}\right)}}=\frac{exp\left(\frac{{ϵ}_{2}u(D_p,c_i)}{2\mathrm{\Delta }u}\right)}{exp\left(\frac{{ϵ}_{2}u(D_p^{\prime },c_i)}{2\mathrm{\Delta }u}\right)}\frac{{\sum }_{c_i\in c}exp\left(\frac{{ϵ}_{2}u(D_p^{\prime },c)}{2\mathrm{\Delta }u}\right)}{{\sum }_{c_i\in c}exp\left(\frac{{ϵ}_{2}u(D_p,c)}{2\mathrm{\Delta }u}\right)}$$

  (35)
  For the left-hand side, the first part can be analyzed as follows.

  $$\frac{exp\left(\frac{{ϵ}_{2}u(D_p,c_i)}{2\mathrm{\Delta }u}\right)}{exp\left(\frac{{ϵ}_{2}u(D_p^{\prime },c_i)}{2\mathrm{\Delta }u}\right)}=exp\left(\frac{{ϵ}_2(u(D_p,c_i)-u(D_p^{\prime },c_i))}{2\mathrm{\Delta }u}\right)\le exp\left(\frac{{ϵ}_2}{2}\right)$$

  (36)
  By symmetry, the other part can be analyzed as follows.

  $$\frac{{\sum }_{c_i\in c}exp\left(\frac{{ϵ}_{2}u(D_p^{\prime },c)}{2\mathrm{\Delta }u}\right)}{{\sum }_{c_i\in c}exp\left(\frac{{ϵ}_{2}u(D_p,c)}{2\mathrm{\Delta }u}\right)}\le exp\left(\frac{{ϵ}_2}{2}\right)$$

  (37)
  Therefore, the feature optimization process for candidate feature set selection by the exponential mechanism satisfies $m{ϵ}_2$-differential privacy, where m is the number of $c_i$. However, it should be noted that the exponential mechanism does not actually add noise and therefore has no effect on the accuracy of the model.
• During the differentially private vertically federated learning, the data organization performs the exchange of intermediate information by adding Laplace noise with a privacy budget of ${ϵ}_3$ each training round. Similar to feature selection, it holds $t{ϵ}_3$-differential privacy.

In summary, since feature selection is operated on a single data center, thus satisfying ${ϵ}_1$-differential privacy, and feature optimization and vertically federated learning are operated on $D_p$, according to Theorem 1 and Theorem 2 that CRDP-FL satisfies $max({ϵ}_1,m{ϵ}_2+t{ϵ}_3)$-differential privacy.

5.2. Correlation Analysis

Our proposed CS-VFL reduces the data correlation with the same number of features, thus improving the utility of differentially private VFL. Since $\mathrm{\Delta }GS$ does not consider correlated degrees, the correlation calculated by $\mathrm{\Delta }GS$ is higher than the weighted correlation $\mathrm{\Delta }CS$. Although, the number of correlation records K is the same for both methods. CS-VFL considers the correlated degree. Furthermore, it provides a more strict standard for the correlation threshold. In distributed datasets, the correlation of local datasets provides a priori knowledge for determining the correlation threshold of the merged dataset, so the selection of $\mathrm{\Delta }C{S}_p$ is more objective than that of $\mathrm{\Delta }CS$. While reflecting the global correlation trend, the correlation threshold of $\mathrm{\Delta }C{S}_p$ is more strict than $\mathrm{\Delta }CS$. Therefore, CS-VFL can reduce correlations more effectively.

Theorem 3. 

For query Q, the distributed correlated sensitivity $\mathrm{\Delta }C{S}_p$ is equal to or less than the correlated sensitivity $\mathrm{\Delta }CS$. Suppose $\mathrm{\Delta }C{S}_p$ and $\mathrm{\Delta }CS$ are correlation analyses about the same set of data, and the same correlation matrix Δ is obtained. Since $C{D}_{Avg}>w_0$ makes the number of correlated records $k\ge k_p$, then $\mathrm{\Delta }C{S}_p\le \mathrm{\Delta }CS$.

Theorem 4. 

When the learning rate of CRDP-FL satisfies 0 < η < $min\{\frac{\sqrt{2}}{2R\sqrt{{\sum }_{i=1}^N{L}_i^2+3L_j^2}},\frac{1}{L}\}$, then for any $T\ge 1$, the following constraint is satisfied constantly.

$$\begin{array}{ccc}\hfill & T\sum _{\tau =0}^{T-1}\mathbb{E}\left[{∥\nabla \mathcal{L}\left({\mathrm{\Theta }}^{\tau }\right)∥}^2\right]\hfill & \\ \hfill & \le \frac{2}{\eta T}(\mathcal{L}\left({\mathrm{\Theta }}^{\left(0\right)}\right)-\mathcal{L}\left({\mathrm{\Theta }}^{*}\right))+2{\eta }^2(N+3)R^2\sum _{n=1}^N{L}_n^2\frac{{\sigma }^2}{S}+2N\frac{{\sigma }^2}{S}\hfill \end{array}$$

(38)

6. Evaluation

6.1. Experimental Setting

To verify the effectiveness of CRDP-FL, we designed comprehensive experiments to measure the impact of the number of different organizations, privacy budget, and feature selection threshold on model accuracy. By comparing with classical algorithms, we verify the advancement of CRDP-FL in model accuracy and data correlation, explore the number of organizations and the threshold value of feature selection, and draw corresponding conclusions. The experimental parameters are set as

Table 1. Experimental Parameters.

Parameters	Symbols	ISOLET	Breast Cancer
Epoch	-	2000	500
Local Training Epoch	R	10	10
Size of Minibatch	-	256	256
Learning Rate	$\eta$	0.01	0.001
Number of Organizations	N	1/4/10–150	1/3/5/7/9/11/13/15
Feature Selection Threshold	$\theta$	0.2/0.4/0.6/0.8	0.3/0.5/0.7
Linear Threshold	${\theta }_{per}$	0.9	0.9
Feature Selection Privacy Budget	${ϵ}_1$	1	1
Feature Optimization Privacy Budget	${ϵ}_2$	1	1
Communication Privacy Budget	${ϵ}_3$	0.2/0.4/0.6/0.8/1	0.2/0.4/0.6/0.8/1
Initial Sensitivity Threshold	$w_0$	0.4	0.4

. The specific experimental settings are as follows.

• Datasets. The common datasets in ML are selected for our experiments. We chose these two datasets because they contain enough feature spaces, which will simulate the federated learning scenarios of feature-partitioned distributed data more realistically. We divide vertically according to the feature space in the experiments. ISOLET [56,57] is a word-and-language phonetic recognition dataset that contains the features of the name of each letter of the English alphabet from 150 volunteers. We retained the records, totaling 7797. The audio data in each entry is quantified as 617 additional features, each of which is a numeric type. Because of its ample feature space, it is suitable for this experiment’s dataset. Breast Cancer [58] consists of 570 entries with 30 features, where M represents malignancy and B represents benignity. It is one of the most commonly used datasets in ML. This dataset contains cytological features of Breast Cancer biopsies that can be used to diagnose breast cancer.
• Training Models. Both of our experiment datasets are textual data that belong to the classification task. Furthermore, the ISOLET dataset is more complex than Breast Cancer. Therefore, in the process of FL, the former uses the full connect neural network (DNN) model for training and inference, and the latter uses the logistic regression (LR) model to classify.
• Comparison Algorithms. We verify the validity of CRDP-FL with different comparison algorithms, including training with a single-party dataset (single), non-private vertically federated learning (Non-private FL) [33], non-private vertically FL with feature selection (FS-FL) [33], and federated learning with a correlation analysis of global sensitivity by calculating the number of correlated records (GDP-FL) [59]. Among them, the training with a single-party dataset refers to non-federated learning. FS-FL is the algorithm that adds feature selection based on non-private vertically federated learning.

6.2. Different Parameter Effect on Accuracy

6.2.1. Organization Number VS Accuracy

This group of experiments reflects the influence of the number of data organizations on accuracy. For different datasets, vertically federated learning without feature selection and communication noise, the impact of the number of organizations on the model accuracy are shown in Figure 2a,b.

In Figure 2a,b, we obtain the highest accuracy when $N=1$, with only a single dataset. When $N>1$, along with the increase in N, the accuracy increases first and then decreases sharply. It can be seen through analysis that in the initial stage, due to too few organizations, the global model is sparse, resulting in a decline in accuracy. Subsequently, with the increase in the number of organizations, the accuracy will rise to a certain extent. However, when the number of organizations gets too large, the data features in a single organization become less, resulting in the scarcity of features shared by local models. At this stage, it is more difficult for local models to capture valuable information, resulting in a decline in accuracy. It can be concluded that in the decentralized FL framework, too few or too many organizations will reduce accuracy. In addition, when the number of organizations is too high, the training time will also rise drastically. Therefore, to balance model accuracy and training time, the following experiments set the number of organizations to 4 for ISOLET and 3 for Breast Cancer datasets. Because of the ample feature space of ISOLET, a more significant number of organizations can be set when configuring the experimental environment. However, when the number of tissue N reaches more than 150, the model’s performance decreases severely, and the training time rises sharply. Hence, we set the maximum number of organizations at 150.

6.2.2. Feature Selection Threshold

This group of experiments reflects the impact of feature selection thresholds on accuracy. We analyze the optimal range of feature selection thresholds through these experiments. As shown in Figure 3a,b, feature selection thresholds on different datasets affect the accuracy of the algorithms FS-FL, CRDP-FL, and GDP-FL. As the threshold of feature selection increases, all the algorithms experience a process of increasing and then decreasing in accuracy, with the highest accuracy reaching between 0.4 and 0.5. For FS-FL, too low a threshold causes uncorrelated features not to be removed entirely, and too high a threshold yields some critical features to be filtered out, resulting in a state of high, middle and low sides. In addition, for CRDP-FL and GDP-FL, when the threshold of feature selection is higher, the decrease in accuracy is much more significant than that of FS-FL. For the noise-adding algorithm, too high a feature selection threshold will increase correlation, thus introducing additional noise, resulting in a further decline in accuracy. Therefore, for CRDP-FL and similar algorithms, an appropriate feature selection threshold is necessary for satisfactory model accuracy.

6.3. Accuracy Evaluation

To verify the accuracy of our CRDP-FL, we set up a group of comparative experiments of model accuracy under different privacy budgets on the ISOLET and Breast Cancer datasets. In this group of experiments, we only consider the influence of the change of ${ϵ}_3$ on the accuracy. Because the sensitivity in the feature selection process is small, the amount of noise is tiny and will not significantly impact the accuracy. At the same time, ${ϵ}_2$ is an exponential mechanism, and there will be no noise injection.

The experimental results are shown in Figure 4a,b, and the performance of the non-private algorithms (Single, Non-private, and FS-FL) is better than the other two private comparison algorithms on different datasets. It is because privacy budgets (${ϵ}_1$, ${ϵ}_2$, and ${ϵ}_3$) do not affect Single, Non-private FL and FS-FL since they are only effective for noise introducing. Non-private and FS-FL followed closely, and the performance of the feature-selected algorithm FS-FL was better than that of the general Non-private algorithm. The results show that feature selection can optimize the utility of vertically FL. Among the private algorithms, CRDP-FL is significantly better than the algorithm GDP-FL under various privacy budget conditions. The correlated sensitivity calculated by the CRDP-FL is lower than the GDP-FL. Therefore, the feature selection by the CRDP-FL further improves the model’s accuracy. Therefore, our algorithm has significant advantages for utility improvement.

To further verify our proposed CS-VFL, we set up a group of experiments to compare the different definitions of correlated sensitivity on different datasets under different feature selection thresholds. We also analyze the change in the correlated sensitivity under different feature selection thresholds. As shown in Figure 5a,b, the correlated sensitivity of CRDP-FL is much smaller than GDP-FL on different datasets. In addition to the advantages of the correlation calculation mentioned for CRDP-FL, the candidate feature set added to the feature optimization process also plays a role. The feature optimization process selects a set of candidate feature sets with the lowest correlation to join the best feature set, which also plays a crucial role in reducing data correlation and sensitivity. In addition, as the threshold of feature selection increases, the feature space of data collected by each data organization becomes smaller, increasing data correlation and sensitivity. Thus, in the intermediate information communication stage, more noise is required, resulting in decreased data utility, so the feature selection threshold should not be too high.

According to the results of the above comprehensive experiment, we can draw the following conclusions. The number of data organizations and the feature selection threshold affect the accuracy, and we can determine the best precision setting through experiments. Regarding accuracy performance, CRDP-FL effectively improves the model accuracy of FL from two aspects. For multiple participants, feature selection is an effective means to improve effectiveness. Furthermore, while providing privacy, our method effectively improves the model accuracy through correlation analysis based on CS-VFL by reducing the additional loss of utility.

7. Conclusions

According to the ubiquitous vertically federated learning scenario, and to break data silos and solve the privacy problem in federated learning, we propose a vertically federated learning algorithm with correlated differential privacy in this paper. We design a decentralized two-layer vertically federated learning framework for different organizations with independent feature space and scenarios with multiple clients downstream of different organizations. A privacy-preserving strategy based on differential privacy is proposed to be integrated into the framework. Moreover, in view of the utility loss caused by large feature space and the possible additional utility loss of correlated data to the application of differential privacy technology in vertically federated scenarios, the solutions of feature selection, correlated sensitivity analysis, and feature optimization are proposed. For the differentially private federated learning, the improving correlation analysis technology presented here can reduce the correlated sensitivity and improve the model accuracy. Comprehensive experiments on ISOLET and Breast Cancer datasets show that feature selection of this method can improve model accuracy, especially for large datasets of the feature set.