Privacy-Preserving Distributed Deep Learning via Homomorphic Re-Encryption

Abstract

The flourishing deep learning on distributed training datasets arouses worry about data privacy. The recent work related to privacy-preserving distributed deep learning is based on the assumption that the server and any learning participant do not collude. Once they collude, the server could decrypt and get data of all learning participants. Moreover, since the private keys of all learning participants are the same, a learning participant must connect to the server via a distinct TLS/SSL secure channel to avoid leaking data to other learning participants. To fix these problems, we propose a privacy-preserving distributed deep learning scheme with the following improvements: (1) no information is leaked to the server even if any learning participant colludes with the server; (2) learning participants do not need different secure channels to communicate with the server; and (3) the deep learning model accuracy is higher. We achieve them by introducing a key transform server and using homomorphic re-encryption in asynchronous stochastic gradient descent applied to deep learning. We show that our scheme adds tolerable communication cost to the deep learning system, but achieves more security properties. The computational cost of learning participants is similar. Overall, our scheme is a more secure and more accurate deep learning scheme for distributed learning participants.

1. Introduction

1.1. Background

In recent years, artificial intelligence (AI) [1] has been applied to more and more fields, such as medical treatment [2], internet of things (IoT) [3] and industrial control [4]. Actually, deep learning [5] is one of the most attractive and representative techniques of AI, which is mainly based on neural network [6]. Thanks to the development of related computer hardware (such as GPU) and the emergence of big data [7], deep learning has achieved striking success in tasks such as image classification [8], traffic identification [9] and self-learning [10]. Therefore deep learning is gaining increasing importance in our more and more intelligent modern society.

Distributed deep learning is gaining increasing popularity nowadays. In this kind of deep learning, the training datasets are collected from multiple distributed data providers, rather than a single one [11]. When the deep learning model is trained with more representative data items, the obtained model could be more generalized, which leads to higher model accuracy. However, the collection and utilization of distributed datasets raise worrying security issues (especially privacy issues), which hinders wider application of distributed deep learning to some extent [12].

There are many examples of privacy leak, which may incur significant threats. For example, in 2018, it was reported that a data analytics company called Cambridge Analytica harvested millions of profiles of voters which revealed voters’ privacy. The data breach influenced choices of the voters which was a threat of the fairness of election. Another real live case is that the Reuters Health recently reported that some health applications in the mobile phone might share information with a host of unrelated companies, some of which have nothing to do with healthcare. A big concern of users is how their data will be used and by whom [13].

With data providers sending more and more data to cloud servers because of the cloud servers owning high computational capability and large storage to process big data, it is reasonable that data providers may worry about their data privacy, even when they encrypt their data before sending their data out [14].

Recently, Aono and Hayashi [15] presented a privacy-preserving deep learning scheme via additively homomorphic encryption. It is one of the most representative works in privacy-preserving deep learning. Their scheme is based on gradients-encrypted asynchronous stochastic gradient descent (ASGD), in combination with learning with errors (LWE)-based encryption and Paillier encryption. They first prove that sharing gradients even partially over an honest-but-curious parameter cloud server as in [16] may leak information. Then they propose an additively homomorphic encryption based scheme, in which all learning participants encrypt their own computed gradients with the same public key and send them to the server. However, the security of the scheme is based on the assumption that the server could not collude with any learning participant.

We analyze potential risks of the system in [15] as follows. (1) Since the learning participants own the same public key and private key, learning participant A could decrypt the gradients of learning participant B as long as it gets B’s encrypted gradients. (2) To make matters worse, it is likely that the server colludes with one of the participants in practice. Once the server and the learning participant collude, they could decrypt the gradients of all learning participants because the server has all encrypted gradients and the learning participant has the private key. According to [15], they could get the private local data of all learning participants through the gradients.

In our scheme, all parties (a cloud server acts as key transform server, a cloud server acts as data service provider, and data providers act as learning participants) are assumed to be honest-but-curious [17]. In other words, they will finish given tasks, but may try to mouse some sensitive information, such as the local data of data providers. We assume that the two servers, key transform server (KTS) and data service provider (DSP), are selected from different cloud companies and they would not collude with each other because of benefit contradiction (different companies take their own benefit as first consideration) and reputation preservation; however, a server may collude with a data provider.

There are two main colluding scenarios between the server and the learning participant. First, the server and a learning participant may belong to the same company. That is to say, they may collude with each other for the common benefit of their company. Specifically, the cloud company a server belongs to may send an entity to be a learning participant. But the two cloud servers are selected from different companies, so they are unlikely to collude with each other for the sake of their companies’ reputation and beneficial contradiction. Second, even if the cloud company a server belongs to cannot send an entity to be a learning participant, it is easier for a cloud sever to corrupt a learning participant than another cloud server. Because the security measures of cloud sever is more than that of a learning participant in general.

In a word, the cloud server is likely to collude with a learning participant in [15]; and it is reasonable for us to assume that the key transform server would not collude with the data service provider in this paper.

1.2. Our Contributions

Due to the security vulnerabilities of the scheme proposed in [15], we propose a multi-key based distributed deep learning scheme to protect the data privacy of learning participants even when a server colludes with one of the learning participants, using homomorphic re-encryption. We introduce a key transform server to re-encrypt the gradients encrypted by learning participants. The data service provider makes the re-encrypted gradients additively homomorphic and finishes the weights update computation. Finally, the learning participants download new weights and decrypt them respectively. The detailed realization steps of our scheme are described in Section 4. In a word, our scheme enjoys the following properties on security, efficiency and accuracy.

• Security: our scheme protects the private local data of learning participants even without the assumption that the server would not collude with any learning participant.
• Efficiency: experimental results show that the computational cost of learning participants are similar to that in [15].
• Accuracy: our scheme provides a little higher accuracy than that in [15].

1.3. More Related Works

Shokri et al. [16] have designed a distributed deep learning system that multiple learning participants could jointly train an ASGD based deep neural network model without sharing their local datasets, but must selectively sharing key parameters of the model. The goal of our paper is designing an ASGD-based distributed deep learning system which do not need sharing key parameters of the model.

Papernot et al. [18] proposed a scheme to preserve the privacy of training data called private aggregation of teacher ensembles (PATE). They used “teachers” for a “student” model instead of public models to protect sensitive data. Essentially, this property of the scheme can be called differential privacy.

NhatHai Phan et al. [19] proposed the deep private auto-encoder (dPA), which was one type of deep learning. Instead of perturbing the result of deep learning, the scheme focuses on the perturbation of objective function of the deep auto-encoder, to realize differential privacy.

Abadi et al. [20] have developed a framework of differential privacy to analyze the privacy cost of crowdsourcing the training of model over large dataset containing sensitive data. They also designed algorithmic techniques for machine learning under modest cost of privacy, computation, efficiency and accuracy.

Hitaj et al. [21] showed that distributed deep learning was susceptible to an attack they devised. They trained a generative adversarial network (GAN) generating samples that came from the same distribution as original training dataset, which should be kept private. Moreover, they showed that existing record-level differential privacy could not resist their attack. Therefore effective method is needed for privacy-preserving distributed collaborative deep learning. They consider one of the learning participants as the adversary, which is practical. We consider the situation in this paper as well.

Mohassel et al. [22] proposed privacy-preserving machine learning protocols for logistic regression, linear regression and stochastic gradient descent based neural network training. In their schemes, the data providers are distributed and there are two servers. Their schemes are based on the assumption that two servers would not collude, which is reasonable in practice because of beneficial conflict. Our system is based on this assumption, too.

Ping Li et al. [23] suggested that distributed deep learning over combined dataset should pay attention to two points. First, all data including intermediate computation results should be encrypted with different keys before being sent out. Second, the computational cost of data providers should be minimal. We fully consider these two points when we design our scheme, so that the data providers can be mobile smart phones. These authors proposed a framework for privacy-preserving outsourced classification in cloud computing (POCC) in [24].

Qingchen Zhang et al. [25] pointed out that offloading some expensive operations of big data feature learning to the cloud server(s) could improve the system efficiency. Meanwhile, the data privacy of enterprises and governments should be concerned. They approximated the activation function as a polynomial function with the Brakerski-Gentry-Vaikuntanathan cryptosystem (BGV). Our scheme do not need the approximation to avoid accuracy decrease.

BD Rouhani et al. [26] proposed a scalable provably-secure deep learning framework called Deepsecure. In this framework, all parties are likely to leak information. The key of their framework is the pre-processing techniques and optimized Yao’s Garbled Circuit protocol [27]. Our scheme considers the situation that one of the learning participants would leak information even its private key.

1.4. Paper Organization and Notations

The rest of the paper is organized as follows. We introduce the definitions of homomorphic re-encryption, ASGD-based deep learning and illustrate that gradients may leak information in Section 2, which are preliminaries of our system. The architecture of our system is proposed in Section 3. The details of realizing our system via proxy-invisible homomorphic re-encryption is given in Section 4. Then we perform security analysis of our system in Section 5. Furthermore, we analyze the communication cost of our system and evaluate its computational cost with experiments in Section 6. Finally, we conclude this paper in Section 7.

To facilitate presentation, we summarize the main notations used in this paper in

Table 1. Notations.

Symbols	Description
${\left[m_i\right]}_k$	The ciphertext of $m_i$ encrypted with key k;
$\alpha$	The learning rate;
N	The number of learning participants;
$N_{wu}$	The number of weights update process;
$Len(*)$	The bit length of input data;
$H(*)$	The hash value of input data;
$\left(k,p,q\right)\to \left(g,n\right)$	Function with the input $k,p,q$ and the output $g,n$;
$p·q$	The product of p and q.

.

2. Preliminaries

2.1. Homomorphic Re-Encryption

Homomorphic re-encryption scheme (HRES) is an asymmetric cryptography that can realize proxy-invisible re-encryption and supports privacy-preserving data processing with access control. In addition, the addition scheme of an improved version of HRES, called the somewhat re-encryption scheme, has the properties of additive homomorphism [28]. There are four roles in this scheme: data providers (DPs), data service provider (DSP), access control server (ACS), and data requesters (DRs). As the names imply, the DPs provide data to ACS; and ACS transforms the received data to realize access control; then DSP processes the transformed data; finally, the DRs request and get data they need. Next, we introduce this improved homomorphic re-encryption scheme in detail which mainly consists of the following five algorithms.

• Key generation (KeyGen): $\left(k,p,q\right)\to \left(g,n,PK\right)$. First choose k as a parameter, and two large primes p and q, where $Len\left(p\right)=Len\left(q\right)=k$. The outputs of this algorithm, denoted as $\left(g,n,PK\right)$, are public system parameters. Determine a generator g of $\mathbb{G}$ with maximal order [29], where $\mathbb{G}$ is the cyclic group of quadratic residues modulo $n^2$. Then compute $n=p·q$. The two servers, DSP and access control server (ACS), respectively generate key pairs: ($s{k}_{DSP}=a,p{k}_{DSP}=g^a$) and ($s{k}_{ACS}=b,p{k}_{ACS}=g^b$). Then they negotiate the Diffie–Hellman key:

  $$PK=p{k}_{DSP}^{s{k}_{ACS}}=p{k}_{ACS}^{s{k}_{DSP}}=g^{a·b}mod{n}^2.$$

  (1)

  Then $PK$ is published to all data providers for them to encrypt their data. Each data provider generates its own key pair. For example, the key pair of data provider i is $\left(s{k}_i,p{k}_i\right)=(k_i,g^{k_i})$.
• Encryption (Enc): $m_j\to {\left[m_j\right]}_{PK}=\left\{T_j,{T_j}^{\prime }\right\}$. This algorithm is performed by data providers. Its function is encrypting plaintext $m_j$ ($m_j\in {\mathbb{Z}}_n$) provided by data provider i under the Diffie–Hellman key $PK$. The resulting ciphertext can be represented by two parts:

  $$\left(1\right)T_j=\left(1+m_j·n\right)·P{K}^{r}mod{n}^2$$

  (2)

  and

  $$\left(2\right){T_j}^{\prime }=g^{r}mod{n}^2,$$

  (3)

  where $r\in \left[1,\frac{n}{4}\right]$ is first selected randomly.
• First phase of re-encryption (FPRE): ${\left[m_j\right]}_{PK}\to {\left[m_j\right]}^{+}$. After receiving ${\left[m_j\right]}_{PK}$ from data provider i, DSP first selects a computation identifier CID, and then implements the following algorithms to compute the intermediate ciphertext ${\left[m_j\right]}^{+}$:

  $$\left(1\right)h_1=H({\left(p{k}_i\right)}^{s{k}_{DSP}}\left|\right|CID);$$

  (4)

  $$\begin{array}{cc}\hfill \left(2\right){\left[m_j\right]}^{+}& =\left\{\widehat{T_j},\widehat{{T_j}^{\prime }}\right\}\hfill \\ \hfill & =\left\{T_j,{\left({T_j}^{\prime }\right)}^{s{k}_{DSP}}·g^{h_1}\right\}.\hfill \end{array}$$

  (5)
• Second phase of re-encryption (SPRE): ${\left[m_j\right]}^{+}\to {\left[m_j\right]}_{p{k}_i}$. After receiving ${\left[m_j\right]}^{+}$ from DSP, ACS performs the following algorithms to compute ${\left[m_j\right]}_{p{k}_i}$:

  $$\left(1\right)h_2=H({\left(p{k}_i\right)}^{s{k}_{ACS}}\left|\right|CID);$$

  (6)

  $$\begin{array}{cc}\hfill \left(2\right){\left[m_j\right]}_{p{k}_i}=& \left\{\overline{T_j},\overline{{T_j}^{\prime }}\right\}\hfill \\ \hfill =& \left\{\widehat{T_j},{\left(\widehat{{T_j}^{\prime }}\right)}^{s{k}_{ACS}}·g^{h_2}\right\}.\hfill \end{array}$$

  (7)

  Here, the final ciphertext ${\left[m_j\right]}_{p{k}_i}$ is obtained.
• Decryption (Dec): ${\left[m_j\right]}_{p{k}_i}\to m_j$. Only when the data requester of ${\left[m_j\right]}_{p{k}_i}$ has the corresponding private key $s{k}_i$, the decryption could be finished correctly. The decryption algorithms are as follows:

  $$\begin{array}{cc}\hfill \left(1\right){h_1}^{\prime }=& H\left({\left(p{k}_{DSP}\right)}^{s{k}_i}\left|\right|CID\right)\hfill \\ \hfill =& H\left(g^{a·s{k}_i}\right|\left|CID\right)\hfill \\ \hfill =& h_1;\hfill \end{array}$$

  (8)

  $$\begin{array}{cc}\hfill \left(2\right){h_2}^{\prime }=& H\left({\left(p{k}_{ACS}\right)}^{s{k}_i}\left|\right|CID\right)\hfill \\ \hfill =& H\left(g^{b·s{k}_i}\right|\left|CID\right)\hfill \\ \hfill =& h_2;\hfill \end{array}$$

  (9)

  $$\begin{array}{cc}\hfill \left(3\right)m_j=& L(\overline{T_j}·p{k}_{ACS}^{{h_1}^{\prime }}·g^{{h_2}^{\prime }}/\overline{{T_j}^{\prime }}mod{n}^2),where\hfill \\ \hfill L\left(x\right)=& (x-1)/n.\hfill \end{array}$$

  (10)

  The computation identifier CID is set to be addition here. Because only addition in HRES is used in this paper, we omit CID in the rest of this paper. According to [28], the properties of the above improved homomorphic re-encryption scheme can be summarized as follows.
  (1) Additive homomorphism:

  $${\left[m_1+m_2\right]}_{p{k}_i}={\left[m_1\right]}_{p{k}_i}\ast {\left[m_2\right]}_{p{k}_i}.$$

  (11)

  Because the proof of additive homomorphism property of HRES is not provided in [28] and this property is important in our scheme, we prove it in this paper as follows.

  Proof Proof of additive homomorphism.

  $$\begin{array}{cc}\hfill left=& {\left[m_1+m_2\right]}_{p{k}_i}\hfill \\ \hfill =& \{{\overline{T}}_{1+2},{\overline{T}}_{1+2}^{\prime }\}\hfill \\ \hfill =& \left\{{\widehat{T}}_{1+2},{\left({\widehat{T}}_{1+2}^{\prime }\right)}^{s{k}_{ACS}}·g^{h_2}\right\}\hfill \\ \hfill =& \left\{T_{1+2},{\left({\left(T_{1+2}^{\prime }\right)}^{s{k}_{DSP}}·g^{h_1}\right)}^{s{k}_{ACS}}·g^{h_2}\right\}\hfill \\ \hfill =& \left\{T_{1+2},{\left({\left(T_{1+2}^{\prime }\right)}^a·g^{h_1}\right)}^b·g^{h_2}\right\}\hfill \\ \hfill =& \left\{T_{1+2},{\left(T_{1+2}^{\prime }\right)}^{ab}·g^{h_1·b+h_2}\right\},\hfill \end{array}$$

  (12)

  where

  $$\begin{array}{cc}\hfill T_{1+2}=& [1+(m_1+m_2)·n]·P{K}^{r}mod{n}^2\hfill \\ \hfill =& [1+(m_1+m_2)·n]·g^{abr}mod{n}^2;\hfill \end{array}$$

  (13)

  $$\begin{array}{cc}\hfill {\left(T_{1+2}^{\prime }\right)}^{ab}·g^{h_1·b+h_2}=& {\left(g^{r}mod{n}^2\right)}^{ab}·g^{h_1·b+h_2}\hfill \\ \hfill =& g^{rab+h_1·b+h_2}mod{n}^2.\hfill \end{array}$$

  (14)

  and

  $$\begin{array}{cc}\hfill right=& {\left[m_1\right]}_{p{k}_i}\ast {\left[m_2\right]}_{p{k}_i}\hfill \\ \hfill =& \left\{\overline{T_1}·\overline{T_2},{\overline{T_1}}^{\prime }·{\overline{T_2}}^{\prime }\right\},\hfill \end{array}$$

  (15)

  where

  $$\begin{array}{cc}\hfill \overline{T_1}·\overline{T_2}=& [\left(1+m_1·n\right)·P{K}^{r}mod{n}^2]·\hfill \\ \hfill & [\left(1+m_2·n\right)·P{K}^{r}mod{n}^2]\hfill \\ \hfill =& [1+(m_1+m_2)·n+m_1{m}_2{n}^2]·P{K}^{2r}mod{n}^2\hfill \\ \hfill =& [1+(m_1+m_2)·n]·P{K}^{2r}mod{n}^2\hfill \\ \hfill =& [1+(m_1+m_2)·n]·g^{2abr}mod{n}^2;\hfill \end{array}$$

  (16)

  $$\begin{array}{cc}\hfill {\overline{T_1}}^{\prime }·{\overline{T_2}}^{\prime }=& {\left\{{[{\left(g^{r}mod{n}^2\right)}^{s{k}_{DSP}}·g^{h_1}]}^{s{k}_{ACS}}·g^{h_2}\right\}}^2\hfill \\ \hfill =& {\left\{{[{\left(g^{r}mod{n}^2\right)}^a·g^{h_1}]}^b·g^{h_2}\right\}}^2\hfill \\ \hfill =& {[\left(g^{rab}mod{n}^2\right)·g^{h_1·b+h_2}]}^2\hfill \\ \hfill =& g^{2(rab+h_1·b+h_2)}mod{n}^2.\hfill \end{array}$$

  (17)

  Because g is the generator with maximal order, we have $T_{1+2}=\overline{T_1}·\overline{T_2}$, ${\left(T_{1+2}^{\prime }\right)}^{ab}·g^{h_1·b+h_2}={\overline{T_1}}^{\prime }·{\overline{T_2}}^{\prime }$. Therefore $left=right$, and HRES has the additive homomorphism property. □

  (2)

  $${\left[t·m\right]}_{p{k}_i}={({\left[m\right]}_{p{k}_i})}^t.$$

  (18)

  (3)

  $${\left[-m\right]}_{p{k}_i}={({\left[m\right]}_{p{k}_i})}^{n-1}.$$

  (19)

  (4) Resistant to impersonation attack and collusion between any server and any distributed data provider thanks to the two hash values $h_1,h_2$. In order to avoid repetition, here we omit the proof of the last three properties. The details can be found in [28].

2.2. ASGD Based Deep Learning

Deep learning can be seen as a series of algorithms over a neural network consisting of multiple layers. There are numerous neuron nodes in each layer. The neurons of one layer are connected with neurons of next layer via weight variables. The weight variables are important for activation function which computes the output of one layer. For example, the output of layer $p+1$ is computed as $ou{t}^{(p+1)}=f(W^{\left(p\right)}·ou{t}^{\left(p\right)}+b^{\left(p\right)})$, where $f\left(x\right)$ is the activation function, and $(W^{\left(p\right)},b^{\left(p\right)})$ denotes the weight vector connecting layer p with layer $p+1$.

The weight vector of the deep neural network consisting of weight variables need to be determined through deep learning. Concretely, take supervised learning for example, where a training dataset is provided firstly, and the cost function J is defined according to the target of the learning task. Then cost function is computed over all data items of the given training dataset. The learning process minimizes J by adjusting values of weight variables.

The most frequently used adjusting method called the stochastic gradient descent (SGD). Denote the weight vector as $W=(w_1,w_2,\dots ,w_n)$, which consists of all weights in the deep neural network. Generally, the cost function is computed iteratively over different subsets (mini-batch) of training dataset, in which elements are selected randomly. For example, the computation result of cost function over a subset consisting of t elements can be denoted as $J_{\left|batch\right|=t}$. Then the gradient vector G of cost function J can be represented as

$$G=\left(\frac{\delta J_{\left|batch\right|=t}}{\delta w_1},\frac{\delta J_{\left|batch\right|=t}}{\delta w_2},\dots ,\frac{\delta J_{\left|batch\right|=t}}{\delta w_n}\right).$$

(20)

In the learning process using SGD, the update rule for weight vector is

$$W:=W-\alpha ·G,$$

(21)

where $\alpha \in \mathbb{R}$ is called learning rate.

In order to make the learning process more efficient, practical asynchronous stochastic gradient descent (ASGD) is proposed in [30], including data parallelism and model parallelism. Data parallelism means the training dataset for updating weight vector can be distributed. That is to say, the dataset consists of data from multiple distributed data providers. Every machine used for training model has a replica of all weights. Model parallelism is separating the model into several parts which can be updated by different machines. A typical example of model parallelism is depicted in Figure 1, which is proposed in [11]. There are $N=4$ learning participants included in four machines (represented by four blue rectangles with dotted lines as boundaries) in this example. The five-layer deep neural network model is separated into $N=4$ parts. The thick lines connecting nodes in different rectangles are called crossing weights, which are responsible for collecting different parts of models trained in different machines. The model is usually separated according to the computational capability of machines.

In model parallelism, the weight variables in the deep neural network can be represented as $W=(W_1,W_2,\dots ,W_N)$. Each component of W, the $W_i(i=1,2,\dots ,N)$ is a weight vector. So the update rule can be transformed to

$$W_i:=W_i-\alpha ·G_i,i=1,2,\dots ,N;$$

(22)

where $W_i$ contains the weight variables of i th part of the model updated by i th machine, $G_i$ means the gradient vector used for updating weights of i th part of the model, i.e., $W_i$.

We adopt model parallelism in our system. The learning participant i, KTS and computing unit $C{U}_i$ of DSP together act as machine i in Figure 1. Machine i is responsible for training the i th part of model assigned to it. That is to say, $W_i$ is updated by the local data of learning participant i. If the weight variable w connects nodes in machine i and machine k, and machine k is responsible for updating w, then the gradient used for updating w should be generated by learning participant k and should be re-encrypted with the public key $p{k}_k$ of learning participant k so that machine k can further process the gradient.

2.3. Gradients Leaking Information

Authors in [15] have shown that a small portion of gradients could leak the data providers’ original training data. They proved that by listing four examples, including the simplest situation that there was only one neuron, the general neural networks, the neural networks with regularization and the most complex one, with Laplace noises added to the gradients. Here we restate the simplest example.

We focus on the learning process of one single neuron. Suppose $(\overrightarrow{x},y)$ is the data item, where $\overrightarrow{x}=\left\{x_1,x_2,\dots ,x_n\right\}$ is the neuron input vector and y is the corresponding truth label. The cost function here is set as the distance between the y and $y_{predict}$, $y_{predict}$ is computed through the activation function f fed with $\overrightarrow{x}$

$$y_{predict}=f(\sum _{i=1}^n{W}_i·x_i+b),$$

(23)

where b is the bias value. Hence the cost function is

$$J(W,b,\overrightarrow{x},y)={(y_{predict}-y)}^2.$$

(24)

Then the gradients can be computed as follows:

$$\begin{array}{cc}\hfill g_i& =\frac{\partial J(W,b,\overrightarrow{x},y)}{\partial W_i}\hfill \\ \hfill & =2(y_{predict}-y)·\frac{\partial (y_{predict}-y)}{\partial W_i}\hfill \\ \hfill & =2(y_{predict}-y)·\frac{\partial f({\sum }_{i=1}^n{W}_i·x_i+b)}{\partial W_i}\hfill \\ \hfill & =2(y_{predict}-y)·f^{\prime }(\sum _{i=1}^n{W}_i·x_i+b)·x_i\hfill \end{array}$$

(25)

and

$$\begin{array}{cc}\hfill g& =\frac{\partial J(W,b,\overrightarrow{x},y)}{\partial b}\hfill \\ \hfill & =2(y_{predict}-y)·\frac{\partial (y_{predict}-y)}{\partial b}\hfill \\ \hfill & =2(y_{predict}-y)·f^{\prime }(\sum _{i=1}^n{W}_i·x_i+b)\hfill \end{array}.$$

(26)

So the data item $x_i$ can be revealed by computing $x_i=\frac{g_i}{g}$ when the gradients are known. The truth value y can be guessed when input data item $\overrightarrow{x}=\left\{x_1,x_2,\dots ,x_n\right\}$ is an image according to [15].

3. System Architecture

The system architecture can be seen in Figure 2. There are three kinds of parties in our system: learning participant, KTS and DSP. The functions of them are described in detail as follows.

3.1. Learning Participant

In our system, the learning participants (LPs) act as data providers, as well as data requesters. In other words, the learning participants provide the newly computed gradients for servers to update weight variables, and request the updated weight variables for next gradients generation process. Since the learning participants are distributed in our system, the training datasets are distributed as well.

To preserve the data privacy of learning participants, encrypting gradients before sending them to DSP for further processing is necessary. Different from the encryption method in [15] that all participants use one key pair $(sk,pk)$ generated jointly, learning participants encrypt their own gradients with the Diffie–Hellman key $PK$ generated by KTS and DSP in our system. $PK$ is known to all learning participants, but its corresponding private key is not overt. That is to say, any learning participant cannot decrypt the ciphertext of other learning participants to get their private data. Therefore each learning participant does not need to set up the independent secure channel to communicate with servers (KTS and DSP).

Every learning participant (suppose there are N participants) implements the following steps.

• Generate its own key pair. The public key is public to all parties of this scheme.
• Select a mini-batch of data from its own local training dataset randomly for this iteration.
• Download the encrypted weights updated in previous iteration from DSP.
• Decrypt the above encrypted weights with its own private key.
• Compute new gradients with data obtained at step 2 and weights obtained at step 4 using partial derivative.
• Encrypt new gradients with the Diffie–Hellman key $PK$ and send the ciphertexts to KTS.

3.2. Key Transform Server

An innovative and important design of our system is that a KTS is responsible for parameters generation and first phase of re-encryption (FPRE). KTS mainly performs the following operations.

• Select a parameter k and two large primes $p,q$ randomly, where $Len\left(p\right)=Len\left(q\right)=k$.
• Compute $n=p·q$ and choose a generator g with maximal order [29].
• Generate its key pair. Then negotiate the Diffie–Hellman key with DSP.
• Receive the ciphertexts from learning participants, and perform the FPRE over them.
• Send the re-encrypted ciphertexts to the corresponding computing unit $C{U}_i$ of DSP $(i=1,2,\dots ,N)$.

3.3. Data Service Provider

Data service provider (DSP) is responsible for the second phase re-encryption and weights update. In order to make full use of the computation power of DSP and facilitate model parallelism, DSP is split into N parts which are named as computing units $CU$ in our system. The steps DSP executes are as follows.

• Generate its key pair.
• Receive re-encrypted ciphertexts from KTS and perform SPRE on them.
• Update the weights using encrypted gradients obtained in step 2.
• Store the updated encrypted weights into the corresponding computing unit $C{U}_i$ of DSP $(i=1,2,\dots ,N)$.

4. System Realization

In this section we use proxy-invisible homomorphic re-encryption to realize our system described in Section 3. All steps of our proposed system are listed in sequence at length as follows, which are shown in Figure 3.

• ParamGen: $\left(k,p,q\right)\to \left(g,n\right)$. First KTS selects k as a security parameter, and two large primes p and q, where $Len\left(p\right)=Len\left(q\right)=k$ ($Len\left(x\right)$ is the bit length of input data x). Then KTS chooses a generator g with maximal order according to [29] and computes $n=p·q$. Finally KTS publishes the public system parameters $\left(g,n\right)$ to all entities in our system.
• KeyGen: $\left(g,n\right)\to \left(\left(sk,pk\right),PK\right)$. Every learning participant generates its own key pair $\left(s{k}_i,p{k}_i\right)(i=1,2,\dots ,N)$.

  $$p{k}_i=g^{s{k}_i}mod{n}^2,i=1,2,\dots ,N;$$

  (27)

  KTS and DSP generate their key pairs $\left(s{k}_{KTS},p{k}_{KTS}\right)$ and $\left(s{k}_{DSP},p{k}_{DSP}\right)$ respectively.

  $$p{k}_{KTS}=g^{s{k}_{KTS}}mod{n}^2=g^{a}mod{n}^2;$$

  (28)

  $$p{k}_{DSP}=g^{s{k}_{DSP}}mod{n}^2=g^{b}mod{n}^2.$$

  (29)

  Then KTS and DSP negotiate their Diffie-Hellman key $PK$. That is to say, KTS sends its public key $p{k}_{KTS}=g^{a}mod{n}^2$ to DSP, and DSP sends its public key $p{k}_{DSP}=g^{b}mod{n}^2$ to KTS. Therefore both KTS and DSP can calculate $PK$ respectively.

  $$PK=p{k}_{DSP}^{s{k}_{KTS}}={\left(g^{b}mod{n}^2\right)}^a=g^{a·b}mod{n}^2.$$

  (30)

  $$PK=p{k}_{KTS}^{s{k}_{DSP}}={\left(g^{a}mod{n}^2\right)}^2=g^{a·b}mod{n}^2.$$

  (31)

  Finally, the public keys are published to all entities in our system.
• Initialization: since we adopt model parallelism, the deep neural network is separated into N parts (N is the number of learning participants). A machine (consists of a learning participant, KTS and a computing unit of DSP) is responsible for training a part of network. The weight variables in each part of the network form a weight vector respectively. So N weight vectors are assigned to N machines respectively. Before performing training process, weight vectors are initialized by KTS and shared by all parties of this system. In order to make representation clear, here we denote $W_i^{\left(j\right)}$ as the weight vector updated by Machine i in the j th weights update iteration, and $G_i^{\left(j\right)}$ is the gradient vector generated by learning participant i in the j th weights update iteration and used for updating the weight vector $W_i^{\left(j\right)}$. The initial weight vectors are denoted as $W_1^{\left(0\right)},W_2^{\left(0\right)},\dots ,W_N^{\left(0\right)}$. The crossing weights connecting machine i and machine $k(k>i)$ are assigned to machine k.
• Data encoding: generally, weights and gradients are real numbers. But the homomorphic re-encryption requires that numbers to be encrypted should be integers. Therefore before encrypting weights and gradients with homomorphic re-encryption, data encoding process should be performed. A real number $x\in \mathbb{R}$ can be represented by an integer $⌊x·2^n⌋$ with n bits of precision. Here we round down $x·2^n$ as the encoding result of real number x.
• Gradients generation and encryption: take the first weights update iteration $(j=1)$ for example. First, every learning participant (take the learning participant i for example here) uses a mini-batch of data selected randomly from its local dataset and the initial weight vector $W_i^{\left(0\right)}$ to calculate a gradient vector $G_i^{\left(1\right)}$. Then calculate $-\alpha ·G_i^{\left(1\right)}$. Next, components of weight vector $W_i^{\left(0\right)}$ and vector $-\alpha ·G_i^{\left(1\right)}$ are encoded into integers. Finally every learning participant encrypts components of its own vector $-\alpha ·G_i^{\left(1\right)}$ with Diffie–Hellman key $PK$ respectively and sends them to KTS. KTS encrypts each component of initial weight vector $W_i^{\left(0\right)}$ with $PK$.
• First phase of re-encryption (FPRE): After receiving the ciphertext $E_{PK}(-\alpha ·G_1^{\left(1\right)})$, $E_{PK}(-\alpha ·G_2^{\left(1\right)})$, …, $E_{PK}(-\alpha ·G_N^{\left(1\right)})$ from learning participant 1, 2, …, N respectively, KTS performs FPRE over them. Take the ciphertext received from learning participant i for example to describe the first phase of re-encryption algorithms. First KTS computes the hash value $h_1$ and then the re-encrypted ciphertext:

  $$h_1=H({\left(p{k}_i\right)}^{s{k}_{KST}})$$

  (32)

  $$\begin{array}{cc}\hfill {\left[E_{PK}(W_i^{\left(0\right)})\right]}^{+}=& \left\{\widehat{W_i},\widehat{{W_i}^{\prime }}\right\}\hfill \\ \hfill =& \left\{W_i,{\left({W_i}^{\prime }\right)}^{s{k}_{KTS}}·g^{h_1}\right\}\hfill \end{array}$$

  (33)

  $$\begin{array}{cc}\hfill {\left[E_{PK}(-\alpha ·G_i^{\left(1\right)})\right]}^{+}=& \left\{\widehat{T_i},\widehat{{T_i}^{\prime }}\right\}\hfill \\ \hfill =& \left\{T_i,{\left({T_i}^{\prime }\right)}^{s{k}_{KTS}}·g^{h_1}\right\}\hfill \end{array}$$

  (34)

  Finally, KTS sends ${\left[E_{PK}(W_i^{\left(0\right)})\right]}^{+}$ and ${\left[E_{PK}(-\alpha ·G_i^{\left(1\right)})\right]}^{+}$ to DSP.
  It is noticeable that if a gradient component is used for updating a crossing weight, then it should be re-encrypted with the public key of the learning participant the crossing weight assigned to.
• SPRE and homomorphic addition: DSP receives ${\left[E_{PK}(W_i^{\left(0\right)})\right]}^{+}$ and ${\left[E_{PK}(-\alpha ·G_i^{\left(1\right)})\right]}^{+}$$(1\le i\le N)$ from KTS and stores them in the corresponding computation unit $C{U}_i$. Each computation unit performs SPRE. Take the $C{U}_i$ for example. It performs following algorithms:

  $$h_2=H({\left(p{k}_i\right)}^{s{k}_{DSP}})$$

  (35)

  $$\begin{array}{cc}\hfill E_{p{k}_i}(W^{\left(0\right)})=& \left\{\overline{W},\overline{W^{\prime }}\right\}\hfill \\ \hfill =& \left\{\widehat{W},{\left(\widehat{W^{\prime }}\right)}^{s{k}_{DSP}}·g^{h_2}\right\}\hfill \end{array}$$

  (36)

  $$\begin{array}{cc}\hfill E_{p{k}_i}(-\alpha ·G_i^{\left(1\right)})=& \left\{\overline{T},\overline{T^{\prime }}\right\}\hfill \\ \hfill =& \left\{\widehat{T},{\left(\widehat{T^{\prime }}\right)}^{s{k}_{DSP}}·g^{h_2}\right\}\hfill \end{array}$$

  (37)

  It is noticeable that $E_{p{k}_i}\left(W^{\left(0\right)}\right)$ and $E_{p{k}_i}(-\alpha ·G_i^{\left(1\right)})$ have the property of additive homomorphism now. So computation unit i of DSP can update the weight vectors:

  $$\begin{array}{c}\hfill E_{p{k}_i}\left(W_i^{\left(1\right)}\right)=E_{p{k}_i}\left(W_i^{\left(0\right)}\right)\ast E_{p{k}_i}\left(-\alpha ·G_i^{\left(1\right)}\right),\\ \hfill 1\le i\le N.\end{array}$$

  (38)

  Finally $E_{p{k}_i}(W_i^{\left(1\right)})$ is stored into the corresponding computation unit $C{U}_i$ of DSP.
• Decryption:From the second iteration of weights update process on, each learning participant downloads updated weight vector from its corresponding computation unit of DSP. That is to say, learning participant i downloads $E_{p{k}_i}(W_i^{\left(j\right)})$ from $C{U}_i$ of DSP $(1\le j\le N_{wu},1\le i\le N)$. Then learning participant i decrypts the downloaded $E_{p{k}_i}(W_i^{\left(j\right)})$ with its private key $s{k}_i$. The decryption algorithms are as follows.

  $${h_1}^{\prime }=H\left({\left(p{k}_{KTS}\right)}^{s{k}_i}\right)$$

  (39)

  $${h_2}^{\prime }=H\left({\left(p{k}_{DSP}\right)}^{s{k}_i}\right)$$

  (40)

  $$\begin{array}{cc}\hfill W_i^{\left(j\right)}=& L(\overline{T}·p{k}_{DSP}^{{h_1}^{\prime }}·g^{{h_2}^{\prime }}/\overline{T^{\prime }}mod{n}^2),where\hfill \\ \hfill L\left(x\right)=& (x-1)/n\hfill \end{array}$$

  (41)

  Finally, downloaded $W_i^{\left(j\right)}$ updated by DSP is decrypted and can be used at next gradients generation iteration or deep learning model configuration when the training is finished.
• Iteration:When the training is not finished, each learning participant repeats the above step 4 to step 8 to iterate the weights update process. In other words, learning participant i computes the gradient vector $G_i^{(j+1)}$ with another mini-batch of data from its local dataset and $W_i^{\left(j\right)}$. In the end of the training process, all the learning participants can get ultimate weight vector $W=(W_1^{\left(N_{wu}\right)},W_2^{\left(N_{wu}\right)},\dots ,W_N^{\left(N_{wu}\right)})$ by downloading and decrypting the $E_{p{k}_i}(W_i^{\left(wu\right)})$ from the $C{U}_i$ of DSP $(1\le i\le N)$ respectively.

5. Security Analysis

In this section, we present the assumption of the computational difficulty problem on which our scheme based, and then study the security of our scheme.

5.1. Assumption

Decisional Diffie-Hellman (DDH) Problem [31] over $Z_{n^2}^{*}$: For each probabilistic polynomial time function F, there is a negligible function $negl\left(\right)$ so that for sufficiently large l:

$$\begin{array}{c}Pr\left[F(n,X,Y,Z_b)=b\left|\begin{array}{c}p,q\leftarrow SP(0.5·l);\\ n=pq;g\leftarrow \mathbb{G};\\ x,y,z\leftarrow [1,ord(\mathbb{G}\left)\right];\\ X=g^{x}mod{n}^2;\\ Y=g^{y}mod{n}^2;\\ Z_0=g^{z}mod{n}^2;\\ Z_1=g^{xy}mod{n}^2;\\ b\leftarrow 0,1;\end{array}\right.\right]\hfill \\ =negl\left(l\right)+0.5\hfill \end{array}$$

(42)

That is to say, when $g^x$ and $g^y$ are given, the possibility for adversaries to distinguish between $g^z$ and $g^{xy}$ is negligible.

5.2. Security of Our Scheme

In this section, we analyze the security of our scheme. As mentioned in Section 1.1, we consider the scenario that KTS and DSP would not collude with each other, but KTS or DSP may collude with one of learning participants. First we give the proof that our scheme is secure in the presence of semi-honest adversaries $(A_{LP},A_{KTS},A_{DSP})$ under non-colluding setting. Then we analyze the security of our system when there are colluding adversaries.

Proof. 

Our scheme is secure in the presence of semi-honest adversaries under non-colluding setting. Our scheme is based on the HRES. The HRES is proved to be semantically secure in [28] based on the difficulty of Decisional Diffie–Hellman Problem above. We omit its proof here in order to avoid repetition. Our scheme involves three types of entities: LP, KTS and DSP. Three kinds of challengers $C_{LP},C_{KTS},C_{DSP}$ are constructed to against three kinds of adversaries $A_{LP},A_{KTS}$, $A_{DSP}$ who want to corrupt LP, KTS and DSP respectively.

When a new gradient G is generated, $C_{LP}$ challenges $A_{LP}$ as follows. First, $C_{LP}$ multiplies it by $-\alpha$ and encodes the result. Then $C_{LP}$ encrypts the encoded result with $PK$ as ${\left[m\right]}_{PK}$. Then $C_{LP}$ sends ${\left[m\right]}_{PK}$ to $A_{LP}$, and outputs the entire view of $A_{LP}$: ${\left[m\right]}_{PK}$. $A_{LP}$’s views in ideal and real executions are indistinguishable because of the security of HRES.

$C_{KTS}$ challenges $A_{KTS}$ as follows. $C_{KTS}$ runs encryption on two randomly chosen integers with $PK$ as ${\left[G\right]}_{PK}$ and ${\left[w\right]}_{PK}$. Then ${\left[G\right]}_{PK}$ is multiplied by ${\left[w\right]}_{PK}$. Next $C_{KTS}$ performs the FPRE on the result to get ${\left[m\right]}^{+}$, which is sent to $A_{KTS}$. If $A_{KTS}$ responds with ⊥, $C_{KTS}$ returns ⊥. $A_{KTS}$’s views are made up of the encrypted data. $A_{KTS}$ gets the same outputs in both ideal and real executions because the LPs are honest and the HRES is proved to be secure. Therefore $A_{KTS}$’s views are indistinguishable.

$C_{DSP}$ challenges $A_{DSP}$ as follows. $C_{DSP}$ first chooses ${\left[m\right]}^{+}$ randomly and performs the SPRE on it with $C_{DSP}$’s private key to get ${\left[m\right]}_{p{k}_j}$. Then ${\left[m\right]}_{p{k}_j}$ is sent to $A_{DSP}$. If $A_{DSP}$ responds with ⊥, $C_{DSP}$ returns ⊥. $A_{DSP}$’s views are made up of the encrypted data. $A_{DSP}$ gets the same output ${\left[m\right]}_{p{k}_j}$ in both ideal and real executions because of the security of HRES. Therefore $A_{DSP}$’s views are indistinguishable.

When it comes to decryption of the updated weight, $C_{LP}$ challenges $A_{LP}$ as follows. $C_{LP}$ chooses ${\left[m^{\prime }\right]}_{p{k}_j}$ randomly and decrypts it. The result $m^{\prime }$ is sent to $A_{LP}$. If $A_{LP}$ responds with ⊥, $C_{LP}$ returns ⊥. The result $m^{\prime }$ is the view of $A_{LP}$, which is indistinguishable in both ideal and real executions because of the security of HRES. Therefore $A_{DSP}$’s views are indistinguishable. □

Therefore our scheme is secure when there are semi-honest adversaries under non-colluding setting. Next, we illuminate that our system is secure even when KTS colludes with one of the learning participants or DSP colludes with one of the learning participants. We suppose that learning participants, KTS and DSP are honest but curious. That is to say, all parties in the system will perform the execution they should do following the system steps but they would try to get local data of learning participants. Here we analyze two situations as follows: KTS colludes with one of the learning participants, or DSP colludes with one of the learning participants.

Situation 1: KTS colludes with one of the learning participants. If KTS colludes with learning participant A, then they can share information with each other and try to get private information of other learning participants. That is to say, KTS can get the gradients of A, even the private key of A. A can get private key of KTS, too. In the scheme of [15], if the cloud server colludes with a learning participant, and the server gets the private key of the learning participant, then the server could decrypt all the gradients of all learning participants because their private keys are the same. Since the gradients leak information of original data, the private data of learning participants are not safe. However, it is impossible in our proposed scheme because all learning participants encrypt their gradients with the Diffie–Hellman key generated by KTS and DSP. KTS can not decrypt the gradients itself even with the private key of a learning participant. Because the generation of hash value $h_2$ can resit impersonation attack and collusion between KTS and any learning participant. KTS should perform steps in the first phase of re-encryption described in Section 4 honestly. Therefore, the private data of learning participants are secure even if KTS colludes with one of the learning participants in our scheme.

Situation 2: DSP colludes with one of the learning participants. If DSP colludes with learning participant B, the situation is similar to Situation 1. In other words, DSP can get the gradients of B, even the private key of B. B can get private key of DSP, as well as the all the re-encrypted weights and gradients. DSP should perform steps in the second phase of re-encryption described in Section 4 honestly. With the private key of learning participant B, DSP could only decrypt the gradients from learning participant B, but has no knowledge about gradients of other learning participants. Because the generation of hash value $h_1$ can resit impersonation attack and collusion between DSP and any learning participant. Therefore, the original data of learning participants are safe even if DSP colludes with one of the learning participants.

To sum up, our scheme is resistant to collusion between any cloud sever and any learning participant, which is not available in schemes of [15].

6. Performance Evaluation

In this section, we evaluate the communication cost and computational cost of our proposed scheme through theoretical analysis and simulation. First, we introduce the concrete parameter settings of the experiment we take.

It is shown in [28] that the length of n, $Len\left(n\right)$ influences the computation efficiency greatly, as well as security of the HRES. Experimental results showed that larger $Len\left(n\right)$ meant longer communication time, but higher security. In order to balance the efficiency and security, here we set $Len\left(n\right)=1024$ bits, which guarantees the learning participants with limited resources can finish decryption process efficiently according to [28].

In the experiment, we generated all the private keys randomly. The influence of length of a private key on HRES was tested in [28]. From the result, we can observe that when length of private key is between 100–200 bits, the computational costs are similar. In order to compare with the scheme in [15], we set the length of private key as 128 bits in our system, which provided the same length of security parameter as that in [15].

In the data coding process, we set $n=32$. That is to say, the precision of data encoding results was 32-bit. We set $N=10$, which means the deep neural network was separated into 10 parts averagely and there are 10 learning participants, 10 machines and 10 weight vectors.

6.1. Communication Cost Analysis

In this section we discuss the total communication cost of our scheme $C_{oursystem}$. There are three kinds of communication as shown in Figure 3, communication between learning participants and KTS $C_{p-k}$, communication between KTS and DSP $C_{k-d}$, and communication between DSP and learning participants $C_{d-p}$.

To estimate the communication cost of our scheme, first we design the following formulas:

$$C_{oneiteration}^{\left(j\right)}=C_{p-k}^{\left(j\right)}+C_{k-d}^{\left(j\right)}+C_{d-p}^{\left(j\right)},$$

(43)

$$C_{oursystem}=\sum _{j=1}^{N_{wu}}{C}_{oneiteration}^{\left(j\right)}.$$

(44)

In Equation (43), $C_{oneiteration}^{\left(j\right)}$ is the communication cost of $j\mathrm{th}$ iteration.

Next, we discuss the communication cost of transmitting one ciphertext between two parties. For example, ciphertext $\left[m_i\right]$ in our proposed scheme is composed of two components: $\left[m_i\right]=\left\{T_i,{T_i}^{\prime }\right\}$, where $T_i=\left(1+m_i·n\right)·P{K}^{r}mod{n}^2$ and ${T_i}^{\prime }=g^{r}mod{n}^2$. The length of $T_i$ and ${T_i}^{\prime }$ are related to $n^2$. Each of them has $2\ast Len\left(n\right)$ bits, so uploading or downloading one ciphertext needs to transmit $4\ast Len\left(n\right)$ bits. We set $Len\left(n\right)=1024$ bits, so transmitting one ciphertext in our scheme means transmitting 4096 bits.

Finally, according to our system realization described in Figure 3, we can obtain the following results. For the first iteration:

$$\begin{array}{cc}\hfill C_{oneiteration}^{\left(1\right)}& =C_{p-k}^{\left(1\right)}+C_{k-d}^{\left(1\right)}+C_{d-p}^{\left(1\right)}\hfill \\ \hfill & =N·4096+2·N·4096+N·4096\hfill \\ \hfill & =4·N·4096.\hfill \end{array}$$

(45)

For the iterations $2\le j\le N_{wu}$:

$$\begin{array}{cc}\hfill C_{oneiteration}^{\left(j\right)}& =C_{p-k}^{\left(j\right)}+C_{k-d}^{\left(j\right)}+C_{d-p}^{\left(j\right)}\hfill \\ \hfill & =N·4096+N·4096+N·4096\hfill \\ \hfill & =3·N·4096.\hfill \end{array}$$

(46)

So the total communication cost of our scheme is

$$\begin{array}{cc}\hfill C_{oursystem}& =\sum _{j=1}^{N_{wu}}{C}_{oneiteration}^{\left(j\right)}\hfill \\ \hfill & =4·N·4096+3·(N_{wu}-1)·N·4096\hfill \\ \hfill & =N·4096+3·N_{wu}·N·4096\hfill \\ \hfill & =(1+3·N_{wu})·N·4096.\hfill \end{array}$$

(47)

In order to compare the communication cost of our scheme with that of scheme in [15], we analyzed the increased communication factor of our scheme. The increased factor is

$$F_{increased-com}={\displaystyle \frac{EncryptedBits}{PlainBits}}.$$

(48)

Since $Len\left(n\right)=lo{g}_{2}n=1024$ bits in our system, we can pack $t=⌊\frac{⌊lo{g}_{2}n⌋}{prec+pad}⌋$ real numbers (after encoding into integers) into one HRES plaintext. We set the precision as $prec=32$ bits in encoding process, and $pad=lo{g}_2{N}_{wu}$ bits to prevent overflows in ciphertext additions [15]. So each ciphertext in our scheme can be used to encrypt t gradients. Therefore the increased factor is

$$\begin{array}{cc}\hfill F_{increased-com}& ={\displaystyle \frac{EncryptedBits}{PlainBits}}\hfill \\ \hfill & ={\displaystyle \frac{(1+3·N_{wu})·N·4096}{t·2·(prec+pad)·N_{wu}·N}}\hfill \\ \hfill & \approx {\displaystyle \frac{(1+3·N_{wu})·N·4096}{2·⌊lo{g}_{2}n⌋·N_{wu}·N}}\hfill \\ \hfill & ={\displaystyle \frac{(1+3·N_{wu})·2}{N_{wu}}}\hfill \\ \hfill & ={\displaystyle \frac{2}{N_{wu}}}+6.\hfill \end{array}$$

(49)

When the number of weights update process $N_{wu}$ is large enough, the increased factor of our scheme was 6. The comparison of communication cost between our scheme and latest works can be found in

Table 2. Communication cost comparison.

Increased Communication Factor	Formula	Value
LWE-based scheme in [15]	$\frac{N·n·lo{g}_{2}q}{N_{wu}·prec}+\frac{lo{g}_{2}q}{prec}$	-
Paillier-based scheme in [15]	$2·(1+\frac{pad}{prec})$	≈2.93
Our scheme	${\displaystyle \frac{2}{N_{wu}}}+6$	≈6

.

Therefore, our scheme was about six times of communication cost of the corresponding ASGD, and two times of communication cost of scheme in [15]. But in terms of learning participants, the communication cost of our scheme was similar to that in [15].

The total number of gradients in the network is $N_g=$ 109,386 in our system. After multiplying $-\alpha$ ($\alpha$ is the learning rate), every gradient is encoded into a 32-bits integer $⌊(-\alpha ·G)·2^n⌋$ and then is encrypted with $PK$. Refer to [15], we calculated the total size of ciphertext as

$$\begin{array}{cc}\hfill L_{ciphertext}& =N_g\ast Len\left(onegradient\right)\ast F_{increased-com}\hfill \\ \hfill & =109386\times 32\times 6\mathrm{bits}\hfill \\ \hfill & =2.6 \mathrm{MB}.\hfill \end{array}$$

(50)

All the ciphertext in our system can be sent in about

$$T_{communication}=L_{ciphertext}/1\mathrm{Gbps}=20.8\mathrm{ms}$$

(51)

via the 1 Gbps channel for communication (suppose the same communication channel speed as that in [15]).

6.2. Computational Cost Analysis

In this section we analyze the computational cost of our scheme. There are three kinds of parties in our system as shown in Figure 2, learning participants, KTS and DSP. We will analyze the computational cost of each of them respectively. We evaluate the computational cost via running time of the algorithms they execute. First we design the following formulas:

$$T_{oneiteration}^{\left(j\right)}=(T_{gradientsgeneration}^{\left(j\right)}+T_{enc}^{\left(j\right)})+T_{FPRE}^{\left(j\right)}+(T_{SPRE}^{\left(j\right)}+T_{add}^{\left(j\right)})+T_{dec}^{\left(j\right)},$$

(52)

$$T_{oursystem}=\sum _{j=1}^{N_{wu}}{T}_{oneiteration}^{\left(j\right)}.$$

(53)

In Equation (52), $T_{oneiteration}^{\left(j\right)}$ is the total running time of the $j_{th}$ weights update process. When a new gradient vector (consists of many gradients generated by all learning participants) was generated, one weight’s update process began. Each learning participant encrypted their own gradients. So the computational cost of the learning participants was $(T_{gradientsgeneration}^{\left(j\right)}+T_{enc}^{\left(j\right)})$. Then KTS carried out the FPRE over the gradients using public keys of different learning participants respectively. So the computational cost of KTS was $T_{FPRE}^{\left(j\right)}$. Next, DSP performs SPRE over the received ciphertexts using public keys of the learning participant and updates the weights via homomorphic addition. Therefore the computational cost of DSP was $(T_{SPRE}^{\left(j\right)}+T_{add}^{\left(j\right)})$. Finally, learning participants download and decrypt the updated weights using their own private keys respectively. Hence, the computational cost of learning participants was $T_{dec}^{\left(j\right)}$.

Next, we estimated the computational cost of our scheme through simulations. In order to compare the computational cost of our scheme with that of the LWE-based scheme in [15], we used the Tensorflow 1.8.0 library and Cuda-9.0 to construct the same fully connected multilayer perceptron (MLP) network as that in [15], with four layers (784-128-64-10 neurons sequentially). The total number of gradients in the network is $N_g=(784+1)\times 128+(128+1)\times 64+(64+1)\times 10=$ 109,386 as well. The detailed experimental settings in model training are shown in

Table 3. Experimental settings in model training.

Schemes	Batch Size	Learning Rate	Precision	Iteration Times	${\mathit{F}}_{\mathit{Activation}}$	Dataset
LWE-MLP [15]	50 images	${10}^{-4}$	32-bits	2 ×${10}^4$	ReLu	MNIST
Our Scheme	50 images	${10}^{-4}$	32-bits	$2\times {10}^4$	ReLu	MNIST

.

We implement our scheme with C++ codes. The network and scheme are run on a computer with NVIDIA GeForce GTX 1080 Ti, Intel(R) Core(TM) I7-4702MQ CPU @ 2.20 GHz and 16 G Memory. The operating system is Windows 10.

According to simulation, the average running time of gradients generation process through training the MLP in our scheme was $T_{gradientsgeneration}=1.3$ ms; and the HRES running time of processing the 109,386 gradients of the network with 10 learning participants in one weights update process was $T_{HRES}=5192.4$ ms. Therefore, the running time of one weights update process in our system was

$$\begin{array}{cc}\hfill T_{oneupdate}& =T_{gradientsgeneration}+T_{HRES}\hfill \\ \hfill & =1.3+5192.4\hfill \\ \hfill & =5193.7\left(\mathrm{ms}\right).\hfill \end{array}$$

(54)

The average running time per weights update iteration for performing steps of homomorphic re-encryption in our scheme, including encryption, FPRE, SPRE, homomorphic addition and decryption time are depicted in Figure 4, which are denoted as encTime, FPRETime, SPRETime, addTime and DecTime respectively.

As mentioned above, learning participants were responsible for the implementation of encryption and decryption; KTS was responsible for FPRE; DSP was responsible for SPRE and addition. So we calculated the computational cost of them respectively in

Table 4. Average computational cost of parties in our scheme.

Party	Every Learning Participant	KTS	DSP
Time item	encTime+DecTime	FPRETime	SPRETime+addTime
Average time	1.55 (μs)	5.76 (μs)	26.20 (μs)
Percentage	$3.3\%$	$12\%$	$55\%$

(accurate to the second decimal place). It can be seen that DSP had the maximum computational overhead ($55\%$). The computational cost of learning participants are $33\%$ of the total computational cost. There were 10 learning participants in our system, and we separated the model averagely, so the average computational cost of every learning participant was just $3.3\%$, which was much lower than that of KTS and DSP. The configuration was reasonable because the computational capability of cloud server is much higher than that of every data provider generally.

The experimental results comparison is shown in Figure 5 and

Table 5. Experimental results comparison.

Schemes	${\mathit{T}}_{\mathit{gradients}\mathit{generation}}$ (ms)	${\mathit{T}}_{\mathit{weights}\mathit{update}}$ (ms)	${\mathit{T}}_{\mathit{Total}}$ (ms)	Accuracy
LWE-MLP [15]	4.6	1963.5	1968.1	$97\%$
Our Scheme	1.3	5192.4	5193.7	$97.1\%$

. In Figure 5, $T\_enc$ of our scheme consists of there parts: running time of encryption, FPRE and SPRE. In Table 5, $T_{weightsupdate}$ means the running time of using our scheme to process the gradients and update the weights. It consists of five parts in our scheme: encryption (1112.4 ms) and decryption (584.1 ms) by learning participants, FPRE (630.2 ms) by KTS, SPRE (615.1 ms) and homomorphic addition (2250.6 ms) by DSP. On the other hand, the LWE-MLP scheme in [15], consists of there parts: encryption (899.2 ms) and decryption (785.4 ms) by learning participants, homomorphic addition (278.9 ms) by DSP.

The running time of our system was about 2.64 times of that in LWE-MLP [15]. Although the total running time of our system was longer, the computational cost of learning participants was similar to that of [15], which is shown in

Table 6. Computational cost of parties comparison.

Time (ms)	Learning Participants	KTS	DSP
LWE-MLP [15]	$899.2+785.4=1684.6$	-	278.9
Our scheme	$1112.4+584.1=1696.5$	630.2	$615.1+2250.6=2865.7$

. The accuracy is $97.1\%$, which was a little higher than that in [15].

7. Conclusions

Considering that previous distributed deep learning scheme sharing single key pair for encryption suffers from collusion between cloud server and any learning participant [15], we propose a novel system using homomorphic re-encryption which makes privacy-preserving distributed deep learning come true. We give the detailed realization steps of our system and implement them for validation. Furthermore, security analysis and performance evaluation are provided. Specifically, the communication cost of our scheme is tolerable. Experimental results show that although the running time of our system is larger than that of LWE-MLP in [15], but the computational overhead of learning participants is nearly the same as that in [15]. More importantly, our system is more secure and accurate because it is collusion-resistant between any server and any learning participant, with higher deep learning accuracy.