Privacy-Preserving Distributed Learning via Newton Algorithm

Abstract

Federated learning (FL) is a prominent distributed learning framework. The main barriers of FL include communication cost and privacy breaches. In this work, we propose a novel privacy-preserving second-order-based FL method, called GDP-LocalNewton. To improve the communication efficiency, we use Newton’s method to iterate and allow local computations before aggregation. To ensure strong privacy guarantee, we make use of the notion of differential privacy (DP) to add Gaussian noise in each iteration. Using advanced tools of Gaussian differential privacy (GDP), we prove that the proposed algorithm satisfies the strong notion of GDP. We also establish the convergence of our algorithm. It turns out that the convergence error comes from the local computation and Gaussian noise for DP. We conduct experiments to show the merits of the proposed algorithm.

1. Introduction

Federated learning (FL) is a popular distributed learning paradigm that enables a large amount of clients to collaboratively train a global model without sharing their individual data [1]. The most popular algorithm is called federated averaging (FedAvg). In each round, the server broadcasts the current model to all the clients. The clients then run multiple steps of stochastic gradient descent (SGD) in a distributed fashion. After that, the server updates the global model by aggregating results from local clients. The convergence rate of FedAvg has been studied extensively in recent years [2,3,4].

One desiderata of a real FL paradigm is the communication efficiency. The communication cost of FL mainly comes from the latency cost, that is, the fixed cost of sending messages, which is proportional to the communication rounds, regardless of the size of the message. In pursuit of this, many first-order gradient-based methods have been developed, such as local stochastic gradient descent (Local SGD) [1,5,6,7] and mini-batch SGD [8,9,10]. These algorithms reduce the communication cost by performing local computations at the client devices before aggregation. It works when the clients are mobile resources which have reasonable computational power but may suffer from the communication latency [11]. However, for some serverless systems, say, the cloud-based systems, the latency cost is severe. The communication failures are more frequent. Hence, the communication rounds should be further reduced. Recent years have seen a few work on using second-order-based methods [11,12,13,14,15] to improve the convergence of first-order-based methods. Wang et al. [12] proposed the GIANT algorithm, a new distributed approximation of Newton’s method, and showed an improved convergence rate over the distributed first-order competitors. Dünner et al. [13] developed a method which approximates the global Hessian matrix by using the block Hessian matrices from users, thereby reducing the computational burden of the global Hessian matrix. Gupta et al. [11] proposed and analyzed a second-order method with local computations, called LocalNewton. Due to its second-order nature and local computations, LocalNewton is superior in terms of the communication cost.

The other desideratum of a real FL paradigm is the privacy preservation. The data from local clients often contain sensitive information of individuals. To some extent, FL could preserve the individuals’ privacy because the original data never leave the clients. However, there exist adversarial attacks, which can result in privacy leakage by using the information during the communication; see [16] for example. Hence, the vanilla FL paradigm does not have a rigorous privacy guarantee. Differential privacy (DP; Dwork [17]) is a standard and well-adopted framework which provides strong guarantee of an individual’s privacy by ensuring that no individual has a significant influence on the algorithm’s output. The worst-case influence is indeed termed the privacy budget. A differentially private (DP) algorithm is often achieved by randomizing the algorithm’s output. Besides the vanilla $(ϵ,\delta )$-DP, several DP notions are developed, say, the Rényi differential privacy (RDP) [18] and the Gaussian differential privacy (GDP) [19]. RDP and GDP enjoy an elegant composition property, that is, the theoretical bound of the privacy leakage of repeated queries can be tight and lossless, which would bring benefits when accounting for the privacy leakage in iterative-based algorithms. Using the notion of DP, there have been several studies on developing a privacy-preserving FL algorithm; see McMahan et al. [2], Geyer et al. [3], Triastcyn and Faltings [4], Noble et al. [20], Wei et al. [21], Girgis et al. [22], Cheu et al. [23], Rastogi and Nath [24], Huang et al. [25], among others. Most of these algorithms are first-order based.

With the two desiderata in mind, we develop a novel privacy-preserving second-order-based method called GDP-LocalNewton within the FL framework. Building upon Newton’s algorithm, each device performs Newton’s iterations locally. To avoid privacy leakage, we take advantage of the notion of GDP [19] to add Gaussian noise to the updates in each iteration. In particular, we propose a novel linear search method to determine the step size in the Newton update. After several local steps, the devices send parameter updates to the central server. The server then aggregates these updates and broadcasts them to synchronize all local parameters. This process is iterated until convergence or specific conditions are attained. It is worth mentioning that we assume a specific adversary termed as the curious onlooker, who can eavesdrop on the communication between the server and the local machines. The server is a typical example of a curious onlooker.

We rigorously show that under proper parameters set-up, the proposed algorithm satisfies GDP. We also analyze the convergence bound of GDP-LocalNewton. It turns out that the convergence error comprises two components. One comes from the DP noises, and the other comes from the local computations. We also conduct experiments to show the merits of GDP-LocalNewton.

2. GDP-LocalNewton

In this section, we first provide the problem formulation and the framework of GDP. Then, we develop the GDP-LocalNewton algorithm. After that, we provide rigorous privacy analysis and convergence analysis of the proposed algorithm.

2.1. Problem Formulation

2.1.1. Some Notations and Symbols

We represent vectors (e.g., $\mathbf{g}$) and matrices (e.g., H) as bold lowercase and uppercase letters, respectively. $\parallel \mathbf{g}\parallel$ denotes its ${\ell }_2$ norm, and the spectral norm of matrix $\mathbf{H}$ is denoted by $\parallel \mathbf{H}\parallel$. $\mathbf{I}$ denotes an identity matrix, and the set $\{1,2,\dots ,n\}$ is denoted by $\left[n\right]$. To distinguish the index of workers and iterations, we denote the worker index as the superscript (e.g., ${\mathbf{g}}^k$) and denote the iteration counter as the subscript (e.g., ${\mathbf{g}}_t$).

Our paper contains a large number of parameters. For ease of reading and distinction, we list them in

Table 1. The parameters set in this paper.

Parameters	Meaning
$\mu$	Privacy parameter.
n	The number of whole data.
s	The number of worker’s data.
K	The number of workers.
T	The number of iteration.
L	The number of local computation.
B	Upper bound of the Hessian matrix of individual loss function.
$\mathsf{\Gamma }$	Upper bound of the gradient of individual loss function.
M	The smooth parameter of global loss function.
$\kappa$	The strongly convex parameter of global loss function.
$\alpha$	Iteration step size.
${\alpha }^{*}$	The max linear search step size.
$\beta$	A parameter in the linear search
${\mathbf{w}}_t$	Global model parameter at the t-th iteration.
${\mathbf{w}}^k$	The k-th worker’s model parameter.
${\mathbf{w}}_t^k$	The k-th worker’s model parameter at the t-th iteration.
${\mathbf{H}}^k$, ${\mathbf{H}}_t$, ${\mathbf{H}}_t^k$, ${\mathbf{g}}^k$, ${\mathbf{g}}_t$, ${\mathbf{g}}_t^k,$ etc.	Like ${\mathbf{w}}_t$, ${\mathbf{w}}^k$ and ${\mathbf{w}}_t^k$.

.

2.1.2. Fundamental Problem

We consider the empirical risk minimization problems of the following form:

$$\underset{\mathbf{w}\in {\mathbb{R}}^d}{\min }\{f\left(\mathbf{w}\right)\triangleq \frac{1}{n}\sum _{j=1}^n{f}_j\left(\mathbf{w}\right)\},$$

(1)

where $f_j(·):{\mathbb{R}}^d\to \mathbb{R}$, for all $j\in \left[n\right]=1,2,\dots ,n$, represents the loss of the j-th observation, given an underlying parameter $\mathbf{w}\in {\mathbb{R}}^d$. Generally, we call the $f(·)$ the global loss function. In the field of machine learning, such problems are general, e.g., logistic and linear regression, support vector machines, neural networks and graphical models. Taking the logistic regression as an example, we have

$$f_j\left(\mathbf{w}\right)={\ell }_j\left({\mathbf{w}}^T{\mathbf{x}}_j\right)=\log (1+{\exp }^{-y_j{\mathbf{w}}^T{\mathbf{x}}_j})+\frac{\gamma }{2}{\parallel \mathbf{w}\parallel }^2,$$

where ${\ell }_j(·)$ is the loss function for sample $j\in \left[n\right]$ and $\gamma$ is an approximately chosen regularization parameter. $\mathbf{X}=[{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\dots ,{\mathbf{x}}_{\mathbf{n}}]\in {\mathbb{R}}^{d\times n}$ is the sample matrix containing n data, and $\mathbf{y}=[y_1,y_2,\dots ,y_n]$ is the corresponding label vector. Hence, $(\mathbf{X},\mathbf{y})$ defines the training dataset, which is composed as $({\mathbf{x}}_j,y_j),j=1,2,..,n$.

At the t-th iteration, the gradient and the Hessian are denoted by

$${\mathbf{g}}_t=\nabla f\left({\mathbf{w}}_t\right)=\frac{1}{n}\sum _{j=1}^n\nabla f_j\left({\mathbf{w}}_t\right),$$

and

$${\mathbf{H}}_t={\nabla }^{2}f\left({\mathbf{w}}_t\right)=\frac{1}{n}\sum _{j=1}^n{\nabla }^2{f}_j\left({\mathbf{w}}_t\right),$$

respectively, where ${\mathbf{w}}_t$ is the updated parameter at the t-th iteration. Specially, in this paper, we make some assumptions about loss functions. Global loss function$f(·)$ is smooth and strongly convex. $f_i(·)$ is twice differentiable, ${\nabla }^2{f}_i\left(\mathbf{w}\right)\preccurlyeq B\mathbf{I}$ and $\parallel \nabla f_i{(·)\parallel }_2⩽\mathsf{\Gamma }$, where $B\in \mathbb{R}$ and $\mathsf{\Gamma }\in \mathbb{R}$ are some fixed constants.

2.1.3. Data Distribution

Assume that the FL system has K workers in total. ${\mathcal{S}}_k$, for all $k\in \left[K\right]=\{1,2,\dots ,K\}$, represents a subset chosen uniformly from $\left[n\right]=\{1,2,\dots ,n\}$ without replacement randomly, and the workers can be identified by the subsets, uniquely. The number of each workers’ sample is equal to $s=|{\mathcal{S}}_k|$ for any $k\in \left[K\right]$. Due to the sampling without replacement assumption, we have ${\mathcal{S}}_1\cup {\mathcal{S}}_2\cup \dots {\mathcal{S}}_K=\left[n\right]$ and ${\mathcal{S}}_i\cap {\mathcal{S}}_j=\mathsf{\Phi }$ for all $i,j\in \left[K\right]$. Also, $K=n/s$ represents the number of workers.

2.2. Gaussian Differential Privacy

In this subsection, we review the definition of Gaussian differential privacy (GDP) [26]. Then, we give some theorems and tools about GDP [26], which are the base of the proof that our algorithm satisfies $\mu -GDP$. Lemmas 1 and 2 show that every user’s output satisfies ${\mu }_k-GDP$. Lemma 3 guarantees DP when the entire dataset is divided as disjointed parts.

Definition 1

(f-$DP$). Let $h:{\mathbb{R}}^{n\times m}\to {\mathbb{R}}^p$ be a randomized function, and $\mathbb{D}=\{x_1,...,x_i,...,x_n\}$ be a dataset. If the output of h for any α-level test between the hypotheses of the formulation $H_0:x_i=t$ vs. $H_1:x_i=s$, which has power function $\beta \left(\alpha \right)⩽1-f\left(\alpha \right)$, where f is a convex, continuous, non-increasing function and for all $\alpha \in [0,1]$ $f\left(\alpha \right)⩽1-\alpha$, then we say h satisfies f-$DP$.

Definition 2

(Gaussian differential privacy). Define $\mathsf{\Phi }(·)$ as the standard Gaussian cumulative distribution. If h is f-$DP$ and

$$f\left(\alpha \right)⩾\mathsf{\Phi }({\mathsf{\Phi }}^{-1}(1-\alpha )-\mu ),\alpha \in [0,1],$$

then h is μ-Gaussian differentially private (μ-GDP).

Definition 3

(Global sensitivity). Let $g:{\mathbb{R}}^{n\times m}\to {\mathbb{R}}^p$ be a deterministic function. The global sensitivity of g is the (possibly infinite) number

$$G{S}_g=\underset{x_{1:n},x_{1:n}^{\prime }\in {\mathbb{R}}^{n\times m}}{\sup }\{\parallel g\left(x_{1:n}^{\prime }\right)-g\left(x_{1:n}\right){\parallel }_2:d_H(x_{1:n},x_{1:n}^{\prime })=1\},$$

(2)

where $x_{1:n}$ means a dataset $\{x_1,x_2,...,x_n\}$, and $x_{1:n}^{\prime }$ is different from $x_{1:n}$ at some datum.

Lemma 1

(Gaussian Mechanism of GDP [26]). Let $g:{\mathbb{R}}^{n\times m}\to {\mathbb{R}}^p$ be a function with finite global sensitivity $G{S}_g$. Let Z be a standard normal p-dimensional random vector. For all $\mu >0$ and $x\in {\mathbb{R}}^{n\times m}$, the random function $h\left(x\right)=g\left(x\right)+\frac{G{S}_g}{\mu }Z$ is $\mu -GDP$.

Lemma 2

(Matrix Gaussian mechanism [26]). Consider a data matrix $A\in {\mathbb{R}}^{n\times m}$ such that each row vector $a_i$ satisfies $\parallel a_i\parallel ⩽ 1$. Further define the function $h\left(A\right)=\frac{1}{n}{A}^{T}A$. Let W be a symmetric random matrix whose upper-triangular elements and the diagonal are $i.i.d.\frac{1}{\mu }N(0,1)$. Then the random function $\tilde{h}\left(A\right)=h\left(A\right)+W$ satisfies $\mu -GDP$.

Lemma 3

(Parallel Composition [27]). Suppose we have a set of privacy mechanisms $\mathbb{M}=\{{\mathbb{M}}_1,\cdots ,{\mathbb{M}}_m\}$. If each ${\mathbb{M}}_i$ provides a ${\mu }_i$-GDP guarantee on a disjointed subset of the entire dataset, $\mathbb{M}$ will provide $\left(\max \{{\mu }_1,\cdots ,{\mu }_m\}\right)$-GDP.

Lemma 4

(Composition of GDP [19]). The n-fold composition of ${\mu }_i-GDP$ mechanisms is $\sqrt{{\mu }_1^2+...+{\mu }_n^2}-GDP$.

2.3. GDP-LocalNewton Algorithm

In this subsection, we propose the GDP-LocalNewton algorithm for privacy-preserving distributed learning; see Algorithm 1.

Algorithm 1 GDP-LocalNewton

1: Input: Initial iteration ${\overline{\mathbf{w}}}_0\in {\mathbb{R}}^d$; Max linear search parameter ${\alpha }^{★}$; Linear search parameter $0<\beta ⩽1/2$; Privacy parameter $\mu$; Loss function parameter B and $\mathsf{\Gamma }$; Iteration parameter T; ${\mathcal{I}}_T\subseteq \{1,2,\dots ,T\}$ is the set of communication nodes. 2: for $k=1toK$ in parallel do 3:     Initialization: ${\mathbf{w}}_0^k={\overline{\mathbf{w}}}_0$ 4:     for $t=0toT-1$ do 5:         if $t\in {\mathcal{I}}_t$ then 6:             ${\overline{\mathbf{w}}}_t=\frac{1}{K}{\sum }_{k=1}^K{\mathbf{w}}_t^k$ 7:             ${\mathbf{w}}_t^k={\overline{\mathbf{w}}}_t$ 8:         end if 9:         ${\widehat{\mathbf{g}}}_t^k={\mathbf{g}}_t^k+\frac{2\mathsf{\Gamma }\sqrt{2T}}{\mu s}{Z}_t^k$ 10:       ${\widehat{\mathbf{H}}}_t^k={\mathbf{H}}_t^k+\frac{2B\sqrt{2T}}{\mu s}{U}_t^k$ 11:       ${\widehat{\mathbf{p}}}_t^k={\widehat{\mathbf{H}}}^k{\left({\mathbf{w}}_t^k\right)}^{-1}{\widehat{\mathbf{g}}}^k\left({\mathbf{w}}_t^k\right)$ 12:       Find step-size ${\alpha }_t^k$ using line search (Equation (10)) 13:       Update model: ${\mathbf{w}}_{t+1}^k={\mathbf{w}}_t^k-{\alpha }_t^k{\widehat{\mathbf{p}}}_t^k$ 14:    end for 15: end for

At the k-th worker in the t-th iteration, we define the local loss function (at the local iteration ${\mathbf{w}}_t^k$) as

$$f^k\left({\mathbf{w}}_t^k\right)=\frac{1}{s}\sum _{j\in {\mathcal{S}}_k}{f}_j\left({\mathbf{w}}_t^k\right).$$

(3)

The k-th worker’s target is to minimize the local loss in Equation (3). We give the corresponding local gradient ${\mathbf{g}}_t^k$ and local Hessian ${\mathbf{H}}_t^k$ at the k-th worker in the t-th iteration as the following:

$${\mathbf{g}}_t^k={\mathbf{g}}^k\left({\mathbf{w}}_t^k\right)=\nabla f^k\left({\mathbf{w}}_t^k\right)=\frac{1}{s}\sum _{j\in {\mathcal{S}}_k}\nabla f_j\left({\mathbf{w}}_t^k\right),$$

$${\mathbf{H}}_t^k={\mathbf{H}}^k\left({\mathbf{w}}_t^k\right)={\nabla }^2{f}^k\left({\mathbf{w}}_t^k\right)=\frac{1}{s}\sum _{j\in {\mathcal{S}}_k}{\nabla }^2{f}_j\left({\mathbf{w}}_t^k\right).$$

The updates of the local workers’ parameter are given by

$${\mathbf{w}}_{t+1}^k=\left\{\begin{array}{cc}{\mathbf{w}}_t^k-{\alpha }_t^k{\left({\widehat{\mathbf{H}}}_t^k\right)}^{-1}{\widehat{\mathbf{g}}}_t^k,\hfill & \mathrm{if}t\notin {\mathcal{I}}_t\hfill \\ {\overline{\mathbf{w}}}_t-{\alpha }_t^k{\widehat{\mathbf{H}}}^k{\left({\overline{\mathbf{w}}}_t\right)}^{-1}{\widehat{\mathbf{g}}}^k\left({\overline{\mathbf{w}}}_t\right),\hfill & \mathrm{if}t\in {\mathcal{I}}_t\hfill \end{array}\right.$$

(4)

where ${\overline{\mathbf{w}}}_t=\frac{1}{K}{\sum }_{k=1}{\mathbf{w}}_t^k$ and

$${\widehat{\mathbf{g}}}_t^k={\mathbf{g}}_t^k+\frac{2\mathsf{\Gamma }\sqrt{2T}}{\mu s}{Z}_t^k,$$

(5)

$${\widehat{\mathbf{g}}}^k\left({\overline{\mathbf{w}}}_t\right)={\mathbf{g}}^k\left({\overline{\mathbf{w}}}_t\right)+\frac{2\mathsf{\Gamma }\sqrt{2T}}{\mu s}{Z}_t^k,$$

(6)

and

$$\begin{array}{c}\hfill {\widehat{\mathbf{H}}}_t^k={\mathbf{H}}_t^k+\frac{2B\sqrt{2T}}{\mu s}{U}_t^k,\end{array}$$

(7)

$$\begin{array}{c}\hfill {\widehat{\mathbf{H}}}^k\left({\overline{\mathbf{w}}}_t\right)={\mathbf{H}}^k\left({\overline{\mathbf{w}}}_t\right)+\frac{2B\sqrt{2T}}{\mu s}{U}_t^k,\end{array}$$

(8)

then,

$${\widehat{\mathbf{p}}}_t^k={\left({\widehat{\mathbf{H}}}_t^k\right)}^{-1}{\widehat{\mathbf{g}}}_t^k,$$

(9)

where $Z_t^k$ is a sequence of i.i.d. standard d-dimensional Gaussian random vectors and $U_t^k$ is a sequence of i.i.d. symmetric random matrices, whose upper triangular elements and the diagonals are i.i.d. standard normal randoms. $\mu$ is the privacy parameter, B is the norm bound of the Hessian matrix and $\mathsf{\Gamma }$ is the norm bound of the gradient. ${\mathcal{I}}_T$ is the set of communication nodes, where every user communicates their own updated parameters to the server, T represents the set of iterations, and L means the number of local computations. We have $\left|T\right|=⌈|{\mathcal{I}}_T| \times L⌉$, e.g., $T=\{1,2,...,L,...,2L,...,⌈|{\mathcal{I}}_T| \times L⌉\}$. In particular, if $L=1$, this means no local computation.

Remark 1.

In practice, ${\widehat{\mathbf{H}}}^k\left({\overline{\mathbf{w}}}_t\right)$ may not be positive definite. So we need to truncate the eigenvalues as $\max \{{\lambda }_j,ϵ\}$, where $ϵ>0$ is equal to the regularization parameter. The matrix is also differentially private as a result of the post-processing property of GDP ([19], Proposition 2.8).

Here, we propose a novel step selection, coping with the negative influence of noises, and it plays an important role in the convergence of GDP-LocalNewton.

Step-size selection: Let each worker locally choose a step size according to the following rule:

$${\alpha }_t^k=\underset{\alpha ⩽{\alpha }^{★}}{\max }\alpha \mathrm{such}\mathrm{that}$$

$$f^k\left({\mathbf{w}}_t^k-\alpha {\widehat{\mathbf{p}}}_t^k\right)⩽ f^k\left({\mathbf{w}}_t^k\right)-\alpha \beta {\left({\widehat{\mathbf{p}}}_t^k\right)}^T{\mathbf{g}}_t^k+\alpha \gamma {\left({\widehat{\mathbf{p}}}_t^k\right)}^T{\mathbf{H}}_t^k{\tilde{N}}_t^k,$$

(10)

for some $\beta \in (0,1/2]$ and $\gamma =1-\beta$ so that

$${\alpha }^{★}⩽\min \left\{\frac{2(1-\beta )\kappa (1-ϵ)}{M(1+ϵ)},\frac{2\beta {\kappa }^2}{3M[M-\kappa /4]L}\right\}<1,$$

where ${\tilde{N}}_t^k={({\widehat{\mathbf{H}}}_t^k)}^{-1}{\widehat{\mathbf{g}}}_t^k-{({\mathbf{H}}_t^k)}^{-1}{\mathbf{g}}_t^k$, M and $\kappa$ are the parameters of the functional smooth and strong convex, proposed at the next subsection.

Remark 2.

In the subsequent theoretical section, we can observe the advantages and significance of the linear search in the algorithm’s convergence analysis. In part of the experiment, we will compare our strategy with the fixed step size and decaying step size. The results show that our strategy is optimal. The decaying step size refers to compressing the step size of the previous iteration at each iteration, which means ${\alpha }_{t+1}={\sigma }^{t+1}{\alpha }_t$, where σ is the decaying rate.

2.4. Theoretical Results

2.4.1. Assumptions on the Loss Functions

We need the following assumptions on the loss functions.

Assumption 1.

$f_i(·)$, for all $i\in \left[n\right]$, is twice differentiable.

Assumption 2.

$f(·)$ is κ-strongly convex, that is, ${\nabla }^{2}f\left(\mathbf{w}\right)\succcurlyeq \kappa \mathbf{I}.$

Assumption 3.

$f(·)$ is M-smooth, that is, ${\nabla }^{2}f\left(\mathbf{w}\right)\preccurlyeq M\mathbf{I}.$

Assumption 4.

For $i\in \left[n\right]$, $\parallel {\nabla }^2{f}_i{(·)\parallel }_2\mathit{and}{\parallel \nabla f_i(·)\parallel }_2$ are upper bounded such that ${\nabla }^2{f}_i\left(\mathbf{w}\right)\preccurlyeq B\mathbf{I}$ and $\parallel \nabla f_i{(·)\parallel }_2⩽\mathsf{\Gamma }$, where $B\in \mathbb{R}$ and $\mathsf{\Gamma }\in \mathbb{R}$ are some fixed constants.

2.4.2. Privacy

Theorem 1 shows that GDP-LocalNewton satisfies $\mu -GDP$.

Theorem 1.

Assuming Assumptions 1–4 hold, the output of the k-worker (4) satisfies $\frac{\mu }{\sqrt{T}}-GDP$. Furthermore, after T iterations, the whole algorithm satisfies $\mu -GDP$.

Proof. 

Firstly, we compute the global sensitivity of $s{\mathbf{g}}_t^k$ and $s{\mathbf{H}}_t^k$ as the following:

$$G{S}_{s{\mathbf{g}}_t^k}= \parallel s{\mathbf{g}}_t^k\left(x_{1:n}^{\prime }\right)-s{\mathbf{g}}_t^k\left(x_{1:n}\right)\parallel = \parallel \nabla f_j(·)-\nabla f_{j^{\prime }}(·)\parallel ⩽2\mathsf{\Gamma },$$

(11)

$$G{S}_{s{\mathbf{H}}_t^k}= \parallel s{\mathbf{H}}_t^k\left(x_{1:n}^{\prime }\right)-s{\mathbf{H}}_t^k\left(x_{1:n}\right)\parallel = \parallel {\nabla }^2{f}_j(·)-{\nabla }^2{f}_{j^{\prime }}(·)\parallel ⩽2B,$$

(12)

where we use Assumption 4, and j and $j^{\prime }$ represent different datasets. Then, using Lemma 4, we know that the output of the k-worker (4) satisfies $\frac{\mu }{\sqrt{T}}-GDP$. Finally, through Lemmas 3 and 4, the whole algorithm satisfies $\mu -GDP$. The proof is completed.    □

2.4.3. Convergence Analysis

The following three lemmas show that we can separate the Gaussian noises added in the algorithm and bound them so as to guarantee the convergence of GDP-LocalNewton.

Lemma 5

([26]). Let $X\in {\mathbb{R}}^d$ be a sub-Gaussian random vector with variance proxy ${\sigma }^2$. For any $\alpha >0$, with probability at least $1-\alpha$,

$${\parallel X\parallel }_2⩽4\sigma \sqrt{d}+2\sigma \sqrt{2\log (1/\alpha )}.$$

Lemma 6

([26]). Let W be a symmetric $d\times d$ random matrix whose upper triangular elements, including the diagonal, are i.i.d. $N(0,1)$. For any $\alpha >0$, with probability at least $1-\alpha$,

$${\parallel W\parallel }_2⩽\sqrt{2d\log (2d/\alpha )}$$

Lemma 7.

Assumptions $1-4$ hold. Fix the total iteration T and the GDP parameter μ, let $s⩾\frac{4B\sqrt{2T}\sqrt{2d\log (4d/{\xi }_0)}}{\kappa (1-ϵ)\mu }$, and use Lemma 1 and Lemma 2, then we can rewrite the noisy term in (4) as

$${({\widehat{\mathbf{H}}}_t^k)}^{-1}{\widehat{\mathbf{g}}}_t^k={({\mathbf{H}}_t^k)}^{-1}{\mathbf{g}}_t^k+{\tilde{N}}_t^k.$$

Here, with probability at least $1-{\xi }_0$, we have

$$\parallel {\tilde{N}}_t^k\parallel ⩽C^{\prime }{\tilde{M}}_{privacy},$$

(13)

where ${\tilde{M}}_{privacy}=\frac{\mathsf{\Gamma }B\sqrt{Td\log (d/{\xi }_0)}}{\mu s{\kappa }^2{(1-ϵ)}^2}$ and $C^{\prime }$ is a constant.

Proof. 

See Appendix B.1.    □

Lemma 8.

Let the function $f(·)$ satisfy Assumptions $1-3$, and suppose that step size ${\alpha }_t^k$ satisfies the line-search condition in (10). Also, let $0<ϵ<1/2$, $0<\delta ,{\xi }_0<1$, μ and T be fixed constants. Moreover, let the sample size $s⩾\frac{4B}{\kappa {ϵ}^2}\log \frac{2d}{\delta }$. Then, defined in Equation (4), at the k-th worker, we have

$$f^k\left({\mathbf{w}}_{t+1}^k\right)-f^k\left({\mathbf{w}}_t^k\right)⩽-\psi \parallel {\mathbf{g}}_t^k{\parallel }^2+ {\alpha }^{★}(1-2\beta ){\left({\mathbf{p}}_t^k\right)}^T{\mathbf{H}}_t^k{\tilde{N}}_t^k+{\alpha }^{★}(1-\beta )M(1+ϵ){\parallel {\tilde{N}}_t^k\parallel }^2,$$

with probability of at least $1-\delta$.

If $s⩾\frac{4B\sqrt{2T}\sqrt{2d\log (4d/{\xi }_0)}}{\kappa (1-ϵ)\mu }$, $\beta =1/2$ and using Lemma 7, we obtain

$$f^k\left({\mathbf{w}}_{t+1}^k\right)-f^k\left({\mathbf{w}}_t^k\right)⩽-\psi {\parallel {\mathbf{g}}_t^k\parallel }^2+\frac{{\alpha }^{★}}{2}M(1+ϵ){\left(C^{\prime }{\tilde{M}}_{privacy}\right)}^2,$$

with at least probability $1-\delta -{\xi }_0$, where $\psi =\frac{{\alpha }^{★}\beta }{M(1+ϵ)}$. Note that ${\mathbf{w}}_{t+1}^k={\mathbf{w}}_t^k-\alpha {\widehat{\mathbf{p}}}_t^k$, which means the workers’ local gradient. ${\tilde{N}}_t^k$, ${\tilde{M}}_{privacy}$ and $C^{\prime }$ are from Lemma 7.

Proof. 

See Appendix B.1.    □

Remark 3.

In Lemma 8, we can find that the latter two terms come from the noise by DP. The second term can be eliminated by setting the $\beta =1/2$ and it is the merit of our new linear selection that can reduce the random negative effectiveness. As a result, Lemma 8 shows that the novel linear selection can improve the convergence of the algorithm. The small $\parallel {\tilde{N}}_t^k{\parallel }^2$ leads to the small third term.

Theorem 2.

/$L=1$ case/Suppose that Assumptions $1-4$ hold, and the step-size ${\alpha }_t^k$ satisfies the line-search condition (10). Also, let T, μ, $0<\delta ,{\xi }_0<1,0<ϵ,{ϵ}_1<1/2$ be fixed constants and let $\beta =1/2$, $\mathsf{\Gamma }={\max }_{1⩽i⩽n}\parallel \nabla f_i(·)\parallel$. Moreover, assume that the sample size for each worker satisfies $s⩾\frac{4B}{\kappa {ϵ}^2}\log \frac{2d}{\delta }$, where the samples are chosen without replacement. When the GDP-LocalNewton is in the $L=1$ case, we obtain the following:

1.

If $s⩾\max \left\{\frac{4B\sqrt{2T}\sqrt{2p\log (4p/{\xi }_0)}}{\kappa (1-ϵ)\mu },\frac{{\mathsf{\Gamma }}^2}{{ϵ}_1^2{G}^2}\log (d/\delta )\right\}$ for $G={\min }_k\parallel {\mathbf{g}}^k\left({\overline{\mathbf{w}}}_t\right)\parallel$, we obtain, with probability at least $1-K(6\delta +{\xi }_0)$,

$$f\left({\overline{\mathbf{w}}}_{t+1}\right)-f\left({\overline{\mathbf{w}}}^{*}\right)⩽{\rho }_1(f\left({\overline{\mathbf{w}}}_t\right)-f\left({\overline{\mathbf{w}}}^{*}\right))+{\tilde{M}}_{\widehat{\phi }},$$

(14)

2.

If $s⩾\frac{2B\sqrt{2T}\sqrt{2p\log (4p/{\xi }_0)}}{\kappa (1-ϵ)\mu }$, we obtain, with probability at least $1-K(6\delta +{\xi }_0)$,

$$f\left({\overline{\mathbf{w}}}_{t+1}\right)-f\left({\overline{\mathbf{w}}}^{*}\right)⩽{\rho }_2(f\left({\overline{\mathbf{w}}}_t\right)-f\left({\overline{\mathbf{w}}}^{*}\right))+{\tilde{M}}_{\phi }+\frac{\eta \mathsf{\Gamma }}{\kappa (1-ϵ)}.$$

(15)

Note that ${\rho }_i=1-2\kappa C_i$, for $i=\{1,2\}$, $C_1=\left(\psi -\frac{{ϵ}_1}{\kappa (1-ϵ)}-\frac{[M-\kappa (1-ϵ)]{({\alpha }^{★})}^2}{{\kappa }^2{(1-ϵ)}^2}\right)$, $C_2=\left(\psi -\frac{[M-\kappa (1-ϵ)]{({\alpha }^{★})}^2}{{\kappa }^2{(1-ϵ)}^2}\right)$, $\psi =\frac{{\alpha }^{★}\beta }{M(1+ϵ)}$, $\eta =(1+\sqrt{2\log \left(\frac{1}{\delta }\right)})\sqrt{\frac{1}{s}}\mathsf{\Gamma }$. ${\tilde{M}}_{\widehat{\phi }}={ϵ}_1\left(C^{\prime }{\tilde{M}}_{privacy}\right)\mathsf{\Gamma }+\{\frac{1}{2}M(1+ϵ)+[M-\kappa (1-ϵ)]{\alpha }^{★}\}\left({\alpha }^{★}\right){\left(C^{\prime }{\tilde{M}}_{privacy}\right)}^2$ and ${\tilde{M}}_{\phi }=\eta \left(C^{\prime }{\tilde{M}}_{privacy}\right)+\{\frac{1}{2}M(1+ϵ)+[M-\kappa (1-ϵ)]{\alpha }^{★}\}\left({\alpha }^{★}\right){\left(C^{\prime }{\tilde{M}}_{privacy}\right)}^2$ are constants. $C^{\prime }{\tilde{M}}_{privacy}$ is from Lemma 7.

Proof. 

See Appendix C.1 and C.2.    □

Remark 4.

Compared to LocalNewton when $L=1$ ([11]), the protection of privacy leads to an extra error term ${\tilde{M}}_{\widehat{\phi }}$. The error term shows the trade-off between the protection of privacy and the usefulness of the model. If we want to decrease the error term ${\tilde{M}}_{\widehat{\phi }}$, it needs more data in every worker.

In this theorem, we choose $\beta =1/2$ so that the second random term in the Lemma 8 is eliminated. That is because the less random terms commonly make the algorithm more stable.

Theorem 3.

/$L>1$ case/. Suppose Assumptions $1-4$ hold and step size ${\alpha }_t^k$ satisfies the line search condition (10). Also, let T, μ, $0<\delta ,{\xi }_0<1,0<ϵ<1/2$ be fixed constants and let $\mathsf{\Gamma }=\underset{1⩽i⩽n}{max}\parallel \nabla f_i(·)\parallel$. Moreover, assume that the sample size for each worker satisfies $s⩾\max \{\frac{4B}{\kappa {ϵ}^2}\log \frac{2d}{\delta },\frac{2B\sqrt{2T}\sqrt{2d\log (4d/{\xi }_0)}}{\kappa (1-ϵ)\mu }\}$. Then, the GDP LocalNewton updates, ${\overline{\mathbf{w}}}_{t⩾0}$, within the case of $L>1$, satisfy

$$f\left({\overline{\mathbf{w}}}_{t_0+1}\right)-f\left({\overline{\mathbf{w}}}_{t_0}\right)⩽-\frac{C}{K}\sum _{k=1}^K\sum _{\tau =t_0}^L{\parallel {\mathbf{g}}_{\tau }^k\parallel }^2+\frac{\eta L\mathsf{\Gamma }}{\kappa (1-ϵ)}+L{\tilde{M}}_{\phi },$$

(16)

with probability at least $1-KL(6\delta +{\xi }_0)$. Here, $C=\psi -\frac{L[M-\kappa (1-ϵ)]{\left({\alpha }^{*}\right)}^2}{{\kappa }^2{(1-ϵ)}^2}$, $\psi =\frac{{\alpha }^{★}\beta }{M(1+ϵ)}$ and $\eta =(1+\sqrt{2\log \left(\frac{1}{\delta }\right)})\sqrt{\frac{1}{s}}\mathsf{\Gamma }$. ${\tilde{M}}_{\phi }=\eta \left(C^{\prime }{\tilde{M}}_{privacy}\right)+\{\frac{1}{2}M(1+ϵ)+[M-\kappa (1-ϵ)]{\alpha }^{★}\}\left({\alpha }^{★}\right){\left(C^{\prime }{\tilde{M}}_{privacy}\right)}^2$ is a constant and $C^{\prime }{\tilde{M}}_{privacy}$ is from Lemma 7.

Proof. 

See Appendix C.3.    □

3. Empirical Evaluation

In this section, we evaluate the numerical performance of GDP-LocalNewton under $L=1$ and $L>1$, respectively. In addition, we design experiments to explore the performance of different step size strategies, such as the linear search step size, fixed step size, and decaying step size. In particular, we conduct both the simulation and the real data experiments.

The simulated datasets are generated according to the logistic model. Supposing $d=10,n=50,000$, each entry of the model parameter vector $w=(w_1,...,w_{10})$ is generated i.i.d. from $U(-0.5,0.5)$. For each sample, each entry of the predictor vector is generated i.i.d. from the standard Gaussian. The labels are then generated from the logistic regression. We also generate 10,000 samples for testing.

The real datasets we use are summarized in

Table 2. The real datasets in this paper.

Dataset	Training Samples (n)	Features (d)	Testing Samples
a9a	32,000	123	16,000
Covtype	500,000	54	80,000
ijcnn1	49,000	22	91,000

, which are publicly available in LIBSVM [28].

3.1. Without Local Computation ($L=1$)

We fix the total clients K to be 50. Hence, the number of samples per client s is $n/50$. We fix the total communication rounds T as 10. The regularization parameter is 0.001. We set the privacy parameter $\mu =1$, 5 and 10, respectively. We compare our method with GDP-GD algorithm with $\mu =1$ and LocalNewton with $\mu =\infty$. The details of GDP-GD can be found in Appendix D. Note that we set the max step size ${\alpha }^{★}$ of GDP-LocalNewton to be the fixed step size of GDP-GD. This is because, due to noise accumulation, the step sizes of algorithms perturbed by GDP should be much smaller than those without DP perturbation in order to ensure convergence.

Figure 1 and Figure 2 display the training loss and test accuracy of each method on one simulated dataset and three real datasets, where the training loss is the empirical average of the loss function evaluated at the training data points, and the test accuracy is the proportion of test samples that are correctly classified. As Figure 1 and Figure 2 show, GDP-LocalNewton with different $\mu$ converges much faster than GDP-GD with $\mu =1$ in all datasets. It means that our privacy-preserving second-order algorithm is much better than the privacy-preserving first-order algorithm GDP-GD. We can also see that as the the privacy parameter $\mu$ increases, the performance of GDP-LocalNewton increases, especially when the communication round is large.

3.2. With Local Computation ($L>1$)

We use the a9a and the ijcnn1 datasets to show how different values of L influence the performance of GDP-LocalNewton. We set the GDP parameter $\mu =2$ and $\mu =4$ for the a9a dataset and set $\mu =1$ and $\mu =2$ for the ijcnn1 dataset, the local computation parameter $L=1$ and $L=3$, and regularization $\gamma =0.5$. The communication round is 8. The other parameter set-ups are the same as those of the former experiments.

As Figure 3 shows, GDP-LocalNewton converges in all datasets. When the values of $\mu$ are equal, $L=3$ can speed up the convergence of the algorithm. That means the noises do not influence the speed of convergence too much. At the 7-th communication of the algorithm, the error from the privacy protection is bigger than the error of the local computation. That means that the privacy protection influences the usefulness of GDP-LocalNewton more than the local computation.

3.3. Different Step Size Strategies

In this section, we explore how different step size strategies, including linear-search step size (${\alpha }^{★}=0.03$), fixed step size (fix ${\alpha }_t^k=0.03$), and decaying step size (the initial step size $\alpha =0.03$), impact GDP-LocalNewton under different levels of privacy protection(different $\mu$ values). In addition, we set different decaying rates to show its impact. Under $L=1$, we use the simulation and the a9a data. Due to the varying characteristics of the datasets, we made different parameter settings for some of them. In the generated data, we set the $\mu =1$ and 5 and a better decaying rate $\sigma$ as $0.5$. In the a9a data, we set the $\mu =5$ and 10 and a better decaying rate $\sigma$ as $0.9$.

As Figure 4 and Figure 5 show, the linear-search step size performs well and stably in every case. This validates the effectiveness of our strategy both theoretically and experimentally. The reason for the similarity between the fixed step size and the search step size is that the noise causes some of the search steps to be equal to the maximum step size set, making them equivalent to the fixed step size. For the decaying step size, the experiments show that the decaying rate is very important for the convergence. An appropriate decaying rate can help the algorithm converge. In the later stages of the algorithm, decaying to a smaller step size helps resist the negative impact of noise on the algorithm and further promotes convergence, such as Figure 4a and Figure 5a. An excessive decaying rate leads to algorithm non-convergence, like the $0.1$ decaying rate in every figure. However, in practice, adjusting the decaying rate will increase the operational costs and raise the risk of privacy leakage. Therefore, the linear search strategy is more effective, efficient, and stable.

4. Conclusions

In this paper, we proposed a novel algorithm called GDP-LocalNewton for privacy-preserving and communication-efficient distributed learning. To improve the communication efficiency, we developed the algorithm based on Newton’s method and traded more local computations between communications. To handle possible privacy leakage via the curious onlooker, we adopted the notion of GDP [19] by adding Gaussian noise to the updates of each local machine. In particular, we developed a step searching strategy to determine the step size in the noisy Newton update. We validated the effectiveness, efficiency, and stability of our strategy through experiments. We theoretically studied the convergence of GDP-LocalNewton, which turns out to have two error terms, one corresponding to the privacy protection and the other corresponding to the local computation. The experiments corroborated the theoretical findings.

There are some interesting problems that deserve further study. First, the DP framework would lead to strong privacy protection; however, a DP algorithm often lead to much accuracy loss compared with its non-DP counterpart. Therefore, it is desirable to consider other weaker notions of privacy [29]. Second, how to handle the heterogeneity among local datasets algorithmically and theoretically in the second-order-based framework deserves further study. Finally, it is worth tackling other issues or challenges that FL faces within the second-order-based framework.