Riemannian SVRG Using Barzilai–Borwein Method as Second-Order Approximation for Federated Learning

Abstract

In this paper, we propose a modified RFedSVRG method by incorporating the Barzilai–Borwein (BB) method to approximate second-order information on the manifold for Federated Learning (FL). Moreover, we use the BB strategy to obtain self-adjustment of step size. We show the convergence of our methods under some assumptions. The numerical experiments on both synthetic and real datasets demonstrate that the proposed methods outperform some used methods in FL in some test problems.

1. Introduction

Federated Learning (FL) [1] has wide applications in machine learning. In this paper, we consider FL as the following form [1,2]:

$$\underset{x\in \mathcal{M}}{min}f\left(x\right):={\displaystyle \frac{1}{n}}\sum _{i=1}^n{f}_i\left(x\right),$$

(1)

where $\mathcal{M}$ is the Riemannian manifold, the loss function $f_i:\mathcal{M}\to \mathbb{R}$ is the local loss function stored in different a client/agent, f is the Lipschitz smooth global objective function and n is the number of clients. Thus, all clients cannot connect to each other [2]. Therefore, there is a central sever that can collect the information to output a consensus that minimize the summation of loss functions from all the clients. This framework makes use of computing resources from different departments while keeping the data private. There is no requirement to share data between them, so there are communications only between the central server and agents.

Some motivating applications of (1) including the Karcher mean problem on the positive define cone (PSD Karcher mean) [3,4]

$$\underset{X\succ 0}{min}f\left(X\right):={\displaystyle \frac{1}{n}}\sum _{i=1}^n{f}_i\left(X\right),$$

(2)

where $f_i\left(X\right)={\parallel log\left(X^{-1/2}{A}_i{X}^{-1/2}\right)\parallel }_F^2$, $\left\{X|X\succ 0\right\}$ denotes symmetric positive-definite matrices, $A_i⪰0$ is the covariance matrix of the data stored on the i-th local agent and the objective function, f, is strongly geodesically convex.

For non-convex objective problems, the examples include the classical Principal Component Analysis (PCA) problem [5]

$$\underset{\{x\in {\mathbb{R}}^d|\parallel x{\parallel }_2=1\}}{min}f\left(x\right):={\displaystyle \frac{1}{n}}\sum _{i=1}^n{f}_i\left(x\right),$$

(3)

where $f_i\left(x\right)=-{\displaystyle \frac{1}{2}}{x}^{\top }{A}_{i}x$.

When $x\in {\mathbb{R}}^{d\times r}$ with $r>1$, (3) evolves into the federated kernel PCA (kPCA) problem [3,6]

$$\underset{X\in {\mathrm{S}}_t(d,r)}{min}f\left(X\right):={\displaystyle \frac{1}{n}}\sum _{i=1}^n{f}_i\left(X\right),$$

(4)

where $f_i\left(X\right)=-{\displaystyle \frac{1}{2}}Tr\left(X^{\top }{A}_{i}X\right)$, ${\mathrm{S}}_t(d,r)=\left\{X\in {\mathbb{R}}^{d\times r}|X^{\top }X=I_r,r\le d\right\}$ denotes the Stiefel manifold and $A_i$ is the covariance matrix of the data stored in the i-th local agent.

Manifold optimization problems in FL of the form (1) also appear in many machine learning tasks, such as low-rank matrix completion [7], diffusion tensor imaging [8,9,10], elasticity theory [11], Electroencephalography (EEG) classification [12] and deep neural network training [13,14]. Still, there are very few federated algorithms on manifolds. In fact, work [3] appears to be the only FL algorithm that can deal with manifold optimization problems with a similar generality as ours.

Handling manifold constraints in an FL setting poses primary obstacles. (i) Existing single-machine methods for manifold optimization [15,16,17] cannot be directly adapted to the federated setting. The server has to average the clients’ local models due to the distributed framework. Because of the non-convexity of $\mathcal{M}$, their average is typically outside of $\mathcal{M}$. (ii) Extending typical FL algorithms to scenarios with manifold constraints is not straightforward. Most existing FL algorithms either are unconstrained [1,2,18,19,20,21,22] or only allow for convex constraints [23,24,25,26], but manifold constraints are typically non-convex. (iii) Compared with non-convex optimization in Euclidean space, manifold optimization necessitates the consideration of the geometric structure of the manifold and properties of the loss functions, which poses challenges for algorithm design and analysis. (iv) Due to manifold constraints, it is difficult to extend some techniques [21,22] for enhancing communication efficiency originally developed for Euclidean spaces to manifold space. For more details, we refer to reference [27].

1.1. Related Work

As $\mathcal{M}={\mathbb{R}}^d$, there are many works on FL algorithms [1,2,18,19,20,21,22,23,24,25,26,28]. The most widely used algorithm is FedAvg [1], which applies two loops of iterations. One is an inner loop for local clients and the other is an outer loop for the server. The inner loop in FedAvg for each client $i\in S_t$ in parallel is as follows

$$x_{\ell +1}^{\left(i\right)}=x_{\ell }-{\eta }^{\left(i\right)}\nabla f_i\left(x_{\ell }^{\left(i\right)}\right),\forall i\in S_t,$$

(5)

where $S_t\subset \left[n\right]$ is uniformly sampled, $\ell \in \{0,1,\cdots ,{\tau }_i-1\}$, ${\eta }^{\left(i\right)}$ is a constant called the learning rate and $x_0^{\left(i\right)}=x_t$ is received from the server. Then, FedAvg averages local gradient descent updates by requiring the server to aggregate the $x_{\ell }^{\left(i\right)}$ in outer loops as

$$x_{t+1}={\displaystyle \frac{1}{k}}\sum _{i=1}^k{x}_{{\tau }_i}^{\left(i\right)},\forall t\in \{0,1,\cdots ,T-1\},$$

(6)

which makes a good empirical convergence. However, FedAvg suffers from the client drift effect, where each client will drift the solution towards the minimum of their own loss function [18]. To alleviate this issue, several papers provide efforts to improve FedAvg. FedProx [18] ensured that the local iterates are not far from the previous consensus point by regularizing each local gradient descent update. Later, FedSplit [19] used operator splitting technology to deal with data heterogeneity, which also reduces the client drift effect. FedNova [20] proposed normalizing the averaged local gradients, but this suffers from a fundamental speed–accuracy conflict under objective heterogeneity [21].

On the other hand, variance reduction techniques are commonly used in client training on local data, which alleviate the data heterogeneity and derive the Federated SVRG (FSVRG) [2] method. FSVRG uses the SVRG method [29] in local updates for clients:

$$x_{\ell +1}^{\left(i\right)}=x_{\ell }-{\eta }^{\left(i\right)}\left(\nabla f_i\left(x_{\ell }^{\left(i\right)}\right)+\nabla f\left(x_t\right)-\nabla f_i\left(x_t\right)\right),\forall i\in S_t,$$

(7)

where $x_0^{\left(i\right)}=x_t$, and the server aggregates the $x_{\ell }^{\left(i\right)}$ as

$$x_{t+1}=x_t+{\displaystyle \frac{1}{k}}\sum _{i=1}^k\left(x_{\tau }^{\left(i\right)}-x_t\right),\mathrm{where}t\in \{0,1,\cdots ,T-1\}.$$

(8)

Compared with method (6), the aggregation method (8) ensures that the next consensus, $x_{t+1}$, is not too far from the previous consensus, $x_t$. Later, FedLin [21] and SCAFFOLD [22] proposed the methods that use variance reduction techniques without calculating the full gradients, $\nabla f\left(x_t\right)$.

Since the aggregation points of non-convex sets tend to be outside of the sets, FL problems with non-convex constraints are rarely be considered [3]. Moreover, the unconvex constraint problems can be transformed into unconstraint problems on the manifold as from (1). Then we can apply many convenient ways to solve the original problems. Ref. [3] proposed the RFedSVRG algorithm, which is the first algorithm to solve FL problems on Riemannian manifolds with convergence guarantee and suitable for solving cases with non-convex constraints. In order to fit the iteration on manifolds, RFedSVRG [3] uses exponential mapping and parallel transport with the SVRG method [29] in local updates:

$$\begin{array}{c}\hfill x_{\ell +1}^{\left(i\right)}={Exp}_{x_{\ell }^{\left(i\right)}}\left\{{\eta }^{\left(i\right)}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}\left(gradf\left(x_t\right)-grad{f}_i\left(x_t\right)\right)\right]\right\},\forall i\in S_t,\\ \hfill \mathrm{where}\ell \in \{0,1,\cdots ,{\tau }_i-1\}.\end{array}$$

(9)

RFedSVRG also requires that the server aggregates the $x_{{\tau }_i}^{\left(i\right)}$ as the tangent space mean

$$x_{t+1}={Exp}_{x_t}\left({\displaystyle {\displaystyle \frac{1}{k}}\sum _{i\in S_t}{Exp}_{x_t}^{-1}\left(x_{{\tau }_i}^{\left(i\right)}\right)}\right),\mathrm{where}t\in \{0,1,\cdots ,T-1\},$$

(10)

which has the “regularization” property $d(x_{t+1},x_t)\le {\displaystyle \frac{1}{k}}{\sum }_{i\in S_t}d(x_{{\tau }_i}^{\left(i\right)},x_t)$, so that the distance between two consensus points can be controlled. Work [30] explores the differential privacy of RFedSVRG. Ref. [27] employs a projection operator and constructs the correction terms in local updates to reduce the computational costs of RFedSVRG. Ref. [31] considers the specific manifold optimization problem that appears in PCA and investigates an ADMM-type method that penalizes the orthogonality constraint.

Recently, many attempts have been devoted to SVRG with second-order information, which can further reduce the variance. It motivated the development of the Hessian-based stochastic variance reduced gradient (SVRG-2) [32] method. SVRG-2 incorporates the second-order information to further reduce the variance via the variance reduction technique of the stochastic gradient as

$$x_{\ell +1}^{\left(i\right)}=x_{\ell }^{\left(i\right)}-{\eta }^{\left(i\right)}\left[\nabla f_i\left(x_{\ell }^{\left(i\right)}\right)+\nabla f\left(x_t\right)-\nabla f_i\left(x_t\right)+\left({\nabla }^{2}f\left(x_t\right)-{\nabla }^2{f}_i\left(x_t\right)\right)\left(x_{\ell }^{\left(i\right)}-x_t\right)\right].$$

(11)

SVRG-2 has been shown to provide better variance reductions than standard SVRG and requires only a small number of epochs to converge. However, Ref. [33] states that the computation of Hessian requires $\mathcal{O}\left(n{d}^2\right)$ time and space complexity. Later, Ref. [33] proposed the the Barzilai–Borwein approximation to further control the variance with lower computational cost as a variant of SVRG-2 (SVRG-2BB) by using the BB method.

Moreover, one important issue for stochastic algorithms and their variants is how to choose an appropriate step size, ${\eta }_t^{\left(i\right)}$, while running the algorithm [34]. In the classical gradient descent method, the line search technique is usually used to obtain the step size. But in the stochastic gradient method, Ref. [35] states that line search is not possible because there is only downsampling information of the function value and gradient. One common method for stochastic algorithms is to to tune a best fixed step size by hand, but it is time consuming [35]. The Barzilai and Borwein [36] (BB) method is a nice choice for us to select the step length, as it can automatically calculate the step size by using the information of the iteration point and the gradient of the function. SVRG-2BBS [33] and SVRG-BB [35] incorporate the BB step size into the SVRG-2BB and SVRG, respectively. These two algorithms further improve the performance of the original algorithms and obtain the linear convergence for the strongly convex objective function.

Moreover, Refs. [37,38,39] have given the BB step size only on the Stiefel manifold with global convergence in solving the minimization problem. Later, Ref. [40] extended the Euclidean BB step size to general manifold spaces, whose numerical results show a good performance with this method.

1.2. Our Contributions

In this paper, our goal is to propose a modified RFedSVRG that incorporates second-order information on the manifold in order to save computation. We use the BB method to reduce the variance, which introduces a new algorithm, RFedSVRG-2BB, on the manifold. In addition, we propose to incorporate the BB step size into RFedSVRG-2BB (RFedSVRG-2BBS). RFedSVRG-2BBS implements automatic calculation of the step size in RFedSVRG-2BB with a faster convergence speed in numerical experiments. It also preserves the global convergence property of RFedSVRG theoretically.

We list the contributions of this paper as follows:

• We propose the Barzilai–Borwein approximation as second-order information to control the variance with lower computational cost (RFedSVRG-2BB). In addition, we incorporate the Barzilai–Borwein step size for RFedSVRG-2BB and lead the RFedSVRG-2BBS.
• We present the convergence results and the corresponding convergence rate of the proposed methods for the strongly geodesically convex objective function and non-convex objective function, respectively.
• We conduct numerical experiments for the proposed methods on solving the PCA, kPCA and PSD Karcher mean problems in some datasets. The numerical results show that our methods are better than RFedSVRG and other algorithms.

The paper is organized as follows. In Section 2, we recall some basic concepts from Riemannian optimization. In Section 3, we first introduce the framework of RFedSVRG and the BB method. Then we propose the RFedSVRG-2BB and RFedSVRG-2BBS algorithms. In Section 4, we provide the convergence analysis of our algorithms. In Section 5, we present the comparing of results of the numerical experiments for the proposed methods and some existing methods in Federated Learning. The conclusions are drawn in Section 6.

2. Preliminaries on Riemannian Optimization

We first briefly review some of the basic concepts related to Riemannian optimization. More details are shown in [4,16,41,42,43,44].

Denote $\mathcal{M}$ is a Riemannian manifold. The tangent space of $x\in \mathcal{M}$ is denoted by $T_x\mathcal{M}$. The inner product on $T_x\mathcal{M}$ is defined as ${\langle ·,·\rangle }_x$ and the corresponding induced norm of $\xi \in T_x\mathcal{M}$ is $\parallel \xi \parallel \triangleq \sqrt{{\langle \xi ,\xi \rangle }_x}$. The Riemannian gradient of a function $f\in C^1\left(\mathcal{M}\right)$ in x is denoted as $gradf\left(x\right):\mathcal{M}\to T_x\mathcal{M}$, which is the unique tangent vector in $T_x\mathcal{M}$, such that

$${\langle gradf\left(x\right),\xi \rangle }_x=\mathrm{D}f\left(x\right)\left[\xi \right]=\langle \nabla f\left(x\right),\xi \rangle ,\forall \xi \in T_x\mathcal{M},$$

where $\mathrm{D}f\left(x\right)\left[\xi \right]$ denotes the directional derivative along the $\xi$ direction [41] and $\langle ·,·\rangle$ is the Euclidean inner product defined by $\langle \xi ,\zeta \rangle =Tr\left({\xi }^{\top }\zeta \right)$. We also can define the angle between $\xi ,\zeta \in T_x\mathcal{M}$ as $arccos{\displaystyle \frac{{\langle \xi ,\zeta \rangle }_x}{\parallel \xi \parallel \parallel \zeta \parallel }}$.

If the function $f\in C^2\mathcal{M}$, the Riemannian Hessian of f is a linear mapping from $T_x\mathcal{M}$ to $T_x\mathcal{M}$, defined by

$$Hessf\left(x\right)\left[\xi \right]={\tilde{\nabla }}_{\xi }gradf\left(x\right),\forall \xi \in T_x\mathcal{M},$$

where $Hessf\left(x\right)\left[\xi \right]$ denotes the action of $Hessf\left(x\right)$ on the tangent vector $\xi \in T_x\mathcal{M}$ and $\tilde{\nabla }$ is the Riemannian connection [41].

Definition 1 

(Geodesic and exponential mapping [43]). A geodesic is a constant speed curve $\gamma :\left[0,1\right]\to \mathcal{M}$ that is the locally distance minimizing. Given a point $x\in \mathcal{M}$ and a tangent vector $v\in T_x\mathcal{M}$, exponential mapping, ${Exp}_x\left(v\right)$, is defined as a mapping from $T_x\mathcal{M}$ to $\mathcal{M}$ s.t. ${Exp}_x\left(v\right)=\gamma \left(1\right)$, where γ is geodesic with $\gamma \left(0\right)=x$ and $\dot{\gamma }\left(0\right)=v$.

Definition 2 

(Inverse of the exponential mapping [43]). If there is a unique geodesic between $x,y\in \mathcal{M}$, there exists the inverse of the exponential mapping ${Exp}_x^{-1}:\mathcal{M}\to T_x\mathcal{M}$, and the geodesic is the unique shortest path with $\parallel {Exp}_x^{-1}\left(y\right)\parallel =\parallel {Exp}_y^{-1}\left(x\right)\parallel$ between x and y.

Exponential mapping is the most accurate mapping for $\forall \xi \in T_x\mathcal{M}$ on the manifold. Moreover, if the manifold is complete, the inverse of the exponential mapping called the logarithm mapping is well defined, and we have

$$d(x,y)=\parallel {Exp}_x^{-1}\left(y\right)\parallel .$$

(12)

Therefore, we use exponential mapping in the paper, whose property (12) is very important for our analysis later. Throughout this paper, we always assume that $\mathcal{M}$ is complete. As we all know, we cannot operate tangent vectors in different tangent spaces directly, so a natural way to deal with this problem is to apply the parallel transport. Next, we give the definition of parallel transport.

Definition 3 

(Parallel transport [43]). Given a Riemannian manifold, $\mathcal{M}$, with two points $x,y$, the parallel transport $P_{x\to y}:T_x\mathcal{M}\to T_y\mathcal{M}$ is a linear operator that keeps the inner product: $\forall \xi ,\zeta \in T_x\mathcal{M}$, i.e., ${\langle \xi ,\zeta \rangle }_x={\langle P_{x\to y}\xi ,P_{x\to y}\zeta \rangle }_y$.

We now present the definition of Lipschitz smooth and convexity on the Riemannian manifold, $\mathcal{M}$, which will be utilized in later convergence analysis.

Definition 4 

(L-smoothness on manifolds [16]). If there exists $L\ge 0$, such that the following inequality holds for function f:

$$\parallel gradf\left(y\right)-P_{x\to y}gradf\left(x\right)\parallel \le Ld(x,y),\forall x,y\in \mathcal{M}.$$

Then f is called Lipschitz smooth (L-smooth) on the manifold. If the manifold is complete, we have [4]:

$$f\left(y\right)\le f\left(x\right)+{\langle gradf\left(x\right),{Exp}_x^{-1}\left(y\right)\rangle }_x+{\displaystyle \frac{L}{2}}{d}^2(x,y),\forall x,y\in \mathcal{M}.$$

Lemma 1 

([16]). Given a Riemannian manifold, $\mathcal{M}$, $f\in C^2\left(\mathcal{M}\right)$, then $Hessf\left(x\right)$ is Lipschitz smooth on the manifold and the inequality holds for f:

$$∥gradf\left(y\right)-P_{x\to y}\left(gradf\left(x\right)+Hessf\left(x\right)\left[{Exp}_x^{-1}\left(y\right)\right]\right)∥\le {\displaystyle \frac{L}{2}}{d}^2(x,y),\forall x,y\in \mathcal{M}.$$

Definition 5 

(Geodesically convex and $\mu$-strongly g-convex [16]). A function $f\in C^1\left(\mathcal{M}\right)$ is called geodesically convex if for all $x,y\in \mathcal{M}$, there exists a geodesic γ s.t. $\gamma \left(0\right)=x$, $\gamma \left(1\right)=y$ and

$$f\left(\gamma \right(t\left)\right)\le (1-t)f\left(x\right)+tf\left(y\right),\forall t\in [0,1],$$

or equivalently

$$f\left(y\right)\ge f\left(x\right)+{\langle gradf\left(x\right),{Exp}_x^{-1}\left(y\right)\rangle }_x.$$

In addition, a function f is μ-strongly g-convex (or geodesically μ-strongly convex) if the following inequality holds for f:

$$f\left(y\right)\ge f\left(x\right)+{\langle gradf\left(x\right),{Exp}_x^{-1}\left(y\right)\rangle }_x+{\displaystyle \frac{\mu }{2}}{d}^2(x,y),$$

where $0<\mu <min\{1,L\}$ is a small constant.

Definition 6 

(h-gradient dominated [17]). We say $f:\mathcal{M}\to \mathbb{R}$ is h-dominated if there exists a constant $h>0$ and $x^{*}$ is a global minimizer of f, for every $x\in \mathcal{M}$

$$f\left(x\right)-f\left(x^{*}\right)\le h{\parallel gradf\left(x\right)\parallel }^2.$$

3. RFedSVRG with Barzilai–Borwein Approximation as Second-Order Information

In this section, we first introduce the framework of the RFedSVRG [3] method and the BB method. Then we propose our method, RFedSVRG-2BB. Finally, we use BB step size along with RFedSVRG-2BB (RFedSVRG-2BBS).

3.1. The RFedSVRG Method

The framework of RFedSVRG has three main steps [3]:

• Uniformly sample clients to obtain the set $S_t\subset \left[n\right]$, and the clients receive $x_t$ and $gradf\left(x_t\right)$ from the server;
• The clients in $S_t$ take the local updates

  $$\begin{array}{c}\hfill x_{\ell +1}^{\left(i\right)}={Exp}_{x_{\ell }^{\left(i\right)}}\left\{{\eta }_t^{\left(i\right)}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}\left(gradf\left(x_t\right)-grad{f}_i\left(x_t\right)\right)\right]\right\}\end{array}$$

  (13)

  to obtain $\widehat{x}=x_{{\tau }_i}^{\left(i\right)}$, where ${\eta }_t^{\left(i\right)}={\eta }^{\left(i\right)}$ is the constant step size specified by users, $x_0^{\left(i\right)}=x_t$ and $\ell \in \{0,1,{\tau }_i-1\}$;
• The central sever aggregates the updated points ${\widehat{x}}^{\left(i\right)}$ from the clients to obtain $x_{t+1}$ by tangent space mean

  $$x_{t+1}={Exp}_{x_t}\left({\displaystyle {\displaystyle \frac{1}{k}}\sum _{i\in S_t}{Exp}_{x_t}^{-1}\left({\widehat{x}}^{\left(i\right)}\right)}\right),t\in \{0,1,\cdots ,T-1\}.$$

  (14)

There are many advantages to this framework. First, RFedSVRG only samples a subset of clients for each communication to iterate to avoid high computational costs. Second, the algorithm utilizes the gradient information at the previous iterate $gradf\left(x_t\right)$ and incorporates variance reduction techniques in the inner loop (13) to estimate the change of the gradient between $x_{\ell }^{\left(i\right)}$ and $x_t$. Because

$$\mathbb{E}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)\right)\right]=gradf\left(x_{\ell }^{\left(i\right)}\right),$$

RFedSVRG correctly converges to the global stationary points and avoids the “client drift” effect. The expectation of variance of $\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)\right)\right]$ has   

$$\begin{array}{cc}\hfill & \mathbb{E}[\parallel grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)\right)-gradf\left(x_{\ell }^{\left(i\right)}\right){\parallel }^2]\hfill \\ \hfill =& \mathbb{E}\left[{∥grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)\right)-gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right]\hfill \\ \hfill =& \mathbb{E}\left[{∥grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}grad{f}_i\left(x_t\right)+\mathbb{E}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}grad{f}_i\left(x_t\right)\right]∥}^2\right]\hfill \\ \hfill \le & \mathbb{E}\left[{∥grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}grad{f}_i\left(x_t\right)∥}^2\right]\le \mathcal{O}\left(d^2(x_{\ell }^{\left(i\right)},x_t)\right),\hfill \end{array}$$

(15)

where the first inequality is due to ${\mathbb{E}[\parallel \beta -\mathbb{E}\left[\beta \right]\parallel }^2{]=\mathbb{E}[\parallel \beta \parallel }^2{]-\parallel \mathbb{E}\left[\beta \right]\parallel }^2\le {\mathbb{E}[\parallel \beta \parallel }^2]$ and the second inequality follows from f is L-smooth. Third, the framework uses the tangent space mean (14) for the outer loop, which is easy to compute with closed-form on Riemannian manifolds and has the “regularization” property $d(x_{t+1},x_t)\le {\displaystyle \frac{1}{k}}{\sum }_{i\in S_t}d(x_{{\tau }_i}^{\left(i\right)},x_t)$, so that the distance between two consensus points can be controlled.

As pointed out by [33], the stochastic gradient methods attempt to estimates the true gradient accurately and the approximation for second-order information can obtain higher accuracy to reduce the number of communication rounds, T. Inspired by the work above, the main proposal of this paper is to generate the technique on the manifold and improve the accuracy with exponential mapping and parallel transport. The local update (13) can be modified as

$$\begin{array}{cc}\hfill x_{\ell +1}^{\left(i\right)}& ={Exp}_{x_{\ell }^{\left(i\right)}}\left\{{\eta }_t^{\left(i\right)}\right[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_t^{\left(i\right)}}(gradf\left(x_t\right)-grad{f}_i\left(x_t\right)\hfill \\ \hfill & +B\xi -B_i\xi \left)\right]\},\xi ={Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right),\hfill \end{array}$$

(16)

where B and $B_i$ are the computationally affordable matrices to approximate the second-order information that meets the following two properties:

• Property 1: unbiased estimate of B and $B_i$, that is,

  $$\mathbb{E}\left[B_i\right]=B.$$
• Property 2: approximation of B and $B_i$ such that

  $$B\xi \approx Hessf\left(x_t\right)\left[\xi \right]\mathrm{and}{B}_i\xi \approx Hess{f}_i\left(x_t\right)\left[\xi \right],$$

where $\xi ={Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)$.

From Property 1, we have

$$\begin{array}{cc}\hfill & \mathbb{E}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_t^{\left(i\right)}}\left(gradf\left(x_t\right)-grad{f}_i\left(x_t\right)\right)+P_{x_t\to x_t^{\left(i\right)}}(B\xi -B_i\xi )\right]\hfill \\ \hfill & =gradf\left(x_{\ell }^{\left(i\right)}\right),\mathrm{where}\xi ={Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right).\hfill \end{array}$$

(17)

Hence, we can expect that the inner loop of the algorithm can still find a correct solution.

From Property 2, we can obtain the variance of $\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_t^{\left(i\right)}}[gradf\left(x_t\right)-grad{f}_i\left(x_t\right)]+P_{x_t\to x_t^{\left(i\right)}}(B\xi -B_i\xi )\right]$ by

$$\begin{array}{cc}\hfill & \mathbb{E}\left[{∥grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}[gradf\left(x_t\right)-grad{f}_i\left(x_t\right)]+P_{x_t\to x_{\ell }^{\left(i\right)}}\left(B\xi -B_i\xi \right)-gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right]\hfill \\ \hfill =& \mathbb{E}[\parallel grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)+B_i\xi \right)\hfill \\ \hfill -& \left(gradf\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(gradf\left(x_t\right)+B\xi \right)\right){\parallel }^2]\hfill \\ \hfill =& \mathbb{E}[\parallel grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)+B_i\xi \right)\hfill \\ \hfill -& \mathbb{E}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)+B_i\xi \right)\right]{\parallel }^2]\hfill \\ \hfill \le & \mathbb{E}\left[{∥grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)+B_i\xi \right)∥}^2\right]\hfill \\ \hfill \approx & \mathbb{E}\left[{∥grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)+Hess{f}_i\left(x_t\right)\left[\xi \right]\right)∥}^2\right]\le \mathcal{O}\left(d^4(x_{\ell }^{\left(i\right)},x_t)\right),\hfill \end{array}$$

(18)

where the second inequality follows from Lemma 1. From (15) and (18), we can see that the variance of the modified local update (16) is smaller than original local update (13) when $x_{\ell }^{\left(i\right)}$ is close to $x_t$.

3.2. Barzilai–Borwein Method

In this subsection, we recall the Barzilai and Borwein method (BB method) deterministically. The BB method belongs to the first-order optimization algorithms. It has proven to be very successful in solving nonlinear optimization problems. In Euclidean space, the idea of calculating BB step size is derived from the quasi-Newton method [45]. Consider the unconstrained optimization problem:

$$\underset{x\in {\mathbb{R}}^d}{min}f\left(x\right),\mathrm{where}f\left(x\right)\in C^1\left({\mathbb{R}}^d\right).$$

(19)

For minimizing the differentiable objective function $f\left(x\right)$, the quasi-Newton method solves this problem by using the approximate of Hessian, which needs to satisfy the secant equation, i.e.,

$$B_t{s}_t=y_t,$$

(20)

where $B_t$ is an approximation of the Hessian of f at $x_t$, $s_t=x_t-x_{t-1}$, $y_t=\nabla f\left(x_t\right)-\nabla f\left(x_{t-1}\right)$ and $t\ge 1$. The BB method in a Riemannian manifold is to use matrix $B_t$ to approximate the action of the Riemannian Hessian of f at a certain point by a multiple of the identity $B_t={\displaystyle \frac{1}{{\eta }_t}}I$ with ${\eta }_t>0$ [40]. $B_t$ also needs to satisfy the secant Equation (20). But the difficultly is that we cannot operate the vectors in different space, and parallel transport and exponential mapping are needed. We should move the vectors in different points and tangent spaces “parallel” to the tangent space $T_{x_t}\mathcal{M}$. Therefore, we denote $s_t$ on the manifold as

$$s_t:=P_{x_{t-1}\to x_t}{Exp}_{x_{t-1}}^{-1}\left(x_t\right),$$

(21)

and denote $y_t$ as

$$y_t:=gradf\left(x_t\right)-P_{x_{t-1}\to x_t}(gradf\left(x_{t-1}\right)).$$

(22)

To solve the the secant equation in a least-square sense, i.e., $\underset{{\eta }_t}{min}{∥{\displaystyle \frac{1}{{\eta }_t}}{s}_t-y_t∥}^2$, when ${\langle s_t,y_t\rangle }_{x_t}>0$, $B_t$ and the Riemannian BB step size, ${\eta }_t$, can be written as

$$B_t={\displaystyle \frac{{\langle s_t,y_t\rangle }_{x_t}}{{\langle s_t,s_t\rangle }_{x_t}}}I\mathrm{and}{\eta }_t={\displaystyle \frac{{\langle s_t,s_t\rangle }_{x_t}}{{\langle s_t,y_t\rangle }_{x_t}}},$$

(23)

Another choice of ${\eta }_t$ is to solve $\underset{{\eta }_t}{min}{∥s_t-{\eta }_t{y}_t∥}^2$, which can be expressed as

$$B_t={\displaystyle \frac{{\langle y_t,y_t\rangle }_{x_t}}{{\langle s_t,y_t\rangle }_{x_t}}}I\mathrm{and}{\eta }_t={\displaystyle \frac{{\langle s_t,y_t\rangle }_{x_t}}{{\langle y_t,y_t\rangle }_{x_t}}}.$$

(24)

3.3. RFedSVRG with Barzilai–Borwein Method as Second-Order Information (RFedSVRG-2BB)

In this subsection, we propose RFedSVRG with the BB method to approximate second-order information. The previous subsection introduces the two different approximations of $B_t$ of (23) and (24). For convenience, we adopt (23) for the rest of the paper.

Now we propose to use the BB method for a Hessian approximation to compute B and $B_i$ in (16). We call it the RFedSVRG-2BB method.

RFedSVRG-2BB: Equation (16) with (23) for the approximate Hessian. We first denote $s_t$ and $y_t$ in (21) and (22) and

$$y_t^i=grad{f}_i\left(x_t\right)-P_{x_{t-1}\to x_t}(grad{f}_i\left(x_{t-1}\right)).$$

(25)

If $t\ge 1$, ${\langle s_t,y_t\rangle }_{x_t}>0$ and ${\langle s_t,y_t^i\rangle }_{x_t}>0$,

$$B={\displaystyle \frac{{\langle s_t,y_t\rangle }_{x_t}}{{\langle s_t,s_t\rangle }_{x_t}}}I\mathrm{and}{B}_i={\displaystyle \frac{{\langle s_t,y_t^i\rangle }_{x_t}}{{\langle s_t,s_t\rangle }_{x_t}}}I,$$

(26)

else, we use first-order RFedSVRG, i.e.,

$$B=B_i=0.$$

(27)

Remark 1. 

In the communication, we compute the full BB approximation, B, using full gradients, and in the local update we compute the stochastic BB approximation, $B_i$, by using stochastic gradients. From (26) and (27), one can observe that it does not require expensive additional computation. It is easy to see that $\mathbb{E}\left[B_i\right]=B$ for the above approximations derived from the BB method.

We summarize our algorithm in the following framework, Algorithm 1.

3.4. RFedSVRG-2BB with Barzilai–Borwein Step Size (RFedSVRG-2BBS)

In this section, we propose RFedSVRG-2BB with the BB step size. Note the Input in Algorithm 1, where we need to specify all step sizes of all clients, which makes us enter a very large number of parameters. As seen from (23), the BB method does not need any parameter and the step size is computed while running the algorithm. To the best of our knowledge, SVRG-BB [35] is the first algorithm to propose the use of BB step size with SVRG. In the t-th ($0\le t\le T$) outer loop, BB step size in [35] can be regarded as in line 8 of Algorithm 1 with

$$\begin{array}{c}\hfill x_{\ell +1}^{\left(i\right)}=x_{\ell }^{\left(i\right)}-{\eta }_t^{\left(i\right)}(\nabla f_i\left(x_{\ell }^{\left(i\right)}\right)+\nabla f\left(x_t\right)-\nabla f_i\left(x_t\right)),{\eta }_t^{\left(i\right)}={\displaystyle \frac{1}{{\tau }_i}}{\displaystyle \frac{\parallel s_t{\parallel }_2^2}{s_t^{\top }{y}_t}},\end{array}$$

(28)

where $s_t=x_t-x_{t-1}$, $y_t=\nabla f\left(x_t\right)-\nabla f\left(x_{t-1}\right)$ and ${\tau }_i$ is the update frequency of the inner loop.

Algorithm 1: Framework of RFedSVRG-2BB

Inspired by the idea of SVRG-BB, we use the BB step size for RFedSVRG-2BB so that we can achieve the step size self-adjusting, which means we do not need to specify a step size for each client and therefore we maintain a faster speed of convergence. We call the new algorithm RFedSVRG-2BBS. We apply (23) to decide step length. Moreover, in order to make the convergence curve flatter in numerical experiments, we use the bounds $[{\eta }_{max},{\eta }_{min}]$ for the step size. This method of calculating the step size is shown in (29) and (30).

RFedSVRG-2BBS: ${\left\{{\eta }^{\left(i\right)}\right\}}_{i=1}^n$ in Input of Algorithm 1 is replaced by $\{\widehat{{\eta }_0},{\eta }_{max},{\eta }_{min}\}$, where $0<{\eta }_{min}<{\eta }_{max}$. If $t\ge 1$, ${\eta }_t^{\left(i\right)}$, in line 8 in Algorithm 1 is changed to

$${\eta }_t^{\left(i\right)}=\widehat{{\eta }_t}/{\tau }_i,$$

(29)

where

$$\widehat{{\eta }_t}=\left\{\begin{array}{cc}min\left\{{\eta }_{max},max\left\{{\eta }_{min},{\eta }_t^{BB}\right\}\right\},\hfill & \mathrm{if}{\langle s_t,y_t\rangle }_{x_t}>0,\hfill \\ {\eta }_{max},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(30)

${\eta }_t^{BB}={\displaystyle \frac{{\langle s_t,s_t\rangle }_{x_t}}{{\langle s_t,y_t\rangle }_{x_t}}}$, $s_t$ and $y_t$ is calculated by (21) and (22). Else, ${\eta }_t^{\left(i\right)}=\widehat{{\eta }_0}/{\tau }_i$.

4. Convergence Analysis

In this section, we present the convergence results with their convergence rate of RFedSVRG-2BB and RFedSVRG-2BBS. Before the convergence results are given, we will give some necessary conclusions and assumptions.

Assumption 1 

(Smoothness). Suppose that $f_i$ is $L_i$-smooth on the manifold. It implies that f is L-smooth with $L={\displaystyle \sum _{i=1}^n}{L}_i$.

Denote $s_t$, $y_t$ and $y_t^i$ in (21), (22) and (25). If the function f is L-smooth, $\parallel y_t\parallel =\parallel gradf\left(x_t\right)-P_{x_{t-1}\to x_t}gradf\left(x_{t-1}\right)\parallel \le Ld(x_{t-1},x_t)=L\parallel {Exp}_{x_{t-1}}^{-1}\left(x_t\right)\parallel =L\parallel s_t\parallel$. Similarly, $\parallel y_t^i\parallel \le L\parallel s_t\parallel$. Then we can obtain the upper bound of ${\langle s_t,y_t\rangle }_{x_t}$ and ${\langle s_t,y_t^i\rangle }_{x_t}$:

$${\langle s_t,y_t\rangle }_{x_t}\le \parallel s_t\parallel ·\parallel y_t\parallel \le L\parallel s_t{\parallel }^2,{\langle s_t,y_t^i\rangle }_{x_t}\le \parallel s_t\parallel ·\parallel y_t^i\parallel \le L\parallel s_t{\parallel }^2.$$

For RFedSVRG-2BBS, we can obtain the lower bound of ${\eta }_t^{BB}$ in (30):

$${\eta }_t^{BB}={\displaystyle \frac{{\langle s_t,s_t\rangle }_{x_t}}{{\langle s_t,y_t\rangle }_{x_t}}}\ge {\displaystyle \frac{1}{L}}·{\displaystyle \frac{\parallel s_t\parallel }{\parallel s_t\parallel }}={\displaystyle \frac{1}{L}}.$$

(31)

It means $max\left\{{\eta }_{min},{\eta }_t^{BB}\right\}\ge {\displaystyle \frac{1}{L}}$. If ${\eta }_{max}\le {\displaystyle \frac{1}{L}}$, from (30) we can see that $\widehat{{\eta }_t}={\eta }_{max}$ is fixed. To avoid this situation, we should set ${\eta }_{max}>{\displaystyle \frac{1}{L}}$. In this case, we have

$$\widehat{{\eta }_t}\ge {\displaystyle \frac{1}{L}}\mathrm{and}{\displaystyle \frac{1}{{\tau }_{i}L}}\le {\eta }_t^{\left(i\right)}\le {\displaystyle \frac{{\eta }_{max}}{{\tau }_i}}.$$

(32)

Moreover, we have

$$0\le \beta ={\displaystyle \frac{{\langle s_t,y_t\rangle }_{x_t}}{{\langle s_t,s_t\rangle }_{x_t}}}\le {\displaystyle \frac{L\parallel s_t{\parallel }^2}{\parallel s_t{\parallel }^2}}=L,0\le {\beta }_i={\displaystyle \frac{{\langle s_t,y_t^{\left(i\right)}\rangle }_{x_t}}{{\langle s_t,s_t\rangle }_{x_t}}}\le {\displaystyle \frac{L\parallel s_t{\parallel }^2}{\parallel s_t{\parallel }^2}}=L.$$

For RFedSVRG-2BB and RFedSVRG-2BBS, B and $B_i$ are computed in (26) and (27), and we can obtain ${∥B-B_i∥}^2={∥\left(\beta -{\beta }_i\right)·I∥}^2={\left(\beta -{\beta }_i\right)}^2·{\parallel I\parallel }^2\le L^2$ and

$$\begin{array}{c}\hfill {∥B\xi -B_i\xi ∥}^2={∥(B-B_i)\xi ∥}^2=\left(∥B-B_i∥\right.∥{\xi \parallel )}^2\le L^2{\parallel \xi \parallel }^2,\forall \xi \in T_{x_{\ell }^{\left(i\right)}}\mathcal{M}.\end{array}$$

(33)

Assumption 2 

(Regularization over manifold). The manifold $\mathcal{M}$ is complete and there exists a compact set, $\mathcal{D}\subset \mathcal{M}$, with diameter bounded by D so that all the iterates of Algorithm 1 and the optimal points are contained in $\mathcal{M}$. The sectional curvature in $\mathcal{M}$ is bounded in $[{\kappa }_{min},{\kappa }_{max}]$. Then we can obtain the following key constant, defined in [4,17], to capture manifold curvature:

$$\zeta =\left\{\begin{array}{cc}{\displaystyle \frac{\sqrt{|{\kappa }_{\min }|}D}{tanh\sqrt{|{\kappa }_{\min }|}D}}\hfill & ,if{\kappa }_{\min }<0.\hfill \\ 1\hfill & ,if{\kappa }_{\min }\ge 0.\hfill \end{array}\right.$$

(34)

Lemma 2 

(Corollary 8 in [4]). If the Riemannian manifold, $\mathcal{M}$, satisfies Assumption 2, for any points $x,x_t\in \mathcal{M}$, the update $x_{t+1}={Exp}_{x_t}(-{\eta }_t{g}_t)$ satisfies

$$\begin{array}{cc}\hfill d^2(x_{t+1},x)\le & d^2(x_t,x)+\zeta {\parallel {\eta }_t{g}_t\parallel }^2-2{〈{Exp}_{x_t}^{-1}\left(x\right),-{\eta }_t{g}_t〉}_{x_t}\hfill \\ \hfill =& d^2(x_t,x)+\zeta d^2(x_t,x_{t+1})-2{〈{Exp}_{x_t}^{-1}\left(x\right),{Exp}_{x_t}^{-1}\left(x_{t+1}\right)〉}_{x_t}.\hfill \end{array}$$

Next, we provide convergence results with their convergence rate for RFedSVRG-2BB and RFedSVRG-2BBS with the $\mu$-strongly g-convex objective function. The proof is inspired from [17]. For the calculation of the convergence rate, we refer to [4].

Lemma 3 

($\mu$-strongly g-convex, RFedSVRG-2BB and RFedSVRG-2BBS with $k=1$). Consider RFedSVRG-2BB and RFedSVRG-2BBS with Option 1 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$ (line 10 in Algorithm 1) and take $k=1$. Assumptions 1 and 2 are satisfied and f is μ-strongly g-convex in problem (1). Denote

$${\alpha }_t^i={\displaystyle \frac{4\zeta {\eta }_t^{\left(i\right)}{L}^2+{\left(1+2{\eta }_t^{\left(i\right)}\left(5\zeta {\eta }_t^{\left(i\right)}{L}^2-\mu \right)\right)}^{{\tau }_i}\left(\mu -9\zeta {\eta }_t^{\left(i\right)}{L}^2\right)}{\mu -5\zeta {\eta }_t^{\left(i\right)}{L}^2}}.$$

(35)

Then we have $\mathbb{E}{d}^2(x_{t+1},x^{*})\le {\alpha }_t^i\mathbb{E}{d}^2(x_t,x^{*})$.

Proof. 

Denote ${\Delta }_{\ell }=grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)\right)$ and

$$v_{i_{\ell }}^t=grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_t^{\left(i\right)}}\left(gradf\left(x_t\right)-grad{f}_i\left(x_t\right)+(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)\right).$$

Taking the expectation with respect to i in the t-th outer loop, we have $\mathbb{E}\left[{\Delta }_{\ell }\right]=gradf\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_t^{\left(i\right)}}gradf\left(x_t\right),$ and then we can bound the squared norm of $v_{i_{\ell }}^t$ as follows

$$\begin{array}{cc}\hfill \mathbb{E}\parallel v_{i_{\ell }}^t{\parallel }^2& =\mathbb{E}{∥{\Delta }_{\ell }+P_{x_t\to x_{\ell }^{\left(i\right)}}gradf\left(x_t\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =\mathbb{E}{∥{\Delta }_{\ell }-\mathbb{E}\left[{\Delta }_{\ell }\right]+gradf\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & \le 2\mathbb{E}{∥{\Delta }_{\ell }-\mathbb{E}\left[{\Delta }_{\ell }\right]∥}^2+2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2\mathbb{E}{∥P_{x_t\to x_{\ell }^{\left(i\right)}}(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & \le 2\mathbb{E}{∥{\Delta }_{\ell }∥}^2+2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}gradf\left(x_t\right)∥}^2+2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)-P_{x^{*}\to x_{\ell }^{\left(i\right)}}gradf\left(x^{*}\right)∥}^2\hfill \\ \hfill & +2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & \le 2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x^{*}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =4L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x^{*}\right)∥}^2\hfill \\ \hfill & \le 4L^2\mathbb{E}{\left(∥{Exp}_{x_t}^{-1}\left(x^{*}\right)∥+∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x^{*}\right)∥\right)}^2+2L^2\mathbb{E}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x^{*}\right)∥}^2\hfill \\ \hfill & \le 8L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x^{*}\right)∥}^2+10L^2\mathbb{E}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x^{*}\right)∥}^2,\hfill \end{array}$$

(36)

where the first inequality is due to ${\parallel a+b+c\parallel }^2\le {2\parallel a\parallel }^2+{2\parallel b\parallel }^2+2{\parallel c\parallel }^2$, the second inequality is due to ${\mathbb{E}\parallel \xi -\mathbb{E}\xi \parallel }^2={\mathbb{E}\parallel \xi \parallel }^2-{\parallel \mathbb{E}\xi \parallel }^2\le \mathbb{E}{\parallel \xi \parallel }^2$ and (33), the third inequality due to Assumption 1 is satisfied, the fourth inequality is due to $d(x,y)\le d(x,z)+d(y,z)$, and the fifth inequality $(\parallel a\parallel +{\parallel b\parallel )}^2\le {2\parallel a\parallel }^2+2{\parallel b\parallel }^2$ and third equality are due to $gradf\left(x^{*}\right)=0$.

Note that $\mathbb{E}\left[v_{i_{\ell }}^t\right]=gradf\left(x_{\ell }^{\left(i\right)}\right)$ and $x_{\ell +1}^{\left(i\right)}={Exp}_{x_{\ell }^{\left(i\right)}}(-{\eta }_t^{\left(i\right)}{v}_{i_{\ell }}^t)$, therefore

$$\begin{array}{c}\mathbb{E}{d}^2(x_{\ell +1}^{\left(i\right)},x^{*})\le \mathbb{E}{d}^2(x_{\ell }^{\left(i\right)},x^{*})+\zeta \mathbb{E}{\left({\eta }_t^{\left(i\right)}\right)}^2{\parallel v_{{\ell }_i}^t\parallel }^2+2{\eta }_t^{\left(i\right)}\mathbb{E}{〈{Exp}_{x_{\ell }^{\left(i\right)}}\left(x^{*}\right),v_{i_{\ell }}^t〉}_{x_{\ell }^{\left(i\right)}}\hfill \\ \hspace{1em}\le \mathbb{E}{d}^2(x_{\ell }^{\left(i\right)},x^{*})+\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{L}^2\mathbb{E}\left(10d^2(x_{\ell }^{\left(i\right)},x^{*})+8d^2(x_t,x^{*})\right)\hfill \\ \hspace{1em}+2{\eta }_t^{\left(i\right)}\mathbb{E}{〈{Exp}_{x_{\ell }^{\left(i\right)}}\left(x^{*}\right),gradf\left(x_{\ell }^{\left(i\right)}\right)〉}_{x_{\ell }^{\left(i\right)}}\hfill \\ \hspace{1em}\le \left(1+10\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{L}^2-{\eta }_t^{\left(i\right)}\mu \right)\mathbb{E}{d}^2(x_{\ell }^{\left(i\right)},x^{*})+8\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{L}^2\mathbb{E}{d}^2(x_t,x^{*})\hfill \\ \hspace{1em}+2{\eta }_t^{\left(i\right)}\mathbb{E}\left(f\left(x^{*}\right)-f\left(x_{\ell }^{\left(i\right)}\right)\right)\hfill \\ \hspace{1em}\le \left(1+10\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{L}^2-2{\eta }_t^{\left(i\right)}\mu \right)\mathbb{E}{d}^2(x_{\ell }^{\left(i\right)},x^{*})+8\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{L}^2\mathbb{E}{d}^2(x_t,x^{*}).\hfill \end{array}$$

(37)

The first inequality uses Lemma 2 and the second one is due to (36). The third and fourth inequalities use the $\mu$-strongly g-convexity of $f\left(x\right)$.

Denote $u_{\ell }^{\left(i\right)}\triangleq \mathbb{E}{d}^2(x_{\ell }^{\left(i\right)},x^{*})$, $q\triangleq 1+2{\eta }_t^{\left(i\right)}\left(5\zeta {\eta }_t^{\left(i\right)}{L}^2-\mu \right)$ and $p\triangleq {\displaystyle \frac{8\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{L}^2}{1-q}}$. Since $k=1$, $x_{t+1}=x_{\tau }^{\left(i\right)}$. Note that $x_t=x_0^{\left(i\right)}$, i.e., $u_t=u_0^{\left(i\right)}$ and from (37) we have $u_{\ell +1}^{\left(i\right)}\le q{u}_{\ell }^{\left(i\right)}+p(1-q)u_t$, i.e., $u_{\ell +1}^{\left(i\right)}-p{u}_t\le q\left(u_{\ell }^{\left(i\right)}-p{u}_t\right)$. Hence, we have $u_{t+1}-p{u}_t\le q^{{\tau }_i}(1-p)u_t$ and

$$u_{t+1}\le \left[p+q^{\tau }(1-p)\right]u_t.$$

(38)

Denoting

$${\alpha }_t^i=p+q^{{\tau }_i}(1-p)={\displaystyle \frac{4\zeta {\eta }_t^{\left(i\right)}{L}^2+{\left(1+2{\eta }_t^{\left(i\right)}\left(5\zeta {\eta }_t^{\left(i\right)}{L}^2-\mu \right)\right)}^{{\tau }_i}\left(\mu -9\zeta {\eta }_t^{\left(i\right)}{L}^2\right)}{\mu -5\zeta {\eta }_t^{\left(i\right)}{L}^2}},$$

from inequation (38) we obtain $\mathbb{E}{d}^2(x_{t+1},x^{*})\le {\alpha }_t^i\mathbb{E}{d}^2(x_t,x^{*})$. □

Theorem 1 

($\mu$-strongly g-convex, RFedSVRG-2BB with $k=1$). Consider RFedSVRG-2BB with Option 1 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$ (line 10 in Algorithm 1) and take $k=1$. Assumptions 1 and 2 are satisfied and f is μ-strongly g-convex in problem (1). Take ${\eta }^{\left(i\right)}<{\displaystyle \frac{\mu }{9\zeta L^2}}(\forall i\in \left[n\right])$, we have

$${\alpha }^i={\displaystyle \frac{4\zeta {\eta }^{\left(i\right)}{L}^2+{\left(1+2{\eta }^{\left(i\right)}\left(5\zeta {\eta }^{\left(i\right)}{L}^2-\mu \right)\right)}^{{\tau }_i}\left(\mu -9\zeta {\eta }^{\left(i\right)}{L}^2\right)}{\mu -5\zeta {\eta }^{\left(i\right)}{L}^2}}<1.$$

Denoting

$$\alpha =\underset{i\in \left[n\right]}{max}\left\{{\alpha }^i\right\}$$

(39)

and $x^{*}$ is the optimal point, the Output of Option 1 in RFedSVRG-2BB has linear convergence in expectation:

$$\mathbb{E}{d}^2(\widehat{x},x^{*})\le {\alpha }^T·\mathbb{E}{d}^2(x_0,x^{*}).$$

In this case, the convergence rate of RFedSVRG-2BB is $\mathcal{O}\left({\displaystyle \frac{\alpha L{D}^2}{T(1-\alpha )}}\right)$.

Proof. 

Since $k=1$, without loss generality, we denote i as the client that we choose at the t-th outer loop. From Lemma 3, we know that $\mathbb{E}{d}^2(x_{t+1},x^{*})\le {\alpha }_t^i\mathbb{E}{d}^2(x_t,x^{*})$, where ${\alpha }_t^i$ is from (35). Because ${\eta }_t^{\left(i\right)}={\eta }^{\left(i\right)}$ in RFedSVRG-2BB, we have ${\alpha }_t^i={\alpha }^i$, where ${\alpha }^i$ is independent from t and denoted as

$${\alpha }^i={\displaystyle \frac{4\zeta {\eta }^{\left(i\right)}{L}^2+{\left(1+2{\eta }^{\left(i\right)}\left(5\zeta {\eta }^{\left(i\right)}{L}^2-\mu \right)\right)}^{{\tau }_i}\left(\mu -9\zeta {\eta }^{\left(i\right)}{L}^2\right)}{\mu -5\zeta {\eta }^{\left(i\right)}{L}^2}}.$$

Since ${\eta }^{\left(i\right)}<{\displaystyle \frac{\mu }{5{\zeta }^2{L}^2}}<{\displaystyle \frac{\mu }{3\zeta L^2}}$, we can obtain

$$\mu -9\zeta {\eta }^{\left(i\right)}{L}^2>0,$$

$$\mu -5\zeta {\eta }^{\left(i\right)}{L}^2>0,$$

$$1+2{\eta }^{\left(i\right)}\left(5\zeta {\eta }^{\left(i\right)}{L}^2-\mu \right)<1.$$

Therefore for $\forall i\in \left[n\right]$,

$${\alpha }^i<{\displaystyle \frac{4\zeta {\eta }^{\left(i\right)}{L}^2+\mu -9\zeta {\eta }^{\left(i\right)}{L}^2}{\mu -5\zeta {\eta }^{\left(i\right)}{L}^2}}=1.$$

Denoting $\alpha =\underset{i\in \left[n\right]}{max}\left\{{\alpha }^i\right\}$, we obtain $\alpha <1$. Therefore, for all outer loops, t, we have $\mathbb{E}{d}^2(x_{t+1},x^{*})<\alpha ·\mathbb{E}{d}^2(x_t,x^{*})$. It follows directly from the algorithm after T outer loops, $\mathbb{E}{d}^2(\widehat{x},x^{*})=\mathbb{E}{d}^2(x_T,x^{*})\le {\alpha }^T·\mathbb{E}{d}^2(x_0,x^{*}).$

By using the L-smooth of f and Assumption 2, we can obtain

$$\mathbb{E}[f\left(x_t\right)-f\left(x^{*}\right)]\le L·\mathbb{E}\left[{\displaystyle \frac{1}{2}}{d}^2(x_t,x^{*})\right]\le {\displaystyle \frac{{\alpha }^{t}L}{2}}·\mathbb{E}\left[d^2(x_T,x^{*})\right]\le {\displaystyle \frac{{\alpha }^{t}L{D}^2}{2}},t\in \{1,2,\cdots ,T\}.$$

Summing the above inequation over $t=1,\cdots ,T$, we have

$${\displaystyle \frac{1}{T}}\sum _{t=1}^T\mathbb{E}[f\left(x_t\right)-f\left(x^{*}\right)]\le {\displaystyle \frac{L{D}^2{\sum }_{t=1}^T{\alpha }^t}{2T}}<{\displaystyle \frac{\alpha L{D}^2}{2T(1-\alpha )}}.$$

Therefore, the convergence rate of RFedSVRG-2BB is $\mathcal{O}\left({\displaystyle \frac{\alpha L{D}^2}{T(1-\alpha )}}\right)$. □

Theorem 2 

($\mu$-strongly g-convex, RFedSVRG-2BBS with $k=1$). Consider RFedSVRG-2BBS with Option 1 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$ (line 10 in Algorithm 1) and take $k=1$. Assumptions 1 and 2 are satisfied and f is μ-strongly g-convex in problem (1). For $\forall i\in \left[n\right]$, take ${\tau }_i=\tau \ge {\displaystyle \frac{9\zeta {\eta }_{max}^2{L}^2}{\mu {\eta }_{min}}}$, we have

$$\alpha ={\displaystyle \frac{4\zeta L^2{\left(\frac{{\eta }_{max}}{\tau }\right)}^2+{\left[1+\frac{2{\eta }_{min}}{\tau }\left(5L^2\zeta \frac{{\eta }_{max}}{\tau }-\mu \right)\right]}^{\tau }\left[\frac{{\eta }_{min}}{\tau }(\mu -5\zeta L^2\frac{{\eta }_{max}}{\tau })-4\zeta L^2{\left(\frac{{\eta }_{max}}{\tau }\right)}^2\right]}{\frac{{\eta }_{min}}{\tau }\left(\mu -5\zeta L^2\frac{{\eta }_{max}}{\tau }\right)}}<1,$$

the Output of Option 1 in RFedSVRG-2BBS has linear convergence in expectation:

$$\mathbb{E}{d}^2(\widehat{x},x^{*})\le {\alpha }^T·\mathbb{E}{d}^2(x_0,x^{*}).$$

In this case, the convergence rate of RFedSVRG-2BBS is $\mathcal{O}\left({\displaystyle \frac{\alpha L{D}^2}{T(1-\alpha )}}\right)$.

Proof. 

Since $k=1$, without loss of generality, we denote i as the client that we choose at the t-th iteration. From Lemma 3, we know that $\mathbb{E}{d}^2(x_{t+1},x^{*})\le {\alpha }_t^i\mathbb{E}{d}^2(x_t,x^{*})$, where ${\alpha }_t^i$ is from (35). Because ${\tau }_i=\tau \ge {\displaystyle \frac{9\zeta {\eta }_{max}^2{L}^2}{\mu {\eta }_{min}}}>{\displaystyle \frac{5\zeta {\eta }_{max}{\eta }_{min}{L}^2+4\zeta L^2{\eta }_{max}^2}{\mu {\eta }_{min}}}>{\displaystyle \frac{5\zeta {\eta }_{max}{L}^2}{\mu }}$, we have

$${\displaystyle \frac{{\eta }_{min}}{\tau }}(\mu -5\zeta L^2{\displaystyle \frac{{\eta }_{max}}{\tau }})-4\zeta L^2{\left({\displaystyle \frac{{\eta }_{max}}{\tau }}\right)}^2>0,$$

$$5\zeta L^2{\eta }_{max}/\tau -\mu <0,$$

$$1+{\displaystyle \frac{2{\eta }_{min}}{\tau }}\left(5L^2\zeta {\displaystyle \frac{{\eta }_{max}}{\tau }}-\mu \right)<1.$$

From the fact that ${\eta }_{min}/\tau \le {\eta }_t^{\left(i\right)}\le {\eta }_{max}/\tau$ in RFedSVRG-2BBS, we can obtain ${\alpha }_t^i\le \alpha$, where $\alpha$ is a constant that is independent from t and i and denoted as

$$\begin{array}{c}\hfill \alpha ={\displaystyle \frac{4\zeta L^2{\left(\frac{{\eta }_{max}}{\tau }\right)}^2+{\left[1+\frac{2{\eta }_{min}}{\tau }\left(5L^2\zeta \frac{{\eta }_{max}}{\tau }-\mu \right)\right]}^{\tau }\left[\frac{{\eta }_{min}}{\tau }(\mu -5\zeta L^2\frac{{\eta }_{max}}{\tau })-4\zeta L^2{\left(\frac{{\eta }_{max}}{\tau }\right)}^2\right]}{\frac{{\eta }_{min}}{\tau }\left(\mu -5\zeta L^2\frac{{\eta }_{max}}{\tau }\right)}}.\end{array}$$

Hence, we obtain the upper bound of $\alpha$:

$$\begin{array}{cc}\hfill \alpha & <{\displaystyle \frac{4\zeta L^2{\left(\frac{{\eta }_{max}}{\tau }\right)}^2+\frac{{\eta }_{min}}{\tau }(\mu -5\zeta L^2\frac{{\eta }_{max}}{\tau })-4\zeta L^2{\left(\frac{{\eta }_{max}}{\tau }\right)}^2}{\frac{{\eta }_{min}}{\tau }\left(\mu -5\zeta L^2\frac{{\eta }_{max}}{\tau }\right)}}=1.\hfill \end{array}$$

For each outer loop, t, it holds that $\mathbb{E}{d}^2(x_{t+1},x^{*})<\alpha ·\mathbb{E}{d}^2(x_t,x^{*})$, It follows directly from the algorithm after T outer loops, $\mathbb{E}{d}^2(\widehat{x},x^{*})=\mathbb{E}{d}^2(x_T,x^{*})\le {\alpha }^T·\mathbb{E}{d}^2(x_0,x^{*}).$

By using the L-smooth of f and Assumption 2, we can obtain

$$\mathbb{E}[f\left(x_t\right)-f\left(x^{*}\right)]\le L·\mathbb{E}\left[{\displaystyle \frac{1}{2}}{d}^2(x_t,x^{*})\right]\le {\displaystyle \frac{{\alpha }^{t}L}{2}}·\mathbb{E}\left[d^2(x_T,x^{*})\right]\le {\displaystyle \frac{{\alpha }^{t}L{D}^2}{2}},t\in \{1,2,\cdots ,T\}.$$

Summing the above inequation over $t=1,\cdots ,T$, we have

$${\displaystyle \frac{1}{T}}\sum _{t=1}^T\mathbb{E}[f\left(x_t\right)-f\left(x^{*}\right)]\le {\displaystyle \frac{L{D}^2{\sum }_{t=1}^T{\alpha }^t}{2T}}<{\displaystyle \frac{\alpha L{D}^2}{2T(1-\alpha )}}.$$

Therefore, the convergence rate of RFedSVRG-2BBS is $\mathcal{O}\left({\displaystyle \frac{\alpha L{D}^2}{T(1-\alpha )}}\right)$. □

Later, we show the convergence results of RFedSVRG-2BB and RFedSVRG-2BBS with ${\tau }_i=1$. The conditions for conclusions do not need the objective function, f, to be g-convex.

Lemma 4 

(Non-convex, RFedSVRG-2BB and RFedSVRG-2BBS with ${\tau }_i=1$). Suppose the problem (1) satisfies Assumptions 1 and 2. If we run RFedSVRG-2BB and RFedSVRG-2BBS with Option 1 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$ and ${\tau }_i=1$, we have

$$f\left(x_{t+1}\right)-f\left(x_t\right)\le \left(-{\eta }_t^{\left(i\right)}+{\displaystyle \frac{{\left({\eta }_t^{\left(i\right)}\right)}^{2}L}{2}}\right){\parallel gradf\left(x_t\right)\parallel }^2.$$

Proof. 

From the local update in RFedSVRG-2BB and RFedSVRG-2BBS, we know that

$${Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right)=-{\eta }_t^{\left(i\right)}\left[grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_t^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)-B\xi +B_i\xi \right)\right],$$

where $\xi ={Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)$. Because ${\tau }_i=1$, for $\forall i\in \left[n\right]$, $x_0^{\left(i\right)}=x_t$ and $\ell \in \left\{0\right\}$, we have

$$\begin{array}{cc}\hfill \xi & ={Exp}_{x_t}^{-1}\left(x_0^{\left(i\right)}\right)={Exp}_{x_t}^{-1}\left(x_t\right)=0,B\xi =B_i\xi =0,\hfill \\ \hfill {Exp}_{x_t}^{-1}\left(x_1^{\left(i\right)}\right)& =-{\eta }_t^{\left(i\right)}\left[grad{f}_i\left(x_t\right)-P_{x_t\to x_0^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)\right)\right]\hfill \\ & =-{\eta }_t^{\left(i\right)}\left[grad{f}_i\left(x_t\right)-P_{x_t\to x_t}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)\right)\right]\hfill \\ & =-{\eta }_t^{\left(i\right)}\left[grad{f}_i\left(x_t\right)-grad{f}_i\left(x_t\right)+gradf\left(x_t\right)\right]=-{\eta }_t^{\left(i\right)}gradf\left(x_t\right).\hfill \end{array}$$

Using the L-smooth of $f_i$ and the aggregation step (line 12 in Algorithm 1), we have

$$\begin{array}{cc}\hfill f\left(x_{t+1}\right)-f\left(x_t\right)& \le {〈{Exp}_{x_t}^{-1}\left(x_{t+1}\right),gradf\left(x_t\right)〉}_{x_t}+{\displaystyle \frac{L}{2}}{d}^2(x_{t+1},x_t)\hfill \\ \hfill & ={〈{\displaystyle \frac{1}{k}}\sum _{i\in S_t}{Exp}_{x_t}^{-1}\left(x_1^{\left(i\right)}\right),gradf\left(x_t\right)〉}_{x_t}+{\displaystyle \frac{L}{2}}{∥{\displaystyle \frac{1}{k}}\sum _{i\in S_t}{Exp}_{x_t}^{-1}\left(x_1^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =-{\eta }_t^{\left(i\right)}\parallel gradf\left(x_t\right){\parallel }^2+{\displaystyle \frac{{\left({\eta }_t^{\left(i\right)}\right)}^{2}L}{2}}{\parallel gradf\left(x_t\right)\parallel }^2\hfill \\ \hfill & =\left(-{\eta }_t^{\left(i\right)}+{\displaystyle \frac{{\left({\eta }_t^{\left(i\right)}\right)}^{2}L}{2}}\right){\parallel gradf\left(x_t\right)\parallel }^2.\hfill \end{array}$$

(40)

□

Theorem 3 

(Non-convex, RFedSVRG-2BB with ${\tau }_i=1$). Suppose the problem (1) satisfies Assumptions 1 and 2. If we run RFedSVRG-2BB with Option 1 in in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$, ${\tau }_i=1$ and ${\eta }^{\left(i\right)}=1/L$, we have

$$\underset{0\le t\le T}{min}{\parallel gradf\left(x_t\right)\parallel }^2\le \mathcal{O}\left({\displaystyle \frac{L[f\left(x_0\right)-f\left(x^{*}\right)]}{T}}\right).$$

Proof. 

From (40) and ${\eta }_t^{\left(i\right)}={\eta }^{\left(i\right)}=1/L$ in RFedSVRG-2BB, we have

$$f\left(x_{t+1}\right)-f\left(x_t\right)\le {\displaystyle \frac{1}{2L}}{\parallel gradf\left(x_t\right)\parallel }^2.$$

Summing t over $t=0,1,\cdots ,T-1$, we obtain

$${\displaystyle \frac{1}{2L}}\sum _{t=1}^T{\parallel gradf\left(x_t\right)\parallel }^2\le f\left(x_0\right)-f\left(x_T\right)\le f\left(x_0\right)-f\left(x^{*}\right),$$

which means that $\underset{0\le t\le T}{min}{\parallel gradf\left(x_t\right)\parallel }^2\le \mathcal{O}\left({\displaystyle \frac{L[f\left(x_0\right)-f\left(x^{*}\right)]}{T}}\right)$. □

Theorem 4 

(Non-convex, RFedSVRG-2BBS with ${\tau }_i=1$). Suppose the problem (1) satisfies Assumptions 1 and 2. If we run RFedSVRG-2BBS with Option 1 in in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$, ${\tau }_i=1$ and ${\eta }_{max}\le 2/L$, we have

$$f\left(x_{t+1}\right)\le f\left(x_t\right).$$

Proof. 

From (32), ${\tau }_i=1$ and ${\eta }_{max}\le 2/L$ in RFedSVRG-2BBS, for $\forall i\in \left[n\right]$, we have ${\displaystyle \frac{1}{L}}\le {\eta }_t^{\left(i\right)}\le {\displaystyle \frac{2}{L}}$, which means that in (40) it holds

$$-{\eta }_t^{\left(i\right)}+{\displaystyle \frac{{\left({\eta }_t^{\left(i\right)}\right)}^{2}L}{2}}\le 0.$$

Therefore, $f\left(x_{t+1}\right)-f\left(x_t\right)\le 0$. □

We also have the convergence results of RFedSVRG-2BB and RFedSVRG-2BBS with ${\tau }_i>1$ and $k=1$. The results show that, when the objective function is only L-smooth, the algorithms can achieve sublinear convergence. Before giving the result we need the following lemma, which is inspired from [3,17].

Lemma 5 

(Non-convex, RFedSVRG-2BB and RFedSVRG-2BBS). Suppose the problem (1) satisfies Assumptions 1 and 2. Denote i as the client we choose at the t-th outer loop in RFedSVRG-2BB and RFedSVRG-2BBS, $\beta >0$ is a free constant and

$$\begin{array}{c}\hfill R_{\ell }=\mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)+c_{\ell }{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right],\end{array}$$

(41)

where $c_{\ell }$ and ${\delta }_{\ell }$ satisfy:

$$\begin{array}{c}\hfill c_{\ell }=c_{\ell +1}\left(1+\beta {\eta }_t^{\left(i\right)}+4\zeta L^2{\left({\eta }_t^{\left(i\right)}\right)}^2\right)+2L^3{\left({\eta }_t^{\left(i\right)}\right)}^2\ge 0,c_{{\tau }_i}=0,\end{array}$$

(42)

$$\begin{array}{c}\hfill {\delta }_{\ell }={\eta }_t^{\left(i\right)}-{\displaystyle \frac{c_{\ell +1}{\eta }_t^{\left(i\right)}}{\beta }}-L{\left({\eta }_t^{\left(i\right)}\right)}^2-2c_{\ell +1}\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2>0.\end{array}$$

(43)

The square of the norm of $gradf$ at iterate $x_{\ell }^{\left(i\right)}$ has the upper bound:

$$\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2\le {\displaystyle \frac{R_{\ell }-R_{\ell +1}}{{\delta }_{\ell }}},\ell =0,1,\cdots ,{\tau }_i-1.$$

(44)

Proof. 

Denote $v_{i_{\ell }}^t=grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)-gradf\left(x_t\right)+(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)\right)$, because f is L-smooth, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[f\left(x_{\ell +1}^{\left(i\right)}\right)\right]& \le \mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)+{〈gradf\left(x_{\ell }^{\left(i\right)}\right),{Exp}_{x_{\ell }^{\left(i\right)}}\left(x_{\ell +1}^{\left(i\right)}\right)〉}_{x_{\ell }^{\left(i\right)}}+{\displaystyle \frac{L}{2}}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right)∥}^2\right]\hfill \\ \hfill & =\mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)\right]+\mathbb{E}{〈gradf\left(x_{\ell }^{\left(i\right)}\right),-{\eta }_t^{\left(i\right)}{v}_{i_{\ell }}^t〉}_{x_{\ell }^{\left(i\right)}}+{\displaystyle \frac{L}{2}}\mathbb{E}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =\mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)\right]-{\eta }_t^{\left(i\right)}\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+{\displaystyle \frac{L{\left({\eta }_t^{\left(i\right)}\right)}^2}{2}}\mathbb{E}{\parallel v_{{\ell }_i}^t\parallel }^2\hfill \\ \hfill & =\mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)-{\eta }_t^{\left(i\right)}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+{\displaystyle \frac{L{\left({\eta }_t^{\left(i\right)}\right)}^2}{2}}{\parallel v_{{\ell }_i}^t\parallel }^2\right],\hfill \end{array}$$

(45)

where the first inequality is due to f being L-smooth, the first equality is due to $\mathbb{E}[a+b+c]=\mathbb{E}\left[a\right]+\mathbb{E}\left[b\right]+\mathbb{E}\left[c\right]$ and ${Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right)=-{\eta }_t^{\left(i\right)}{v}_{i_{\ell }}^t$, and the second equality is due to $\mathbb{E}\left[v_{i_{\ell }}^t\right]=gradf\left(x_{\ell }^{\left(i\right)}\right)$. Then the following inequality holds:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[{∥{Exp}_{x_t}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right)∥}^2\right]\le \mathbb{E}[{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+\zeta {∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right)∥}^2\hfill \\ \hfill -& 2{〈{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_{\ell +1}^{\left(i\right)}\right),{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_t\right)〉}_{x_{\ell }^{\left(i\right)}}]\hfill \\ \hfill =& \mathbb{E}\left[{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{∥v_{i_{\ell }}^t∥}^2\right]+2{\eta }_t^{\left(i\right)}\mathbb{E}{〈gradf\left(x_{\ell }^{\left(i\right)}\right),{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_t\right)〉}_{x_{\ell }^{\left(i\right)}}\hfill \\ \hfill \le & \mathbb{E}\left[{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{∥v_{i_{\ell }}^t∥}^2\right]\hfill \\ \hfill +& 2{\eta }_t^{\left(i\right)}\mathbb{E}\left[{\displaystyle \frac{1}{2\beta }}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+{\displaystyle \frac{\beta }{2}}{∥{Exp}_{x_{\ell }^{\left(i\right)}}^{-1}\left(x_t\right)∥}^2\right],\hfill \end{array}$$

(46)

where the first inequality follows from Lemma 2 and the second inequality is due to $2{\langle a,b\rangle }_x\le {\displaystyle \frac{1}{\beta }}{\parallel a\parallel }^2+\beta {\parallel b\parallel }^2$, where $\beta >0$ is a free constant. Denote $R_{\ell }=\mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)+c_{\ell }{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right]$, where $c_{\ell }$ is a parameter that varies with ℓ. Substituting (45) and (46) into $R_{\ell +1}$, we can obtain the following bound:

$$\begin{array}{cc}\hfill R_{\ell +1}& \le \mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)-{\eta }_t^{\left(i\right)}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+{\displaystyle \frac{L{\left({\eta }_t^{\left(i\right)}\right)}^2}{2}}{\parallel v_{{\ell }_i}^t\parallel }^2\right]+c_{\ell +1}\mathbb{E}[{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & +\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2{∥v_{i_{\ell }}^t∥}^2]+2c_{\ell +1}{\eta }_t^{\left(i\right)}\mathbb{E}\left[{\displaystyle \frac{1}{2\beta }}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+{\displaystyle \frac{\beta }{2}}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right]\hfill \\ \hfill & =\mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)-\left({\eta }_t^{\left(i\right)}-{\displaystyle \frac{c_{\ell +1}{\eta }_t^{\left(i\right)}}{\beta }}\right){∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right]\hfill \\ \hfill & +\left[{\displaystyle \frac{L{\left({\eta }_t^{\left(i\right)}\right)}^2}{2}}+c_{\ell +1}\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2\right]\mathbb{E}\left[\parallel v_{i_{\ell }}^t{\parallel }^2\right]+c_{\ell +1}\left(1+{\eta }_t^{\left(i\right)}\beta \right)\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2.\hfill \end{array}$$

(47)

Denoting ${\Delta }_{\ell }=grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}\left(grad{f}_i\left(x_t\right)\right)$, we have

$$\mathbb{E}\left[{\Delta }_{\ell }\right]=gradf\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}gradf\left(x_t\right),$$

and

$$\begin{array}{cc}\hfill \mathbb{E}\parallel v_{i_{\ell }}^t{\parallel }^2& =\mathbb{E}{∥{\Delta }_{\ell }+P_{x_t\to x_{\ell }^{\left(i\right)}}gradf\left(x_t\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =\mathbb{E}{∥{\Delta }_{\ell }-\mathbb{E}\left[{\Delta }_{\ell }\right]+gradf\left(x_{\ell }^{\left(i\right)}\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & \le 2\mathbb{E}{∥{\Delta }_{\ell }-\mathbb{E}\left[{\Delta }_{\ell }\right]∥}^2+2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2\mathbb{E}{∥P_{x_t\to x_{\ell }^{\left(i\right)}}(B-B_i){Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & \le 2\mathbb{E}{∥{\Delta }_{\ell }∥}^2+2L^2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}gradf\left(x_t\right)∥}^2+2L^2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & \le 2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =4L^2\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2+2L^2\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2,\hfill \end{array}$$

(48)

where the first inequality is due to ${\parallel a+b+c\parallel }^2\le {2\parallel a\parallel }^2+{2\parallel b\parallel }^2+2{\parallel c\parallel }^2$, the second inequality is due to ${\mathbb{E}\parallel \xi -\mathbb{E}\xi \parallel }^2={\mathbb{E}\parallel \xi \parallel }^2-{\parallel \mathbb{E}\xi \parallel }^2\le \mathbb{E}{\parallel \xi \parallel }^2$ and (33) and the third inequality follows from Assumption 1. Substituting (48) into (47), we obtain

$$\begin{array}{cc}\hfill R_{\ell }& \le \mathbb{E}\left[f\left(x_{\ell }^{\left(i\right)}\right)-\left({\eta }_t^{\left(i\right)}-{\displaystyle \frac{c_{\ell +1}{\eta }_t^{\left(i\right)}}{\beta }}-L{\left({\eta }_t^{\left(i\right)}\right)}^2-2c_{\ell +1}\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2\right){∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2\right]\hfill \\ \hfill & +\left[c_{\ell +1}\left(1+\beta {\eta }_t^{\left(i\right)}+4\zeta L^2{\left({\eta }_t^{\left(i\right)}\right)}^2\right)+2L^3{\left({\eta }_t^{\left(i\right)}\right)}^2\right]\mathbb{E}{∥{Exp}_{x_t}^{-1}\left(x_{\ell }^{\left(i\right)}\right)∥}^2\hfill \\ \hfill & =R_{\ell +1}-\left({\eta }_t^{\left(i\right)}-{\displaystyle \frac{c_{\ell +1}{\eta }_t^{\left(i\right)}}{\beta }}-L{\left({\eta }_t^{\left(i\right)}\right)}^2-2c_{\ell +1}\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2\right)\mathbb{E}{∥gradf\left(x_t^{\left(i\right)}\right)∥}^2.\hfill \end{array}$$

(49)

From inequation (49), we can obtain $c_{\ell }=c_{\ell +1}\left(1+\beta {\eta }_t^{\left(i\right)}+4\zeta L^2{\left({\eta }_t^{\left(i\right)}\right)}^2\right)+2L^3{\left({\eta }_t^{\left(i\right)}\right)}^2\ge 0$, $c_{{\tau }_i}=0$, ${\delta }_{\ell }={\eta }_t^{\left(i\right)}-{\displaystyle \frac{c_{\ell +1}{\eta }_t^{\left(i\right)}}{\beta }}-L{\left({\eta }_t^{\left(i\right)}\right)}^2-2c_{\ell +1}\zeta {\left({\eta }_t^{\left(i\right)}\right)}^2>0$ and

$$\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2\le {\displaystyle \frac{R_{\ell }-R_{\ell +1}}{{\delta }_{\ell }}}.$$

□

Theorem 5 

(Non-convex, RFedSVRG-2BB with $k=1$). Suppose the problem (1) satisfies Assumptions 1 and 2. Consider RFedSVRG-2BB with Option 2 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$, if we set $k=1$, ${\tau }_i=\tau =⌈m^{3{\alpha }_1/2}/{\zeta }^{1-2{\alpha }_2}⌉$ and ${\eta }^{\left(i\right)}={\displaystyle \frac{1}{6L{m}^{{\alpha }_1}{\zeta }^{{\alpha }_2}}}(\forall i\in \left[n\right])$, where $m\in {\mathbb{N}}^{+}$, ${\alpha }_1\in (0,1)$ and ${\alpha }_2\in (0,2)$, the Output with Option 2 in RFedSVRG-2BB satisfies:

$$\mathbb{E}\parallel gradf\left(\widehat{x}\right){\parallel }^2\le \mathcal{O}\left({\displaystyle \frac{L{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}\left[f\left(x_0\right)-f\left(x^{*}\right)\right]}{T}}\right).$$

(50)

When the function f is h-gradient dominated, the convergence rate of RFedSVRG-2BB is

$$\mathcal{O}\left({\displaystyle \frac{Lh{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}{D}^2}{T}}\right).$$

Proof. 

Since $k=1$, without loss of generality, we denote i as the client that we choose at the t-th outer loop. We take $\beta =L{\zeta }^{1-{\alpha }_2}/m^{{\alpha }_1/2}$ in (43), where $m\in {\mathbb{N}}^{+}$. Because in RFedSVRG-2BB, ${\eta }_t^{\left(i\right)}={\eta }^{\left(i\right)}$, recursively from (42), it can be seen that:

$$c_0=2L^3{\left({\eta }^{\left(i\right)}\right)}^2{\displaystyle \frac{{(1+\theta )}^{{\tau }_i}-1}{\theta }}={\displaystyle \frac{L}{18m^{2{\alpha }_1}{\zeta }^{2{\alpha }_2}}}{\displaystyle \frac{{(1+\theta )}^{{\tau }_i}-1}{\theta }},$$

where $\theta ={\eta }^{\left(i\right)}\beta +4\zeta L^2{\left({\eta }^{\left(i\right)}\right)}^2={\displaystyle \frac{{\zeta }^{1-2{\alpha }_2}}{6m^{3{\alpha }_1/2}}}+{\displaystyle \frac{{\zeta }^{1-2{\alpha }_2}}{9m^{2{\alpha }_1}}}\in \left({\displaystyle \frac{{\zeta }^{1-2{\alpha }_2}}{6m^{3{\alpha }_1/2}}},{\displaystyle \frac{5{\zeta }^{1-{\alpha }_2}}{18m^{3{\alpha }_1/2}}}\right)$ is a parameter and ${\left\{c_{\ell }\right\}}_{\ell =0}^{\tau -1}$ is a set of decreasing numbers.

Note that $\theta <1/\tau$, which means that ${(1+\theta )}^{\tau }<e$, then the upper bound of $c_0$ satisfies $c_0\le {\displaystyle \frac{L(e-1)}{3m^{{\alpha }_1/2}\zeta }}$. Therefore, the lower bound of ${\delta }_{\ell }$ in (43) is:

$$\begin{array}{cc}\hfill {\delta }_{min}& =\underset{\ell }{min}\left\{{\delta }_{\ell }\right\}\hfill \\ \hfill & \ge \left({\eta }^{\left(i\right)}-{\displaystyle \frac{c_0{\eta }^{\left(i\right)}}{\beta }}-{\left({\eta }^{\left(i\right)}\right)}^{2}L-2c_0\zeta {\left({\eta }^{\left(i\right)}\right)}^2\right)\hfill \\ \hfill & \ge {\eta }^{\left(i\right)}\left(1-{\displaystyle \frac{e-1}{3{\zeta }^{2-{\alpha }_2}}}-{\displaystyle \frac{1}{6m^{{\alpha }_1}{\zeta }^{{\alpha }_2}}}-{\displaystyle \frac{e-1}{18m^{{\alpha }^{3{\alpha }_1/2}}{\zeta }^{{\alpha }_2}}}\right)\hfill \\ \hfill & \ge {\eta }^{\left(i\right)}\left(1-{\displaystyle \frac{e-1}{3}}-{\displaystyle \frac{1}{6}}-{\displaystyle \frac{e-1}{18}}\right)>{\displaystyle \frac{33}{200}}{\eta }^{\left(i\right)}={\displaystyle \frac{11}{600L{m}^{{\alpha }_1}{\zeta }^{{\alpha }_2}}}.\hfill \end{array}$$

Note that this lower bound, ${\delta }_{min}$, is independent from the choice of the client, i. From (41), we have $R_0=f\left(x_t\right)$ and

$$R_{\tau }=\mathbb{E}\left[f\left(x_{\tau }^{\left(i\right)}\right)+c_{\ell }∥{Exp}_{x_t}^{-1}\left(x_{\tau }^{\left(i\right)}\right)∥\right]\ge \mathbb{E}\left[f\left(x_{\tau }^{\left(i\right)}\right)\right],$$

Because $x_0^{\left(i\right)}=x_t$ and we choose Option 2 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$, summing (44) over $\ell =0,1,\cdots ,{\tau }_i-1$, we have

$$\mathbb{E}[\parallel gradf\left(x_{t+1}\right){\parallel }^2]={\displaystyle \frac{1}{{\tau }_i}}\sum _{\ell =1}^{{\tau }_i-1}\mathbb{E}{\parallel gradf\left(x_{\ell }^{\left(i\right)}\right)\parallel }^2\le {\displaystyle \frac{R_0-R_{{\tau }_i}}{{\tau }_i{\delta }_{min}}}\le \mathbb{E}\left({\displaystyle \frac{f\left(x_t\right)-f\left(x_{t+1}\right)}{\tau {\delta }_{min}}}\right).$$

Summing the above inequation over $t=0,1,\cdots ,T-1$, we obtain

$$\mathbb{E}{∥gradf\left(\widehat{x}\right)∥}^2={\displaystyle \frac{1}{T}}\sum _{t=0}^{T-1}{\displaystyle \frac{1}{{\tau }_i}}\sum _{\ell =0}^{{\tau }_i-1}\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2<{\displaystyle \frac{600L{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}\left[f\left(x_0\right)-f\left(x^{*}\right)\right]}{11T}}.$$

(51)

Therefore, we have $\mathbb{E}\parallel gradf\left(\widehat{x}\right){\parallel }^2\le \mathcal{O}\left({\displaystyle \frac{L{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}\left[f\left(x_0\right)-f\left(x^{*}\right)\right]}{T}}\right).$

When the function f is h-gradient dominated, we have $f\left(x_t\right)-f\left(x^{*}\right)\le h{\parallel gradf\left(x_t\right)\parallel }^2$. From (51), L-smooth of f and Assumption 2, we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}\mathbb{E}[f\left(x_t\right)-f\left(x^{*}\right)]& \le {\displaystyle \frac{h}{T}}{\displaystyle \sum _{t=1}^T}\mathbb{E}{∥gradf\left(x_t\right)∥}^2<{\displaystyle \frac{600hL{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}\left[f\left(x_0\right)-f\left(x^{*}\right)\right]}{11Th}}\hfill \\ \hfill & \le {\displaystyle \frac{300Lh{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}{d}^2(x_0,x^{*})}{11T}}\le {\displaystyle \frac{300Lh{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}{D}^2}{11T}}.\hfill \end{array}$$

Therefore, the convergence rate of RFedSVRG-2BB is $\mathcal{O}\left({\displaystyle \frac{Lh{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}{D}^2}{T}}\right)$. □

Theorem 6 

(Non-convex, RFedSVRG-2BBS with $k=1$). Suppose the problem (1) satisfies Assumptions 1 and 2. Consider RFedSVRG-2BBS with Option 2 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$, if we set $k=1$, ${\tau }_i\ge \tau =24{\zeta }^2(\forall i\in \left[n\right])$ and ${\eta }_{max}\le 3/\left(2L\right)$, the Output with Option 2 in RFedSVRG-2BBS satisfies:

$$\mathbb{E}\parallel gradf\left(\widehat{x}\right){\parallel }^2\le \mathcal{O}\left({\displaystyle \frac{L[f\left(x_0\right)-f\left(x^{*}\right)]}{T}}\right).$$

When the function f is h-gradient dominated, the convergence rate of RFedSVRG-2BBS is $\mathcal{O}\left({\displaystyle \frac{Lh{D}^2}{T}}\right)$.

Proof. 

Since $k=1$, without loss of generality, we denote i as the client that we choose at the t-th outer loop. In RFedSVRG-2BB, ${\eta }_t^i=\widehat{{\eta }_t}/{\tau }_i$, by recursive (42), we obtain $c_0=L^3{\left({\displaystyle \frac{\widehat{{\eta }_t}}{{\tau }_i}}\right)}^2{\displaystyle \frac{{(1+\theta )}^{\tau }-1}{\theta }}$, where $\theta =\beta \left({\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\right)+2\zeta L^2{\left({\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\right)}^2$ is a parameter and ${\left\{c_{\ell }\right\}}_{\ell =0}^{{\tau }_i-1}$ is a series of decreasing numbers.

We fixed $\beta =L/\zeta$ in (42), then we have $2\zeta L^2{\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\le 2\zeta L^2{\displaystyle \frac{3}{2\times 24{\zeta }^{2}L}}={\displaystyle \frac{L}{8\zeta }}$ and

$$\theta =\beta {\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}+\left(2\zeta L^2{\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\right){\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\le {\displaystyle \frac{L}{\zeta }}{\displaystyle \frac{3}{2\tau L}}+{\displaystyle \frac{L}{8\zeta }}{\displaystyle \frac{3}{2\tau L}}={\displaystyle \frac{27}{16\zeta \tau }}.$$

Because $\zeta \ge 1$, ${(1+\theta )}^{\tau }\le {\left(1+{\displaystyle \frac{27}{16\zeta \tau }}\right)}^{\tau }<e^{27/\left(16\zeta \right)}\le e^{27/16}$. From $\widehat{{\eta }_t}/{\tau }_i\le \widehat{{\eta }_t}/\tau \le {\eta }_{max}/\tau$, we have

$$\begin{array}{cc}\hfill c_0& <{\displaystyle \frac{L^3{\left(\widehat{{\eta }_t}/{\tau }_i\right)}^2{e}^{27/16}}{\beta \widehat{{\eta }_t}/{\tau }_i}}={\displaystyle \frac{L^3{e}^{117/48}\zeta \widehat{{\eta }_t}}{{\tau }_{i}L}}\le {\displaystyle \frac{L^2{e}^{27/16}\zeta {\eta }_{max}}{\tau }}\hfill \\ \hfill & \le L^2{e}^{27/16}\zeta ·{\displaystyle \frac{3}{2\tau L}}\le L^2{e}^{27/16}\zeta ·{\displaystyle \frac{3}{2\times 24{\zeta }^{2}L}}={\displaystyle \frac{L{e}^{27/16}}{16\zeta }}.\hfill \end{array}$$

Then we can obtain the lower bound of ${\delta }_{min}:=\underset{\ell =1,\cdots ,{\tau }_i}{min}{\left\{{\delta }_{\ell }\right\}}_{\ell =0}^{{\tau }_i-1}$, where ${\left\{{\delta }_{\ell }\right\}}_{\ell =0}^{{\tau }_i-1}$ is calculated from (43):

$$\begin{array}{cc}\hfill {\delta }_{min}& \ge \left({\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}-{\displaystyle \frac{c_0\widehat{{\eta }_t}}{\tau \beta }}-{\left({\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\right)}^{2}L-2c_0\zeta {\left({\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\right)}^2\right)\hfill \\ \hfill & \ge {\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\left(1-{\displaystyle \frac{e^{81/32}}{24}}-{\displaystyle \frac{1}{16{\zeta }^2}}-{\displaystyle \frac{e^{27/16}}{128\zeta }}\right)\hfill \\ \hfill & \ge {\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\left(1-{\displaystyle \frac{e^{27/16}}{16}}-{\displaystyle \frac{1}{16}}-{\displaystyle \frac{e^{27/16}}{128}}\right)>{\displaystyle \frac{1}{2}}·{\displaystyle \frac{\widehat{{\eta }_t}}{\tau }}\ge {\displaystyle \frac{1}{2\tau L}}>0.\hfill \end{array}$$

where the last inequality is due to (32). Note that the lower bound, ${\delta }_{min}>0$, is independent from the choice of the client, i.

From (41), we have $R_0=f\left(x_t\right)$ and $R_{{\tau }_i}=\mathbb{E}\left[f\left(x_{{\tau }_i}^{\left(i\right)}\right)+c_{\ell }∥{Exp}_{x_t}^{-1}\left(x_{{\tau }_i}^{\left(i\right)}\right)∥\right]\ge \mathbb{E}\left[f\left(x_{{\tau }_i}^{\left(i\right)}\right)\right]=\mathbb{E}\left[f\left(x_{t+1}\right)\right]$. Because we choose Option 2 in each inner loop to obtain ${\widehat{x}}^{\left(i\right)}$, summing (44) over $\ell =0,1,\cdots ,{\tau }_i$, we obtain

$$\mathbb{E}[\parallel gradf\left(x_{t+1}\right){\parallel }^2]={\displaystyle \frac{1}{{\tau }_i}}\sum _{\ell =1}^{{\tau }_i-1}\mathbb{E}{\parallel gradf\left(x_{\ell }^{\left(i\right)}\right)\parallel }^2\le {\displaystyle \frac{R_0-R_{{\tau }_i}}{{\tau }_i{\delta }_{min}}}\le \mathbb{E}\left({\displaystyle \frac{f\left(x_t\right)-f\left(x_{t+1}\right)}{\tau {\delta }_{min}}}\right).$$

(52)

Summing (52) over $t=0,1,\cdots ,T-1$, we have

$$\mathbb{E}{∥gradf\left(\widehat{x}\right)∥}^2={\displaystyle \frac{1}{T}}\sum _{t=0}^{T-1}{\displaystyle \frac{1}{{\tau }_i}}\sum _{\ell =0}^{{\tau }_i-1}\mathbb{E}{∥gradf\left(x_{\ell }^{\left(i\right)}\right)∥}^2<{\displaystyle \frac{2L(f\left(x_0\right)-f^{*})}{T}}.$$

(53)

Therefore, we have $\mathbb{E}\parallel gradf\left(\widehat{x}\right){\parallel }^2\le \mathcal{O}\left({\displaystyle \frac{L[f\left(x_0\right)-f\left(x^{*}\right)]}{T}}\right).$

When the function f is h-gradient dominated, we have $f\left(x_t\right)-f\left(x^{*}\right)\le h{\parallel gradf\left(x_t\right)\parallel }^2$. From (53), L-smooth of f and Assumption 2, we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}\mathbb{E}[f\left(x_t\right)-f\left(x^{*}\right)]& \le {\displaystyle \frac{h}{T}}{\displaystyle \sum _{t=1}^T}\mathbb{E}{∥gradf\left(x_t\right)∥}^2<{\displaystyle \frac{2Lh(f\left(x_0\right)-f^{*})}{T}}\hfill \\ \hfill & \le {\displaystyle \frac{Lh{d}^2(x_0,x^{*})}{T}}\le {\displaystyle \frac{Lh{D}^2}{T}}.\hfill \end{array}$$

Therefore, the convergence rate of RFedSVRG-2BBS is $\mathcal{O}\left({\displaystyle \frac{Lh{D}^2}{T}}\right)$. □

Based on the theorems in this section, we briefly summarize the convergence rates of our algorithms when function f satisfies different properties in

Table 1. This table summarizes the convergence rates we have proved for our algorithms. Where T is total number of communication of algorithms, L is the Lipschitz constant of f and D is the diameter of the domain in Assumption 2. For the conditions that need to be met and other parameter sources, please refer to the corresponding theorem.

Properties of f	Algorithm	Convergence Rate	Parameter Source and Conditions
L-smooth and $\mu$-strongly g-convex	RFedSVRG-2BB	$\mathcal{O}\left({\displaystyle \frac{\alpha L{D}^2}{T(1-\alpha )}}\right)$	Theorem 1
L-smooth and $\mu$-strongly g-convex	RFedSVRG-2BBS	$\mathcal{O}\left({\displaystyle \frac{\alpha L{D}^2}{T(1-\alpha )}}\right)$	Theorem 2
L-smooth and h-gradient dominated	RFedSVRG-2BB	$\mathcal{O}\left({\displaystyle \frac{Lh{m}^{-{\alpha }_1/2}{\zeta }^{3{\alpha }_2-1}{D}^2}{T}}\right)$	Theorem 5
L-smooth and h-gradient dominated	RFedSVRG-2BBS	$\mathcal{O}\left({\displaystyle \frac{Lh{D}^2}{T}}\right)$	Theorem 6

. For more details, please refer to the corresponding theorem and its proof.

5. Numerical Experiments

In this section, we will demonstrate the performance of RFedSVRG-2BB and RFedSVRG-2BBS in solving problem (1) and compare it with RFedSVRG [3], RFedAvg [3] and RFedProx [3]. We use the Pymanopt package [46]. Since the inverse of exponential mapping (logarithm mapping) on the Stiefel manifold is not easy to calculate, we use the projection-like retraction [47] and its inverse [48] to approximate the exponential mapping and the logarithm mapping, respectively.

We test the five algorithms on the PSD Karcher mean (2), PCA (3) and kPCA (4) on synthetic or real datasets. For all problems, we measure the norm of the global Riemannian gradients. Because the optimal solution of $f\left(X\right)$ in problem (4) only represents the eigen-space corresponding to the r-largest eigenvalues, we also measure the principal angles [3,49] between the subspaces for kPCA.

5.1. Experiments on Synthetic Data

In this section, we demonstrate the results of the five algorithms for solving PCA (3) and the PSD Karcher mean (2) on synthetic data. We first generate the data $C_i\in {\mathbb{R}}^{p\times d}$, whose row vectors are generated from standard normal distribution. The $A_i$ in (3) and (2) are set to

$$A_i:=C_i^{\top }{C}_i\mathrm{with}{C}_i\in {\mathbb{R}}^{p\times d}\mathrm{and}{A}_i\in {\mathbb{R}}^{d\times d}.$$

(54)

Under these experiments, the datas in different agents are homogeneous in distribution, which provides a milder environment for comparing the behavior of all the algorithms. All algorithms iterate through the same random initial point. We terminate the algorithms if they meet one of the following conditions:

(a)

$\parallel gradf\left(x_t\right)\parallel \le {10}^{-13}$

(b)

the communication exceeds the specified number of rounds.

The y-axis of the figures in this subsection denotes $\parallel gradf\left(x_t\right)\parallel$ and the x-axis denotes the outer loop in the algorithms, i.e., the round of communication in Federated Learning.

5.1.1. Experiments on PSD Karcher Mean

In this subsection, we test RFedAvg, RFedProx, RFedSVRG and RFedSVRG-BB to solve the PSD Karcher mean (2). In these experiments, we set $d=p=20$ in (54), ${\tau }_i=\tau$ and $n=10$. For RFedAvg, RFedProx, RFedSVRG and RFedSVRG-2BB, we set ${\eta }^{\left(i\right)}=2\times {10}^{-1}$. We set $({\eta }_{max},{\eta }_{min})=(8\times {10}^{-1},8\times {10}^{-3})$ for RFedSVRG-2BBS. The results are given in Figure 1. The convergence curves of RFedSVRG-2BB and RFedSVRG-2BBS in Figure 1a–c show a linear rate of convergence, largely due to the fact that (2) is strongly geodesic convex [4].

From Figure 1a, we can see that RFedAvg and RFedProx are unable to reduce the norm of the Riemannian gradient to an acceptable level. RFedSVRG-2BB and RFedSVRG-2BBS can decrease the norm of the Riemannian gradient faster than RFedAvg, RFedProx and RFedSVRG, and RFedSVRG-2BBS is the fastest of all algorithms.

From Figure 1b, we can see that RFedSVRG-2BB and RFedSVRG-2BBS with different $\tau$ can convergence linearly. It is noted that the effect of taking ${\tau }_i=7$ is not as good as the effect of taking ${\tau }_i=5$. The central idea in SVRG and its variants is that the stochastic gradients are used to estimate the change in the gradient between point $x_{\ell }^{\left(i\right)}$ and $x_t$ [2,29], and it is clear that, if $x_{\ell }^{\left(i\right)}$ and $x_t$ are close to each other, the variance of the estimate

$$grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)-P_{x_t\to x_{\ell }^{\left(i\right)}}grad{f}_i\left(x_t\right)+P_{x_t\to x_{\ell }^{\left(i\right)}}\left(B\xi -B_i\xi \right)$$

line 8 of Algorithm 1 should be small, resulting in an estimate of $grad{f}_i\left(x_{\ell }^{\left(i\right)}\right)$ with small variance. As the inner iterate $x_{\ell }^{\left(i\right)}$ proceeds further, variance grows, and the algorithms need to start a new outer loop to compute a new full gradient $gradf\left(x_{t+1}\right)$ and reset the variance. Therefore it is important to set ${\tau }_i$ not too large to avoid a large variance.

From Figure 1c, the cases when RFedSVRG-2BB and RFedSVRG-2BBS with $k\ge 1$ can decrease the norm of the Riemannian gradient in convergence can be seen. RFedSVRG-2BB with $k>1$ is able to reach a more acceptable level of the norm of the Riemannian gradient than RFedSVRG-2BB with $k=1$.

5.1.2. Experiments on PCA

We now test the five algorithms on the standard PCA problem (3). In this test, we set the algorithms with different numbers of the agents, n, and pick up $k=n/10$ as the number of clients for each round. We partition the data sampled from 10,000 data points in ${\mathbb{R}}^{600}$ into n agents on average, i.e., $p=\lfloor 10,000/n\rfloor$, $d=600$ in (54), and each of them contains equal numbers of data. The number of data in these experiments are relatively large. For all algorithms, we set ${\tau }_i=2$. We use the constant step size ${\eta }^{\left(i\right)}=2\times {10}^{-2}$ for RFedAvg, RFedProx, RFedSVRG and RFedSVRG-2BB, and set $({\eta }_{max},{\eta }_{min})=(3\times {10}^{-1},3\times {10}^{-4})$ for RFedSVRG-2BBS. We specify that the number of terminated rounds is 500. The results are given in Figure 2.

From all the figures in Figure 2, we can see that RFedAvg and RFedProx are unable to reduce the norm of the Riemannian gradient to an acceptable level for different numbers of n. RFedSVRG-2BB and RFedSVRG-2BBS decrease the norm of the Riemannian gradient to a better level than RFedAvg, RFedProx and RFedSVRG with the same number of communications. Moreover, RFedSVRG-2BB and RFedSVRG-2BBS converge faster than RFedSVRG, and RFedSVRG-2BBS is the fastest.

5.2. Experiments on Real Data

In this subsection, we focus on the kPCA problem (4) here with four real datasets: the Iris dataset [50], the wine dataset [50], the breast cancer dataset [51] and the MNIST hand-written dataset [52]. They are highly heterogeneous real data and we use them to generate $A_i$ in problem (4).

We first normalize the four datasets. For one of the normalized datasets we denote it as $D_0={[{\beta }_i^{\top },\cdots ,{\beta }_m^{\top }]}^{\top }\in {\mathbb{R}}^{m\times d}$, where each data sample, ${\beta }_i$, in $D_0$ satisfies ${\beta }_i\in {\mathbb{R}}^{1\times d}$ and m is the number of data in dataset $D_0$. Then we randomly divide data in $D_0$ into n parts and denote the divided data $D_1,\cdots ,D_n$ as the dataset in clients. The covariance matrix, $A_i$, in (4) is calculated as

$$A_i=D_i^{\top }{D}_i\in {\mathbb{R}}^{d\times d}.$$

Utilizing the features of (4), we can obtain the optimal point, $x^{*}$, of (4) by directly solving the first r eigenvectors of $A={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{A}_i,$ i.e., if the the first r eigenvectors of A are ${\alpha }_1,\cdots ,{\alpha }_r$, the optimal point, $x^{*}$, in (4) is $x^{*}=[{\alpha }_1,\cdots ,{\alpha }_r]\in {\mathbb{R}}^{d\times r}$ [5].

Then we can compute the principal angles between the iterate point, $x_t$, and $x^{*}$ directly. Moreover, the Output, $\widehat{x}$, of our algorithms is the projection matrix that could be used to extract the principal components of dataset $D_0$ by

$$D=D_0·\widehat{x}\in {\mathbb{R}}^{m\times r},$$

(55)

which effectively reduces the dimension of data in $D_0$ from ${\mathbb{R}}^d$ to ${\mathbb{R}}^r$, and D is a reduced dimensional dataset [5]. We randomly choose the initial point for each experiment and terminate the algorithms if they meet one of the following conditions:

• $\parallel gradf\left(x_t\right)\parallel \le {10}^{-13}$ and the principal angle between $x_t$ and $x^{*}$ is less than ${10}^{-13}$;
• the communication exceeds the specified number of rounds.

We present our results in Figure 3, Figure 4, Figure 5 and Figure 6. The x-axis of the figures denotes the round of communication in Federated Learning.

In Figure 3a, Figure 4a, Figure 5a and Figure 6a, we plot the principal angle between $x^{*}$ and $x_t$ as the round of communication increases, i.e., the y-axis of the figures denotes the value of the principal angle between $x^{*}$ and $x_t$. We can observe that RFedAvg and RFedProx are unable to reduce the principal angle between $x^{*}$ and $x_t$ to an acceptable level. RFedSVRG-2BB and RFedSVRG-2BBS are able to effectively decrease the principal angle. Comparing the five algorithms, RFedSVRG-2BB and RFedSVRG-2BBS have a faster convergence rate and RFedSVRG-2BBS is the fastest.

In Figure 3b, Figure 4b, Figure 5b and Figure 6b, we plot the norm of gradient of $\parallel gradf\left(x_t\right)\parallel$ as the round of communication increases, i.e., the y-axis of the figures denotes $\parallel gradf\left(x_t\right)\parallel$. We can observe that RFedAvg and RFedProx are unable to reduce the norm of the Riemannian gradient to an acceptable level. RFedSVRG-2BB and RFedSVRG-2BBS are able to effectively decrease the norm of gradient of $\parallel gradf\left(x_t\right)\parallel$. Compared with RFedAvg, RFedProx and RFedSVRG, RFedSVRG-2BB and RFedSVRG-2BBS have a faster convergence rate and RFedSVRG-2BBS is the fastest.

In Figure 3c,d, Figure 4c,d and Figure 5c,d, because we take $r=3$, we can draw the scatter plots for D in (55) in 3D space to obtain an intuitive feel, where $\widehat{x}$ in (55) is the output point of RFedSVRG-2BB and RFedSVRG-2BBS, respectively. The colors of points in Figure 3c,d, Figure 4c,d and Figure 5c,d are based on the label of the original datasets. It shows that the algorithms indeed grasp the principal direction of the datasets and effectively reduce the dimensionality of the datasets.

6. Conclusions

In this paper we applied the BB technique to approximate the second-order information in RFedSVRG and obtain a new Riemannian FL algorithm, RFedSVRG-2BB. Then we incorporated the BB step size for RFedSVRG-2BB as a new variant, RFedSVRG-2BBS. We analyzed the RFedSVRG-2BB and RFedSVRG-2BBS convergence rates for a strongly geodesically convex function and L-smooth non-convex function. Additionally, we conducted some numerical experiments on synthetic and real datasets. The results show that our algorithms are better than RFedSVRG and also outperform RFedAvg and RFedProxk, which are widely applied algorithms. Therefore, the RFedSVRG-2BB and RFedSVRG-2BBS methods can be regarded as competitive alternatives to the classic methods of solving Federated Learning problems.