Symmetric ADMM-Based Federated Learning with a Relaxed Step

Abstract

Federated learning facilitates the training of global models in a distributed manner without requiring the sharing of raw data. This paper introduces two novel symmetric Alternating Direction Method of Multipliers (ADMM) algorithms for federated learning. The two algorithms utilize a convex combination of current local and global variables to generate relaxed steps to improve computational efficiency. They also integrate two dual-update steps with varying relaxation factors into the ADMM framework to boost the accuracy and the convergence rate. Another key feature is the use of weak parametric assumptions to enhance computational feasibility. Furthermore, the global update in the second algorithm occurs only at certain steps (e.g., at steps that are a multiple of a pre-defined integer) to improve communication efficiency. Theoretical analysis demonstrates linear convergence under reasonable conditions, and experimental results confirm the superior convergence and heightened efficiency of the proposed algorithms compared to existing methodologies.

1. Introduction

Federated learning serves as a widely adopted distributed machine learning methodology that has garnered substantial interest in recent years due to its effectiveness in addressing issues related to data privacy, security, and the accessibility of heterogeneous data [1,2,3]. This methodology has been employed extensively in various domains, such as health care [4] (Yang et al., 2019), finance [5] (Liu et al., 2021), the Internet of Things (IoT) [6] (Zeng et al., 2023), and intelligent transportation [7,8] (Manias et al., 2021). By providing solutions for data security and compliance, federated learning facilitates improved utilization of decentralized data and enhances the performance and efficiency of machine learning models. As a contemporary research hot spot, federated learning’s significance spans beyond boosting performance of machine learning models, extending to the safeguarding of data privacy, economizing computational resources, and supporting heterogeneous devices.

1.1. Related Work

(a) Improving computational efficiency and accuracy.

Each node independently addresses local optimization sub-problems in federated learning. Early research [6,9,10] used a strategy of splitting the computation across multiple devices for local optimization, although shortcomings persist regarding computational efficiency and accuracy. The Alternating Direction Method of Multipliers (ADMM), as a distributed method, has been used to solve many optimization problems due to its simplicity and efficiency. In federated learning, there are two types of ADMM available, namely exact [11] and inexact [12] ADMM. The former requires the clients to update the parameters by solving the sub-problems accurately, thereby increasing the computational burden [12,13,14,15,16,17]. The latter updates the parameters by solving sub-problems approximately, which reduces the computational complexity for clients [18,19,20,21]. However, these algorithms implement a single dual-update step per iteration and necessitate relatively stringent assumptions with respect to the parameters to ensure convergence properties.

(b) Saving computing resources. In distributed learning, local clients and the central server engage in frequent, often inefficient communication. Thus, extensive research efforts focus on devising algorithms to minimize the number of communication rounds. The stochastic gradient descent method, which aggregates in a cyclic fashion, is a widely used approach [22,23,24,25,26,27] that has demonstrated promising results in reducing the number of communication rounds. To further alleviate the burden of communication rounds, McMahan et al. [6,28,29] introduced the Federated Averaging Algorithm (FedAvg) and its improved variants. These algorithms minimize communications by performing local iterations multiple times before periodically conducting global aggregation. Li et al. [30] further refined the FedAvg method by introducing the Federated Proximal (FedProx) algorithm, allowing available device-based system resources to perform varying amounts of local work before aggregating partial solutions. Both FedAvg and FedProx have seen extensive application in distributed learning.

1.2. Our Contribution

The main contributions of this paper include the introduction of two federated learning algorithms based on Symmetric ADMM with Relaxed Step (Fed-RSADMM; see Algorithms 2 and 3), characterized as follows:

(I) Relaxed step. In contrast to the conventional ADMM, the presented algorithms employ a convex combination of the current local and global variables to generate the relaxed steps.

(II) $SymmetricADMM.$ We integrate two dual-update steps into the ADMM framework to construct a symmetric ADMM algorithm with varying relaxation factors, which is different from the general ADMM.

(III) $Weakparametricassumptions.$ Differing from conventional algorithmic assumptions, only simple assumptions are made with respect to the parameters.

1.3. Organization

This paper is organized as follows. In the next section, we provide the symbolic definitions and some common mathematical definitions that are used in this paper. In Section 3, we present the Fed-RSADMM and FedAvg-RSADMM algorithms, then prove their convergence. In Section 4, we design some comparative numerical experiments to illustrate the performance of our two proposed algorithms. The conclusions of this paper are presented in Section 5.

2. Preliminaries

The present section introduces some notations and definitions employed in this paper.

2.1. Notations

${\mathbb{R}}^n$ denotes the n-dimensional Euclidean space, $〈x,y〉=x^{\mathrm{T}}y$, and $\Vert ·\Vert$ denotes the Euclidean norm. Let the set $S\subseteq {\mathbb{R}}^n$ and the point $x\in {\mathbb{R}}^n$. If S is non-empty, the distance from point x to set S is denoted as $dist(x,S)=inf\{\Vert y-x\Vert :y\in S\}$. When $S=⌀$, let $dist(x,S)=+\infty$.

Definition 1

($L-$Lipschitz continuity). In mathematics, a function ($g:{\mathbb{R}}^n\to \mathbb{R}$) is said to be L-Lipschitz continuous (or simply L-Lipschitz) if, for any $x,y\in {\mathbb{R}}^n$, one has

$$\begin{array}{c}∥g\left(x\right)-g\left(y\right)∥\le L∥x-y∥,\hfill \end{array}$$

(1)

where $∥·∥$ denotes the Euclidean norm.

If the function (f) is continuously differentiable and its gradient ($\nabla f$) is L-Lipschitz continuous, then we have

$$\left|f\left(y\right)-f\left(x\right)-〈\nabla f\left(x\right),y-x〉\right|\le {\displaystyle \frac{L}{2}}\Vert y-x{\Vert }^2,\forall x,y\in {\mathbb{R}}^n.$$

(2)

Definition 2.

Let the function $f:{\mathbb{R}}^n\to \mathbb{R}$ be normally lower semicontinuous. The authors of [31] provided the following definitions:

(I) 

The Frechet subdifferential of f at $x\in domf$ is denoted as

$$\widehat{\partial }f\left(x\right)=\left\{x^{*}\in {\mathbb{R}}^n:\underset{y\ne x}{lim}\underset{y\to x}{inf}{\displaystyle \frac{f\left(y\right)-f\left(x\right)-〈x^{*},y-x〉}{\Vert y-x\Vert }}\right\},$$

(3)

and when $x\notin \mathit{domf}$, $\widehat{\partial }f\left(x\right)=⌀$.

(II) 

The limiting subdifferential of f at $x\in domf$ is denoted as

$$\partial f\left(x\right)=\left\{x^{*}\in {\mathbb{R}}^n:\exists x_k\to x,f\left(x_k\right)\to f\left(x\right),{\widehat{x}}_k\in \widehat{\partial }f\left(x_k\right),{\widehat{x}}_k\to x^{*}\right\},$$

(4)

and assuming that $x\in {\mathbb{R}}^n$ is a minimal-value point of f, then $0\in \partial f\left(x\right)$. If $0\in \partial f\left(x\right)$, then x is said to be a stable point of f, and the set of stable points of f is denoted as $critf$.

The proximal operator of a normal closed, convex function f at $v\in {\mathbb{R}}^n$ is defined as follows [32]:

$$Pro{x}_{\lambda f}\left(v\right)=arg\underset{x\in {\mathbb{R}}^n}{min}\left\{f\left(x\right)+{\displaystyle \frac{1}{2\lambda }}\Vert x-v{\Vert }^2\right\},$$

(5)

where $∥·∥$ denotes the Euclidean norm.

2.2. Loss Function

A machine learning model encompasses a set of parameters that are refined based on the training data. Typically, the training data samples include the following two components: input features represented as vectors ($a_j$) and desired outputs known as output labels ($b_j$). Each model is equipped with a loss function defined on its parameter vector (x) for each data sample. The loss function records the model error in the training data. The learning process aims to minimize this loss function over a set of training data samples ($a_j^i$). For each data sample, the loss function is designated as $f(x_i,a_j,b_j)$ or abbreviated to $f\left(x_i\right)$ for convenience.

Table 1. Loss functions for some models.

Model	Loss Function
Linear regression	${\displaystyle \frac{\lambda }{2}}\Vert x{\Vert }^2+\frac{1}{2}max{\left\{0;1-b_j{x}^{\mathrm{T}}{a}_j\right\}}^2$
Squared-SVM	$\frac{1}{2}\Vert b_j-x^{\mathrm{T}}{a}_j{\Vert }^2$
K-means	$\frac{1}{2}{\displaystyle \underset{l}{min}}\Vert a_j-x\left(l\right){\Vert }^2\mathrm{where}x\triangleq {\left[x_{\left(1\right)}^{\mathrm{T}},x_{\left(2\right)}^{\mathrm{T}},\dots \right]}^{\mathrm{T}}$

 [33,34,35,36] summarizes loss functions for popular machine learning models. For convenience, suppose there are m edge nodes and that the local datasets are $D_1,D_2,\dots ,D_m$. For each dataset ($D_i$) at node i, the loss function of the set of data samples at that node is

$$\begin{array}{c}{F}_i\left(x\right)\triangleq {\displaystyle \frac{1}{\left|{\mathcal{D}}_i\right|}}{\displaystyle \sum _{j\in {\mathcal{D}}_i}}{f}_j\left(x\right).\hfill \end{array}$$

(6)

We define $\left|D_i\right|$, where $\left|D_i\right|$ denotes the size of set $D_i$, i.e.,

$$\left|D\right|={\sum }_{i=1}^m\left|D_i\right|.$$

The global loss function of all distributed datasets is defined as

$$\begin{array}{c}F\left(x\right)\triangleq {\displaystyle \frac{{\displaystyle \sum _{j\in \underset{i}{\cup }{\mathcal{D}}_i}}{f}_j\left(x\right)}{\left|\underset{i}{\cup }{\mathcal{D}}_i\right|}}={\displaystyle \frac{{\displaystyle \underset{i=1}{\sum ^N}}\left|D_i\right|F_i\left(x\right)}{\left|D\right|}}.\hfill \end{array}$$

(7)

2.3. Symmetric ADMM

The following convex minimization model with linear constraints and a separable objective function is considered.

$$\underset{x\in {\mathbb{R}}^n,y\in {\mathbb{R}}^q}{min}f\left(x\right)+g\left(y\right),s.t.Ax+By=b,$$

(8)

where $A\in {\mathbb{R}}^{p\times n},B\in {\mathbb{R}}^{p\times q}$ and $b\in {\mathbb{R}}^p$. Such divisible convex optimization problems can be solved by using the ADMM algorithm. The augmented Lagrangian function for the above optimization problem is expressed as follows.

$$\mathcal{L}(x,y,u):=f\left(x\right)+g\left(y\right)+〈Ax+By-b,u〉+{\displaystyle \frac{\rho }{2}}\Vert Ax+By-b{\Vert }^2,$$

(9)

where $\rho$ is the penalty parameter and u is the Lagrange multiplier. ADMM follows the following update process [37]:

$$\begin{array}{cc}\hfill & x^{k+1}=arg{min}_{x\in {\mathbb{R}}^n}\mathcal{L}\left(x,y^k,u^k\right),\hfill \\ \hfill & y^{k+1}=arg{min}_{y\in {\mathbb{R}}^q}\mathcal{L}\left(x^{k+1},y,u^k\right),\hfill \\ \hfill & u^{k+1}=u^k-\rho \left(A{x}^{k+1}+B{y}^{k+1}-b\right).\hfill \end{array}$$

(10)

Based on the Peaceman–Rachford splitting method [38], symmetric ADMM (S-ADMM) was proposed to solve (9). The iterative process is expressed as follows:

$$\begin{array}{cc}\hfill & x^{k+1}=arg{min}_{x\in {\mathbb{R}}^n}\mathcal{L}\left(x,y^k,u^k\right),\hfill \\ \hfill & u^{k+\frac{1}{2}}=u^k-\rho \left(A{x}^{k+1}+B{y}^k-b\right)\hfill \\ \hfill & y^{k+1}=arg{min}_{y\in {\mathbb{R}}^q}\mathcal{L}\left(x^{k+1},y,u^{k+\frac{1}{2}}\right),\hfill \\ \hfill & u^{k+1}=u^{k+\frac{1}{2}}-\rho \left(A{x}^{k+1}+B{y}^{k+1}-b\right).\hfill \end{array}$$

(11)

2.4. Federated Learning

Suppose we have m local nodes, each with a local dataset ($D_i$). Each node has a local total loss ($f_i\left(x\right)$) as a lower-bounded function. The global loss function can, therefore, be derived as

$$\begin{array}{c}f\left(x\right):={\sum }_{i=1}^m{w}_i{f}_i\left(x\right),\hfill \end{array}$$

(12)

where $w_i,(i=1,2,\dots ,m)$ are positive weights and satisfy

$${\sum }_{i=1}^m{w}_i=1.$$

The goal of federated learning is to minimize the loss function at the central node to obtain the optimal parameter ($x^{*}$), which can be described as the following problem:

$$\begin{array}{c}{x}^{*}:=argmi{n}_{x\in {\mathbb{R}}^n}f\left(x\right).\hfill \end{array}$$

(13)

By introducing the auxiliary variable (y) and adding the constraint of $x_i=y$, the original problem can be rewritten in the following form:

$$\begin{array}{c}{\displaystyle \underset{x_i\in {\mathbb{R}}^n,y\in \mathbb{R}}{min}}{\sum }_{i=1}^m{w}_i{f}_i\left(x_i\right),\mathrm{s}.\mathrm{t}.x_i=y.\hfill \end{array}$$

(14)

Based on the above optimization problem, the conventional federated learning Algorithm 1 can be summarized in the following form [39]:

Algorithm 1 Federated Learning

1: Initialize: $x_i^0=x^0,m,\gamma >0.$ 2: for $k=1$ to n do 3:     $\underline{Globalupdate:}$ 4:     The central server calculates the average parameter $y^{k+1}$ by: 5:      $$y^k={\sum }_{i=1}^m{w}_i{x}_i^k.$$ 6:     Broadcasts the parameter $y^{k+1}$ to every local client: 7:     for $i=1$ to n do 8:         $\underline{Localupdate:}$ 9:         Each client updates its parameter locally and in parallel by: 10:          $$x_i^{k+1}=x_i^k-\gamma \nabla f\left(y^k\right).$$ 11:     end for 12: end for 13: $Return:x_i^{k+1},y^{k+1}(i=1,2,\dots ,m)$

To implement the algorithms proposed in this paper, we construct the augmented Lagrangian function for problem (14). The augmented Lagrangian function of (14) is

$$\begin{array}{c}{\mathcal{L}}_{\rho }(y,X,U):={\sum }_{i=1}^n{L}_{{\rho }_i}(y,x_i,u_i),\hfill \end{array}$$

(15)

where $X=(x_1,x_2,\cdots ,x_n)$, $U=(u_1,u_2,\dots ,u_n)$, $\rho =({\rho }_1,{\rho }_2,\cdots ,{\rho }_m)$, and

$$\begin{array}{c}{L}_{{\rho }_i}(y,x_i,u_i)=w_i{f}_i\left(x\right)-〈x_i-y,u_i〉+{\displaystyle \frac{{\rho }_i}{2}}\Vert x_i-y{\Vert }^2,\hfill \end{array}$$

(16)

where $u_i,i=1,2,\dots ,m$ are the Lagrange multipliers and ${\rho }_i,i=1,2,\dots ,m$ are the penalty parameters.

2.5. Stationary Points

Here, we present the optimality condition for problem (14).

Definition 3.

Point $(y^{*},X^{*},U^{*})$ is a stationary point of problem (14) and satisfies the following conditions:

$$\begin{array}{c}\left\{\begin{array}{c}{w}_i\nabla f_i\left(x_i^{*}\right)+u_i^{*}=0\\ x_i^{*}-y^{*}=0\\ {\displaystyle \sum _{i=1}^m}{u}_i^{*}=0\end{array}\right.,i=1,2,\dots ,m.\hfill \end{array}$$

(17)

Point $x^{*}$ is deemed a stationary point of problem (14) if it satisfies the following condition:

$$\begin{array}{c}\nabla f\left(x^{*}\right)=0.\hfill \end{array}$$

(18)

One can readily observe that any local optimal solution satisfies (17), and if each $f_i$ is a convex function, a point fulfilling (18) constitutes a globally optimal solution.

Based on the definition of the proximal operator and Definition 3, we obtain the following lemma.

Lemma 1

([39,40]). Suppose that $f_i:{\mathbb{R}}^n\to \mathbb{R},i=1,2,\cdots ,m$ are properly convex lower semicontinuous functions. Then, solving problem (15) reduces to a zero point of

$$\begin{array}{c}e(p,\varrho )=\left(\begin{array}{c}{e}_{x_i}(p,\varrho ):=x_i-Pro{x}_{\varrho f_i}\left(x_i+\varrho u_i\right)\hfill \\ e_y(p,\varrho ):=\varrho \left({\sum }_{i=1}^m{u}_i\right)\hfill \\ e_{u_i}(p,\varrho ):=\varrho (x_i-y)\hfill \end{array}\right)(i=1,2,\cdots ,m),\hfill \end{array}$$

(19)

where $p:=(X,y,U),\varrho \in {\mathbb{R}}_{+}$ for any given positive constant. For $p^{*}=\left(X^{*},y^{*},U^{*}\right)\in crit{\mathcal{L}}_{\rho }$, $e\left(p^{*},\varrho \right)=0$. Thus, $e(p,\varrho )$ can be used to measure the distance between point p and stable set $crit{\mathcal{L}}_{\rho }$.

Now, we provide the following lemma for $e(p,\varrho )$, which is important for Remark 2 and Lemma 5 in Section 3.

Lemma 2

([39,40]). Suppose that $f_i:{\mathbb{R}}^n\to \mathbb{R},i=1,2,\cdots ,m$ are properly convex and lower semicontinuous. If p is not a stable point of $crit{\mathcal{L}}_{\rho }$ and $\overline{\varrho }\ge \varrho >0$, then

$$\begin{array}{c}\Vert e(p,\overline{\varrho })\Vert \ge \Vert e(p,\varrho )\Vert ,\hfill \end{array}$$

(20)

and

$$\begin{array}{c}{\displaystyle \frac{\Vert e(p,\overline{\varrho })\Vert }{\overline{\varrho }}}\le {\displaystyle \frac{\Vert e(p,\varrho )\Vert }{\varrho }}.\hfill \end{array}$$

(21)

3. Symmetric ADMM-Based Federated Learning with a Relaxed Step and Convergence

Based on the above augmented Lagrangian functions, in this section, we construct two symmetric ADMM-based federated learning algorithms, the first of which is Fed-RSADMM, which utilizes the federated learning framework and symmetric ADMM with a relaxed step (RSADMM). The second is FedAvg-RSADMM, based on Fed-RSADMM, which allows local clients to update multiple times, then upload their parameters to the central server.

3.1. Fed-RSADMM

Given an original dataset comprising m nodes, the local parameter for the $i$-th node is set as $x_i$, and data for the $i$-th node is assigned as $D_i$. The specific algorithmic workflow proceeds as follows (Algorithm 2).

Algorithm 2 Fed-RSADMM

Input: $\alpha ,\tau ,\gamma ,{\rho }_i>0.\mathrm{S}=\left[m\right]$, $\left[m\right]:=\{1,2\dots ,m\}.$ Initialize: $x_i^0,y^0,u_i^0,i\in \left[m\right].Setk\Leftarrow 0.$ for $k=1$ to n do     for $i=1$ to m do         $\underline{Localrelaxedupdate:}$          $$x_{rs\left(i\right)}^{k+1}=\alpha x_i^k+(1-\alpha )y^k$$ (22)     end for     $\underline{Globalupdate:}$     The central server calculates the global parameter $y^{k+1}$ by      $$y^{k+1}=arg\underset{y}{min}\underset{i=1}{\sum ^m}\left(w_i{f}_i\left(x_{rs\left(i\right)}^{k+1}\right)-〈x_{rs\left(i\right)}^{k+1}-y,u_i^k〉+{\displaystyle \frac{{\rho }_i}{2}}{∥x_{rs\left(i\right)}^{k+1}-y∥}^2\right)$$ (23)     Broadcasts the parameter $y^{k+1}$ to every local client:     for $i=1$ to m do         $\underline{Localrelaxedupdate:}$          $$u_i^{k+\frac{1}{2}}=u_i^k-\tau {\rho }_i\left(x_{rs\left(i\right)}^{k+1}-y^{k+1}\right)$$ (24)          $$x_i^{k+1}=arg\underset{x_i}{min}\left(w_{i}f\left(x_i\right)-〈x_i-y^{k+1},u_i^{k+\frac{1}{2}}〉+{\displaystyle \frac{\gamma {\rho }_i}{2}}{∥x_i-y^{k+1}∥}^2\right)$$ (25)          $$u_i^{k+1}=u_i^{k+\frac{1}{2}}-\gamma {\rho }_i\left(x_i^{k+1}-y^{k+1}\right)$$ (26)              end for end for $Return:x_i^{k+1},y^{k+1}(i=1,2,\dots ,m)$

Remark 1.

In Algorithm 2, subproblem (23) can be solved by the following equation:

$$\begin{array}{c}{y}^{k+1}=arg{min}_y\mathcal{L}(y,X^k,U^k)={\displaystyle \sum _{i=1}^m}{\displaystyle \frac{{\rho }_i{x}_{rs\left(i\right)}^{k+1}}{\widehat{\rho }}}+{\displaystyle \sum _{i=1}^m}{\displaystyle \frac{u_i^k}{\widehat{\rho }}},\hfill \end{array}$$

(27)

where

$$\begin{array}{c}\widehat{\rho }:={\displaystyle \sum _{i=1}^m}{\rho }_i,\hfill \end{array}$$

(28)

Compared to traditional federated learning, we adopt the update methods outlined in (23) and (27) for the global parametersinstead of using the average of all local parameters, denoted as $x_i^{k+1}$. In contrast to the symmetric ADMM algorithm, we introduce a relaxation step to accelerate the convergence rate [19].

3.2. FedAvg-RSADMM

The communications in FedAvg-RSADMM only occur when $k\in \mathcal{K}$$=\{0,k_0,2k_0,\dots \}$, where $k_0$ is a predefined positive integer. To facilitate local updates in Algorithm 2, an auxiliary variable ($z^{k+1}=y^{\tau k+1}$) is introduced, where ${\Gamma }_k=⌊k/k_0⌋k_0$. It can be readily observed that if $k={\Gamma }_k$, then $k\in \mathcal{K}$, and when ${\Gamma }_k<k<{\Gamma }_k+k_0$, then $k\notin \mathcal{K}$, i.e.,

$$\begin{array}{c}{z}^{k+1}=\left\{\begin{array}{ccc}{y}^{k+1},\hfill & if& k\in \mathcal{K},\\ y^{{\tau }_k+1},\hfill & if& k\notin \mathcal{K}.\end{array}\right.\hfill \end{array}$$

(29)

This approach decreases the number of communication rounds (e.g., parameter feedback and parameter upload), resulting in substantial cost savings with a convergence rate of $\mathit{0}(1/K)$, where K is the number of iterations. A convex combination between local and global variables is used to formulate the relaxed step, which is then employed to perform the parameter update. The corresponding Algorithm 3 proceeds as follows.

Algorithm 3 FedAvg-RSADMM

1: Input: $\alpha ,\tau ,\gamma ,{\rho }_i>0.\mathrm{S}=\left[m\right].$ 2: Initialize: $x_i^0,y^0,u_i^0,i\in \left[m\right].Setk\Leftarrow 0.$ 3: for $k=1$ to n do 4:     for $i=1$ to m do 5:         $\underline{Localrelaxedupdate:}$ 6:          $$x_{rs\left(i\right)}^{k+1}=\alpha x_i^k+(1-\alpha )z^k$$ (30) 7:     end for 8:     if $k\in \mathcal{K}:=\left\{0,k_0,2k_0,3k_0,\cdots \right\}$ then 9:         $\underline{Globalupdate:}$ 10:         The central server calculates the global parameter $z^{k+1}$ by 11:          $$z^{k+1}=arg\underset{z}{min}\underset{i=1}{\sum ^m}\left(w_i{f}_i\left(x_{rs\left(i\right)}^{k+1}\right)-〈x_{rs\left(i\right)}^{k+1}-z,u_i^k〉+{\displaystyle \frac{{\rho }_i}{2}}{∥x_{rs\left(i\right)}^{k+1}-z∥}^2\right)$$ (31) 12:         Broadcasts the parameter $z^{k+1}$ to every local client: 13:     end if 14:     for $i=1$ to m do 15:         $\underline{Localrelaxedupdate:}$ 16:          $$u_i^{k+\frac{1}{2}}=u_i^k-\tau {\rho }_i\left(x_{rs\left(i\right)}^{k+1}-z^{k+1}\right)$$ (32) 17:          $$x_i^{k+1}=arg\underset{x_i}{min}\left(w_{i}f\left(x_i\right)-〈x_i-z^{k+1},u_i^{k+\frac{1}{2}}〉+{\displaystyle \frac{\gamma {\rho }_i}{2}}{∥x_i-z^{k+1}∥}^2\right)$$ (33) 18:          $$u_i^{k+1}=u_i^{k+\frac{1}{2}}-\gamma {\rho }_i\left(x_i^{k+1}-z^{k+1}\right)$$ (34) 19:          20:     end for 21: end for 22: $Return:x_i^{k+1},z^{k+1},u_i^{k+1}(i=1,2,\dots ,m)$

3.3. Convergence

In this section, we only provide the corresponding convergence lemmas and theorem for Algorithm 2, as those for Algorithm 3 follow a similar process. The following assumption is important for the proof.

Assumption 1.

(a) 

The function $f_i:{\mathbb{R}}^n\to \mathbb{R},i=1,2,\cdots ,m$ is lower semi-continuous.

(b) 

The function $f_i:{\mathbb{R}}^n\to \mathbb{R},i=1,\cdots$, is continuous and has the same L-Lipschitz continuous gradient.

(c) 

The parameters in the algorithm satisfy the following:

$$\gamma +\tau \ge 0,\gamma +\tau \alpha >0,0<\alpha <1,0<\gamma <1,\tau <1.$$

The penalty parameter (${\rho }_i$) complies with the following:

$${\rho }_i>{\displaystyle \frac{c+\sqrt{c^2-4h{L}^2}}{2h}},$$

where $c=\gamma \left(\alpha -\gamma \right)-\tau (1+\gamma \alpha )$, $h=\left(\tau -1\right)\left(\alpha -1\right)L^2+2\left(1-\gamma \right)L$.

(d) 

The datasets of all devices are independently and identically distributed (i.i.d).

We first prove the decreasing property of the $\left\{{\mathcal{L}}_{\rho }\left(p^k\right)\right\}$ sequence; these properties enable us to obtain Lemma 4, which, together with the optimality conditions, shows the convergence of the $\left\{p^k\right\}$ sequence.

Lemma 3.

Assuming that Assumption 1 holds, there exist $a>0$ and $b>0$ such that

$$\begin{array}{c}\hfill {\mathcal{L}}_{\rho }\left(p^k\right)-{\mathcal{L}}_{\rho }\left(p^{k+1}\right)\ge a\underset{i=1}{\sum ^m}{∥\Delta x_i^{k+1}∥}^2+b{∥\Delta y^{k+1}∥}^2,\forall k.\end{array}$$

(35)

Then, the ${\mathcal{L}}_{\rho }\left(p^k\right)$ sequence is monotonically decreasing, where m is a finite constant.

Lemma 4.

Assuming assumption A holds and $w^k$ is bounded, then

$$\begin{array}{c}\sum _{k=0}^{+\infty }|p^{k+1}-p^k|<+\infty .\hfill \end{array}$$

(36)

Theorem 1 establishes the subsequence convergence property of the iterative sequence generated by Algorithm 2.

Theorem 1

(Subsequence Convergence). Assuming the conditions in Lemma 4 are met, the set of accumulation points of the $w^k$ sequence is denoted as Ω. Then, the following conclusions hold:

(1) 

Ω is a non-empty compact set, and $dist(w^k,\Omega )\to 0$ as $k\to +\infty$;

(2) 

$\Omega \subseteq crit{\mathcal{L}}_{\rho }$.

3.4. Linear Convergence Rate

To obtain the local linear convergence rates of sequences {$p^k$} and {${\mathcal{L}}_{\rho }\left(p^k\right)$} generated by Algorithm 2, the following results require that functions $f_i,i=1,2,\dots ,m$ be convex. We also make the following assumptions:

Assumption 2.

For any $\zeta \ge {inf}_p{\mathcal{L}}_{\rho }\left(p\right)$, there exist $\epsilon >0$ and $\varsigma >0$ such that $\left|e\right(p,1\left)\right|\le \epsilon$ and ${\mathcal{L}}_{\rho }\left(p\right)\le \zeta$, which implies

$$dist\left(p,crit{\mathcal{L}}_{\rho }\right)\le \varsigma \Vert e(p,1)\Vert .$$

Remark 2.

According to Lemma 1, the expression for $∥e(p,1)∥$ is

$$\begin{array}{c}e(p,1)=\left(\begin{array}{c}{e}_{x_i}(p,1):=x_i-Pro{x}_{f_i}\left(x_i+u_i\right)\hfill \\ e_y(p,1):=\left({\sum }_{i=1}^m{u}_i\right)\hfill \\ e_{u_i}(p,1):=(x_i-y)\hfill \end{array}\right),(i=1,2,\cdots ,m).\hfill \end{array}$$

According to Lemma 2, for any given $\varrho >0$, we have

$$\Vert e(p,1)\Vert \le max\left\{\varrho ,{\displaystyle \frac{1}{\varrho }}\right\}\Vert e(p,\varrho )\Vert .$$

Thus, Assumption 2, we can also set $dist(p,crit{\mathcal{L}}_{\rho })\le \varsigma \left|e(p,\varrho )\right|$, where $\left|e\right(p,\varrho \left)\right|\le \epsilon$ and ${\mathcal{L}}_{\rho }\left(p\right)\le \zeta$.

Assumption 3.

For any given $\overline{p}=(\overline{X},\overline{y},\overline{U})\in crit{\mathcal{L}}_{\rho }$ and $\tilde{p}=(\tilde{X},\tilde{y},\tilde{U})\in crit{\mathcal{L}}_{\rho }$, there exists $\delta >0$ such that when $|\overline{p}-\tilde{p}|\le \delta$, ${\mathcal{L}}_{\rho }\left(\overline{p}\right)={\mathcal{L}}_{\rho }\left(\tilde{p}\right)$ holds.

To prove the convergence rate, we also need Lemma 5.

Lemma 5.

Suppose Assumption 1 holds and the functions $f_i,i=1,2,\cdots ,m$ are convex; then, there exists ${\theta }_1,$${\theta }_2>0$ such that

$$\begin{array}{c}e\left(p^{k+1},1\right)\le {\theta }_1{\sum }_{i=1}^m\Vert \Delta {x_i}^{k+1}\Vert +{\theta }_2\Vert \Delta y^{k+1}\Vert ,\forall k.\hfill \end{array}$$

(37)

We have shown that Algorithm 2 converges. Now, we would like to see how fast this convergence is based on Lemma 5, as expressed by Theorems 3 and 4.

Theorem 2.

Suppose Assumptions 1 and 2, and the conditions in Lemma 4 hold and the that functions $f_i,i=1,2,\dots ,m$ are convex; then, the following conclusions are valid:

(1) 

$dist\left(p^k,crit{\mathcal{L}}_{\rho }\right)\to 0,k\to +\infty$;

(2) 

For any given ${\overline{p}}^k\in crit{\mathcal{L}}_{\rho }$ and $∥{\overline{p}}^k-p^k∥=dist(w^k,crit\mathcal{L}\rho )$, there exists a positive integer ($\widehat{k}$) such that

$${\mathcal{L}}_{\rho }\left({\overline{p}}^k\right)=\underset{k\to +\infty }{lim}{\mathcal{L}}_{\rho }\left(p^k\right)=\underset{k}{inf}{\mathcal{L}}_{\rho }\left(p^k\right),\forall k\ge \widehat{k};$$

(3) 

The {${\mathcal{L}}_{\rho }\left(p^k\right)$} sequence is Q-linearly convergent.

Theorem 3.

Assuming the conditions in Theorem 2 hold, the $p^k$ sequence converges to $crit{\mathcal{L}}_{\rho }$ with an R-linear convergence rate.

See the proofs of Lemmas 3 and 4, and Theorem 1, as well as Lemma 5, Theorems 2 and 3, in Appendix A.

4. Numerical Experiment

In this section, the performances of Algorithms 2 and 3 are demonstrated by two numerical arithmetic examples, namely linear regression and logistic regression, respectively. All numerical experiments were conducted on a laptop computer with 16 GB of RAM and an Intel(R) Core(TM) i5-12500H 2.3 GHz CPU, using MATLAB (R2021a) for implementation.

4.1. Testing Examples

In our examples, each local client is designated with its own objective function ($f_i,i=1,2,\dots ,m$, where m signifies the number of client nodes). Subsequently, every client generates random datasets ($A_i=[a_1^i,a_2^i,\cdots ,a_n^i]$ and ${\mathrm{b}}_i=[b_1^i,b_2^i,\cdots ,b_n^i]$). In these datasets, $a_j^i,(i=1,\dots ,m;j=1,2,\dots ,n)$ represents the feature data of dimension d, whereas $b_j^i,(i=1,\dots ,m;j=1,2,\dots ,n)$ denotes the label data of dimension 1.

Regarding Algorithms 2 and 3, we establish identical parameters across each node. These parameters include a relaxed step weight of $\alpha =0.5$, a penalty parameter (${\rho }_i$) set to 1 for each node, a relaxed factor ($\tau$) of 0.1 for the initial dual step, and a relaxed factor ($\gamma$) of 0.5 for the subsequent dual step at every node.

Example 1

(Linear regression). Linear regression, a canonical problem in machine learning, seeks to construct a linear function from a specified dataset, enabling the prediction of relationships between input and output variables. In this context, the objective functions for local nodes are given by

$$\begin{array}{c}{f}_i\left(x\right)=\frac{1}{2}{\displaystyle \underset{j=1}{\sum ^n}}{\left(x^T{a}_j^i-b_j^i\right)}^2,i=1,2,\cdots ,m.\hfill \end{array}$$

(38)

In this function, $a_j^i\in {\mathbb{R}}^n$ and $b_j^i\in \mathbb{R}$ denote the j-th sample for client i. It should be noted that the above objective function formulates a convex quadratic optimization problem. For this scenario, we randomly generate features ($A_i$) and corresponding labels ($b_i$) from a uniform distribution within the interval of $[0,1]$. For the purpose of simplification, we initially set $n=100$ while selecting $m\in [10,100]$. Subsequently, we solidify $m=30$ and let n fall in the range of $[100,200]$.

Example 2

(Logistic regression). Logistic regression, a prevalently utilized classification algorithm, is especially apt for handling binary classification predicaments. Within this context, local clients define their objective functions as

$$\begin{array}{c}{f}_i\left(x\right)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{j=1}^n}log\left(1+exp\left(-b_j^i{x}^T{a}_j^i\right)\right),i=1,2,\cdots ,m,\hfill \end{array}$$

(39)

where $a_j^i\in {\mathbb{R}}^n$ and $b_j^i\in \mathbb{R}$ correspond to the j-th sample of client i. Features ($A_i$) are randomly generated in accordance with a uniform distribution spanning the interval of $[0,1]$, and labels ($b_i$) are derived from the set of $\left\{-1,1\right\}$. Each nodal dataset is designated with a unique dimensionality. In the first instance, datasets are defined such that $m=100$ and $n=1000$, with $k_0$ permitted to be selected from the set of $\left\{5,8,10,20,25\right\}$. In a subsequent iteration, the dataset configuration persists with $m=100$, while n is expanded to 2000, continuously allowing for the selection of $k_0$ from $\left\{5,8,10,20,25\right\}$.

4.2. Numerical Results

Data generation was conducted in accordance with Example 1. Principal component analysis was then employed to illustrate the distribution of the ensuing data. In this case, the five-dimensional feature data were condensed into a two-dimensional plane, with a color bar employed to signify the continuum of label values, thereby delineating the different random distributions within the $[0,1]$ interval that each original sample encountered at the data nodes.

In Figure 1a–j, the first principal component resulting from the dimensionality reduction via principal component analysis is represented along the x axis, while the second principal component is plotted along the y axis. Figure 1 suggests that the data are variably randomly distributed across the ten nodes.

Upon employing Algorithm 2 for the case illustrated in Example 1 and stipulating the number of communication rounds as 20, a two-stage process was adopted. To initiate the process, the number of data points for a single node is set to $n=100$, and m, denoting the number of nodes, is varied. This produces the graphical output depicted on the left. Subsequently, with a constant node count of $m=30$, the quantity of sample points per node is varied, resulting in the representation displayed on the right.

Figure 2 presents the variation in average node loss for different data quantities. Figure 2a depicts a scenario with a single-node data sample of $n=100$ and a node count m ranging within $\{10,30,50,80,100$}. As evidenced in Figure 2a, the average loss function for nodes under Algorithm 2 descends most rapidly for $m=100$ and most sluggishly for $m=10$. Consequently, it can be inferred that an increase in the number of nodes accelerates the decrease in the loss function, implying higher accuracy. Figure 2b considers five scenarios where $m=30$ and n varies within the range of $\left\{100,130,150,180,200\right\}$. The average node loss function under Algorithm 2 is found to descend most swiftly for $n=100$ and most slowly for $n=200$. This indicates that a reduction in the number of nodes expedites the descent of the loss function.

In addressing Example 1, Algorithm 2 was employed to determine parameters, which were then integrated into the linear model. Subsequently, the original feature data were incorporated, and labels for the original feature data were computed, which were then compared with the original data labels. Dimensionality reduction was achieved via principal component analysis to display a comparative graph of the original and model-predicted data.

Figure 3a–j employ distinct symbols to represent raw and predicted data, with color gradients depicting the corresponding label values. Dot markers indicate the sample data points post dimensionality reduction of the original data via principal component analysis. Cross markers denote data processed through the linear regression model generated by Algorithm 2, with the subsequent predicted labels presented in reduced dimensionality, also by principal component analysis. A comparison between the original and predicted data, as showcased in Figure 3, reveals a high level of accuracy attained by the linear regression model in conjunction with Algorithm 2.

A comparison is also made with conventional Distributed Machine Learning (DML) [34] and Federated Learning (FL) [28]. The loss function progression of these three algorithms is depicted accordingly.

Figure 4 displays the three corresponding loss function curves. From top to bottom, the curves represent the following distinct models. The first pertains to conventional FL, the second to DML, and the third to Algorithm 2. It is observed that the loss function value for Algorithm 2 (Fed-RSADMM) decreases more rapidly than that for DML and FL and also yields the smallest value upon convergence. Figure 4 suggests that Algorithm 2 exhibits superior accuracy and faster convergence compared to traditional algorithms.

To compare the time efficiency of the algorithms, we conducted a comparative analysis of their execution times, yielding the following results.

Figure 5 provides a comparative analysis of the execution times for FL, DML, and Fed-RSADMM applied to test Example 1 over a series of iterations. The performance of the Fed-RSADMM algorithm is traced by the dashed green line, which consistently shows reduced execution times, in contrast to FL (solid blue line) and DML (dash–dot red line). The flatter trajectory of the Fed-RSADMM line across the iteration spectrum underscores its time-efficiency advantage. In essence, the results from test Example 1 endorse the Fed-RSADMM algorithm’s superior time efficiency, with its modest increase in execution time demonstrating potential for scalable and efficient processing in iterative tasks.

Subsequently, Algorithm 3 is subjected to verification via its application to Example 2. To commence, $k_0=5$ is established within Algorithm 3, followed by a comparison with the conventional federated learning algorithm, resulting in the accompanying comparative results.

As depicted in Figure 6, the upper curve corresponds to the loss function for the traditional FL algorithm, whereas the lower curve represents Algorithm 3 (FedAvg-RSADMM). The graph illustrates that Algorithm 3 exhibits a more rapid rate of descent than the FL algorithm, indicating its superior performance over the conventional FL algorithm.

The subsequent analysis focuses on discerning the effect of $k_0$ on the performance of Algorithm 3. While resolving the logistic regression problem of Example 2 using Algorithm 3, the sample data size of a single node is kept constant at 1000 and 2000, while $k_0$ varies within the range of $\left\{5,8,10,20,25\right\}$, leading to the ensuing comparison.

Figure 7 plots the number of iterations of the local variable (x) on the x axis against the average loss of the nodes on the y axis for different m, n, and $k_0$ values. As per the flow of Algorithm 3, a larger $k_0$ implies fewer global updates. Consequently, Figure 7 reveals a slower descent of the node’s loss function with larger $k_0$ values, which is attributable to the reduced number of steps in the global update, thereby conserving computational and communication resources. However, it is noted that Algorithm 3 exhibits convergence similar to that achieved with various $k_0$ values. This suggests that an increased $k_0$ effectively reduces communication and computational resource consumption, inducing only a minor error loss. Accordingly, a modest increase in $k_0$ in Algorithm 3 can boost its computational efficiency. The data sample size for a single node is then elevated as $n=2000$ based on the experiment illustrated in Figure 7a for n = 1000, yielding the results portrayed in Figure 7b, which mirroring those presented in the left figure. Thus, it follows that the accuracy of Algorithm 3 remains unimpacted by the escalation of local sample data size, rendering the algorithm suitable for federated learning problems involving extensive data volumes and multiple nodes.

In Figure 8, the number of iterations of the global variable y is denoted by the x-axis, while the average loss of the nodes forms the y-axis, resulting in the displayed loss curve. The figure exhibits faster node loss function decreases with larger $k_0$, attributable to increased local updates for sizeable $k_0$ during global variable updating. Consequently, fewer global update steps are required for convergence, thus significantly diminishing the number of communications and global update steps. Therefore, Algorithm 3 can be deemed effective in reducing communication losses. Again, the data sample size for a single node is amplified to $n=2000$ based on the experiments of Figure 8a, culminating the results in Figure 8b. These demonstrate that Algorithm 3 is fitting for distributed optimization problems involving considerable local node data.

To investigate the impact of varying initial iteration counts ($k_0$) and the number of client nodes (m) on the temporal efficiency of the algorithm, we conducted a series of experiments. The parameter influences the number of local updates ($k_0$), while m corresponds to the number of client nodes involved in the computation. Our objective was to ascertain whether an increased number of nodes affects the algorithm’s parallelism. The experimental outcomes are listed in Figure 9.

Figure 9a illustrates the relationship between the iterative time and the communication rounds at different $k_0$ values for Algorithm 3 . An increment in $k_0$ corresponds to a decrease in the total number of iterations required, allowing Algorithm 3 to halt earlier and utilize less iterative time. Each line in the graph represents the algorithm’s performance with a different $k_0$ value, demonstrating that higher values lead to quicker convergence, as evidenced by the curves leveling off sooner. This suggests that optimizing the $k_0$ parameter can enhance the algorithm’s efficiency, reducing computational time while maintaining convergence integrity.

Figure 9b demonstrates the relationship between the iterative time and $k_0$ with different numbers of nodes (m) for Algorithm 3. It is observed that an increase in m leads to a longer iteration time. However, the overall time decreases with an increase in $k_0$, indicating that our algorithm exhibits favorable parallelism. Despite this, the influence of node count on computation time is not entirely negated; therefore, the potential time cost due to an increase in nodes must be considered in computational evaluations. This underscores the necessity of balancing the number of nodes against the performance gains achieved through parallel processing when deploying the algorithm in distributed computing environments.

Obviously, Algorithm 3 considers the communication and computation costs in relation to $k_0$. Specifically, the framework states that the global update occurs only at certain steps (e.g., at step km being a multiple of a pre-defined integer ($k_0$)). It is shown that the larger the $k_0$, the less time for our algorithm to converge (see Figure 7, Figure 8 and Figure 9). In addition, the local computational complexity at each node is $O\left(k\right)$, where k is the iteration time. By adjusting $k_0$, we can optimize the trade-off between communication efficiency and computational load, demonstrating the scalability and adaptability of the algorithm for federated learning.

5. Conclusions

This study introduces two symmetric ADMM-based federated learning algorithms relaxed steps. Algorithm 2 bolsters computational efficiency in federated learning, while Algorithm 3 capitalizes on Algorithm 2 to further optimize communication efficiency. Relevant numerical experiments were set up to illustrate the feasibility and efficiency of the algorithms. In conclusion, the two proposed algorithms exhibit rapid convergence and excellent performance. While experiments based on linear and logistic regressions were conducted on small scales and only serve as proofs of concept, in contrast to [41,42], which researched large-scale cases and realistic usage. Therefore, exploring applications related to large-scale optimization problems will be the subject of further research in the future.