An Adaptive Low Computational Cost Alternating Direction Method of Multiplier for RELM Large-Scale Distributed Optimization

Abstract

In a class of large-scale distributed optimization, the calculation of RELM based on the Moore–Penrose inverse matrix is prohibitively expensive, which hinders the formulation of a computationally efficient optimization model. Attempting to improve the model’s convergence performance, this paper proposes a low computing cost Alternating Direction Method of Multipliers (ADMM), where the original update in ADMM is solved inexactly with approximate curvature information. Based on quasi-Newton techniques, the ADMM approach allows us to solve convex optimization with reasonable accuracy and computational effort. By introducing this algorithm into the RELM model, the model fitting problem can be decomposed into a set of subproblems that can be executed in parallel to achieve efficient classification performance. To avoid the storage of expensive Hessian for large problems, BFGS with limited memory is proposed with computational efficiency. And the optimal parameter values of the step-size search method are obtained through Wolfe line search strategy. To demonstrate the superiority of our methods, numerical experiments are conducted on eight real-world datasets. Results on problems arising in machine learning suggest that the proposed method is competitive with other similar methods, both in terms of better computational efficiency as well as accuracy.

1. Introduction

The extreme learning machine (ELM) [1] has received much attention in recent years, owing to its fast training speed and good generalization. Note that traditional extreme learning machines suffer from a limitation of memory with large scale datasets. Especially in the era of big data, the dataset scale is usually extremely large and the data are often very high-dimensional for detailed information [2,3,4,5,6] because the increasing complexity of the datasets enlarges the dimension of the hidden output matrix, which leads to a huge memory space and heavy computational load in matrix-inversion based (MI-based) solutions.

To address these limitations, some enhanced ELMs with parallel or distributed structures have been implemented to meet the challenge of large-scale datasets, as shown in Table 1. For example, ELM based on the MapReduce framework can effectively calculate the matrix multiplication in parallel [7,8], and has an efficient learning ability in massive rapidly updated datasets [9]. However, the parallel ELM based on MapReduce creates a large amount of extra overhead and degrades the learning speed. The algorithm based on the Spark parallel framework is then proposed to speed up the whole computing process of ELM for big data [10].

The methods discussed above focus on the computation of MI-based solutions using parallel and distributed hardware structures and programming models. The alternating direction method of multipliers (ADMM), without the time-consuming MI operation, is also an effective method for distributed optimization [3,11,12]. By using the ADMM framework, the model fitting problem can be decomposed into a set of subproblems that can be executed in parallel to achieve efficient classification performance so as to meet the needs of large-scale data processing in the real environment. To achieve optimal performance without user oversight, an adaptive method that automatically tunes the key algorithm parameters is applied to improve the Relaxed ADMM [13]. Appropriate selection of the penalty parameter is crucial to obtaining good performance from the ADMM. While analytic results for optimal selection of the penalty parameter are very limited, an adaptive penalty strategy based on residual balancing is proposed to obtain good performance from the ADMM [14] because a convex model fitting problem can be split into a set of concurrently executable subproblems. Therefore, in a big data environment, the regularized least-squares problem is split across the coefficients and incorporates a relaxation technique to achieve good convergence [15]. Furthermore, elastic-net theory is employed to simultaneously improve the sparsity and stability of the model, which develops an accelerated ADMM algorithm [16].

Table 1. Review of various approaches of enhanced ELMs in the literature.

Framework	Utilized Techniques	Metrics	Datasets	Main Characteristics
Parallel or distributed learning	MapReduce [7]	Running time, Speed up	Synthetic datasets	Parallel computing ability, Efficient learning of large-scale data
	MapReduce [9]	Running time, Update ratio	Synthetic datasets	Efficient learning in massive rapidly updated dataset
	MapReduce [8]	Speed up, Scaleup, Sizeup	Real datasets	Parallelism, Low runtime memory, Good scalability
	Spark [10]	Running time, Speed up, Accuracy	Synthetic datasets	Fault tolerance, Persist/cache strategies
ADMM	Residuals normalization [14]	Iteration number	Synthetic datasets	Robust in sparse coding
	Adaptive penalty, Relaxation technique [13]	Iteration number	Real datasets	Without user oversight or parameter tuning
	Maximally splitting, Relaxation technique [15]	Convergence ratio, Acceleration ratios	Real datasets	Fast convergence, Less computations, High parallelism
	Inertial technique, Bregman distance [17]	Training time, Constraint errors	Synthetic datasets	Global convergence, High acceleration

Table 1. Review of various approaches of enhanced ELMs in the literature.

Framework	Utilized Techniques	Metrics	Datasets	Main Characteristics
Parallel or distributed learning	MapReduce [7]	Running time, Speed up	Synthetic datasets	Parallel computing ability, Efficient learning of large-scale data
	MapReduce [9]	Running time, Update ratio	Synthetic datasets	Efficient learning in massive rapidly updated dataset
	MapReduce [8]	Speed up, Scaleup, Sizeup	Real datasets	Parallelism, Low runtime memory, Good scalability
	Spark [10]	Running time, Speed up, Accuracy	Synthetic datasets	Fault tolerance, Persist/cache strategies
ADMM	Residuals normalization [14]	Iteration number	Synthetic datasets	Robust in sparse coding
	Adaptive penalty, Relaxation technique [13]	Iteration number	Real datasets	Without user oversight or parameter tuning
	Maximally splitting, Relaxation technique [15]	Convergence ratio, Acceleration ratios	Real datasets	Fast convergence, Less computations, High parallelism
	Inertial technique, Bregman distance [17]	Training time, Constraint errors	Synthetic datasets	Global convergence, High acceleration

For real-time data classification in the convex optimization problem, the prime problem can be decomposed into several subproblems by leveraging ADMM [18]. The global optimal solution to the original problem can be obtained by processing the subproblem in parallel. Fast convergence speed and parallelism make ADMM suitable for solving large-scale distributed optimization problems. However, subproblem optimization must be solved at each iteration, which imposes a heavy calculation burden [19]. Numerical experience has shown that the effective solution of subproblems is critical to the performance of ADMM [20].

Several alternatives are available for unconstrained optimization such as Newton-type methods [21], Chebyshev-like methods [22], the quasi-Newton method(QNM) [23], and so on [24]. These methods require less computational effort to calculate the search direction, thus demonstrating rapid convergence. The RLS problem in RELM mainly involves the computation of a Hessian matrix and the gradient of the cost function. The second-order partial derivatives of the Hessian matrix can be avoided by putting forth the displacement and first derivative information of two adjacent entries [25,26]. Combined with line search techniques, it could achieve attractive global convergence properties.

However, in machine learning and image processing, the computation of the Hessian matrix at each iteration is not a trivial task [13]. The cost of storing and working with Hessian approximations can be excessive for large matrixes. In order to reduce the storage of the Hessian approximations, variants of the quasi-Newton approach, such as limited-memory BFGS (LBFGS) and stochastic QNM [27,28,29,30,31], are developed to store Hessian approximations compactly. Azam et al. [32] analyzed the convergence performance of L-BFGS in convex optimization problems and further proved their practical value in solving large-scale optimization problems. Aryan et al. [28] proposed stochastic QNM(SQN), which uses second-order information to accelerate stochastic convergence, and modified the update formula of BFGS to ensure that the eigenvalues of the Hessian approximation matrix remain bounded, so as to ensure that the function can obtain extreme values. Chen et al. [30] proposed the stochastic damped L-BFGS to ensure positivity and avoid ill-conditioned outputs in the Hessian update process by introducing damping parameters. The computation of Hessian matrixes often involves the step-size parameter, which can be determined by line search method. Backtracking line search is commonly used to guarantee convergence in case the linear model assumptions break down and an unstable stepsize is produced. It is time-consuming and may lose its advantage in other types of ELMs.

In this paper, we study a low-cost computational scheme for the ADMM and jointly devise an adaptive step-size selection. The stochastic damped optimal L-BFGS(R-SDL-BFGS) is therefore derived, which improves the computational efficiency of the ADMM. Our contributions can be summarized as follows:

(1)

Low-cost computational scheme: The curvature information from recent iterations is used to reduce computational cost;

(2)

Damped BFGS correction scheme: The damping technology is introduced into the BFGS to make up for the deficiency of the Hessian approximation matrix in the positive definiteness of the solution, and to ensure the positive definiteness of the BFGS matrix in non-convex optimization;

(3)

Step factor selection scheme: The non-monotonic Wolfe-type strategy is applied to the memory gradient method, combined with the BB spectral gradient descent, to obtain the optimal step-size factor.

Finally, we compare the proposed method to other ADMM variants by experiments of real-world classification and image processing problems.

2. Preliminaries

2.1. Convex Optimization

There are great advantages to recognizing or formulating a problem as a convex optimization problem. The most basic advantage is that the problem can then be solved, very reliably and efficiently, using interior-point methods or other special methods for convex optimization. These solution methods are reliable enough to be embedded in a machine learning model, or even a real-time reactive or automatic control system. An example would be the single-node perceptron neural network, in which the parameter optimization can be formulated as a convex optimization problem.

The idea behind the convex optimization problem can be extended to a more general form, in which the convex minimization problem can be expressed as an unconstrained optimization problem. There are also theoretical or conceptual advantages in formulating a problem as an unconstrained problem in optimization research. For the purpose of achieving good numerical performance, it is helpful to make use of convex functions to find bounds on the optimal value. Thus, in most cases where combinatorial optimization and global optimization is needed, it is highly advantageous to make use of it.

In particular, the unconstrained optimization problem can be summarized as follows.

$$minf\left(\mathit{x}\right),\mathit{x}\in {\mathit{R}}^n$$

(1)

where $f\left(\mathit{x}\right)$ denotes the objective function with variable $x\in R$.

There is in general no analytical formula for the solution of convex optimization problems, but there are very effective methods for solving them. The ability to analytically solve the unconstrained optimization problem is the basis for BFGS [31], a powerful method for convex optimization. For convex optimization problem, the BFGS resorts to an iterative algorithm to achieve accurate solutions. A general iterative numerical method has the form:

$${\mathit{x}}_{k+1}={\mathit{x}}_k+{\alpha }_k{\mathit{d}}_k$$

(2)

$${\mathit{d}}_k=-{{\mathit{B}}_k}^{-1}{\mathit{g}}_k$$

(3)

where ${\mathit{d}}_k$ is search direction, ${\mathit{g}}_k$ indicates the gradient of $f\left({\mathit{x}}_k\right)$ at ${\mathit{x}}_k$, and ${\alpha }_k$ is the step size. Here, ${\mathit{B}}_k$ denotes the approximate Hessian matrix. The BFGS algorithm approximates the inverse of the Hessian matrix by constructing a symmetric, positive definite initial moment ${\mathit{B}}_0$, and then iteratively updating the $\mathit{B}$.

The essence of BFGS is to approximate the Hessian using finite differences computed by successive iterations and gradients, so the gradient estimation and the iterative difference are defined as follows:

$${\mathit{y}}_k={\mathit{g}}_{k+1}-{\mathit{g}}_k$$

(4)

$${\mathit{s}}_k={\mathit{x}}_{k+1}-{\mathit{x}}_k$$

(5)

Because the size of the Hessian may vary across iterations, it is not trivial to calculate an appropriate Hessian at each iteration. To address this problem, this paper seeks to approximate the Hessian to avoid calculating the inverse of the Hessian matrix for each iteration.

2.2. Limited-Memory BFGS Algorithm

Different correction formulas of the quasi-Newton matrix ${\mathit{B}}_{k+1}$ represent different QNM. The BFGS correction formula can maintain the symmetric positive definiteness of the iterative matrix ${\mathit{B}}_k$. And the BFGS algorithm has global convergence for the convex minimization problem, and therefore the BFGS correction formula is commonly used to solve ${\mathit{B}}_{k+1}$. The BFGS correction formula is as follows:

$${\mathit{B}}_{k+1}={\mathit{B}}_k-\frac{{\mathit{B}}_k{\mathit{s}}_k{\mathit{s}}_k{\mathit{B}}_k^T}{{\mathit{s}}_k^T{\mathit{B}}_k{\mathit{s}}_k}+\frac{{\mathit{y}}_k{\mathit{y}}_k^T}{{\mathit{y}}_k^T{\mathit{s}}_k}$$

(6)

The line search method has been proved useful in practice since selection of the step size ${\alpha }_k$ determines where along the line $\left\{x_k+{\alpha }_k{d}_k|x\in R\right\}$ the next iterate will be. The exact line search is used when the cost of the minimization problem is with one variable. However, in an exact line search method, the values of ${\mathit{B}}_k$ and one or more derivatives at multiple values of k must be found. It usually gives a very small improvement in efficiency. For these reasons, most practical implementations use an inexact line search, which is guaranteed to be feasible for the unconstrained problem. Common inexact line search criteria are given by [23]:

(1)

Armijo criteria:

$$f(x_k+{\alpha }_k{d}_k)\le f\left(x_k\right)+c_1{g}_k^T{\alpha }_k{d}_k$$

(7)

(2)

Wolfe criteria:

$$g{(x_k+{\alpha }_k{d}_k)}^T{d}_k\le f\left(x_k\right)+c_2{g}_k^T{d}_k$$

(8)

where $0<c_1<c_2<1$.

The BFGS algorithm finds it difficult to solve large-scale unconstrained optimization problems on account of the calculation and storage of metrics in each iteration. That is, in each iteration, the BFGS approximates the inverse of the Hessian with gradient information, and the computional cost is expensive. In this section, the LBFGS method [33] is introduced to reduce the computational cost. It stores the matrix in a vector form that makes it possible to reduce computational effort for transmission or storage of data (assuming the vectors are stored using an appropriate data structure). To simplify the calculation of the inversion matrix, by applying the the Sherman–Morrison formula, the corresponding BFGS correction formula is transformed into:

$$\begin{array}{c}{\mathit{B}}_{k+1}^{-1}=(\mathit{I}-\frac{{\mathit{s}}_k{\mathit{y}}_k^T}{{\mathit{y}}_k^T{\mathit{s}}_k}){\mathit{B}}_k^{-1}(\mathit{I}-\frac{{\mathit{y}}_k{\mathit{s}}_k^T}{{\mathit{y}}_k^T{\mathit{s}}_k})+\frac{{\mathit{s}}_k{\mathit{s}}_k^T}{{\mathit{y}}_k^T{\mathit{s}}_k}\hfill \end{array}$$

(9)

$$\begin{array}{c}{\mathit{y}}_k=\frac{1}{M_k}\sum _{k=1}^{M_k}{f}^{\prime }\left({\mathit{x}}_{k+1}\right)-f^{\prime }\left({\mathit{x}}_k\right)\hfill \end{array}$$

(10)

where $M_k$ is storage length.

3. Adaptive Stochastic Damping Optimization for Limited Memory

The positive definite matrix plays a significant role in convex optimization. And the BFGS uses the positive definite matrix ${\mathit{B}}_k$ to approximate the Hessian. Note that, during iterations, the ${\mathit{B}}_k$ may become a singular matrix, which will significantly affect the convergence of the algorithm. Simultaneously, BFGS requires that the optimization problem must be convex. Otherwise, the ${\mathit{B}}_k$ may become non-positive, and the decreasing step size may not be positive. Therefore, we need to deal with non-convexity and ill-conditioned behavior to guarantee the positivity of ${\mathit{B}}_k$.

3.1. Proposed Damped SL-BFGS Method

In the optimization process of BFGS, the Hessian approximation ${\mathit{B}}_k$ may become a non-positive definite matrix if the property ${\mathit{s}}_k^T{\mathit{y}}_k>0$ is not satisfied. In this case, it may not be possible to ensure that the algorithm is along the best search direction. Considering that the convergence of BFGS relies heavily on the positive definite matrix, damping technology [23] is used to correct the BFGS update formula. This makes up the deficiency of the solution to the positive definiteness of the Hessian approximation, which therefore maintains the positive definiteness of the ${\mathit{B}}_k$.

The optimization of LBFGS reduces the computational cost from the perspective of transmitting or storing data. Nevertheless, such modifications do affect the accuracy of the Hessian approximation. Stochastic optimization methods, as a popular optimization tool, can effectively obtain good analytical solutions. By applying the stochastic gradient information to approximate the curvature of the objective function in the convex optimization, the optimal analytical solution can be obtained, which also speeds up the convergence.

Because the noise of the stochastic gradient may be infinitely amplified in the curvature estimation, the Hessian approximation matrix will be negatively affected, which reduces the convergence speed. We shall adjust the gradient estimation ${\mathit{y}}_k$ and descent distance estimation ${\mathit{s}}_k$ by different batch sizes, thereby decoupling the computations of stochastic gradient and curvature estimation. By extending the random damping technique to the LBFGS, ${\mathit{y}}_k$ and ${\mathit{s}}_k$ can be represented by ${\widehat{\mathit{y}}}_k$ and ${\widehat{\mathit{s}}}_k$, respectively.

$${\widehat{\mathit{s}}}_k={\widehat{\mathit{x}}}_{k+1}-{\widehat{\mathit{x}}}_k,{\widehat{\mathit{x}}}_k=\left\{\begin{array}{c}\frac{1}{b}{\displaystyle \sum _{k=(k-1)b}^{jb-1}}{\mathit{x}}_k,j\ge 1\\ \\ {\mathit{x}}_0,j=0\end{array}\right.$$

(11)

$${\widehat{\mathit{y}}}_k={\omega }_k{\mathit{y}}_k+(1-{\omega }_k){\mathit{B}}_k{\widehat{\mathit{s}}}_k$$

(12)

$${\omega }_k=\left\{\begin{array}{c}\frac{0.8{\widehat{\mathit{s}}}_k^T{\mathit{B}}_k{\mathit{s}}_k}{{\widehat{\mathit{s}}}_k^T{\mathit{B}}_k{\mathit{s}}_k-{\widehat{\mathit{s}}}_k^T{\mathit{y}}_k},{\widehat{\mathit{s}}}_k^T{\mathit{y}}_k<0.2{\widehat{\mathit{s}}}_k^T{\mathit{B}}_k{\widehat{\mathit{s}}}_k\\ \\ 1,\mathrm{others}\end{array}\right.$$

(13)

where b is interval length (also called batch size), and scalar ${\omega }_k$ denotes the damping factor.

It is possible to design a more efficient model that computes the stochastic gradient and target variable with a batch update of batch size b in each iteration.

$${\widehat{\mathit{x}}}_{k+1}=\left\{\begin{array}{c}{\widehat{\mathit{x}}}_{{}_k}-{\gamma }_k{\mathit{B}}_k^{-1}{\mathit{g}}_k,mod(d,k)=0\\ \\ {\widehat{\mathit{x}}}_k-{\gamma }_k{\mathit{g}}_k,\mathrm{others}\end{array}\right.$$

(14)

$${\mathit{g}}_k=f^{\prime }\left({\widehat{\mathit{x}}}_k\right){\gamma }_k=\frac{{\gamma }_0{\tau }_0}{{\tau }_0+k}$$

(15)

where ${\gamma }_k$ represents the batch step size, and ${\gamma }_0{\tau }_0$ are constant.

Although knowledge of gradient information allows BFGS to gradually approximate the inverse of the Hessian, the search direction also plays a crucial role in the global convergence. We should ensure that the algorithm makes reasonable progress along the given search direction and focus on finding a suitable step length along this direction.

3.2. Robust Optimization Approach for Limited Memory

As a common descending direction search criterion, the key to the success of the inexact line search is that the convergence performance of each step must be monotonically decreasing. In many cases, it is possible to leverage a non-monotonic search technique [34] to relax the convergence conditions while overcoming the oscillation phenomenon, whereas this method makes it easy to obtain the local extrema when the initial value is taken near the local function valley.

To avoid the above problems, a non-monotonic Wolfe search strategy can be devised. This method combines current and past function iteration information to find global solutions. By introducing this method for a convex optimization problem, the objective value changes by rules based on

$${\mathit{x}}_k=\left\{\begin{array}{c}-{\mathit{x}}_k,k=1\\ -[(1-{\xi }_k){\mathit{x}}_k+{\xi }_k{\mathit{x}}_k],k\ge 2\end{array}\right.$$

(16)

$${\xi }_k=\frac{\zeta {∥{\mathit{x}}_k∥}^2}{{∥{\mathit{x}}_k∥}^2+\left|{\mathit{x}}_k^T{\mathit{x}}_{k-1}\right|},\zeta \in (0,1)$$

(17)

4. The Adaptive Stochastic Optimization to RELM Design

Consider a large-scale optimization problem, in which the training data has high dimension and large volume. The MI-based RELM method will require more computations. As a powerful tool for solving large-scale optimization problems, ADMM can greatly improve the speed of convergence by parallel computing. Using ADMM, a large-scale optimization problem can be split into a set of concurrently executable subproblems, each with just a subset of model coefficients. As for ADMM, efficiency and robustness of the numerical computation greatly depend on the effective solution of the subproblem. The complexity of subproblems therefore limits the convergence of the algorithm.

4.1. AADMM with Low Computing Cost

ADMM has advantages of dealing with convex optimization problems owing to its good convergence and parallel structure [35]. In practice, optimal trade-off solutions are explored for convex optimization problems, based on the separable structure. Define $[\mathit{A}=[{\mathit{a}}_1,{\mathit{a}}_2,\dots ,{\mathit{a}}_P]\in {\mathit{R}}^{N\times P},{\mathit{a}}_i\in {\mathit{R}}^N]$ as the data matrix. $\mathit{x}\in {\mathit{R}}^P$ indicates the model coefficient, $\mathit{b}\in {\mathit{R}}^N$ denotes the target output, $f(·)$ and $h(·)$ are the convex loss function and the convex regularization function, respectively. $\mathit{x}$ can be partitioned as ${[{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_L]}^T,{\mathit{x}}_l\in {\mathit{R}}^{p_l}$ where ${\sum }_{l=1}^L{\mathit{p}}_l=P$. Similarly, partition $\mathit{A}$ as $[{\mathit{A}}_1,{\mathit{A}}_2,\dots ,{\mathit{A}}_L],{\mathit{A}}_l\in {\mathit{R}}^{N\times p_l}$. According to the above definitions, $h\left(\mathit{x}\right)={\sum }_{l=1}^L{h}_l\left({\mathit{x}}_l\right)$. Then, a convex optimization problem can be given by

$$minf(\sum _{l=1}^L{\mathit{z}}_l-\mathit{b})+\sum _{l=1}^L{h}_l({\mathit{x}}_l)$$

(18)

$${\mathit{A}}_l{\mathit{x}}_l-{\mathit{z}}_l=0$$

(19)

where $\mathit{Z}=[{\mathit{z}}_1,{\mathit{z}}_2,\dots ,{\mathit{z}}_L],{\mathit{z}}_l\in {\mathit{R}}^N$.

Its augmented Lagrangian function is given by

$$\begin{array}{c}{L}_{\rho }(\mathit{x},\mathit{Z},\Lambda )=f\left({\displaystyle \sum _{l=1}^L}{\mathit{z}}_l-\mathit{b}\right)+{\displaystyle \sum _{l=1}^L}{h}_l\left({\mathit{x}}_l\right)+{\displaystyle \sum _{l=1}^L}{\lambda }_l^T({\mathit{A}}_l{\mathit{x}}_l-{\mathit{z}}_l)+\frac{\rho }{2}{\displaystyle \sum _{l=1}^L}{∥{\mathit{A}}_l{\mathit{x}}_l-{\mathit{z}}_l∥}_2^2\hfill \end{array}$$

(20)

where $\rho >0$ represents the penalty factor, and $\Lambda =[{\lambda }_1,{\lambda }_2,\dots ,{\lambda }_L]\in {\mathit{R}}^{N\times L}$ denotes the dual variable.

The key technology that affects the performance of ADMM lies in the division of subproblems and the selection of hyperparameters. Built upon the maximally split technique, a method for basic vector and matrix operations that ensures that each partition subproblem only contains one scalar component to take advantage of specific computer architectures. Consider the L partition ADMM with $L=P$; the optimization problem (18) is therefore maximally split into P subproblems. In this case, the matrix ${\mathit{A}}_l$ is reduced to a vector ${\mathit{a}}_l$, and vector ${\mathit{x}}_l$ is reduced to a scalar $x_l$. To also account for past iterates when computing the next ones, an alternative relaxation technique is used, in which ${\mathit{z}}_l$ is replaced by $(1-\alpha ){\mathit{a}}_l{x}_l^{k+1}+\alpha {\mathit{z}}_l$. Then, the corresponding iterative solution process turns into

$$\begin{array}{cc}\hfill & x_l^{k+1}=\underset{x_l}{argmin}\left\{h_l\left(x_l\right)+\frac{\rho }{2}{∥{\mathit{a}}_l{x}_l-{\mathit{z}}_l^k+\frac{{\lambda }_l^k}{\alpha \rho })∥}_2^2\right\}\hfill \\ \hfill & {\mathit{Z}}^{k+1}=\underset{Z}{argmin}\left\{f\left(\sum _{l=1}^L{\mathit{z}}_l-\mathit{b}\right)+\frac{\rho {\alpha }^2}{2}\sum _{l=1}^L{∥{\mathit{a}}_l{x}_l^{k+1}-{\mathit{z}}_l+\frac{{\lambda }_l^k}{\alpha \rho }∥}_2^2\right\}\hfill \\ \hfill & {\lambda }_l^{k+1}={\lambda }_l^k+\alpha \rho ({\mathit{a}}_l{x}_l^{k+1}-{\mathit{z}}_l^{k+1})\hfill \end{array}$$

(21)

where $\alpha >0$ is the relaxation parameter.

In view of the fact that the number of iterations required for ADMM convergence depends on the penalty factor, we try to automatically adjust the value of the key parameters in each iterative process to improve the convergence speed. This can be achieved by balancing the original residual and dual residual to measure the algorithm convergence.

$$\begin{array}{c}{\mathit{r}}_k=\mathit{A}{\mathit{x}}_k-{\mathit{z}}_k\hfill \\ {\theta }_k=\rho {\mathit{A}}^T({\mathit{z}}_k-{\mathit{z}}_{k-1})\hfill \end{array}$$

(22)

where ${\mathit{r}}_k$ and ${\mathit{d}}_k$ are primal residual and dual residual, respectively. According to [36], the iteration is generally stopped when

$$\begin{array}{cc}\hfill & {∥{\mathit{r}}_k∥}^2\le {\epsilon }^{tol}max\left\{{∥\mathit{A}{\mathit{x}}_k∥}^2,{∥{\mathit{z}}_k∥}^2\right\},\hfill \\ \hfill & {∥{\mathit{d}}_k∥}^2\le {\epsilon }^{tol}max\left\{{∥{\mathit{A}}^T{\lambda }_k∥}^2\right\}\hfill \end{array}$$

(23)

where ${\epsilon }^{tol}$ denotes the termination tolerance. It can be observed that the higher penalty term results in smaller primal residuals but larger dual ones, while a smaller penalty term yields larger primal and smaller dual residuals. In order to maintain small residuals at convergence, the penalty parameter $\rho$ should be adaptively tuned to balance the residuals.

Following [37] the Barzilai–Borwein(BB) and Douglas–Rachdord splitting (DRS) method, the optimal $\rho$ that minimizes the residual is given by

$${\rho }_k=arg\underset{\rho }{min}\frac{1+{\theta }_k{\sigma }_k{\rho }^2}{({\theta }_k+{\sigma }_k)\rho }=\frac{1}{\sqrt{{\theta }_k{\sigma }_k}}$$

(24)

where ${\theta }_k>0$ and ${\sigma }_k>0$ are the spectral gradient descent stepsizes for the dual variable of problem (18) at iteration k. Under this choice of ${\rho }_k$, we then have the optimal relaxation parameter ${\alpha }_k$

$${\alpha }_k=1+\frac{1+{\theta }_k{\sigma }_k{{\rho }_k}^2}{({\theta }_k+{\sigma }_k){\rho }_k}\le 2$$

(25)

The model parameters ${\theta }_k$ and ${\sigma }_k$ can be estimated based on the results from iteration k and an older iteration $k_0<k$. Let $SD$ stand for the steepest descent and $MG$ for the minimum gradient, and then

$${\widehat{\theta }}_k=\left\{\begin{array}{c}{\widehat{\theta }}_k^{MG},2{\widehat{\theta }}_k^{MG}>{\widehat{\theta }}_k^{SD}\\ \\ {\widehat{\theta }}_k^{SD}-{\widehat{\theta }}_k^{MG}/2,\mathrm{otherwise}\end{array}\right.$$

(26)

where $\widehat{\theta }=1/\theta$. ${\sigma }_k$ can be obtained in a similar way. For a detailed account of these formulas, see [13].

The specific steps of the proposed LCC-AADMM are shown in Algorithm 1.

Algorithm 1: LCC-AADMM

Data:  optimization variable ${\mathit{x}}_k$, batch step size ${\gamma }_k$, interval length b, initial Hessian approximation matrix ${\mathit{B}}_{k+1}^{-1}$, storage length $M_k$, number of iterations k, constant ${\gamma }_0$,${\tau }_0$  Result:  the optimal variable $\mathit{x}$ 1 Initialization; 2 Randomly select samples with batch size b and calculate the stochastic gradient $g_k$ of the objective function; 3 Update the target variable ${\mathit{x}}_k$ by Equation (14); 4 Calculate the distance estimation ${\mathit{s}}_k$ and gradient estimation ${\mathit{y}}_k$ of the objective function by Formulas (11) and (12); 5 Calculate the dual variable ${\lambda }_k$ and auxiliary variable ${\mathit{z}}_k$ by Formula (21); 6 Judging whether the iteration termination condition (23) is satisfied; if it is satisfied, terminate the iteration, or calculate and update the BFGS approximate matrix by Formula (9), and return to iterative step 1, and the number of iterations is increased by 1;

4.2. RELM Models with Stochastic Optimization Constraints

For an M-category classification problem, assuming that the training sample is $\mathit{X}=[{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_S]\in R^{L\times S}$ and the number of hidden layer nodes is I, the model output $f_m\left({\mathit{x}}_s\right)$ is given by

$$\begin{array}{c}{f}_m\left({\mathit{x}}_s\right)={\displaystyle \sum _{i=1}^I}F({{\mathit{w}}_i}^T{\mathit{x}}_s+{\tau }_i){\beta }_{im}\hfill \end{array}$$

(27)

where ${\mathit{w}}_i\in R^L$ is the weight vector, and ${\tau }_i$ is the bias of i-th node. Final output can be obtained with the combination of the activation function $F(·)$ and the weight matrix $\mathit{B}\in R^{I\times M}$.

To improve the generalization performance and stability, a regularization theory is introduced to simplify the ELM model. If we define the target output as $\mathit{T}\in R^{S\times M}$ and the hidden-layer output as $\mathit{N}\in R^{S\times I}$ with $n_{si}=F({{\mathit{w}}_i}^T{\mathit{x}}_s+{\tau }_i)$, then the least squares problem is turned into

$$\underset{\mathit{B}}{min}\frac{1}{2}{∥\mathit{N}\mathit{B}-\mathit{T}∥}_F^2+\frac{1}{2}{\mu }^2{∥\mathit{B}∥}_F^2$$

(28)

where ${∥·∥}_F$ indicates the Frobenius norm, and $\mu >0$ represents the regulation factor.

Formula (28) is the objective function, which is a convex optimization calculation model with separable structure. To simplify the objective function, the ADMM calculation framework is introduced to decompose the objective function into a set of concurrent executable subproblems. Therefore, each subproblem has a subset of the model coefficients; that is, Formula (28) is equivalent to Formula (18). Then, the RELM method (28) can be derived from LCC-AADMM.

Denote ${\mathit{v}}_m^k={[v_{1m}^k,v_{1m}^k,\dots ,v_{(S+I)m}^k]}^T$, ${\mathit{z}}_m^k={[z_{1m}^k,z_{1m}^k,\dots ,z_{(S+I)m}^k]}^T$ and ${\lambda }_m^k=$$[{\lambda }_{1m}^k,$${\lambda }_{1m}^k,$$\dots ,{\lambda }_{(S+I),m}^k{]}^T$. Then, the RELM is expressed as

$$\begin{array}{cc}\hfill & {\beta }_{im}^{k+1}={\beta }_{im}^k-\widehat{\alpha }{{\widehat{\mathit{n}}}_i}^{-1}({\mathit{v}}_m^k+{\lambda }_m^k-{\mathit{z}}_m^k)\hfill \\ \hfill & v_{jm}^{k+1}={\widehat{\mathit{h}}}_j^T{\mathit{\beta }}_m^{k+1}\hfill \\ \hfill & z_{jm}^{k+1}=\frac{1}{1+\left(I^{-1}{\alpha }^2\rho \right)}{\widehat{t}}_{jm}+\frac{I^{-1}{\alpha }^2\rho }{1+\left(I^{-1}{\alpha }^2\rho \right)}(v_{jm}^{k+1}+\frac{{\lambda }_{jm}^k}{\alpha \rho })\hfill \\ \hfill & {\lambda }_{jm}^{k+1}={\lambda }_{jm}^k+\alpha \rho (v_{jm}^{k+1}-z_{jm}^{k+1})\hfill \end{array}$$

(29)

where $\widehat{\alpha }=I^{-1}\alpha$, $i=1,2,\dots ,I$ and $j=1,2,\dots ,S+I$. $\widehat{\mathit{N}}=I^{-1}{[{\mathit{N}}^T,\mu {\mathit{I}}_I]}^T$. Then, $\widehat{\mathit{N}}=$${[{\widehat{\mathit{h}}}_1,{\widehat{\mathit{h}}}_2,\dots ,{\widehat{\mathit{h}}}_{S+I}]}^T$ = $[{\widehat{\mathit{n}}}_1,{\widehat{\mathit{n}}}_2,\dots ,{\widehat{\mathit{n}}}_I]$. $\widehat{\mathit{T}}=I^{-1}{[{\mathit{T}}^T,{\mathit{0}}_{M\times I}]}^T$ with $\widehat{\mathit{T}}=[{\widehat{\mathit{t}}}_1,{\widehat{\mathit{t}}}_2,\dots ,{\widehat{\mathit{t}}}_M]$ and ${\widehat{\mathit{t}}}_m=$${[{\widehat{t}}_{1m},{\widehat{t}}_{2m},\dots ,{\widehat{t}}_{(S+I)m}]}^T$.

By reconstructing the RELM method, all variable updates can be decomposed into scalar variable updates, which has a highly parallel structure. This is the desired structure for optimization applications with high computational cost.

5. Simulation Experiment and Result Analysis

As with the R-SDL-BFGS method discussed so far, it can not only be well applied to unconstrained optimization problems but also solve convex optimization problems. In unconstrained optimization problems, the ability to find the bounds of the optimal value and the stochastic robust approximation make the proposed method superior to other quasi-Newton algorithms. To verify this claim, simulations are carried out on four benchmark functions (Branin Function, Levy Function N.13, Matyas Function, and Three-Hump Camel Function) to compare the performance of R-SDL-BFGS, SD-BFGS, LBFGS, andBFGS methods. Experiments are conducted using MATLAB 2019(The MathWorks, Inc., Natick, MA, US) on a desktop with an Intel Core i7-10700 8-core CPU and 16GB of RAM. At the same time, in order to ensure that the performance of the algorithm is not accidental, it is necessary to take the average value of multiple experiments to ensure whether the algorithm has good convergence performance in each experiment, so as to ensure that the algorithm has good robustness. The specific description is shown in

Table 2. Benchmark function description.

Function	Function Optimal Value	Optimal Solution
Branin Function	0.389	[3.1416, 2.2750]
Levy Function N.13	0	[0, 0]
Matyas Function	0	[0, 0]
Three-Hump Camel Function	0	[0, 0]

.

For convex optimization problems, the R-SDL-BFGS method can simplify the solution process of subproblems with the Hessian approximation matrix, thereby reducing the ADMM computational cost and improving the speed of convergence. To verify this claim, simulations are carried out on eight benchmark datasets to compare the performance of R-SDL-BFGS with LCC-AADMM, MS-AADMM and RB-ADMM algorithms. Here, the benchmark datasets include Gisette, USPS, Magic, BASEHOCK, Pendigits, Optical-Digits, statlog, and PCMAC. The characteristics of the eight datasets are shown in

Table 3. Datasets specifications.

Dataset	Attributes	Training Samples	Testing Samples	Classes
Gisette	5000	5600	1400	2
USPS	256	7439	1859	10
Magic	10	15,216	3804	2
Pendigits	16	8794	2198	10
Optical-Digits	64	4496	1124	10
statlog	36	5148	1287	6
PCMAC	3289	1555	388	2
BASEHOCK	4862	1595	438	2

. The first six sets of data in the table are from the UCI machine learning library, and the last two are from the ASU feature selection dataset.

5.1. Comparative Analysis of Convergence Performance of Quasi-Newton Algorithms

This section reports the the convergence speed of different algorithms, based on the number of iterations required before $f\left(x_{k+1}\right)-f\left({\mathit{x}}_k\right)\le Q$, where Q is the error stop station. In general, the quasi-Newton method needs simple line search procedures to satisfy the termination condition. This property leads to a low computational cost during the training phase. Therefore, it is convenient to use the iteration step to evaluate the effectiveness of the method.

As shown in

Table 4. Comparison of Convergence Speeds of Different Quasi-Newton Methods.

Function	BFGS	SD-BFGS	R-SDL-BFGS
Branin Function	15	12	6
Levy Function N.13	5	5	1
Matyas Function	20	1	1
Three-Hump Camel Function	8	5	3

and

Table 5. Comparison of Convergence Speeds of BFGS based Methods.

CEC Benchmark Function	Ratio (BFGS)	Ratio (SD-BFGS)
Branin Function	88.0000	81.2500
Levy Function N.13	95.2380	88.8888
Matyas Function	96.6666	66.6666
Three-Hump Camel Function	75.0000	40.0000

, the standard BFGS algorithm avoids the problem of singular matrices by replacing inverse matrices with Hessian approximation matrices. However, this algorithm needs to calculate and store the matrix in each iteration, which leads to an expensive computational cost as well as a slow convergence.

To ensure the positive definiteness, the SD-BFGS algorithm is devised by making use of damping technology so as to maintain the positive definiteness of the BFGS matrix in non-convex optimization. Since the global convergence of the algorithm depends on the monotonicity of the function, a suitable numerical solution of non-monotonic equations is in general not feasible.

In order to attain asymptotic linear convergence, a non-monotonic Wolfe-type strategy can be applied to the memory gradient method (R-SDL-BFGS). By combining the current function iteration information and the function information of multiple points in the past, it overcomes the oscillation phenomenon and improves the global convergence of the algorithm.

From a theoretical point of view, the R-SDL-BFGS algorithm has a better global convergence and convergence speed compared with the BFGS, SD-BFGS, and R-SDL-BFGS methods. Experiments on benchmark function are presented in Table 4, which shows the speed of convergence can be quite acceptable. As can be seen from Table 5, the convergence rate of R-SDL-BFGS is $88.7260\%$ higher than that of BFGS in the CEC benchmark function. The convergence rate of R-SDL-BFGS is $69.20135\%$ higher than that of SD-BFGS in the CEC benchmark function. According to the numerical experiment results, it can be seen that the performance of the R-SDL-BFGS is completely consistent with the theoretical results and achieves good classification performance in practice. The results of experiments in Figure 1, Figure 2, Figure 3 and Figure 4 and Table 5 demonstrate that the R-SDL-BFGS algorithm converges faster than other QNMs.

5.2. Convergence Performance Comparative Analysis

The complexity of the iterative algorithm is determined by the complexity of the unit computation and the number of iterations, where the unit computation complexity refers to the number of floating-point operations required by the optimization algorithm for single iteration, and the complexity of the iteration times refers to the number of iterations required to calculate the solution with a given precision. However, since the unit computation complexity in the iterative process is almost the same, the convergence performance of the algorithm is evaluated by comparing the complexity of iterations.

In this section, the convergence performance of different ADMM algorithms under the same error condition are proved, and the iterative termination condition (27) is the key to evaluating the convergence speed of the algorithm. Under the same iteration termination condition, LCC-AADMM, MS-AADMM, and RB-ADMM methods are evaluated on eight benchmark datasets. At the same time, in order to ensure that the performance of the algorithm is not accidental, it is necessary to take the average value of multiple experiments to ensure whether the algorithm has good convergence performance in each experiment so as to ensure that the algorithm has good robustness. Additionally, these algorithms for computing and approximating the matrix are analyzed by comparing the iteration number.

From a theoretical point of view, for the M-category classification problem, we can rewrite (28) as

$$\begin{array}{cc}\hfill & \underset{\mathit{\beta }}{min}{∥\mathit{N}\mathit{\beta }-\mathit{t}∥}_F^2\hfill \\ \hfill & \mathit{\beta }=\mathrm{vec}\left(\mathit{B}\right)\in R^{IM\times 1}\hfill \\ \hfill & \mathit{N}={\mathit{I}}_M\otimes {[{\mathit{N}}^T,\mu {\mathit{I}}_I^T]}^T\in R^{(S+I)M\times IM}\hfill \\ \hfill & \mathit{t}=\mathrm{vec}\left({[{\mathit{T}}^T,{\mathbf{0}}_{I\times M}^T]}^T\right)\in R^{(S+I)M\times 1}\hfill \end{array}$$

(30)

where ⊗ is the Kronecker product, and $\mathrm{vec}(·)$ means the concatenation of all columns of the matrix. Then, the MS-ADMM (31) and RB-AADMM (32) are as follows:

$$\begin{array}{cc}\hfill & {\beta }_{im}^{k+1}=\underset{{\beta }_{im}}{argmin}\left\{g_{im}\left({\beta }_{im}\right)+\frac{\rho }{2}{∥{\mathit{n}}_{im}{\beta }_{im}-{\mathit{z}}_{im}^k+\frac{{\lambda }_{im}^k}{\rho }∥}_2^2\right\}\hfill \\ \hfill & {\mathit{Z}}^{k+1}=\underset{\mathit{Z}}{argmin}\left\{\sum _{im=1}^{IM}{\mathit{z}}_{im}-\mathit{t}+\frac{\rho }{2}\sum _{im=1}^{IM}{∥{\mathit{n}}_{im}{\beta }_{im}^k-{\mathit{z}}_i+\frac{{\lambda }_{im}^k}{\rho }∥}_2^2\right\}\hfill \\ \hfill & {\lambda }_{im}^{k+1}={\lambda }_{im}^k+\rho ({\mathit{n}}_{im}{\beta }_{im}^{k+1}-{\mathit{z}}_{im}^{k+1})\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & {\beta }_{im}^{k+1}=\underset{{\beta }_{im}}{argmin}\left\{\frac{\rho }{2}{∥{\mathit{n}}_{im}({\beta }_{im}-{\beta }_{im}^k)-\alpha ({\mathit{z}}_{im}^k-{\mathit{n}}_{im}{\beta }_{im}^k-\frac{{\lambda }_{im}^k}{\rho \alpha })∥}_2^2\right\}\hfill \\ \hfill & {\mathit{Z}}^{k+1}=\underset{\mathit{Z}}{argmin}\left\{\sum _{im=1}^{IM}{\mathit{z}}_{im}-\mathit{t}+\frac{\rho {\alpha }^2}{2}\sum _{im=1}^{IM}{∥{\mathit{z}}_{im}-{\mathit{n}}_{im}{\beta }_{im}^{k+1}-\frac{{\lambda }_{im}^k}{\rho \alpha }∥}_2^2\right\}\hfill \\ \hfill & {\lambda }_{im}^{k+1}={\lambda }_{im}^k+\rho \alpha ({\mathit{n}}_{im}{\beta }_{im}^{k+1}-{\mathit{z}}_{im}^{k+1})\hfill \end{array}$$

(32)

where $im=1,2,\dots ,IM$ and ${\mathit{z}}_{im}={\mathit{n}}_{im}{\beta }_{im}$.

5.2.1. Convergence Performance Analysis Compared with RB-ADMM

Combining (29) and (31), it can be found that most of iterative steps in LCC-AADMM are linear, which reduces the computational cost and improves the speed of convergence. From a theoretical point of view, the choice of the penalty factor is of practical importance in improving the overall performance of the model. Although RB-ADMM can auto-adjust the penalty factor by balancing relationship between the dual residuals and the primitive residuals, it can be seen that the computational cost of RB-ADMM varies greatly with the size of the problem. At the same time, if no proper penalty factor is chosen, then the algorithm may not converge.

The parameter selection scheme can provide fast and accurate estimates of the optimal parameters of the algorithm. To improve the convergence performance, LCC-AADMM use step-size selection constraints to construct the adaptive parameter selection scheme, where the step size is chosen to satisfy the Wolfe conditions. Also, instead of computing Hessian approximation afresh at every iteration, LCC-AADMM updates it in a simple manner to account for the curvature measured during the most recent step. This makes LCC-AADMM converge more rapidly than RB-ADMM.

The simulation results of the objective function are given in

Table 6. Comparison of Convergence Analysis Results.

Dataset	RB-AADMM		MS-AADMM		LCC-AADMM
	Time	Iteration	Time	Iteration	Time	Iteration
Gisette	737.5244	912	33.4513	39	5.0.96	13
USPS	2621.8914	546	152.5741	31	39.1072	13
BASEHOCK	332.8139	1674	9.0221	41	2.6267	19
Magic	1156.6178	674	46.0599	25	19.2866	14
Pendigits	3912.3798	742	176.3753	33	46.7936	14
Optical-Digits	1825.9015	538	139.2235	41	18.4858	9
statlog	1626.4322	704	92.0177	39	16.8493	13
PCMAC	353.2414	1753	3.3102	15	1.4095	13

, showing that R-SDL-BFGS has the desired effect of reducing ADMM computational costs. For comparison, the improvement rate of different algorithms is shown in

Table 7. Convergence Speed Improvement Ratio.

Dataset	Classes	RB-ADMM	MS-AADMM
Gisette	2	97.9808	31.5789
Magic	2	94.6984	55.1724
BASEHOCK	2	98.3636	29.6296
PCMAC	2	95.4008	54.2857
statlog	6	96.8750	39.1304
USPS	10	94.6494	68.9655
Pendigits	10	95.0425	62.5871
Optical-Digits	10	98.2264	58.0645

. As can be seen from Table 7, the convergence rate of LCC-ADMM is $96.6109\%$ higher than that of RB-ADMM in the two types of classified datasets. In the 6-classification datasets, the convergence speed is improved by $96.8750\%$ on average. In the 10-classification datasets, the convergence rate is improved by $95.9727\%$ on average.

5.2.2. Convergence Performance Analysis Compared with MS-AADMM

One powerful approach to obtaine the optimal output weights starts from an appropriate parameter selection scheme, which allows us to use an adjustable step size to speed up algorithm convergence. For the actual numerical performance of ADMM, the subproblem solving process is the key to determining the performance of the algorithm. However, MS-AADMM has ignored this key factor.

It can be known from (31) and (32) that LCC-AADMM converts the exact solution into an approximate solution by performing inexact optimization with the help of the Hessian approximation matrix, which greatly reduces the computational cost and thereby improves the speed of convergence. Theoretically, it can be obviously found that the convergence performance of LCC-AADMM is better than MS-AADMM.

A comparison of the convergence performance of different methods is shown in Table 6. It can be seen that the R-SDL-BFGS algorithm obviously performs the best in terms of classification performance. And Table 7 shows that the R-SDL-BFGS method has a certain improvement over MS-AADMM in the efficiency of classification.

As can be seen from Table 7, the convergence rate of LCC-ADMM is $42.6666\%$ higher than that of RB-ADMM in the two types of classified datasets. In the 6-classification datasets, the convergence speed is improved by $39.1304\%$ on average. In the 10-classification datasets, the convergence rate is improved by $63.2057\%$ on average.

5.2.3. Overall Convergence Performance Analysis

The LCC-AADMM method divides the convex optimization problem of RELM into univariate subproblems that can be executed in parallel by using the maximum partitioning technique, which reduces the computational complexity in iterative updates. By introducing the R-SDL-BFGS algorithm, AADMM achieves inexact optimization with the help of the Hessian approximation matrix, which reduces the computational cost while maintaining a fast convergence speed.

Theoretically, LCC-AADMM usually has better convergence performance than other algorithms in solving classification problems. It can be seen from Table 6 that the LCC-AADMM algorithm has the fastest convergence speed. The difference in performance of these methods is also evident as the size of the error is varied. Under the same conditions, the LCC-AADMM algorithm always has the best convergence according to Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12.

5.3. Accuracy Analysis for LCC-AADMM

Classification accuracy is one of the important indicators to evaluate the performance of classification models and assesses the quality of the model. The fatal flaw of RELM is that it cannot be applied to large-scale distributed optimization problems owing to its high computational cost. In view of the above shortcomings, LCC-AADMM is adopted to decompose the convex optimization problem into a set of subproblems that can be executed in parallel, thereby achieving efficient classification performance. We performed experiments on the eight benchmark classification datasets in Table 2 using the MI-based, MS-AADMM, and proposed methods. The accuracy of the test results is listed in Figure 13.

Figure 13 shows the performance of different methods under the big data classification task. It is obvious that the LCC-AADMM methods consistently outperform the two competitive ELM algorithms on all eight datasets. The best overall performance is provided by the the proposed LCC-AADMMs as shown in Figure 12. In addition, the LCC-AADMM algorithm shows significant improvement over the best results obtained by the other two competitive ADMM methods, showing good performance in terms of classification accuracy and suitability for applications that require superior accuracy.

It can be concluded that the proposed method performs well on a wide variety of problems and does not require excessive computer time or storage compared with MI-based and MS-AADMM methods. In practice, this technique can be expected to provide good learning ability and satisfactory generalization performance.

6. Conclusions

In this paper, we consider implementing distributed learning through the effective solution of subproblems. That is, the regularized LS problem in the RELM is split into a set of optimization subproblems. To achieve high computational efficiency in solving subproblems, an efficient LCC-AADMM based on the R-LBFGS algorithm is proposed. The novelty of this method mainly lies in three aspects: (1) A SL-BFGS method is devised, which uses a limited amount of storage and updates the quasi-Newton matrix continuously; (2) random damping technology is proposed, which adopts a new strategy for determining the step size at each iteration and guarantees the positive definiteness of the BFGS matrix to achieve a learning ability with high quality; (3) based on the residual balancing scheme, an adaptive penalty factor selection strategy is applied to balance the relationship between the distance from convergence and the residuals and obtain good convergence.

The impact of this issue is demonstrated on eight benchmark dataset example problems. These experiments show that the proposed method achieves good performance in certain cases and converges faster than other ADMM methods. The high parallelism of LCC-AADMM is further demonstrated by comparison with an MI-based method. Therefore, the LCC-AADMM method offers a complementary alternative to optimization problems in large-scale applications.