Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Abstract

For large-scale optimization that covers a wide range of optimization problems encountered frequently in machine learning and deep neural networks, stochastic optimization has become one of the most used methods thanks to its low computational complexity. In machine learning and deep learning problems, nonconvex problems are common, while convex problems are rare. How to find the global minimum for nonconvex optimization and reduce the computational complexity are challenges. Inspired by the phenomenon that the stagewise stepsize tuning strategy can empirically improve the convergence speed in deep neural networks, we incorporate the stagewise stepsize tuning strategy into the iterative framework of Nesterov’s acceleration- and variance reduction-based methods to reduce the computational complexity, i.e., the stagewise stepsize tuning strategy is incorporated into randomized stochastic accelerated gradient and stochastic variance-reduced gradient. The proposed methods are theoretically derived to reduce the complexity of the nonconvex and convex problems and improve the convergence rate of the frameworks, which have the complexity $O(1/\mu ϵ)$ and $O(1/\mu \sqrt{ϵ})$, respectively, where $\mu$ is the PL modulus and L is the Lipschitz constant. In the end, numerical experiments on large benchmark datasets validate well the competitiveness of the proposed methods.

1. Introduction

In this thesis, we consider the following empirical risk minimization problem:

$$\underset{x\in {\mathbb{R}}^d}{min}F\left(x\right):=\frac{1}{n}\sum _{i=1}^n{f}_i\left(x\right)$$

(1)

where x represents the model parameters and $f_i:{\mathbb{R}}^d\to \mathbb{R}$ denotes a smooth but possibly nonconvex function. In particular, $f_i\left(x\right):=\ell (x,a_i,b_i)$ often denotes the loss function on given training sample $\{(a_i,b_i)\in {\mathbb{R}}^{d\times 1}\}$. Therefore, in problem (1), for example, when $\ell (x,a_i,b_i)=log(1+exp(-b_i{x}^T{a}_i))$, (1) reduces to the logistic regression [1]; or if we let $\ell (x,a_i,b_i)={({\sigma }_l(w_l^T\dots {\sigma }_1\left(w_1^T{a}_i\right))-b_i)}^2$ where $x:=[w_1,\dots ,w_l]$ and ${\sigma }_s$, $s=1,\dots ,l$ denote activation functions, we obtain the training model of DNNs [2].

To solve problem (1), one of the standard methods is the gradient descent (GD) that carries out the following updates:

$$x_{k+1}=x_k-\frac{{\eta }_k}{n}\sum _{i=1}^n\nabla f_i\left(x_k\right)$$

where ${\eta }_k$ is the stepsize. Since the above recursion needs to evaluate derivatives n times at each iteration, it is impracticable for problems in larger scales. To break this bottleneck, there has been a growing interest on stochastic methods to reduce the computational cost, among which the method of stochastic gradient descent (SGD) [3,4,5] is a typical one. In particular, the SGD reads as

$$x_{k+1}=x_k-{\eta }_k\nabla f_{i_k}\left(x_k\right),$$

where $i_k$ is an i.i.d. random variable taking value in $\{1,2,\dots ,n\}$ uniformly. Obviously, the computational complexity of SGD is independent with the size n. Thereupon, SGD becomes one of the most popular first-order methods to solve large-scale optimization problems [6,7,8]. However, because of the random interference caused by the stochastic gradient, SGD can only tolerate a relatively small stepsize that makes the convergence rate of SGD slower than its non-stochastic counterpart (i.e., GD). Consequently, it has become a hot topic to develop accelerated algorithms.

1.1. Related Works

The accelerated methods for optimization can be traced back to the 1960s. Inspired by the motion of a heavy ball in a potential field, Polyak [9] sped up the convergence of GD numerically, but without a rigorous analysis. In [10], Nesterov devised a different iterative framework and obtained a rigorous result in that, by running the Nesterov algorithm with the complexity at most $O(1/\sqrt{ϵ})$ iterations, one can find an $ϵ$-minimum, i.e., a point $\mathring{x}$ such that $F\left(\mathring{x}\right)-F\left(x^{\ast }\right)\le ϵ$ where $x^{\ast }$ denotes the minimizer of objective F over ${\mathbb{R}}^d$. Henceforth, various variations of Nesterov’s acceleration-based methods emerged, see [11,12,13,14,15] and references therein. However, please note that the convexity has played a pivotal role in establishing the convergence of the above methods.

In this paper, we mainly focus on acceleration playing a role in stochastic nonconvex optimization. Different from convex optimization, it is impractical to find a global minimum for nonconvex optimization in general. Consequently, one aims to seek a weaker guarantee, i.e., an $ϵ$-stationary point, that is, a point $\mathring{x}$ with a sufficiently small gradient $\Vert \nabla F\left(\mathring{x}\right)\Vert \le ϵ$ or $\mathbb{E}[\Vert \nabla F\left(\mathring{x}\right)\Vert ]\le ϵ$, to surrogate the local optimal. When the objective is assumed to meet Lipschitz continuously differentiable (l.c.d.) with constant L, Ghadimi and Lan [16] showed that SGD needs to be run $O(L^2/ϵ+L\sigma /{ϵ}^2)$ times to find an $ϵ$-stationary point. To improve convergence rate, naturally, a number of researchers [17,18] applied Nesterov’s acceleration to the nonconvex case, but no theoretical guarantees for faster rate were given. Recently, Ghadimi and Lan [19] devised a new elaborate iterative framework, termed the randomized stochastic accelerated gradient (RSAG) method, and showed that the iterative complexity can be reduced to $O(L/ϵ+L\sigma /{ϵ}^2)$ to find an $ϵ$-stationary point, where $\sigma$ denotes the upper bound of the standard deviation of the stochastic gradient.

The reason why SGD converges slowly is that, to avoid divergence caused the random interference, only a relatively small stepsize can be tolerated. Then, reducing the random interference in stochastic gradient becomes another method for acceleration. For this purpose, Johnson and Zhang [20] proposed an easy and feasible approach named VR; that is, after dividing the total iterations into a number of epochs firstly, once the iteration is performed through an epoch, the full gradient is calculated once to avoid the stochastic gradient deviating too far. Considering strongly convex objectives, Johnson and Zhang followed the ideal of VR to propose the stochastic variance reduced gradient (SVRG), for which a relatively large stepsize becomes acceptable. In particular, Johnson and Zhang showed that SVRG can achieve a linear convergence rate when the stepsize is chosen appropriately. However, in modern learning problems, the strong convexity is often unsatisfied. For this, Reddi et al. [21] extended the SVRG to handle nonconvex objectives and showed that SVRG can achieve iterative complexity $O(n^{2/3}L/ϵ)$ to find an $ϵ$-stationary point. Shang et al. [22] proposed a simple stochastic variance reduction method for machine learning, termed as VR-SGD.

As discussed earlier, in SGD, a large stepsize may amplify the random interference to cause the iterates to diverge. On the other hand, a too-small stepsize may make it difficult for the iterates to escape saddle points. In the process of solving DNN and other nonconvex models [23,24,25,26], one empirically finds a phenomenon that the optimization algorithms equipped with SSTS, which start from a relatively large stepsize and decrease it geometrically after a number of iterations, can improve the convergence speed effectively. In terms of theoretical analysis, Xu et al. [27] showed that the convergence rate of stagewise SGD (i.e., SGD equipped with SSTS), abbreviated as S-SGD, for convex objectives can be significantly improved with different degrees under different local growth conditions. However, the convexity plays an important role in deriving the above result, which is not met for modern learning problems in general, such as training DNNs. Recently, the Polyak–Łojasiewicz (PL) condition has been observed and proved for training DNNs [28,29,30,31,32]. For the vanilla SGD under PL condition, Arjevani et al. [33] studied the lower bounds of ${ϵ}^{-3}$ to find an $ϵ$ -stationary point by using stochastic first-order methods in the certain condition. Horváath et al. [34] showed the iterative complexity to find an $ϵ$-approximate solution that can fall in between $O(1/\left({\mu }^2{ϵ}^2\right))$ and $O(1/(\mu ϵ\left)\right)$ where $\mu$ denotes the PL modulus. In learning DNNs, $\mu \ll 1$ generally, which dampens its performance severely. Wang et al. [35] proposed the momentum stochastic method to achieve an $ϵ$-stationary solution under a constant step size with $O(1/\left({ϵ}^2\right))$ computation complexity. Yuan et al. [36] considered SSTS work for the nonconvex objective meeting PL condition, and then answered the question in the affirmative in [37]. Particularly, they showed that the iterative complexity of SGD for nonconvex objective meeting PL condition can be reduced to $O(L/\mu ϵ)$ (which is smaller than $O(1/\left({\mu }^2ϵ\right))$ significantly due to $\mu \ll 1$) when SSTS is adopted.

In this paper, we mainly consider whether we can further improve the convergence rate or reduce iterative complexity by incorporating the SSTS into the iterative framework of Nesterov’s acceleration based methods and VR based methods. We will answer this question in the affirmative. Specifically, we will develop SSTS-equipped RSAG and SVRG algorithms respectively and give their corresponding their theoretical analysis.

1.2. Contributions

In this paper, we mainly develop and analyze two accelerated algorithms, namely, SSTS equipped RSAG and SVRG. Specifically, the main contributions of the paper are summarized as follows:

• Incorporating the SSTS into the iterative framework of RSAG, we propose the stagewise RSAG, abbreviated as S-RSAG, and show that the iterative complexities of it are at most $O(L/\mu ϵ)$ to find an $ϵ$-stationary point for nonconvex objective and $O(1/\mu ϵ)$ to find an $ϵ$-minimum for convex objective, which are significantly reduced with respect to its non-stagewise counterpart RSAG, the complexities of which are $O(L^2/ϵ+L/{ϵ}^2)$ and $O(L/\sqrt{ϵ}+1/{ϵ}^2)$ respectively, where $\mu$ is the PL modulus and L is the Lipschitz constant. Compared to existing stagewise algorithm S-SGD, the complexities of our S-RSAG are more superior under the convex condition (i.e., $O(1/\mu ϵ)$ with respect to $O(L/\mu ϵ)$, where $L\gg 1$ in general) and at the same level under the nonconvex condition.
• With the same methodology, we propose the stagewise SVRG, abbreviated as S-SVRG and show that the iterative complexities of it are at most $O(Lm/\left(\mu \sqrt{ϵ}\right))$ to find an $ϵ$-stationary point for nonconvex objective and $O(Lm/\left({\mu }^2\sqrt{ϵ}\right))$ to find an $ϵ$-minimum for convex objective, where m is an arbitrary constant and denotes the number of inner iterations for VR. It is worth mentioning that the iterative complexities of S-SVRG are both significantly superior to its non-stagewise counterpart SVRG and existing stagewise algorithm S-SGD under convex and nonconvex conditions.
• We also numerically evaluate our algorithms through the experiments on a number of benchmark datasets. The obtained results are consistent with our theoretical findings.

The remainder of this paper is organized as follows. Section 2 provides some notions and preliminaries. The accelerated methods based on SSTS are proposed and analyzed in Section 3. Experiments performed on several real-world datasets are presented in Section 4. Discussion about the proposed methods are reported in Section 5. Lastly, Section 6 gives some concluding remarks. All the proofs are presented in the Appendix A.

2. Notions and Preliminaries

In this paper, we use $\Vert ·\Vert$ to denote a general norm without specific mention. Given any $X\subseteq {\mathbb{R}}^d$, we say f is Lipschitz continuously differentiable (l.c.d.) with Lipschitz constant $L>0$ over X, if $\Vert \nabla f\left(y\right)-\nabla f\left(x\right)\Vert \le L\Vert y-x\Vert$ for any $x,y\in X$. What is more, we can also verify the following inequality:

$$\begin{array}{c}\hfill |f\left(y\right)-f\left(x\right)-\langle \nabla f\left(x\right),y-x\rangle |\le \frac{L}{2}{\Vert y-x\Vert }^2.\end{array}$$

(2)

We say f is convex over X, if any intermediate value is at most the average value, i.e.,

$$f(\lambda x+(1-\lambda \left)y\right)\le \lambda f\left(x\right)+(1-\lambda )f\left(y\right)$$

for any $x,y\in X$ and $\lambda \in (0,1)$. We say f is a PL function or meets the PL condition over X with modulus $\mu >0$, if

$$2\mu (f\left(x\right)-f\left(x^{\ast }\right))\le {\Vert \nabla f\left(x\right)\Vert }^2$$

(3)

where $x^{\ast }$ is the minimizer of f over X. Note that such a function f need not be convex. However, it is also easy to show that a $\lambda$-strongly convex function is a PL function with modulus $1/2\lambda$. Furthermore, for f meeting the PL condition, it holds that

$$\Vert x-x^{\ast }{\Vert }^2\le \frac{1}{2\mu }(f\left(x\right)-f\left(x^{\ast }\right)).$$

(4)

The proof of above inequality can be found in [38] directly.

3. Stagewise Accelerated Algorithms Development

In this section, we propose to consider the SSTS playing an accelerated role in Nesterov’s acceleration-based method (RSAG) and VR-based method (SVRG). Particularly, we incorporate SSTS into the iterative framework of RSAG or SVRG to create a new way to accelerate the convergence rate further. Next, let us discuss stagewise RSAG and stagewise SVRG one by one.

3.1. Stagewise RSAG Development

Nesterov’s acceleration is devised according to Polyak’s heavy ball, the acceleration principle behind which is mainly based on the effectiveness of momentum on physics. Nesterov’s acceleration has attracted much interest due to the increasing need to solve large-scale problems. However, Nesterov’s acceleration requires explicitly convexity assumption for establishing convergence. Recently, Ghadimi and Lan [19], based on Nesterov’s acceleration, redesigned an elaborate iterative framework RSAG. On the other hand, in the process of optimizing nonconvex models such as DNNs, sparse regularization, one empirically finds out a phenomenon that the optimization algorithms equipped with SSTS, which starts from a relatively large stepsize and decreases it geometrically after a number of iterations, can improve the convergence speed effectively. SSTS makes the objective function to fastly find the space of the optimal. Thereupon, SSTS has been viewed as another simply implemented way to accelerate convergence rate.

In this section, we incorporate the SSTS into the iterative framework of RSAG which carries out the following updates (Algorithm 1): In Algorithm 1, the number of total iterations has been divided into T epochs, and the iteration is implemented $S_k$ times at the kth epoch. At the first epoch, we choose a relatively large ${\eta }_0$, and decrease it by half, i.e., ${\eta }_1={\eta }_0/2$, at the second epoch, and so on. The SSTS procedure contains the T epochs times $S_k$ epoch, which starts from a relatively large stepsize and decreases it geometrically after a number of iterations. The stepsize is decreased in half through each epoch in Algorithm 1. It is easy to verify that the convergence result also holds when the the stepsize is decayed slower somewhat, i.e., decayed by a factor $1<c<2$, ${\eta }_{k+1}={\eta }_k/c$. In particular, the recursion shown in the 6$th$ line of Algorithm 1 is the so called RSAG.

Algorithm 1 Stagewise RSAG (S-RSAG).

1: Input: $T\ge 1$, parameters ${\{S_k,{\eta }_k,{\alpha }_s\}}_{k=1}^T\ge 1$ and random variables $R_{S_k}$ taking values in $\{1,\dots ,S_k\}$; 2: Initialize: $x_0=0$; 3: for $k=0,1,\dots ,T-1$ do 4:    $x_{k,0}=x_{k,0}^{md}=x_{k,0}^{ag}=x_k$; 5:    for $s=0,1,\dots ,S_k-1$ do 6:       $$\left\{\begin{array}{cc}\hfill x_{k,s+1}^{md}=& (1-{\alpha }_s)x_{k,s}^{ag}+{\alpha }_s{x}_{k,s}\hfill \\ \hfill x_{k,s+1}=& x_{k,s}-{\lambda }_{k,s}\nabla f_{i_s}\left(x_{k,s+1}^{md}\right)\hfill \\ \hfill x_{k,s+1}^{ag}=& x_{k,s}^{md}-{\eta }_k\nabla f_{i_s}\left(x_{k,s+1}^{md}\right)\end{array}\right.$$ 7:       8:    end for 9:     $$\left\{\begin{array}{cc}\hfill x_{k+1}=& x_{k,R_{S_k}}^{ag},\mathrm{if}F\mathrm{is}\mathrm{convex}\hfill \\ \hfill x_{k+1}=& x_{k,R_{S_k}}^{md},\mathrm{elsewise}\hfill \end{array}\right.$$ 10: end for 11: Output: $x_T.$

3.2. Theoretical Aspects of S-RSAG

The goal of this section is to show the results of the convergence rate of Algorithm 1. For DNNs, the PL condition has been satisfied and proved. So we assume the objective F meets the PL condition. In particular, we give the following theorem in which the convergence rate of the Algorithm 1 is characterized.

Theorem 1.

Suppose that F is l.c.d. with constant L and meets the PL condition with modulus μ, and its gradient is bounded by G uniformly.

1.

If the parameters are chosen as ${\alpha }_s=\frac{2}{s+1}$, ${\eta }_k\le min\left\{\frac{1}{2L},\frac{{ϵ}_k}{8L{\sigma }^2}\right\}$, ${\lambda }_{k,s}\in \left[{\eta }_k,\frac{2s+3}{2(s+1)}{\eta }_k\right]$, $S_k=\frac{4}{\mu {\eta }_k}$, and probability mass function of $S_k$ is chosen such that $\mathbb{P}(R_{S_k}=s)=\frac{C_{k,s}{\lambda }_{k,s}}{{\sum }_{i=1}^{S_k}{C}_{k,i}{\lambda }_{k,i}}$ for any $s=1,\dots ,S_k$, where ${ϵ}_0\ge {\Vert \nabla F\left(x_0\right)\Vert }^2$, ${ϵ}_{k+1}={ϵ}_k/2$, and $C_{k,s}=1-L\left[{\lambda }_{k,s}+\frac{{({\lambda }_{k,s}-{\eta }_k)}^2{(s+1)}^2}{8{\lambda }_{k,s}(S_k+1)}\right]$ we can find an ϵ-stationary point (a point $\mathring{x}$ such that $\mathbb{E}[\Vert \nabla F\left(\mathring{x}\right){\Vert }^2]\le ϵ$) by performing Algorithm 1 at most $O(L/(\mu ϵ\left)\right)$ times.

2.

If we further assume F is convex and the parameters are chosen as ${\alpha }_s=\frac{2}{s+1}$, ${\eta }_k\le min\left\{\frac{1}{L},\frac{{ϵ}_k}{16{\sigma }^2}\sqrt{\frac{\mu }{3L}}\right\}$, ${\lambda }_{k,s}={\eta }_k$, $S_k=\sqrt{\frac{12}{L\mu {\eta }_k^2}}$, and probability mass function of $S_k$ is chosen such that $\mathbb{P}(R_{S_k}=s)=\frac{s(s+1)}{{\sum }_{i=1}^{S_k}i(i+1)}$ for any $s=1,\dots ,S_k$, where ${ϵ}_0\ge F\left(x_0\right)-F\left(x^{\ast }\right)$ and ${ϵ}_{k+1}={ϵ}_k/2$, we can find an ϵ-minimizer (a point $\mathring{x}$ such that $\mathbb{E}[F\left(\mathring{x}\right)-F\left(x^{\ast }\right)]\le ϵ$) by performing Algorithm 1 at most $O(1/(\mu ϵ\left)\right)$ times.

where σ denotes the upper bound of the standard deviation of the stochastic gradient.

The proof of Theorem 1 is presented in Appendix A.1.

Remark 1.

By the above Theorem 1, the iterative complexities of S-RSAG are at most $O(L/\mu ϵ)$ to find an ϵ-stationary point for nonconvex objective and $O(1/\mu ϵ)$ to find an ϵ-minimum for convex objective. For RSAG, the complexities are $O(L^2/ϵ+L/{ϵ}^2)$ and $O(L/\sqrt{ϵ}+1/{ϵ}^2)$, respectively, where μ is the PL modulus and L is the Lipschitz constant. S-RSAG reduces the complexity $O(1/{ϵ}^2)$ to $O(1/ϵ)$ under convex and nonconvex conditions. Compared to existing stagewise algorithm S-SGD, the complexities of our S-RSAG are more superior under a convex condition (i.e., $O(1/\mu ϵ)$ v.s. $O(L/\mu ϵ)$, where $L\gg 1$ in general) and at the same level under a nonconvex condition. The detail comparison is reported in

Table 1. Some recent results in the accelerated stochastic gradient methods.

Methods	PL Condition	Generally Convex	Nonconvex
SGD [16]		$O(L/ϵ+\sigma /{ϵ}^2)$	$O(L^2/ϵ+L\sigma /{ϵ}^2)$
SGD [39]	✓		$O(1/\left({\mu }^2ϵ\right))$
RSAG [19]		$O(L/\sqrt{ϵ}+\sigma /{ϵ}^2)$	$O(L^2/ϵ+L\sigma /{ϵ}^2)$
SVRG [21]			$O(n^{2/3}L/ϵ)$
S-SGD [37]	✓	$O(L/\mu ϵ)$	$O(L/\mu ϵ)$
S-RSAG	✓	$O(1/\mu ϵ)$	$O(L/\mu ϵ)$
S-SVRG	✓	$O(Lm/\left({\mu }^2\sqrt{ϵ}\right))$	$O(Lm/\left(\mu \sqrt{ϵ}\right))$

. Meanwhile, our algorithm S-RSAG does not deny the optimality of RSAG for first-order stochastic gradient methods, and further considers the PL condition.

3.3. Stagewise SVRG Development

As discussed earlier, because SGD can only tolerate a relatively small stepsize, it suffers from a slow convergence rate consequently. Apart from Nesterov’s method, another way (VR) is to implement acceleration by reducing the variance of random interference. The core ideal of the VR technique is realized by calculating a full gradient once at each epoch and incorporating it into the iteration to adjust the current stochastic gradient so that it does not deviate too far away from the full one. Benefiting from the VR technique, SVRG has been shown to possess strong competitiveness with respect to fast convergence rate.

In this section, we mainly consider whether the convergence rate can be further improved by combining the VR technique and SSTS together. In particular, we incorporate the SSTS into the iterative framework of SVRG, which carries out the following updates (Algorithm 2):

Algorithm 2 Stagewise SVRG (S-SVRG).

1: Input: $T\ge 1$, $m\ge 1$, parameters ${\{S_k,{\eta }_k\}}_{k=1}^T\ge 1$ and random variables $R_{S_k}$ taking values in $\{1,\dots ,S_k\}$; 2: Initialize: $x_0=0$; 3: for $k=0,1,\dots ,T-1$ do 4:    $x_{k,0}=x_k$; 5:    for $s=0,1,\dots ,S_k-1$ do 6:      $x_{k,s}^0=x_{k,s}$; 7:      for $t=0,1,\dots ,m-1$ do 8:         $x_{k,s}^{t+1}=x_{k,s}^t-{\eta }_k[\nabla f_{i_t}\left(x_{k,s}^t\right)-\nabla f_{i_t}\left(x_{k,s}^0\right)+\nabla F\left(x_{k,s}^0\right)]$; 9:      end for 10:      $x_{k,s+1}=x_{k,s}^m$; 11:    end for 12:    $x_{k+1}=x_{k,R_{S_k}}$; 13: end for 14: Output: $x_T.$

In Algorithm 2, m is an arbitrary constant and denotes the number of inner iteration for VR, and the rest parameters are set as in Algorithm 1. In particular, the process of the so-called VR technique has been shown in the 7–9th line of Algorithm 2.

3.4. Theoretical Aspects of S-SVRG

The goal of this section is to show the results of the convergence rate of Algorithm 2. In particular, we give the following theorem in which the convergence rate of the Algorithm 2 is characterized.

Theorem 2.

Suppose that F is l.c.d. with constant L and meets the PL condition with modulus μ, and its gradient is bounded by G uniformly.

1.

If the parameters are chosen as ${\eta }_k\le min\left\{\frac{1}{2mGL}\sqrt{\frac{(1-m^2{\eta }_0^2{L}^2){ϵ}_k}{4+m{\eta }_{0}L}},\frac{1}{2mL}\right\}$, $S_k=\frac{4}{\mu m{\eta }_k(1-2{\eta }_{0}mL)}$, and probability mass function of $R_{S_k}$ is chosen such that $\mathbb{P}(R_{S_k}=s)=\frac{1}{S_k}$ for any $s=0,\dots ,S_k-1$, where ${ϵ}_0\ge {\Vert \nabla F\left(x_0\right)\Vert }^2$ and ${ϵ}_{k+1}={ϵ}_k/2$, we can find an ϵ-stationary point (a point $\mathring{x}$ such that $\mathbb{E}[\Vert \nabla F\left(\mathring{x}\right){\Vert }^2]\le ϵ$) by performing Algorithm 2 at most $O(Lm/\left(\mu \sqrt{ϵ}\right))$ times.

2.

If we further assume F is convex and the parameters are chosen as $S_k=\frac{1+log\left(4\right)}{\mu m{\eta }_k}$, ${\eta }_k\le min\left\{\frac{\mu \sqrt{(1-m^2{\eta }_0^2{L}^2){ϵ}_k}}{2mGL},\frac{1}{mL},\frac{1}{\mu m}\right\}$ and probability mass function are chosen such that $\mathbb{P}(R_{S_k}=S_k)=1$ and $\mathbb{P}(R_{S_k}=s)=0$ for $s=1,\dots ,S_k-1$, where ${ϵ}_0\ge {\Vert x_0-x^{\ast }\Vert }^2$ and ${ϵ}_{k+1}={ϵ}_k/2$, we can find an ϵ-minimizer (a point $\mathring{x}$ such that $\Vert \mathring{x}-x^{\ast }{\Vert }^2\le ϵ$) by performing Algorithm 1 at most $O(mL/\left({\mu }^2\sqrt{ϵ}\right))$ times.

The proof of Theorem 2 is presented in Appendix A.2.

Remark 2.

From the above theorem, it easy to verify that the iterative complexities of S-SVRG have been significantly reduced, which are both more superior than its non-stagewise counterpart SVRG and existing stagewise algorithm S-SGD under convex and nonconvex conditions. In other words, the convergence rates of S-SVRG have been significantly improved by SSTS. A detailed comparison is reported in Table 1.

So far, we have answered the main question considered in this paper in the affirmative; namely, the incorporation between SSTS and RSAG or SVRG does further improve the convergence rate.

4. Numerical Experiments

In the previous sections, we proposed two stagewise algorithms S-RSAG and S-SVRG, and analyzed the acceleration of their convergence rate. Now, we turn to consider their experimental performances.

4.1. Learning DNNs

In this subsection, we focus on testing our algorithm under the nonconvex condition, i.e., training DNNs. Particularly, we choose two familiar networks, MLP and VGG net, to examine the performances of our algorithms. Note that we are not trying to show that these two networks are the most efficient, but are attempting to show the superiority of our algorithms based on these two nonconvex models.

Firstly, we compare our stagewise algorithms S-RSAG and S-SVRG with their non-stagewise counterparts RSAG and SVRG. Then, we also compare them with the other state-of-art methods including stagewise SGD (S-SGD), VR-SGD [22], and Katyusha [40]. Experiments are performed on two commonly used datasets:

• MNIST: This dataset contains $28\times 28$ gray images from ten-digit classes. To improve learning efficiency, we load 10 samples per batch. We use 60,000 ($6000\times 10$) images for training, and the remaining 10,000 for testing. We adopt the 4-layer MLP network

  $$784FC-2048FC-1024FC-512FC-256FC-10SF$$

  to training, where $FC$ denotes a ReLU full-connected layer and $SF$ denotes a softmax output layer, for which the “CrossEntropyLoss” is adopted.
• CIFAR-10: This dataset contains $32\times 32$ color images from ten object classes. Similarly, we load 4 samples per batch. We use 50,000 (12,500 × 4) images for training, and the remaining 10,000 for testing. We adopt the VGG-like architecture

  $$\begin{array}{cc}\hfill & 32\times 32\times 3C3-MP2-16\times 16\times 64C3-MP2\hfill \\ \hfill & -8\times 8\times 128C3-MP2-4\times 4\times 512C3-MP2\hfill \\ \hfill & -2\times 2\times 256C3-MP2-10FC\hfill \end{array}$$

  where $C3$ denotes a $3\times 3$ ReLU convolution layer, and $MP2$ denotes a $2\times 2$ max-pooling layer, and $FC$ denotes a ReLU full-connected output layer, for which the “CrossEntropyLoss” is adopted.

In this section, the initial stepsize values for stagewise algorithms S-RSAG, S-SVRG and S-SGD are set as ${\eta }_0=0.5$ for MLP and ${\eta }_0=0.05$ for VGG. The iterations of these algorithms are divided into 5 epochs ($T=5$) for MLP and 10 epochs ($T=10$) for VGG. At each epoch, we run the iterations over the entire training set ergodicly, namely, the number of inner iterations at each epoch satisfies $S_k=6000$ for MLP and $S_k$ = 12,500 for VGG. At last, the stepsize for MLP decays by a factor of 2 and for VGG it decays by a factor of 1.5. In addition, the more the algorithm can tolerate large stepsize, the faster the convergence rate of the algorithm is, usually [22]. For other comparison algorithms, we tried several times to select a large as possible stepsize under the premise of ensuring convergence. We evaluate the performances of these algorithms in three aspects, i.e., value of loss, classification error rate on training set (training accuracy), and rate on testing set (testing accuracy). Next, we design the two following comparative experiments to verify our previous claims.

4.1.1. S-RSAG and S-SVRG vs. Their Non-Stagewise Counterparts

In this test, we attempt to verify the effectiveness of SSTS via comparing our stagewise algorithms S-RSAG and S-SVRG with their non-stagewise counterparts RSAG and SVRG. Figure 1 shows the behaviors of all the algorithms considered, in the three aspects (i.e., value of loss, training error, and testing error). Let us take a close look at the decay of loss function values: it is easy to find that the proposed S-RSAG and S-SVRG converge faster than their non-stagewise counterparts RSAG and SVRG, respectively. With respect to the training and testing accuracy, in most cases, our proposed RSAG and SVRG also can achieve the best accuracy more quickly.

4.1.2. S-RSAG and S-SVRG vs. Other Methods

In the above test, we have shown the effectiveness of SSTS via comparing the performances of our proposed algorithms with their non-stagewise counterparts. In this section, we verify whether the combination of SSTS and Nesterov’s acceleration or VR can further accelerate the convergence rate. Specifically, for the full-gradient-free S-RSAG, we compare it with other full-gradient-free method S-SGD; for the full-gradient-calibration S-SVRG, we compare it with VR-SGD and Katyusha.

Figure 2 shows the behaviors of all the algorithms considered. For full-gradient-free methods, as can be seen, S-RSAG outperforms S-SGD obviously in all the three aspects we considered. For full-gradient-calibration methods, we can see that our proposed S-SVRG achieves the fastest convergence rate with respect to the value of loss. Since our S-SVRG takes the last iteration as the output, the stability of the model is susceptible to random noise interference. From

Table 2. The computational time (s) for different methods in the nonconvex condition.

Methods	SGD	RASG	SVRG	S-SGD	Katyusha	S-RSAG	S-SVRG
MLP on MNIST	247.57	685.94	292.72	293.61	435.22	293.15	326.78

, it can be seen that S-SVRG and S-RASG have lower computational times. Therefore, S-SVRG’s performance in terms of training and testing accuracy is better than other methods.

In the end, from the above numerical results, it can be seen that the combination of SSTS with Nesterov’s acceleration or VR does further accelerate the convergence rate under nonconvex optimization. Next, we will examine the performances of our proposed algorithms under the convex condition.

4.2. Logistic Regression

In the above section, we have examined the performance of our proposed algorithms under the nonconvex condition (training DNNs). In this section, we test our algorithms on logistic regression problem, which is under the convex condition and can be viewed as a one-layer fully connected network with SoftMarginLoss. Experiments are performed on two commonly used datasets downloaded from the libsvm website (https://www.csie.ntu.edu.tw/~cjlin/libsvm/).

• REAL-SIM: This contains 72,309 data points of 20,958 features from two object classes. We divide it into two sets, i.e., one for training and the other for testing.
• RCV1: This contains 20,242 data points of 47,236 features from two object classes. Similarly, we also divide it into two sets, averagely, one for training and the other for testing.

In this section, the initial values of stepsize for stagewise algorithm S-RSAG, S-SVRG and S-SGD are chosen as ${\eta }_0=5$. The iterations are divided into 5 epochs ($T=5$). At each epoch, we set the batch size as 10 and run the iterations over the entire training set ergodicly, namely, the number of inner iteration at each epoch satisfies $S_k=1500$ for RCV1 and $S_k=5000$ for REAL-SIM. For other comparison algorithms, we tried several times to select as large as possible a stepsize under the premise of ensuring convergence. We evaluate the performances of these algorithms in the aspects of loss, training, and testing accuracy.

4.2.1. S-RSAG and S-SVRG vs. Their Non-Stagewise Counterparts

In this test, we attempt to verify the effectiveness of SSTS under the convex condition. Figure 3 shows the behaviors of all the algorithms considered. With respect to the decay of loss function values, it is easy to find that the proposed S-RSAG and S-SVRG converge faster than their non-stagewise counterparts RSAG and SVRG, respectively. With respect to the training accuracy, S-RSAG and S-SVRG also outperform their non-stagewise counterparts. As shown in subfigure (f), with respect to the testing accuracy, our proposed methods are inferior to the comparison methods. This is due to overfitting; that is, the faster the algorithm converges, the weaker the generalization ability is.

4.2.2. S-RSAG and S-SVRG vs. S-SGD

Figure 4 shows the behaviors of all the algorithms considered. As can be seen, S-RSAG outperforms S-SGD, and S-SVRG achieves the best performance other than full-gradient-calibrated methods, in most cases. We have verified the effectiveness of SSTS under the convex condition.

5. Discussion

Inspired by the phenomenon of optimization algorithms equipped into SSTS and the continuous strategy in nonconvex optimization, we incorporated SSTS into the iterative framework of RSAG and SVRG, and proposed S-RSAG and S-SVRG. We show that the iterative complexities of S-RSAG are significantly reduced with respect to its non-stagewise counterpart RSAG, and are more superior than the existing stagewise algorithm S-SGD under the convex condition and at the same level under the nonconvex condition, and the iterative complexities of S-SVRG are significantly superior than both its non-stagewise counterpart SVRG and the existing stagewise algorithm S-SGD under convex and nonconvex conditions. We will further discuss the lower bound for our proposed methods and explore the performance of gradient in higher-dimensional constrained optimization in the future.

6. Conclusions

In this paper, we mainly proposed a conjecture as to whether the incorporation between SSTS, which is a common method to train DNNs empirically, and Nesterov’s acceleration or VR-based methods can further improve the convergence rate. Particularly, we propose two SSTS equipped accelerated algorithms and answered the above conjecture in the affirmative theoretically under both convex and nonconvex conditions, respectively. Furthermore, we examine the performance of our equipped algorithms by designing contrast experiments on training DNNs and logistic regression, which validates the competitiveness of our methods well.