On the Convergence Rate of Quasi-Newton Methods on Strongly Convex Functions with Lipschitz Gradient

Abstract

The main results of the study of the convergence rate of quasi-Newton minimization methods were obtained under the assumption that the method operates in the region of the extremum of the function, where there is a stable quadratic representation of the function. Methods based on the quadratic model of the function in the extremum area show significant advantages over classical gradient methods. When solving a specific problem using the quasi-Newton method, a huge number of iterations occur outside the extremum area, unless there is a stable quadratic approximation of the function. In this paper, we study the convergence rate of quasi-Newton-type methods on strongly convex functions with a Lipschitz gradient, without using local quadratic approximations of a function based on the properties of its Hessian. We proved that quasi-Newton methods converge on strongly convex functions with a Lipschitz gradient with the rate of a geometric progression, while the estimate of the convergence rate improves with the increasing number of iterations, which reflects the fact that the learning (adaptation) effect accumulates as the method operates. Another important fact discovered during the theoretical study is the ability of quasi-Newton methods to eliminate the background that slows down the convergence rate. This elimination is achieved through a certain linear transformation that normalizes the elongation of function level surfaces in different directions. All studies were carried out without any assumptions regarding the matrix of second derivatives of the function being minimized.

1. Introduction

Quasi-Newton (QN) methods for solving nonlinear optimization problems are based on the idea of reconstructing the matrix of second derivatives of a function from its gradients. The reconstructed matrix is used similarly to the second derivative matrix in Newton’s method. Quasi-Newton methods are effective tools for solving smooth minimization problems. Methods from QN class are less costly than the Newton method for solving large-scale optimization problems because their iterations do not use second-order derivatives. QN methods are applied to various areas such as physics, biology, engineering, geophysics, chemistry, and industry to solve the nonlinear systems of equations. QN methods can be applied in the Deep Learning area as a method for the empirical risk minimization, where the number of samples as well as the number of variables is large [1,2,3]. In microscopy, QN methods help to achieve high resolution imaging [4]. In modeling the spread of infections, QN is useful for the identification of the unknown model coefficients [5]. QN methods are also useful for the modeling of complex crack propagation [6], fluid–structure interaction [7,8,9], melting and solidification of alloys [10], heat transfer systems [11], etc.

Nowadays, there are a significant number of matrix reconstruction formulas in QN methods, and hundreds of papers have been written on the topic of quasi-Newton methods (see, for example, [12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29] and their bibliography). The first QN method was proposed in [20] and improved in [24]. This matrix update formula is known as DFP (Davidon–Fletcher–Powell). Symmetric Rank 1 (SR1) method [19,20] is another way to update the Hessian. Today, it is generally accepted [12,13,14,30] that the BFGS (Broyden–Fletcher–Goldfarb–Shanno) matrix updating formula [19,23,26,31] is the most efficient among the family of QN methods.

The main results of the convergence of quasi-Newton methods in the extremum region are given in [13,15]. These results refer to the convergence rate of quasi-Newton minimization methods under the assumption that the method operates in the extremum area of the function, where there is a stable quadratic representation of the function and where methods based on the quadratic model show significant advantages over standard gradient methods.

An incremental quasi-Newton method with a local superlinear convergence rate is presented in [32,33]. The proposed incremental algorithm reduces computational cost by restricting the update to a single function per iteration and relative to incremental second-order methods by removing the need to compute the inverse of the Hessian.

The authors in [34] showed that a range of QN methods are first-order methods in the Nesterov definition [35]. They extended the complexity analysis for smooth strongly convex problems to finite dimensions and showed that in a worst-case scenario, the local superlinear or faster convergence rates of QN methods cannot be improved unless the number of iterations exceeds half of the problem size.

In [36], the authors confirmed certain superlinear convergence rates for the QN method, depending on the problem size and the specifically defined condition number. The analysis was developed based on the trace potential function, which was scaled by the logarithm of determinant of the inverse Hessian approximation to extend the proof to the general nonlinear case. The results of [36] were further improved in [37], where the authors demonstrated that the convergence rate of the BFGS method depends only on the product of the problem dimensionality and the logarithm of its condition number.

Another analysis of local non-asymptotic superlinear convergence of the DFP and BFGS methods was presented in [38]. The authors showed that in a local neighborhood of the optimal solution, the iterations generated by both DFP and BFGS converge to the optimal solution in a superlinear rate (1/k)^k/2, where k is the number of performed iterations.

A sampled version of the BFGS method named limited-memory BFGS (L-BFGS) [39] was presented to handle with high dimensional problems. The algorithm stores only a few vectors that represent the approximation of Hessian instead of the entire matrix. A version with bound constraints was proposed in [40]. The algorithm developed in [1] generates points randomly around the current iterate at every iteration to produce approximations that do not depend on information about past iterations.

Randomized variants of QN algorithms have been recently investigated. Such random methods employ a random direction at each iteration for updating the approximate Hessian matrix. The online L-BFGS method [41] adapts the L-BFGS method to make use of subsampled gradients. The regularized BFGS method [42,43] modifies the BFGS update by adding a regularizer to the metric matrix. The stochastic block BFGS method was proposed in [44]. This method enables the incorporation of curvature information in stochastic approximation. The estimate of the inverse Hessian matrix is updated at each iteration using a randomly generated compressed form of the Hessian. Such an approach was called a “sketch” technique. Then, the authors developed an adaptive variant of the randomized block BFGS, AdaRBFGS in [45], in which they modified the distribution underlying the stochasticity of the method throughout the iterative process. Further, in [46], it was shown that the block BFGS method also converges superlinearly, and a framework using a curvature-adaptive step size was introduced. In [47], a stochastic QN method is proposed that employs the classical BFGS update formula in its limited memory form and is based on collecting curvature information pointwise and at regular intervals through Hessian-vector products. In [48], the authors study stochastic QN methods in the interpolation setting and prove that these algorithms, including L-BFGS, can achieve global linear convergence with a constant batch size. The authors in [30] provide a semi-local convergence rate for the randomized BFGS method under the assumption that the function is self-concordant. An extension of BFGS proposed in [49] generates an estimate of the true objective function by taking the empirical mean over a sample drawn at each step and attains R-superlinear convergence. A regularized stochastic accelerated QN method (RES-NAQ) that combines the concept of the regularized stochastic BFGS method (RES) with the Nesterov accelerating technique by introducing a new momentum coefficient was proposed in [50].

Greedy variants of the QN method were introduced in [51]. In contrast to the classical QN methods, which use the difference of successive iterations for updating the Hessian approximations, the method in [40] applies basis vectors, greedily selected to maximize a certain measure of progress. An explicit non-asymptotic bound on the local superlinear convergence rate was established. This approach was further improved in [52,53] with methods of condition-number-free superlinear convergence speed.

When solving a specific problem using the QN method, a huge number of iterations occur outside the extremum area where there is no stable quadratic approximation of the function. In this paper, our aim is to study the convergence rate of quasi-Newton-type methods without assuming the existence of second derivatives of the function. Strongly convex functions with a Lipschitz gradient are studied, and local quadratic approximations of the function based on information about the properties of its Hessian are not used.

The obtained results are related to the estimates for the convergence rate of quasi-Newton methods on strongly convex functions with a Lipschitz gradient by means of a geometric progression. It is shown that the indicators for estimating the convergence speed improve with the increase in the number of iterations of the method, which indicates the benefit of adjusting the metric matrices in the method.

It is known that it is possible to both reduce the spread of elongation of level surfaces along different directions and increase it with the help of a linear transformation of coordinates. Quasi-Newton methods eliminate this scatter on quadratic functions. The work shows that in the case of strongly convex functions, there is a scattering that can be eliminated using a linear transformation of coordinates, and then, the quasi-Newton method also eliminates it. That is, it is possible to improve the behavior of a strongly convex function using some linear coordinate transformation, and then, the quasi-Newton method can recreate this coordinate transformation. This property of the method is based on its invariance under a linear coordinate transformation, which allows us to consider the method in a coordinate system with better characteristics in terms of estimates of the convergence rate.

The rest of the paper is organized as follows. In Section 2, we provide basic information about quasi-Newton methods. In Section 3, we restate necessary information about strongly convex functions and obtain an estimate for the convergence rate of quasi-Newton methods on strongly convex functions with a Lipschitz gradient, depending on the convexity constants and Lipschitz constants. Accelerating properties of quasi-Newton methods on strongly convex functions with a Lipschitz gradient are considered in Section 4. Numerical results are presented in Section 5. A short conclusion of the obtained results is given in the last section.

2. Quasi-Newton Methods

The iteration of the quasi-Newton method has the following form (see, for example, [12]):

$$x_{k+1}=x_k+{\mathit{\beta }}_k{s}_k,$$

(1)

$$s_k=-H_k\nabla f\left(x_k\right), {\mathit{\beta }}_k=\underset{\mathit{\beta }\ge 0}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}f\left(x_k+\mathit{\beta }{s}_k\right),$$

(2)

where $\nabla f\left(x_k\right)$ is the gradient of the objective function f at $x_k$, $H_k$ denotes an approximation of the Hessian inverse ${\left[{\nabla }^{2}f\left(x_k\right)\right]}^{-1}$, $s_k$ is a search direction, and ${\mathit{\beta }}_k$ is chosen to satisfy the Wolfe conditions. The following notation will be used:

$$\Delta x_k=x_{k+1}-x_k, y_k=\nabla f\left(x_{k+1}\right)-\nabla f\left(x_k\right),$$

(3)

$$H_{k+1}=H\left(H_k,\Delta x_k,y_k\right),$$

(4)

where $H_{k+1}=H\left(H_k,\Delta x_k,y_k\right)$ denotes an appropriate formula for updating matrices $H_k$. The initial iterative point is denoted by x₀ and the approximation of ${\left[{\nabla }^{2}f\left(x_0\right)\right]}^{-1}$ satisfies H₀ > 0 (H₀ = I is usually assumed).

We will denote by A($A_k,\Delta x_k,y_k$) the operator of the Hessian ${\nabla }^{2}f\left(x_k\right)$ approximation. Process (1)–(4) is a certain approximation of the Newton optimization method. We will be interested in the accelerating properties of QN methods and the conditions for their appearance.

Well-known rules for updating the matrices $H_k$ and $A_k$ are as follows. The Davidon–Fletcher–Powell (DFP) updating formula [20,24] is given by the following:

$${H_{k+1}=H}_{DFP}\left(H_k,\Delta x_k,y_k\right)=H_k-\frac{H_k{y}_k{y}_k^T{H}_k}{\left(y_k,H_k{y}_k\right)}+\frac{{\Delta x}_k{\Delta x}_k^T}{\left({\Delta x}_k, y_k\right)},$$

(5)

$$A_{k+1}=A_{DFP}\left(A_k,\Delta x_k,y_k\right)=A_k-\frac{\left(y_k-A_k{\Delta x}_k,{\Delta x}_k\right)y_k{y}_k^T}{{\left(y_k,{\Delta x}_k\right)}^2}+\frac{\left(y_k-A_k{\Delta x}_k\right)y_k^T+y_k{\left(y_k-A_k{\Delta x}_k\right)}^T}{\left(y_k,{\Delta x}_k\right)},$$

(6)

such that $H_k=A_k^{-1}$.

The Broyden–Fletcher–Goldfarb–Shanno (BFGS) updating formula [19,23,26,31] is defined as follows:

$${H_{k+1}=H}_{BFGS}\left(H_k,\Delta x_k,y_k\right)=H_k-\frac{\left({\Delta x}_k-H_k{y}_k,y_k\right){\Delta x}_k{\Delta x}_k^T}{{\left(y_k,{\Delta x}_k\right)}^2}+\frac{\left({\Delta x}_k-H_k{y}_k\right){\Delta x}_k^T+{\Delta x}_k{\left({\Delta x}_k-H_k{y}_k\right)}^T}{\left(y_k,{\Delta x}_k\right)}.$$

(7)

The one-parameter family of formulas combines (5) and (7) (see, for example, [12]):

$$H(H_k,\Delta x_k,y_k)=\mathit{\gamma }{H}_{BFGS}(H_k,\Delta x_k,y_k)+(1-\mathit{\gamma })H_{DFP}(H_k,\Delta x_k,y_k), \mathit{\gamma }\in \left[\mathrm{0,1}\right].$$

(8)

Equation (8) can be represented as follows:

$$H_{k+1}=H(H_k,\Delta x_k,y_k)=H_k-\frac{H_k{y}_k{y}_k^T{H}_k}{(y_k,H_k{y}_k)}+\frac{{\Delta x}_k{\Delta x}_k^T}{({\Delta x}_k,y_k)}+\mathit{\gamma }{v}_k{v}_k^T,$$

(9)

where

$$v_k=(y_k,H_k{y}_k{)}^{0.5}\left[\frac{{\Delta x}_k}{\left({\Delta x}_k,y_k\right)}-\frac{H_k{y}_k}{\left(y_k,H_k{y}_k\right)}\right],\mathit{\gamma }\in \left[\mathrm{0,1}\right].$$

(10)

In the exact one-dimensional search (2), the approximations $x_k$ obtained from Formulas (1)–(4) coincide, where one of the recalculation formulas for a one-parameter family is used with an arbitrary choice of β ∈ [0, 1] (see [22]).

In what follows, a symmetric positive definiteness of the matrix H will be denoted by H > 0. If $H_0$ > 0, then the family (8) generates symmetric matrices $H_k$.

3. Convergence Rate of Quasi-Newton Methods on Strongly Convex Functions with Lipschitz Gradient

The main known studies on the convergence rate of QN methods were carried out in the region of function extremum in the presence of a stable quadratic representation of the function. The fact of the geometric progression convergence rate for the DFP method was established in [15] under the condition that the function is three times continuously differentiable and the matrix of second derivatives is bounded. In our work, an estimate of the convergence rate of a one-parameter family of QN methods on strongly convex functions was considered without the assumption of the existence of second derivatives. Accelerating properties of the QN family were substantiated in comparison with the gradient method. The obtained results indicate that QN methods initially based on the assumption that the quadratic representation of a function exists in the neighborhood of a point are able to approximate a coordinate transformation that reduces the degree of function degeneracy even in the absence of a quadratic representation of the function in the neighborhood of a point. Due to this fact, QN methods have an advantage in convergence speed compared to the steepest descent method. This result forms the content of this section.

For the simplicity, the notations g(x) and $g\left(x_k\right)$ will be used instead of ∇f(x) and ∇f($x_k$). In what follows, we will assume the following condition.

Condition 1. 

The objective function f(x), x ∈ $R^n$ is differentiable and strongly convex in $R^n$, i.e., there exists ρ > 0 such that the inequality,

$$f\left(\mathit{\alpha }x+\left(1-\mathit{\alpha }\right)y\right)\le \mathit{\alpha }f\left(x\right)+(1-\mathit{\alpha })f\left(y\right)-\mathit{\alpha }(1-\mathit{\alpha })\mathit{\rho }{‖x-y‖}^2/2,$$

(11)

Holds for all x ∈ $R^n$, y ∈ $R^n$ and α ∈ [0, 1], and the gradient g(x) satisfies the Lipschitz condition,

$$‖g\left(x\right)-g\left(y\right)‖\le L‖x-y‖\hspace{1em}\forall x,y\in R^n, L>0.$$

(12)

Functions which fulfill Condition 1 satisfy the following relations [5]:

$$\left(g\left(x\right)-g\left(y\right),x-y\right)\ge \mathit{\rho }{‖x-y‖}^2\hspace{1em}\forall x,\mathrm{y}\in R^n,$$

(13)

$$f\left(x\right)-f^{*}\le \frac{{‖g\left(x\right)‖}^2}{2\mathit{\rho }}, \forall x\in R^n,$$

(14)

$$\left(g\left(x\right)-g\left(y\right),x-y\right)\ge \frac{{‖g\left(x\right)-g\left(y\right)‖}^2}{L}, \forall x,y\in R^n,$$

(15)

$$f\left(x\right)-f^{*}\ge \frac{\mathit{\rho }{‖x-x^{*}‖}^2}{2, },\hspace{1em}\forall x\in R^n,$$

(16)

where x* is the minimum point and f* = f(x*) is the function value at the minimum point.

Since the sequences of approximations x_k obtained utilizing Formulas (1)–(4) during an exact one-dimensional search (2) coincide for an arbitrary choice in the matrix transformation formula (9) [22], then all further reasoning will be carried out using the sequence of matrices H_k generated by the DFP Formula (5).

If the matrix H₀ is symmetric, then the family (8) generates symmetric matrices. If the condition

$$(y_k,\Delta x_k)>0$$

(17)

holds and the matrix H₀ is strictly positive definite, then the matrices H_k retain strict positive definiteness [13]. If the function f satisfies Condition 1, then (13) implies the validity of (17). This proves that the sequence H_k obtained by the rules (8) or (9) will be strictly positive definite when the objective function satisfies Condition 1.

In Lemma 1 we estimate the reduction coefficient of the function at iteration depending on changes in the gradient.

Lemma 1. 

Let the objective function f satisfy Condition 1. Then, as a result of iterations defined by (1)–(4), the function decreases with the estimate as follows:

$$f_{k+1}-f^{*}\le q_k\left(f_k-f^{*}\right),$$

(18)

where

$$q_k={\left[1+\frac{{\mathit{\rho }}^2{‖y_k‖}^2}{L^2{‖g_{k+1}‖}^2}\right]}^{-1}, ‖g_{k+1}‖>0.$$

(19)

In addition, the following inequality holds for a sequence of iterations $f_j$= 0, 1,…, k:

$$f_{k+1}-f^{*}\le Q_k(f_0-f^{*}), Q_k={\prod }_{j=0}^k{q}_{j.}$$

(20)

Proof of Lemma 1. 

The exact value of the indicator is as follows:

$$q_k=\frac{f_{k+1}-f^{*}}{f_k-f^{*}}=\frac{f_{k+1}-f^{*}}{(f_{k+1}-f^{*})+(f_k-f_{k+1})}={\left[1+\frac{f_k-f_{k+1}}{f_{k+1}-f^{*}}\right]}^{-1}.$$

(21)

Let us make estimates for the denominator terms in (21). According to (14),

$$f_{k+1}-f^{*}\le \frac{{‖g_{k+1}‖}^2}{2\mathit{\rho }}.$$

(22)

The Lipschitz condition gives the following inequality:

$$||\Delta x_k{||}^2\ge \frac{{‖y_k‖}^2}{L^2}.$$

(23)

In view of (16), which is also true for a one-dimensional function along the direction $\Delta x_k$, the following holds:

$$f_k-f_{k+1}\ge \mathit{\rho }\left|\right|\Delta x_k|{|}^2/2.$$

(24)

Combining (24) with (23) leads to the following:

$$f_k-f_{k+1}\ge \mathit{\rho }\frac{|\left|y_k\right|{|}^2}{2L^2}.$$

(25)

Then, the application of the estimates (22) and (25) in (21) imply (19). A recursive application of (19) produces (20). □

Denote by Sp(H) the trace of a matrix H. Applying the formulas for the matrix traces of H_k and $A_k=H_k^{-1}$ obtained in [15], we generate estimates based on which evaluations of the convergence rate are constructed.

Lemma 2. 

Let the function satisfy Condition 1. The following estimates hold for sequences {H_j}, {y_j}, {g_j}, j = 0, 1,…, k generated as a result of the iterative process (1)–(4):

$${\sum }_{j=0}^k{a}_j\le (k+1)c_a, a_j=\frac{(g_{j+1},H_j{g}_{j+1})}{||y_{j+1}|{|}^2}, c_a=\frac{1}{\mathit{\rho }}+\frac{Sp(H_0)}{k+1},$$

(26)

$${\sum }_{j=0}^k{b}_j\le (k+1)c_b, b_j=\frac{||g_{j+1}|{|}^2}{(g_{j+1},H_j{g}_{j+1})}, c_b=L+\frac{Sp(A_0)}{(k+1)},$$

(27)

$$f_{k+1}-f\ast \le q_k(f_k-f^{*}),$$

(28)

where

$$q_k={\left[1+\frac{{\mathit{\rho }}^2{‖y_k‖}^2}{L^2{‖g_{k+1}‖}^2}\right]}^{-1}={\left[1+c_0{a}_k{b}_k\right]}^{-1}=\frac{a_k{b}_k}{c_0+a_k{b}_k},{ c}_0=\frac{{\mathit{\rho }}^2}{L^2}, ‖g_{k+1}‖>0.$$

(29)

In addition, the following inequality holds for a sequence of iterations $f_j$ , j = 0, 1,…, k:

$$f_{k+1}-f^{*}\le Q_k(f_0-f^{*}), Q_k={\prod }_{j=0}^k{q}_j={\prod }_{j=0}^k\frac{a_j{b}_j}{c_0+a_j{b}_j}.$$

(30)

Proof of Lemma 2. 

Expressions for the trace of the matrices A_k and H_k were calculated in [15] through Formulas (5) and (6) as follows:

$$Sp\left(H_{k+1}\right)=Sp\left(H_0\right)-\sum _{j=0}^k\frac{||H_j{y}_j|{|}^2}{\left(y_j,H_j{y}_j\right)}+\sum \frac{||\Delta x_j|{|}^2}{\left(y_j,\Delta x_j\right)},$$

(31)

$$Sp(A_{k+1})=S_P(A_0)+\frac{||g_{k+1}|{|}^2}{(g_{k+1},H_{k+1}{g}_{k+1})}-\frac{||g_0|{|}^2}{(g_0,H_0{g}_0)}-\sum _{j=0}^k\frac{||g_{j+1}|{|}^2}{(g_{j+1},H_j{g}_{j+1})}+\sum _{j=0}^k\frac{||y_j|{|}^2}{(\Delta x_j,y_j)}.$$

(32)

As noted earlier, the matrices H_k and therefore A_k are strictly positive definite. Due to the remark made above about the identity of sequences x_k for different γ ∈ [0, 1] in (9), we will carry out the proof for the sequence H_k generated by (5).

Due to the inequality

$${‖z‖}^2\le S_P\left(A_{k+1}\right)\cdot \left(z,H_{k+1}z\right),$$

valid for every $z\in R^n$, the next inequality is as follows:

$$S_P(A_{k+1})-\frac{{‖g_{k+1}‖}^2}{\left(g_{k+1},H_{k+1}{g}_{k+1}\right)}\ge 0.$$

From (32), considering the last inequality and (15), one obtains the following:

$$\begin{gathered} \sum _{j=0}^k\frac{{‖g_{j+1}‖}^2}{\left(g_{j+1},H_j{g}_{j+1}\right)}=S_P(A_0)-\frac{{‖g_0‖}^2}{\left(g_0,H_0{g}_0\right)}-\left[S_P(A_{k+1})-\frac{{‖g_{k+1}‖}^2}{\left(g_{k+1},H_{k+1}{g}_{k+1}\right)}\right] \\ +\sum _{j=0}^k\frac{{‖y_j‖}^2}{(\Delta x_j,y_j)}\le S_P\left(A_0\right)+\sum _{j=0}^k\frac{{‖y_j‖}^2}{\left(\Delta x_j,y_j\right)}\le S_P\left(A_0\right)+\left(k+1\right)L. \end{gathered}$$

(33)

The inequality (13) implies the following:

$$\frac{{‖\Delta x_j‖}^2}{\left(y_j,\Delta x_j\right)}\le \frac{1}{\mathit{\rho }}.$$

(34)

The Schwartz’s inequality leads to the following:

$${\left(y_j,H_j{y}_j\right)}^2\le {‖H_j{y}_j‖}^2{‖y_j‖}^2.$$

(35)

Due to the exact one-dimensional search condition (3.8), the next equality holds:

$$-\left(g_{j+1},H_j{g}_j\right)=\left(g_{j+1},\Delta x_j\right)=0.$$

From this and from the positive definiteness of the matrices, the following holds:

$$\begin{array}{ll}\left(y_j,H_j{y}_j\right)& =\left(g_{j+1},H_j{g}_{j+1}\right)+\left(g_j,H_j{g}_j\right)-2\left(g_{j+1},H_j{g}_j\right)\\ & =\left(g_{j+1},H_j{g}_{j+1}\right)+\left(g_j,H_j{g}_j\right)\ge \left(g_{j+1},H_j{g}_{j+1}\right).\end{array}$$

(36)

Considering (34)–(36), the equality (31) is transformed as follows:

$$\begin{array}{ll}{\displaystyle \sum _{j=0}^k}\frac{\left(g_{j+1},H_j{g}_{j+1}\right)}{{‖y_j‖}^2}& \le {\displaystyle \sum _{j=0}^k}\frac{{‖H_j{y}_j‖}^2}{\left(y_j,H_j{y}_j\right)}=\\ & =S_P\left(H_0\right)-S_P\left(H_{k+1}\right)+{\displaystyle \sum _{j=0}^k}\frac{{‖\Delta x_j‖}^2}{\left(y_j,\Delta x_j\right)}\le S_P\left(H_0\right)+\frac{k+1}{\mathit{\rho }}.\end{array}$$

(37)

From (37) and (33), we arrive at the following inequalities:

$$\sum _{j=0}^k\frac{\left(g_{j+1},H_j{g}_{j+1}\right)}{{‖y_j‖}^2}\le S_p\left(H_0\right)+\frac{k+1}{\mathit{\rho }}=\frac{k+1}{\mathit{\rho }}\left[1+\frac{\mathit{\rho }Sp\left(H_0\right)}{k+1}\right],$$

(38)

$$\sum _{j=0}^k\frac{{‖g_{j+1}‖}^2}{\left(g_{j+1},H_j{g}_{j+1}\right)}\le S_P\left(A_0\right)+\left(k+1\right)L=\left(k+1\right)L\left[1+\frac{Sp\left(A_0\right)}{L\left(k+1\right)}\right].$$

(39)

Inequalities (38) and (39) are identical to (26) and (27). Estimate (30) follows from (20). □

The convergence rate of the QN method is determined by the indicator $Q_k$ in (20). In the next lemma, we find an upper bound for this indicator. The problem is formulated as follows:

$$\begin{gathered} \{-Q_k\}\to \underset{a,b}{\mathrm{m}\mathrm{i}\mathrm{n}} \\ \mathrm{subject} \mathrm{to} \left(26\right) \mathrm{and} \left(27\right), \end{gathered}$$

(40)

where a^T = (a₀, a₁, …, a_k), b^T = (b₀, b₁, …, b_k).

Lemma 3. 

Under the conditions of Lemma 2, the solution to the problem (40) is of the following form:

$$a_j^{*}=c_a=\frac{1}{\mathit{\rho }}+\frac{Sp\left(H_0\right)}{k+1}, b_j^{*}=c_b=L+\frac{Sp\left(A_0\right)}{(k+1)}, j=0,1,\dots k.$$

(41)

The optimal value is equal to the following:

$$Q_k^{*}=(q_k^{*}{)}_{}^{k+1},$$

(42)

where

$$q_k^{*}=\frac{c_a{c}_b}{c_0+c_a{c}_b}=\frac{1}{1+c_0/\left(c_a{c}_b\right)}.$$

(43)

Proof of Lemma 3. 

To solve the problem (40), we form the Lagrange function:

$$L\left(a,b,y_a,y_b\right)=-\prod _{j=0}^k\frac{a_j{b}_j}{c_0+a_j{b}_j}+y_a\left(\sum _{j=0}^k{a}_j-\left(k+1\right)c_a\right)+y_b\left(\sum _{j=0}^k{b}_j-\left(k+1\right)c_b\right).$$

The partial derivatives of L are equal to the following:

$$\begin{array}{ll}\frac{\mathsf{\partial }L}{\mathsf{\partial }{a}_j}& =-\frac{Q_k}{q_j}\times \frac{b_j(a_j{b}_j+c_0)-b_j\left(a_j{b}_j\right)}{(c_0+a_j{b}_j{)}^2}+y_a\\ & =-\frac{Q_k}{q_j} \frac{b_j{c}_0}{(c_0+a_j{b}_j{)}^2}+y_a=-Q_k\frac{c_0}{a_j\left(c_0+a_j{b}_j\right)}+y_a=0,\end{array}$$

(44)

which implies

$$a_j(c_0+a_j{b}_j)=c_0{Q}_k/y_{a.}$$

(45)

Similarly, the coefficients b_j fulfill the following:

$$b_j\left(c_0+a_j{b}_j\right)=c_0{Q}_k/y_b.$$

(46)

From the expressions (45) and (46), it is easy to obtain that all elements $a_j$ are the same. This property is also true for elements $b_j$.

If the solution lies on the boundary, we write the subsequent equalities to find the parameters $a_j$ and $b_j$:

$${\sum }_{j=0}^k{a}_j=(k+1)c_a, {\sum }_{j=0}^k{b}_j=(k+1)c_b.$$

From here, we obtain the statement (41).

The matrix of second derivatives of the Lagrange function is diagonal, and its elements are positive. Consequently, at the point with parameters (41), the sufficient conditions for the extremum are satisfied as the parameters from (41) are a solution to the problem (40). □

Theorem 1. 

Let the function satisfy Condition 1. Then, for the sequence of iterations $f_j$, j = 0, 1,…, k, the convergence rate of the goal function is estimated as follows:

$$f_{k+1}-f^{*}\le Q_k^{*}(f_0-f^{*}), Q_k^{*}=(q_k^{*}{)}_{}^{k+1},$$

(47)

where

$$q_k^{*}\le 1-\frac{{\mathit{\rho }}^3}{2L^3\left[1+\frac{\mathit{\rho }n{M}_0}{k+1}\right]\times \left[1+\frac{n}{L{m}_0\left(k+1\right)}\right]}.$$

(48)

Here, M₀ and m₀ are maximal and minimal eigenvalues of the matrix H₀, respectively.

Proof of Theorem 1. 

We use the worst-case scenario for estimating the reduction index of the function (43) in (42), and transform it considering (26), (27), (40) and (41):

$$q_k^{*}=\frac{1}{1+c_0/\left(c_a{c}_b\right)}={\left[1+\frac{{\mathit{\rho }}^3}{L^3\left[1+\frac{\mathit{\rho }Sp\left(H_0\right)}{k+1}\right]\times \left[1+\frac{Sp\left(H_0^{-1}\right)}{L(k+1)}\right]}\right]}^{-1}.$$

(49)

To transform (49), we use the following inequality:

$$\frac{1}{1+d}\le 1-\frac{d}{2},d\in \left[\mathrm{0,1}\right].$$

(50)

Multiplying (50) by 1 + d gives the following:

$$1\le \left(1+d\right)\left(1-\frac{d}{2}\right)=1+\frac{d}{2}\left(1-d\right).$$

Based on this inequality, (50) is valid for d ∈ [0, 1].

Since in (49) the quantity below is positive and limited by

$$d=\frac{{\mathit{\rho }}^3}{L^3\left[1+\frac{\mathit{\rho }Sp\left(H_0\right)}{k+1}\right]\times \left[1+\frac{Sp\left(H_0^{-1}\right)}{L(k+1)}\right]}\le 1,$$

we use (50) to transform (49). The result is the following inequality:

$$q_k^{*}\le 1-\frac{{\mathit{\rho }}^3}{2L^3\left[1+\frac{\mathit{\rho }Sp\left(H_0\right)}{k+1}\right]\times \left[1+\frac{Sp\left(H_0^{-1}\right)}{L(k+1)}\right]}.$$

(51)

In view of the relations for the matrix traces

$$Sp\left(H_0\right)\le n{M}_0, Sp\left(H_0^{-1}\right)\le \frac{n}{m_0},$$

(51) leads to the estimate (48). □

For a sufficiently large k, the convergence rate of QN methods can be clearly represented in the following form:

$$q\approx 1-\frac{{\mathit{\rho }}^3}{2L^3}.$$

(52)

Note that we did not involve any information about the matrix of second derivatives when obtaining estimates of the convergence rate. If the objective function is twice differentiable, then the eigenvalues of the matrix of second derivatives are bounded by the interval of the strong convexity parameter and Lipschitz parameter [ρ,L].

Let us analyze the convergence rate indicator (48).

• The estimate (48) is determined by the strong convexity parameter ρ, Lipschitz parameter L and the properties of the initial matrix H₀.
• As the number of iterations k increases, the estimate of the indicator $q_k^{*}$ decreases and tends to the value from the expression (52). This fact is consistent with the expected improvements in convergence rates resulting from the matrix transformation process in QN methods.

Due to the invariance of QN methods, estimates (48) can be considered in different coordinate systems. In the next section, we will consider the method in a coordinate system, where the ratio ρ/L is maximal.

4. Accelerating Properties of QN Methods on Strongly Convex Functions with Lipschitz Gradient

Further research is aimed at determining the conditions, except for the trivial case of minimizing a quadratic function, under which QN methods are superior in convergence rate to the steepest descent method. Quasi-Newton methods are invariant under variable transformations,

$$\widehat{x}=Px,$$

where P is a non-singular (n × n) matrix [13]. This means that the type of process (1)–(4) is completely identical to the type of process in the new coordinate system. In this case, for identical process values, the relationship

$$\Delta \widehat{x}=P\Delta x, \widehat{y}=P^{-T}y, \widehat{\mathrm{g}}=P^{-T}g, \widehat{H}=PH{P}^T, \widehat{x}=Px,$$

(53)

is valid if the initial conditions are related by the following relations:

$${\widehat{x}}_0=P{x}_0,{\widehat{H}}_0=P{H}_0{P}^T.$$

(54)

Here, $P^{-T}={\left(P^T\right)}^{-1}$. In the new coordinate system, the process (1)–(4) is equivalent to minimizing the function,

$$\phi (\widehat{x})=f\left(P^{-1}\widehat{x}\right)=f(x),$$

which, as is easy to show, satisfies Condition 1 with strong convexity constant ${\mathit{\rho }}_P$ and Lipschitz constant $L_P$.

Define the following transformation:

$$\widehat{x}=Vx,$$

where V is a non-singular matrix such that

$$\frac{{\mathit{\rho }}_V}{L_V}\ge \frac{{\mathit{\rho }}_p}{L_p}$$

(55)

for an arbitrary non-singular (n × n) matrix P.

Theorem 2. 

Let the conditions of Theorem 1 be satisfied. Then, the sequence $f\left(x_k\right)$, k = 0, 1, 2,…, given by the process (1)–(4), fulfills the following estimate,

$$f_k-f^{*}\le \left(f_0-f^{*}\right)(q_k^{*}{)}^k,$$

(56)

$$q_k^{*}\le 1-\frac{{\mathit{\rho }}_V^3}{2L_V^3\left[1+\frac{\mathit{\rho }n{M}_0^v}{k+1}\right]\times \left[1+\frac{n}{L{m}_0^v\left(k+1\right)}\right]},$$

(57)

where M₀^ν and m₀^ν are maximal and minimal eigenvalues of matrix$P^{-T}{H}_0{P}^{-1}$, respectively.

Proof of Theorem 2. 

Due to the identity of the form of the minimization process in the old and new coordinate systems and the connection of the initial conditions (54) to estimate the speed of the process (1)–(4), an estimate in any coordinate system can be used and, particularly, in the system $\widehat{x}=Vx$. This fact and considering (48) proves (56). □

The estimate (56) was obtained in the coordinate system selected by condition (55). If the function satisfies the relation

$${\mathit{\rho }}_V/L_V>>\mathit{\rho }/L,$$

then the advantages of QN methods, compared to the steepest descent method, are indisputable. The result (56) was obtained without the assumption of the existence of second derivatives of the function being minimized. Under such weakened conditions on the goal function given in Condition 1, QN methods converge at the rate of a geometric progression and can eliminate the background that slows down the convergence rate. The elimination is enabled by the corresponding linear transformation of coordinates.

5. Numerical Experiment

The purpose of a numerical experiment is to study the ability of quasi-Newton methods to eliminate the background that slows down the convergence rate through some linear transformation which normalizes the elongation of function level surfaces in different directions. For comparison, methods were chosen in which the background that slows down the convergence rate is active during the solution process. The gradient descent (GR) method, Hestenes–Stiefel (HS) conjugate gradient method, and the quasi-Newtonian (BFGS) method with a one-dimensional search procedure with cubic interpolation were implemented and compared.

Due to the fact that the use of QN methods is justified primarily on functions with a high degree of conditionality, where conjugate gradient methods do not work, the test functions were selected based on this position. The QN method is based on a quadratic model of a function; therefore, its local convergence rate in a certain neighborhood of the current minimum is largely determined by its efficiency in minimizing ill-conditioned quadratic functions. Therefore, studies were carried out on quadratic functions and functions of their derivatives.

If the function is twice differentiable, then the eigenvalues of the matrix of second derivatives are limited by the interval of the strong convexity and Lipschitz parameters [ρ,L]. When creating tests, we used the representation of a quadratic function and the analysis of its conditionality, relying on its eigenvalues. The test functions simulate the oscillatory nature of the second derivatives of the function.

The following function is accepted as the basic quadratic function:

$$f_1(x,[a\max ])=\frac{1}{2}{\displaystyle \sum _{i=1}^n{a}_i^{}\hspace{0.17em}{x}_i^2},\hspace{0.17em}\hspace{0.17em}{a}_i^{}=a{\max }_{}^{\frac{i-1}{n-1}}.$$

To simulate random fluctuations of second derivatives, a function $f_2$ was created based on the basic function $f_1$:

$$f_2=f_1(x,[a\max ]).$$

Its gradients were distorted randomly according to the following scheme:

$$\nabla f_2=\nabla f_1\times (1+r\mathit{\xi })\hspace{0.17em}.$$

where ξ ∈ [−1, 1] is a uniformly distributed random number and r = 0.3 is gradient distortion parameter. It should be noted that distortion of gradients significantly reduces the accuracy of a one-dimensional search, where gradients are used to estimate directional derivatives in the cubic approximation.

The point $x_0=(\mathrm{100,100},\dots ,100)$ was chosen as the starting point. The stopping criterion was as follows:

$$f(x_{}^k)-f*\le \epsilon ={10}^{-10}.$$

Minimization results are presented in

Table 1. Function f₂ minimization results. Parameter [a max] = 10³.

n	GR			HS			BFGS
	N_it	nfg	f	N_it	nfg	f	N_it	nfg	f
100	596	1337	9.2491 × 10−11	464	998	9.7246 × 10−11	193	416	9.7600 × 10−11
200	1006	2223	6.8119 × 10−11	442	953	8.8826 × 10−11	223	477	8.3226 × 10−11
300	1218	2650	3.0812 × 10−11	452	971	6.7694 × 10−11	242	522	8.9270 × 10−11
400	545	1202	2.3238 × 10−12	454	974	8.5506 × 10−11	266	579	9.1770 × 10−11
500	1110	2417	9.9534 × 10−11	465	1012	9.3773 × 10−11	247	544	8.8586 × 10−11
600	499	1109	3.4604 × 10−11	494	1071	9.6048 × 10−11	265	575	8.5970 × 10−11
700	941	2081	9.5699 × 10−11	458	994	7.8899 × 10−11	272	586	8.7071 × 10−11
800	761	1689	9.9708 × 10−11	442	963	9.1321 × 10−11	270	593	9.7620 × 10−11
900	736	1636	6.5657 × 10−11	472	1010	9.5092 × 10−11	284	626	9.0551 × 10−11
1000	944	2111	9.5688 × 10−11	435	945	9.2528 × 10−11	285	625	8.4472 × 10−11

and

Table 2. Function f₂ minimization results. Parameter [a max] = 10⁶.

n	GR			HS			BFGS
	N_it	nfg	f	N_it	nfg	f	N_it	nfg	f
100	10,001	22,163	18,623	10,001	23,266	1,380,105	452	1010	1.7175 × 10−11
200	10,001	22,146	261,315	10,001	23,274	756,738	753	1696	4.1613 × 10−11
300	10,001	22,199	0.8183	10,001	23,235	1,319,239	972	2183	9.3183 × 10−11
400	10,001	22,094	541,444	10,001	23,218	823,225	1272	2909	7.2557 × 10−11
500	10,001	22,182	3,456,399	10,001	23,250	83,606	1354	3072	8.7855 × 10−11
600	10,001	21,986	3,485,875	10,001	23,238	1,303,868	1544	3525	9.8429 × 10−11
700	10,001	22,184	1,875,235	10,001	23,297	1,016,413	1784	4066	9.8578 × 10−11
800	10,001	22,141	1,892,176	10,001	23,262	1,484,428	1830	4203	9.5674 × 10−11
900	10,001	22,239	344,032	10,001	23,187	1,368,998	2154	4912	8.0695 × 10−11
1000	10,001	22,246	51,892	10,001	23,170	1,627,625	2141	4879	8.8202 × 10−11

. As shown in the tables for the methods studied, N_it—number of iterations (one-dimensional searches along the direction); nfg—number of calls to the procedure for simultaneous calculation of a function and gradient; f—achieved value of the function.

It can be seen from Table 1 that the BFGS method is the most effective, the second is the HS method, and the last is the GR method. Table 2 shows that, unlike the HS and GR methods, the BFGS method remains operational due to the removal of the background noise that worsens the convergence rate.

6. Conclusions

An overwhelming number of iterations occur outside the extremum area when solving a specific minimization problem using a quasi-Newton method if there is no stable quadratic approximation of the objective function. This paper presents a study of the convergence rate of quasi-Newton-type methods on strongly convex functions with a Lipschitz gradient without assuming the existence of second derivatives of the goal function. In our work, the convergence of quasi-Newton methods on strongly convex functions with a Lipschitz gradient is estimated in the form of a geometric progression.

The estimate of the convergence rate includes the dependence on the strong convexity parameter, Lipschitz parameter, and the initial matrix parameter. The convergence rate is determined by the ratio of constants ρ/L, which characterizes the spread of elongation of level surfaces in different directions. The greater this ratio, the higher the convergence rate.

It is shown that an increase in the number of iterations of the method improves the indicators for estimating the convergence rate, which demonstrates the benefit of adjusting the metric matrices in the method.

The property of invariance of quasi-Newton methods with respect to a linear transformation of coordinates enables us to consider the method in a coordinate system where the ratio ρ/L is maximal, that is, the spread of elongation of level surfaces in different directions is minimal, and to obtain a conclusion about the accelerating properties of quasi-Newton methods without relying on the matrix of the second derivatives of the function.

Based on the computational experiment, we can conclude that the theoretically predicted ability of quasi-Newton methods to eliminate the background noise slowing down the convergence rate has been numerically confirmed. The research results can be applied in practice, for example, when choosing a method for training neural networks. As a suggestion for future work, the numerical experiment can be extended to other functions.

The study of the convergence rate of quasi-Newton minimization methods was developed under the assumption of the exact line search (2). One area for future research may be convergence analysis based on various inexact line search procedures.