A Three-Dimensional Subspace Algorithm Based on the Symmetry of the Approximation Model and WYL Conjugate Gradient Method

Abstract

In this paper, a three-dimensional subspace method is proposed, in which the search direction is generated by minimizing the approximation model of the objective function in a three-dimensional subspace. The approximation model of the objective function is not unique, and alternatives can be chosen between a symmetric quadratic model and a conic model by specific criteria. Moreover, the idea of a WLY conjugate gradient method is applied to characterize the change of gradient direction between adjacent iteration points. The strategy of initial stepsize and nonmonotone line search are adopted, and the global convergence of the presented algorithm is established under mild assumptions. In numerical experiments, we use a collection of 80 unconstrained optimization test problems to show the competitive performance of the presented method.

1. Introduction

Considering the following unconstrained optimization problem:

$$\underset{x\in {\Re }^n}{\min }f\left(x\right),$$

(1)

with an initial point $x_0$, the following iterative formula is often used to solve (1):

$$x_{k+1}=x_k+{\alpha }_k{d}_k,$$

(2)

where $x_k$ is the kth iteration point, ${\alpha }_k\in \Re$ is the stepsize determined by a line search procedure, and $d_k$ is the search direction obtained by special ways.

The search direction of the conjugate gradient (CG) methods is obtained by

$$d_{k+1}=-g_{k+1}+{\beta }_k{d}_k,k=0,1,...,$$

(3)

where $d_0=-g_0$, $g_{k+1}=\nabla f\left(x_{k+1}\right)$ and ${\beta }_k\in \Re$ is a scalar called the conjugate gradient parameter. Corresponding to different choices for the parameter ${\beta }_k$, various nonlinear conjugate gradient methods have been proposed. Some classical CG methods include HS method (Hestenes and Stiefel [1]), FR method (Fletcher and Reeves [2]), PRP method (Polak et al. [3]), CD method (Fletcher [4]), LS method (Liu and Storey [5]) and DY method (Dai and Yuan [6]).

Among these mentioned CG methods, the PRP, HS and LS methods share the common numerator $g_{k+1}^T{y}_k$, where $y_k=g_{k+1}-g_k$. When the step $x_{k+1}-x_k$ is small, the factor $y_k$ tends to zero, so ${\beta }_k$ becomes small and the next search direction is essentially the steepest descent direction which can avoid jamming. Hence, these methods have better numerical performance than those lacking in $g_{k+1}^T{y}_k$. However, the PRP, HS and LS methods possess weak convergence properties. Based on the PRP method, Wei et al. [7] proposed a new formula

$${\beta }_k^{WYL}=\frac{g_{k+1}^T{y}_k^{*}}{g_k^T{g}_k},$$

where $y_k^{*}=g_{k+1}-\frac{\parallel g_{k+1}\parallel }{\parallel g_k\parallel }{g}_k$ and $\parallel .\parallel$ denotes the Euclidean norm. When step $x_{k+1}-x_k$ is small, $y_k^{*}$ also tends to zero so that the WYL method can also avoid jamming. Moreover, $y_k^{*}$ can avoid the drawback: if the gradients of $x_{k+1}$ and $x_k$ are significantly different, the structure of $y_k$ makes the smaller one between $g_{k+1}$ and $g_k$ nearly insignificant so that the method cannot make full use of the information. Furthermore, the WYL method not only has nice numerical performance but also has global convergence properties. More applications and properties of $y_k^{*}$ can be found in [7,8,9,10,11,12,13].

The procedure for determining the stepsize ${\alpha }_k$ used in (2) is usually called line search, which can be classified into an exact one and an inexact one. One of the most used inexact line search is the so-called standard Wolfe line search [14]:

$$f(x_k+{\alpha }_k{d}_k)\le f\left(x_k\right)+\delta {\alpha }_k{g}_k^T{d}_k,$$

(4)

$$\nabla f{(x_k+{\alpha }_k{d}_k)}^T{d}_k\ge \sigma g_k^T{d}_k,$$

(5)

where $0<\delta <\sigma <1$. Obviously, it is a monotone procedure that seeks a suitable ${\alpha }_k$, making the function value decrease to some extent. Zhang and Hager [15] proposed a nonmonotone version (ZH line search) that modifies condition (4) to

$$f(x_k+{\alpha }_k{d}_k)\le C_k+\delta {\alpha }_k{g}_k^T{d}_k,$$

(6)

where $C_0=f\left(x_0\right)$, $Q_0=1$, $C_{k+1}$ and $Q_{k+1}$ are updated by

$$C_{k+1}=\frac{{\eta }_k{Q}_k{C}_k+f\left(x_{k+1}\right)}{Q_{k+1}},Q_{k+1}={\eta }_k{Q}_k+1,$$

(7)

where ${\eta }_k\in [{\eta }_{min},{\eta }_{max}]$ and $0\le {\eta }_{min}\le {\eta }_{max}\le 1$. The choice of ${\eta }_k$ controls the degree of nonmonotonicity. Such a line search can not only overcome some drawbacks in the monotone line search, but is particularly efficient for unconstrained problems in numerical experiments [15]. In addition to ZH line search, there are many other efficient nonmonotone line search procedures, which can be found in [16,17,18,19,20,21,22,23,24,25].

The subspace technique is one of the effective means for solving large-scale optimization problems, which is receivinging more and more attention. Yuan reviewed various subspace techniques that have been used in constructing numerical methods for solving nonlinear optimization problems in [26,27]. There are many optimization methods adopting the subspace technique, such as the limited-memory quasi-Newton method [28], the subspace trust region methods [29,30], the subspace SQP method [31], and more research can be found in [32,33,34,35,36,37,38]. Moreover, the combination between subspace technique and conjugate gradient method has been extensively studied. In the earliest research (see [39]), Yuan and Stoer computed the search direction $d_{k+1}$ by minimizing the approximation quadratic model in the two dimensional subspace spanned by $g_{k+1}$ and $s_k$, namely ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k\}$ where $s_k=x_{k+1}-x_k$, and proposed the subspace minimization conjugate gradient method (SMCG), in which $d_{k+1}$ is formed by

$$d_{k+1}=t{g}_{k+1}+\mu s_k,$$

(8)

where t and $\mu$ are undetermined parameters. Based on the above idea, Andrei [40] extended the subspace to ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k,y_k\}$ and exploited the acceleration scheme, finally presenting a three-term conjugate gradient method (TTS), in which

$$d_{k+1}=-g_{k+1}+\mu s_k+\nu y_k,$$

and $\mu$, $\nu$ are also scalar parameters. Inspired by Andrei, Yang et al. [41] changed the subspace into ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k,s_{k-1}\}$, and put forward the subspace three-term conjugate gradient method (STT). For the same subspace, Li et al. [42] added more parameters to the computation of search direction so that

$$d_{k+1}=t{g}_{k+1}+\mu s_k+\nu s_{k-1},$$

and adopted the strategy of nonmonotone line search, eventually proposing the subspace minimization conjugate gradient method with nonmonotone line search (SMCG_NLS). Huo et al. [13] combined the idea of the WYL method with subspace method, then constructed the subspace ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k,y_k^{*}\}$, in which

$$d_{k+1}=t{g}_{k+1}+\mu s_k+\nu y_k^{*},$$

and finally proposed a new three-dimensional subspace conjugate gradient method (TSCG).

On the other hand, Dai and Kou [43] also focused on the analysis of Yuan and Stoer, but they paid more attention to the estimate of the parameter ${\rho }_{k+1}=g_{k+1}^T{B}_{k+1}{g}_{k+1}$ during the calculation of $d_{k+1}$. They combined the Barzilai–Borwein [44] idea and provided some efficient Barzilai–Borwein conjugate gradient methods (BBCG). It is remarkable that the idea of BBCG to estimate ${\rho }_{k+1}$ is employed in this paper.

It is noteworthy that all of the above mentioned subspace minimization conjugate gradient methods obtain the search direction by minimizing the approximate quadratic model of objective function in the presented subspace. However, Sun and Yuan [45,46] have pointed out that, when the current iterative point is not close to the minimizer, the quadratic model may lead to a poor prediction of the minimizer if the objective function possesses strong non-quadratic behavior. Furthermore, a quadratic model does not take into account more information instead of the gradient value in the current iteration, which means that it does not have enough degrees freedom for incorporating all of the information in the iterative procedure.

Thus, the research for approximate nonquadratic model is of the essence. Up to now, many nonquadratic models have been applied to optimization problems, such as conic model, tensor model and regularization model. The conic model can incorporate more function information than the quadratic model, and its application in unconstrained optimization was first studied by Davidon [47]. A typical conic model for unconstrained optimization is

$${\varphi }_{k+1}\left(s\right)=\frac{g_{k+1}^{T}s}{1+b_{k+1}^{T}s}+\frac{1}{2}\frac{s^T{B}_{k+1}s}{{(1+b_{k+1}^{T}s)}^2},$$

which is an approximation to $f(x_k+s)-f\left(x_k\right)$, and $B_{k+1}$ is a symmetric positive definite matrix approximating to the Hessian of $f\left(x\right)$ at $x_{k+1}$ which satisfies the secant equation $B_{k+1}{s}_k=y_k$. The vector $b_{k+1}$ is normally called the horizontal vector satisfying $1+b_{k+1}^{T}s>0$. Such a conic model has been investigated by many scholars. Sorensen [48] discussed a class of conic methods called “optimization by collinear scaling” for unconstrained optimization and shown that a particular member of this algorithm class has a Q-superlinear convergence. Ariyawansa [49] modified the procedure of Sorensen [48] and established the duality between the collinear scaling DFP and BFGS methods. Sheng [50] further discussed the interpolation properties of conic model method. Di and Sun [51] proposed a trust region method for conic models to solve unconstrained optimization problems. The trust region methods based on conic model have brought about a great number of research.

Li et al. [52] paid attention to the combination of subspace method and conic model. They considered the following conic approximation model:

$${\varphi }_{k+1}\left(d\right)=\frac{g_{k+1}^{T}d}{1+b_{k+1}^{T}d}+\frac{1}{2}\frac{d^T{B}_{k+1}d}{{(1+b_{k+1}^{T}d)}^2}$$

(9)

where

$$\begin{array}{cc}\hfill b_{k+1}& =-\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}{g}_{k+1}^T{s}_k}{g}_{k+1},\hfill \\ \hfill {\gamma }_{k+1}& =\frac{-g_k^T{s}_k}{\sqrt{{\mathsf{\Delta }}_{k+1}}+f_k-f_{k-1}},\hfill \\ \hfill {\mathsf{\Delta }}_{k+1}& ={(f_k-f_{k+1})}^2-\left(g_{k+1}^T{s}_k\right)\left(g_k^T{s}_k\right).\hfill \end{array}$$

In the two-dimensional subspace ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k\}$, they minimized the above conic model to compute the search direction, and finally developed a subspace minimization conjugate method based on the conic model (SMCG_Conic). Sun et al. [53] extended the subspace to ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k,s_{k-1}\}$ and presented a three-dimensional subspace minimization conjugate gradient method based on a conic model (CONIC_CG3).

The adoption of a nonquadratic model does not mean abandoning the quadratic model; here we refer to the analysis of Yuan [54], in which a quantity $u_k$ is defined by

$$u_k=\left|\frac{2(f_k-f_{k+1}+g_{k+1}^T{s}_k)}{s_k^T{y}_k}-1\right|,$$

which shows the extent of how the objective function $f\left(x\right)$ is close to a quadratic on the line segment between $x_k$ and $x_{k+1}$. Dai et al. [55] indicate that if the following condition

$$u_k\le c_{1}or\max \{u_k,u_{k-1}\}\le c_2$$

(10)

holds, where $0<c_1<c_2$ are two small constants, then they think that $f\left(x\right)$ is very close to a quadratic on the line segment between $x_k$ and $x_{k+1}$. If the condition (10) is satisfied, then the choice of the quadratic approximation model is preferable; otherwise, the conic model is more suitable. The utilization of such a quantity can be found in [56,57].

Since the combination of subspace technique and conic model has been investigated, we come up with the idea of whether we can make further studies. SMCG_Conic and CONIC_CG3 are efficient methods that possess good theoretical and numerical results, but they only considered the two-dimensional subspace $Span\{g_{k+1},s_k\}$ and three-dimensional subspace $Span\{g_{k+1},s_k,s_{k-1}\}$, respectively. Hence, we expect to extend the application of conic model in the subspace conjugate gradient method. In addition, the WYL method with the ingredient $y_k^{*}$ has good performance both in theory and numerical experiment, and the subspace method TSCG based on quadratic model and subspace $Span\{g_{k+1},s_k,y_k^{*}\}$ also is an efficient method. In this paper, we investigates a new application of WYL method and subspace method based on the conic model by extending SMCG_Conic to subspace $Span\{g_{k+1},s_k,y_k^{*}\}$. Furthermore, we adopt the initial stepsize strategy and nonmonotone line search, under which we establish the sufficient descent property and global convergence property. Since our method is based on three-dimensional subspace and conic model, it has efficient performance in numerical experiment and can solve more problems than subspace method based on two-dimensional subspace and quadratic model.

This paper is organized as follows: in Section 2, the search directions on the subspace ${\mathsf{\Omega }}_{k+1}$ based on two different models are derived, and the criteria for how to choose the approximate model and search direction are presented. In Section 3, we obtain the stepsize by the strategies of initial stepsize and nonmonotone line search, and elaborate the generated algorithm. In Section 4, we give the proofs for some important lemmas of the search direction, then state the convergence performance of the generated algorithm under suitable assumptions. In Section 5, we compare the numerical results of our algorithm with another two methods.

2. The Search Direction

For an optimization method, the search direction attaches great importance to the iterative formula, and the computation of search direction is always the first job. Hence, the main content of this section is to construct the formula for the search direction. Since the approximation model is selected between conic model and quadratic model, the computation will be divided into two parts.

2.1. Conic Model

In this subsection, we will give the computation of the search direction in the case that conic model is used.

When (10) is violated, the conic model is more suitable to approximate the objective function, so we compute the search direction by minimizing the conic model in subspace ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k,y_k^{*}\}$. We consider the subproblem

$$\underset{d\in {\mathsf{\Omega }}_{k+1}}{\min }{\varphi }_{k+1}\left(d\right),$$

(11)

where ${\varphi }_{k+1}\left(d\right)$ is the same as (9).

The discussion will be divided into three parts depending on the three different dimensions of the subspace ${\mathsf{\Omega }}_{k+1}$.

Situation 1:

$dim\left({\mathsf{\Omega }}_{k+1}\right)=3.$

In this situation, the search direction is

$$d_{k+1}=t{g}_{k+1}+\mu s_k+\nu y_k^{*}.$$

(12)

By substituting (12) into (11) and using the secant equation, the problem (11) turns into

$$\underset{(t,\mu ,\nu )}{\min }{\varphi }_{k+1}\left(u\right)=\frac{a^{T}u}{1+c^{T}u}+\frac{1}{2}\frac{u^T{A}_{k+1}u}{{(1+c^{T}u)}^2},$$

(13)

where

$$A_{k+1}=\left(\begin{array}{ccc}{\rho }_{k+1}& g_{k+1}^T{y}_k& {\omega }_k\\ g_{k+1}^T{y}_k& s_k^T{y}_k& y_k^T{y}_k^{*}\\ {\omega }_k& y_k^T{y}_k^{*}& {\tau }_k\end{array}\right),a=\left(\begin{array}{c}\parallel g_{k+1}{\parallel }^2\\ g_{k+1}^T{s}_k\\ g_{k+1}^T{y}_k^{*}\end{array}\right),c=\left(\begin{array}{c}{b}_{k+1}^T{g}_{k+1}\\ b_{k+1}^T{s}_k\\ b_{k+1}^T{y}_k^{*}\end{array}\right),u=\left(\begin{array}{c}t\\ \mu \\ \nu \end{array}\right),$$

and ${\rho }_{k+1}=g_{k+1}^T{B}_{k+1}{g}_{k+1},{\omega }_k=g_{k+1}^T{B}_{k+1}{y}_k^{*},{\tau }_k={y_k^{*}}^T{B}_{k+1}{y}_k^{*}.$

To solve (13), we first figure out the zero solution of the function

$$\nabla {\varphi }_{k+1}\left(u\right)=\frac{1}{1+c^{T}u}\left(I-\frac{c{u}^T}{1+c^{T}u}\right)\left(a+\frac{A_{k+1}u}{1+c^{T}u}\right),$$

and obviously $I-\frac{c{u}^T}{1+c^{T}u}$ is invertible, so the problem is reduced to $a+\frac{A_{k+1}u}{1+c^{T}u}=0$. After calculation, we can deduce that, when $A_{k+1}$ is positive definite and $1+c^T{A}_{k+1}^{-1}a\ne 0$, the solution of (13) is

$$u_{k+1}=\frac{-A_{k+1}^{-1}a}{1+c^T{A}_{k+1}^{-1}a}.$$

(14)

By using the relationship $A_{k+1}^{-1}=\frac{A_{k+1}^{*}}{|A_{k+1}|}$, where

$$\begin{array}{c}\hfill A_{k+1}^{*}=\left(\begin{array}{ccc}X& {\theta }_1& {\theta }_2\\ {\theta }_1& \theta & {\theta }_3\\ {\theta }_2& {\theta }_3& Y\end{array}\right)\end{array}$$

(15)

is the adjugate matrix of $A_{k+1}$, and $X,{\theta }_1,{\theta }_2,\theta ,{\theta }_3,Y$ denote the cofactor of $A_{k+1}^{11},A_{k+1}^{21}$, $A_{k+1}^{31},A_{k+1}^{22},A_{k+1}^{32},A_{k+1}^{33}$, respectively, in which $A_{k+1}^{ij}$ denotes the ij-th element of $A_{k+1}$ and

$$\begin{array}{cc}\hfill X& =\left(s_k^T{y}_k\right){\tau }_k-{\left(y_k^T{y}_k^{*}\right)}^2,\hfill \\ \hfill {\theta }_1& =\left(y_k^T{y}_k^{*}\right){\omega }_k-\left(g_{k+1}^T{y}_k\right){\tau }_k,\hfill \\ \hfill {\theta }_2& =\left(g_{k+1}^T{y}_k\right)\left(y_k^T{y}_k^{*}\right)-\left(s_k^T{y}_k\right){\omega }_k,\hfill \\ \hfill \theta & ={\rho }_{k+1}{\tau }_k-{\omega }_k^2,\hfill \\ \hfill {\theta }_3& =\left(g_{k+1}^T{y}_k\right){\omega }_k-{\rho }_{k+1}\left(y_k^T{y}_k^{*}\right),\hfill \\ \hfill Y& ={\rho }_{k+1}\left(s_k^T{y}_k\right)-{\left(g_{k+1}^T{y}_k\right)}^2,\hfill \end{array}$$

then we finally obtain the solution of (13),

$$u_{k+1}=\left(\begin{array}{c}{t}_{k+1}\\ {\mu }_{k+1}\\ {\nu }_{k+1}\end{array}\right)=-\frac{1}{D_{k+1}}\left(\begin{array}{ccc}X& {\theta }_1& {\theta }_2\\ {\theta }_1& \theta & {\theta }_3\\ {\theta }_2& {\theta }_3& Y\end{array}\right)\left(\begin{array}{c}\parallel g_{k+1}{\parallel }^2\\ g_{k+1}^T{s}_k\\ g_{k+1}^T{y}_k^{*}\end{array}\right)=\frac{1}{D_{k+1}}\left(\begin{array}{c}-q_1\\ -q_2\\ -q_3\end{array}\right),$$

(16)

where

$$\begin{array}{cc}\hfill |A_{k+1}|& ={\rho }_{k+1}X+{\theta }_1{g}_{k+1}^T{y}_k+{\theta }_2{\omega }_k,\hfill \\ \hfill D_{k+1}& =|A_{k+1}|+q_1{b}_{k+1}^T{g}_{k+1}+q_2{b}_{k+1}^T{s}_k+q_3{b}_{k+1}^T{y}_k^{*},\hfill \\ \hfill q_1& =X\parallel g_{k+1}{\parallel }^2+{\theta }_1{g}_{k+1}^T{s}_k+{\theta }_2{g}_{k+1}^T{y}_k^{*},\hfill \\ \hfill q_2& ={\theta }_1{\parallel g_{k+1}\parallel }^2+\theta g_{k+1}^T{s}_k+{\theta }_3{g}_{k+1}^T{y}_k^{*},\hfill \\ \hfill q_3& ={\theta }_2{\parallel g_{k+1}\parallel }^2+{\theta }_3{g}_{k+1}^T{s}_k+Y{g}_{k+1}^T{y}_k^{*}.\hfill \end{array}$$

So the solution of (11) in the three-dimensional subspace ${\mathsf{\Omega }}_{k+1}=Span\{g_{k+1},s_k,y_k^{*}\}$ is

$$d_{k+1}=\left(\begin{array}{ccc}{g}_{k+1}& s_k& y_k^{*}\end{array}\right)u_{k+1}.$$

During evaluation, it is important to find appropriate ways to estimate ${\rho }_{k+1},{\omega }_k$ and ${\tau }_k$ in order to avoid matrix–vector multiplication and improve efficiency. Yuan and Stoer [39] proposed two ways to calculate such quantities containing $B_{k+1}$, one of which is to obtain $B_{k+1}$ by using the scaled memoryless BFGS formula. The approach of Dai and Kou [43] is to combine the Barzilai–Borwein [44] idea by approximating the Hessian by $(1/{\alpha }_{k+1}^{B{B}_1})I$ or $(1/{\alpha }_{k+1}^{B{B}_2})I$, where

$$\begin{array}{c}\hfill {\alpha }_{k+1}^{B{B}_1}=\frac{\parallel s_k{\parallel }^2}{s_k^T{y}_k},{\alpha }_{k+1}^{B{B}_2}=\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}.\end{array}$$

For ${\omega }_k$, we utilize the memoryless BFGS formula to obtain $B_{k+1}$ so that

$$\begin{array}{cc}\hfill {\omega }_k& =g_{k+1}^T\left(I+\frac{y_k{y}_k^T}{s_k^T{y}_k}-\frac{s_k{s}_k^T}{s_k^T{s}_k}\right)y_k^{*}\hfill \\ \hfill & =g_{k+1}^T{y}_k^{*}+\frac{g_{k+1}^T{y}_k{y}_k^T{y}_k^{*}}{s_k^T{y}_k}-\frac{g_{k+1}^T{s}_k{s}_k^T{y}_k^{*}}{s_k^T{s}_k}.\hfill \end{array}$$

(17)

Then for ${\tau }_k$, we combine the idea of [42,43], and estimate ${\tau }_k$ by

$${\tau }_k={\zeta }_k\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}{\parallel y_k^{*}\parallel }^2,$$

(18)

where

$${\zeta }_k=\left\{\begin{array}{cc}\max \{0.9{\zeta }_{k-1},1.2\},\hfill & \mathrm{if}{\alpha }_k>1.0,\hfill \\ \min \{1.1{\zeta }_{k-1},1.75\},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

and ${\zeta }_0=1.5$. It is obvious that ${\zeta }_k\in [1.2,1.75]$.

For the estimate of ${\rho }_{k+1}$, we adopt the idea of Li et al. [42], because it can guarantee some good properties. Firstly, according to the analysis regarding (14), the positive definitiveness of $A_{k+1}$ is essential, which requires $|A_{k+1}|>0$. It follows that

$${\rho }_{k+1}>\frac{-{\theta }_1{g}_{k+1}^T{y}_k-{\theta }_2{\omega }_k}{X}.$$

(19)

By setting $X=m_k{s}_k^T{y}_k{\tau }_k$ with $m_k\triangleq 1-\frac{{\left(y_k^T{y}_k^{*}\right)}^2}{s_k^T{y}_k{\tau }_k}$, and combining (15), we have

$${\rho }_{k+1}>\left[\frac{{\left(g_{k+1}^T{y}_k\right)}^2}{s_k^T{y}_k}+\frac{{\omega }_k^2}{{\tau }_k}-2\frac{g_{k+1}^T{y}_k{\omega }_k{y}_k^T{y}_k^{*}}{s_k^T{y}_k{\tau }_k}\right]/m_k\triangleq n_k,$$

(20)

if and only if $m_k$ is positive, which can be guaranteed by (18). In addition, the positive definitiveness of $A_{k+1}$ also requires that its first and second order leading principal minors are positive, and it means

$${\rho }_{k+1}>\frac{{\left(g_{k+1}^T{y}_k\right)}^2}{s_k^T{y}_k}.$$

(21)

Secondly, $D_{k+1}>0$ is also a necessary condition to keep the sufficient descent property, from which follows

$${\rho }_{k+1}>\frac{S_k}{s_k^T{y}_k{\tau }_k}/M_k,$$

(22)

if $M_k>0$, where

$$\begin{array}{cc}\hfill S_k=-{\theta }_1{g}_{k+1}^T{y}_k-{\theta }_2{\omega }_k& -b_{k+1}^T{g}_{k+1}(X\parallel g_{k+1}{\parallel }^2+{\theta }_1{g}_{k+1}^T{s}_k+{\theta }_2{g}_{k+1}^T{y}_k^{*})\hfill \\ \hfill & -b_{k+1}^T{s}_k({\theta }_1\parallel g_{k+1}{\parallel }^2-{\omega }_k^2{g}_{k+1}^T{s}_k+{\omega }_k{g}_{k+1}^T{y}_k{g}_{k+1}^T{y}_k^{*})\hfill \\ \hfill & -b_{k+1}^T{y}_k^{*}({\theta }_2\parallel g_{k+1}{\parallel }^2+{\omega }_k{g}_{k+1}^T{y}_k{g}_{k+1}^T{s}_k-{\left(g_{k+1}^T{y}_k\right)}^2{g}_{k+1}^T{y}_k^{*}),\hfill \end{array}$$

and

$$M_k=1-\frac{{\left(y_k^T{y}_k^{*}\right)}^2}{s_k^T{y}_k{\tau }_k}+\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\left[2\frac{g_{k+1}^T{y}_k^{*}{y}_k^T{y}_k^{*}}{s_k^T{y}_k{\tau }_k}-\frac{g_{k+1}^T{s}_k}{s_k^T{y}_k}-\frac{{\left(g_{k+1}^T{y}_k^{*}\right)}^2}{g_{k+1}^T{s}_k{\tau }_k}\right].$$

Likewise, we define the right-hand side of (22) as $N_k$, i.e., $N_k\triangleq \frac{S_k}{s_k^T{y}_k{\tau }_k}/M_k$.

After the above discussion, we can estimate ${\rho }_{k+1}$ as follows,

$${\rho }_{k+1}={\zeta }_k\max \{K,N_k,n_k\},$$

(23)

where $K=K_1{\parallel g_{k+1}\parallel }^2$,

$$K_1=\max \left\{\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k},\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}\right\},$$

and ${\zeta }_k$ is the same as that in (18). One of the purposes of K is to ensure (21).

Before computing $d_{k+1}$ by (12) and (16), we should verify the following conditions:

$${\mathsf{\Delta }}_{k+1}\ge 0,$$

(24)

$$M_k\ge {\rho }_0,$$

(25)

$${\xi }_1\le \frac{s_k^T{y}_k}{\parallel s_k{\parallel }^2}\le \frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}\le {\xi }_2,$$

(26)

$$\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}\le {\xi }_3,$$

(27)

$$\frac{|A_{k+1}|}{s_k^T{y}_k{\tau }_k{\rho }_{k+1}}\ge {\xi }_5,$$

(28)

where ${\rho }_0\in (0,1)$ and ${\xi }_1,{\xi }_2,{\xi }_3,{\xi }_5$ are positive constants. The expressions (24) and (25) are fundamental premises of the conic model (5) and relation (22), respectively. On the basis of the Barzilai–Borwein [44] idea, (26) might indicate the suitable condition numbers of the approximation Hessian matrix. Moreover, (27) is vital to guarantee the descent property of the search direction. As for (28), obviously it makes $A_{k+1}$ more positive definite, and is also important for establishing the sufficient descent property of the search direction.

For convenience, we call (24)–(28) the first conic model conditions. Hence, if the first conic model conditions hold, then we compute the search direction by (12) and (16).

Situation 2:

$dim\left({\mathsf{\Omega }}_{k+1}\right)=2$ or 1.

Li et al. [52] have made a deep study of the subspace conjugate gradient method based on the conic model in this case. Here we refer to their works. When $dim\left({\mathsf{\Omega }}_{k+1}\right)=2$, the search direction is formed by

$$d_{k+1}=t{g}_{k+1}+\mu s_k,$$

which is the same as (8). The solution of t and $\mu$ is

$$\left(\begin{array}{c}{t}_{k+1}\\ {\mu }_{k+1}\end{array}\right)=\frac{1}{{\overline{D}}_{k+1}}\left(\begin{array}{c}-c_1\\ -c_2\end{array}\right),$$

(29)

which is the same as (13) in [52].

In this case, conditions (24) and (26) are still essential, but before obtaining $d_{k+1}$ by (8) and (29), the following conditions are also necessary:

$$\begin{array}{cc}\hfill {\overline{m}}_k& \ge {\overline{\rho }}_0,\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill \left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}& \le {\xi }_3,\mathrm{if}\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}{g}_{k+1}^T{s}_k}<0,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \frac{\parallel g_{k+1}{\parallel }^2\parallel y_k\parallel \parallel s_k\parallel }{{\left(g_{k+1}^T{s}_k\right)}^2}& \le {\xi }_4,\mathrm{if}\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}{g}_{k+1}^T{s}_k}\ge 0,\hfill \end{array}$$

(32)

where ${\overline{\rho }}_0\in (0,1)$, ${\xi }_4$ is a positive constant and ${\xi }_3$ is identical to that in (27). We call (24), (26) and (30)–(32) the second conic model conditions. If the first conic model conditions do not all hold but the second conic model conditions hold, we compute the search direction by (8) and (29).

If the first and second conic model conditions are violated but (33) and (34) hold,

$${\vartheta }_1\le \frac{s_k^T{y}_k}{\parallel s_k{\parallel }^2},$$

(33)

$$\frac{|g_{k+1}^T{y}_k{g}_{k+1}^T{d}_k|}{d_k^T{y}_k{\parallel g_{k+1}\parallel }^2}\le {\vartheta }_4,$$

(34)

where ${\vartheta }_4\in [0,1)$ and ${\vartheta }_1$ are positive constants, the HS direction is considered, i.e.,

$$d_{k+1}=-g_{k+1}+{\beta }_k^{HS}{d}_k.$$

(35)

We call (33) and (34) the HS conditions.

Therefore, there are two choices of search direction when $dim\left({\mathsf{\Omega }}_{k+1}\right)=2$: one is to compute $d_{k+1}$ by (8) and (29) if the second conic model conditions hold; the other is to compute $d_{k+1}$ by (35) if the HS conditions hold. Otherwise, we use the negative gradient direction $-g_{k+1}$ as our search direction.

2.2. Quadratic Model

The content of this subsection is to calculate the search direction in the case that the objective function is approximated by a quadratic model.

When the condition (10) holds, the quadratic approximation model seems to perform well in approximating the objective function, so we consider the subproblem based on the quadratic approximation model as follows:

$$\underset{d\in {\mathsf{\Omega }}_{k+1}}{\min }{\psi }_{k+1}\left(d\right)=g_{k+1}^{T}d+\frac{1}{2}{d}^T{B}_{k+1}d,$$

(36)

and related work can be found in [13]. The corresponding results can be applied to this situation. We state the concrete results in the following.

Firstly, there are some necessary conditions:

$${\vartheta }_1\le \frac{s_k^T{y}_k}{\parallel s_k{\parallel }^2}\le \frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}\le {\vartheta }_2,$$

(37)

$${\vartheta }_1\le \frac{{\tau }_k}{\parallel y_k^{*}{\parallel }^2},\frac{4\parallel y_k{\parallel }^4{\parallel y_k^{*}\parallel }^2}{{\left(s_k^T{y}_k\right)}^2{\tau }_k}\le {\vartheta }_2,$$

(38)

$$\frac{\parallel s_k{\parallel }^2}{\parallel g_{k+1}{\parallel }^2}\ge {\vartheta }_3,$$

(39)

we call (37)–(39) the quadratic model conditions.

When the quadratic model conditions hold,

$$d_{k+1}=t_{k+1}^1{g}_{k+1}+{\mu }_{k+1}^1{s}_k+{\nu }_{k+1}^1{y}_k^{*},$$

(40)

where

$$\begin{array}{c}\hfill \left(\begin{array}{c}{t}_{k+1}^1\\ {\mu }_{k+1}^1\\ {\nu }_{k+1}^1\end{array}\right)=-\frac{1}{|A_{k+1}|}\left(\begin{array}{ccc}X& {\theta }_1& {\theta }_2\\ {\theta }_1& \theta & {\theta }_3\\ {\theta }_2& {\theta }_3& Y\end{array}\right)\left(\begin{array}{c}\parallel g_{k+1}{\parallel }^2\\ g_{k+1}^T{s}_k\\ g_{k+1}^T{y}_k^{*}\end{array}\right).\end{array}$$

When the quadratic model conditions do not all hold but (37) holds,

$$d_{k+1}=t_{k+1}^2{g}_{k+1}+{\mu }_{k+1}^2{s}_k,$$

(41)

where

$$\begin{array}{c}\hfill \left(\begin{array}{c}{t}_{k+1}^2\\ {\mu }_{k+1}^2\end{array}\right)=\frac{1}{|{\overline{A}}_{k+1}|}\left(\begin{array}{c}{g}_{k+1}^T{y}_k{g}_{k+1}^T{s}_k-{\parallel g_{k+1}\parallel }^2{s}_k^T{y}_k\\ \parallel g_{k+1}{\parallel }^2{g}_{k+1}^T{y}_k-{\rho }_{k+1}{g}_{k+1}^T{s}_k\end{array}\right).\end{array}$$

$${\overline{A}}_{k+1}=\left(\begin{array}{cc}{\rho }_{k+1}& g_{k+1}^T{y}_k\\ g_{k+1}^T{y}_k& s_k^T{y}_k\end{array}\right).$$

When (37) does not holds but the HS conditions hold,

$$d_{k+1}=-g_{k+1}+{\beta }_k^{HS}{d}_k.$$

Otherwise, $d_{k+1}=-g_{k+1}.$

To sum up, the generated directions possess several forms as follows:

 

when the approximation conic model is selected, i.e., (10) is violated,

-

$d_{k+1}$ is calculated by (12) and (16), if the first conic model conditions hold,

-

$d_{k+1}$ is calculated by (8) and (29), if the second conic model conditions hold,

-

$d_{k+1}$ is calculated by (35), if the HS conditions hold;

 

when the approximation quadratic model is selected, i.e., (10) holds,

-

$d_{k+1}$ is calculated by (12) and (40), if the quadratic model conditions hold,

-

$d_{k+1}$ is calculated by (8) and (41), if only (37) holds,

-

$d_{k+1}$ is calculated by (35), if the HS conditions hold;

 

otherwise, $d_{k+1}=-g_{k+1}$.

3. The Stepsize and Algorithm

In this section, we will present the obtainment of another essential ingredient in the iterative formula of optimization method, i.e., stepsize. It requires two steps: the first is the choice of initial stepsize, and the second is the line search procedure. After these discussion, then we can propose our whole algorithm.

3.1. Strategy for the Initial Stepsize

Firstly, we show how to choose the initial stepsize.

It is generally acknowledged that the initial stepsize is of great significance for an optimization method, especially for the conjugate gradient method. For Newton and quasi-Newton methods, the choice of the initial trial stepsize may always be unit step ${\alpha }^0=1$. However, it is different for methods that do not produce well-scaled search directions, such as the steepest descent or the conjugate gradient methods. Thus, it is significant to make a reasonable initial guess of the stepsize by considering the current information about the objective function and algorithm for such methods [58,59]. Many strategies of the initial stepsize have been proposed which can be found in the achievements of Hager and Zhang [60], Nocedal and Wright [58], Dai and Kou [19], Liu and Liu [57].

In the strategy of [19], Dai and Kou presented a condition,

$$\frac{|{\phi }_{k+1}\left({\alpha }_{k+1}^0\right)-{\phi }_{k+1}\left(0\right)|}{{\epsilon }_1+\left|{\phi }_{k+1}\left(0\right)\right|}\le {\epsilon }_2,$$

(42)

where ${\epsilon }_1,{\epsilon }_2$ are small positive constants, ${\alpha }_{k+1}^0$ denotes the initial trial stepsize, and ${\phi }_{k+1}\left(\alpha \right)=f(x_{k+1}+\alpha d_{k+1})$. If condition (42) holds, it implies that the points $x_{k+1}+{\alpha }_{k+1}^0{d}_{k+1}$ and $x_{k+1}$ are not far away from each other, so it is reasonable to use the minimizer of the quadratic interpolation function for ${\phi }_{k+1}\left(0\right)$, ${\phi }_{k+1}^{{}^{\prime }}\left(0\right)$ and ${\phi }_{k+1}\left({\alpha }_{k+1}^0\right)$ as the new initial stepsize, where ${\phi }_{k+1}^{{}^{\prime }}\left(0\right)$ denotes the first derivative of ${\phi }_{k+1}\left(0\right)$.

In this paper, the selection of the initial stepsize has two parts, depending on whether the negative gradient direction is adopted or not, and is presented via modification of that in Liu and Liu [56].

When the search direction is negative gradient direction, according to the analysis of Liu and Liu [57], it is desirable to take the initial trial stepsize by

$${\overline{\overline{\alpha }}}_{k+1}=\left\{\begin{array}{cc}\max \left\{\min \right\{s_k^T{y}_k/\parallel y_k{\parallel }^2,{\lambda }_{max}\},{\lambda }_{min}\},\hfill & \mathrm{if}{g}_{k+1}^T{s}_k>0,\hfill \\ \max \left\{\min \right\{\parallel s_k{\parallel }^2/s_k^T{y}_k,{\lambda }_{max}\},{\lambda }_{min}\},\hfill & \mathrm{if}{g}_{k+1}^T{s}_k\le 0,\hfill \end{array}\right.$$

(43)

where ${\lambda }_{min}$ and ${\lambda }_{max}$ are two positive constants controlling the initial stepsize within the interval $[{\lambda }_{min},{\lambda }_{max}]$ which is preferable in numerical experiments.

Andrei [61] thinks that, the more accurate the step length is, the faster convergence a conjugate gradient algorithm has. It therefore makes sense to verify if the initial trial stepsize ${\overline{\overline{\alpha }}}_{k+1}$ satisfies (42) or not. If so and $d_k\ne -g_k$, $\parallel g_{k+1}{\parallel }^2\le 1$, we update the initial stepsize by

$${\tilde{\alpha }}_{k+1}=\max \{\min \{{\tilde{\tilde{\alpha }}}_{k+1},{\lambda }_{max}\},{\lambda }_{min}\},$$

where

$${\tilde{\tilde{\alpha }}}_{k+1}=\min q({\phi }_{k+1}\left(0\right),{\phi }_{k+1}^{{}^{\prime }}\left(0\right),{\phi }_{k+1}\left({\overline{\overline{\alpha }}}_{k+1}\right)),$$

where $q({\phi }_{k+1}\left(0\right),{\phi }_{k+1}^{{}^{\prime }}\left(0\right),{\phi }_{k+1}\left({\overline{\overline{\alpha }}}_{k+1}\right))$ denotes the quadratic interpolation function for ${\phi }_{k+1}\left(0\right)$, ${\phi }_{k+1}^{{}^{\prime }}\left(0\right)$ and ${\phi }_{k+1}\left({\overline{\overline{\alpha }}}_{k+1}\right)$.

Therefore, the initial stepsize for the negative gradient direction is

$${\alpha }_{k+1}^0=\left\{\begin{array}{cc}{\tilde{\alpha }}_{k+1},\hfill & \mathrm{if}\left(42\right)\mathrm{holds},d_k\ne -g_k,{\parallel g_{k+1}\parallel }^2<1\mathrm{and}{\tilde{\tilde{\alpha }}}_{k+1}>0,\hfill \\ {\overline{\overline{\alpha }}}_{k+1},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(44)

When it comes to the search direction that is not negative gradient direction, the similarity of the calculation between our direction and that in quasi-Newton methods implies that the unit stepsize ${\alpha }_{k+1}^0=1$ might be a reasonable initial trial stepsize. Again we figure out the minimizer of the quadratic interpolation function

$${\overline{\alpha }}_{k+1}=\min q({\phi }_{k+1}\left(0\right),{\phi }_{k+1}^{{}^{\prime }}\left(0\right),{\phi }_{k+1}\left(1\right)).$$

If ${\alpha }_{k+1}^0=1$ satisfies (42) and ${\overline{\alpha }}_{k+1}>0$, we update the initial stepsize by

$${\widehat{\alpha }}_{k+1}=\max \{\min \{{\overline{\alpha }}_{k+1},{\lambda }_{max}\},{\lambda }_{min}\}.$$

Therefore, the initial stepsize for the search direction except negative gradient direction is

$${\alpha }_{k+1}^0=\left\{\begin{array}{cc}{\widehat{\alpha }}_{k+1},\hfill & \mathrm{if}\left(42\right)\mathrm{holds}\mathrm{and}{\overline{\alpha }}_{k+1}>0,\hfill \\ 1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(45)

3.2. The Nonmonotone Line Search

Secondly, we introduce the line search used in our algorithm.

For the variable ${\eta }_k$ in ZH line search, Zhang and Hager [15] proved that if ${\eta }_{max}=1$, then the generated sequence $\left\{x_k\right\}$ only has the property that

$$\underset{k\to \infty }{\lim \inf }\parallel g_k\parallel =0.$$

Liu and Liu [57] presented a formula of ${\eta }_k$,

$${\eta }_k=\left\{\begin{array}{c}c,mod(k,n)=n-1,\hfill \\ 1,mod(k,n)\ne n-1,\hfill \end{array}\right.$$

where $0<c<1$ and $\mathrm{mod}(k,n)$ denotes the residue for k modulo n, and it resulted in a better convergence.

Referring to the above study, Li et al. [52] and Sun et al. [53] made some modifications so that the improved line search is more appropriate for their algorithms.

In order to gain the decent convergence result and performance, this paper modifies the improved ZH line search used in [53]. To be specific, we set

$$\begin{array}{cc}\hfill C_{k+1}=& \left\{\begin{array}{cc}{f}_{k+1}+\min \{1,0.9(C_k-f_{k+1})\},\hfill & k<5,\hfill \\ {\eta }_k{Q}_k{C}_k+f\left(x_{k+1}\right)]/Q_{k+1},\hfill & k\ge 5,\hfill \end{array}\right.\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill Q_{k+1}=& \left\{\begin{array}{cc}6,\hfill & k=4,\hfill \\ {\eta }_k{Q}_k+1,\hfill & k\ge 5,\hfill \end{array}\right.\hfill \end{array}$$

(47)

where

$${\eta }_k=\left\{\begin{array}{c}\eta ,mod(k,n)=0,\hfill \\ 1,mod(k,n)\ne 0,\hfill \end{array}\right.$$

(48)

where $\eta =0.999$.

3.3. Algorithm

In this subsection, we will detail our new three-dimensional subspace method based on conic model.

First, we incorporate a special restart technique proposed by Dai and Kou [19]. They defined a quantity

$$r_k=\frac{2(f_{k+1}-f_k)}{{\alpha }_k(g_k^T{d}_k+g_{k+1}^T{d}_k)},$$

where $f_{k+1}={\phi }_k\left({\alpha }_k\right)$ and $f_k={\phi }_k\left(0\right)$. If $r_k$ is close to 1, they think that the line search function ${\phi }_k$ is close to some quadratic function. According to their analysis, the exact approach of this quantity is that if there are continuously many iterations such that $r_k$ is close to 1, we restart the algorithm with the negative gradient direction. In addition, if the number of the iterations since the last restart reaches the MaxRestart threshold, we also restart out algorithm.

The details of the three-dimensional subspace conjugate gradient method based on conic model are given in the next algorithm, which is Algorithm 1.

Algorithm 1 WYL_TSCO

Require: initial point $x_0$, initial stepsize ${\alpha }_0^0$, positive constants $ϵ,{ϵ}_1,{ϵ}_2,0<\delta <\sigma <1,$ ${\xi }_1,{\xi }_2,{\xi }_3,{\xi }_4,{\xi }_5,{\vartheta }_1,{\vartheta }_2,{\vartheta }_3,{\vartheta }_4,$ ${\rho }_0,{\overline{\rho }}_0\in (0,1),{\lambda }_1,{\lambda }_2,{\lambda }_{min},{\lambda }_{max}$

Ensure: optimal $x^{*}$

1: Set MaxRestart: = 4n, IterRestart: = 0, IterQuad: = 0, MinQuad: = 3, Numnongrad: = 0, $C_0=f_0,d_0=-g_0$ and k: = 0. 2: If $\parallel g_0{\parallel }_{\infty }\le ϵ$, stop. 3: Calculate the stepsize ${\alpha }_k$ by (5) and (6) with ${\alpha }_k^0$. 4: Update $x_{k+1}=x_k+{\alpha }_k{d}_k$. If $\parallel g_{k+1}{\parallel }_{\infty }\le ϵ$, stop; otherwise, let IterRestart: = IterRestart +1. If $|r_k-1|\le {10}^{-8}$ or $|f_{k+1}-f_k-0.5(g_{k+1}^T{s}_k+g_k^T{s}_k)|\le 6.0\times {10}^{-8}$, IterQuad: = IterQuad+1; else, IterQuad: = 0. 5: Calculate the search direction $d_k$. If Numnongrad = MaxRestart, or (IterQuad = MinQuad and IterRestart ≠IterQuad), go to 5.6; else if the condition (10) holds, go to 5.1; else, go to 5.3. 5.1: If the quadratic model conditions hold, compute $d_{k+1}$ by (12) and (40), set Numnongrad: = Numnongrad+1; else, go to 5.2. 5.2: If condition (37) holds, compute $d_{k+1}$ by (8) and (41), set Numnongrad: = Numnongrad+1; else, go to 5.5. 5.3: If the first conic model conditions hold, compute $d_{k+1}$ by (12) and (16), set Numnongrad: = Numnongrad+1; else, go to 5.4. 5.4: If the second conic model conditions, compute $d_{k+1}$ by (8) and (29), set Numnongrad: = Numnongrad+1; else, go to 5.5. 5.5: If the HS conditions hold, compute $d_{k+1}$ by (35), set Numnongrad: = Numnongrad +1; else, go to 5.6. 5.6: Compute $d_{k+1}=-g_{k+1}$, set Numnongrad: = 0 and IterRestart: = 0. 6: If $d_{k+1}=-g_{k+1}$, calculate ${\alpha }_{k+1}^0$ by (44); otherwise, calculate ${\alpha }_{k+1}^0$ by (45). 7: Update $Q_{k+1}$ and $C_{k+1}$ by (46)–(48). 8: Set $k:=k+1$, go to Step 3.

4. Properties of the Proposed Algorithm

The theoretical nature is an important criterion to determine the performance of an algorithm. In this section, we will give some properties of WYL_TSCO, including the sufficient descent property and the global convergence.

4.1. Some Lemmas

This subsection will make a deep discussion on our algorithm and prove some properties of the presented directions before establishing the global convergence. At first, we propose two assumptions as follows:

Assumption A1.

The objective function $f\left(x\right)$ is continuously differentiable and bounded from below on ${\Re }^n$.

Assumption A2.

The gradient function $g\left(x\right)$ is Lipschitz continuous on the level set $D=\{x\in {\Re }^n:f\left(x\right)\le f\left(x_0\right)\}$, which means that there exists a positive constant $L>0$ satisfying

$$\parallel g\left(x\right)-g\left(y\right)\parallel \le L\parallel x-y\parallel ,\forall x,y\in D,$$

which implies that$\parallel y_k\parallel \le L\parallel s_k\parallel$.

With these assumptions, we can prove some properties of the new directions in the following.

Lemma 1.

For the search direction $d_{k+1}$ calculated by WYL_TSCO, there exists a constant ${\varsigma }_1>0$ such that

$$g_{k+1}^T{d}_{k+1}\le -{\varsigma }_1{\parallel g_{k+1}\parallel }^2.$$

(49)

Proof. 

We will discuss in several parts based on different situations and different approximation models.

Case I: if the negative direction is adopted, i.e., $d_{k+1}=-g_{k+1}$, then

$$g_{k+1}^T{d}_{k+1}=-\parallel g_{k+1}{\parallel }^2\le -\frac{1}{2}{\parallel g_{k+1}\parallel }^2.$$

Case II: if $d_{k+1}$ is determined by (35), i.e., the HS direction, combining (34), then we have

$$\begin{array}{cc}\hfill g_{k+1}^T{d}_{k+1}& =-\parallel g_{k+1}{\parallel }^2+{\beta }_k^{HS}{g}_{k+1}^T{d}_k\hfill \\ \hfill & \le -\parallel g_{k+1}{\parallel }^2+\frac{|g_{k+1}^T{y}_k{g}_{k+1}^T{d}_k|}{d_k^T{y}_k}\hfill \\ \hfill & \le -\parallel g_{k+1}{\parallel }^2+{\vartheta }_4{\parallel g_{k+1}\parallel }^2\hfill \\ \hfill & =-(1-{\vartheta }_4){\parallel g_{k+1}\parallel }^2.\hfill \end{array}$$

Case III (conic): if $d_{k+1}$ is calculated by (8) and (29), Li, Liu and Liu [52] have proved that

$$g_{k+1}^T{d}_{k+1}\le -\overline{\varsigma }{\parallel g_{k+1}\parallel }^2,$$

where

$$\overline{\varsigma }=\min \left\{\frac{{\overline{\rho }}_0}{(8-6{\overline{\rho }}_0\max \{{\xi }_2,{\xi }_4\})},\frac{1}{6\overline{c}}\right\},$$

and $\overline{c}$ is a positive constant.

Case IV (conic): when $d_{k+1}$ is obtained by (12) and (16), we have

$$\begin{array}{cc}\hfill g_{k+1}^T{d}_{k+1}& ={\left(\begin{array}{c}\parallel g_{k+1}{\parallel }^2\\ g_{k+1}^T{s}_k\\ g_{k+1}^T{y}_k^{*}\end{array}\right)}^T\left(\begin{array}{c}{t}_{k+1}\\ {\mu }_{k+1}\\ {\nu }_{k+1}\end{array}\right)\hfill \\ \hfill & =-\frac{1}{D_{k+1}}{\left(\begin{array}{c}\parallel g_{k+1}{\parallel }^2\\ g_{k+1}^T{s}_k\\ g_{k+1}^T{y}_k^{*}\end{array}\right)}^T\left(\begin{array}{ccc}X& {\theta }_1& {\theta }_2\\ {\theta }_1& \theta & {\theta }_3\\ {\theta }_2& {\theta }_3& Y\end{array}\right)\left(\begin{array}{c}\parallel g_{k+1}{\parallel }^2\\ g_{k+1}^T{s}_k\\ g_{k+1}^T{y}_k^{*}\end{array}\right)\hfill \\ \hfill & =-\frac{\parallel g_{k+1}{\parallel }^4}{D_{k+1}}h(x,y),\hfill \end{array}$$

where $x\triangleq \frac{g_{k+1}^T{y}_k^{*}}{\parallel g_{k+1}{\parallel }^2}$, $y\triangleq \frac{g_{k+1}^T{s}_k}{\parallel g_{k+1}{\parallel }^2}$. Moreover, $h(x,y)$ is a binary quadratic function of x and y which can be expressed as

$$h(x,y)=Y{x}^2+2{\theta }_{3}xy+\theta y^2+2{\theta }_{2}x+2{\theta }_{1}y+X.$$

It is easy to acquire the Hessian of $h(x,Y)$

$$H_h=\left(\begin{array}{cc}2Y& 2{\theta }_3\\ 2{\theta }_3& 2\theta \end{array}\right),$$

we have $Y>0$ because $A_{k+1}$ is positive definite, and the determinant of $H_h$

$$4Y\theta -4{\theta }_3^2=4{\rho }_{k+1}\left|A_{k+1}\right|,$$

is also positive, so $h(x,y)$ has a minimizer, that is

$$h{(x,y)}_{min}=\frac{|A_{k+1}|}{{\rho }_{k+1}}.$$

Therefore, we can derive

$$g_{k+1}^T{d}_{k+1}\le -\frac{\parallel g_{k+1}{\parallel }^4}{D_{k+1}}h{(x,y)}_{min}\le -\frac{|A_{k+1}|}{D_{k+1}{\rho }_{k+1}}{\parallel g_{k+1}\parallel }^4.$$

(50)

Because ${\rho }_{k+1},D_{k+1}$ and $|A_{k+1}|$ are all positive, we just need to prove that $\frac{|A_{k+1}|}{D_{k+1}{\rho }_{k+1}}{\parallel g_{k+1}\parallel }^2$ has a lower bound. Since $D_{k+1}$ contains ${\rho }_{k+1}$, we first prove that ${\rho }_{k+1}$ has an upper bound; that is the upper bound of $N_k,n_k$ and K.

For $N_k$, we use the Cauchy inequality and can obtain

$$\begin{array}{cc}\hfill |N_k|& =\left|\frac{S_k}{s_k^T{y}_k{\tau }_k}\right|/M_k\hfill \\ \hfill & \le \left[\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\left(\frac{\parallel g_{k+1}{\parallel }^4}{|g_{k+1}^T{s}_k|}+4\frac{\parallel g_{k+1}{\parallel }^4\parallel y_k{\parallel }^2{\parallel y_k^{*}\parallel }^2}{|g_{k+1}^T{s}_k|s_k^T{y}_k{\tau }_k}+4\frac{\parallel g_{k+1}{\parallel }^2\parallel y_k\parallel \parallel y_k^{*}\parallel |{\omega }_k|}{s_k^T{y}_k{\tau }_k}\right.\right.\hfill \\ \hfill & \left.+2\frac{\parallel g_{k+1}{\parallel }^3\parallel y_k\parallel }{s_k^T{y}_k}+2\frac{\parallel g_{k+1}{\parallel }^3\parallel y_k^{*}\parallel |{\omega }_k|}{|g_{k+1}^T{s}_k|{\tau }_k}+\frac{\parallel g_{k+1}\parallel \parallel s_k\parallel {\omega }_k^2}{s_k^T{y}_k{\tau }_k}\right)\hfill \\ \hfill & \left.+\frac{\parallel g_{k+1}{\parallel }^2{\parallel y_k\parallel }^2}{s_k^T{y}_k}+\frac{{\omega }_k^2}{{\tau }_k}+2\frac{\parallel g_{k+1}\parallel \parallel y_k{\parallel }^2\parallel y_k^{*}\parallel |{\omega }_k|}{s_k^T{y}_k{\tau }_k}\right]/M_k,\hfill \end{array}$$

combining (17), (18) and using the Cauchy inequality again, we have

$$\begin{array}{c}\hfill |N_k|\le \frac{1}{M_k}{\parallel g_{k+1}\parallel }^2\left[\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\left(\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}+4\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}+4\frac{2+{\xi }_2}{{\zeta }_k}\frac{\parallel g_{k+1}\parallel }{\parallel y_k\parallel }\right.\right.\\ \hfill \left.+2\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel }{s_k^T{y}_k}+2\frac{2+{\xi }_2}{{\zeta }_k}\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}+\frac{{(2+{\xi }_2)}^2}{{\zeta }_k}\frac{\parallel g_{k+1}\parallel \parallel s_k\parallel }{\parallel y_k{\parallel }^2}\right)\\ \hfill \left.+\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}+\frac{{(2+{\xi }_2)}^2}{{\zeta }_k}\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}+2\frac{2+{\xi }_2}{{\zeta }_k}\right].\end{array}$$

Under condition (26), we have $\frac{\parallel y_k\parallel \parallel s_k\parallel }{s_k^T{y}_k}\le \sqrt{\frac{{\xi }_2}{{\xi }_1}}$, $\frac{\parallel s_k\parallel }{\parallel y_k\parallel }\le \frac{1}{{\xi }_1}$, and

$$\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel }{s_k^T{y}_k}=\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel |g_{k+1}^T{s}_k|}{s_k^T{y}_k\left|g_{k+1}^T{s}_k\right|}\le \frac{\parallel y_k\parallel \parallel s_k\parallel }{s_k^T{y}_k}\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|},$$

and in the same way,

$$\frac{\parallel g_{k+1}\parallel }{\parallel y_k\parallel }\le \frac{\parallel s_k\parallel }{\parallel y_k\parallel }\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}.$$

Note that ${\zeta }_k\ge 1$, the above inequality of $|N_k|$ can be simplified to

$$\begin{array}{c}\hfill |N_k|\le \frac{1}{M_k}{\parallel g_{k+1}\parallel }^2\left[\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\left(5+4(2+{\xi }_2)\frac{\parallel s_k\parallel }{\parallel y_k\parallel }+2\frac{\parallel y_k\parallel \parallel s_k\parallel }{s_k^T{y}_k}+2(2+{\xi }_2)\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}\right.\right.\\ \hfill \left.\left.+{(2+{\xi }_2)}^2\frac{\parallel s_k{\parallel }^2}{\parallel y_k{\parallel }^2}\right)\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}+{\left(1+(2+{\xi }_2)\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}\right)}^2\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}\right].\end{array}$$

Utilizing conditions (25)–(27) and the expression of K, we can obtain the upper bound of $N_k$,

$$\begin{array}{cc}\hfill |N_k|& \le \frac{\parallel g_{k+1}{\parallel }^2}{M_k}\left[\left(\frac{{(2+{\xi }_2)}^2}{{\xi }_1^2}+6\frac{(2+{\xi }_2)}{{\xi }_1}+2\sqrt{\frac{{\xi }_2}{{\xi }_1}}+5\right)\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}^T{s}_k|}\right.\hfill \\ \hfill & \left.+{\left(\frac{2+{\xi }_2}{{\xi }_1}+1\right)}^2\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}\right]\hfill \\ \hfill & \le \frac{1}{{\rho }_0}\left(2\frac{{(2+{\xi }_2)}^2}{{\xi }_1^2}+8\frac{(2+{\xi }_2)}{{\xi }_1}+2\sqrt{\frac{{\xi }_2}{{\xi }_1}}+6\right)K_1{\parallel g_{k+1}\parallel }^2\hfill \\ \hfill & =\left(2\frac{{(2+{\xi }_2)}^2}{{\xi }_1^2}+8\frac{(2+{\xi }_2)}{{\xi }_1}+2\sqrt{\frac{{\xi }_2}{{\xi }_1}}+6\right)\frac{K}{{\rho }_0}.\hfill \end{array}$$

Next, the upper bound of $n_k$ is acquired by the same procedure,

$$\begin{array}{cc}\hfill |n_k|& \le \frac{1}{m_k}{\parallel g_{k+1}\parallel }^2\left(\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}+\frac{{(2+{\xi }_2)}^2}{{\zeta }_k}\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}+2\frac{(2+{\xi }_2)}{{\zeta }_k}\right)\hfill \\ \hfill & \le \frac{1}{m_k}{\parallel g_{k+1}\parallel }^2{\left(1+(2+{\xi }_2)\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}\right)}^2\frac{\parallel y_k{\parallel }^2}{s_k^T{y}_k}\hfill \\ \hfill & \le \frac{1}{1-\frac{1}{{\zeta }_k}}\frac{{({\xi }_1+{\xi }_2+2)}^2}{{\xi }_1^2}{K}_1{\parallel g_{k+1}\parallel }^2\hfill \\ \hfill & \le 6\frac{{({\xi }_1+{\xi }_2+2)}^2}{{\xi }_1^2}K.\hfill \end{array}$$

For $K_1$, it is easy to know that

$$K_1\le \max \{{\xi }_2,{\xi }_3\}.$$

Through the above discussion, now we can give the upper bound of ${\rho }_{k+1}$,

$$\begin{array}{cc}\hfill {\rho }_{k+1}& ={\zeta }_k\max \{K,N_k,n_k\}\hfill \\ \hfill & \le 2\max \left\{\left(2\frac{{(2+{\xi }_2)}^2}{{\xi }_1^2}+8\frac{(2+{\xi }_2)}{{\xi }_1}+2\sqrt{\frac{{\xi }_2}{{\xi }_1}}+6\right)/{\rho }_0,6\frac{{({\xi }_1+{\xi }_2+2)}^2}{{\xi }_1^2}\right\}{K}_1{\parallel g_{k+1}\parallel }^2\hfill \\ \hfill & \le L_0{\parallel g_{k+1}\parallel }^2,\hfill \end{array}$$

where

$$L_0\triangleq 2\max \left\{\left(2\frac{{(2+{\xi }_2)}^2}{{\xi }_1^2}+8\frac{(2+{\xi }_2)}{{\xi }_1}+2\sqrt{\frac{{\xi }_2}{{\xi }_1}}+6\right)/{\rho }_0,6\frac{{({\xi }_1+{\xi }_2+2)}^2}{{\xi }_1^2}\right\}\max \{{\xi }_2,{\xi }_3\}.$$

Since we have found the upper bound of ${\rho }_{k+1}$, then we now consider $D_{k+1}$. According to (22), $D_k$ can be expressed as

$$D_{k+1}=s_k^T{y}_k{\tau }_k{\rho }_{k+1}{M}_k-S_k=s_k^T{y}_k{\tau }_k(M_k{\rho }_{k+1}-M_k{N}_k).$$

(51)

Using the formula of $M_k$ and the upper bound of ${\rho }_{k+1}$, we have

$$\begin{array}{c}\hfill |D_{k+1}|=D_{k+1}\le {\parallel g_{k+1}\parallel }^2{s}_k^T{y}_k{\tau }_k\left[L_0+L_0\frac{\parallel y_k{\parallel }^2{\parallel y_k^{*}\parallel }^2}{s_k^T{y}_k{\tau }_k}+L_0\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\left(2\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel \parallel y_k^{*}{\parallel }^2}{s_k^T{y}_k{\tau }_k}\right.\right.\\ \hfill \left.\left.+\frac{\parallel g_{k+1}\parallel \parallel s_k\parallel }{s_k^T{y}_k}+\frac{\parallel g_{k+1}{\parallel }^2{\parallel y_k^{*}\parallel }^2}{|g_{k+1}{s}_k|{\tau }_k}\right)+\frac{M_k\left|N_k\right|}{\parallel g_{k+1}{\parallel }^2}\right].\end{array}$$

Note that

$$\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel \parallel y_k^{*}{\parallel }^2}{s_k^T{y}_k{\tau }_k}=\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel \parallel y_k^{*}{\parallel }^2\left|g_{k+1}{s}_k\right|}{s_k^T{y}_k{\tau }_k\left|g_{k+1}{s}_k\right|}\le \frac{\parallel s_k\parallel }{\parallel y_k\parallel }\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}{s}_k|},$$

$$\frac{\parallel g_{k+1}\parallel \parallel s_k\parallel }{s_k^T{y}_k}=\frac{\parallel g_{k+1}\parallel \parallel s_k\parallel |g_{k+1}{s}_k|}{s_k^T{y}_k\left|g_{k+1}{s}_k\right|}\le \frac{\parallel s_k{\parallel }^2}{s_k^T{y}_k}\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}{s}_k|},$$

so the above inequality of $D_{k+1}$ can be simplified to

$$\begin{array}{c}\hfill D_{k+1}\le {\parallel g_{k+1}\parallel }^2{s}_k^T{y}_k{\tau }_k\left[2L_0+L_0\left(2\frac{\parallel s_k\parallel }{\parallel y_k\parallel }+\frac{\parallel s_k{\parallel }^2}{s_k^T{y}_k}+\frac{s_k^T{y}_k}{\parallel y_k{\parallel }^2}\right)\left|\frac{1-{\gamma }_{k+1}}{{\gamma }_{k+1}}\right|\frac{\parallel g_{k+1}{\parallel }^2}{|g_{k+1}{s}_k|}\right.\\ \hfill \left.+\left(4\frac{{\xi }_2}{{\xi }_1}+6\sqrt{\frac{{\xi }_2}{{\xi }_1}}+16\right)K_1\right].\end{array}$$

Finally, using (26) and (27), we can obtain

$$\begin{array}{cc}\hfill D_{k+1}& \le \parallel g_{k+1}{\parallel }^2{s}_k^T{y}_k{\tau }_k\left[2L_0+\left(4\frac{L_0}{{\xi }_1}+4\frac{{\xi }_2}{{\xi }_1}+6\sqrt{\frac{{\xi }_2}{{\xi }_1}}+16\right)K_1\right]\hfill \\ \hfill & \le \parallel g_{k+1}{\parallel }^2{s}_k^T{y}_k{\tau }_k\left[2L_0+\left(4\frac{L_0}{{\xi }_1}+4\frac{{\xi }_2}{{\xi }_1}+6\sqrt{\frac{{\xi }_2}{{\xi }_1}}+16\right)\max \{{\xi }_2,{\xi }_3\}\right]\hfill \\ \hfill & \le L_1{\parallel g_{k+1}\parallel }^2{s}_k^T{y}_k{\tau }_k,\hfill \end{array}$$

(52)

where

$$L_1\triangleq \left[2L_0+\left(4\frac{L_0}{{\xi }_1}+4\frac{{\xi }_2}{{\xi }_1}+6\sqrt{\frac{{\xi }_2}{{\xi }_1}}+16\right)\max \{{\xi }_2,{\xi }_3\}\right].$$

With (28), (50) and (52), the sufficient descent property of $d_{k+1}$ under this case can be established by

$$g_{k+1}^T{d}_{k+1}\le -\frac{{\xi }_5}{L_1}{\parallel g_{k+1}\parallel }^2.$$

Case V (quadratic): when $d_{k+1}$ is generated by (8) and (41), the proof can be found in [43], in which Dai and Kou proved that

$$g_{k+1}^T{d}_{k+1}\le -\frac{1}{2{\vartheta }_2}{\parallel g_{k+1}\parallel }^2.$$

(53)

Next, if $d_{k+1}$ is generated by (12) and (40), Yang et al. [13] have drawn that

$$g_{k+1}^T{d}_{k+1}\le -\frac{1}{16{\vartheta }_2}{\parallel g_{k+1}\parallel }^2.$$

(54)

Combining (54) with (53), we finally have

$$g_{k+1}^T{d}_{k+1}\le -\frac{1}{16{\vartheta }_2}{\parallel g_{k+1}\parallel }^2.$$

After the above discussion, we can prove that there indeed exists a constant ${\varsigma }_1$ such that

$$g_{k+1}^T{d}_{k+1}\le -{\varsigma }_1{\parallel g_{k+1}\parallel }^2,$$

where

$${\varsigma }_1=\min \left\{\overline{\varsigma },\frac{1}{2},1-{\vartheta }_4,\frac{{\xi }_5}{L_1},\frac{1}{16{\vartheta }_2}\right\}.$$

The proof is complete.      □

Lemma 2.

Assume that f satisfies Assumption 2. If the search direction $d_{k+1}$ is calculated by WYL_TSCO, then there exists a constant ${\varsigma }_2>0$ such that

$$\parallel d_{k+1}\parallel \le {\varsigma }_2\parallel g_{k+1}\parallel .$$

(55)

Proof. 

Similar to Lemma 1, the proof is divided into several cases.

Case I: if $d_{k+1}$ is calculated by negative gradient, then there certainly holds

$$\parallel d_{k+1}\parallel \le \parallel g_{k+1}\parallel .$$

Case II: if the search direction is the HS direction, according to Assumption 2 and the condition (33), we have

$$\begin{array}{cc}\hfill \parallel d_{k+1}\parallel & =\parallel -g_{k+1}+{\beta }_k^{HS}{d}_k\parallel \hfill \\ \hfill & \le \parallel g_{k+1}\parallel +\frac{\parallel g_{k+1}\parallel \parallel y_k\parallel \parallel d_k\parallel }{d_k^T{y}_k}\hfill \\ \hfill & \le \left(1+\frac{L}{{\vartheta }_1}\right)\parallel g_{k+1}\parallel .\hfill \end{array}$$

Case III (conic): if $d_{k+1}$ is calculated by (8) and (29), from [52] we conclude that

$$\parallel d_{k+1}\parallel \le \left(\frac{10{\xi }_2+5{\xi }_1+5m}{{\overline{\rho }}_0{\xi }_1^2}\right)\parallel g_{k+1}\parallel ,$$

where $m=2n_0\overline{\xi }/{\overline{\rho }}_0$ with

$$n_0=\max \left\{4-3{\overline{\rho }}_0,1+\frac{2{\xi }_2}{{\xi }_1}\right\},\overline{\xi }=\max \{{\xi }_2,{\xi }_3,{\xi }_4\}.$$

Case IV (conic): if $d_{k+1}$ is formed by (12) and (16), we have

$$\begin{array}{cc}\hfill \parallel d_{k+1}\parallel & =\parallel t_{k+1}{g}_{k+1}+{\mu }_{k+1}{s}_k+{\nu }_{k+1}{y}_k^{*}\parallel \hfill \\ \hfill & \le \frac{1}{D_{k+1}}\left(\right|q_1|\parallel g_{k+1}\parallel +|q_2|\parallel s_k\parallel +|q_3|\parallel y_k^{*}\parallel ),\hfill \end{array}$$

(56)

so in order to prove Lemma 2 in this case, we first need to obtain a lower bound of $D_{k+1}$. Combining (23), (25) and (51), and the value range of ${\zeta }_k$, we can derive

$$\begin{array}{cc}\hfill D_{k+1}& =s_k^T{y}_k{\tau }_k{M}_k({\zeta }_k\max \{K,N_k,n_k\}-N_k)\hfill \\ \hfill & \ge s_k^T{y}_k{\tau }_k{\rho }_0(\frac{6}{5}\max \{K,N_k,n_k\}-N_k)\hfill \\ \hfill & \ge s_k^T{y}_k{\tau }_k{\rho }_0\frac{K}{5}.\hfill \end{array}$$

Then using the above inequality, we can transform (56) into

$$\begin{array}{cc}\hfill \parallel d_{k+1}\parallel & \le \frac{5}{{\rho }_{0}K{s}_k^T{y}_k{\tau }_k}\left(\right|q_1|\parallel g_{k+1}\parallel +|q_2|\parallel s_k\parallel +|q_3|\parallel y_k^{*}\parallel )\hfill \\ \hfill & \le \frac{5}{{\rho }_{0}K}\parallel g_{k+1}\parallel \left[\parallel g_{k+1}{\parallel }^2+4\frac{\parallel g_{k+1}{\parallel }^2\parallel y_k{\parallel }^2{\parallel y_k^{*}\parallel }^2}{s_k^T{y}_k{\tau }_k}+2\frac{\parallel g_{k+1}{\parallel }^2\parallel s_k\parallel \parallel y_k\parallel }{s_k^T{y}_k}\right.\hfill \\ \hfill & +4\frac{\parallel g_{k+1}\parallel \parallel s_k\parallel \parallel y_k\parallel \parallel y_k^{*}\parallel |{\omega }_k|}{s_k^T{y}_k{\tau }_k}+2\frac{\parallel g_{k+1}\parallel \parallel y_k^{*}\parallel |{\omega }_k|}{{\tau }_k}+\frac{{\rho }_{k+1}{\parallel s_k\parallel }^2}{s_k^T{y}_k}\hfill \\ \hfill & \left.+\frac{\parallel s_k{\parallel }^2{\omega }_k^2}{s_k^T{y}_k{\tau }_k}+2\frac{{\rho }_{k+1}\parallel s_k\parallel \parallel y_k\parallel \parallel y_k^{*}{\parallel }^2}{s_k^T{y}_k{\tau }_k}+\frac{{\rho }_{k+1}{\parallel y_k^{*}\parallel }^2}{{\tau }_k}\right]\hfill \\ \hfill & \le \frac{5\parallel g_{k+1}\parallel }{{\rho }_0}\frac{\parallel g_{k+1}{\parallel }^2}{K}\left(\frac{{(2+{\xi }_2)}^2}{{\xi }_1^2}+6\frac{(2+{\xi }_2)}{{\xi }_1}+4\frac{M_0}{{\xi }_1}+2\sqrt{\frac{{\xi }_2}{{\xi }_1}}+5\right)\hfill \\ \hfill & \le \frac{\parallel g_{k+1}\parallel }{{\rho }_0{K}_1}\left(\frac{25{\xi }_1^2+5{\xi }_2^2+30{\xi }_1{\xi }_2+10{\xi }_1\sqrt{{\xi }_1{\xi }_2}+(60+20M_0){\xi }_1+20{\xi }_2+20}{{\xi }_1^2}\right).\hfill \end{array}$$

Since (26) implies that $K_1\ge {\xi }_1$, we finally obtain an upper bound of $d_{k+1}$, that is

$$\begin{array}{c}\hfill \parallel d_{k+1}\parallel \le \left(\frac{25{\xi }_1^2+5{\xi }_2^2+30{\xi }_1{\xi }_2+10{\xi }_1\sqrt{{\xi }_1{\xi }_2}+(60+20M_0){\xi }_1+20{\xi }_2+20}{{\rho }_0{\xi }_1^3}\right)\parallel g_{k+1}\parallel .\end{array}$$

For convenience, we define

$$L_2\triangleq \left(\frac{25{\xi }_1^2+5{\xi }_2^2+30{\xi }_1{\xi }_2+10{\xi }_1\sqrt{{\xi }_1{\xi }_2}+(60+20M_0){\xi }_1+20{\xi }_2+20}{{\rho }_0{\xi }_1^3}\right),$$

namely, $\parallel d_{k+1}\parallel \le L_2\parallel g_{k+1}\parallel .$

Case V (quadratic): if $d_{k+1}$ is formed by (8) and (41), it is similar to Lemma 4 in Li, Liu and Liu [42]. According to their proof, we can deduce

$$\parallel d_{k+1}\parallel \le \frac{20}{{\vartheta }_1}\parallel g_{k+1}\parallel .$$

If $d_{k+1}$ is calculated by (12) and (40), it follows from [13] that

$$\parallel d_{k+1}\parallel \le \frac{750}{{\vartheta }_1}\parallel g_{k+1}\parallel .$$

Since ${\vartheta }_1>0$, we finally have

$$\parallel d_{k+1}\parallel \le \frac{750}{{\vartheta }_1}\parallel g_{k+1}\parallel .$$

According to the above analysis, the proof of Lemma 2 is completed by setting

$${\varsigma }_2=\max \left\{\left(\frac{10{\xi }_2+5{\xi }_1+5m}{{\overline{\rho }}_0{\xi }_1^2}\right),1,1+\frac{L}{{\vartheta }_1},L_2,\frac{750}{{\vartheta }_1}\right\}.$$

□

4.2. Convergence Analysis

In this subsection, we will give the global convergence of the presented algorithm for general functions.

Lemma 3.

Suppose ${\alpha }_k$ satisfies the line search conditions (5) and (6), and f satisfies Assumption 2, then

$${\alpha }_k\ge \frac{(1-\sigma )|g_k^T{d}_k|}{L\parallel d_k{\parallel }^2}.$$

(57)

Proof. 

According to the line search condition (5), we can easily obtain

$$(\sigma -1)g_k^T{d}_k\le {(g_{k+1}^T-g_k)}^T{d}_k=y_k^T{d}_k\le \parallel y_k\parallel \parallel d_k\parallel \le L\parallel s_k\parallel \parallel d_k\parallel ={\alpha }_{k}L{\parallel d_k\parallel }^2,$$

because $\sigma -1<0$ and $g_k^T{d}_k<0$, it is obvious that

$${\alpha }_k\ge \frac{(1-\sigma )|g_k^T{d}_k|}{L\parallel d_k{\parallel }^2}.$$

□

Theorem 1.

Suppose the objective function f satisfies Assumptions 1 and 2, and sequence $\left\{x_k\right\}$ is generated by WYL_TSCO, then we have

$$\underset{k\to \infty }{\lim }\parallel g_k\parallel =0.$$

(58)

Proof. 

According to (57), (49) and (55), it follows

$$\delta {\alpha }_k{g}_k^T{d}_k\le -\frac{(1-\sigma )\delta }{L}\frac{{\left(g_k^T{d}_k\right)}^2}{\parallel d_k{\parallel }^2}\le -\frac{(1-\sigma )\delta {\varsigma }_1^2}{L{\varsigma }_2}\parallel g_k{\parallel }^2=-H{\parallel g_k\parallel }^2,$$

where $H\triangleq \frac{(1-\sigma )\delta {\varsigma }_1^2}{L{\varsigma }_2}$, then combining (6), we have

$$f_{k+1}\le C_k+\delta {\alpha }_k{g}_k^T{d}_k\le C_k-H{\parallel g_k\parallel }^2.$$

(59)

With the reasonable assumptions that $Q_0=1$, $Q_1=2$, $Q_2=3$, $Q_3=4$, $Q_4=5$ and ${\eta }_0={\eta }_1={\eta }_2={\eta }_3={\eta }_4=1$ according to the update formulas (7) and (48), we can acquire that

$$Q_{k+1}=1+\sum _{j=0}^k\prod _{i=0}^j{\eta }_{k-i},k=0,1,2...$$

Combining (48), we can derive the general formula of $Q_{k+1}$,

$$Q_{k+1}=\left\{\begin{array}{cc}{\displaystyle 1+{\eta }^{k/n}+n\sum _{i=1}^{k/n}{\eta }^i,}\hfill & mod(k,n)=0,\hfill \\ {\displaystyle 1+\mathrm{mod}(k,n)+{\eta }^{\lfloor k/n\rfloor }+n\sum _{i=1}^{\lfloor k/n\rfloor }{\eta }^i,}\hfill & mod(k,n)\ne 0,\hfill \end{array}\right.$$

where $\lfloor ·\rfloor$ denotes the floor function. Thus, it is easy to obtain an upper bound of $Q_{k+1}$,

$$\begin{array}{cc}\hfill Q_{k+1}& \le 1+\mathrm{mod}(k,n)+{\eta }^{\lfloor k/n\rfloor +1}+n\sum _{i=1}^{\lfloor k/n\rfloor +1}{\eta }^i\hfill \\ \hfill & \le 1+n+(n+1)\sum _{i=1}^{\lfloor k/n\rfloor +1}{\eta }^i\hfill \\ \hfill & \le 1+n+(n+1)\sum _{i=1}^{k+1}{\eta }^i\hfill \\ \hfill & \le (1+n)\sum _{i=0}^{k+1}{\eta }^i\hfill \\ \hfill & \le \frac{1+n}{1-\eta }\hfill \\ \hfill & =W,\hfill \end{array}$$

(60)

where $W=\frac{1+n}{1-\eta }$.

When $k\ge 5$, combining (7), (59) and (60), we obtain

$$C_{k+1}=C_k+\frac{f_{k+1}-C_k}{Q_{k+1}}\le C_k-\frac{H}{W}{\parallel g_k\parallel }^2,$$

which means

$$\frac{H}{W}{\parallel g_k\parallel }^2\le C_k-C_{k+1}.$$

(61)

When $k<5$, (6) and (46) imply that

$$C_{k+1}\le f_{k+1}+0.9(C_k-f_{k+1})=C_k+0.1(f_{k+1}-C_k)<C_k.$$

Hence $C_k$ is monotonically decreasing. According to Lemma 1.1 in [15], we have $f_{k+1}\le C_{k+1}$ for each $k\ge 5$, and (46) indicates that $f_{k+1}\le C_{k+1}$ for each $k<5$. Thus, with the assumption that f is bounded below, $C_k$ is certainly bounded from below.

Summing up the above analysis and (61), we can finally obtain that

$$\sum _{k=0}^{\infty }\frac{H}{W}{\parallel g_k\parallel }^2<\infty ,$$

which implies

$$\underset{k\to \infty }{\lim }{\parallel g_k\parallel }^2=0.$$

The proof is completed.      □

5. Numerical Results

In this section, the results of the numerical experiments are shown below. The unconstrained test functions were taken from [62] with the given initial points. To prove the efficiency of the proposed WYL_TSCO algorithm, we compare its numerical performance with SMCG_Conic and TSCG. The performance profile proposed by Dolan and Moré [63] is used to evaluate the performance of these methods. The dimension of the test functions is 10,000. All the programs were written in C code.

SMCG_Conic is also a subspace minimization conjugate gradient algorithm based on the conic model, and is a pioneering one that combines the subspace technique with a conic model to seek the search direction. The numerical experiments in [52] show that the performance of SMCG_Conic is very efficient. Since the biggest difference between WYL_TSCO and SMCG_Conic is the dimension of the used subspace, the comparison between WYL_TSCO and SMCG_Conic can not only reflect the high efficiency of our algorithm, but also reveal the influence on the numerical result generated by the change of dimension for the adopted subspace in the subspace minimization conjugate gradient algorithm.

As for TSCG, it is an efficient subspace minimization conjugate gradient method, and it successfully applies the idea of subspace method and WYL method which are employed in our algorithm as well. However, TSCG only uses the quadratic model to approximate the objective function. Thus, the comparison between the numerical performance of WYL_TSCO and TSCG can reveal the influence of the approximation model in the subspace method.

For the initial stepsize of the first iteration, we adopt the adaptive strategy used in [42]. The other parameters of WYL_TSCO are selected as follows.

$ϵ={10}^{-6},{ϵ}_1={10}^{-3},{ϵ}_2={10}^{-4},\delta =0.001,\sigma =0.9999,{\lambda }_{min}={10}^{-30},{\lambda }_{max}={10}^{30},$

${\xi }_1=0.15\times {10}^2,{\xi }_2=8.5\times {10}^4,{\xi }_3=4\times {10}^8,{\xi }_4=6.5\times {10}^7,{\xi }_5=0.1,{\rho }_0=0.3,$

${\overline{\rho }}_0=0.9,{\vartheta }_1={10}^{-7},{\vartheta }_2={10}^4,{\vartheta }_3={10}^{-2},{\vartheta }_4={10}^{-4},{\lambda }_1={10}^{-7},{\lambda }_2=0.05.$

SMCG_Conic and TSCG use the original parameters in their paper respectively. In addition to stopping when the stopping criterion $\parallel g_k{\parallel }_{\infty }\le \epsilon$ holds, the algorithm also stops when the number of iterations exceeds 200,000.

For the 80 test problems from [62], WYL_TSCO successfully solves 100% of them, while TSCG and SMCG_Conic both successfully solve 76 problems. We compare the numerical results of problems that are solved successfully by all the three algorithms, the number of which are 72. The numerical results of the three algorithms are presented in the following figures.

Figure 1 depicts the performance based on the number of iterations for the three methods. It shows that the performance of WYL_TSCO is similar to that of TSCG, but is a little inferior to SMCG_Conic’s only when $\tau <1.31$. However, WYL_TSCO outperforms TSCG and SMCG_Conic when $\tau >1.31$.

In Figure 2, we observe that WYL_TSCO and TSCG outperforms SMCG_Conic on the number of function evaluations, and TSCG is slightly ahead of WYL_TSCG only in the case of $\tau <1.5$.

Similar to Figure 1, Figure 3 illustrates that WYL_TSCO is at a little disadvantage compared with SMCG_Conic only when $\tau <1.318$, but is competitive with TSCG.

For the CPU time, Figure 4 shows that WYL_TSCO has an improvement on TSCG and SMCG_Conic, which shows the high efficiency of WYL_TSCO.

For the 80 test problems, the numerical results show that WYL_TSCO and TSCG may perform a little worse than SMCG_Conic when $\tau$ is very small, but they overall have significant improvements over SMCG_Conic, and WYL_TSCO also is superior to TSCG for almost all $\tau$. Moreover, WYL_TSCO can solve more problems than TSCG and SMCG_Conic, which might show the advantage of the three-dimensional subspace method and conic model. To sum up, the WYL_TSCO is an efficient algorithm for solving the unconstrained optimization problem.

6. Conclusions

Combining the idea of WYL method, this paper proposes a three-dimensional subspace method based on conic models for unconstrained optimization (WYL_TSCO). The sufficient descent property of the search direction and the global convergence of this method are obtained under some suitable assumptions. The selection of an approximation model depends on whether certain criterions are satisfied. Furthermore, the strategies of initial stepsize and nonmonotone line search are exploited which are beneficial to the convergence and efficiency. According to the numerical results and theoretical analysis, WYL_TSCO is very competitive and promising.