Comparative Analysis of Accelerated Models for Solving Unconstrained Optimization Problems with Application of Khan’s Hybrid Rule

Abstract

In this paper, we follow a chronological development of gradient descent methods and its accelerated variants later on. We specifically emphasise some contemporary approaches within this research field. Accordingly, a constructive overview over the class of hybrid accelerated models derived from the three-term hybridization process proposed by Khan is presented. Extensive numerical test results illustrate the performance profiles of hybrid and non-hybrid versions of chosen accelerated gradient models regarding the number of iterations, CPU time, and number of function evaluation metrics. Favorable outcomes justify this hybrid approach as an accepted method in developing new efficient optimization schemes.

1. Class of Accelerated Gradient Descent Methods and Its Benefits

Many contemporary scientific, engineering, medical and problems from various other research areas, are closely related to mathematical optimization theory. Among all others, the unconstrained optimization problems are the most frequently considered [1,2,3,4,5,6,7,8]. Owing to the duality principle, an optimization problem may be viewed as a minimization problem. Unconstrained minimization problems can simply be stated as finding

$$minf\left(x\right),x\in R^n,$$

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is an objective function, that is solved by the general iteration:

$$x_{k+1}=x_k+t_k{d}_k.$$

(1)

In (1), $x_k$ presents the current iterative point, $x_{k+1}$ is the next one; the positive iterative step length value is denoted by $t_k$ while $d_k$ stands for the k-th search direction vector. As can be observed, two main elements that measure the efficiency and robustness of the iterative rule (1) are the adequately calculated iterative step size $t_k$ and the properly chosen iterative search direction $d_k$. Since we are dealing with minimization problems, it is a natural choice to define the search direction so that it fulfils the descent condition, i.e.,

$$g_k^T{d}_k<0,$$

(2)

where $g_k$ is the gradient of f at the point $x_k$. Apart from that, we use standard notations for the gradient and the Hessian of the objective function f:

$$g\left(x\right)=▽f\left(x\right),G\left(x\right)={▽}^{2}f\left(x\right),g_k=▽f\left(x_k\right),G_k={▽}^{2}f\left(x_k\right).$$

(3)

Following condition (2), it is easy to conclude that the $d_{\equiv }-g_k$ produces the most certain descent direction, known as the gradient descent direction. Iteration (1) that includes the gradient descent direction is known as the gradient descent method (or GD method)

$$x_{k+1}=x_k-t_k{g}_k.$$

(4)

Step length parameter $t_k$ in iterations (1) and (4), is determined either by the exact or by some of the inexact line search procedures. Using the exact line search technique, iterative step length value $t_k$ is computed by solving the following minimization task:

$$f(x_k+t_k{d}_k)=minf(x_k+t{d}_k),t>0.$$

(5)

It is clear that solving the previous minimization problem in each iterative value presents a time- and resource-consuming task regarding CPU time requirements and the number of required objective function evaluations. For this reason, most contemporary optimization methods use the inexact line search algorithms to calculate iterative step size instead of the exact line search procedure. The convergence properties of line search methods for unconstrained optimization are specifically examined in [9]. Some of the frequently used inexact line search algorithms are weak and strong Wolfe’s algorithms [10], Backtracking algorithm proposed in [11] with Armijo’s rule [12], etc.:

• Weak Wolfe’s line search: 

  $$f(x_k+t_k{d}_k)\le f\left(x_k\right)+\delta t_k{g}_k^T{d}_k$$

  $$g{(x_k+t_k{d}_k)}^T{d}_k\ge \sigma g_k^{T}t{d}_k;$$
• Strong Wolfe’s line search: 

  $$f(x_k+t_k{d}_k)\le f\left(x_k\right)+\delta t_k{g}_k^T{d}_k$$

  $$|g{(x_k+t_k{d}_k)}^T{d}_k|\ge -\sigma g_k^{T}t{d}_k;$$
• Backtracking algorithm:

• Objective function $f\left(x\right)$, the direction $d_k$ of the search at the point $x_k$ and numbers $0<\sigma <0.5$ and $\beta \in (0,1)$ are required;
• $t=1$;
• $f(x_k+t{d}_k)>f\left(x_k\right)+\sigma t{g}_k^T{d}_k$, take $t:=t\beta$;
• Return $t_k=t$.

Subsequently, the Newton method with line search is given as

$$x_{k+1}=x_k-t_k{G}_k^{-1}{g}_k.$$

(6)

In (6), $G_k^{-1}$ stands for the inverse of the function Hessian, according to the previously adopted notation. Step length parameter $t_k$ is obtained by applying some chosen inexact procedure. Instead of calculating the inverse of the function Hessian, which is often a demanding task, in quasi-Newton methods the adequate approximation of the Hessian (or of its inverse) is used

$$x_{k+1}=x_k-t_k{H}_k{g}_k.$$

(7)

Herein, $H_k\equiv B_k^{-1}$ and $B_k$ is derived as a symmetric positive definite Hessian’s approximation. Updating of $\left\{B_i\right\},i\in \mathbb{N}$ is conducted using the quasi-Newton property of secant equation

$$B_{k+1}{s}_k=y_k,$$

where parameters $s_k$ and $y_k$ are the differences in two successive iterative points and iterative gradients, respectively, i.e., $s_k=x_{k+1}-x_k,y_k=g_{k+1}-g_k$. In [13], the classification of methods for updating the matrix $B_k$ is presented. Therein, the three updating methods are extricated:

• matrix $B_k$ is defined as a scalar matrix, i.e., $B_k={\gamma }_{k}I,{\gamma }_k>0;$
• matrix $B_k$ is defined as a diagonal matrix, i.e., $B_k=diag({\lambda }_1,\cdots ,{\lambda }_n),{\lambda }_i>0,i=\overline{1,n};$
• matrix $B_k$ is defined as a full matrix.

Taking the simplest updating approach, the first one, i.e., $B_k={\gamma }_{k}I\backsim G_k,{\gamma }_k>0$, the quasi-Newton method (7) is transformed to

$$x_{k+1}=x_k-t_k{\gamma }_k^{-1}{g}_k.$$

(8)

In [14], the authors named the iterative methods (8) as a class of accelerated gradient methods. They named it so for their good convergence and performance metrics. Previously in [15], Andrei defined this accelerated iteration by calculating parameter ${\theta }_k(={\gamma }_k^{-1})$ as follows:

Algorithm for generating scalar ${\theta }_k$ from [15]:

• Objective function $f\left(x\right)$, the direction $d_k$ of the search at the point $x_k$ and numbers $0<\sigma <0.5$ and $\beta \in (0,1)$ are required;
• Apply Backtracking algorithm to calculate $t_k\in (0,1]$;
• Compute $z=x_k-t_k{g}_k,g_z=\nabla f\left(z\right),y_k=g_z-g_k$;
• Compute $a_k=t_k{g}_k^T{g}_k,b_k=-t_k{y}_k^T{g}_k$
• Return ${\theta }_k=\frac{a_k}{b_k}$.

Stanimirović and Miladinović in [14] determined the parameter ${\gamma }_k^{-1}$ from (8) on the basis of the second-order Taylor’s expansion, and denoted it as $SM$ method. Results obtained by several researchers on this topic confirmed that this way of deriving the accelerated parameter (as named in [16]), is justifiable regarding the convergence and numerical performance aspects [17,18]. Several forms of this important variable in chosen accelerated gradient schemes are listed as expressions (5)–(9) in [19] and some other approaches are presented in [20,21,22].

Among the various accelerated gradient iterations, for this research we specifically chose three among which Khan’s hybridization three-term process was later applied. The first of the three is the already mentioned $SM$ method, defined by relation (8) and presented in [14]. The second one is the $ADD$ method (i.e., accelerated double direction method) from [16], with the iterative representation

$$x_{k+1}=x_k+{\alpha }_k^2{d}_k-{\alpha }_k{{\gamma }_k}^{-1}{g}_k.$$

(9)

In (9), ${\alpha }_k>0$ is the iterative step length value, while $d_k$ is the second search vector calculated under the assumption $\parallel d_k\parallel =1,k=1,2,\dots$, by the next procedure:

$$d_k\left(t\right)=\left\{\begin{array}{cc}{d}_k^{*},\hfill & k\le m-1\hfill \\ \sum _{i=2}^m{t}^{i-1}{d}_{k-i+1}^{*}\hfill & k\ge m\hfill \end{array}\right.$$

(10)

and $d_k^{*}$ is the solution of the problem ${min}_{x\in R}{\mathsf{\Phi }}_k\left(d\right)$,

$${\mathsf{\Phi }}_k\left(d\right)=▽f{\left(x_k\right)}^{T}d+\frac{1}{2}{\gamma }_{k+1}I=g{\left(x_k\right)}^{T}d+\frac{1}{2}{\gamma }_{k+1}I.$$

(11)

The iterative value of the accelerated parameter ${\gamma }_k$, obtained through Taylor’s series of (9), is

$${\gamma }_{k+1}^{ADD}=2\frac{f\left(x_{k+1}\right)-f\left(x_k\right)-{\alpha }_k{g}_k^T\left({\alpha }_k{d}_k-{\gamma }_k^{-1}{g}_k\right)}{{\left({\alpha }_k{d}_k-{\gamma }_k^{-1}{g}_k\right)}^T\left(t_k{d}_k-{\gamma }_k^{-1}{g}_k\right)}.$$

(12)

The positive step length value ${\alpha }_k$ of the $ADD$ scheme is derived using the Backtracking algorithm, starting with initial $t=1.$ Taking the following substitutions in (9)

$${\alpha }_k^2\to {\beta }_k,$$

where ${\beta }_k$ is calculated by a different Backtracking procedure, and

$$d_k\to -g_k,$$

lead to the ADSS (accelerated double step size) method [17]:

$$x_{k+1}=x_k-{\alpha }_k{{\gamma }_k}^{-1}{g}_k-{\beta }_k{g}_k=x_k-\left({\alpha }_k{\gamma }_k^{-1}+{\beta }_k\right)g_k.$$

(13)

Finally, under assumption

$${\alpha }_k+{\beta }_k=1,$$

$ADSS$ iteration is transformed to $TADSS$ scheme [18], our third chosen accelerated gradient descent model:

$$x_{k+1}=x_k-[{\alpha }_k({{\gamma }_k}^{-1}-1)+1]g_k.$$

(14)

As in (9), the iterative step length value ${\alpha }_k$ in (14) is calculated on the basis of Backtracking algorithm. The accelerated parameter of the $TADSS$ scheme is

$${\gamma }_{k+1}^{TADSS}=2\frac{f\left(x_{k+1}\right)-f\left(x_k\right)+{\psi }_k{\parallel g_k\parallel }^2}{{\psi }_k^2{\parallel g_k\parallel }^2},{\psi }_k=[{\alpha }_k{\gamma }_k^{-1}-{\alpha }_k^2)+1].$$

(15)

In the following proposition, we prove that (8), (9) and (14) iterations are a gradient descending process.

Proposition 1.

The search directions in iterations defined by (8), (9) and (14) fulfil the descending condition (2).

Proof. 

We separately analyze the search directions of all three listed methods.

• According to the general iteration form (1), the search direction in $SM$ method, defined by relation (8), is $d_k\equiv -{\gamma }_k^{-1}{g}_k.$ One of the essential properties of the accelerated parameter ${\gamma }_k$ is its positiveness. If in some iterative step k of the accelerated gradient algorithms with leading iterative rules (8), (9) and (14) this necessary condition is not fulfilled, then the k-th accelerated scalar value is set to be ${\gamma }_k=1$. Bearing this fact in mind, we easily conclude that

  $${\mathbf{g}}_k^T{\mathbf{d}}_k={\mathbf{g}}_k^T(-{\gamma }_k^{-1}{\mathbf{g}}_k)=-{\gamma }_k^{-1}{\parallel {\mathbf{g}}_k\parallel }^2<0,$$

  which confirms that the condition (2) in $SM$ method, defined by (8), is fulfilled.
• Accelerated double direction $ADD$ scheme (9) contains two search vectors. The first one, denoted as $d_k$, is defined by (10). The second one is of the same form as in the $SM$ iteration, i.e., $-{{\gamma }_k}^{-1}{g}_k$. In the procedure (10), crucial element $d_k^{*}$ in deriving vector $d_k$ is defined as a solution of the minimization problem (11) that depends on the gradient $g_k$, under the assumption $\parallel d_k\parallel =1.$ Thus, the defined vector direction is a relaxed differentiable variant of the procedure for determination of the search vector $d_k$ (rule 2) in [23] and accordingly, the $d=0$ is globally optimum of the problem (11). Subsequently, we consider only the second direction, which is already performed in the previous item.
• In $TADSS$ scheme, vector direction can be seen as $-[{\alpha }_k({{\gamma }_k}^{-1}-1)+1]g_k$. Checking the descent condition (2), we get

  $${\mathbf{g}}_k^T·\left(-[{\alpha }_k({{\gamma }_k}^{-1}-1)+1]{\mathbf{g}}_k\right)=-[{\alpha }_k({{\gamma }_k}^{-1}-1)+1]{\parallel {\mathbf{g}}_k\parallel }^2<1,$$

  since ${\gamma }_k^{-1}>1$ and ${\alpha }_k\in (0,1]$ according to $TADSS$ algorithm.

□

Further on, in Section 2, we analyse Khan’s hybridization rule applied on various gradient methods. Finally, Dolan–Moré representations and parallels among the hybrid and non-hybrid versions of $SM,ADD$ and $TADSS$ schemes, obtained over large-scale numerical outcomes, are illustrated in Section 3.

2. Three-Term Khan’s Hybridization Principle over the Accelerated Gradient Descent Models

For a nonempty convex subset $\mathbb{C}$ of a normed space $\mathbb{E}$ , let $T:\mathbb{C}\to \mathbb{C}$ be a mapping defined on $\mathbb{C}$ . Then, for some sequences $u_k,v_k,z_k$ and $y_k$ defined on $\mathbb{C}$, the Picard, Mann and Ishikawa iterative processes [24,25,26] are, respectively, given as:

$$\left\{\begin{array}{c}{u}_1=u\in \mathbb{C},\hfill \\ u_{k+1}=T{u}_k,\hfill & k\in \mathbb{N},\hfill \end{array}\right.$$

$$\left\{\begin{array}{c}{v}_1=v\in \mathbb{C},\hfill \\ v_{k+1}=(1-{\alpha }_k)v_k+{\alpha }_{k}T{v}_k,\hfill & k\in \mathbb{N},\hfill \end{array}\right.$$

$$\left\{\begin{array}{c}{z}_1=z\in \mathbb{C},\hfill \\ z_{k+1}=(1-{\alpha }_k)z_k+{\alpha }_{k}Tyk,\hfill \\ y_k=(1-{\beta }_k)z_k+{\beta }_{k}T{z}_k.\hfill & k\in \mathbb{N}\hfill \end{array}\right.$$

In the listed relations, parameters $\left\{{\alpha }_k\right\},\left\{{\beta }_k\right\}\in (0,1)$ are the sequences of positive numbers, which in the Ishikawa process [26] fulfil the following assumptions

• $0\le {\alpha }_k\le {\beta }_k\le 1,k\ge 0$,
• ${lim}_{k\to \infty }{\beta }_k=0$,
• ${\sum }_{k=1}^{\infty }{\alpha }_k{\beta }_k=\infty$.

In [27], Khan proposed a new three-term iterative process as follows

$$\left\{\begin{array}{c}{x}_1=x\in \mathbb{R},\hfill \\ x_{k+1}=T{y}_k,\hfill \\ y_k=(1-{\alpha }_k)x_k+{\alpha }_{k}T{x}_k,\hfill & k\in \mathbb{N}\hfill \end{array}\right.$$

(16)

with the sequence of positive numbers $\left\{{\alpha }_k\right\}\in (0,1)$, which is considered as a set of constant values, i.e., $(\alpha ={\alpha }_k\in (0,1)\forall k\in \mathbb{N},)$ in practical numerical tests, as proposed in [27]. Khan confirmed in [27] that the process (16) converges faster than the processes of Picard, Mann and Ishikawa.

Khan developed this iterative process (16) as a hybrid variant of previously mentioned, well-known iterations and with that managed to improve these types of methods. Further, some authors used auspicious aspects of this hybrid rule and applied it to some accelerated gradient optimization methods. The hybridization principle consists of taking the objective accelerated iteration as a guiding operator in (16). As a result, several accelerated-hybridization processes arose [28,29,30,31,32]. We list them below, together with their accelerated parameters.

• Hybrid accelerated gradient descent method ($HSM$) [28]

  $$x_{k+1}=x_k-({\alpha }_k+1)t_k{\gamma }_k^{-1}{g}_k,$$

  (17)

  $${\gamma }_{k+1}^{HSM}=2{\gamma }_k\frac{{\gamma }_k\left[f\left(x_{k+1}\right)-f\left(x_k\right)\right]+({\alpha }_k+1)t_k{\parallel g_k\parallel }^2}{{({\alpha }_k+1)}^2{t}_k^2{\parallel g_k\parallel }^2}.$$

  (18)
• Hybrid accelerated double direction method ($HADD$) [29]

  $$x_{k+1}=x_k-\alpha t_k{\gamma }_k^{-1}{g}_k+\alpha t_k^2{d}_k,\alpha \in (1,2),$$

  (19)

  $${\gamma }_{k+1}=2\frac{f\left(x_{k+1}\right)-f\left(x_k\right)-\alpha g_k^T\left(t_k^2{d}_k-t_k{\gamma }_k^{-1}{g}_k\right)}{{\alpha }^2{t}_k^2{\left(t_k{d}_k-{\gamma }_k^{-1}{g}_k\right)}^T\left(t_k{d}_k-{\gamma }_k^{-1}{g}_k\right)}.$$

  (20)
• Hybrid accelerated double step size method ($HADSS$) [30]

  $$x_{k+1}=x_k-g_k\alpha (t_k{\gamma }_k^{-1}+p_k),\alpha \equiv {\alpha }_k+1\in (1,2)\forall k,$$

  (21)

  $${\gamma }_{k+1}^{HADSS}=2\frac{f\left(x_{k+1}\right)-f\left(x_k\right)+\alpha \left(t_k{\gamma }_k^{-1}+p_k\right){\parallel g_k\parallel }^2}{{\alpha }^2{\left(t_k{\gamma }_k^{-1}+p_k\right)}^2{\parallel g_k\parallel }^2}.$$

  (22)
• Hybrid transformed double step size method ($HTADSS$) [31]

  $$x_{k+1}=x_k-\alpha (t_k({\gamma }_k^{-1}-1)+1)g_k,\alpha \in (1,2),$$

  (23)

  $${\gamma }_{k+1}^{HTADSS}=2\frac{f\left(x_{k+1}\right)-f\left(x_k\right)+\alpha {\phi }_k{\parallel g_k\parallel }^2}{{\alpha }^2{\phi }_k^2{\parallel g_k\parallel }^2},$$

  (24)

  where

  $${\phi }_k=t_k({\gamma }_k^{-1}-1)+1.$$

  (25)
• Hybrid gradient descent method ($HGD$) [32]

  $$x_{k+1}=x_k-({\alpha }_k+1)t_k{g}_k,{\alpha }_k\in (0,1)\forall k.$$

  (26)
• Hybrid accelerated gradient descent method ($HAGD$) [32]

  $$x_{k+1}=x_k-({\alpha }_k+1){\theta }_k{t}_k{g}_k,\alpha \in (0,1)\forall k,{\theta }_k=\frac{{\gamma }_k}{t_k{\gamma }_{k+1}}.$$

  (27)
• Hybrid modified accelerated gradient descent method ($HMAGD$) [32]

  $$x_{k+1}=x_k-({\alpha }_k+1){\theta }_k(t_k+t_k^2-t_k^3)g_k,\alpha \in (0,1)\forall k,{\theta }_k=\frac{{\gamma }_k}{t_k{\gamma }_{k+1}}.$$

  (28)
• Hybrid modified improved gradient descent method ($HMIGD$) [32]

  $$x_{k+1}=x_k-({\alpha }_k+1){\gamma }_k^{-1}(t_k+t_k^2-t_k^3)g_k,\alpha \in (0,1)\forall k.$$

  (29)

As shown above, from Khan’s hybridization rule at least eight hybrid models have appeared. Convergence properties as well as performance efficiency of these iterative schemes are presented and illustrated in the literature. The leading model of all listed ((17), (19), (21), (23), (26), (27), (28), (29)), and therewith the first one that was developed on the basis of the Khan’s process, is certainly the $HSM$ method from [28]. In paper [28], the authors examined the performance of the defined method for various values of the so-called correction parameter ${\alpha }_k\in (0,1)$, i.e., $\alpha ={\alpha }_k+1\in (1,2)$, which is a necessary factor of all hybrid methods generated through Khan’s hybridization. They experimentally concluded that the $HSM$ method achieves best performance characteristics when the correction parameter is taken closely to its left limit. Later in [33], the authors improved the $HSM$ model by reducing the initial step length parameter in the Backtracking procedure.

3. Dolan–Moré Performance Profiles and Comparisons

In this section, we apply the aspects of Dolan–Moré benchmarking optimization software from [34] on hybrid and non-hybrid variants of chosen accelerated gradient minimization models. In conducted numerical tests, we follow performance metrics regarding the number of iterations, CPU time and number of function evaluations.

For all obtained numerical outcomes, the following points are valid:

• Codes are written in the visual C++ programming language and run on a Workstation Intel(R) Core(TM) 2.3 GHz.
• The Backtracking parameters values taken are: $\sigma =0.0001$ and $\beta =0.8$. These are standard values for the Backtracking parameters applied in various optimization models with Backtracking algorithm [2,14,15,20,21,22,28,29,30,31]. This set of values means that a small portion of the decrease predicted by the linear approximation of the current point is accepted.
• The stopping criteria are:

  $$\parallel g_k\parallel \le {10}^{-6}\mathrm{and}\frac{|f\left(x_{k+1}\right)-f\left(x_k\right)|}{1+\left|f\right(x_k\left)\right|}\le {10}^{-16}.$$
• Chosen test functions are taken from the unconstrained test functions collection presented in [35]. More precisely, all specific test functions that were used for this research are listed in Listing 1.

Listing 1. Test functions.

1. Extended Penalty

2. Perturbed Quadratic

3. Raydan-1

4. Diagonal 1

5. Diagonal 3

6. Generalized Tridiagonal-1

7. Diagonal 4

8. Extended Himmelblau

9. Quadr. Diag. Perturbed

10. Quadratic QF1

11. Exten. Quadr. Penalty QP1

12. Exten. Quadr. Penalty QP2

13. Quadratic QF2

14. Extended EP1

15. Arwhead

16. Almost Perturbed Quadratic

17. Engval1

18. Quartc

19. Generalized Quartic

20. LIARWHD

21. Diagonal 6

22. Tridia

23. Indef

24. Diagonal 9

25. DIXON3DQ

26. NONSCOMP

27. BIGGSB1

28. Power (Cute)

29. Hager

30. Raydan 2

Listing 1. Test functions.

1. Extended Penalty

2. Perturbed Quadratic

3. Raydan-1

4. Diagonal 1

5. Diagonal 3

6. Generalized Tridiagonal-1

7. Diagonal 4

8. Extended Himmelblau

9. Quadr. Diag. Perturbed

10. Quadratic QF1

11. Exten. Quadr. Penalty QP1

12. Exten. Quadr. Penalty QP2

13. Quadratic QF2

14. Extended EP1

15. Arwhead

16. Almost Perturbed Quadratic

17. Engval1

18. Quartc

19. Generalized Quartic

20. LIARWHD

21. Diagonal 6

22. Tridia

23. Indef

24. Diagonal 9

25. DIXON3DQ

26. NONSCOMP

27. BIGGSB1

28. Power (Cute)

29. Hager

30. Raydan 2

Further, by $i_{p,s},t_{p,s}$ and $e_{p,s}$ we denote the number of iterations, the CPU time and the number of function evaluations, respectively, needed for solving problem p when the solver s is applied. The main observation arises from a comparison of performance profiles, considering analyzed metrics, of hybrid and non-hybrid versions of the same scheme. For this investigation, we chose the following three accelerated gradient methods: $SM$ (8), $ADD$ (9) and $TADSS$ (14). Accordingly, we analyzed their hybrid forms $HSM$ (17), $HADD$ (19) and $HTADSS$ (23), as well. So in these tests, the solver s belongs to the set of six elements, $s\in \{SM,HSM,ADD,HADD,TADSS,HTADSS\}$. We specifically chose this set of comparative non-hybrid and hybrid pairs among the others mentioned above in Section 2, since the selected models are the most cited of Khan’s hybrid methods.

According to the benchmark presented in [34] and regarding the comparisons obtained within this paper, for each pair of comparative hybrid and non-hybrid variants we have two solvers, i.e., $n_s=2$, where the comparative pairs are:

$$\left\{\right(SM,HSM),(ADD,HADD),(TADSS,HTADSS\left)\right\}.$$

The total number of experiments for each pair is minimal $n_p=210$. Precisely, for the pair $(SM,HSM)$ we have conducted numerical tests for 25 test function and 11 different numbers of variables, so $n_p=11·25=275$, the same as for the pair $(TADSS,HTADSS)$. The pair $(ADD,HADD)$ included 21 test functions with 10 different numbers of variables, so in this case $n_p=210$. In order to apply Dolan–Moré benchmarking optimization software with performance profiles over the chosen accelerated and hybrid models, we use the original outcomes presented in the papers in which these models were generated [28,29,31]. So, since in [28,31] numerical experiments included 25 test functions, while in [29] the number of tested functions is 21, the total number of tests for all test functions and all three pairs of models is $n_p^{SM,HSM}+n_p^{TADSS,HTADSS}+n_p^{ADD,HADD}=760.$

Considering defined parameters, we are now able to expose performance ratios defined for the number of iterations, the CPU time and the number of function evaluations, respectively:

$$\begin{array}{l}{r}_{p,s}=\frac{i_{p,s}}{min\{i_{p,s}:s\in \{SM,HSM\}\}}\\ =\frac{i_{p,s}}{min\{i_{p,s}:s\in \{ADD,HADD\}\}}=\frac{i_{p,s}}{min\{i_{p,s}:s\in \{TADSS,HTADSS\}\}},\end{array}$$

$$\begin{array}{l}{r}_{p,s}=\frac{t_{p,s}}{min\{t_{p,s}:s\in \{SM,HSM\}\}}\\ =\frac{t_{p,s}}{min\{t_{p,s}:s\in \{ADD,HADD\}\}}=\frac{t_{p,s}}{min\{t_{p,s}:s\in \{TADSS,HTADSS\}\}},\end{array}$$

$$\begin{array}{l}{r}_{p,s}=\frac{e_{p,s}}{min\{e_{p,s}:s\in \{SM,HSM\}\}}\\ =\frac{e_{p,s}}{min\{e_{p,s}:s\in \{ADD,HADD\}\}}=\frac{e_{p,s}}{min\{e_{p,s}:s\in \{TADSS,HTADSS\}\}}.\end{array}$$

As in [34], we define the performance profile for each solver s with respect to all three measured metrics

$${\rho }_s\left(\tau \right)=\frac{1}{n_p}size\{p\in \mathcal{P}:r_{p,s}\le \tau \},$$

(30)

which presents the cumulative distribution function. In (30) parameter $\tau \in \mathbb{R},$ while $\mathcal{P}$ is the set of problems.

In Figure 1, Figure 2 and Figure 3, we present the performance profiles of $SM$ and $HSM$ regarding the number of iterations, the CPU time and the number of function evaluations, respectively. Comparisons between pairs ($ADD,HADD$) and ($TADSS,HTADSS$), regarding all three tested metrics are similarly illustrated in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9.

From Figure 1, Figure 2 and Figure 3, we clearly observe that the $HSM$ algorithm outperforms the non-hybrid $SM$ model, with respect to the number of iterations and the CPU time metrics, while regarding the number of evaluations metric, the hybrid and non-hybrid models perform similarly.

From Figure 5 and Figure 6, we see that the hybrid accelerated double direction model shows conspicuously better features regarding CPU time and the number of evaluations metrics. Nevertheless, concerning the number of iterations metric, forerunner $ADD$ is more efficient, as shown in Figure 4.

Finally, comparisons of the $TADSS$ and the $HTADSS$ methods are displayed in Figure 7, Figure 8 and Figure 9. From these three presented graphs, we observe that the hybrid version of the transformed double step size method convincingly upgrades its counterpart non-hybrid model. In this case, the dominance of the hybrid variant with respect to all three analyzed metrics is more than evident.

To obtain Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9, a total of 4560 numerical outcomes were included. More precisely, for 6 analyzed methods ($SM,ADD,TADSS,HSM,HADD,HTADSS$) we followed 3 metrics (number of iterations, CPU time, number of function evaluations) on 25 test functions for $SM,HSM,TADSS$ and $HTADSS$ solvers and 21 test functions for $ADD$ and $HADD$ solvers from [35]. For each test function, the tests were conducted for at least 10 different numbers of variables. With that, the execution time for each test is limited by the time-limiter parameter defined in [16].

4. Conclusions

In this research, we present an overview of two gradient method classes: accelerated gradient descent models and its hybrid variants derived from Khan’s three-term iterative rule. This is an important and useful retrospective of one, confirmed efficient approach in defining the robust accelerated methods for solving unconstrained optimization problems. The obtained results, achieved on the basis of comprehensive Dolan–Moré performance profiles [34], conducted on total 4560 numerical outcomes, confirm that Khan’s hybridization rule is justified for use as an applicable technique in generating effective minimization processes. Accordingly, this research paves the way for new possibilities aimed at generating similar hybridization rules and their applications to accelerated gradient schemes.