Application of the Parabola Method in Nonconvex Optimization

Abstract

We consider the Golden Section and Parabola Methods for solving univariate optimization problems. For multivariate problems, we use these methods as line search procedures in combination with well-known zero-order methods such as the coordinate descent method, the Hooke and Jeeves method, and the Rosenbrock method. A comprehensive numerical comparison of the obtained versions of zero-order methods is given in the present work. The set of test problems includes nonconvex functions with a large number of local and global optimum points. Zero-order methods combined with the Parabola method demonstrate high performance and quite frequently find the global optimum even for large problems (up to 100 variables).

1. Introduction

Zero-order methods are usually applied to problems in which the function being optimized has no explicit analytic expression. To calculate the values of such functions, it is necessary to solve non-trivial auxiliary problems of computational mathematics and optimization. In some cases, it is assumed that the calculation of a single objective function value may take from a few minutes to several hours of continuous computer operation. Developing an effective global optimization method requires identifying certain properties of the objective function (and constraints), for example, determining a good estimate of the Lipschitz constant [1,2,3,4] or representing a function as a difference between two convex functions [5,6,7]. Such auxiliary problems do not always have a unique solution and are not often easily solvable. The effectiveness of global optimization methods often depends on the ability to quickly find a good local solution [8], which can significantly speed up some global optimization methods (for example, the branch and bounds method).

The aim of the paper is to test several well-known zero-order methods on nonconvex global optimization problems a using special version of the multistart strategy. We check the ability of these methods not only to find a good local solution but also to find a global optimum of the problem. The multistart strategy was considered as a starting case for population-based optimization. It is necessary to note that the multistart approach can be considered as an overabundant population. Our aim is to check the efficiency of the suggested approach under this condition.

Special attention is given to the Parabola Method in the present work. The reason for such an investigation is the following: if the objective function is differentiable, then in a small neighborhood of an interior local minimum point, the objective function behaves like a convex function. In [9], a more general statement of this property based on the concept of local programming is given. A combination of this method with different zero-order methods in multivariate optimization problems demonstrates quite high performance and global optimization method qualities. We tested several well-known zero-order multivariate optimization methods; nevertheless, application of this modification of the Parabola Method as the line-search procedure can be used in many other methods.

The paper is organized as follows. In Section 2, we give a description of the Golden Section and Parabola Methods and perform numerical experiments on a set of univariate optimization problems. A numerical comparison of zero-order methods in combination with the Golden Section and Parabola Methods is presented in Section 3. We conclude the paper with a short discussion of the results in Section 4 and Section 5.

2. Univariate Optimization: Parabola Method

We consider the following optimization problem

$$\begin{array}{cc}\hfill \mathrm{minimize}& f\left(x\right),\hfill \\ \hfill \mathrm{subject}\mathrm{to}& x\in P\subset R,P=\left\{x\in R:\alpha \le x\le \beta \right\},\hfill \end{array}$$

(1)

where $f:R\to R$ is a continuously differentiable function. It is assumed that f has a finite number of local minima over P, and only values of f are available. Problems with such an assumption are described in [10]. In this section, we consider two well-known univariate optimization methods: the Golden Section Method (GSM) [11] and the Quadratic Interpolation Method or the Powell M.J.D. Method or the Parabola Method (PM) [12]. Let us give brief descriptions of both methods.

2.1. The Golden Section Method

Step 0.

Choose the accuracy of the algorithm $\epsilon >0$. Set $\gamma =\frac{\sqrt{5}-1}{2},{\alpha }_0=\alpha ,{\beta }_0=\beta ,$ ${\lambda }_0={\alpha }_0+(1-\gamma )({\beta }_0-{\alpha }_0),{\mu }_0={\alpha }_0+\gamma ({\beta }_0-{\alpha }_0)$. Evaluate $f\left({\lambda }_0\right)$ and $f\left({\mu }_0\right)$. Set $k=0$.

Step 1.

If ${\beta }_k-{\alpha }_k\le \epsilon$, stop: $x^{*}$ is $\epsilon$-optimal point. Otherwise, if $f\left({\lambda }_k\right)>f\left({\mu }_k\right)$, go to Step 2; else, go to Step 3.

Step 2.

Set ${\alpha }_{k+1}={\lambda }_k,{\beta }_{k+1}={\beta }_k,{\lambda }_{k+1}={\mu }_k,{\mu }_{k+1}={\alpha }_{k+1}+\gamma ({\beta }_{k+1}-{\alpha }_{k+1}).$ Evaluate $f\left({\mu }_{k+1}\right)$. Go to Step 4.

Step 3.

Set ${\alpha }_{k+1}={\alpha }_k,{\beta }_{k+1}={\mu }_k,{\mu }_{k+1}={\lambda }_k,{\lambda }_{k+1}={\alpha }_{k+1}+(1-\gamma )({\beta }_{k+1}-{\alpha }_{k+1})$. Evaluate $f\left({\lambda }_{k+1}\right)$. Go to Step 4.

Step 4.

Increase $k\to k+1$, and go to Step 1.

2.2. The Parabola Method

Step 0.

Choose the accuracy $\epsilon >0$. Choose points $x_1^0<x_2^0<x_3^0\in [\alpha ,\beta ]$ such that

$$f\left(x_1^0\right)>f\left(x_2^0\right)<f\left(x_3^0\right).$$

(2)

Set $z=x_2^0,k=0$.

Step 1.

Find the minimum ${\overline{x}}^k$ of the quadratic interpolation polynomial in the following way.

$${\overline{x}}^k=\frac{1}{2}\left(x_1^k+x_2^k-\frac{a_1^k}{a_2^k}\right),$$

$$a_1^k=\frac{f\left(x_2^k\right)-f\left(x_1^k\right)}{x_2^k-x_1^k},a_2^k=\frac{1}{x_3^k-x_2^k}\left(\frac{f\left(x_3^k\right)-f\left(x_1^k\right)}{x_3^k-x_1^k}-a_1^k\right).$$

Step 2.

Check stop criterion

$$|{\overline{x}}^k-z|\le \epsilon .$$

(3)

If (3) is held, then terminate the algorithm, and $x^{★}={\overline{x}}^k$ is $\epsilon$-optimal point. Otherwise, go to Step 3.

Step 3.

Choose ${\widehat{x}}^k=arg\underset{x=x_1^k,x_3^k}{min}f\left(x\right)$.

Set $z={\overline{x}}^k$. Denote points $\widehat{x},x_2^k,{\overline{x}}^k$ in ascending order as $x_1^{k+1},x_2^{k+1},x_3^{k+1}$. Increase $k\to k+1$, and go to Step 1.

When f is a unimodal (semi-strictly quasiconvex [13]) over P function, both methods determine an approximate point of minimum in a finite number of iterations. In the case of the Golden Section Method the number of function evaluations is equal to the number of iterations plus two; in the case of the Parabola Method, the number of function evaluations is equal to the number of iterations plus three. It is well known that, in general, the Golden Section Method is more efficient than the Parabola Method. However, in the continuously differentiable case, the behavior of the Parabola Method can be improved.

Consider the following two examples from [14].

Example 1.

In problem (1), $f\left(x\right)=-16(x^2-24x+5)e^{-x},P=[1.9,3.9]$. The Golden Section Method with $\epsilon =0.001$ finds the approximate solution $x^{*}=2.867996734$ with $f\left(x^{*}\right)=-3.850450707$ in 16 iterations. The Parabola Method with the same ε finds the approximate solution $x^{*}=2.868823736$ with $f\left(x^{*}\right)=-3.850448184$ in six iterations.

Example 2.

In problem (1), $f\left(x\right)=2{(x-3)}^2+e^{\frac{x^2}{2}},P=[-3,3]$. The Golden Section Method with $\epsilon =0.001$ finds the approximate solution $x^{*}=1.590558077$ with $f\left(x^{*}\right)=7.515924361$ in 19 iterations. The Parabola Method with the same ε finds the approximate solution $x^{*}=1.584929941$ with $f\left(x^{*}\right)=7.516292947$ in 35 iterations.

In both examples, the objective functions are unimodal. In the first example, the Parabola Method worked two times faster, and in the second example, the Parabola Method worked about two times slower than the Golden Section Method. From a geometrical point of view, the objective function in the first example is more like a parabola than in the second example, and we are going to show how the efficiency of the Parabola Method can be improved in cases similar to Example 2.

We start from checking efficiency of the Golden Section Method on a number of multimodal functions.

Table 1. Test problems 1–5.

No.	Function and Interval	Solution	$\mathbf{g}/\mathbf{l}$	Record	${\mathit{k}}_{\mathit{f}}$
1.	$f\left(x\right)=\frac{x^6}{6}-\frac{52}{25}{x}^5+\frac{39}{80}{x}^4+$	$f^{*}=-29763.23$,	g	$f^{♯}=-29763.23$	22
	$\frac{71}{10}{x}^3-\frac{79}{20}{x}^2-x+\frac{1}{10},$	$x^{*}=10$		$x^{♯}=9.9996$
	$x\in [-1.5,11]$
2.	$f\left(x\right)=sin\left(x\right)+sin\left(\frac{10x}{3}\right),$	$f^{*}=-1.899599$,	g	$f^{♯}=-1.899599$	20
	$x\in [2.7,7.5]$	$x^{*}=5.145735$		$x^{♯}=5.145733$
3.	$f\left(x\right)=-{\displaystyle \sum _{k=1}^5}ksin((k+1)x+k),$	$f^{*}=-12.031249$,	g	$f^{♯}=-12.031234$	22
	$x\in [-10,10]$	$x^{*,1}=-6.774576$		$x^{♯}=-0.491707$
		$x^{*,2}=-0.491390$
		$x^{*,3}=5.791794$
4.	$f\left(x\right)=(3x-1.4)sin\left(18x\right),$	$f^{*}=-1.489072$,	l	$f^{♯}=-0.4612863$	17
	$x\in [0,1.2]$	$x^{*}=0.966086$		$x^{♯}=0.6291664$
5.	$f\left(x\right)=sin\left(x\right)+sin\left(\frac{10x}{3}\right)+$	$f^{*}=-1.601308$,	g	$f^{♯}=-1.601308$	20
	$ln\left(x\right)-0.84x+3$	$x^{*}=5.199778$		$x^{♯}=5.199901$
	$x\in [2.7,7.5]$

and

Table 2. Test problems 6–17.

No.	Function and Interval	Solution	g/l	Record	${\mathit{k}}_{\mathit{f}}$
6.	$f\left(x\right)=-{\displaystyle \sum _{k=1}^5}kcos((k+1)x+k),$	$f^{*}=-14.50801$,	g	$f^{♯}=-14.50780$	22
	$x\in [-10,10]$	$x^{*,1}=-7.083506$		$x^{♯}=5.483102$
		$x^{*,2}=-0.800321$
		$x^{*,3}=5.482864$
7.	$f\left(x\right)=sin\left(x\right)+sin\left(\frac{2x}{3}\right),$	$f^{*}=-1.905961$,	l	$f^{♯}=-1.215982$	22
	$x\in [3.1,20.4]$	$x^{*}=17.039199$		$x^{♯}=5.361825$
8.	$f\left(x\right)=-xsin\left(x\right),$	$f^{*}=-7.916727$,	g	$f^{♯}=-7.916727$	22
	$x\in [0,10]$	$x^{*}=7.978666$		$x^{♯}=7.978632$
9.	$f\left(x\right)=2cos\left(x\right)+cos\left(2x\right),$	$f^{*}=-1.500000$,	g	$f^{♯}=-1.499999$	21
	$x\in [-1.57,6.28]$	$x^{*,1}=2.094395$		$x^{♯}=2.094447$
		$x^{*,2}=4.188790$
10.	$f\left(x\right)={sin}^3\left(x\right)+{cos}^3\left(x\right),$	$f^{*}=-1.000000$,	g	$f^{♯}=-0.999999$	21
	$x\in [0,6.28]$	$x^{*,1}=3.141593$		$x^{♯}=4.712287$
		$x^{*,2}=4.712389$
11.	$f\left(x\right)=-e^{-x}sin\left(2\pi x\right),$	$f^{*}=-0.788595$,	l	$f^{♯}=-0.039202$	20
	$x\in [0,4]$	$x^{*}=0.224982$		$x^{♯}=3.226282$
12.	$f\left(x\right)=\frac{x^2-5x+6}{x^2+1},$	$f^{*}=-0.035534$,	g	$f^{♯}=-0.035533$	22
	$x\in [-5,5]$	$x^{*}=2.414214$		$x^{♯}=2.414418$
13.	$f\left(x\right)=x^6-15x^4+27x^2+250,$	$f^{*}=7.000000$,	g	$f^{♯}=7.000001$	21
	$x\in [-4,4]$	$x^{*,1}=-3.0$		$x^{♯}=-3.000009$
		$x^{*,2}=3.0$
14.	$f\left(x\right)=-x+sin\left(3x\right)-1,$	$f^{*}=-7.815675$,	l	$f^{♯}=-5.721279$	21
	$x\in [0,6.5]$	$x^{*}=5.872866$		$x^{♯}=3.778193$
15.	$f\left(x\right)=cos\left(x\right)-sin\left(5x\right)+1,$	$f^{*}=-0.952897$,	g	$f^{♯}=-0.952896$	21
	$x\in [0,7]$	$x^{*}=2.839347$		$x^{♯}=2.839196$
16.	$f\left(x\right)=-x{e}^{-sin\left(3x\right)}+1,$	$f^{*}=-3.363290$,	g	$f^{♯}=-3.363290$	20
	$x\in [-3,2]$	$x^{*}=1.639062$		$x^{♯}=1.638984$
17.	$f\left(x\right)=ln\left(3x\right)ln\left(2x\right)-1,$	$f^{*}=-1.041100$,	g	$f^{♯}=-1.041100$	21
	$x\in [0.1,7]$	$x^{*}=0.408248$		$x^{♯}=0.408014$

present the results of 17 test problems from [14].

The sign g in the g/l column corresponds to the case when a global minimum point was determined, and the sign l corresponds to the case when only a local minimum point was determined. Symbol $k_f$ corresponds to the number of function evaluations. Surprisingly enough, only in four problems was a local minimum determined; in all other problems, the Golden Section Method found global solutions.

The geometrical interpretation of failure of the Golden Section Method in finding a global minimum can be given on the basis of problems 7 and 11 from Table 2. In these cases, only local minimum points were determined, and, as can be seen from Figure 1 and Figure 2, the global minimum points are located closer to the endpoints of the feasible intervals. Hence, we can make the following assumption: univariate multimodal problems with global minimum points located more or less in the middle of the intervals are more efficiently tractable by the Golden Section Method from the global optimization point of view. This topic needs further theoretical investigation and is not the subject of our paper.

Let us turn now to the Parabola Method. In order to start the method, property (2) must be satisfied. The points satisfying property (2) are known as a three-point pattern (TPP) in [11]. In order to find the TPP, we propose the following procedure.

2.3. TPP Procedure

Step 0.

Choose integer $N>0$, and calculate $\Delta =\frac{\beta -\alpha }{N}$. Set $k=0,k_p=0$.

Step 1.

Calculate $y_1^k=\alpha +k\Delta ,y_2^k=\alpha +(k+1)\Delta ,y_3^k=\alpha +(k+2)\Delta$.

Step 3.

If $f\left(y_1^k\right)>f\left(y_2^k\right)<f\left(y_3^k\right)$, then $k_p=k_p+1,{\nu }_1^{k_p}=y_1^k,{\nu }_2^{k_p}=y_2^k,{\nu }_3^{k_p}=y_3^k$.

Step 4.

If $y_3^{k_p}\le \beta -\Delta$, then increase $k\to k+1$, and go to Step 1. Otherwise, stop.

When the TPP procedure stops, we have $k_p$ three-point patterns, and in order to find the best solution, we can start the Parabola Method for all $k_p$ patterns. In practice, we used the TPP procedure several times. We started from a small enough value of N, say $N=3$, and ran the TPP procedure. Then, we increased $N\to 2N$ and ran the TPP procedure again. When the number of three-point patterns was the same in both runs, we stopped. In this case, we say that the TPP procedure stabilized. Otherwise, we increased N and ran the TPP procedure again, and so on. Since we assumed that the number of local minima of f over P is finite, such a repetition is also finite. Finally, we obtained $k_p$ TPP subintervals $[{\nu }_1^i,{\nu }_3^i]$ and the three-point corresponding patterns ${\nu }_1^i,{\nu }_2^i,{\nu }_3^i$, $i=1,\dots ,k_p$. We cannot guarantee that each subinterval contains exactly one local minimum; but, in practice, this property is true rather often. Then, we ran the Parabola Method for each subinterval, found the corresponding local minima, and selected the best ones as solutions. If $k_p=0$, then no three-point patterns were detected. In this case, the Parabola Method is not applicable (it can be disconvergent), and we switch to the Golden Section Method. We call this strategy a two-stage local search approach (two-stage approach for short).

Let us apply the two-stage approach to solving problems 11 (Figure 1) and 7 (Figure 2) from Table 1 for which the Golden Section Method did not find global minimum points and then run the Parabola Method over each obtained subinterval.

Example 3.

Problem 11: $f\left(x\right)=-e^{-x}sin\left(2\pi x\right),P=[0,4]$. Initial value $N=3$. The TPP procedure was run four times. After the first run, one subinterval with one three-point pattern was obtained, after the second run, two subintervals with two three-point patterns were obtained, after the third run, four subintervals with four three-point patterns were obtained, and after the fourth run, four smaller subintervals with four three-point patterns were obtained again. The TPP procedure was stopped, and $k_p=4$. The total number of function evaluations was equal to 24. Then, with the accuracy $\epsilon =0.001$, we ran the Parabola Method, which stopped after two additional function evaluations for each subinterval. The final subintervals, the three-point pattern (TPP), the corresponding starting parabolas, and additional function evaluations ($k_f$) of the Parabola Method over each subinterval are given in

Table 3. Results for Example 3.

i	Subinterval	TPP	Parabola	${\mathit{k}}_{\mathit{f}}$
	$[{\mathit{\nu }}_{\mathbf{1}}^{\mathit{i}},{\mathit{\nu }}_{\mathbf{3}}^{\mathit{i}}]$	$\{{\mathit{\nu }}_{\mathbf{1}}^{\mathit{i}},{\mathit{\nu }}_{\mathbf{2}}^{\mathit{i}},{\mathit{\nu }}_{\mathbf{3}}^{\mathit{i}}\}$
1	$[0,0.333]$	$\{0,0.166,0.333\}$	$15.206x^2-6.931x$	2
2	$[1,1.333]$	$\{1,1.166,0.333\}$	$5.590x^2-13.733x+8.144$	2
3	$[2,2.333]$	$\{2,2.166,2.333\}$	$2.055x^2-9.159x+10.101$	2
4	$[3,3.333]$	$\{3,3.166,3.333\}$	$0.755x^2-4.878x+7.837$	2

.

Therefore, in total, after 32 function evaluations, all minima of the given function were determined. The geometrical interpretation of the results of the TPP procedure application corresponding to Table 3 is presented in Figure 3.

Example 4.

Problem 7: $f\left(x\right)=sin\left(x\right)+sin\left(\frac{2x}{3}\right),P=[3.1,20.4]$. Initial value $N=3$. Three subintervals were determined after four runs of the TPP procedure, $k_p=3$ (see Figure 4). The results of the TPP procedure and the Parabola Method are given in

Table 4. Results for Example 4.

i	Subinterval	TPP	Parabola	${\mathit{k}}_{\mathit{f}}$
	$[{\mathit{\nu }}_{\mathbf{1}}^{\mathit{i}},{\mathit{\nu }}_{\mathbf{3}}^{\mathit{i}}]$	$\{{\mathit{\nu }}_{\mathbf{1}}^{\mathit{i}},{\mathit{\nu }}_{\mathbf{2}}^{\mathit{i}},{\mathit{\nu }}_{\mathbf{3}}^{\mathit{i}}\}$
1	$[4.542,5.983]$	$\{4.542,5.263,5.983\}$	$0.486x^2-5.238x+12.887$	4
2	$[9.588,11.029]$	$\{9.588,10.308,11.029\}$	$0.249x^2-5.181x+26.728$	5
3	$[16.075,17.517]$	$\{16.075,16.796,17.517\}$	$0.638x^2-21.713x+182.968$	3

.

After 36 function evaluations, all minima were detected.

Now, we apply the two-stage approach to the problems from Table 1 and Table 2. The results are given in

Table 5. Application of the two-stage approach to problems 1–5, 8–10, and 12–17.

Problem	Minima		TPP
No.	Structure	${\mathit{k}}_{\mathit{f}}^{\mathit{TPP}}$	Subintervals	${\mathit{k}}_{\mathit{f}}^{\mathit{GSM}}$	${\mathit{k}}_{\mathit{f}}^{\mathit{PM}}$	g/l
1.	2 local	12	$[8.917,11.0]$	16	6	g
	1 global
2.	2 local	12	$[3.100,3.900]$	14	4	l
	1 global		$[4.700,5.500]$	14	4	g
			$[6.700,7.500]$	14	4	l
3.	17 local	24	$[-7.500,-5.833]$	16	4	g
	3 global		$[-2.500,-0.833]$	16	5	l
			$[1.666,3.333]$	16	6	l
			$[5.000,6.667]$	16	5	g
4.	4 local	24	$[0.050,0.150]$	10	3	l
	1 global		$[0.350,0.450]$	10	3	l
			$[0.600,0.700]$	10	2	l
			$[0.900,1.000]$	10	3	g
5.	2 local	12	$[3.100,3.900]$	14	5	l
	1 global		$[4.700,5.500]$	14	4	g
			$[6.700,7.500]$	14	4	l
8.	1 local	12	$[0.833,2.500]$	16	5	l
	1 global		$[7.500,9.167]$	16	4	g
9.	1 local	6	$[1.047,3.663]$	17	8	g
	2 global
10.	1 local	6	$[2.093,4.187]$	16	4	g
	2 global
12.	1 local	6	$[1.666,5.000]$	17	14	g
	2 global
13.	1 local	12	$[-3.333,-2.000]$	15	7	g
	2 global		$[-0.667,0.667]$	15	1	l
			$[2.000,3.333]$	15	7	g
14.	3 local	24	$[1.354,1.896]$	14	4	l
	1 global		$[3.521,4.063]$	14	1	l
			$[5.687,6.229]$	14	4	g
15.	5 local	6	$[2.333,4.667]$	17	6	g
	1 global
16.	1 local	24	$[-1.750,-1.333]$	13	3	l
	1 global		$[1.375,1.792]$	13	4	g
17.	1 global	12	$[0.100,1.250]$	15	19	g

and

Table 6. Application of the two-stage approach to problem 6.

Problem	Minima		TPP
No.	Structure	${\mathit{k}}_{\mathit{f}}^{\mathit{TPP}}$	Subintervals	${\mathit{k}}_{\mathit{f}}^{\mathit{GSM}}$	${\mathit{k}}_{\mathit{f}}^{\mathit{PM}}$	g/l
6.	17 local	96	$[-9.583,-9.167]$	13	3	l
	3 global		$[-8.541,-8.125]$	13	3	l
			$[-7.291,-6.875]$	13	3	g
			$[-6.250,-5.833]$	13	4	l
			$[-5.208,-4.791]$	13	4	l
			$[-4.166,-3.750]$	13	4	l
			$[-3.125,-2.708]$	13	4	l
			$[-2.291,-1.875]$	13	4	l
			$[-1.042,-0.625]$	13	4	g
			$[0.208,0.625]$	13	4	l
			$[1.042,1.458]$	13	4	l
			$[2.083,2.500]$	13	3	l
			$[3.125,3.542]$	13	4	l
			$[4.166,4.583]$	13	2	l
			$[5.208,5.625]$	13	4	g
			$[6.458,6.875]$	13	3	l
			$[7.500,7.917]$	13	2	l
			$[8.333,8.750]$	13	4	l
			$[9.375,9.792]$	13	2	l

. Since problems 7 and 11 were considered in Examples 3 and 4, they are not included in the Table 5. Problem 6 is described in Table 5.

In these tables, the Minima structure column shows how many local and global minima each problem has, the $k_f^{TPP}$ column shows the number of performed function evaluations until the TPP procedure is stabilized, the TPP subintervals column shows the subintervals obtained from application of the TPP procedure, the $k_f^{GSM}$ column shows the number of function evaluations of the Golden Section Method, the $k_f^{PM}$ column shows the number of function evaluations of the Parabola Method, and the g/l column shows the type of the calculated point: g—global minimum, l—local minimum. For example, for problem 2 with two local minima and one global minima, the TPP procedure found three subintervals after 12 function evaluations. The first subinterval contains a local minimum, the second subinterval contains a global minimum, and the third subinterval contains a local minimum. The minimum over the first subinterval was found by the Golden Section method in 14 function evaluations and by the Parabola Method in 4 function evaluations. The same results from both methods were demonstrated over the second and the third subintervals. Therefore, the total number of function evaluations spent by the two-stage approach with the Golden Section Method at the second stage was $12+3\times 14=54$ and $12+3\times 4=24$ function evaluations with the Parabola Method.

Table 6 shows the results of the application of the two-stage approach to problem 6. Application of the TPP procedure finished in 19 subintervals, and for each of them, the Golden Section Method and the Parabola Method were used; all global minima were found, as well as all interior local minima, and one local minimum was attained at the endpoint of the feasible interval.

We can see from the presented test results that the direct application of the two-stage approach may involve a lot of computation. If we are interested in finding a good local solution faster than the pure Golden Section Method we can use the following Accelerated Two-Stage Approach (ATSA).

3. Accelerated Two-Stage Approach

In this section, we propose a modification of the two-stage approach described in the previous section. We set the integer parameter N in the TPP procedure in advance and do not change it. When the TPP procedure is finished, $N+1$ values of the objective function at points $x_k=\alpha +k\Delta ,k=0,\dots ,N$ with $\Delta =\frac{\beta -\alpha }{N}$ are available. We determine the record value $f^{Rec}=min\{f\left(x_i\right):i=0,\dots ,N\}$ and a corresponding record point $x^{Rec}\in \mathrm{Arg}min\{f\left(x_i\right):i=0,\dots ,N\}$. If the number $k_p$ of the TPP subintervals is positive, $k_p>0$, then we choose the TPP subinterval, which contains the record point $x^{Rec}$, and run the Parabola Method over this TPP subinterval. Let $x^{PM}$ be a point found by the Parabola Method. We define the point $x^{*}=argmin\{f\left(x^{Rec}\right),f\left(x^{PM}\right)$ and the corresponding objective value $f^{*}=f\left(x^{*}\right)$. We deliver the pair $(x^{*},f^{*})$ as the result of the Accelerated Two-Stage Approach. If the number $k_p$ is equal to zero, i.e., no TPP subintervals were detected, then we determine

$$m\in \{0,\dots ,N-1\}:x_m\le x^{Rec}\le x_{m+1},$$

(4)

and run the Golden Section Method over the interval $[x_m,x_{m+1}]$. Let $x^{GSM}$ be the corresponding point determined by the Golden Section Method. As in the previous case, we define the point $x^{*}=argmin\{f\left(x^{GSM}\right),f\left(xRec\right)$ and the value $f^{*}=f\left(x^{*}\right)$, and we deliver the pair $x^{*},f^{*}$, as a result of the Accelerated Two-Stage Approach (ATSA).

Let us now give the description of the ATSA procedure.

The ATSA Procedure

Step 1.

Apply the TPP procedure. Let $k_p$ and $\Delta$ be parameters calculated by the TPP procedure. Let $f^{Rec}$ be the record value over all calculated points and $x^{Rec}:f\left(x^{Rec}\right)=f^{Rec}$.

Step 2.

If $k_p>0$, then select the subinterval containing $x^{Rec}$ and run the Parabola Method over the selected subinterval, obtaining the point $x^{PM}$. Define the point $x^{*}=argmin\{f\left(x^{Rec}\right),f\left(x^{PM}\right)\}$ and value $f^{*}=f\left(x^{*}\right)$. Stop.

Step 3.

If $k_p=0$, then determine the subinterval $[x_m,x_{m+1}]$ according to (4) and run the Golden Section Method, obtaining the point $x^{GSM}$. Define the point $x^{*}=argmin\{f\left(x^{Rec}\right),f\left(x^{GSM}\right)\}$ and value $f^{*}=f\left(x^{*}\right)$. Stop.

The results of testing the accelerated two-stage approach are given in

Table 7. Results of application of the accelerated two-stage approach.

Problem No.	Number of TPP Subintervals, ${\mathit{k}}_{\mathit{p}}$	GSM/PM	Total ${\mathit{k}}_{\mathit{f}}$	g/l
1.	0	GSM, $k_f^{GSM}=17$	23	g
2.	2	PM, $k_f^{PM}=5$	11	g
3.	2	PM, $k_f^{PM}=9$	15	g
4.	2	PM, $k_f^{PM}=4$	10	g
5.	2	PM, $k_f^{PM}=5$	11	g
6.	1	PM, $k_f^{PM}=8$	14	g
7.	2	PM, $k_f^{PM}=4$	10	g
8.	2	PM, $k_f^{PM}=4$	10	g
9.	1	PM, $k_f^{PM}=2$	8	g
10.	1	PM, $k_f^{PM}=8$	14	g
11.	1	PM, $k_f^{PM}=6$	10	l
12.	1	PM, $k_f^{PM}=22$	28	g
13.	2	PM, $k_f^{PM}=17$	23	g
14.	2	PM, $k_f^{PM}=6$	10	l
15.	1	PM, $k_f^{PM}=6$	10	g
16.	0	GSM, $k_f^{GSM}=15$	21	g
17.	1	PM, $k_f^{PM}=21$	26	g

. The integer parameter N for the TPP procedure was equal to 3. The GSM/PM column shows the number of function evaluations of the corresponding method. If, after the first stage, the number of TPP subintervals was equal to zero (as for problem 1 $k_p=0$), then the Golden Section Method was used, and the number of function evaluations $k_f^{GSM}=17$. If, after the first stage, the number of the TPP subintervals was positive (as for problem 2 $k_p=2$), then the Parabola Method was used, and the number of function evaluations $k_f^{PM}=5$. The Total $k_f$ column shows the total number of function evaluations of the ATSA procedure, and the g/l column shows a global (g) or local (l) minimum was found.

We have to mention that the integer parameter N (the number of the subdivision points in the TPP procedure) is the most crucial. If, for example, N is equal to 20, then for all testing problems, global minimum points were determined. However, in this case, the numbers of objective function evaluations were rather large, more than several tens. The parameter N can be chosen according to the number of expected local minima. In the general case, we aim to find a good local minimum, and from the results of our testing, we recommend choosing an N between 5 and 10.

We can see from Table 7 that the accelerated two-stage approach finds solutions with lower computational efforts compared to the pure Golden Section Method, still finding global solutions in almost all the test problems. As for problems 12 and 17, some further modifications can be invented. Nevertheless, the current results are encouraging.

4. Numerical Comparison of Zero-Order Methods: Multivariate Optimization Problems

We tested and compared the following zero-order methods: the Hooke and Jeeves method combined with the ATSA, the Rosenbrock method with a discrete step in the search procedure, the Rosenbrock method combined with the ATSA, and the coordinate descent method combined with the ATSA. In the current section, we give a brief description of these methods. Many of them are described in monographs and review articles on optimization methods [10,11,15,16,17,18,19].

Hooke and Jeeves method. The pattern search method of Hooke and Jeeves consists of a sequence of exploratory moves about a base point that, if successful, are followed by pattern moves.

The exploratory moves acquire information about the function $f\left(x\right)$ in the neighborhood of the current base point $b^k=(x_1^k,\dots ,x_n^k)$. Each variable $x_i^k$, in turn, is given an increment ${\epsilon }_i$ (first, in the positive direction and then, if necessary, in the negative direction), and a check is made of the new function value. If any move results in a reduced function value, the new value of that variable will be retained. After all the variables have been considered, a new base point $b^{k+1}$ will be reached. If $b^{k+1}$ = $b^k$, no function reduction has been achieved. The step length ${\epsilon }_i$ is reduced, and the procedure is repeated. If $b^{k+1}\ne b^k$, a pattern move from $b^k$ is made.

A pattern move attempts to speed up the search by using the information already acquired about $f\left(x\right)$ to identify the best search direction. A move is made from $b^{k+1}$ in the direction $d=b^{k+1}-b^k$, since a move in this direction leads to a decrease in the function value. In this step, we use the ATSA to solve a univariate optimization problem $f\left(\lambda \right)={min}_{\lambda }f(b^{k+1}+\lambda d)$ and obtain a new point $p^k=b^{k+1}+{\lambda }^{★}d$. The search continues with a new sequence of exploratory moves about $p_k$. If the lowest function value obtained is less than $f\left(b^k\right)$, then a new base point $b^{k+2}$ has been reached. In this case, a second pattern move is made. If not, the pattern move from $b^{k+1}$ is abandoned, and we continue with a new sequence of exploratory moves about $b^{k+1}$.

Rosenbrock method. The main idea of the method is to iteratively find the descending direction of a function along linearly independent and orthogonal directions. A successful step in the current direction leads to an increase in this step on the following iteration by means of a stretch coefficient $\rho >0$; otherwise, the coefficient $0<\rho <1$ is used to decrease the step. The search within the current direction system is implemented until all possibilities of function reduction are exhausted. If there are no successful directions, a new set of linearly independent and orthogonal directions is constructed by means of rotating the previous ones in an appropriate manner. To obtain a new direction system, the Gram–Schmidt procedure is used.

Iterative coordinate descent method. This method is a simplified version of the Hooke and Jeeves method. It uses only the analogue of exploratory moves with an appropriate step size obtained from the line search procedure along the current basis vector.

Combined versions. We designed three modifications of the described methods to estimate the potential for finding the global optimum. The iterative coordinate descent method, the Hooke and Jeeves method, and the Rosenbrock method are combined with the ATSA and presented in this section. The following test problems were used to perform numerical experiments.

• Branin function

  $$f\left(x\right)={\left(x_2-\frac{5.1x_1^2}{4{\pi }^2}+\frac{5x_1}{\pi }-6\right)}^2+10\left(1-\frac{1}{8\pi }\right)cos{x}_1+10,$$

  $$X=\left\{x\in R^2:-5\le x_1\le 10,0\le x_2\le 15,\right\}.$$
  Global optimum: $x^{★}=(3.141593,2.275)$, $x^{★★}=(-3.141593,12.275)$, $f^{★}=0.397667$.
• Treccani function

  $$f\left(x\right)=x_1^4+4x_1^3+4x_1^2+x_2^2,$$

  $$X=\left\{x\in R^2:-3\le x_1\le 3,-3\le x_2\le 3,\right\}.$$
  Global optimum: $x^{★}=(0,0)$, $x^{★★}=(2,0)$, $f^{★}=0$.
• Shubert function

  $$f\left(x\right)=\left({\displaystyle \sum _{i=1}^5}icos((i+1)x_1+i)\right)·\left({\displaystyle \sum _{i=1}^5}icos((i+1)x_2+i)\right).$$

  $$X=\left\{x\in R^2:-10\le x_1\le 10,-10\le x_2\le 10,\right\}.$$
  The function has 760 local minimum points for $x\in X$, 18 of them are global optimum points, and $f^{*}=-186.730909$.
• 3-hump camel function

  $$f\left(x\right)=2x_1^2-1.05x_1^4+\frac{x_1^6}{6}-x_1{x}_2+x_2^2,$$

  $$X=\left\{x\in R^2:-3\le x_1\le 3,-3\le x_2\le 3,\right\}.$$
  This function has three minimums for $x\in X$, one of them is global: $x^{★}=(0,0)$, and $f^{★}=0$.
• 6-hump camel function

  $$f\left(x\right)=4x_1^2-2.1x_1^4+\frac{x_1^6}{3}+x_1{x}_2-4x_2^2+4x_2^4,$$

  $$X=\left\{x\in R^2:-3\le x_1\le 3,-1.5\le x_2\le 1.5,\right\}.$$
  The 6-hump camel function has six minimum points for $x\in X$, two of them are global optima: $x^{★}=(-0.089842,0.712656)$, $x^{★★}=(0.089842,-0.712656)$, and $f^{★}=-1.031628$.
• Rosenbrock function

  $$f\left(x\right)=100{\left(x_1^2-x_2\right)}^2+{(x_1-1)}^2,$$

  $$X=\left\{x\in R^2:-5\le x_1\le 5,-5\le x_2\le 5,\right\}.$$
  The only global minimum of the function is $x^{★}=(1,1)$, and $f^{★}=0$.
• Levy-1 function

  $$f\left(x\right)=\frac{\pi }{n}\left(10{sin}^2\left(\pi x_1\right)+{\displaystyle \sum _{i=1}^{n-1}}{(x_i-1)}^2[1+10{sin}^2\left(\pi x_{i+1}\right)]+{(x_n-1)}^2\right),$$

  $$X=\left\{x\in R^n:-10\le x\le 10\right\}.$$
  The function has approximately $5^n$ local minima for $x\in X$, the global minimum is $x_i^{★}=1$, $i=1,\dots ,n$, and $f^{★}=0$.
• Levy-2 function

  $$f\left(x\right)=\frac{\pi }{n}\left(10{sin}^2\left(\pi y_1\right)+{\displaystyle \sum _{i=1}^{n-1}}{(y_i-1)}^2[1+10{sin}^2\left(\pi y_{i+1}\right)]+{(y_n-1)}^2\right),$$

  $$y_i=\frac{x_i-1}{4}+1,$$

  $$X=\left\{x\in R^n:-10\le x\le 10\right\}.$$
  As with Levy-2 function, the function has approximately $5^n$ local minima for $x\in X$, the global minimum is $x_i^{★}=1$, $i=1,\dots ,n$, and $f^{★}=0$.
• Levy-3 function

  $$f\left(x\right)=\frac{1}{10}{sin}^2\left(3\pi x_1\right)+\frac{1}{10}{\displaystyle \sum _{i=1}^{n-1}}{(x_i-1)}^2\left(1+\mathrm{s}\mathrm{i}{\mathrm{n}}^2\left(3\pi x_{i+1}\right)\right)$$

  $$+\frac{1}{10}{(x_n-1)}^2\left(1+{sin}^2\left(2\pi x_n\right)\right),$$

  $$X=\left\{x\in R^n:-10\le x\le 10\right\}.$$
  This function has approximately ${30}^n$ local minima for $x\in X$, the only global minimum is $x_i^{★}=1$, $i=1,\dots ,n$, and $f^{★}=0$.

The results of the numerical experiments are presented in

Table 8. Coordinate descent method and Hooke–Jeeves method.

	Coordinate Descent–Par			Hooke–Jeeves–Par
Problem	${\mathit{f}}_{\mathit{best}}-{\mathit{f}}^{★}$	${\mathit{k}}_{\mathit{f}}$	$\mathit{m}/\mathit{M}$	${\mathit{f}}_{\mathit{best}}-{\mathit{f}}^{★}$	${\mathit{k}}_{\mathit{f}}$	$\mathit{m}/\mathit{M}$
Branin	${10}^{-4}$	1752	154/1000	${10}^{-4}$	18,117	64/200
Trecani	${10}^{-6}$	808	1000/1000	${10}^{-6}$	1318	688/1000
Shubert	${10}^{-6}$	1056	1000/1000	${10}^{-1}$	4451	33/1000
3-hump camel	${10}^{-6}$	1061	673/1000	${10}^{-6}$	12,172	33/200
6-hump camel	${10}^{-4}$	1828	1000/1000	${10}^{-2}$	13,515	1/200
Rosenbrock	${10}^{-3}$	37,768	95/1000	${10}^{-6}$	8306	26/200
Levy-1 (n = 5)	${10}^{-6}$	1323	735/1000	${10}^{-2}$	10,383	44/200
Levy-1 (n = 50)	${10}^{-6}$	12,543	780/1000	${10}^{-3}$	13,853	5/200
Levy-1 (n = 100)	${10}^{-6}$	25,535	750/1000	${10}^0$	14,950	0/200
Levy-2 (n = 5)	${10}^{-6}$	1339	1000/1000	${10}^{-3}$	49,081	14/50
Levy-2 (n = 50)	${10}^{-6}$	14,285	50/50	${10}^0$	35,751	12/50
Levy-2 (n = 100)	${10}^{-6}$	31,372	50/50	${10}^0$	40,945	11/50
Levy-3 (n = 5)	${10}^{-6}$	4864	1000/1000	${10}^{-1}$	33,975	3/50
Levy-3 (n = 50)	${10}^{-6}$	31,945	50/50	${10}^0$	16,244	11/50
Levy-3 (n = 100)	${10}^{-6}$	75,752	50/50	${10}^0$	46,882	0/50

and

Table 9. Rosenbrock methods.

	Rosenbrock–Dis			Rosenbrock–Par
Problem	${\mathit{f}}_{\mathit{best}}-{\mathit{f}}^{★}$	${\mathit{k}}_{\mathit{f}}$	$\mathit{m}/\mathit{M}$	${\mathit{f}}_{\mathit{best}}-{\mathit{f}}^{★}$	${\mathit{k}}_{\mathit{f}}$	$\mathit{m}/\mathit{M}$
Branin	${10}^{-4}$	296	200/200	${10}^{-4}$	9840	12/200
Trecani	${10}^{-6}$	115	200/200	${10}^{-6}$	7817	53/200
Shubert	${10}^{-5}$	4451	9/200	${10}^{-6}$	10,964	100/200
3-hump camel	${10}^{-6}$	127	85/200	${10}^{-6}$	11,107	22/200
6-hump camel	${10}^{-6}$	90	131/200	${10}^{-2}$	14,210	60/200
Rosenbrock	${10}^{-6}$	489	172/200	${10}^{-4}$	8201	15/200
Levy-1 (n = 5)	${10}^{-6}$	705	20/200	${10}^0$	33,418	1/200
Levy-1 (n = 50)	${10}^{-3}$	22,550	1/20	${10}^0$	241,667	0/20
Levy-1 (n = 100)	${10}^{-3}$	63,142	1/20	${10}^0$	576,160	0/20
Levy-2 (n = 5)	${10}^{-3}$	573	7/20	${10}^0$	39,955	1/20
Levy-2 (n = 50)	${10}^{-3}$	6765	3/20	${10}^0$	373,840	0/20
Levy-2 (n = 100)	${10}^{-3}$	17,145	4/20	${10}^0$	871,650	0/20
Levy-3 (n = 5)	${10}^{-1}$	33,975	4/50	${10}^{-1}$	49,301	1/20
Levy-3 (n = 50)	${10}^{-1}$	22,235	7/20	${10}^0$	398,060	0/20
Levy-3 (n = 100)	${10}^0$	42,040	1/20	${10}^0$	886,740	0/20

. The following notations are used: n is the number of variables; $f_{best}$ is the best value of the objective function found during the executing of the algorithm; $f^{★}$ is the optimal value of the objective function; $k_f$ is the average number of function calculations during the execution of the algorithm considering all launches of the current problem; m is the number of problems solved successfully using the multistart procedure; M is the total number of randomly generated points. The multistart procedure launches the algorithm from one of the generated points. The designation $m/M$ means that m launches of the algorithm from M starting points resulted in a successful problem solution (the global minimum point was found). Coordinate descent–par is the coordinate descent method in combination with the ATSA; Hooke–Jeeves–par is the Hooke and Jeeves method combined with the ATSA; Rosenbrock–dis is the Rosenbrock method with a discrete step in the search procedure; Rosenbrock–par is the Rosenbrock method in combination with the ATSA.

5. Discussion of the Results

The coordinate descent method in combination with the ATSA proved to be an effective method in searching for the global optimum even in quite difficult optimization problems. For instance, notice the results for Levy functions. Despite the large number of local minima in the search area of these functions, the algorithm in most cases found the global minimum points. The coordinate descent method in combination with the ATSA demonstrated an acceptable number of function calculations and quite high accuracy of the best function value for all tested problems.

The Hooke and Jeeves method combined with the ATSA attained global optimum points in most tested problems but not as frequently as the coordinate descent method. Nevertheless, it is possible to obtain quite a high accuracy of the best-found solution for some problems. The price for this accuracy is a large number of objective function calculations due to the careful selection of start points in the ATSA.

The Rosenbrock method with the ATSA has the same obvious drawback as the Hooke and Jeeves method, namely, the number of function calculations is quite large. However, we can notice that the Rosenbrock method with a discrete step demonstrated acceptable performance considering the number of successive launches and function calculations.

6. Conclusions

We tested some of the well-known zero-order methods and added to their algorithms a line search based on the ATSA for univariate problems. Some of the tested problems, for instance, the Shubert problem and Levy problems are quite difficult in terms of searching for the global optimum; however, according to the numerical experiments, it is possible to use zero-order methods to find the global optimum with quite high accuracy and with acceptable performance. The coordinate descent method combined with the ATSA deserves attention in terms of its ability to find a global optimum with high frequency in most tested problems.

It would be very interesting to combine the suggested Parabola Method with other zero-order methods like the Downhill Simplex, Genetic Algorithm, Particle Swarm Optimization, Cuckoo Search Algorithm, and the YUKI algorithm. We will continue our investigations and are working on a new paper devoted to such extension and testing.