Toward an Ideal Particle Swarm Optimizer for Multidimensional Functions

Abstract

The Particle Swarm Optimization (PSO) method is a global optimization technique based on the gradual evolution of a population of solutions called particles. The method evolves the particles based on both the best position of each of them in the past and the best position of the whole. Due to its simplicity, the method has found application in many scientific areas, and for this reason, during the last few years, many modifications have been presented. This paper introduces three modifications to the method that aim to reduce the required number of function calls while maintaining the accuracy of the method in locating the global minimum. These modifications affect important components of the method, such as how fast the particles change or even how the method is terminated. The above modifications were tested on a number of known universal optimization problems from the relevant literature, and the results were compared with similar techniques.

1. Introduction

The global optimization problem is usually defined as:

$$x^{*}=\arg \underset{x\in S}{min}f\left(x\right)$$

(1)

with S:

$$S=\left[a_1,b_1\right]\otimes \left[a_2,b_2\right]\otimes \dots \left[a_n,b_n\right]$$

The function f is considered a continuous and differentiable function, which is formulated as $f:S\to R,S\subset R^n$. This problem finds application in a variety of objective problems in the real world, such as problems of physics [1,2,3], chemistry [4,5,6], economics [7,8], medicine [9,10], etc. Global optimization methods are grouped into two broad categories: deterministic and stochastic methods. The most common methods of the first category are the so-called Interval methods [11,12], where the set S is divided iteratively into subregions, and some subregions that do not contain the global solution are discarded using some pre-defined criteria. In stochastic methods, the finding of the global minimum is guided by randomness. In these methods, there is no guarantee to find the global minimum, but they constitute the vast majority of global optimization methods, which is mainly due to the simplicity in their implementation. There have been proposed many stochastic methods by various researchers such as Controlled Random Search methods [13,14,15], Simulated Annealing methods [16,17,18], Differential Evolution methods [19,20], Particle Swarm Optimization methods [21,22,23], Ant Colony Optimization [24,25], Genetic Algorithms [26,27,28], etc. In addition, many works have appeared utilizing the modern GPU processing units [29,30,31]. The reader can find some complete surveys about metaheuristic algorithms in some recent works [32,33,34].

The method of Particle Swarm Optimization is a method based on a population of candidate solutions that are also called particles. These particles have two basic characteristics: their position at any given time $\overrightarrow{x}$ and the speed $\overrightarrow{u}$ at which they moved. The position $x_i$ of every particle can be defined as the summation of the current position $x_i$ and the associated velocity $u_i$ : $x_i=x_i+u_i$, where the velocity $u_i$ is defined as as a combination of the old velocity and the best located values $p_i$ and $p_{\mathrm{best}}$:

$$u_i=\omega u_i+r_1{c}_1\left(p_i-x_i\right)+r_2{c}_2\left(p_{\mathrm{best}}-x_i\right)$$

(2)

where $r_1$ and $r_2$ are random numbers and $c_1,c_2$ are user-defined constants. The vaue $\omega$ is called the inertia of the algorithm and usually is calculated using some efficient algorithm. The purpose of the method is to move the particles repetitively, and their next position is calculated as a function not only of their position but also of the best position they had in the past as well as the best position of the population. Various researchers have provided an analytic review of the PSO process such as the review of Jain et al. [35] where 52 papers have been reviewed, the work of Khare et al. [36] where a systematic review of the PSO algorithm is provided along with the application of PSO in solar photovoltaic systems. The method was successfully used in a variety of scientific and practical problems in areas such as physics [37,38], chemistry [39,40], medicine [41,42], economics [43], etc.

Many researchers have proposed a variety of modifications in the PSO method during the last few years; such methods aimed to estimate a more efficient calculation for the inertia parameter $\omega$ of the speed calculation [44,45,46]. The majority of methods used to calculate the inertia value calculate the inertia as a time-varying parameter, and hence, there is lack of diversity, which could be used to better explore the search space of the objective function. This paper introduces a new scheme for the calculation of inertia which takes into account the speed of the algorithm to locate new local minima in the search space of the objective function.

Other modifications of the PSO are used to alter the calculation of the factor $c_{1,}{c}_2$, which usually are called acceleration factors: for example, the work of Ratnaweera et al. [47], where a time-varying mechanism is proposed for the calculation of these factors. In addition, the PSO algorithm has been combined with the mutation mechanism [48,49,50] in order to improve the diversity of the algorithm and to better explore the search space of the objective function. Engelbrecht [51] proposed a novel method for the initialization of the velocity vector, where the velocity is initialized randomly rather than with zero value. In addition, in the relevant literature, there have been hybrid techniques [52,53,54], parallel techniques [55,56,57], methods aim to improve the velocity calculation [58,59,60], etc.

The method of PSO has been integrated into other optimization techniques. For example, Bogdanova et al. [61] combined Grammatical Evolution [62] with swarm techniques such as PSO. Their work is divided in two phases: during the first phase, the PSO method is decomposed into a list of basic grammatical rules. During the second phase, instances of the PSO method are created using the previous rules. Another interesting combination with an optimization technique is the work of Pan et al. [63], who combined the PSO method with simulated annealing to improve the local search ability of the PSO method. In addition, Mughal et al. [64] used a hybrid technique of PSO and simulated annealing for photovoltaic cell parameter estimation. Similarly, the work of Lin et al. [65] utilized a hybrid method of PSO and differential evolution for numerical optimization problems. In addition, Epitropakis et al. [66] suggested a hybrid PSO method with differential evolution, where the at each step of PSO, a differential evolution is applied to the best particle of the population. Among others, Wang et al. proposed a new PSO method based on chaotic neighbourhood search [67].

The PSO method is an iterative process, during which a series of particles evolve through a process that involves updating the position of the particles and fitness computation, i.e., evaluation of the objective function. Variations of PSO that aim at the global minimum in a shorter time may include the use of a local optimization method in each iteration of the algorithm. Of course, the above process can be extremely time consuming, and depending on the termination method used and the number of local searches performed, it may require a long execution time.

This text introduces three distinct modifications to the original method, which drastically improve the time required to find the total minimum by reducing the required number of function evaluations. These modifications cover a large part of the method: the speed calculation, a new method of avoiding running local search methods and a new adaptive termination rule. The first modification is used to enable the method to explore the search space more efficiently by calculating the inertia parameter with respect to the ability of the PSO to discover the local minima of the objective function. The second modification is used to avoid finding the same local minimum over and over again by the method. This way, on the one hand, the algorithm will not be trapped in local minima, and on the other hand, calls to the objective function will not be wasted, which will lead with some certainty to minima that have already been found. The third modification introduces a novel termination rule based on asymptotic calculations. Termination of methods such as PSO is an important process, as efficient termination methods will terminate the method in time without wasting many calls on the objective function, but at the same time, they should guarantee the ability of the method to find the global minimum.

The proposed modifications were applied to a number of problems from the relevant literature and compared with similar techniques, and the results are presented in a separate section. From the experiments performed, it was shown that the proposed modifications significantly reduce the required execution time of the method by drastically reducing the number of function calls required to find the total minimum. In addition, these modifications can be applied either alone or in combination with the same positive results. This means that they are quite general and can be included in other techniques based on PSO.

The rest of this article is divided as follows: in Section 2, the initial method and the proposed modifications are discussed, in Section 3, the experiments are listed, and finally, in Section 4, some conclusions and guidelines for future improvements are presented.

2. Method Description

The base algorithm of PSO and the proposed modifications are outlined in detail in the following subsections. The discussion starts with a new mechanism that calculates the velocity of the populations, continues with a discarding procedure used to minimize the number of local searches performed, and ends with a discussion about the new stopping rule proposed here.

2.1. The Base Algorithm

The base Algorithm 1 is listed below with the periodical application of the local search method in order to enhance the estimation of the global minimum, i.e., at every iteration, a decision with probability $p_l$ is made in order to apply a local search procedure to some of the particles. Usually, this probability is small, for example 0.05 (5%).

Algorithm 1 The base algorithm of PSO and the proposed modifications

1.  Initialization.             (a)   Set $\mathrm{iter}=0$ (iteration counter).             (b)   Set the number of particles m.             (c)   Set the maximum number of iterations allowed ${\mathrm{iter}}_{\max }$.             (d)   Set the local search rate $p_l\in [0,1]$.             (e)   Initialize randomly the positions of the m particles $x_1,x_2,\dots ,x_m$, with $x_i\in S\subset R^n$.             (f)   Initialize randomly the velocities of the m particles $u_1,u_2,\dots ,u_m$, with $u_i\in S\subset R^n$.             (g)   For $i=1..m$ do $p_i=x_i$. The $p_i$ vectors are the best located values for every particle i.             (h)   Set $p_{\mathrm{best}}=arg{min}_{i\in 1..m}f\left(x_i\right)$.     2.  Termination Check. Check for termination. If termination criteria are met, then stop.     3.  For $i=1..m$ Do             (a)   Update the velocity $u_i$ as a function of $u_i,p_i$ and $p_{\mathrm{best}}$.             (b)   Update the position $x_i=x_i+u_i$.             (c)   Set $r\in [0,1]$ a random number. If $r\le p_m$ then $x_i=\mathrm{LS}\left(x_i\right)$, where $\mathrm{LS}\left(x\right)$ is a local search procedure.             (d)   Evaluate the fitness of the particle i, $f\left(x_i\right)$.             (e)   If $f\left(x_i\right)\le f\left(p_i\right)$ then $p_i=x_i$.     4.  End For     5.  Set $p_{\mathrm{best}}=arg{min}_{i\in 1..m}f\left(x_i\right)$.     6.  Set $\mathrm{iter}=\mathrm{iter}+1$.     7.  Goto Step 2.

The current work modifies the above algorithm in three key points:

1.

In Step 2, a new termination rule based on asymptotic considerations is introduced.

2.

In Step 3b, the algorithm calculates the new position of the particles. The proposed methodology modifies the position of the particles based on the average speed of the algorithm to discover new minimums.

3.

In Step 3c, a method based on gradient calculations will be used to prevent the PSO method from executing unnecessary local searches.

2.2. Velocity Calculation

The algorithm of Section 2.1 calculates at every iteration the new position $x_i$, which is calculated using the old position $x_i$, and the associated velocity $u_i$ according to the scheme:

$$x_i=x_i+u_i$$

(3)

Typically, the velocity is calculated as a combination of the old velocity and the best located values $p_i$ and $p_{\mathrm{best}}$ and may be defined as:

$$u_i=\omega u_i+r_1{c}_1\left(p_i-x_i\right)+r_2{c}_2\left(p_{\mathrm{best}}-x_i\right)$$

(4)

where

1.

The parameters $r_1,r_2$ are random numbers with $r_1\in [0,1]$ and $r_2\in [0,1]$.

2.

The constant numbers $c_1,c_2$ are in the range $[1,2]$.

3.

The variable $\omega$ is called inertia, with $\omega \in [0,1]$.

The inertia was proposed by Shi and Eberhart [21]. They argued that high values of the inertia coefficient cause better exploration of the search area, while small values of this variable are used when we want to achieve better local research around promising areas for the global minimum. The value of the inertia factor generally starts with large values and decreases with repetition. This article proposes a new adaptive technique for the inertia parameter, and this mechanism is compared against three others from the relevant literature.

2.2.1. Random Inertia

The inertia calculation is proposed in [68], and it is defined as

$${\omega }_{\mathrm{iter}}=0.5+\frac{r}{2}$$

(5)

where r is a random number with $r\in [0,1]$. This inertia calculation will be called I1 in the tables with the experimental results.

2.2.2. Linear Time-Varying Inertia (Min Version)

This inertia schema has been proposed and used in various studies [68,69,70], and it is defined as:

$${\omega }_{\mathrm{iter}}=\frac{{\mathrm{iter}}_{\max }-\mathrm{iter}}{{\mathrm{iter}}_{\max }}\left({\omega }_{\max }-{\omega }_{\min }\right)+{\omega }_{\min }$$

(6)

where ${\omega }_{\min }$ is the minimum value of inertia and ${\omega }_{\max }$ the maximum value for inertia. This inertia calculation will be called I2 in the tables with the experimental results.

2.2.3. Linear Time-Varying Inertia (Max Version)

This method is proposed in [71,72], and it is defined as:

$${\omega }_{\mathrm{iter}}=\frac{{\mathrm{iter}}_{\max }-\mathrm{iter}}{{\mathrm{iter}}_{\max }}\left({\omega }_{\min }-{\omega }_{\max }\right)+{\omega }_{\max }$$

(7)

This inertia calculation will be called I3 in the tables with the experimental results.

2.2.4. Proposed Technique

This calculation of inertia involves the number of iterations where the method manages to find a new minimum. In the first iterations and when the method has to do more exploration of the research area, the inertia will be great. When the method should focus on a minimum, then the inertia will decrease. For this reason, at every iteration $\mathrm{iter}$, the quantity

$${\delta }^{\left(\mathrm{iter}\right)}=\left|\sum _{i=1}^m\left|f_i^{\left(\mathrm{iter}\right)}\right|-\sum _{i=1}^m\left|f_i^{(\mathrm{iter}-1)}\right|\right|$$

(8)

is measured. In the first steps of the algorithm, this quantity will change from repetition to repetition at a fast pace, and at some point, it will no longer change at the same rate or will be zero.

Hence, we define a metric to model the changes in ${\delta }^{\left(\mathrm{iter}\right)}$ as

$${\zeta }^{\left(\mathrm{iter}\right)}=\left\{\begin{array}{cc}1,\hfill & {\delta }^{\left(i\right)}=0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(9)

Using this observation, two additional metrics are created, $S_{\delta }^{\left(\mathrm{iter}\right)}$ and $C_{\delta }^{\left(\mathrm{iter}\right)}$. The metric $S_{\delta }^{\left(\mathrm{iter}\right)}$ is given by

$$S_{\delta }^{\left(\mathrm{iter}\right)}=\sum _{i=1}^{\mathrm{iter}}{\zeta }^{\left(i\right)}$$

(10)

and the metric $C_{\delta }$ is given by:

$$C_{\delta }^{\left(\mathrm{iter}\right)}=\frac{S_{\delta }^{(\mathrm{iter})}}{\mathrm{iter}}$$

(11)

The following definition for the inertia calculation is proposed:

$${\omega }_{\mathrm{iter}}={\omega }_{\max }-\frac{\mathrm{iter}}{C_{\delta }^{\left(\mathrm{iter}\right)}}\left({\omega }_{\max }-{\omega }_{\min }\right)$$

(12)

This mechanism will be called IP in the tables with the experimental results.

2.3. The Discarding Procedure

The method in each iteration performs under the condition of a series of local searches. However, these searches will often either lead to local minima already found or locate values far below the global minimum. This means that much of the computing time will be wasted on actions that could have been avoided. In order to be able to group points that would lead by local search to the same local minimum, we introduce the concept of cluster, which refers to a set of points that are believed, under some asymptotic considerations, to belong to the same region of attraction of the function. The region of attraction for a local minimum $x^{*}$ is defined as:

$$A\left(x^{*}\right)=\left\{x:x\in S\subset R^n,\mathrm{LS}\left(x\right)=x^{*}\right\}$$

(13)

where $\mathrm{LS}\left(x\right)$ is a local search procedure that starts from a given point x and terminates when a local minimum is discovered. The discarding procedure suggested here prevents the method from starting a local search from a point x if that point belongs to the same region of attraction with other points. This procedure is composed by two two major parts:

1.

The first part is the so-called typical distance, which is a measure calculated after every local search, and it is given by

$$r_C=\frac{1}{M}\sum _{i=1}^M∥x_i-x_{iL}∥$$

(14)

where the local search procedure $\mathrm{LS}\left(x\right)$ initiates from $x_i$ and $x_{iL}$ is the outcome of $\mathrm{LS}\left(x_i\right)$. This measure has been used also in [73]. If a point x is close enough to an already discovered local minima, then it is highly possible that the point belongs to the so-called region of attraction of the minima.

2.

The second part is a check using the gradient values between a candidate starting point and an already discovered local minimum. The function value $f\left(x\right)$ near to some local minimum z can be calculated using:

$$f\left(x\right)\simeq f\left(z\right)+\frac{1}{2}{\left(x-z\right)}^{T}B\left(x-z\right)$$

(15)

where B is the Hessian matrix at the minimum $z.$ By taking gradients in both sides of Equation (15), we obtain:

$$\nabla f\left(x\right)\simeq B\left(x-z\right)$$

(16)

Of course, Equation (16) holds for any other point y near to z

$$\nabla f\left(y\right)\simeq B\left(y-z\right)$$

(17)

By subtracting Equation (17) from Equation (16) and by multiplying with ${\left(x-y\right)}^T$, we have the following equation:

$${\left(x-y\right)}^T\left(\nabla f\left(x\right)-\nabla f\left(y\right)\right)\simeq {\left(x-y\right)}^{T}B{\left(x-y\right)}^T>0$$

(18)

Hence, a candidate start point x can be rejected if

$$∥x-z∥\le r_C\mathrm{AND}{\left(x-y\right)}^T\left(\nabla f\left(x\right)-\nabla f\left(z\right)\right)$$

(19)

for any already discovered local minimum z.

2.4. Stopping Rule

A common used way to terminate a global optimization method is to utilize a maximum number of allowed iterations ${\mathrm{iter}}_{\max }$, i.e., stop when $\mathrm{iter}\ge {\mathrm{iter}}_{\max }$. Although it is a simple criterion, it is not an efficient one, since if ${\mathrm{iter}}_{\max }$ is too small, then the algorithm will terminate without locating the global optimum. In addition, when ${\mathrm{iter}}_{\max }$ is too high, the optimization algorithm will spend computation time in unnecessary function calls. In this paper, a new termination rule is proposed to terminate the PSO process, and it is compared against two other methods used in various optimization methods.

2.4.1. Ali’s Stopping Method

A method is proposed in the work of Ali and Kaelo [74] where at every generation, the measure

$$\alpha =\left|f_{\max }-f_{\min }\right|$$

(20)

is calculated. The $f_{\max }$ stands for the maximum function value of the population and $f_{\min }$ represents the lowest function value of the population. The method will terminate when

$$\alpha \le ϵ$$

(21)

where $ϵ$ is a predefined small positive value, for example $ϵ={10}^{-3}$.

2.4.2. Double Box Method

The second method utilized is a method initially proposed by [75]. In this method, we denote with ${\sigma }^{\left(\mathrm{iter}\right)}$ the variance of $f_{\min }$ calculated at iteration $\mathrm{iter}$. If the algorithm cannot locate a new lower value for $f_{\min }$ for a number of iterations, then the global minimum has already located, and the algorithm should terminate when

$${\sigma }^{\left(\mathrm{iter}\right)}\le \frac{{\sigma }^{\left({\mathrm{iter}}_{\mathrm{last}}\right)}}{2}$$

(22)

where ${\mathrm{iter}}_{\mathrm{last}}$ stands for the last iteration where a new lower value for $f_{\min }$ was discovered.

2.4.3. Proposed Technique

In the proposed termination technique, in each iteration k, the difference between the current best value and the previous best value is measured, i.e., the difference $\left|f_{\min }^{\left(k\right)}-f_{\min }^{(k-1)}\right|.$ If this difference is zero for a predefined number of iterations $k_{\max }$, then the method terminates.

3. Experiments

To measure the effect of the proposed modifications on the original method, a series of experiments were performed on test functions from the relevant literature [76,77], and they have been used widely by various researchers [74,78,79,80]. The experiments evaluated both the effect of the new method of calculating inertia as well as the criterion for avoiding local minima as well as the new termination rule. The experiments were recorded in separate tables, so that it is more possible to understand the effect of each modification separately.

3.1. Test Functions

The definition of the test functions used are given below

• Bf1 (Bohachevsky 1) function:

$$f\left(x\right)=x_1^2+2x_2^2-\frac{3}{10}cos\left(3\pi x_1\right)-\frac{4}{10}cos\left(4\pi x_2\right)+\frac{7}{10}$$

with $x\in {[-100,100]}^2$.

• Bf2 (Bohachevsky 2) function:

  $$f\left(x\right)=x_1^2+2x_2^2-\frac{3}{10}cos\left(3\pi x_1\right)cos\left(4\pi x_2\right)+\frac{3}{10}$$

  with $x\in {[-50,50]}^2$.
• Branin function: $f\left(x\right)={\left(x_2-\frac{5.1}{4{\pi }^2}{x}_1^2+\frac{5}{\pi }{x}_1-6\right)}^2+10\left(1-\frac{1}{8\pi }\right)cos\left(x_1\right)+10$ with $-5\le x_1\le 10,0\le x_2\le 15$. with $x\in {[-10,10]}^2$.
• CM function:

  $$f\left(x\right)=\sum _{i=1}^n{x}_i^2-\frac{1}{10}\sum _{i=1}^{n}cos\left(5\pi x_i\right)$$

  where $x\in {[-1,1]}^n$. In the current experiments, we used $n=4$.
• Camel function:

  $$f\left(x\right)=4x_1^2-2.1x_1^4+\frac{1}{3}{x}_1^6+x_1{x}_2-4x_2^2+4x_2^4,x\in {[-5,5]}^2$$
• Easom function:

  $$f\left(x\right)=-cos\left(x_1\right)cos\left(x_2\right)exp\left({\left(x_2-\pi \right)}^2-{\left(x_1-\pi \right)}^2\right)$$

  with $x\in {[-100,100]}^2.$
• Exponential function, defined as:

  $$f\left(x\right)=-exp\left(-0.5\sum _{i=1}^n{x}_i^2\right),-1\le x_i\le 1$$
  In the current experiments, we used this function with $n=2,4,8,16,32$.
• Goldstein and Price function

  $$\begin{array}{ccc}\hfill f\left(x\right)& =& \left[1+{\left(x_1+x_2+1\right)}^2\right.\hfill \\ & & \left(19-14x_1+3x_1^2-14x_2+6x_1{x}_2+3x_2^2\right)]\times \hfill \\ & & [30+{\left(2x_1-3x_2\right)}^2\hfill \\ & & \left(18-32x_1+12x_1^2+48x_2-36x_1{x}_2+27x_2^2\right)]\hfill \end{array}$$

  with $x\in {[-2,2]}^2$.
• Griewank 2 function:

  $$f\left(x\right)=1+\frac{1}{200}\sum _{i=1}^2{x}_i^2-\prod _{i=1}^2\frac{cos\left(x_i\right)}{\sqrt{\left(i\right)}},x\in {[-100,100]}^2$$
• Gkls function. $f\left(x\right)=\mathrm{Gkls}(x,n,w)$, is a function with w local minima, which was described in [81] with $x\in {[-1,1]}^n$ and n being a positive integer between 2 and 100. The value of the global minimum is -1, and in our experiments, we have used $n=2,3$ and $w=50,100$.
• Hansen function: $f\left(x\right)={\sum }_{i=1}^{5}icos\left[(i-1)x_1+i\right]{\sum }_{j=1}^{5}jcos\left[(j+1)x_2+j\right]$,
  $x\in {[-10,10]}^2$.
• Hartman 3 function:

  $$f\left(x\right)=-\sum _{i=1}^4{c}_{i}exp\left(-\sum _{j=1}^3{a}_{ij}{\left(x_j-p_{ij}\right)}^2\right)$$

  with $x\in {[0,1]}^3$ and $a=\left(\begin{array}{ccc}3& 10& 30\\ 0.1& 10& 35\\ 3& 10& 30\\ 0.1& 10& 35\end{array}\right),c=\left(\begin{array}{c}1\\ 1.2\\ 3\\ 3.2\end{array}\right)$ and

  $$p=\left(\begin{array}{ccc}0.3689& 0.117& 0.2673\\ 0.4699& 0.4387& 0.747\\ 0.1091& 0.8732& 0.5547\\ 0.03815& 0.5743& 0.8828\end{array}\right)$$
• Hartman 6 function:

  $$f\left(x\right)=-\sum _{i=1}^4{c}_{i}exp\left(-\sum _{j=1}^6{a}_{ij}{\left(x_j-p_{ij}\right)}^2\right)$$

  with $x\in {[0,1]}^6$ and $a=\left(\begin{array}{cccccc}10& 3& 17& 3.5& 1.7& 8\\ 0.05& 10& 17& 0.1& 8& 14\\ 3& 3.5& 1.7& 10& 17& 8\\ 17& 8& 0.05& 10& 0.1& 14\end{array}\right),c=\left(\begin{array}{c}1\\ 1.2\\ 3\\ 3.2\end{array}\right)$ and

  $$p=\left(\begin{array}{cccccc}0.1312& 0.1696& 0.5569& 0.0124& 0.8283& 0.5886\\ 0.2329& 0.4135& 0.8307& 0.3736& 0.1004& 0.9991\\ 0.2348& 0.1451& 0.3522& 0.2883& 0.3047& 0.6650\\ 0.4047& 0.8828& 0.8732& 0.5743& 0.1091& 0.0381\end{array}\right)$$
• Potential function. The molecular conformation corresponding to the global minimum of the energy of N atoms interacting via the Lennard–Jones potential [82] is used as a test function here, and it is defined by:

  $$V_{LJ}\left(r\right)=4ϵ\left[{\left(\frac{\sigma }{r}\right)}^{12}-{\left(\frac{\sigma }{r}\right)}^6\right]$$

  (23)
  For our experiments, we used: $N=3,4,5$.
• Rastrigin function.

  $$f\left(x\right)=x_1^2+x_2^2-cos\left(18x_1\right)-cos\left(18x_2\right),x\in {[-1,1]}^2$$
• Rosenbrock function.

  $$f\left(x\right)=\sum _{i=1}^{n-1}\left(100{\left(x_{i+1}-x_i^2\right)}^2+{\left(x_i-1\right)}^2\right),-30\le x_i\le 30.$$
  In our experiments, we used the values $n=4,8,16$.
• Shekel 7 function.

$$f\left(x\right)=-\sum _{i=1}^7\frac{1}{(x-a_i){(x-a_i)}^T+c_i}$$

with $x\in {[0,10]}^4$ and $a=\left(\begin{array}{cccc}4& 4& 4& 4\\ 1& 1& 1& 1\\ 8& 8& 8& 8\\ 6& 6& 6& 6\\ 3& 7& 3& 7\\ 2& 9& 2& 9\\ 5& 3& 5& 3\end{array}\right),c=\left(\begin{array}{c}0.1\\ 0.2\\ 0.2\\ 0.4\\ 0.4\\ 0.6\\ 0.3\end{array}\right)$.

• Shekel 5 function.

$$f\left(x\right)=-\sum _{i=1}^5\frac{1}{(x-a_i){(x-a_i)}^T+c_i}$$

with $x\in {[0,10]}^4$ and $a=\left(\begin{array}{cccc}4& 4& 4& 4\\ 1& 1& 1& 1\\ 8& 8& 8& 8\\ 6& 6& 6& 6\\ 3& 7& 3& 7\end{array}\right),c=\left(\begin{array}{c}0.1\\ 0.2\\ 0.2\\ 0.4\\ 0.4\end{array}\right)$.

• Shekel 10 function.

$$f\left(x\right)=-\sum _{i=1}^{10}\frac{1}{(x-a_i){(x-a_i)}^T+c_i}$$

with $x\in {[0,10]}^4$ and $a=\left(\begin{array}{cccc}4& 4& 4& 4\\ 1& 1& 1& 1\\ 8& 8& 8& 8\\ 6& 6& 6& 6\\ 3& 7& 3& 7\\ 2& 9& 2& 9\\ 5& 5& 3& 3\\ 8& 1& 8& 1\\ 6& 2& 6& 2\\ 7& 3.6& 7& 3.6\end{array}\right),c=\left(\begin{array}{c}0.1\\ 0.2\\ 0.2\\ 0.4\\ 0.4\\ 0.6\\ 0.3\\ 0.7\\ 0.5\\ 0.6\end{array}\right)$.

• Sinusoidal function:

  $$f\left(x\right)=-\left(2.5\prod _{i=1}^{n}sin\left(x_i-z\right)+\prod _{i=1}^{n}sin\left(5\left(x_i-z\right)\right)\right),0\le x_i\le \pi .$$
  The case of $n=4,8,16,32$ and $z=\frac{\pi }{6}$ was used in the experimental results.
• Test2N function:

  $$f\left(x\right)=\frac{1}{2}\sum _{i=1}^n{x}_i^4-16x_i^2+5x_i,x_i\in [-5,5].$$
  The function has $2^n$ in the specified range and in our experiments we used $n=4,5,6,7$.
• Test30N function:

  $$f\left(x\right)=\frac{1}{10}{sin}^2\left(3\pi x_1\right)\sum _{i=2}^{n-1}\left({\left(x_i-1\right)}^2\left(1+{sin}^2\left(3\pi x_{i+1}\right)\right)\right)+{\left(x_n-1\right)}^2\left(1+{sin}^2\left(2\pi x_n\right)\right)$$

  with $x\in [-10,10]$, with ${30}^n$ local minima in the search space. For our experiments, we used $n=3,4$.

3.2. Experimental Setup

All the experiments have been performed 30 times using a different seed for the random number generator each. The code has been implemented in ANSI C++, and the well-known function drand48() was used to produce random numbers. The local search method used was the BFGS method [83]. All the parameters used in the conducted experiments are listed in

Table 1. Values for the experimental parameters.

Parameter	Value
m	100
${\mathrm{iter}}_{\max }$	100
$p_l$	0.05
$c_1$	1.0
$c_2$	1.0
${\omega }_{\min }$	0.4
${\omega }_{\max }$	0.9
$ϵ$	0.001
$k_{\max }$	15

.

3.3. Experimental Results

For every stopping rule, two tables are listed: the first one contains experiments with the relevant stopping rule without the gradient discarding procedure, and the second table contains experiments with the gradient discarding procedure enabled. The numbers in table cells stand for average function calls. The fraction in parentheses denotes the fraction of runs where the global optimum was found. The absence of these fractions means that the global minimum was discovered in every run (100% success). The experimental results using the stopping rule of Equation (21) are listed in

Table 2. Experiments with the Ali stopping rule without gradient check.

FUNCTION	I1	I2	I3	IP
BF1	24,929	22,874	18,739	22,088
BF2	24,043	22,254	17,172	20,743
BRANIN	17,691	16,205	13,397	12,471
CM4	20,117	22,568	26,867	14,941
CAMEL	19,474	17,813	14,461	13,492
EASOM	13,327	13,106	9969	9212
EXP2	6339	8243	7853	3501
EXP4	7816	10,066	10,900	4458
EXP8	8667	10,937	13,126	4761
EXP16	8748	11,402	15,754	5098
EXP32	9567	12,323	18,189	5471
GKLS250	10,907	12,562	9673	8552
GKLS2100	12,960	13,403	9930	9541
GKLS350	15,410(0.97)	14,722	10,542	9298
GKLS3100	16,639	14,495	10,412	13,075(0.97)
GOLDSTEIN	20,437	22,877	16,410	8935
GRIEWANK2	27,620	24,230	18,473	20,133
HANSEN	21,513	20,279	16,326	15,046
HARTMAN3	16,233	17,152	12,305	6511
HARTMAN6	47,038	48,947	46,852	23,431
POTENTIAL3	31,684	32,175	36,930	24,463
POTENTIAL4	184,602	181,231	168,962	129,267
POTENTIAL5	74,508	70,519	76,890	54,042
RASTRIGIN	23,574	20,865	15,596	16,198
ROSENBROCK4	145,178	161,136	160,341	129,891
ROSENBROCK8	95,290	97,035	96,687	80,408
ROSENBCROK16	118,614	116,454	115,122	97,004
SHEKEL5	27,458	27,088	25,927	18,036
SHEKEL7	27,521	27,271	25,967	18,805
SHEKEL10	29,699(0.97)	28,082	25,511	20,823
TEST2N4	26,740	27,050	22,905	19,495
TEST2N5	20,243(0.97)	20,290	17,729	16,024(0.97)
TEST2N6	33,118	33,366	30,118	25,235(0.93)
TEST2N7	23,266(0.90)	22,804	21,294	18,218(0.90)
SINU4	17,035	20,487	18,971	11,079
SINU8	22,827	27,176	27,732	12,379
SINU16	31,055	35,998	42,984	15,692
SINU32	44,736(0.97)	51,624	82,114	25,991
TEST30N3	18,733	20,119	17,803	17,543
TEST30N4	20,348	22,191	20,679	20,085
TOTAL	1,365,704(0.99)	1,399,419	1,367,612	1,001,436(0.99)

and

Table 3. Experiments with the Ali stopping rule with the gradient check enabled.

FUNCTION	I1	I2	I3	IP
BF1	9709	8918	9531	10,932
BF2	10,196	9588	9089	10,730
BRANIN	10,718	9597	9813	9501
CM4	6242	7503	12,531	6985
CAMEL	10,422	9306	8491	9624
EASOM	11,565	11,366	8497	8196
EXP2	3364	4443	4558	1926
EXP4	3558	4767	6023	2122
EXP8	3716	4787	7753	2186
EXP16	3784	5076	9696	2211
EXP32	4137	5698	11,379	2323
GKLS250	5917	7080	7517	5273
GKLS2100	6843	8261	7449	7296
GKLS350	6845	8076	7833	5881(0.97)
GKLS3100	10,290(0.93)	10,187	7828	8066(0.97)
GOLDSTEIN	7977	9035	8505	4381
GRIEWANK2	12,567	12,222	12,000	12,037
HANSEN	13,441	13,360	11,876	10,818
HARTMAN3	9758	9548	8123	4114
HARTMAN6	12,893(0.90)	12,889(0.93)	22,309	10,126(0.93)
POTENTIAL3	17,912	16,420	21,904	15,969
POTENTIAL4	73,629	64,886	95,707	67,084
POTENTIAL5	40,585	35,239	47,807	33,661
RASTRIGIN	11,305	10,101	11,141	10,046
ROSENBROCK4	18,115	21,919	38,407	43,093
ROSENBROCK8	12,869	14,192	31,923	25,405
ROSENBCROK16	12,096	13,023	38,486	23,165
SHEKEL5	10,347	11,466	14,446	11,802
SHEKEL7	11,511	10,521	13,944	10,399
SHEKEL10	10,834	10,842	13,785	12,253
TEST2N4	11,133	10,869	12,161	11,546
TEST2N5	10,923(0.97)	10,315	10,868	11,072(0.97)
TEST2N6	12,331(0.97)	12,345	14,123	15,652
TEST2N7	11,342(0.93)	11,354	12,118	12,370(0.93)
SINU4	7724	9845	12,294	6575
SINU8	8468	10,969	18,122	5382
SINU16	9334	13,213	31,589	9294
SINU32	13,290	17,502(0.97)	63,111(0.97)	14,959
TEST30N3	12,675	12,954	12,472	12,482
TEST30N4	13,964	14,903	14,999	15,389
TOTAL	494,119(0.99)	504,585(0.99)	720,208(0.99)	502,326(0.99)

. The experimental results for the Double Box stopping rule of Equation (22) are listed in

Table 4. Experiments with the double box stopping rule without gradient check.

FUNCTION	I1	I2	I3	IP
BF1	6807	6866	6712	6757
BF2	6102	6150	6057	6207
BRANIN	4551	4596	4470	4435
CM4	9814	10,101	9580	9342
CAMEL	5055	5202	4897	5004
EASOM	2975	2788	3014	3000
EXP2	4436	4541	4377	4543
EXP4	5443	5562	5331	5290
EXP8	5682	5754	5614	5504
EXP16	5707	5799	5638	5526
EXP32	5871	5797	5769	5659
GKLS250	3973	3906	3971	3921
GKLS2100	4009	3862	4073	3958
GKLS350	4558	3965	4525(0.97)	4266
GKLS3100	4701(0.87)	4266	4361(0.90)	4465
GOLDSTEIN	10,259	9145	7945	7625
GRIEWANK2	5932	6194	5700	5915
HANSEN	6386	6260	5688(0.97)	5874
HARTMAN3	4681	4694	4625	4675
HARTMAN6	14,245	14,091	13,793	13,825
POTENTIAL3	7219	7206	7532	7234
POTENTIAL4	38,053	37,924	38,421	38,897
POTENTIAL5	15,196	14,459	15,708	15,358
RASTRIGIN	5915	5797	5944(0.83)	5844
ROSENBROCK4	91,574	101,485	117,512	76,367
ROSENBROCK8	66,648	61,974	58,831	41,591
ROSENBCROK16	62,029	54,550	63,406	55,800
SHEKEL5	9119	10,271(0.97)	8975	8538
SHEKEL7	9197	9831	9638	8732
SHEKEL10	10,417	10,449	9373	9721(0.90)
TEST2N4	8512	8272	8884	7992
TEST2N5	5793	5704	5511(0.90)	5515
TEST2N6	9797(0.93)	9731	9657(0.83)	9666(0.97)
TEST2N7	6435(0.80)	6659	6713(0.77)	5990(0.87)
SINU4	7567	7774	7334	7063
SINU8	9882	10,083	9643	9331
SINU16	12,750	12,947	12,569	12,207
SINU32	20,164	21,112	19,684(0.90)	19,239
TEST30N3	6388	7942	5934	5855
TEST30N4	7611	9251	6385	8284
TOTAL	531,453(0.99)	532,690(0.99)	543,794(0.98)	475,015(0.99)

and

Table 5. Experiments with the double box stopping rule with the gradient check enabled.

FUNCTION	I1	I2	I3	IP
BF1	3296	3038	3063	3003
BF2	2922	2762	2863	2845
BRANIN	2562	2641	2538	2564
CM4	3569	4277	2944	3230
CAMEL	2646	2854	2467	2577
EASOM	2490	2390	2479	2464
EXP2	2377	2489	2261	2315
EXP4	2456	2669	2282	2389
EXP8	2429	2671	2268	2385
EXP16	2358	2569	2227	2326
EXP32	2337	2533	2248	2312
GKLS250	2394	2535	2274	2321
GKLS2100	2384	2511	2267	2333
GKLS350	2492	2410	2212	2339
GKLS3100	2800(0.90)	2708	2648(0.83)	2571
GOLDSTEIN	3161	3701	3166	2799
GRIEWANK2	3910	4520	3543	3641
HANSEN	4409	4268	3755	4325
HARTMAN3	2423	2518	2374	2425
HARTMAN6	3913	4390	4199(0.93)	3700
POTENTIAL3	3951	4093	4482	4021
POTENTIAL4	18,555	19,559	19,506	18,691
POTENTIAL5	8771	8397	9677	9154
RASTRIGIN	3111	3244(0.97)	3031	3146
ROSENBROCK4	9729	12,980	11,453	8587
ROSENBROCK8	4987	6738	4688	5512
ROSENBCROK16	4410	5939	4553	4002
SHEKEL5	3906	4095	3203	3495
SHEKEL7	3119	3965	2950	3528
SHEKEL10	3497(0.97)	4464	3142(0.97)	3353
TEST2N4	3468(0.97)	4059	4167	3881(0.93)
TEST2N5	3318(0.97)	3786	2926(0.90)	3157(0.97)
TEST2N6	4523(0.93)	5046(0.93)	5537(0.83)	4066(0.97)
TEST2N7	3364(0.80)	4191(0.90)	4183(0.80)	3315(0.87)
SINU4	3173	3807	2610	3004
SINU8	3055	3742	2592	2857
SINU16	3160	3746	3854	3290
SINU32	6613	7377	6327	6450
TEST30N3	5129	6367	5605	4451
TEST30N4	5649	6441	6074	5543
TOTAL	162,816(0.99)	182,490(0.99)	164,638(0.98)	158,367(0.99)

. Finally, for the proposed stopping rule, the results are listed in

Table 6. Experiments with the proposed stopping rule without the gradient check.

FUNCTION	I1	I2	I3	IP
BF1	5305	5326	5240	5209
BF2	4760	4841	4750	4856
BRANIN	3599	3703	3520	3443
CM4	7674	7835	7430	7057
CAMEL	3996	4131	3864	3825
EASOM	2370	2292	2425	2478
EXP2	3528	3613	3455	3675
EXP4	4292	4350	4178	4020
EXP8	4579	4632	4515	4278
EXP16	4576	4637	4505	4236
EXP32	4692	4771	4588	4296
GKLS250	3105	3065	3115	3024
GKLS2100	3193	3049	3193	3099
GKLS350	3308	3000	3560	3401
GKLS3100	2935(0.97)	2777	3158(0.83)	3088
GOLDSTEIN	5534	5595	5332	5265
GRIEWANK2	4225	4332	4413	4489
HANSEN	3865	3762	3824	3769
HARTMAN3	3724	3770	3714	3705
HARTMAN6	11,901(0.97)	11,829(0.97)	11,386	10,573
POTENTIAL3	5910	5850	6134	6501
POTENTIAL4	30,880	30,570	31,180	30,682
POTENTIAL5	12,021	11,643	12,521	13,475
RASTRIGIN	4583	4595	4625	4360
ROSENBROCK4	58,299	61,266	55,759	35,517
ROSENBROCK8	31,778	30,888	30,989	22,055
ROSENBCROK16	32,719	30,503	30,957	24,478
SHEKEL5	6806	7047(0.97)	6636	6233
SHEKEL7	6807	7001	6626	6270
SHEKEL10	6774	6987	6583	6534
TEST2N4	6111	6127	5909	5893
TEST2N5	4455(0.97)	4558	4372(0.97)	4271(0.93)
TEST2N6	7446(0.97)	7419	7218(0.87)	7122(0.93)
TEST2N7	4992(0.90)	5057	4888(0.83)	4680(0.90)
SINU4	5948	6043	5750	5229
SINU8	7965	8095	7778	6963
SINU16	10,121	10,252	9968	9219
SINU32	16,093	16,509	15,663	14,478
TEST30N3	4331	4953	4230	3957
TEST30N4	6290	6341	4288	4717
TOTAL	361,490(0.99)	363,013(0.99)	352,239(0.99)	310,420(0.99)

and

Table 7. Experiments with the proposed stopping rule with the gradient check enabled.

FUNCTION	I1	I2	I3	IP
BF1	2276	2379	2266	2250
BF2	2157	2274	2098	2191
BRANIN	2132	2178	2051	2170
CM4	3098	3717	2538	2791
CAMEL	2198	2335	1974	2058
EASOM	2007	2011	2031	2084
EXP2	1952	2030	1842	1861
EXP4	2046	2266	1877	1909
EXP8	1990	2240	1849	1879
EXP16	1944	2110	1828	1838
EXP32	1953	2126	1859	1867
GKLS250	1982	2079	1850	1900
GKLS2100	1983	2064	1859	1891
GKLS350	1882	1944	1793	1831
GKLS3100	1898	1909(0.97)	1850(0.83)	1833(0.83)
GOLDSTEIN	2523	2670	2110	2164
GRIEWANK2	2893	2885	2791	2681
HANSEN	2766	2879	2731	2804
HARTMAN3	1988	2093	1949	2015
HARTMAN6	3366	3871(0.97)	2767	3133
POTENTIAL3	3312	3487	3613	3892
POTENTIAL4	15,392	16,390	16,223	17,497
POTENTIAL5	7109	7104	7732	8477
RASTRIGIN	2591	2648	2474	2732
ROSENBROCK4	8023	12,179	4433	6025
ROSENBROCK8	4376	6081	2721	3314
ROSENBCROK16	3643	4954	2746	2485
SHEKEL5	2849	3296	2274	2390
SHEKEL7	2696	3294	2262	2283
SHEKEL10	2624	3251	2338(0.93)	2359
TEST2N4	2536	2637	2427	2782
TEST2N5	2266(0.97)	2336(0.97)	2163(0.90)	2342(0.90)
TEST2N6	2724(0.93)	2832	2694(0.80)	3133(0.90)
TEST2N7	2283(0.80)	2370	2279(0.80)	2585(0.90)
SINU4	2789	3245	2228	2436
SINU8	2601	3151	2233	2348
SINU16	2721	3086	2443	2624
SINU32	4652	5135	4086	4089
TEST30N3	3031	3349	3007	2562
TEST30N4	3747	3797	3250	3237
TOTAL	126,999(0.99)	142,682(0.99)	115,539(0.98)	122,742(0.99)

. In addition, the boxplots for the proposed stopping rules without and with the gradient check of Equation (19) are illustrated in the Figure 1 and Figure 2 respectively. The above results lead to a number of observations:

1.

The PSO method is a robust method, and this is evident by the high success rate in finding the global minimum, although the number of particles used was relatively low (100).

2.

The proposed inertia calculation method as defined in Equation (12) achieves a significant reduction in the number of calls between 11 and 25% depending on the termination criterion used. However, the presence of the gradient check mechanism of Equation (19) nullifies any gain of the method, as the rejection criterion significantly reduces the number of calls regardless of the inertia calculation mechanism used.

3.

The local optimization avoidance mechanism of the gradient check drastically reduces the required number of calls for each termination criterion while maintaining the success rate of the method at extremely high levels.

4.

The proposed termination criterion is significantly superior to the other two with which the comparison was made. In addition, if the termination criterion is combined with the mechanism for avoiding local optimizations, then the gain in the number of calls grows even more.

To show the effect of the proposed termination method, an additional experiment was performed in which the dimension of the sinus problem increased from 2 to 32, and in each case, all three termination techniques were tested. The result of this experiment is graphically represented in Figure 3. This graph shows that the double box method is significantly superior to the Ali method but, of course, the new proposed method further reduces the required number of function calls. In addition, the effect of the application of the rejection mechanism based on gradients of Equation (19) is illustrated graphically in Figure 4, where the proposed termination rule is applied on a series of test functions with the gradient check and without the gradient check.

4. Conclusions

In the current work, three new modifications of the PSO method for locating the global minimum of continuous and differentiable functions were presented. The first modification alters the population velocity calculation in an attempt to cause large changes in velocity when the method is in its infancy and constantly finds new local minima and small velocity changes when the method is to be centered around a promising area of a global minimum. The second modification limits the number of local searches performed by the method through an asymptotic criterion based on derivative computation. The third modification introduces a new termination criterion based on the observation that the method from some iteration onwards will not be able to detect a new minimum, and therefore, its termination should be considered. All of these modifications have low computational requirements.

The proposed modifications were applied to the PSO method either one by one or all together in combination. The purpose of the method is to find the total minimum of continuous functions using the smallest possible number of function calls. The experimental results showed that the modifications significantly reduce the number of function calls even when not used in combination. This means that they can be used individually and in other variations of PSO. The reduction in the number of function calls reaches up to 80%. In addition, the amendments did not reduce the ability of the PSO to find the total minimum of the objective function. In addition, the first modification reduces the number of required calls but only when the criterion for avoiding local minimization is not present.