A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

Abstract

The Levenberg–Marquardt (LM) method is one of the most significant methods for solving nonlinear equations as well as symmetric and asymmetric linear equations. To improve the method, this paper proposes a new adaptive LM algorithm by modifying the LM parameter, combining the trust region technique and the non-monotone technique. It is interesting that the new algorithm is constantly optimized by adaptively choosing the LM parameter. To evaluate the effectiveness of the new algorithm, we conduct tests using various examples. To extend the convergence results, we prove the convergence of the new algorithm under the Hölderian local error bound condition rather than the commonly used local error bound condition. Theoretical analysis and numerical results show that the new algorithm is stable and effective.

1. Introduction

Nonlinear equations are widely used in key fields such as electricity, optics, mechanics, economic management, engineering technology, biomedicine, and alternative energy [1,2,3,4,5,6]. This paper discusses the following nonlinear equations:

$$\left\{\begin{array}{c}{f}_1(x_1,x_2,\cdots ,x_n)=0,\hfill \\ f_2(x_1,x_2,\cdots ,x_n)=0,\hfill \\ ⋮\hfill \\ f_n(x_1,x_2,\cdots ,x_n)=0,\hfill \end{array}\right.$$

which can be written in a vector form:

$$F\left(x\right)=0,$$

(1)

where $F\left(x\right):{\mathbb{R}}^n\to {\mathbb{R}}^n$ is continuously differentiable and $x={\left(x_1,x_2,\cdots ,x_n\right)}^T$. We denote the solution set of Equation (1) by $X^{\ast }$ and assume that $X^{\ast }$ is nonempty.

Several promising numerical methods [7,8,9,10,11] have been proposed for solving nonlinear equations. One of the classical methods to solve Equation (1) is the Gauss–Newton method, which at each iteration computes the trial step

$$d_k=-{\left(J_k^T{J}_k\right)}^{-1}{J}_k^T{F}_k,$$

where $F_k=F\left(x_k\right)$, $J_k=F^{\prime }\left(x_k\right)$ is the Jacobian matrix of $F\left(x\right)$ at $x_k$.

However, in the actual calculation, the trial step of the Gauss–Newton method may not be well defined when $J\left(x\right)$ is singular or near-singular. To overcome this difficulty, the Levenberg–Marquardt(LM) method [10,11] was proposed. At the kth iteration, the LM method computes the trial step

$$d_k=-{\left(J_k^T{J}_k+{\lambda }_{k}I\right)}^{-1}{J}_k^T{F}_k,$$

(2)

where ${\lambda }_k\ge 0$ is the LM parameter and $I\in {\mathbb{R}}^{n\times n}$ is the identity matrix. The trial step of the LM method is actually a modification of the trial step of the Gauss–Newton method, where the parameter ${\lambda }_k$ is introduced to prevent the steps from being undefined or too large when $J\left(x\right)$ is singular or nearly singular.

The LM method has quadratic convergence when $J\left(x\right)$ is Lipschitz continuous and nonsingular at the solution of Equation (1) [12]. Nevertheless, the theoretical research shows that the condition of the nonsingularity of $J\left(x\right)$ is too strong. To solve this problem, some scholars [13,14,15,16,17,18,19] have analysed the convergence of the LM method under the following local error bound condition, which is weaker than nonsingularity of $J\left(x\right)$:

$$c·dist\left(x,X^{\ast }\right)\le \parallel F\left(x\right)\parallel ,\forall x\in N\left(x^{\ast }\right),$$

(3)

where $c>0$ is a positive constant, $dist\left(x,X^{\ast }\right)$ is the distance from x to $X^{\ast }$, and $N\left(x^{\ast }\right)$ is some neighbourhood of $x^{\ast }\in X^{\ast }$. In this paper, $\parallel ·\parallel$ is the 2-norm.

Although the local error bound condition is weaker than the nonsingularity of $J\left(x\right)$, this condition is not always satisfied with some ill-conditioned nonlinear equations in biochemical systems and certain applications. Recently, some scholars [20,21,22,23] have studied the convergence of LM method under the following Hölderian error bound condition, which is weaker than the local error bound condition:

$$c·dist\left(x,X^{\ast }\right)\le {\parallel F\left(x\right)\parallel }^{\gamma },\forall x\in N\left(x^{\ast }\right),$$

(4)

where c is a positive constant and $\gamma \in (0,1]$. Obviously, the Hölderian error bound condition (4) is a generalization of the local error bound condition (3), where the exponent $\gamma$ of $\parallel F\left(x\right)\parallel$ is extended to an interval $(0,1]$. In this paper, we study the convergence of the new algorithm under the Hölderian error bound condition.

The LM parameter ${\lambda }_k$ is vital to the efficiency of LM algorithms. Several scholars have done interesting research [13,14,15,16,17,18,19,21,22,23] on ${\lambda }_k$. Yamashita and Fukushima [13] took ${\lambda }_k={∥F_k∥}^2$, although the disadvantage of choosing parameters in this way is that the value of ${\lambda }_k={∥F_k∥}^2$ may be too small to be effective when the sequence $\left\{x_k\right\}$ is close to the solution set of Equation (1), which affects the local convergence rate. In order to solve this disadvantage and reduce the impact, Fan and Yuan [14] chose ${\lambda }_k={∥F_k∥}^{\delta }$, which is a generalization of ${\lambda }_k={∥F_k∥}^2$, and extended the exponent $\delta$ of ${\lambda }_k={∥F_k∥}^{\delta }$ to an interval $[1,2]$. The numerical results when solving some equations showed better performance when $\delta =1$; however, the disadvantage of choosing parameters in this way is that it may make ${\lambda }_k=∥F_k∥$ too large and step $d_k$ too small when $\left\{x_k\right\}$ is far away from the solution set, causing the sequence to move slowly to the solution set and affecting the global convergence rate. To compensate for this flaw, Fan [17] used ${\lambda }_k={\mu }_k∥F_k∥$, where ${\mu }_k$ is updated every iteration by the trust region technique. Numerical results showed that this change improved the performance of the algorithm. Chen and Ma [23] took ${\lambda }_k=\theta \parallel F_k{\parallel }^{\delta }+(1-\theta ){\parallel J_k^T{F}_k\parallel }^{\delta }$ for $\theta \in [0,1]$ and $\delta \in [1,2]$, finding that this improved the numerical results of the LM algorithm. Recently, Li et al. [24] proposed a new adaptive accelerated LM algorithm by choosing the LM parameter as ${\lambda }_{k+1}=\frac{{\mu }_{k+1}\parallel F_{k+1}\parallel }{1+\parallel F_{k+1}\parallel }$, with numerical results showing that the algorithm is efficient for solving symmetric and asymmetric linear equations.

Inspired by the above literature, we take a new adaptive LM parameter to enhance the computing performance of the LM algorithm, as follows:

$${\lambda }_k=\left\{\begin{array}{c}{\mu }_k(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{\delta }),\mathrm{if}∥F_k∥\le 1,\hfill \\ {\mu }_k(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{-\delta }),\mathrm{otherwise},\hfill \end{array}(0\le \theta \le 1,1\le \delta \le 2),\right.$$

where ${\mu }_k$ is updated every iteration via trust region technology. When $\left\{x_k\right\}$ is close to a solution set, $∥F_k∥$ is close to 0; thus, ${\lambda }_k$ is close to ${\mu }_k{∥F_k∥}^{\delta }$ if $\delta =1$, as used in [17]. Conversely, when $\left\{x_k\right\}$ is far from the solution set, the leading $∥F_k∥$ may be very large; thus, ${\lambda }_k$ will be close to ${\mu }_k\theta$. This effectively regulates the range of ${\lambda }_k$ to prevent the LM step from becoming excessively small, thereby enhancing computational efficiency. Therefore, it seems that this choice of ${\lambda }_k$ is more effective for the LM algorithm.

The following sections outline the remaining contents of this paper. In Section 2, we propose a new algorithm with a new LM parameter in more detail and prove its global convergence. In Section 3, we analyse the convergence rate of the new algorithm. In Section 4, we present numerical results verifying that the new algorithm is effective. Finally, some key conclusions are put forward in Section 5.

2. The New Adaptive LM Algorithm and Its Global Convergence

In this section, we introduce our new adaptive algorithm and establish its global convergence.

If we define the merit function for Equation (1) as

$$\varphi \left(x\right)={\parallel F\left(x\right)\parallel }^2,$$

then, at the kth iteration, the actual reduction of $\varphi \left(x\right)$ is provided by

$${Ared}_k={∥F_k∥}^2-{∥F\left(x_k+d_k\right)∥}^2$$

(5)

and the predicted reduction of $\varphi \left(x\right)$ by

$${Pred}_k={∥F_k∥}^2-{∥F_k+J_k{d}_k∥}^2,$$

(6)

where $d_k$ is computed by Equation (2). The ratio of ${Ared}_k$ to ${Pred}_k$ is

$$r_k=\frac{{Ared}_k}{{Pred}_k},$$

(7)

which determines whether to accept $d_k$ and update ${\mu }_k$. Several studies have suggested that algorithms employing non-monotone strategies outperform those with monotone strategies [18,25,26,27,28]. To carry out the non-monotone strategy, Amini et al. [18] used the following actual reduction to replace Equation (5):

$$\overline{A}re{d}_k=F_{l\left(k\right)}^2-{∥F\left(x_k+d_k\right)∥}^2$$

(8)

where

$$F_{l\left(k\right)}=\underset{0\le j\le n\left(k\right)}{max}\left\{∥F_{k-j}∥\right\},k=0,1,2,\dots ,$$

(9)

$n\left(k\right)=min\left\{N_0,k\right\}$, and $N_0$ is a positive integer constant. With this change, $∥F\left(x_{k+1}\right)∥$ is compared with $\underset{0\le j\le n\left(k\right)}{max}\left\{∥F_{k-j}∥\right\}$ at each iteration. To combine the non-monotone strategy with the new adaptive LM parameter, we use the following ratio:

$${\widehat{r}}_k=\frac{\overline{A}{red}_k}{{Pred}_k}$$

to replace the original role of the ratio $r_k$ in the algorithm.

Next, we present a new adaptive LM algorithm, named the ALLM algorithm (Algorithm 1).

Algorithm 1 (ALLM Algorithm)

Step 1. Given $x_0\in {\mathbb{R}}^n$, $N_0>0,{\mu }_0>m>0,\epsilon >0,0<p_0\le p_1\le p_2<1$. Set $k:=0$. Step 2. If $∥J_k^T{F}_k∥\le \epsilon$, stop. Otherwise let $${\lambda }_k=\left\{\begin{array}{cc}{\mu }_k(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{\delta }),\hfill & \mathrm{if} ∥F_k∥\le 1,\hfill \\ {\mu }_k(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{-\delta }),\hfill & \mathrm{otherwise},\hfill \end{array}\right.(0\le \theta \le 1,1\le \delta \le 2).$$ (10) Step 3. Compute $d_k$ $$\left(J_k^T{J}_k+{\lambda }_{k}I\right)d=-J_k^T{F}_k,$$ (11) Step 4. Compute $F_{l\left(k\right)}$, $Pre{d}_k$ and $\overline{A}re{d}_k$ by Equations (9), (6) and (8). Set $${\widehat{r}}_k=\frac{\overline{A}{red}_k}{{Pred}_k}.$$ (12) Step 5. Set $$x_{k+1}=\left\{\begin{array}{cc}{x}_k+d_k,& \mathrm{if}{\widehat{r}}_k\ge p_0\\ x_k,& \mathrm{otherwise}\end{array}\right.$$ (13) Step 6. Choose ${\mu }_{k+1}$ as $${\mu }_{k+1}=\left\{\begin{array}{cc}4{\mu }_k,& \mathrm{if}{\widehat{r}}_k<p_1\hfill \\ {\mu }_k,& \mathrm{if}{\widehat{r}}_k\in \left[p_1,p_2\right]\hfill \\ max\left\{\frac{{\mu }_k}{4},m\right\},& \mathrm{otherwise}.\hfill \end{array}\right.$$ (14) Step 7. Set $k=k+1$ and return to Step 2.

To prevent excessively large steps, we impose the following condition:

$${\mu }_k\ge m,\forall k\in \mathbb{N}$$

(15)

where m is a positive constant.

Lemma 1.

$Pre{d}_k\ge ∥J_k^T{F}_k∥min\{∥d_k∥,\frac{∥J_k^T{F}_k∥}{∥J_k^T{J}_k∥}\}$ for all $k\in \mathbb{N}$.

Proof. 

This proof comes from famous result in [29]. □

Lemma 2

([18]). Assume that sequence $\left\{x_k\right\}$ is generated by the ALLM algorithm; then, the sequence $\left\{F_{l\left(k\right)}\right\}$ converges.

Assumption 1.

(a) $J\left(x\right)$ is Hölderian continuous, i.e., there exists a constant ${\kappa }_{hj}>0$ such that

$$\parallel J\left(x\right)-J\left(y\right)\parallel \le {\kappa }_{hj}{\parallel x-y\parallel }^v,\forall x,y\in {\mathbb{R}}^n$$

(16)

where the exponent $v\in (0,1]$.

(b) $J\left(x\right)$ is bounded, i.e., there exists a constant ${\kappa }_{bj}>0$ such that

$$\parallel J\left(x\right)\parallel \le {\kappa }_{bj},\forall x\in {\mathbb{R}}^n.$$

(17)

It follows from Equation (16) that

$$\parallel F\left(y\right)-F\left(x\right)-J\left(x\right)(y-x)\parallel \le \frac{{\kappa }_{hj}}{1+v}{\parallel y-x\parallel }^{1+v}.$$

(18)

Thus, there exists a constant ${\kappa }_{bf}>0$ that makes

$$\parallel F\left(y\right)-F\left(x\right)\parallel \le {\kappa }_{bf}\parallel y-x\parallel .$$

(19)

Theorem 1.

Under Assumption 1, the ALLM algorithm satisfies

$$\underset{k\to \infty }{lim}inf\parallel J_k^T{F}_k\parallel =0.$$

(20)

Proof. 

Assuming that Theorem 1 is not true, we obtain

$$\parallel J_k^T{F}_k\parallel \ge {ϵ}_0,\forall k\ge k_0$$

(21)

where ${ϵ}_0$ is a positive constant and $k_0\in \mathbb{N}$.

If $d_k$ is accepted by the ALLM algorithm, then

$$F_{l\left(k\right)}^2-{∥F\left(x_k+d_k\right)∥}^2\ge p_0{Pred}_k.$$

Per Lemma 1, Equations (17) and (21) indicate that, for all $k\ge k_0$,

$$\begin{array}{cc}\hfill F_{l\left(k\right)}^2-{∥F_{k+1}∥}^2& \ge p_0\parallel J_k^T{F}_k\parallel min\{∥d_k∥,\frac{\parallel J_k^T{F}_k\parallel }{\parallel J_k^T{J}_k\parallel }\}\hfill \\ & \ge p_0{ϵ}_{0}min\{∥d_k∥,\frac{{ϵ}_0}{{\kappa }_{bj}^2}\}.\hfill \end{array}$$

Then, substituting k for $l\left(k\right)-1$,

$$F_{l\left(l\right(k)-1)}^2-{\parallel F_{l\left(k\right)}\parallel }^2\ge p_0{ϵ}_{0}min\left\{\parallel d_{l\left(k\right)-1}\parallel ,\frac{{ϵ}_0}{{\kappa }_{bj}^2}\right\}$$

holds for all sufficiently large k.

Per Lemma 1, we obtain

$$\underset{k\to \infty }{lim}\left(F_{l\left(l\right(k)-1)}^2-{\parallel F_{l\left(k\right)}\parallel }^2\right)=0;$$

thus,

$$\underset{k\to \infty }{lim}min\left\{\parallel d_{l\left(k\right)-1}\parallel ,\frac{{ϵ}_0}{{\kappa }_{bj}^2}\right\}=0,$$

as $\frac{{ϵ}_0}{{\kappa }_{bj}^2}$ is a positive constant, meaning that

$$\underset{k\to \infty }{lim}\parallel d_{l\left(k\right)-1}\parallel =0.$$

Per Equation (19), the last equality implies that

$$\underset{k\to \infty }{lim}\parallel F\left(x_{l\left(k\right)}\right)\parallel =\underset{k\to \infty }{lim}\parallel F\left(x_{l\left(k\right)-1}\right)\parallel .$$

Next, by considering the proof process of Theorem 2.4 in [18], we can prove that

$$\underset{k\to \infty }{lim}∥d_k∥=0.$$

(22)

Along with Equations (10), (11), (17) and (21), this implies that

$${\mu }_k\to \infty ,\mathrm{as}k\to \infty .$$

(23)

Next, per Equation (18), we obtain

$$\left|∥F\left(x_k+d_k\right)∥-∥F_k+J_k{d}_k∥\right|\le \frac{{\kappa }_{hj}}{1+v}{∥d_k∥}^{1+v},$$

which yields

$$|\parallel F(x_k+d_k){\parallel }^2-\parallel F_k+J_k{d}_k{\parallel }^2|\le \frac{2{\kappa }_{hj}}{1+v}∥F_k+J_k{d}_k∥{∥d_k∥}^{1+v}+\frac{{\kappa }_{hj}^2}{{(1+v)}^2}{∥d_k∥}^{2+2v}.$$

From Lemma 1 and Equations (17), (21), (22) and $∥F_k+J_k{d}_k∥\le ∥F_k∥\le ∥F_1∥$, we obtain

$$\begin{array}{cc}\hfill \left|r_k-1\right|& =\left|\frac{{Ared}_k-{Pred}_k}{{Pred}_k}\right|\hfill \\ & \le \frac{\frac{2{\kappa }_{hj}}{1+v}∥F_k+J_k{d}_k∥{∥d_k∥}^{1+v}+\frac{{\kappa }_{hj}^2}{{(1+v)}^2}{∥d_k∥}^{2+2v}}{∥J_k^T{F}_k∥min\left\{∥d_k∥,\frac{∥J_k^T{F}_k∥}{∥J_k^T{J}_k∥}\right\}}\hfill \\ & \to 0;\hfill \end{array}$$

thus,

$$\underset{k\to +\infty }{lim}{r}_k=1.$$

Combined with Equations (6), (8), (9) and (12), we obtain

$${\widehat{r}}_k=\frac{{\overline{A}red}_k}{{Pred}_k}=\frac{F_{l\left(k\right)}^2-{∥F\left(x_k+d_k\right)∥}^2}{{Pred}_k}\ge \frac{{∥F_k∥}^2-{∥F_{k+1}∥}^2}{{Pred}_k}=r_k\to 1.$$

In view of the ALLM algorithm, for all large k there exists a positive constant $\overline{\mu }>m$ that makes ${\mu }_k<\overline{\mu }$, which contradicts Equation (23). Thus, Theorem 1 holds. □

3. Convergence Rate

This section discusses the convergence rate of the ALLM algorithm. Here, we let $\left\{x_k\right\}$ generated by ALLM algorithm lie within a neighborhood of $x^{\ast }\in X^{\ast }$ and converge to the solution set $X^{\ast }$ of Equation (1).

Assumption 2.

(a) $F\left(x\right)$ provides a Hölderian local error bound, i.e., there exist constants $c>0$ and $0<b<1$ that make

$$c·dist\left(x,X^{\ast }\right)\le {\parallel F\left(x\right)\parallel }^{\gamma },\forall x\in N\left(x^{\ast },b\right),$$

(24)

where the exponent $\gamma \in (0,1]$, $N\left(x^{\ast },b\right)=\left\{x\in {\mathbb{R}}^n\mid ∥x-x^{\ast }∥\le b\right\}$.

(b) $J\left(x\right)$ is Hölderian continuous, i.e., there exists a constant ${\kappa }_{hj}>0$ such that

$$\parallel J\left(x\right)-J\left(y\right)\parallel \le {\kappa }_{hj}{\parallel x-y\parallel }^v,\forall x,y\in N\left(x^{\ast },b\right)$$

(25)

where the exponent $v\in (0,1]$.

From Equation (25), we have

$$\parallel F\left(y\right)-F\left(x\right)-J\left(x\right)(y-x)\parallel \le \frac{{\kappa }_{hj}}{1+v}{\parallel y-x\parallel }^{1+v},\forall x,y\in N\left(x^{\ast },b\right).$$

(26)

Thus,

$$\parallel F\left(y\right)-F\left(x\right)\parallel \le {\kappa }_{bf}\parallel y-x\parallel ,\forall x,y\in N\left(x^{\ast },b\right),$$

(27)

where ${\kappa }_{bf}$ is a positive constant.

Defining by ${\overline{x}}_k\in X^{\ast }$ satisfies

$$∥{\overline{x}}_k-x_k∥=dist\left(x_k,X^{\ast }\right),$$

which implies that ${\overline{x}}_k$ is closest to $x_k$.

Next, we discuss the important property of $∥d_k∥$ and ${\mu }_k$; finally, we study the convergence rate of the ALLM algorithm using the singular value decomposition (SVD) technique. Without loss of generality, we assume that $x_k\in N(x^{\ast },\frac{b}{4})$.

Lemma 3.

Under Assumption 2, we have

(1) If $∥F_k∥\le 1$; then, the following relationship holds:

$$∥d_k∥\le \overline{c}dist{\left(x_k,X^{\ast }\right)}^{min\{1,1+v-\frac{\delta }{2\gamma }\}}$$

(28)

where $\overline{c}$ is a positive constant.

(2) If $∥F_k∥>1$, then the following relationship holds:

$$∥d_k∥\le \tilde{c}dist\left(x_k,X^{\ast }\right)$$

(29)

where $\tilde{c}$ is a positive constant.

Proof. 

(1) As $x_k\in N(x^{\ast },\frac{b}{4})$, we have

$$\parallel {\overline{x}}_k-x^{\ast }\parallel \le \parallel {\overline{x}}_k-x_k\parallel +\parallel x_k-x^{\ast }\parallel \le \frac{b}{2};$$

thus, ${\overline{x}}_k\in N(x^{\ast },\frac{b}{2})$.

We define

$${\phi }_k\left(d\right)=\parallel F_k+J_k{d\parallel }^2+{\lambda }_k{\parallel d\parallel }^2.$$

It can be concluded from (11) that $d_k$ is the minimizer of ${\phi }_k\left(d\right)$. From (26) and $F\left({\overline{x}}_k\right)=0$, we have

$$\begin{array}{cc}\hfill \parallel d_k{\parallel }^2& \le \frac{{\phi }_k\left(d_k\right)}{{\lambda }_k}\hfill \\ & \le \frac{{\phi }_k({\overline{x}}_k-x_k)}{{\lambda }_k}\hfill \\ & =\frac{\parallel F_k+J_k({\overline{x}}_k-x_k){\parallel }^2+{\lambda }_k{\parallel {\overline{x}}_k-x_k\parallel }^2}{{\lambda }_k}\hfill \\ & =\frac{\parallel F\left({\overline{x}}_k\right)-F_k-J_k({\overline{x}}_k-x_k){\parallel }^2+{\lambda }_k{\parallel {\overline{x}}_k-x_k\parallel }^2}{{\lambda }_k}\hfill \\ & \le \frac{1}{{\lambda }_k}{\left(\frac{{\kappa }_{hj}}{1+v}\right)}^2\parallel {\overline{x}}_k-x_k{\parallel }^{2+2v}+{\parallel {\overline{x}}_k-x_k\parallel }^2.\hfill \end{array}$$

If $\parallel F_k\parallel \le 1$, then $\parallel F_k{\parallel }^{\delta }\le 1$ and $1+\parallel F_k{\parallel }^{\delta }\le 2$. In conjunction with (15) and (24), this yields

$$\begin{array}{cc}\hfill {\lambda }_k& ={\mu }_k\left(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{\delta }\right)\hfill \\ & \ge m\left(\theta \frac{c^{\frac{\delta }{\gamma }}{∥x_k-{\overline{x}}_k∥}^{\frac{\delta }{\gamma }}}{2}+(1-\theta )c^{\frac{\delta }{\gamma }}{∥x_k-{\overline{x}}_k∥}^{\frac{\delta }{\gamma }}\right)\hfill \\ & \ge \frac{1}{2}(3m-m\theta )c^{\frac{\delta }{\gamma }}{∥x_k-{\overline{x}}_k∥}^{\frac{\delta }{\gamma }}.\hfill \end{array}$$

Thus,

$$\begin{array}{cc}\hfill {∥d_k∥}^2& \le \frac{1}{{\lambda }_k}{\left(\frac{{\kappa }_{hj}}{1+v}\right)}^2{∥{\overline{x}}_k-x_k∥}^{2+2v}+{∥{\overline{x}}_k-x_k∥}^2\hfill \\ & \le \frac{2{\kappa }_{hj}^2{c}^{-\frac{\delta }{\gamma }}}{(3m-m\theta ){(1+v)}^2}{∥{\overline{x}}_k-x_k∥}^{2+2v-\frac{\delta }{\gamma }}+{∥{\overline{x}}_k-x_k∥}^2\hfill \\ & \le \left(\frac{2{\kappa }_{hj}^2{c}^{-\frac{\delta }{\gamma }}}{(3m-m\theta ){(1+v)}^2}+1\right){∥{\overline{x}}_k-x_k∥}^{2min\{1,1+v-\frac{\delta }{2\gamma }\}}.\hfill \end{array}$$

Setting $\overline{c}=\sqrt{2{\kappa }_{hj}^2{c}^{-\frac{\delta }{\gamma }}/(3m-m\theta ){(1+v)}^2}$, we obtain Equation $\left(28\right)$.

(2) If $∥F_k∥>1$, then ${∥F_k∥}^{\delta }>1$ and $1+{∥F_k∥}^{\delta }\le 2{∥F_k∥}^{\delta }$. Along with (19), this allows us to conclude that

$$\begin{array}{cc}\hfill {\lambda }_k& ={\mu }_k\left(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{-\delta }\right)\hfill \\ & \ge m\left(\theta \frac{{∥F_k∥}^{\delta }}{2{∥F_k∥}^{\delta }}+(1-\theta )k_{bf}^{-\delta }{∥x_k-{\overline{x}}_k∥}^{-\delta }\right)\hfill \\ & \ge \frac{m\theta }{2}+m(1-\theta )k_{bf}^{-\delta }{∥x_k-{\overline{x}}_k∥}^{-\delta }.\hfill \end{array}$$

Thus, there exists a constant $\tilde{c}>0$ such that

$$\parallel d_k{\parallel }^2\le {\tilde{c}}^{2}dist{(x_k,X^{\ast })}^2.$$

Therefore, $\parallel d_k\parallel \le \tilde{c}dist(x_k,X^{\ast })$. □

Lemma 4.

Under Assumption 2, we have the following:

(1) If $\parallel F_k\parallel \le 1$, $v>max\{\frac{1}{\gamma }-1,\frac{1}{\gamma (1+v)-\frac{\delta }{2}}-1,\frac{1-\gamma }{\gamma (1+v)-\frac{\delta }{2}}\},$ then ${\mu }_k$ is bounded above, i.e., there exists a positive constant $M_1$ such that ${\mu }_k\le M_1$ holds for all large k.

(2) If $\parallel F_k\parallel >1,$$v>\frac{1}{\gamma }-1,$ then ${\mu }_k$ is bounded above, i.e., there exists a positive constant $M_2$ such that ${\mu }_k\le M_2$ holds for all large k.

Proof. 

(1) Considering Lemma 3.3 in [21], we can see that

$$\begin{array}{cc}\hfill \left|r_k-1\right|& =\left|\frac{{Ared}_k-{Pred}_k}{{Pred}_k}\right|\hfill \\ & =\left|\frac{{∥F_k+J_k{d}_k∥}^2-{∥F\left(x_k+d_k\right)∥}^2}{{Pred}_k}\right|\hfill \\ & \le \frac{{\left(\frac{{\kappa }_{hj}}{1+v}\right)}^2{∥d_k∥}^{2+2v}+\frac{2{\kappa }_{hj}}{1+v}∥F_k+J_k{d}_k∥{∥d_k∥}^{1+v}}{c_4∥F_k∥{∥d_k∥}^{max\{\frac{1}{\gamma },\frac{1}{\gamma (1+v)-\frac{\delta }{2}},\frac{1-\gamma }{\gamma (1+v)-\frac{\delta }{2}}+1\}}}\hfill \\ & \to 0;\hfill \end{array}$$

thus,

$$\underset{k\to +\infty }{lim}{r}_k=1.$$

This, along with Equations (6), (8), (9), and (12), yields

$${\widehat{r}}_k=\frac{\overline{A}re{d}_k}{{Pred}_k}=\frac{F_{l\left(k\right)}^2-{∥F\left(x_k+d_k\right)∥}^2}{Pre{d}_k}\ge \frac{{∥F_k∥}^2-{∥F_{k+1}∥}^2}{{Pred}_k}=r_k\to 1.$$

Considering the updating rule from (14), we can ascertain the existence of a positive constant $M_1>m$, ensuring that ${\mu }_k\le M_1$ holds for sufficiently large k.

(2) Consider the following two cases.

Case 1: $∥{\overline{x}}_k-x_k∥\le ∥d_k∥$. Per Lemma 3 (2), Equations (24), (26) and $v>\frac{1}{\gamma }-1$, we have

$$\begin{array}{cc}\hfill ∥F_k∥-∥F_k+J_k{d}_k∥& \ge ∥F_k∥-∥F_k+J_k\left({\overline{x}}_k-x_k\right)∥\hfill \\ \hfill & \ge c^{\frac{1}{\gamma }}{∥{\overline{x}}_k-x_k∥}^{\frac{1}{\gamma }}-\frac{{\kappa }_{hj}}{1+v}{∥{\overline{x}}_k-x_k∥}^{1+v}\hfill \\ \hfill & \ge c_1{∥{\overline{x}}_k-x_k∥}^{\frac{1}{\gamma }}\hfill \\ \hfill & \ge c_2{∥d_k∥}^{\frac{1}{\gamma }}\hfill \end{array}$$

(30)

which holds for some $c_1$, $c_2>0$.

Case 2: $∥{\overline{x}}_k-x_k∥>∥d_k∥$. It follows from Equation (30) that

$$\begin{array}{cc}\hfill ∥F_k∥-∥F_k+J_k{d}_k∥& \ge ∥F_k∥-∥F_k+\frac{∥d_k∥}{∥{\overline{x}}_k-x_k∥}{J}_k\left({\overline{x}}_k-x_k\right)∥\hfill \\ \hfill & \ge ∥F_k∥-∥\left(1-\frac{∥d_k∥}{∥{\overline{x}}_k-x_k∥}\right)F_k+\frac{∥d_k∥}{∥{\overline{x}}_k-x_k∥}\left(F_k+J_k\left({\overline{x}}_k-x_k\right)\right)∥\hfill \\ \hfill & \ge \frac{∥d_k∥}{∥{\overline{x}}_k-x_k∥}\left(∥F_k∥-∥F_k+J_k\left({\overline{x}}_k-x_k\right)∥\right)\hfill \\ \hfill & \ge c_1∥d_k∥{∥{\overline{x}}_k-x_k∥}^{\frac{1}{\gamma }-1}\hfill \\ \hfill & \ge c_3{∥d_k∥}^{\frac{1}{\gamma }}\hfill \end{array}$$

(31)

holds for some $c_3>0$.

Therefore, from Equations (30) and (31), we have

$$\begin{array}{cc}\hfill Pre{d}_k& =\left(∥F_k∥+∥F_k+J_k{d}_k∥\right)\left(∥F_k∥-∥F_k+J_k{d}_k∥\right)\hfill \\ \hfill & \ge ∥F_k∥\left(∥F_k∥-∥F_k+J_k{d}_k∥\right)\hfill \\ \hfill & \ge c_4∥F_k∥{∥d_k∥}^{\frac{1}{\gamma }},\hfill \end{array}$$

(32)

which holds for some $c_4>0$.

Because $∥F_k+J_k{d}_k∥\le ∥F_k∥,v>\frac{1}{\gamma }-1$, from Equations (26) and (32) we have

$$\begin{array}{cc}\hfill \left|r_k-1\right|& =\left|\frac{Are{d}_k-Pre{d}_k}{Pre{d}_k}\right|\hfill \\ & =\left|\frac{{∥F_k+J_k{d}_k∥}^2-{∥F\left(x_k+d_k\right)∥}^2}{{\mathrm{Pred}}_k}\right|\hfill \\ & \le \frac{{\left(\frac{{\kappa }_{hj}}{1+v}\right)}^2{∥d_k∥}^{2+2v}+\frac{2{\kappa }_{hj}}{1+v}∥F_k+J_k{d}_k∥{∥d_k∥}^{1+v}}{c_5∥F_k∥{∥d_k∥}^{\frac{1}{\gamma }}}\hfill \\ & \to 0;\hfill \end{array}$$

thus,

$$\underset{k\to +\infty }{lim}{r}_k=1.$$

This, along with Equations (6), (8), (9) and (12), yields

$${\widehat{r}}_k=\frac{\overline{A}re{d}_k}{Pre{d}_k}=\frac{F_{l\left(k\right)}^2-{∥F\left(x_k+d_k\right)∥}^2}{Pre{d}_k}\ge \frac{{∥F_k∥}^2-{∥F_{k+1}∥}^2}{Pre{d}_k}=r_k\to 1.$$

Therefore, there exists a positive constant $M_2>m$ such that ${\mu }_k\le M_2$ holds for sufficiently large k. □

Next, we consider SVD technology. In view of the findings provided by Behling and Iusem in [30], without loss of generality, we set $rank\left(J\left(\overline{x}\right)\right)=r$ for all $\overline{x}\in N\left(x^{\ast },b\right)\cap X^{\ast }$. Suppose that the SVD of $J\left({\overline{x}}_k\right)$ is

$$J\left({\overline{x}}_k\right)={\overline{U}}_k{\overline{\mathrm{\Sigma }}}_k{\overline{V}}_k^T=\left({\overline{U}}_{k,1},{\overline{U}}_{k,2}\right)\left(\begin{array}{cc}{\overline{\mathrm{\Sigma }}}_{k,1}\hfill & \\ & 0\hfill \end{array}\right)\left(\begin{array}{c}{\overline{V}}_{k,1}^T\\ {\overline{V}}_{k,2}^T\end{array}\right)={\overline{U}}_{k,1}{\overline{\mathrm{\Sigma }}}_{k,1}{\overline{V}}_{k,1}^T,$$

where ${\overline{\mathrm{\Sigma }}}_{k,1}=diag\left({\overline{\sigma }}_{k,1},\dots ,{\overline{\sigma }}_{k,r}\right)>0$.

Correspondingly,

$$J_k=\left(U_{k,1},U_{k,2}\right)\left(\begin{array}{cc}{\mathrm{\Sigma }}_{k,1}& \\ & {\mathrm{\Sigma }}_{k,2}\end{array}\right)\left(\begin{array}{c}{V}_{k,1}^T\\ V_{k,2}^T\end{array}\right)=U_{k,1}{\mathrm{\Sigma }}_{k,1}{V}_{k,1}^T+U_{k,2}{\mathrm{\Sigma }}_{k,2}{V}_{k,2}^T,$$

where ${\mathrm{\Sigma }}_{k,2}=diag\left({\sigma }_{k,r+1},\dots ,{\sigma }_{k,n}\right)>0$.

For clearness, we let

$$J_k=U_1{\mathrm{\Sigma }}_1{V}_1^T+U_2{\mathrm{\Sigma }}_2{V}_2^T,$$

which neglects the subscription k in $U_{k,i}$, ${\mathrm{\Sigma }}_{k,i}$ and $V_{k,i}$.

Lemma 5

([21]). Under Assumption 2, the following relationship holds:

(1) $∥U_1{U}_1^T{F}_k∥\le {\kappa }_{bf}∥{\overline{x}}_k-x_k∥$

(2) $∥U_2{U}_2^T{F}_k∥\le 2{\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^{1+v}$.

Theorem 2.

Under the conditions of Lemma 3, we have the following:

(1) If $∥F_k∥\le 1$, then the $\left\{x_k\right\}$ generated by the ALLM algorithm converges to the solution set of Equation (1) with order $min\{\gamma (1+\delta ),\gamma (1+v),\gamma (1+v)(1+v-\frac{\delta }{2\gamma })\}$.

(2) If $∥F_k∥>1$, then the $\left\{x_k\right\}$ generated by the ALLM algorithm converges to the solution set of Equation (1) with order γ.

Proof. 

(1) It follows from the SVD of $J_k$ that

$$d_k=-V_1{\left({\mathrm{\Sigma }}_1^2+{\lambda }_{k}I\right)}^{-1}{\mathrm{\Sigma }}_1{U}_1^T{F}_k-V_2{\left({\mathrm{\Sigma }}_2^2+{\lambda }_{k}I\right)}^{-1}{\mathrm{\Sigma }}_2{U}_2^T{F}_k$$

and

$$\begin{array}{cc}\hfill F_k+J_k{d}_k& =F_k-U_1{\mathrm{\Sigma }}_1{\left({\mathrm{\Sigma }}_1^2+{\lambda }_{k}I\right)}^{-1}{\mathrm{\Sigma }}_1{U}_1^T{F}_k-U_2{\mathrm{\Sigma }}_2{\left({\mathrm{\Sigma }}_2^2+{\lambda }_{k}I\right)}^{-1}{\mathrm{\Sigma }}_2{U}_2^T{F}_k\hfill \\ \hfill & ={\lambda }_k{U}_1{\left({\mathrm{\Sigma }}_1^2+{\lambda }_{k}I\right)}^{-1}{U}_1^T{F}_k+{\lambda }_k{U}_2{\left({\mathrm{\Sigma }}_2^2+{\lambda }_{k}I\right)}^{-1}{U}_2^T{F}_k.\hfill \end{array}$$

(33)

According to the theory of matrix perturbation [31] and Equation (25), we have

$$∥diag\left({\mathrm{\Sigma }}_1-{\overline{\mathrm{\Sigma }}}_{k,1},{\mathrm{\Sigma }}_2\right)∥\le ∥J_k-J\left({\overline{x}}_k\right)∥\le {\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^v,$$

which indicates

$$∥{\mathrm{\Sigma }}_1-{\overline{\mathrm{\Sigma }}}_{k,1}∥\le {\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^v,∥{\mathrm{\Sigma }}_2∥\le {\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^v.$$

(34)

As $\left\{x_k\right\}$ converges to $X^{\ast }$, without loss of generality, we let ${\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^v\le \frac{\overline{\sigma }}{2}$ hold for all large k. From Equation (34), we have

$$∥{\left({\mathrm{\Sigma }}_1^2+{\lambda }_{k}I\right)}^{-1}∥\le ∥{\mathrm{\Sigma }}_1^{-2}∥\le \frac{1}{{\left(\overline{\sigma }-{\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^v\right)}^2}\le \frac{4}{{\overline{\sigma }}^2}.$$

(35)

From Equations (34), (35), Lemma 5, and ${\left({\mathrm{\Sigma }}_2^2+{\lambda }_{k}I\right)}^{-1}\parallel \le {\lambda }_k^{-1}$, we have

$$∥F_k+J_k{d}_k∥\le \frac{4{\lambda }_k{\kappa }_{bf}}{{\overline{\sigma }}^2}∥{\overline{x}}_k-x_k∥+2{\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^{1+v}.$$

(36)

If $∥F_k∥\le 1$, then ${∥F_k∥}^{\delta }\le 1$, while from Equation (27) and Lemma 4 we have

$$\begin{array}{cc}\hfill {\lambda }_k& ={\mu }_k\left(\theta \frac{{∥F_k∥}^{\delta }}{1+{∥F_k∥}^{\delta }}+(1-\theta ){∥F_k∥}^{\delta }\right)\hfill \\ & \le M_1\left(\theta {∥F_k∥}^{\delta }+(1-\theta ){∥F_k∥}^{\delta }\right)\hfill \\ & \le M_1{k}_{bf}^{\delta }{∥x_k-{\overline{x}}_k∥}^{\delta }.\hfill \end{array}$$

This, along with Equation (36), yields

$$\begin{array}{cc}\hfill ∥F_k+J_k{d}_k∥& \le \frac{4M_1{\kappa }_{bf}^{1+\delta }}{{\overline{\sigma }}^2}{∥{\overline{x}}_k-x_k∥}^{1+\delta }+2{\kappa }_{hj}{∥{\overline{x}}_k-x_k∥}^{1+v}\hfill \\ \hfill & \le \left(\frac{4M_1{\kappa }_{bf}^{1+\delta }}{{\overline{\sigma }}^2}+2{\kappa }_{hj}\right){∥{\overline{x}}_k-x_k∥}^{min\{1+\delta ,1+v\}}.\hfill \end{array}$$

(37)

Letting $c_5=\frac{4M_1{\kappa }_{bf}^{1+\delta }}{{\overline{\sigma }}^2}+2{\kappa }_{hj}$, from Equations (24), (26), (28) and (37) we obtain

$$\begin{array}{cc}\hfill {\left(c∥{\overline{x}}_{k+1}-x_{k+1}∥\right)}^{\frac{1}{\gamma }}& \le ∥F\left(x_k+d_k\right)∥\hfill \\ & \le ∥F_k+J_k{d}_k∥+{\kappa }_{hj}{∥d_k∥}^{1+v}\hfill \\ & \le c_5{∥{\overline{x}}_k-x_k∥}^{min\{1+\delta ,1+v\}}+{\kappa }_{hj}{\overline{c}}^{1+v}{∥{\overline{x}}_k-x_k∥}^{min\{1+v,(1+v)(1+v-\frac{\delta }{2\gamma })\}}\hfill \\ & \le \left(c_5+{\kappa }_{hj}{\overline{c}}^{1+v}\right){∥{\overline{x}}_k-x_k∥}^{min\{1+\delta ,1+v,(1+v)(1+v-\frac{\delta }{2\gamma })\}}.\hfill \end{array}$$

Thus,

$$c∥{\overline{x}}_{k+1}-x_{k+1}∥\le {\left(c_5+{\kappa }_{hj}{\overline{c}}^{1+v}\right)}^{\gamma }{∥{\overline{x}}_k-x_k∥}^{min\{\gamma (1+\delta ),\gamma (1+v),\gamma (1+v)(1+v-\frac{\delta }{2\gamma })\}},$$

(38)

which indicates that $\left\{x_k\right\}$ converges to the solution set $X^{\ast }$ of Equation (1) with convergence rate $min\left\{\gamma \right(1+$$\delta ),\gamma (1+v),\gamma (1+v)(1+v-\frac{\delta }{2\gamma })\}$.

(2) The proof of $∥F_k∥>1$ is similar to the proof of ${∥F_k∥}^{-\delta }\le 1$. We obtain

$$c∥{\overline{x}}_{k+1}-x_{k+1}∥\le {\left(c_6+{\kappa }_{hj}{\overline{c}}^{1+v}\right)}^{\gamma }{∥{\overline{x}}_k-x_k∥}^{\gamma };$$

thus, $\left\{x_k\right\}$ converges to the solution set $X^{\ast }$ of Equation (1) with order $\gamma$. □

Theorem 3.

Under Assumption 2, we have the following:

(1) If $\parallel F_k\parallel \le 1,$$v>\frac{1}{\gamma }-1$, $\frac{1}{\gamma }-1<\delta \le 2\gamma v$, then $\left\{x_k\right\}$ generated by the ALLM algorithm converges to some solution of Equation (1) with order $min\left\{\gamma \right(1+\delta ),\gamma (1+v\left)\right\}$.

(2) If $\parallel F_k\parallel >1,$$v>\frac{1}{\gamma }-1$, then $\left\{x_k\right\}$ generated by the ALLM algorithm converges to some solution of Equation (1) with order γ.

Proof. 

(1) If $\parallel F_k\parallel \le 1$ and $v\ge \frac{\delta }{2\gamma }$, then from Equation (28) we obtain

$$∥d_k∥\le \overline{c}∥{\overline{x}}_k-x_k∥.$$

(39)

It follows from $v>\frac{1}{\gamma }-1$ and $v\ge \frac{\delta }{2\gamma }$ that

$$\begin{array}{c}\hfill \left(\frac{1}{\gamma }-1\right)-\left(\frac{1-\gamma }{\gamma (1+v)-\frac{\delta }{2}}\right)\ge 0\end{array}$$

and

$$\left(\frac{1-\gamma }{\gamma (1+v)-\frac{\delta }{2}}\right)-\left(\frac{1}{\gamma (1+v)-\frac{\delta }{2}}-1\right)\ge 0.$$

Therefore, the conditions of Lemma 4 (1) hold. In conjunction with $\delta >\frac{1}{\gamma }-1$ and $\delta \le 2\gamma v$, this yields

$$\begin{array}{cc}\hfill min& \left\{\gamma (1+\delta ),\gamma (1+v),\gamma (1+v)\left(1+v-\frac{\delta }{2\gamma }\right)\right\}\hfill \\ \hfill =& min\left\{\gamma \right(1+\delta ),\gamma (1+v\left)\right\}\hfill \\ \hfill >& 1.\hfill \end{array}$$

(40)

Thus, $\left\{x_k\right\}$ converges superlinearly to $X^{\ast }$.

For clearness,

$$∥{\overline{x}}_k-x_k∥\le ∥{\overline{x}}_{k+1}-x_{k+1}∥+∥d_k∥.$$

(41)

In view of Equations (38) and (40), we know the existence of a constant $M>0$, meaning that

$$∥{\overline{x}}_k-x_k∥\le M∥d_k∥$$

(42)

holds for large k. Thus, from Equations (38), (39), (40), and (42), we have

$$∥d_{k+1}∥\le O\left({∥d_k∥}^{min\left\{\gamma \right(1+\delta ),\gamma (1+v\left)\right\}}\right),$$

which means that the ALLM algorithm converges with order $min\left\{\gamma \right(1+\delta ),\gamma (1+v\left)\right\}$.

(2) The proof of $∥F_k∥>1$ is similar to the proof of $\parallel F_k\parallel \le 1$. We obtain

$$∥d_{k+1}∥\le O\left({∥d_k∥}^{\gamma }\right);$$

thus, ALLM algorithm converges with order $\gamma$. □

4. Numerical Experiments

In this section, we verify the effectiveness of the ALLM algorithm by presenting some numerical experiments. Algorithm 1 (named the AELM algorithm) from [22] is used for comparison. All algorithms were tested in the MATLAB R2022b programming environment on a personal PC with an i7-7500U CPU and 2.7 GHz. We selected the parameters of the AELM algorithm as follows: $p_0={10}^{-4}$, $p_1=0.25,$$p_2=0.75,N_0=5,{\mu }_1=0.01,m={10}^{-8}$. We selected the parameters of the ALLM algorithm as follows: $p_0={10}^{-4},p_1=0.25,p_2=0.75,N_0=5,{\mu }_1=0.01,m={10}^{-8},\theta =0,0.5,1,\delta =1,2.$ All algorithms were terminated when $∥J_k^T{F}_k∥\le {10}^{-5}$ or when the number of iterations surpassed 1000.

Example 1.

We consider four special functions [22] to verify that the ALLM algorithm satisfies more theoretical applications. Functions 1–4 satisfy the Hölderian local error bound condition around the zero point but do not satisfy the local error bound condition. Here, the $J\left(x\right)$ for Functions 3–4 are Hölderian continuous but not Lipschitz continuous, while the $J\left(x\right)$ for Functions 1–2 are both Lipschitz continuous and Hölderian continuous.

Function 1

$$\begin{array}{cc}\hfill f_1\left(x\right)& =x_1+10x_2,\hfill \\ \hfill f_2\left(x\right)& =\sqrt{5}(x_3-x_4),\hfill \\ \hfill f_3\left(x\right)& ={\left(x_2-2x_3\right)}^2,\hfill \\ \hfill f_4\left(x\right)& =\sqrt{10}{(x_1-x_4)}^2.\hfill \end{array}$$

Initial point: $x_0={(3,-1,0,1)}^T$, zero point: ${(0,0,0,0)}^T$.

Function 2

$$\begin{array}{cc}\hfill f_1\left(x\right)& =x_1{x}_2,\hfill \\ \hfill f_2\left(x\right)& =x_1^2+x_2^2.\hfill \end{array}$$

Initial point: $x_0={(1,1)}^T$, zero point: ${(0,0)}^T$.

Function 3

$$\begin{array}{cc}& f_1\left(x\right)=x_1+10x_2,\hfill \\ & f_2\left(x\right)=x_3-x_4,\hfill \\ & f_3\left(x\right)={(x_2-2x_3)}^{\frac{3}{2}},\hfill \\ & f_4\left(x\right)={(x_1-x_4)}^{\frac{3}{2}}.\hfill \end{array}$$

Initial point: $x_0={(3,1,0,1)}^T$, zero point: ${(0,0,0,0)}^T$.

Function 4

$$\begin{array}{cc}& f_1\left(x\right)=x_1+10x_2,\hfill \\ & f_2\left(x\right)=x_3-x_4,\hfill \\ & f_3\left(x\right)={(x_2-2x_3)}^{\frac{4}{3}},\hfill \\ & f_4\left(x\right)={(x_1-x_4)}^{\frac{4}{3}}.\hfill \end{array}$$

Initial point: $x_0={(3,-1,0,1)}^T$, zero point: ${(0,0,0,0)}^T$.

We tested each function for three starting points, $x_0,10x_0$ and $100x_0$, to study the global convergence of the ALLM algorithm. Table 1 lists the numerical results achieved by the AELM and ALLM algorithms on the four test functions. The symbols in Table 1 have the following meanings:

• NF: The number of function calculations.
• NJ: The number of Jacobian calculations.
• NT: We generally use the ‘$\mathrm{NT}=\mathrm{NF}+\mathrm{NJ}\ast n$’ to indicate the total computations.

Table 1. Numerical results of the AELM and ALLM algorithms with various choices of $\delta$ and $\theta$.

			AELM				ALLM
					$\mathit{\delta }=\mathbf{1}$			$\mathit{\delta }=\mathbf{2}$
Function	$\mathit{n}$	${\mathit{x}}_{\mathbf{0}}$		$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$	$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$
			NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT
1	4	1	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50
		10	13/13/65	13/13/65	13/13/65	13/13/65	13/13/65	13/13/65	13/13/65
		100	16/16/80	16/16/80	16/16/80	16/16/80	16/16/80	16/16/80	16/16/80
2	2	1	8/8/24	8/8/24	8/8/24	8/8/24	8/8/24	8/8/24	8/8/24
		10	11/11/33	11/11/33	11/11/33	11/11/33	11/11/33	11/11/33	11/11/33
		100	15/15/45	15/15/45	15/15/45	15/15/45	15/15/45	15/15/45	15/15/45
3	4	1	8/8/40	8/8/40	8/8/40	8/8/40	8/8/40	8/8/40	8/8/40
		10	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50
		100	12/12/60	12/12/60	12/12/60	12/12/60	12/12/60	12/12/60	12/12/60
4	4	1	13/13/65	7/7/35	7/7/35	7/7/35	7/7/35	7/7/35	7/7/35
		10	16/16/80	9/9/45	9/9/45	9/9/45	9/9/45	9/9/45	9/9/45
		100	61/50/261	11/11/55	11/11/55	11/11/55	11/11/55	11/11/55	11/11/55

Table 1. Numerical results of the AELM and ALLM algorithms with various choices of $\delta$ and $\theta$.

			AELM				ALLM
					$\mathit{\delta }=\mathbf{1}$			$\mathit{\delta }=\mathbf{2}$
Function	$\mathit{n}$	${\mathit{x}}_{\mathbf{0}}$		$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$	$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$
			NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT	NF/NJ/NT
1	4	1	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50
		10	13/13/65	13/13/65	13/13/65	13/13/65	13/13/65	13/13/65	13/13/65
		100	16/16/80	16/16/80	16/16/80	16/16/80	16/16/80	16/16/80	16/16/80
2	2	1	8/8/24	8/8/24	8/8/24	8/8/24	8/8/24	8/8/24	8/8/24
		10	11/11/33	11/11/33	11/11/33	11/11/33	11/11/33	11/11/33	11/11/33
		100	15/15/45	15/15/45	15/15/45	15/15/45	15/15/45	15/15/45	15/15/45
3	4	1	8/8/40	8/8/40	8/8/40	8/8/40	8/8/40	8/8/40	8/8/40
		10	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50	10/10/50
		100	12/12/60	12/12/60	12/12/60	12/12/60	12/12/60	12/12/60	12/12/60
4	4	1	13/13/65	7/7/35	7/7/35	7/7/35	7/7/35	7/7/35	7/7/35
		10	16/16/80	9/9/45	9/9/45	9/9/45	9/9/45	9/9/45	9/9/45
		100	61/50/261	11/11/55	11/11/55	11/11/55	11/11/55	11/11/55	11/11/55

As can be seen from Table 1, the ALLM algorithm is obviously superior to the AELM algorithm for the numerical results of Function 4, while the two algorithms are the same for the numerical results of Functions 1–3.

Example 2.

We consider some singular problems which are created by the following form [32]:

$$\widehat{F}\left(x\right)=F\left(x\right)-J\left(x^{\ast }\right)A{\left(A^{T}A\right)}^{-1}{A}^T\left(x-x^{\ast }\right),$$

where the test function $F\left(x\right)$ is provided by Moré, Garbow, and Hillstrom in [33], $x^{\ast }$ is the root of $F\left(x\right)$, and $A\in {\mathbb{R}}^{n\times k}$ has full column rank. It is clear that the Jacobian of $\widehat{F}\left(x^{\ast }\right)$ is

$$\widehat{J}\left(x^{\ast }\right)=J\left(x^{\ast }\right)\left(I-A{\left(A^{T}A\right)}^{-1}{A}^T\right),$$

with rank $n-k(1\le k\le n)$ and $\widehat{F}\left(x^{\ast }\right)=0$. Similar to [33], we choose

$$A={[1,1,\dots ,1]}^T\in {\mathbb{R}}^{n\times 1},$$

which implies $rank\left(\widehat{J}\left(x^{\ast }\right)\right)=n-1$.

Next, we ran all test problems for three starting points: $-10x_0,-x_0,x_0,10x_0$, and $100x_0$, where $x_0$ derives from [33]. Table 2 and Table 3 display the numerical results achieved by the algorithms for all test functions. The meanings of the symbols in Table 2 and Table 3 are as follows:

• Iter: Number of iterations.
• F: Final value of the norm of the function.
• Time: CPU time in seconds.

From Table 2 and Table 3, it is evident that the ALLM algorithm generally outperforms the AELM algorithm in terms of CPU time across most test functions. Compared with the AELM algorithm, the performance of the ALLM algorithm exhibits superior performance when $\theta =0$ and $\delta =2$, dominating approximately 90% of the CPU time results; about $4\%$ of the results of iterations of the two algorithms are the same. In particular, for certain test functions it can be seen that the ALLM algorithm consistently outperforms the AELM algorithm in terms of both iteration count and CPU time when the initial point is distant from the solution set. From Table 2, for the extended helical valley function, when $n=501$ and the initial point is $-10x_0$, the number of iterations and the CPU time of the ALLM algorithm are better than those of the AELM algorithm. From Table 3, for the discrete boundary value function, when $n=1000$ and the initial point is $-10x_0$ or $-x_0$ or $x_0$ or $10x_0$ or $100x_0$, the number of iterations and CPU time of the ALLM algorithm are better than those of the AELM algorithm.

To compare the numerical performance profile of the AELM and ALLM algorithms, we chose the performance analysis method proposed by Dolan [34]. As can be seen from Figure 1, when $\theta =0$ and $\delta =2$, the ALLM algorithm demonstrates the best performance in terms of iteration count, while when $\theta =1$ and $\delta =1$, the performance in terms of the number of iterations for both algorithms. As can be seen from Figure 2, when $\theta =0$ and $\delta =2$, the CPU time of the ALLM algorithm has the best performance, while when $\theta$ and $\delta$ take other values the ALLM algorithm maintains advantages in CPU time performance.

In general, the ALLM algorithm proves more effective in solving nonlinear equations compared to the AELM algorithm. In particular, when $\delta$ is larger and $\theta$ is smaller, the ALLM algorithm demonstrates superior performance. According to the needs of practical applications, the selection of ${\lambda }_k$ is continuously optimized by changing the values of $\delta$ and $\theta$.

Table 2. Numerical results of the AELM and ALLM algorithms with $\delta =1$ and various choices of $\theta$.

			AELM	ALLM
				$\mathit{\delta }=\mathbf{1}$
Function	$\mathit{n}$	${\mathit{x}}_{\mathbf{0}}$		$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$
			Iters/F/Time	Iters/F/Time	Iters/F/Time	Iters/F/Time
Extended Rosenbrock	500	−10	19/2.7631 × 10−7/0.30	19/2.7858 × 10−7/0.27	19/2.7744 × 10−7/0.29	19/2.7631 × 10−7/0.28
		−1	16/1.7341 × 10−7/0.23	14/2.5509 × 10−7/0.19	15/3.3852 × 10−7/0.20	16/1.7341 × 10−7/0.25
		1	17/2.2186 × 10−7/0.24	17/1.9553 × 10−7/0.25	17/2.0901 × 10−7/0.22	17/2.2186 × 10−7/0.23
		10	19/3.9252 × 10−7/0.29	19/3.8895 × 10−7/0.25	19/3.9066 × 10−7/0.27	19/3.9252 × 10−7/0.29
		100	23/1.3154 × 10−7/0.42	23/1.3142 × 10−7/0.30	23/1.3150 × 10−7/0.34	23/1.3154 × 10−7/0.37
	1000	−10	19/3.9156 × 10−7/1.63	19/3.9442 × 10−7/1.52	19/3.9299 × 10−7/1.53	19/3.9156 × 10−7/1.64
		−1	16/2.5034 × 10−7/1.43	14/3.8204 × 10−7/1.10	16/1.2047 × 10−7/1.28	16/2.5034 × 10−7/1.34
		1	17/3.1868 × 10−7/1.46	17/2.8057 × 10−7/1.40	17/2.9943 × 10−7/1.36	17/3.1868 × 10−7/1.46
		10	20/1.3911 × 10−7/1.67	20/1.3781 × 10−7/1.93	20/1.3866 × 10−7/1.65	20/1.3911 × 10−7/1.57
		100	23/1.8652 × 10−7/2.06	23/1.8646 × 10−7/1.90	23/1.8638 × 10−7/1.90	23/1.8652 × 10−7/1.98
Extended Helical valley	501	−10	42/1.3356 × 10−6/0.68	3/5.1316 × 10−7/0.03	13/3.7044 × 10−7/0.17	14/3.2573 × 10−7/0.23
		−1	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01
		1	8/3.1758 × 10−7/0.13	8/1.4137 × 10−7/0.09	8/2.1648 × 10−7/0.10	8/3.1758 × 10−7/0.12
		10	8/1.8981 × 10−9/0.12	8/8.0024 × 10−10/0.10	8/1.2568 × 10−9/0.10	8/1.8981 × 10−9/0.11
		100	8/5.9124 × 10−10/0.12	8/3.9747 × 10−10/0.11	8/4.8399 × 10−10/0.11	8/5.9124 × 10−10/0.12
	1000	−10	7/2.3766 × 10−13/0.53	6/3.0134 × 10−13/0.45	6/1.5143 × 10−9/0.44	7/2.3766 × 10−13/0.55
		−1	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04
		1	8/1.8337 × 10−8/0.69	8/7.2043 × 10−9/0.68	8/1.1804 × 10−8/0.62	8/1.8337 × 10−8/0.67
		10	8/1.6817 × 10−11/0.66	8/1.0758 × 10−11/0.61	8/1.4744 × 10−11/0.60	8/1.6817 × 10−11/0.65
		100	26/9.3824 × 10−13/2.51	35/1.0739 × 10−7/3.29	26/6.6675 × 10−8/2.18	26/6.9835 × 10−11/2.16
Discrete boundary value	500	−10	6/3.3487 × 10−3/0.11	6/3.3666 × 10−3/0.07	6/3.3631 × 10−3/0.09	6/3.3487 × 10−3/0.12
		−1	4/1.2234 × 10−3/0.06	4/1.2579 × 10−3/0.04	4/1.2417 × 10−3/0.05	4/1.2234 × 10−3/0.04
		1	3/3.5633 × 10−4/0.04	3/3.6008 × 10−4/0.03	3/3.5823 × 10−4/0.03	3/3.5633 × 10−4/0.03
		10	5/6.7290 × 10−3/0.08	5/6.7739 × 10−3/0.06	5/6.7614 × 10−3/0.06	5/6.7290 × 10−3/0.08
		100	12/1.3651 × 10−4/0.23	13/1.3834 × 10−5/0.16	12/1.5752 × 10−4/0.17	12/1.3651 × 10−4/0.16
	1000	−10	6/3.6656 × 10−3/0.52	6/3.6804 × 10−3/0.50	6/3.6780 × 10−3/0.47	6/3.6656 × 10−3/0.48
		−1	4/1.4253 × 10−3/0.30	4/1.4669 × 10−3/0.30	4/1.4474 × 10−3/0.30	4/1.4253 × 10−3/0.31
		1	3/1.3022 × 10−4/0.22	3/1.3092 × 10−4/0.20	3/1.3058 × 10−4/0.21	3/1.3022 × 10−4/0.21
		10	5/6.5900 × 10−3/0.40	5/6.6346 × 10−3/0.38	5/6.6209 × 10−3/0.35	5/6.5900 × 10−3/0.39
		100	13/9.9458 × 10−5/1.09	13/1.0869 × 10−4/1.06	13/1.0505 × 10−4/1.07	13/9.9458 × 10−5/1.08
Discrete integral equation	500	−10	12/1.2304 × 10−5/1.06	12/1.2171 × 10−5/1.03	12/1.2238 × 10−5/1.04	12/1.2304 × 10−5/1.06
		−1	9/1.5928 × 10−5/0.76	9/1.4153 × 10−5/0.76	9/1.5162 × 10−5/0.75	9/1.5928 × 10−5/0.76
		1	7/1.3357 × 10−5/0.59	7/1.3770 × 10−5/0.58	7/1.3592 × 10−5/0.58	7/1.3357 × 10−5/0.58
		10	10/9.3502 × 10−6/0.86	8/9.0151 × 10−6/0.67	9/1.5419 × 10−5/0.76	10/9.3502 × 10−6/0.86
		100	10/4.5155 × 10−9/0.91	10/4.5463 × 10−9/0.89	10/4.5306 × 10−9/0.88	10/4.5155 × 10−9/0.91
	1000	−10	12/1.7452 × 10−5/4.50	12/1.7265 × 10−5/4.48	12/1.7358 × 10−5/4.50	12/1.7452 × 10−5/4.51
		−1	10/6.0308 × 10−6/3.71	10/5.2005 × 10−6/3.73	10/5.6998 × 10−6/3.70	10/6.0308 × 10−6/3.67
		1	8/5.1495 × 10−6/2.86	8/5.3838 × 10−6/2.90	8/5.2754 × 10−6/2.50	8/5.1495 × 10−6/2.86
		10	10/1.4251 × 10−5/3.73	9/5.0675 × 10−6/3.30	10/5.7297 × 10−6/3.59	10/1.4251e × 10−5/3.66
		100	10/6.3828 × 10−9/3.86	10/6.4261 × 10−9/3.83	10/6.4040 × 10−9/3.81	10/6.3828 × 10−9/3.83
Broyden banded	500	−10	10/3.8446 × 10−12/0.17	10/3.9166 × 10−12/0.16	10/4.3882 × 10−12/0.14	10/3.8446 × 10−12/0.17
		−1	26/6.9212 × 10−6/0.52	31/1.2468 × 10−5/0.53	28/1.6756 × 10−5/0.50	25/1.2128 × 10−5/0.50
		1	12/1.5063 × 10−5/0.20	12/1.5060 × 10−5/0.20	12/1.5061 × 10−5/0.19	12/1.5063 × 10−5/0.20
		10	18/1.7636 × 10−5/0.33	18/1.7636 × 10−5/0.30	18/1.7636 × 10−5/0.28	18/1.7636 × 10−5/0.28
		100	24/1.0280 × 10−5/0.44	24/1.0280 × 10−5/0.36	24/1.0280 × 10−5/0.37	24/1.0280 × 10−5/0.37
	1000	−10	10/3.5499 × 10−12/0.90	10/3.8124 × 10−12/0.91	10/4.6220 × 10−12/0.90	10/3.5499 × 10−12/0.99
		−1	33/9.9927 × 10−6/3.35	27/9.6110 × 10−6/2.54	33/2.6949 × 10−5/3.04	28/9.7912 × 10−6/2.62
		1	12/2.1201 × 10−5/1.22	12/2.1196 × 10−5/1.08	12/2.1199 × 10−5/1.09	12/2.1201 × 10−5/1.17
		10	18/2.4886 × 10−5/1.82	18/2.4886 × 10−5/1.59	18/2.4886 × 10−5/1.68	18/2.4886 × 10−5/1.68
		100	24/1.4499 × 10−5/2.80	24/1.4499 × 10−5/2.42	24/1.4499 × 10−5/2.37	24/1.4499 × 10−5/2.54

Table 2. Numerical results of the AELM and ALLM algorithms with $\delta =1$ and various choices of $\theta$.

			AELM	ALLM
				$\mathit{\delta }=\mathbf{1}$
Function	$\mathit{n}$	${\mathit{x}}_{\mathbf{0}}$		$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$
			Iters/F/Time	Iters/F/Time	Iters/F/Time	Iters/F/Time
Extended Rosenbrock	500	−10	19/2.7631 × 10−7/0.30	19/2.7858 × 10−7/0.27	19/2.7744 × 10−7/0.29	19/2.7631 × 10−7/0.28
		−1	16/1.7341 × 10−7/0.23	14/2.5509 × 10−7/0.19	15/3.3852 × 10−7/0.20	16/1.7341 × 10−7/0.25
		1	17/2.2186 × 10−7/0.24	17/1.9553 × 10−7/0.25	17/2.0901 × 10−7/0.22	17/2.2186 × 10−7/0.23
		10	19/3.9252 × 10−7/0.29	19/3.8895 × 10−7/0.25	19/3.9066 × 10−7/0.27	19/3.9252 × 10−7/0.29
		100	23/1.3154 × 10−7/0.42	23/1.3142 × 10−7/0.30	23/1.3150 × 10−7/0.34	23/1.3154 × 10−7/0.37
	1000	−10	19/3.9156 × 10−7/1.63	19/3.9442 × 10−7/1.52	19/3.9299 × 10−7/1.53	19/3.9156 × 10−7/1.64
		−1	16/2.5034 × 10−7/1.43	14/3.8204 × 10−7/1.10	16/1.2047 × 10−7/1.28	16/2.5034 × 10−7/1.34
		1	17/3.1868 × 10−7/1.46	17/2.8057 × 10−7/1.40	17/2.9943 × 10−7/1.36	17/3.1868 × 10−7/1.46
		10	20/1.3911 × 10−7/1.67	20/1.3781 × 10−7/1.93	20/1.3866 × 10−7/1.65	20/1.3911 × 10−7/1.57
		100	23/1.8652 × 10−7/2.06	23/1.8646 × 10−7/1.90	23/1.8638 × 10−7/1.90	23/1.8652 × 10−7/1.98
Extended Helical valley	501	−10	42/1.3356 × 10−6/0.68	3/5.1316 × 10−7/0.03	13/3.7044 × 10−7/0.17	14/3.2573 × 10−7/0.23
		−1	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01
		1	8/3.1758 × 10−7/0.13	8/1.4137 × 10−7/0.09	8/2.1648 × 10−7/0.10	8/3.1758 × 10−7/0.12
		10	8/1.8981 × 10−9/0.12	8/8.0024 × 10−10/0.10	8/1.2568 × 10−9/0.10	8/1.8981 × 10−9/0.11
		100	8/5.9124 × 10−10/0.12	8/3.9747 × 10−10/0.11	8/4.8399 × 10−10/0.11	8/5.9124 × 10−10/0.12
	1000	−10	7/2.3766 × 10−13/0.53	6/3.0134 × 10−13/0.45	6/1.5143 × 10−9/0.44	7/2.3766 × 10−13/0.55
		−1	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04
		1	8/1.8337 × 10−8/0.69	8/7.2043 × 10−9/0.68	8/1.1804 × 10−8/0.62	8/1.8337 × 10−8/0.67
		10	8/1.6817 × 10−11/0.66	8/1.0758 × 10−11/0.61	8/1.4744 × 10−11/0.60	8/1.6817 × 10−11/0.65
		100	26/9.3824 × 10−13/2.51	35/1.0739 × 10−7/3.29	26/6.6675 × 10−8/2.18	26/6.9835 × 10−11/2.16
Discrete boundary value	500	−10	6/3.3487 × 10−3/0.11	6/3.3666 × 10−3/0.07	6/3.3631 × 10−3/0.09	6/3.3487 × 10−3/0.12
		−1	4/1.2234 × 10−3/0.06	4/1.2579 × 10−3/0.04	4/1.2417 × 10−3/0.05	4/1.2234 × 10−3/0.04
		1	3/3.5633 × 10−4/0.04	3/3.6008 × 10−4/0.03	3/3.5823 × 10−4/0.03	3/3.5633 × 10−4/0.03
		10	5/6.7290 × 10−3/0.08	5/6.7739 × 10−3/0.06	5/6.7614 × 10−3/0.06	5/6.7290 × 10−3/0.08
		100	12/1.3651 × 10−4/0.23	13/1.3834 × 10−5/0.16	12/1.5752 × 10−4/0.17	12/1.3651 × 10−4/0.16
	1000	−10	6/3.6656 × 10−3/0.52	6/3.6804 × 10−3/0.50	6/3.6780 × 10−3/0.47	6/3.6656 × 10−3/0.48
		−1	4/1.4253 × 10−3/0.30	4/1.4669 × 10−3/0.30	4/1.4474 × 10−3/0.30	4/1.4253 × 10−3/0.31
		1	3/1.3022 × 10−4/0.22	3/1.3092 × 10−4/0.20	3/1.3058 × 10−4/0.21	3/1.3022 × 10−4/0.21
		10	5/6.5900 × 10−3/0.40	5/6.6346 × 10−3/0.38	5/6.6209 × 10−3/0.35	5/6.5900 × 10−3/0.39
		100	13/9.9458 × 10−5/1.09	13/1.0869 × 10−4/1.06	13/1.0505 × 10−4/1.07	13/9.9458 × 10−5/1.08
Discrete integral equation	500	−10	12/1.2304 × 10−5/1.06	12/1.2171 × 10−5/1.03	12/1.2238 × 10−5/1.04	12/1.2304 × 10−5/1.06
		−1	9/1.5928 × 10−5/0.76	9/1.4153 × 10−5/0.76	9/1.5162 × 10−5/0.75	9/1.5928 × 10−5/0.76
		1	7/1.3357 × 10−5/0.59	7/1.3770 × 10−5/0.58	7/1.3592 × 10−5/0.58	7/1.3357 × 10−5/0.58
		10	10/9.3502 × 10−6/0.86	8/9.0151 × 10−6/0.67	9/1.5419 × 10−5/0.76	10/9.3502 × 10−6/0.86
		100	10/4.5155 × 10−9/0.91	10/4.5463 × 10−9/0.89	10/4.5306 × 10−9/0.88	10/4.5155 × 10−9/0.91
	1000	−10	12/1.7452 × 10−5/4.50	12/1.7265 × 10−5/4.48	12/1.7358 × 10−5/4.50	12/1.7452 × 10−5/4.51
		−1	10/6.0308 × 10−6/3.71	10/5.2005 × 10−6/3.73	10/5.6998 × 10−6/3.70	10/6.0308 × 10−6/3.67
		1	8/5.1495 × 10−6/2.86	8/5.3838 × 10−6/2.90	8/5.2754 × 10−6/2.50	8/5.1495 × 10−6/2.86
		10	10/1.4251 × 10−5/3.73	9/5.0675 × 10−6/3.30	10/5.7297 × 10−6/3.59	10/1.4251e × 10−5/3.66
		100	10/6.3828 × 10−9/3.86	10/6.4261 × 10−9/3.83	10/6.4040 × 10−9/3.81	10/6.3828 × 10−9/3.83
Broyden banded	500	−10	10/3.8446 × 10−12/0.17	10/3.9166 × 10−12/0.16	10/4.3882 × 10−12/0.14	10/3.8446 × 10−12/0.17
		−1	26/6.9212 × 10−6/0.52	31/1.2468 × 10−5/0.53	28/1.6756 × 10−5/0.50	25/1.2128 × 10−5/0.50
		1	12/1.5063 × 10−5/0.20	12/1.5060 × 10−5/0.20	12/1.5061 × 10−5/0.19	12/1.5063 × 10−5/0.20
		10	18/1.7636 × 10−5/0.33	18/1.7636 × 10−5/0.30	18/1.7636 × 10−5/0.28	18/1.7636 × 10−5/0.28
		100	24/1.0280 × 10−5/0.44	24/1.0280 × 10−5/0.36	24/1.0280 × 10−5/0.37	24/1.0280 × 10−5/0.37
	1000	−10	10/3.5499 × 10−12/0.90	10/3.8124 × 10−12/0.91	10/4.6220 × 10−12/0.90	10/3.5499 × 10−12/0.99
		−1	33/9.9927 × 10−6/3.35	27/9.6110 × 10−6/2.54	33/2.6949 × 10−5/3.04	28/9.7912 × 10−6/2.62
		1	12/2.1201 × 10−5/1.22	12/2.1196 × 10−5/1.08	12/2.1199 × 10−5/1.09	12/2.1201 × 10−5/1.17
		10	18/2.4886 × 10−5/1.82	18/2.4886 × 10−5/1.59	18/2.4886 × 10−5/1.68	18/2.4886 × 10−5/1.68
		100	24/1.4499 × 10−5/2.80	24/1.4499 × 10−5/2.42	24/1.4499 × 10−5/2.37	24/1.4499 × 10−5/2.54

Table 3. Numerical results of the AELM and ALLM algorithms with $\delta =2$ and various choices of $\theta$.

			AELM	ALLM
				$\mathit{\delta }=\mathbf{2}$
Function	$\mathit{n}$	${\mathit{x}}_{\mathbf{0}}$		$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$
			Iters/F/Time	Iters/F/Time	Iters/F/Time	Iters/F/Time
Extended Rosenbrock	500	−10	19/2.7631 × 10−7/0.30	19/2.7841 × 10−7/0.26	19/2.7734 × 10−7/0.26	19/2.7635 × 10−7/0.30
		−1	16/1.7341 × 10−7/0.23	14/1.7729 × 10−7/0.17	15/3.2343 × 10−7/0.21	16/1.7288 × 10−7/0.20
		1	17/2.2186 × 10−7/0.24	17/1.8677 × 10−7/0.25	17/2.0435 × 10−7/0.25	17/2.2121 × 10−7/0.22
		10	19/3.9252 × 10−7/0.29	19/3.8875 × 10−7/0.24	19/3.9061 × 10−7/0.26	19/3.9243 × 10−7/0.25
		100	23/1.3154 × 10−7/0.42	23/1.3134 × 10−7/0.29	23/1.3145 × 10−7/0.33	23/1.3149 × 10−7/0.33
	1000	−10	19/3.9156 × 10−7/1.63	19/3.9423 × 10−7/1.50	19/3.9284 × 10−7/1.54	19/3.9140 × 10−7/1.56
		−1	16/2.5034 × 10−7/1.43	14/2.3878 × 10−7/1.05	15/4.5237 × 10−7/1.24	16/2.4969 × 10−7/1.32
		1	17/3.1868 × 10−7/1.46	17/2.7048 × 10−7/1.33	17/2.9369 × 10−7/1.36	17/3.1809 × 10−7/1.33
		10	20/1.3911 × 10−7/1.67	20/1.3786 × 10−7/1.53	20/1.3857 × 10−7/1.57	20/1.3919e × 10−7/1.59
		100	23/1.8652 × 10−7/2.06	23/1.8649 × 10−7/1.83	23/1.8655 × 10−7/1.80	23/1.8633 × 10−7/1.81
Extended Helical valley	501	−10	42/1.3356 × 10−6/0.68	3/7.5853 × 10−13/0.04	13/1.7643 × 10−7/0.18	14/2.0341 × 10−7/0.19
		−1	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01
		1	8/3.1758 × 10−7/0.13	8/1.0595 × 10−7/0.11	8/1.9230 × 10−7/0.11	8/3.2101 × 10−7/0.09
		10	8/1.8981 × 10−9/0.12	8/4.7841 × 10−10/0.11	8/9.5278 × 10−10/0.10	8/1.6935 × 10−9/0.10
		100	8/5.9124 × 10−10/0.12	8/2.0055 × 10−10/0.10	8/3.2729 × 10−10/0.13	8/4.9161 × 10−10/0.13
	1000	−10	7/2.3766 × 10−13/0.53	5/7.4481 × 10−10/0.36	6/1.7998 × 10−12/0.51	6/4.6416 × 10−7/0.42
		−1	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04
		1	8/1.8337 × 10−8/0.69	8/4.1226 × 10−9/0.59	8/9.1462 × 10−9/0.62	8/1.7753 × 10−8/0.63
		10	8/1.6817 × 10−11/0.66	8/2.2703 × 10−11/0.59	8/3.2386 × 10−11/0.60	8/4.0803 × 10−11/0.64
		100	26/9.3824 × 10−13/2.51	46/2.6770 × 10−11/3.70	26/8.5493 × 10−9/2.09	26/2.8466 × 10−10/2.15
Discrete boundary value	500	−10	6/3.3487 × 10−3/0.11	4/3.1555 × 10−3/0.04	4/4.2613 × 10−3/0.04	4/4.7919 × 10−3/0.05
		−1	4/1.2234 × 10−3/0.06	3/7.2588 × 10−4/0.03	3/7.1034 × 10−4/0.03	3/6.9394 × 10−4/0.04
		1	3/3.5633 × 10−4/0.04	3/4.1479 × 10−6/0.03	3/4.1402 × 10−6/0.04	3/4.1324 × 10−6/0.04
		10	5/6.7290 × 10−3/0.08	4/2.4328 × 10−3/0.05	4/ 3.0758 × 10−3/0.05	4/3.4218 × 10−3/0.05
		100	12/1.3651 × 10−4/0.23	11/4.2188 × 10−5/0.16	12/1.5591 × 10−5/0.18	12/1.3814 × 10−5/0.18
	1000	−10	6/3.6656 × 10−3/0.52	4/3.5180 × 10−3/0.29	4/4.5349 × 10−3/0.29	4/5.0429 × 10−3/0.28
		−1	4/1.4253 × 10−3/0.30	3/9.3230 × 10−4/0.21	3/9.1090 × 10−4/0.21	3/8.8813 × 10−4/0.23
		1	3/1.3022 × 10−4/0.22	2/2.6311 × 10−4/0.13	2/2.6303 × 10−4/0.13	2/2.6296 × 10−4/0.13
		10	5/6.5900 × 10−3/0.40	4/2.5604 × 10−3/0.29	4/3.0932 × 10−3/0.30	4/3.4004 × 10−3/0.27
		100	13/9.9458 × 10−5/1.09	11/1.2743 × 10−4/0.88	11/2.1884 × 10−4/0.93	11/2.1536 × 10−4/0.85
Discrete integral equation	500	−10	12/1.2304 × 10−5/1.06	12/1.2047 × 10−5/1.03	12/1.2171 × 10−5/1.05	12/1.2294 × 10−5/1.04
		−1	9/1.5928 × 10−5/0.76	9/1.0655 × 10−5/0.75	9/1.2735 × 10−5/0.76	9/ 1.4195 × 10−5/0.76
		1	7/1.3357 × 10−5/0.59	7/1.0869 × 10−5/0.57	7/1.0772 × 10−5/0.58	7/1.0633 × 10−5/0.57
		10	10/9.3502 × 10−6/0.86	9/4.8669 × 10−6/0.75	9/1.3758 × 10−5/0.76	10/9.7578 × 10−6/0.84
		100	10/4.5155 × 10−9/0.91	10/4.5453 × 10−9/0.89	10/4.5300 × 10−9/0.90	10/4.5154 × 10−9/0.88
	1000	−10	12/1.7452 × 10−5/4.50	12/1.7133 × 10−5/4.41	12/1.7285 × 10−5/4.46	12/1.7441 × 10−5/4.47
		−1	10/6.0308 × 10−6/3.71	9/1.5092 × 10−5/3.25	9/1.8533 × 10−5/3.27	10/5.2150 × 10−6/3.66
		1	8/5.1495 × 10−6/2.86	7/1.5749 × 10−5/2.48	7/1.5610 × 10−5/2.50	7/1.5367 × 10−5/2.62
		10	10/1.4251 × 10−5/3.73	9/8.5398 × 10−6/3.25	10/5.1845 × 10−6/3.62	10/1.4626 × 10−5/3.71
		100	10/6.3828 × 10−9/3.86	10/6.4246 × 10−9/3.76	10/6.4031 × 10−9/3.79	10/6.3825 × 10−9/3.77
Broyden banded	500	−10	10/3.8446 × 10−12/0.17	10/3.795 × 10−12/0.16	10/4.3814 × 10−12/0.17	10/3.8105 × 10−12/0.16
		−1	26/6.9212 × 10−6/0.52	29/6.5177 × 10−6/0.45	28/1.6459 × 10−5/0.44	25/1.1907 × 10−5/0.42
		1	12/1.5063 × 10−5/0.20	12/1.5059 × 10−5/0.20	12/1.5061 × 10−5/0.20	12/1.5063 × 10−5/0.18
		10	18/1.7636 × 10−5/0.33	18/1.7636 × 10−5/0.30	18/1.7636 × 10−5/0.33	18/1.7636 × 10−5/0.31
		100	24/1.0280 × 10−5/0.44	24/1.0280 × 10−5/0.36	24/1.0280 × 10−5/0.40	24/1.0280 × 10−5/0.36
	1000	−10	10/3.5499 × 10−12/0.90	10/4.5936 × 10−12/0.86	10/3.810 × 10−12/0.89	10/5.7143 × 10−12/0.93
		−1	33/9.9927 × 10−6/3.35	29/1.5374 × 10−5/2.76	31/1.8408 × 10−5 /2.84	28/9.7968 × 10−6/2.61
		1	12/2.1201 × 10−5/1.22	12/2.1194 × 10−5/1.07	12/2.1198 × 10−5/1.08	12/2.1201 × 10−5/1.13
		10	18/2.4886 × 10−5/1.82	18/2.4886 × 10−5/1.69	18/2.4886 × 10−5/1.64	18/2.4886 × 10−5/1.65
		100	24/1.4499 × 10−5/2.80	24/1.4499 × 10−5/2.30	24/1.4499 × 10−5/2.33	24/1.4499 × 10−5/2.51

Table 3. Numerical results of the AELM and ALLM algorithms with $\delta =2$ and various choices of $\theta$.

			AELM	ALLM
				$\mathit{\delta }=\mathbf{2}$
Function	$\mathit{n}$	${\mathit{x}}_{\mathbf{0}}$		$\mathit{\theta }=\mathbf{0}$	$\mathit{\theta }=\mathbf{0.5}$	$\mathit{\theta }=\mathbf{1}$
			Iters/F/Time	Iters/F/Time	Iters/F/Time	Iters/F/Time
Extended Rosenbrock	500	−10	19/2.7631 × 10−7/0.30	19/2.7841 × 10−7/0.26	19/2.7734 × 10−7/0.26	19/2.7635 × 10−7/0.30
		−1	16/1.7341 × 10−7/0.23	14/1.7729 × 10−7/0.17	15/3.2343 × 10−7/0.21	16/1.7288 × 10−7/0.20
		1	17/2.2186 × 10−7/0.24	17/1.8677 × 10−7/0.25	17/2.0435 × 10−7/0.25	17/2.2121 × 10−7/0.22
		10	19/3.9252 × 10−7/0.29	19/3.8875 × 10−7/0.24	19/3.9061 × 10−7/0.26	19/3.9243 × 10−7/0.25
		100	23/1.3154 × 10−7/0.42	23/1.3134 × 10−7/0.29	23/1.3145 × 10−7/0.33	23/1.3149 × 10−7/0.33
	1000	−10	19/3.9156 × 10−7/1.63	19/3.9423 × 10−7/1.50	19/3.9284 × 10−7/1.54	19/3.9140 × 10−7/1.56
		−1	16/2.5034 × 10−7/1.43	14/2.3878 × 10−7/1.05	15/4.5237 × 10−7/1.24	16/2.4969 × 10−7/1.32
		1	17/3.1868 × 10−7/1.46	17/2.7048 × 10−7/1.33	17/2.9369 × 10−7/1.36	17/3.1809 × 10−7/1.33
		10	20/1.3911 × 10−7/1.67	20/1.3786 × 10−7/1.53	20/1.3857 × 10−7/1.57	20/1.3919e × 10−7/1.59
		100	23/1.8652 × 10−7/2.06	23/1.8649 × 10−7/1.83	23/1.8655 × 10−7/1.80	23/1.8633 × 10−7/1.81
Extended Helical valley	501	−10	42/1.3356 × 10−6/0.68	3/7.5853 × 10−13/0.04	13/1.7643 × 10−7/0.18	14/2.0341 × 10−7/0.19
		−1	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01	1/0.0000 × 100/0.01
		1	8/3.1758 × 10−7/0.13	8/1.0595 × 10−7/0.11	8/1.9230 × 10−7/0.11	8/3.2101 × 10−7/0.09
		10	8/1.8981 × 10−9/0.12	8/4.7841 × 10−10/0.11	8/9.5278 × 10−10/0.10	8/1.6935 × 10−9/0.10
		100	8/5.9124 × 10−10/0.12	8/2.0055 × 10−10/0.10	8/3.2729 × 10−10/0.13	8/4.9161 × 10−10/0.13
	1000	−10	7/2.3766 × 10−13/0.53	5/7.4481 × 10−10/0.36	6/1.7998 × 10−12/0.51	6/4.6416 × 10−7/0.42
		−1	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04	1/0.0000 × 100/0.04
		1	8/1.8337 × 10−8/0.69	8/4.1226 × 10−9/0.59	8/9.1462 × 10−9/0.62	8/1.7753 × 10−8/0.63
		10	8/1.6817 × 10−11/0.66	8/2.2703 × 10−11/0.59	8/3.2386 × 10−11/0.60	8/4.0803 × 10−11/0.64
		100	26/9.3824 × 10−13/2.51	46/2.6770 × 10−11/3.70	26/8.5493 × 10−9/2.09	26/2.8466 × 10−10/2.15
Discrete boundary value	500	−10	6/3.3487 × 10−3/0.11	4/3.1555 × 10−3/0.04	4/4.2613 × 10−3/0.04	4/4.7919 × 10−3/0.05
		−1	4/1.2234 × 10−3/0.06	3/7.2588 × 10−4/0.03	3/7.1034 × 10−4/0.03	3/6.9394 × 10−4/0.04
		1	3/3.5633 × 10−4/0.04	3/4.1479 × 10−6/0.03	3/4.1402 × 10−6/0.04	3/4.1324 × 10−6/0.04
		10	5/6.7290 × 10−3/0.08	4/2.4328 × 10−3/0.05	4/ 3.0758 × 10−3/0.05	4/3.4218 × 10−3/0.05
		100	12/1.3651 × 10−4/0.23	11/4.2188 × 10−5/0.16	12/1.5591 × 10−5/0.18	12/1.3814 × 10−5/0.18
	1000	−10	6/3.6656 × 10−3/0.52	4/3.5180 × 10−3/0.29	4/4.5349 × 10−3/0.29	4/5.0429 × 10−3/0.28
		−1	4/1.4253 × 10−3/0.30	3/9.3230 × 10−4/0.21	3/9.1090 × 10−4/0.21	3/8.8813 × 10−4/0.23
		1	3/1.3022 × 10−4/0.22	2/2.6311 × 10−4/0.13	2/2.6303 × 10−4/0.13	2/2.6296 × 10−4/0.13
		10	5/6.5900 × 10−3/0.40	4/2.5604 × 10−3/0.29	4/3.0932 × 10−3/0.30	4/3.4004 × 10−3/0.27
		100	13/9.9458 × 10−5/1.09	11/1.2743 × 10−4/0.88	11/2.1884 × 10−4/0.93	11/2.1536 × 10−4/0.85
Discrete integral equation	500	−10	12/1.2304 × 10−5/1.06	12/1.2047 × 10−5/1.03	12/1.2171 × 10−5/1.05	12/1.2294 × 10−5/1.04
		−1	9/1.5928 × 10−5/0.76	9/1.0655 × 10−5/0.75	9/1.2735 × 10−5/0.76	9/ 1.4195 × 10−5/0.76
		1	7/1.3357 × 10−5/0.59	7/1.0869 × 10−5/0.57	7/1.0772 × 10−5/0.58	7/1.0633 × 10−5/0.57
		10	10/9.3502 × 10−6/0.86	9/4.8669 × 10−6/0.75	9/1.3758 × 10−5/0.76	10/9.7578 × 10−6/0.84
		100	10/4.5155 × 10−9/0.91	10/4.5453 × 10−9/0.89	10/4.5300 × 10−9/0.90	10/4.5154 × 10−9/0.88
	1000	−10	12/1.7452 × 10−5/4.50	12/1.7133 × 10−5/4.41	12/1.7285 × 10−5/4.46	12/1.7441 × 10−5/4.47
		−1	10/6.0308 × 10−6/3.71	9/1.5092 × 10−5/3.25	9/1.8533 × 10−5/3.27	10/5.2150 × 10−6/3.66
		1	8/5.1495 × 10−6/2.86	7/1.5749 × 10−5/2.48	7/1.5610 × 10−5/2.50	7/1.5367 × 10−5/2.62
		10	10/1.4251 × 10−5/3.73	9/8.5398 × 10−6/3.25	10/5.1845 × 10−6/3.62	10/1.4626 × 10−5/3.71
		100	10/6.3828 × 10−9/3.86	10/6.4246 × 10−9/3.76	10/6.4031 × 10−9/3.79	10/6.3825 × 10−9/3.77
Broyden banded	500	−10	10/3.8446 × 10−12/0.17	10/3.795 × 10−12/0.16	10/4.3814 × 10−12/0.17	10/3.8105 × 10−12/0.16
		−1	26/6.9212 × 10−6/0.52	29/6.5177 × 10−6/0.45	28/1.6459 × 10−5/0.44	25/1.1907 × 10−5/0.42
		1	12/1.5063 × 10−5/0.20	12/1.5059 × 10−5/0.20	12/1.5061 × 10−5/0.20	12/1.5063 × 10−5/0.18
		10	18/1.7636 × 10−5/0.33	18/1.7636 × 10−5/0.30	18/1.7636 × 10−5/0.33	18/1.7636 × 10−5/0.31
		100	24/1.0280 × 10−5/0.44	24/1.0280 × 10−5/0.36	24/1.0280 × 10−5/0.40	24/1.0280 × 10−5/0.36
	1000	−10	10/3.5499 × 10−12/0.90	10/4.5936 × 10−12/0.86	10/3.810 × 10−12/0.89	10/5.7143 × 10−12/0.93
		−1	33/9.9927 × 10−6/3.35	29/1.5374 × 10−5/2.76	31/1.8408 × 10−5 /2.84	28/9.7968 × 10−6/2.61
		1	12/2.1201 × 10−5/1.22	12/2.1194 × 10−5/1.07	12/2.1198 × 10−5/1.08	12/2.1201 × 10−5/1.13
		10	18/2.4886 × 10−5/1.82	18/2.4886 × 10−5/1.69	18/2.4886 × 10−5/1.64	18/2.4886 × 10−5/1.65
		100	24/1.4499 × 10−5/2.80	24/1.4499 × 10−5/2.30	24/1.4499 × 10−5/2.33	24/1.4499 × 10−5/2.51

Figure 1. Performance profile of AELM and ALLM based on number of iterations for example 1–10.

Figure 1. Performance profile of AELM and ALLM based on number of iterations for example 1–10.

Figure 2. Performance profile of AELM and ALLM based on CPU time for example 1–10.

Figure 2. Performance profile of AELM and ALLM based on CPU time for example 1–10.

5. Conclusions

In this paper, inspired by the Hölderian local error bound condition, we studied the convergence properties of our ALLM algorithm under different conditions. We used the new modified adaptive LM parameter and incorporated the non-monotone technique to modify the Levenberg–Marquardt algorithm. The numerical results show that our new algorithm is efficient and stable.