A New Forward–Backward Algorithm with Line Searchand Inertial Techniques for Convex Minimization Problems with Applications

Abstract

For the past few decades, various algorithms have been proposed to solve convex minimization problems in the form of the sum of two lower semicontinuous and convex functions. The convergence of these algorithms was guaranteed under the L-Lipschitz condition on the gradient of the objective function. In recent years, an inertial technique has been widely used to accelerate the convergence behavior of an algorithm. In this work, we introduce a new forward–backward splitting algorithm using a new line search and inertial technique to solve convex minimization problems in the form of the sum of two lower semicontinuous and convex functions. A weak convergence of our proposed method is established without assuming the L-Lipschitz continuity of the gradient of the objective function. Moreover, a complexity theorem is also given. As applications, we employed our algorithm to solve data classification and image restoration by conducting some experiments on these problems. The performance of our algorithm was evaluated using various evaluation tools. Furthermore, we compared its performance with other algorithms. Based on the experiments, we found that the proposed algorithm performed better than other algorithms mentioned in the literature.

1. Introduction

The convex minimization problem has been studied intensively for the past few decades due to its wide range of applications in various real-world problems. Some major problems in physics, economics, data science, engineering, and medical science can be viewed as convex minimization problems, for instance a reaction–diffusion equation, which is a mathematical model describing physical phenomena, such as chemical reactions, heat flow models, and population dynamics. The problem of finding an unknown reaction term of such an equation can be formulated as a coefficient inverse problem (CIP) for a partial differential equation (PDE). Numerical methods for solving CIPs for PDEs, as well as their applications to various subjects have been widely studied and analyzed; for more comprehensive information on this topic, see [1,2,3,4,5]. To obtain a globally convergent method for solving CIPs for PDEs, many authors have employed the convexification technique, which reformulates CIPs as convex minimization problems; for a more in-depth development and discussion, we refer to [6,7,8]. Therefore, a numerical method for convex minimization problems can be applied. More examples of convex minimization problems are signal and image processing, compressed sensing, and machine learning tasks such as regression and classification; see [9,10,11,12,13,14,15,16] and the references therein for more information.

The problem is formulated, in the form of the summation of two convex functions, as follows:

$$\underset{x\in H}{min}\{f\left(x\right)+g\left(x\right)\},$$

(1)

where $f,g:H\to \mathbb{R}\cup \{+\infty \}$ are proper, lower semicontinuous convex functions and H is a Hilbert space.

If f is differentiable, then x solves (1) if and only if:

$$x=pro{x}_{\alpha g}(I-\alpha ▿f)\left(x\right),$$

(2)

where $\alpha >0,$ $pro{x}_{\alpha g}\left(x\right)=J_{\alpha }^{\partial g}\left(x\right)={(I-\alpha \partial g)}^{-1}\left(x\right),$ I is an identity mapping and $\partial g$ is a subdifferential of $g.$ In other words, x is a fixed point of $pro{x}_{\alpha g}(I-\alpha ▿f).$ Under some conditions, the Picard iteration converges to the solution. As a result, the forward–backward algorithm [17], which is defined as follows:

$$x_{n+1}=pro{x}_{{\alpha }_{n}g}(I-{\alpha }_n▿f)\left(x_n\right),\mathrm{for} \mathrm{all} n\in \mathbb{N},$$

(3)

where ${\alpha }_n$ is a suitable step size, converges to a solution of (1). Due to its simplicity, this method has been intensively studied and improved by many researchers; see [10,16,18] and the references therein for more information. One well-known method, which notably improves the convergence rate of (3), is the fast iterative shrinkage-thresholding algorithm (FISTA) introduced by Beck and Teboulle [19]. To the best of our knowledge, most of these works assumed that $▿f$ is Lipschitz continuous. However, in general, such an assumption might be too strong, and finding a Lipschitz constant of $▿f$ is sometimes difficult. To relax this strong assumption, Cruz and Nghia [20] proposed a line search technique and replaced the Lipschitz continuity assumption of $▿f$ with weaker assumptions, as seen in the following conditions:

Assumption A1.

$f,g$ are proper lower semicontinuous convex functions with $domg\subseteq domf;$

Assumption A2.

f is differentiable on an open set containing $domg,$ and $▿f$ is uniformly continuous on any bounded subset of $domg$ and maps any bounded subset of $domg$ to a bounded set in $H.$

Note that if $▿f$ is L-Lipschitz continuous, then A2 holds. They also established an algorithm using a line search technique and obtained a weak convergence result under Assumptions 1 and 2. In the same work, they also proposed an accelerated algorithm based on FISTA and a line search technique. They showed that this accelerated algorithm performed better than the other introduced algorithm.

Recently, inspired by [20], Kankam et al. [9] proposed a new line search technique and a new algorithm to solve (1). They conducted some experiments on signal processing and found that their method performed better than that of Cruz and Nghia [20].

In recent years, many authors have utilized the inertial technique in order to accelerate their algorithms. This was first introduced by Polyak [21] to solve smooth convex minimization problems. After that, many inertial-type algorithms were proposed and studied by many authors; see [22,23,24,25] and the references therein.

The algorithms introduced in [9,20] are easy to employ, since a Lipschitz constant of the gradient of an objective function f is not required to exist. Although the convergence of these algorithms is guaranteed, some improvements are still welcome, specifically utilizing an inertial technique in order to improve the performance. Hence, in this work, motivated by the works mentioned above, our main objective was to propose a new algorithm that utilizes both line search and inertial techniques to not only guarantee its convergence to a solution of (1) without assuming an L-Lipschitz continuity on $▿f$, but also to improve its performance by mean of an inertial technique. We established a weak convergence theorem under Assumptions 1 and 2. Moreover, a complexity theorem of this new algorithm was studied. Furthermore, we employed our proposed algorithm to solve an image restoration problem, as well as data classification problems. Then, we evaluated its performance and compared it with various other algorithms. The proposed method is also of great interest in solving nonlinear coefficient inverse problems for partial differential equations, along with other possible applications of convex minimization problems.

This work is organized as follows: In Section 2, we recall important definitions, lemmas that will be used in later sections, as well as various methods introduced in [9,19,20,22]. In Section 3, we introduce a new algorithm using new line search and inertial techniques and establish its weak convergence to a solution of (1). Moreover, the complexity of this method is also analyzed. In Section 4, in order to compare the performance of the studied algorithms, some numerical experiments on data classification problems and image restoration problem are conducted and discussed. In the last section, the brief conclusion of this paper is presented.

2. Preliminaries

Throughout this work, we denote $x_n\to x$ and $x_n\rightharpoonup x$ as $\left\{x_n\right\}$ converges strongly and weakly to x, respectively. For $h:H\to R\cup \{+\infty \}$, we also denote $domh:=\{x\in H:h(x)<+\infty \}.$

First, we recall some methods for solving (1) introduced by many authors mentioned in Section 1.

The fast iterative shrinkage-thresholding algorithm (FISTA) was introduced by Beck and Teboulle[19] as follows (Algorithm 1).

Algorithm 1: FISTA.

1: Input Given $y_1=x_0\in {\mathbb{R}}^n$, and $t_1=1,$ for $n\in \mathbb{N},$ $$\begin{array}{l}{x}_n=pro{x}_{\frac{1}{L}g}(y_n-\frac{1}{L}▿f(y_n)),\\ t_{n+1}=\frac{1+\sqrt{1+4t_n^2}}{2},{\theta }_n=\frac{t_n-1}{t_{n+1}},\\ y_{n+1}=x_n+{\theta }_n(x_n-x_{n-1}),n\in \mathbb{N},\end{array}$$ where L is a Lipschitz constant of $▿f$.

In 2016, Cruz and Nghia [20] introduced a line search step as follows (Algorithm 2).

Algorithm 2: Line Search 1$(x,\sigma ,\theta ,\delta )$.

1: Input Given $x\in domg,$$\sigma >0$, $\theta \in (0,1),$ and $\delta \in (0,\frac{1}{2}).$ 2: Set $\alpha =\sigma .$ 3: while$\alpha \parallel ▿f\left(pro{x}_{\alpha g}(x-\alpha ▿f\left(x\right))\right)-▿f\left(x\right)\parallel >\delta \parallel pro{x}_{\alpha g}(x-\alpha ▿f\left(x\right))-x\parallel$do 4:     Set $\alpha =\theta \alpha$ 5: end while 6: Output$\alpha .$

They asserted that Line Search 1 stops after finitely many steps and proposed the following algorithm (Algorithm 3).

Algorithm 3: Line Search 1 stops after finitely many steps.

1: Input Given $x_0\in domg,$$\sigma >0$, $\theta \in (0,1),$ and $\delta \in (0,\frac{1}{2}),$ for $n\in \mathbb{N}$, $$x_{n+1}=pro{x}_{{\alpha }_{n}g}(I-{\alpha }_n▿f)\left(x_n\right),$$ where ${\alpha }_n:=$Line Search 1$(x_n,\sigma ,\theta ,\delta ).$

They showed that a sequence generated by Algorithm 3 converges weakly to a solution of (1) under Assumptions 1 and 2.

Recently, Kankam et al. [9] proposed a new line search technique as follows (Algorithm 4).

Algorithm 4: Line Search 2$(x,\sigma ,\theta ,\delta )$.

1: Input Given $x\in domg,$ $\sigma >0$, $\theta \in (0,1),$ and $\delta >0$. Set $$\begin{array}{c}L(x,\alpha )=pro{x}_{\alpha g}(x-\alpha ▿f\left(x\right)), \mathrm{and}\\ S(x,\alpha )=pro{x}_{\alpha g}(L(x,\alpha )-\alpha ▿f\left(L(x,\alpha )\right)).\end{array}$$ 2: Set $\alpha =\sigma .$ 3: while $$\begin{array}{ll}\alpha max\{\parallel ▿f(S(x,\alpha ))-▿f(L(x,& \alpha \left)\right)\parallel ,\parallel ▿f\left(L\right(x,\alpha \left)\right)-▿f\left(x\right)\parallel \}\\ & >\delta (\parallel S(x,\alpha )-L(x,\alpha )\parallel +\parallel L(x,\alpha )-x\parallel )\end{array}$$ do 4:     Set $\alpha =\theta \alpha ,L(x,\alpha )=L(x,\theta \alpha ),S(x,\alpha )=S(x,\theta \alpha )$ 5: end while 6: Output$\alpha .$

They proved that Line Search 2 stops at finitely many steps and introduced the following algorithm (Algorithm 5).

Algorithm 5: Line Search 2 stops at finitely many steps.

1: Input Given $x_0\in domg,$$\sigma >0$, $\theta \in (0,1),$ and $\delta \in (1,\frac{1}{8}),$ for $n\in \mathbb{N},$ $$\begin{array}{l}{y}_n=pro{x}_{{\gamma }_{n}g}(x_n-{\gamma }_n▿f\left(x_n\right)),\\ x_{n+1}=pro{x}_{{\gamma }_{n}g}(y_n-{\gamma }_n▿f\left(y_n\right)),\end{array}$$ where ${\gamma }_n:=\mathbf{Line}\mathbf{Search}\mathbf{2}(x_n,\sigma ,\theta ,\delta ).$

They established a weak convergence theorem for Algorithm 5, under Assumptions 1 and 2. As we can see, Algorithms 3 and 5 do not utilize an inertial step.

Inspired by Algorithm 1 (FISTA), Cruz and Nghia [20] also proposed the following accelerated algorithm (Algorithm 6).

Algorithm 6: Accelerated algorithm.

1: Input Given $x_0,x_1\in domg,{\alpha }_{-1}>0$, $\theta \in (0,1),t_0=1,$ and $\delta \in (0,\frac{1}{2}),$ for $n\in \mathbb{N},$ $$\begin{array}{l}{t}_{n+1}=\frac{1+\sqrt{1+4t_n^2}}{2},\\ y_n=x_n+\left(\frac{t_n-1}{t_{n+1}}\right)(x_n-x_{n-1}),\\ {\tilde{y}}_n=P_{domg}\left(y_n\right),\\ x_{n+1}=pro{x}_{{\alpha }_{n}g}({\tilde{y}}_n-{\alpha }_n▿f\left({\tilde{y}}_n\right)),\end{array}$$ where ${\alpha }_n:=$Line Search 1$({\tilde{y}}_n,{\alpha }_{n-1},\theta ,\delta ),$ and P is a metric projection.

They showed that Algorithm 6 has better complexity than Algorithm 3. However, the convergence to a solution of (1) is not guaranteed under the inertial parameter ${\beta }_n=\frac{t_n-1}{t_{n+1}}$.

In 2019, Attouch and Cabot [22] analyzed the convergence rate of a method called the relaxed inertial algorithm (RIPA) for solving monotone inclusion problems, which is closely related to convex minimization problems. This method utilizes an inertial step $x_n+{\beta }_n(x_n-x_{n-1})$ to improve its performance. It was defined as follows (Algorithm 7).   

Algorithm 7: RIPA.

1: Input Given $x_0,x_1\in H,{\beta }_n\ge 0,{\rho }_n\in (0,1],{\mu }_n>0,$ for $n\in \mathbb{N},$ $$\begin{array}{c}{y}_n=x_n+{\beta }_n(x_n-x_{n-1}),\\ x_{n+1}=(1-{\rho }_n)y_n+{\rho }_n{J}_{{\mu }_n}^A\left(y_n\right),\end{array}$$ where A is a maximal monotone operator and $J_{{\mu }_n}^A\left(y_n\right)={(I+{\mu }_{n}A)}^{-1}$.

Under mild restrictions of the control parameters, they showed that Algorithm 7 (RIPA) gives a fast convergence rate.

Next, we recall some important tools that will be used in the later sections.

Definition 1.

For $x\in H,$ a subdifferential of h at x is defined as follows:

$$\partial h\left(x\right):=\{u\in H:\langle u,y-x\rangle +h(x)\le h(y),y\in H\}.$$

The following can be found in [26].

Lemma 1 ([26]).

A subdifferential $\partial h$ is maximal monotone. Moreover, a graph of $\partial h,$$Gph(\partial h):=\left\{\right(u,v)\in H\times H:v\in \partial h(u\left)\right\}$ is demiclosed, i.e., for any sequence $(u_n,v_n)\in Gph(\partial h)$ such that $\left\{u_n\right\}$ converges weakly to u and $\left\{v_n\right\}$ converges strongly to $v,$ we have $(u,v)\in Gph(\partial h)$.

The proximal operator, $pro{x}_g:H\to domg$ with $pro{x}_g\left(x\right)={(I+\partial g)}^{-1}\left(x\right)$, is single-valued with the full domain, and the following holds:

$$\frac{x-pro{x}_{\alpha g}x}{\alpha }\in \partial g\left(pro{x}_{\alpha g}x\right), \mathrm{for} \mathrm{all} x\in H \mathrm{and} \alpha >0.$$

(4)

The following lemmas are crucial for the main results.

Lemma 2 ([27]).

Let H be a real Hilbert space. Then, the following hold, for all $x,y\in H$ and $\alpha \in [0,1]$:

(i) 

${\parallel x\pm y\parallel }^2={\parallel x\parallel }^2\pm 2\langle x,y\rangle +{\parallel y\parallel }^2;$

(ii) 

${\parallel \alpha x+(1-\alpha )y\parallel }^2={\alpha \parallel x\parallel }^2+{(1-\alpha )\parallel y\parallel }^2-\alpha (1-\alpha ){\parallel x-y\parallel }^2;$

(iii) 

${\parallel x+y\parallel }^2\le {\parallel x\parallel }^2+2\langle y,x+y\rangle .$

Lemma 3 ([16]).

Let $\left\{a_n\right\}$ and $\left\{{\beta }_n\right\}$ be sequences of non-negative real numbers such that:

$$a_{n+1}\le (1+{\beta }_n)a_n+{\beta }_n{a}_{n-1}, for all n\in \mathbb{N}.$$

Then, the following holds:

$${\displaystyle a_{n+1}\le K\prod _{j=1}^n(1+2{\beta }_j), where K=max\{a_1,a_2\}.}$$

Moreover, if ${\displaystyle \sum _{n=1}^{+\infty }{\beta }_n<+\infty ,}$ then $\left\{a_n\right\}$ is bounded.

Lemma 4 ([27]).

Let $\left\{a_n\right\},\left\{b_n\right\}$ and $\left\{{\delta }_n\right\}$ be sequences of non-negative real numbers such that:

$$a_{n+1}\le (1+{\delta }_n)a_n+b_n, for all n\in \mathbb{N}.$$

If ${\displaystyle \sum _{n=1}^{+\infty }{\delta }_n<+\infty }$ and ${\displaystyle \sum _{n=1}^{+\infty }{b}_n<+\infty ,}$ then ${\displaystyle \underset{n\to +\infty }{lim}{a}_n}$ exists.

Lemma 5 ([28]).

Let H be a Hilbert space and $\left\{x_n\right\}$ a sequence in H such that there exists a nonempty subset Ω of H satisfying the following:

(i) 

for any ${\displaystyle x^{*}\in \Omega ,\underset{n\to }{lim}\parallel x_n-x^{*}\parallel }$ exists;

(ii) 

every weak-cluster point of $\left\{x_n\right\}$ belongs to $\Omega .$

Then, $\left\{x_n\right\}$ converges weakly to an element in Ω.

3. Main Results

In this section, we assume the existence of a solution of (1) and denote $S_{*}$ the set of all such solutions. We modify Line Search 2 as follows (Algorithm 8).

Algorithm 8: Line Search 3$(x,\sigma ,\theta ,\delta )$.

1: Input Given $x\in domg,$$\sigma >0$, $\theta \in (0,1),$ and $\delta >0$. Set $$\begin{array}{c}L(x,\alpha )=pro{x}_{\alpha g}(x-\alpha ▿f\left(x\right)), \mathrm{and}\\ S(x,\alpha )=pro{x}_{\alpha g}(L(x,\alpha )-\alpha ▿f\left(L(x,\alpha )\right)).\end{array}$$ 2: Set $\alpha =\sigma .$ 3: while $$\begin{array}{ll}\frac{\alpha }{2}(\parallel ▿f\left(S(x,\alpha )\right)-▿f\left(L(x,\alpha )\right)\parallel & +\parallel ▿f\left(L\right(x,\alpha \left)\right)-▿f\left(x\right)\parallel )\hfill \\ & >\delta (\parallel S(x,\alpha )-L(x,\alpha )\parallel +\parallel L(x,\alpha )-x\parallel ),\end{array}$$ do 4:     Set $\alpha =\theta \alpha ,L(x,\alpha )=L(x,\theta \alpha ),S(x,\alpha )=S(x,\theta \alpha )$ 5: end while 6: Output$\alpha .$

We know that Line Search 3 terminates at a lower step than Line Search 2, since:

$$\begin{array}{cc}\hfill \frac{\alpha }{2}(\parallel ▿f\left(S\right(x,& \alpha \left)\right)-▿f\left(L\right(x,\alpha \left)\right)\parallel +\parallel ▿f\left(L\right(x,\alpha \left)\right)-▿f\left(x\right)\parallel )\hfill \\ \hfill & \le \alpha max\{\parallel ▿f(S(x,\alpha ))-▿f(L(x,\alpha ))\parallel ,\parallel ▿f(L(x,\alpha ))-▿f(x)\parallel \}.\hfill \end{array}$$

Therefore, Line Search 3 stops after finitely many steps. We introduce an accelerated algorithm by utilizing Line Search 3 as follows (Algorithm 9).

Algorithm 9.

1: Input Given $x_0,x_1\in dom g,{\beta }_n\ge 0,\sigma >0,\theta \in (0,1) \mathrm{and} \delta \in (0,\frac{1}{8}), \mathrm{for} n\in \mathbb{N},$ $$\begin{array}{l}{\hat{x}}_n = x_n+{\beta }_n(x_n-x_{n-1}),\\ y_n = P_{dom g}{\hat{x}}_n,\\ z_n = {prox}_{{\gamma }_{n}g}(y_n-{\gamma }_n▿f(y_n)),\\ x_{n+1} = {prox}_{{\gamma }_{n}g}(z_n-{\gamma }_n▿f(z_n)),\end{array}$$ for $y_n:=$ Line Search 3($y_n,\sigma ,\theta ,\delta$), and $P_{dom g}$ is a metric projection map onto $dom g$.

Next, we establish our first theorem.

Theorem 1.

Let H be a Hilbert space, $f:H\to \mathbb{R}\cup \{+\infty \}$ and $g:H\to \mathbb{R}\cup \{+\infty \}$ proper lower semicontinuous convex functions satisfying A1 and A2. Suppose that $domg$ is closed and the following also hold, for all $n\in \mathbb{N}$:

C1. 

${\gamma }_n\ge \gamma >0;$

C2. 

${\displaystyle {\beta }_n\ge 0 and \sum _{n=1}^{+\infty }{\beta }_n<+\infty .}$

Then, a sequence $\left\{x_n\right\}$ generated by Algorithm 9 converges weakly to a point in $S_{*}$.

Proof. 

For any $x\in domg$ and $n\in \mathbb{N},$ we claim that:

$$\begin{array}{cc}\hfill \parallel y_n{-x\parallel }^2-{\parallel x_{n+1}-x\parallel }^2\ge & 2{\gamma }_n[(f+g)\left(z_n\right)+(f+g)\left(x_{n+1}\right)-2(f+g)\left(x\right)]\hfill \\ \hfill & +(1-8\delta )(\parallel z_n-y_n{\parallel }^2+\parallel x_{n+1}-z_n{\parallel }^2).\hfill \end{array}$$

(5)

To prove our claim, we know, from (4) and the definition of $\partial g,$ that:

$$\frac{y_n-z_n}{{\gamma }_n}-▿f\left(y_n\right)\in \partial g\left(z_n\right), \mathrm{and}\frac{z_n-x_{n+1}}{{\gamma }_n}-▿f\left(z_n\right)\in \partial g\left(x_{n+1}\right).$$

Then,

$$g\left(x\right)-g\left(z_n\right)\ge \langle \frac{y_n-z_n}{{\gamma }_n}-▿f\left(y_n\right),x-z_n\rangle , \mathrm{and}$$

$$g\left(x\right)-g\left(x_{n+1}\right)\ge \langle \frac{z_n-x_{n+1}}{{\gamma }_n}-▿f\left(z_n\right),x-x_{n+1}\rangle , \mathrm{for} \mathrm{all} n\in \mathbb{N}.$$

Moreover,

$$f\left(x\right)-f\left(y_n\right)\ge \langle ▿f\left(y_n\right),x-y_n\rangle ,$$

$$f\left(x\right)-f\left(z_n\right)\ge \langle ▿f\left(z_n\right),x-z_n\rangle ,$$

$$f\left(y_n\right)-f\left(z_n\right)\ge \langle ▿f\left(z_n\right),y_n-z_n\rangle , \mathrm{and}$$

$$f\left(z_n\right)-f\left(x_{n+1}\right)\ge \langle ▿f\left(x_{n+1}\right),z_n-x_{n+1}\rangle , \mathrm{for} \mathrm{all} n\in \mathbb{N}.$$

From the definition of ${\gamma }_n$ and the above inequalities, we have, for all $x\in domg$ and $n\in \mathbb{N}$,

$$\begin{array}{cc}\hfill f\left(x\right)& -f\left(y_n\right)+f\left(x\right)-f\left(z_n\right)+g\left(x\right)-g\left(z_n\right)+g\left(x\right)-g\left(x_{n+1}\right)\hfill \\ \hfill & \ge \frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\langle ▿f\left(y_n\right),z_n-y_n\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle \hfill \\ \hfill & +\langle ▿f\left(z_n\right),x_{n+1}-z_n\rangle \hfill \\ \hfill & =\frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\langle ▿f\left(y_n\right)-▿f\left(z_n\right),z_n-y_n\rangle +\langle ▿f\left(z_n\right),z_n-y_n\rangle \hfill \\ \hfill & +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle +\langle ▿f\left(z_n\right)-▿f\left(x_{n+1}\right),x_{n+1}-z_n\rangle \hfill \\ \hfill & +\langle ▿f\left(x_{n+1}\right),x_{n+1}-z_n\rangle \hfill \\ \hfill & \ge \frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle \hfill \\ \hfill & -\parallel ▿f\left(z_n\right)-▿f\left(y_n\right)\parallel \parallel z_n-y_n\parallel +\langle ▿f\left(z_n\right),z_n-y_n\rangle \hfill \\ \hfill & -\parallel ▿f\left(x_{n+1}\right)-▿f\left(z_n\right)\parallel \parallel x_{n+1}-z_n\parallel +\langle ▿f\left(x_{n+1}\right),x_{n+1}-z_n\rangle \hfill \\ \hfill & \ge \frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle +\langle ▿f\left(z_n\right),z_n-y_n\rangle \hfill \\ \hfill & -\parallel ▿f\left(z_n\right)-▿f\left(y_n\right)\parallel (\parallel z_n-y_n\parallel +\parallel x_{n+1}-z_n\parallel )\hfill \\ \hfill & -\parallel ▿f\left(x_{n+1}\right)-▿f\left(z_n\right)\parallel (\parallel z_n-y_n\parallel +\parallel x_{n+1}-z_n\parallel )+\langle ▿f\left(x_{n+1}\right),x_{n+1}-z_n\rangle \hfill \\ \hfill & =\frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle +\langle ▿f\left(z_n\right),z_n-y_n\rangle \hfill \\ \hfill & +\langle ▿f\left(x_{n+1}\right),x_{n+1}-z_n\rangle \hfill \\ \hfill & -(\parallel ▿f\left(z_n\right)-▿f\left(y_n\right)\parallel +\parallel ▿f\left(x_{n+1}\right)-▿f\left(z_n\right)\parallel \left)\right(\parallel z_n-y_n\parallel +\parallel x_{n+1}-z_n\parallel )\hfill \\ \hfill & \ge \frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle +\langle ▿f\left(z_n\right),z_n-y_n\rangle \hfill \\ \hfill & +\langle ▿f\left(x_{n+1}\right),x_{n+1}-z_n\rangle -\frac{2\delta }{{\gamma }_n}(\parallel z_n-y_n\parallel +\parallel x_{n+1}-z_n{\parallel )}^2\hfill \\ \hfill & \ge \frac{1}{{\gamma }_n}\langle y_n-z_n,x-z_n\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x-x_{n+1}\rangle +f\left(x_{n+1}\right)-f\left(y_n\right)\hfill \\ \hfill & -\frac{4\delta }{{\gamma }_n}(\parallel z_n-y_n{\parallel }^2+\parallel x_{n+1}-z_n{\parallel }^2).\hfill \end{array}$$

Hence,

$$\begin{array}{cc}\hfill \frac{1}{{\gamma }_n}& \langle y_n-z_n,z_n-x\rangle +\frac{1}{{\gamma }_n}\langle z_n-x_{n+1},x_{n+1}-x\rangle \hfill \\ \hfill & \ge (f+g)\left(z_n\right)+(f+g)\left(x_{n+1}\right)-2(f+g)\left(x\right)-\frac{4\delta }{{\gamma }_n}\parallel z_n-y_n{\parallel }^2-\frac{4\delta }{{\gamma }_n}{\parallel x_{n+1}-z_n\parallel }^2.\hfill \end{array}$$

Moreover, the following also hold, for all $n\in \mathbb{N},$

$$\langle y_n-z_n,z_n-x\rangle =\frac{1}{2}(\parallel y_n{-x\parallel }^2-\parallel y_n-z_n{\parallel }^2-\parallel z_n-x{\parallel }^2),and,$$

$$\langle z_n-x_{n+1},x_{n+1}-x\rangle =\frac{1}{2}(\parallel z_n{-x\parallel }^2-\parallel z_n-x_{n+1}{\parallel }^2-\parallel x_{n+1}-x{\parallel }^2).$$

As a result, we obtain:

$$\begin{array}{cc}\hfill \frac{1}{2{\gamma }_n}& (\parallel y_n{-x\parallel }^2-\parallel y_n-z_n{\parallel }^2)-\frac{1}{2{\gamma }_n}(\parallel z_n-x_{n+1}{\parallel }^2+\parallel x_{n+1}-x{\parallel }^2)\hfill \\ \hfill & \ge (f+g)\left(z_n\right)+(f+g)\left(x_{n+1}\right)-2(f+g)\left(x\right)-\frac{4\delta }{{\gamma }_n}\parallel z_n-y_n{\parallel }^2-\frac{4\delta }{{\gamma }_n}{\parallel x_{n+1}-z_n\parallel }^2,\hfill \end{array}$$

for all $x\in domg$ and $n\in \mathbb{N}$. Therefore,

$$\begin{array}{cc}\hfill \parallel y_n{-x\parallel }^2-{\parallel x_{n+1}-x\parallel }^2\ge & 2{\gamma }_n[(f+g)\left(z_n\right)+(f+g)\left(x_{n+1}\right)-2(f+g)\left(x\right)]\hfill \\ \hfill & +(1-8\delta )(\parallel z_n-y_n{\parallel }^2+\parallel x_{n+1}-z_n{\parallel }^2),\hfill \end{array}$$

for all $x\in domg$ and $n\in \mathbb{N}$. Furthermore, putting $x=x^{*}\in S_{*}$, we obtain:

$$\begin{array}{c}\hfill \parallel y_n-x^{*}{\parallel }^2-\parallel x_{n+1}-x^{*}{\parallel }^2\ge (1-8\delta )(\parallel z_n-y_n{\parallel }^2+\parallel x_{n+1}-z_n{\parallel }^2).\end{array}$$

(6)

Next, we show that ${\displaystyle \underset{n\to \infty }{lim}\parallel x_n-x^{*}\parallel }$ exists. From (6), we have:

$$\begin{array}{cc}\hfill \parallel x_{n+1}-x^{*}\parallel & \le \parallel y_n-x^{*}\parallel ,\hfill \\ \hfill & =\parallel P_{domg}{\widehat{x}}_n-P_{domg}{x}^{*}\parallel ,\hfill \\ \hfill & \le \parallel {\widehat{x}}_n-x^{*}\parallel ,\hfill \\ \hfill & \le \parallel x_n-x^{*}\parallel +{\beta }_n\parallel x_n-x_{n-1}\parallel ,\hfill \\ \hfill & \le (1+{\beta }_n)\parallel x_n-x^{*}\parallel +{\beta }_n\parallel x_{n-1}-x^{*}\parallel ,\mathrm{for} \mathrm{all} n\in \mathbb{N}.\hfill \end{array}$$

(7)

This implies by Lemma 3 and C2 that $\left\{x_n\right\}$ is bounded.

Consequently, ${\displaystyle \sum _{n=1}^{+\infty }{\beta }_n\parallel x_n-x_{n-1}\parallel <+\infty ,}$ and:

$$\parallel {\widehat{x}}_n-x_n\parallel ={\beta }_n\parallel x_n-x_{n-1}\parallel \to 0,asn\to +\infty .$$

By (7) and Lemma 4, we obtain that ${\displaystyle \underset{n\to +\infty }{lim}\parallel x_n-x^{*}\parallel }$ exists. By the definitions of $z_{n-1}$ and $x_n,$ we see $x_n\in domg,$ for all $n\in \mathbb{N}$. As a result, we have:

$$\parallel {\widehat{x}}_n-y_n\parallel \le \parallel {\widehat{x}}_n-x_n\parallel ,\mathrm{for} \mathrm{all} n\in \mathbb{N},$$

so, ${\displaystyle \underset{n\to +\infty }{lim}\parallel {\widehat{x}}_n-y_n\parallel =0.}$

Thus, we obtain from (7) that ${\displaystyle \underset{n\to +\infty }{lim}\parallel x_n-x^{*}\parallel =\underset{n\to +\infty }{lim}\parallel y_n-x^{*}\parallel .}$ Moreover, it follows from (6) that ${\displaystyle \underset{n\to +\infty }{lim}\parallel z_n-y_n\parallel =0,}$ and hence, ${\displaystyle \underset{n\to +\infty }{lim}\parallel z_n-x_n\parallel =0.}$

Now, we prove that every weak-cluster point of $\left\{x_n\right\}$ belongs to $S_{*}.$ In order to accomplish this, we first let w be a weak-cluster point of $\left\{x_n\right\}.$ Therefore, there exists a subsequence $\left\{x_{n_k}\right\}$ of $\left\{x_n\right\}$ such that $x_{n_k}\rightharpoonup w,$ and hence, $z_{n_k}\rightharpoonup w$. Next, we prove that w belongs to $S^{*}.$ Since $▿f$ is uniformly continuous, we have ${\displaystyle \underset{k\to +\infty }{lim}\parallel ▿f{z}_{n_k}-▿f{y}_{n_k}\parallel =0.}$ From (4), we obtain:

$$\frac{y_{n_k}-{\gamma }_{n_k}▿f{y}_{n_k}-z_{n_k}}{{\gamma }_{n_k}}\in \partial g\left(z_{n_k}\right),\mathrm{for} \mathrm{all} k\in \mathbb{N}.$$

Hence,

$$\frac{y_{n_k}-z_{n_k}}{{\gamma }_{n_k}}-▿f{y}_{n_k}+▿f{z}_{n_k}\in \partial g\left(z_{n_k}\right)+▿f{z}_{n_k}=\partial (f+g)\left(z_{n_k}\right), \mathrm{for} \mathrm{all} k\in \mathbb{N}.$$

By letting $k\to +\infty$ in the above inequality, the demiclosedness of $Gph(\partial (f+g\left)\right)$ implies that $0\in \partial (f+g)\left(w\right),$ and hence, $w\in S_{*}.$ Therefore, every weak-cluster point of $\left\{x_n\right\}$ belongs to $S_{*}.$

We derive from Lemma 5 that $\left\{x_n\right\}$ converges weakly to $w^{*}$ in $S_{*}$. Therefore, $\left\{x_n\right\}$ converges weakly to a solution of (1), and the proof is complete. □

In the next theorem, we provide the complexity theorem of Algorithm 9. First, we introduce the control sequence $\left\{t_n\right\}$ defined in [22] by:

$${\displaystyle t_n=1+\sum _{k=n}^{+\infty }\left(\prod _{i=n}^k{\beta }_i\right), \mathrm{for} \mathrm{all} n\in \mathbb{N}.}$$

(8)

This sequence is well defined if the following assumption holds:

$${\displaystyle \sum _{k=n}^{+\infty }\left(\prod _{i=n}^k{\beta }_i\right)<+\infty , \mathrm{for} \mathrm{all} n\in \mathbb{N}.}$$

Hence, from (8), we can see that:

$${\beta }_n{t}_{n+1}=t_n-1, \mathrm{for} \mathrm{all} n\in \mathbb{N}.$$

(9)

Next, we establish the following theorem.

Theorem 2.

Given $x_0=x_1\in domg$ and letting $\left\{x_n\right\}$ be a sequence generated by Algorithm 9, assume that all assumptions in Theorem 1 are satisfied. Furthermore, suppose that the following conditions hold, for all $n\in \mathbb{N}$:

D1. 

${\displaystyle \sum _{k=n}^{+\infty }\left(\prod _{i=n}^k{\beta }_i\right)<+\infty ,}$ and $2t_{n+1}^2-2t_{n+1}\le t_n^2,$

D2. 

${\gamma }_n\ge {\gamma }_{n+1}.$

Then,

$${\displaystyle (f+g)\left(x_{n+1}\right)-\underset{x\in H}{min}(f+g)\left(x\right)\le \frac{{\displaystyle d{(x_1,S_{*})}^2+2{\gamma }_1{t}_1^2[(f+g)\left(x_1\right)-\underset{x\in H}{min}(f+g)\left(x\right)]}}{2\gamma t_{n+1}^2},}$$

for all $n\in \mathbb{N}.$ In other words,

$${\displaystyle (f+g)\left(x_{n+1}\right)-\underset{x\in H}{min}(f+g)\left(x\right)=\mathcal{O}\left(\frac{1}{t_{n+1}^2}\right),foralln\in \mathbb{N}.}$$

Proof. 

For any $x\in domg,$ we know that:

$$\parallel y_n{-x\parallel }^2-{\parallel x_{n+1}-x\parallel }^2\ge 2{\gamma }_n[(f+g)\left(z_n\right)+(f+g)\left(x_{n+1}\right)-2(f+g)\left(x\right)]$$

for all $n\in \mathbb{N}.$ Since $x\in domg,$ we obtain $\parallel {\widehat{x}}_n-x\parallel \ge \parallel y_n-x\parallel .$ Thus, we conclude that:

$$\parallel {\widehat{x}}_n{-x\parallel }^2-{\parallel x_{n+1}-x\parallel }^2\ge 2{\gamma }_n[(f+g)\left(z_n\right)+(f+g)\left(x_{n+1}\right)-2(f+g)\left(x\right)],\mathrm{for} \mathrm{all} n\in \mathbb{N}.$$

(10)

Let $x^{*}$ be an element in $S_{*}$. We know that $x_n,x\in domg$ and $t_{n+1}\ge 1,$ so $(1-\frac{1}{t_{n+1}})x_n+\frac{1}{t_{n+1}}{x}^{*}\in domg.$ Next, we put $x=(1-\frac{1}{t_{n+1}})x_n+\frac{1}{t_{n+1}}{x}^{*}$ in (10) and obtain the following:

$$\begin{array}{cc}\hfill \parallel x_{n+1}& -(1-\frac{1}{t_{n+1}})x_n-\frac{1}{t_{n+1}}{x}^{*}{\parallel }^2-{\parallel {\widehat{x}}_n-(1-\frac{1}{t_{n+1}})x_n-\frac{1}{t_{n+1}}{x}^{*}\parallel }^2\hfill \\ \hfill & \le 2{\gamma }_n[(f+g)((1-\frac{1}{t_{n+1}})x_n+\frac{1}{t_{n+1}}{x}^{*})-(f+g)\left(x_{n+1}\right)]\hfill \\ \hfill & +2{\gamma }_n[(f+g)((1-\frac{1}{t_{n+1}})x_n+\frac{1}{t_{n+1}}{x}^{*})-(f+g)\left(z_n\right)]\hfill \\ \hfill & \le 2{\gamma }_n[(1-\frac{1}{t_{n+1}})(f+g)\left(x_n\right)+\frac{1}{t_{n+1}}(f+g)\left(x^{*}\right)-(f+g)\left(x_{n+1}\right)]\hfill \\ \hfill & +2{\gamma }_n[(1-\frac{1}{t_{n+1}})(f+g)\left(x_n\right)+\frac{1}{t_{n+1}}(f+g)\left(x^{*}\right)-(f+g)\left(z_n\right)]\hfill \\ \hfill & =2{\gamma }_n[(1-\frac{1}{t_{n+1}})[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]-[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)]]\hfill \\ \hfill & +2{\gamma }_n[(1-\frac{1}{t_{n+1}})[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]-[(f+g)\left(z_n\right)-(f+g)\left(x^{*}\right)]]\hfill \\ \hfill & \le 4{\gamma }_n(1-\frac{1}{t_{n+1}})[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]-2{\gamma }_n[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)],\hfill \end{array}$$

(11)

for all $n\in \mathbb{N}$. From D1 and D2, we know that $t_n^2\ge 2t_{n+1}^2-2t_{n+1}$ and ${\gamma }_n\ge {\gamma }_{n+1},$ so:

$$\begin{array}{cc}\hfill 4{\gamma }_n(1-\frac{1}{t_{n+1}})& [(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]-2{\gamma }_n[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)]\hfill \\ \hfill & \le \frac{1}{t_{n+1}^2}[2{\gamma }_n(2t_{n+1}^2-2t_{n+1})[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]\hfill \\ \hfill & -2{\gamma }_{n+1}{t}_{n+1}^2[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)]]\hfill \\ \hfill & \le \frac{1}{t_{n+1}^2}[2{\gamma }_n{t}_n^2[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]\hfill \\ \hfill & -2{\gamma }_{n+1}{t}_{n+1}^2[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)]],foralln\in \mathbb{N}.\hfill \end{array}$$

(12)

Moreover, we also have:

$$\begin{array}{cc}\hfill \parallel & x_{n+1}-(1-\frac{1}{t_{n+1}})x_n-\frac{1}{t_{n+1}}{x}^{*}{\parallel }^2-{\parallel {\widehat{x}}_n-(1-\frac{1}{t_{n+1}})x_n-\frac{1}{t_{n+1}}{x}^{*}\parallel }^2\hfill \\ \hfill & =\frac{1}{t_{n+1}^2}(\parallel t_{n+1}{x}_{n+1}-(t_{n+1}-1)x_n-x^{*}{\parallel }^2\hfill \\ \hfill & -\parallel t_{n+1}{x}_n+{\beta }_n{t}_{n+1}(x_n-x_{n-1})-(t_{n+1}-1)x_n-x^{*}{\parallel }^2)\hfill \\ \hfill & =\frac{1}{t_{n+1}^2}(\parallel t_{n+1}{x}_{n+1}-(t_{n+1}-1)x_n-x^{*}{\parallel }^2-\parallel (t_n-1)(x_n-x_{n-1})+x_n-x^{*}{\parallel }^2)\hfill \\ \hfill & =\frac{1}{t_{n+1}^2}(\parallel t_{n+1}{x}_{n+1}-(t_{n+1}-1)x_n-x^{*}{\parallel }^2-\parallel t_n{x}_n-(t_n-1)x_{n-1}-x^{*}{\parallel }^2),\hfill \end{array}$$

(13)

for all $n\in \mathbb{N}$. By (11)–(13), we obtain:

$$\begin{array}{cc}\hfill \parallel t_{n+1}& x_{n+1}-(t_{n+1}-1)x_n-x^{*}{\parallel }^2-{\parallel t_n{x}_n-(t_n-1)x_{n-1}-x^{*}\parallel }^2\hfill \\ \hfill & \le 2{\gamma }_n{t}_n^2[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]-2{\gamma }_{n+1}{t}_{n+1}^2[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)],\hfill \end{array}$$

for all $n\in \mathbb{N}$. It follows that:

$$\begin{array}{cc}\hfill 2{\gamma }_{n+1}{t}_{n+1}^2[(f+g)\left(x_{n+1}\right)-(f+g)\left(x\right)]& \le \parallel t_n{x}_n-(t_n-1)x_{n-1}-x^{*}{\parallel }^2\hfill \\ \hfill & -\parallel t_{n+1}{x}_{n+1}-(t_{n+1}-1)x_n-x^{*}{\parallel }^2\hfill \\ \hfill & +2{\gamma }_n{t}_n^2[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)],\hfill \end{array}$$

(14)

for all $n\in \mathbb{N}$. Moreover, we can inductively prove, from (14), that:

$$\begin{array}{cc}\hfill 2{\gamma }_{n+1}{t}_{n+1}^2[(f+g)\left(x_{n+1}\right)-(f+g)\left(x^{*}\right)]& \le \parallel t_n{x}_n-(t_n-1)x_{n-1}-x^{*}{\parallel }^2\hfill \\ \hfill & +2{\gamma }_n{t}_n^2[(f+g)\left(x_n\right)-(f+g)\left(x^{*}\right)]\hfill \\ \hfill & \le \parallel t_{n-1}{x}_{n-1}-(t_{n-1}-1)x_{n-2}-x^{*}{\parallel }^2\hfill \\ \hfill & +2{\gamma }_{n-1}{t}_{n-1}^2[(f+g)\left(x_{n-1}\right)-(f+g)\left(x^{*}\right)]\hfill \\ \hfill & ⋮\hfill \\ \hfill & \le \parallel t_1{x}_1-(t_1-1)x_0-x^{*}{\parallel }^2\hfill \\ \hfill & +2{\gamma }_1{t}_1^2[(f+g)\left(x_1\right)-(f+g)\left(x^{*}\right)],\hfill \end{array}$$

for all $n\in \mathbb{N}.$ From C1, we know that ${\gamma }_{n+1}\ge \gamma .$ Therefore, we obtain, for all $n\in \mathbb{N},$

$$\begin{array}{cc}\hfill {\displaystyle (f+g)\left(x_{n+1}\right)-\underset{x\in H}{min}(f+g)\left(x\right)}& \le \frac{1}{2{\gamma }_{n+1}{t}_{n+1}^2}{\parallel x_1-x^{*}\parallel }^2\hfill \\ \hfill & +\frac{2{\gamma }_1{t}_1^2}{2{\gamma }_{n+1}{t}_{n+1}^2}[(f+g)\left(x_1\right)-(f+g)\left(x^{*}\right)]\hfill \\ \hfill & \le \frac{{\displaystyle \parallel x_1-x^{*}{\parallel }^2+2{\gamma }_1{t}_1^2[(f+g)\left(x_1\right)-\underset{x\in H}{min}(f+g)\left(x\right)]}}{2\gamma t_{n+1}^2}.\hfill \end{array}$$

Since $x^{*}$ is arbitrarily chosen from $S_{*}$, we have:

$${\displaystyle (f+g)\left(x_{n+1}\right)-\underset{x\in H}{min}(f+g)\left(x\right)\le \frac{{\displaystyle d{(x_1,S_{*})}^2+2{\gamma }_1{t}_1^2[(f+g)\left(x_1\right)-\underset{x\in H}{min}(f+g)\left(x\right)]}}{2\gamma t_{n+1}^2},}$$

for all $n\in \mathbb{N},$ and the proof is complete. □

Remark 1.

To justify that there exists a sequence $\left\{{\beta }_n\right\}$ satisfying D1, we choose:

$${\beta }_n=\frac{1}{{(n+1)}^2},\mathrm{for} \mathrm{all} n\in \mathbb{N}.$$

We see that, for all $n\in \mathbb{N},$

$$t_n=1+\frac{1}{{(n+1)}^2}+\frac{1}{{(n+1)}^2{(n+2)}^2}+\frac{1}{{(n+1)}^2{(n+2)}^2{(n+3)}^2}+ \dots .$$

Therefore, we have:

$$t_{n+1}\le t_n\mathrm{for} \mathrm{all} n\in \mathbb{N}.$$

Furthermore, it can be seen that ${\displaystyle t_{n+1}<1+\sum _{k=3}^{+\infty }\frac{1}{k^2}=\frac{{\pi }^2}{6}-\frac{1}{4}}$, for all $n\in \mathbb{N}.$ Then:

$$\begin{array}{cc}\hfill 2t_{n+1}^2-2t_{n+1}& =\left(t_{n+1}\right)(2t_{n+1}-2)\hfill \\ \hfill & \le \left(t_n\right)(\frac{{\pi }^2}{3}-\frac{5}{2}),foralln\in \mathbb{N}.\hfill \end{array}$$

Since $\frac{{\pi }^2}{3}-\frac{5}{2}<1,$ we can conclude that

$$2t_{n+1}^2-2t_{n+1}<t_n<t_n^2,$$

for all $n\in \mathbb{N}.$ Obviously, ${\displaystyle \sum _{k=n}^{+\infty }\left(\prod _{i=n}^k{\beta }_i\right)<+\infty ;}$, hence D1 is satisfied.

We also note that to obtain a sequence $\left\{{\gamma }_n\right\}$ satisfying D2, one could simply modify Algorithm 9 by choosing ${\gamma }_n=\mathbf{Line}\mathbf{Search}\mathbf{3}(y_n,{\gamma }_{n-1},\theta ,\delta ).$

4. Applications to Data Classification and Image Restoration Problems

In this section, the proposed algorithm is used to solve classification and image restoration problems. The performance of Algorithm 9 is evaluated and compared with Algorithms 3, 5, and 6.

4.1. Data Classification

Data classification is a major branch of problems in machine learning, which is an application of artificial intelligence (AI) possessing the ability to learn and improve from experience without being programmed. In this work, we focused on one particular learning technique called extreme learning machine (ELM) introduced by Huang et al. [29]. It is defined as follows.

Let $S:=\{(x_k,t_k):x_k\in {\mathbb{R}}^n,t_k\in {\mathbb{R}}^m,k=1,2,...,N\}$ be a training set of N samples, where $x_k$ is an input and $t_k$ is a target. The output of ELM with M hidden nodes and activation function G is defined by:

$${\displaystyle o_j=\sum _{i=1}^M{\eta }_{i}G(\langle w_i,x_j\rangle +b_i),}$$

where $w_i$ is the weight vector connecting the i-th hidden node and the input node, ${\eta }_i$ is the weight vector connecting the i-th hidden node and the output node, and $b_i$ is the bias. The hidden layer output matrix $\mathbf{H}$ is formulated as:

$$\mathbf{H}=\left[\begin{array}{ccc}G(\langle w_1,x_1\rangle +b_1)& \cdots & G(\langle w_M,x_1\rangle +b_M)\\ ⋮& \ddots & ⋮\\ G(\langle w_1,x_N\rangle +b_1)& \cdots & G(\langle w_M,x_N\rangle +b_M)\end{array}\right].$$

The main goal of ELM is to find an optimal weight $\eta ={[{\eta }_1^T,...,{\eta }_M^T]}^T$ such that $\mathbf{H}\eta =\mathbf{T},$ where $\mathbf{T}={[t_1^T,...,t_N^T]}^T$ is the training set. If the Moore–Penrose generalized inverse${\mathbf{H}}^{†}$ of $\mathbf{H}$ exists, then $\eta ={\mathbf{H}}^{†}\mathbf{T}$ is the desired solution. However, in general cases, ${\mathbf{H}}^{†}$ may not exist or be challenging to find. Hence, to avoid such difficulties, we applied the concept of convex minimization to find $\eta$ without relying on ${\mathbf{H}}^{†}$.

To prevent overfitting, we used the least absolute shrinkage and selection operator (LASSO) [30], formulated as follows:

$$\underset{\eta }{min}{\{\mathbf{H}\eta -\mathbf{T}\parallel }_2^2+{\lambda \parallel \eta \parallel }_1\},$$

(15)

where $\lambda$ is a regularization parameter. In the setting of convex minimization, we set $f\left(x\right)={\parallel \mathbf{H}x-\mathbf{T}\parallel }_2^2$ and $g\left(x\right)={\lambda \parallel x\parallel }_1$.

In the experiment, we aimed to classify three datasets https://archive.ics.uci.edu, accessed on 1 May 2021.

Iris dataset [31]: Each sample in this dataset has four attributes, and the set contains three classes with fifty samples for each type.

Heart disease dataset [32]: This dataset contains 303 samples, each of which has 13 attributes. In this dataset, we classified two classes of data.

Wine dataset [33]: In this dataset, we classified three classes of one-hundred seventy-eight samples. Each sample contained 13 attributes.

In all experiments, we used the sigmoid as the activation function with the number of hidden nodes $M=30.$ The accuracy of the output is calculated by:

$$\mathrm{accuracy} = \frac{\mathrm{correctly} \mathrm{predicted} \mathrm{data}}{\mathrm{all} \mathrm{data}}\times 100.$$

We also utilized 10-fold cross-validation to evaluate the performance of each algorithm and used the average accuracy as the evaluation tool. It is defined as follows:

$${\displaystyle \mathrm{Average} \mathrm{ACC}=\sum _{i=1}^N\frac{x_i}{y_i}\times 100\%/N.}$$

where N is the number of sets considered during cross-validation ($N=10$), $x_i$ is the number of correctly predicted data at fold i, and $y_i$ is the number of all data at fold i.

We used 10-fold cross-validation to split the data into training sets and testing sets; more information can be seen in

Table 1. Number of samples in each fold for all datasets.

	Iris		Heart Disease		Wine
	Train	Test	Train	Test	Train	Test
Fold 1	135	15	273	30	161	17
Fold 2	135	15	272	31	160	18
Fold 3	135	15	272	31	160	18
Fold 4	135	15	272	31	160	18
Fold 5	135	15	273	30	160	18
Fold 6	135	15	273	30	160	18
Fold 7	135	15	273	30	160	18
Fold 8	135	15	273	30	160	18
Fold 9	135	15	273	30	160	18
Fold 10	135	15	273	30	161	17

.

All parameters of Algorithms 3, 5, 6 and 9 were chosen as in

Table 2. Chosen parameters for each algorithm.

	Algorithm 3	Algorithm 5	Algorithm 6	Algorithm 9
$\sigma$	0.49	0.124	0.49	0.124
$\delta$	0.1	0.1	0.1	0.1
$\theta$	0.1	0.1	0.1	0.1

.

The inertial parameters ${\beta }_n$ of Algorithm 9 may vary depending on the dataset, since some ${\beta }_n$ work well on specific datasets. We used the following two choices of ${\beta }_n$ in our experiments.

$${\beta }_n^1=\frac{{10}^8}{\parallel x_n-x_{n-1}{\parallel }^3+n^3+{10}^8}, \mathrm{and} {\beta }_n^2=\left\{\begin{array}{c}0.9, \mathrm{if} n\le 1000\hfill \\ \frac{1}{{(n+1)}^2}, \mathrm{if} n\ge 1001\hfill \end{array}\right..$$

The regularization parameters $\lambda$ for each dataset and algorithm were chosen to prevent overfitting, i.e., a model obtained from the algorithm achieves high accuracy on the training set, but low accuracy on the testing set in comparison, so it cannot be used to predict the unknown data. It is known that when $\lambda$ is too large, the model tends to underfit, i.e., low accuracy on the training set, and cannot be used to predict the future data. On the other hand, if $\lambda$ is too small, then it may not be enough to prevent a model from overfitting. In our experiment, for each algorithm, we chose a set of $\lambda$ that satisfies $|Acc\_train-Acc\_test|<2\%$, where $Acc\_train$ and $Acc\_test$ are the average accuracy of the training set and testing set, respectively. Under this criterion, we can prevent the studied models from overfitting. Then, from these candidates, we chose $\lambda$, which yields high $ACC\_test$ for each algorithm. Therefore, the models obtained from Algorithms 3, 5, 6 and 9 can be effectively used to predict the unknown data.

By this process, the regularization parameters $\lambda$ for Algorithms 5, 6, and 9 were as in

Table 3. Chosen $\lambda$ for each algorithm.

	Regularization Parameter $\mathit{\lambda }$
	Iris	Heart Disease	Wine
Algorithm 3	$0.001$	$0.003$	$0.02$
Algorithm 5	$0.01$	$0.03$	$0.006$
Algorithm 6	$0.9$	$0.2$	$0.003$
Algorithm 9	$0.003$	$0.16$	$0.17$

.

We assessed the performance of each algorithm at the 300th iteration with the average accuracy. The results can be seen in

Table 4. Average accuracy of each algorithm at the 300th iteration with 10-fold cv.

	Iris		Heart Disease		Wine
	Train	Test	Train	Test	Train	Test
Algorithm 3	92.37	90.67	81.85	80.52	97.57	97.16
Algorithm 5	94.37	94.00	83.53	81.84	98.00	97.19
Algorithm 6	96.67	96.00	84.42	83.49	99.25	98.33
Algorithm 9	98.52	98.67	84.30	83.82	99.38	99.44

.

As we see from Table 4, from the choice of $\lambda$, all models obtained from Algorithms 3, 5, 6 and 9 had reasonably high average accuracy on both the training and testing sets for all datasets. Moreover, we observed that a model from Algorithm 9 performed better than the models from other algorithms in terms of the accuracy in all experiments conducted.

4.2. Image Restoration

We first recall that an image restoration problem can be formulated as a simple mathematical model as follows:

$$Ax=b+w$$

(16)

where $x\in {\mathbb{R}}^{n\times 1}$ is the original image, $A\in {\mathbb{R}}^{m\times n}$ is a blurring matrix, b is an observed image, and w is noise. The main objective of image restoration is to find x from given image $b,$ blurring matrix A, and noise w.

In order to solve (16), one could implement LASSO [30] and reformulate the problem in the following form.

$$\underset{x}{min}{\{\parallel Ax-b\parallel }_2^2+{\lambda \parallel x\parallel }_1\},$$

(17)

where $\lambda$ is a regularization parameter. Hence, it can be viewed as a convex minimization problem. Therefore, Algorithms 3, 5, 6 and 9 can be used to solve an image restoration problem.

In our experiment, we used the $256\times 256$ color image as the original image. We used Gaussian blur of size $9^2$ and standard deviation four on the original image and obtained the blurred image. In order to assess the performance of each algorithm, we implemented the peak-signal-to-noise ratio (PSNR) [34] defined by:

$$PSNR\left(x_n\right)=10{log}_{10}\left(\frac{{255}^2}{\mathrm{MSE}}\right).$$

(18)

For any original image x and deblurred image $x_n$, the mean squared error (MSE) is calculated by $\mathrm{MSE} = \frac{1}{M}{\parallel x_n-x\parallel }^2,$ where M is the number of pixels of x. We also need to mention that an algorithm with a higher PSNR performs better than one with a lower PSNR.

The control parameters of each algorithm were chosen as $\delta =\theta =\sigma =0.1$. As the inertial parameter ${\beta }_n$ of Algorithm 9, we used the following:

$${\beta }_n=\left\{\begin{array}{c}0.95, \mathrm{if} n\le 1000\hfill \\ \frac{1}{n^2}, \mathrm{if} n\ge 1001.\hfill \end{array}\right..$$

As for the regularization parameter $\lambda$, we experimented on $\lambda$ varying from zero to one for each algorithm. In Figure 1, we show the PSNR of Algorithms 3 and 5 with respect to $\lambda$ at the 200th iteration. In Figure 2, we show the PSNR of Algorithms 6 and 9 with respect to $\lambda$ at the 200th iteration.

We observe from Figure 1 and Figure 2 that the PSNRs of Algorithms 3, 5, 6 and 9 increased as $\lambda$ became smaller. Based on this, for the next two experiments, we chose $\lambda$ to be small to obtain a high PSNR for all algorithms. Next, we observed the PSNR of each algorithm when $\lambda$ was small ($\lambda <{10}^{-4}$). In Figure 3, we show the PSNR of each algorithm with respect to $\lambda <{10}^{-4}$ at the 200th iteration.

We see from Figure 3 that Algorithm 9 offered a higher PSNR than the other algorithms.

In the next experiment, we chose $\lambda =5\times {10}^{-5}$ for each algorithm and evaluated the performance of each algorithm at the 200th iteration; see

Table 5. PSNR of each algorithm at the 200th iteration.

	PSNR (dB)
Algorithm 3	77.62
Algorithm 5	78.55
Algorithm 6	78.95
Algorithm 9	81.29

for the results.

In Figure 4, we show the PSNR of each algorithm at each step of iteration.

In Figure 5, we show the original test image, blurred image and deblurred images obtained from Algorithms 3, 5, 6 and 9. As we see from Table 5 and Figure 4, Algorithm 9 achieved the highest PSNR.

5. Conclusions

We introduced a new line search technique inspired by [9,19] and used this technique along with the inertial step to construct a new accelerated algorithm for solving convex minimization problems in the form of the sum of two lower semicontinuous convex functions. We proved its weak convergence to a solution of (1), which did not require an L-Lipschitz continuity of $▿f,$ as well as its complexity theorem. We note that many forward–backward-type algorithms require $▿f$ to be L-Lipschitz continuous in order to obtain the convergence theorem; see [10,16,18] for examples. Our proposed algorithm is easy to employ, since an L-Lipschitz constant of $▿f$ does not need to be calculated. Moreover, we also utilized an inertial technique to improve the convergence behavior of our proposed algorithm. In order to show that our algorithm performed better than other line search algorithms mentioned in the literature, we conducted some experiments on classification and image restoration problems. In our experiments, we evaluated the performance of each algorithm based on selecting its suitable parameters, especially the regularization parameter. It was evidenced that our proposed algorithm performed better than the other algorithms on both data classification and image restoration problems. Moreover, we observed from [19,20] that the inertial parameter ${\beta }_n=\frac{t_n-1}{t_{n+1}}$ of FISTA and Algorithm 6 satisfied the condition ${\displaystyle \underset{n}{sup}{\beta }_n=1,}$ while our algorithm required a more strict condition, ${\displaystyle \sum _{n=1}^{+\infty }{\beta }_n<+\infty ,}$ to ensure its convergence to a solution of (1), which was a limitation of our algorithm. We also note that the convergence of FISTA and Algorithm 6 cannot be obtained under the condition ${\displaystyle \underset{n}{sup}{\beta }_n=1}$. Therefore, it is very interesting to find a weaker condition on ${\beta }_n$ that still ensure the convergence of the algorithm to a solution of (1).