An Efficient Three-Term Iterative Method for Estimating Linear Approximation Models in Regression Analysis

Abstract

This study employs exact line search iterative algorithms for solving large scale unconstrained optimization problems in which the direction is a three-term modification of iterative method with two different scaled parameters. The objective of this research is to identify the effectiveness of the new directions both theoretically and numerically. Sufficient descent property and global convergence analysis of the suggested methods are established. For numerical experiment purposes, the methods are compared with the previous well-known three-term iterative method and each method is evaluated over the same set of test problems with different initial points. Numerical results show that the performances of the proposed three-term methods are more efficient and superior to the existing method. These methods could also produce an approximate linear regression equation to solve the regression model. The findings of this study can help better understanding of the applicability of numerical algorithms that can be used in estimating the regression model.

1. Introduction

The steepest descent (SD) method, founded in 1847 by [1], is said to be the simplest gradient and iterative method for minimization of nonlinear optimization problems without constraints. This method is categorized in a single-objective optimization problem which attempts to obtain only one optimal solution [2]. However, due to the low-dimensional property of this method, it converges very slowly. Therefore, since far too little attention has been paid to the modification of the search direction for this method, this study suggests the three-term direction to solve large-scale unconstrained optimization functions.

The standard SD method for solving unconstrained optimization function is defined as

$$\underset{x\in R^r}{\min }f(x)$$

has the following form of direction

$$d_k=-g_k$$

where $f\left(x\right)$ is a continuous differential function in $R^n$ and $g_k=\nabla f\left(x_k\right)$. This minimization method has the following iterative form

$$x_{k+1}=x_k+{\lambda }_k{d}_k$$

(1)

where ${\lambda }_k$ is the step size. This study is particularly interested in using exact line search procedures to obtain ${\lambda }_k$ given by

$${\lambda }_k^{*}=\underset{\lambda >0}{\min }\left\{f\left(x_k+{\lambda }_k{d}_k\right)\right\}$$

(2)

Throughout this paper, without specification, $g_k$ is used to denote the gradient of $f$ at the current iterate point, $x_k$ and $\Vert .\Vert$ to denote the Euclidean norm of vectors. The study will also use $f_k$ as the abbreviation of $f\left(x_k\right)$. The superscript T signifies the transpose.

Line search rules is one of the methods to compute (1) by estimating the direction, $d_k$ and the step size, ${\lambda }_k$. Generally, it can be classified into two types, exact line search and inexact line search rules. The inexact line search represents methods known as Armijo [3], Wolfe [4] and Goldstein [5]. Despite the fact that the exact line search is quite slow compared to inexact line search, in recent years, an increasing number of studies adopting the exact line search was discovered due to faster computing powers such as in [6]. This research emphasized the exact line search as we assume that this new era of fast computer processors will give an advantage in using this line search.

The remainder of this study is organized as follows: in Section 2, the evolution of the SD method is discussed while in Section 3, the proposed three-term SD methods with two different scaled parameters and their convergence analysis are presented. Next, numerical results of the proposed methods are illustrated and discussed in Section 4 while in Section 5, the implementation in regression analysis of all proposed methods is demonstrated. A brief conclusion and some future recommendations are provided in the last section of this paper.

2. Evolution of Steepest Descent Method

The issue on search direction modification for SD method has grown importance in light of recent as in 2018, [7] introduced a new descent method that used a three-step discretization method which has an intermediate step between the initial point, x₀ to the next iterate point, $x_{k+1}$. In 2016, [8] proposed a search direction of the SD method that possessed global convergence properties. The search direction of the proposed, named as ZMRI taken by the name of the researches Zubai’ah, Mustafa, Rivaie and Ismail, has improved the behavior of the SD method where a proportion of previous search directions is added to the current negative gradient. This search direction is given by

$$d_k^{ZMRI}=-g_k-\Vert g_k\Vert g_{k-1}$$

(3)

The numerical result of the method revealed that ZMRI has superior performance compared to the standard SD and the method was also 11 times faster than SD.

Recently, inspired by (3), [9] proposed a scaled SD method that also satisfied global convergence properties. The search direction is known as $d_k^{RRM}$ which abbreviated from the researcher’s name Rashidah, Rivaie and Mustafa, is given by

$$d_k^{RRM}=\{\begin{array}{ll}-g_k\hfill & \hfill \mathrm{if} k=0\\ -{\theta }_k{g}_k-\Vert g_k\Vert g_{k-1}\hfill & \hfill \mathrm{if} k\ge 1\end{array}$$

The value of ${\theta }_k$ was taken from the coefficient in [10] and defined as

$${\theta }_k=\frac{d_{k-1}^T{y}_{k-1}}{{\Vert g_{k-1}\Vert }^2}$$

where $y_{k-1}=g_k-g_{k-1}$. The method was then compared with standard SD and (3). The results showed that RRM is the fastest solver for about 76.79% of the 14 selected test problems and solved 100% of the problem.

Several modifications to the SD method have been made. Recently, [11] presented a three-term iterative method for unconstrained optimization problems motivated from [12,13,14] defined as follows:

$$d_k=\{\begin{array}{ll}-g_k,\hfill & \hfill \mathrm{if} k=0\\ -g_k+{\beta }_k{g}_{k-1}-{\theta }_k{y}_{k-1},\hfill & \hfill \mathrm{if} k\ge 1\end{array}$$

(4)

where

$${\beta }_k=\frac{g_k^T{y}_{k-1}}{{\Vert g_{k-1}\Vert }^2},{\theta }_k=\frac{{\Vert g_k\Vert }^2}{{\Vert g_{k-1}\Vert }^2}$$

As researchers can see, the author put a restart feature that directly addresses the jamming problem. When the step $x_k-x_{k-1}$ is too small, then the factor $y_{k-1}$ approaches the zero vector. The author has also proven that the method is globally convergent under standard Armijo-type line search and modified Armijo-type line search. As a result, the numerical experiments for the proposed method is much better than the methods in [12,13,14].

3. Algorithm and Convergence Analysis of New Three-Term Search Direction

This section presents the new three-term search direction for SD method to solve large-scale unconstrained optimization problems. This research highlights the development of SD method that can lessen the number of iterations and CPU time while establishing the theoretical proofs under exact line searches. Motivated by the above evolutions on SD, the new direction formula is obtained as follows:

$$d_k=\{\begin{array}{ll}-g_k,\hfill & \hfill \mathrm{if} k=0\\ -g_k-{\beta }_k{g}_{k-1}+{\theta }_k{y}_{k-1},\hfill & \hfill \mathrm{if} k\ge 1\end{array}$$

(5)

In this research, by employing the parameter from the conjugate gradient method which is said to have faster convergence and lower memory requirements [15], two different scaled parameters, ${\beta }_k$ and ${\theta }_k$, are presented. For the first direction, the parameters are called as three-term SD and abbreviated as TTSD1 are

$${\beta }_k=\frac{{\Vert g_k\Vert }^2}{{\Vert g_{k-1}\Vert }^2}\mathrm{and}{\theta }_k=\frac{g_k^T{g}_{k-1}}{{\Vert g_{k-1}\Vert }^2}$$

while the second direction known as TTSD2 which is an extension of TTSD1, the parameters are

$${\beta }_k=\frac{{\Vert g_k\Vert }^2+{\Vert g_{k-1}\Vert }^2}{{\Vert g_{k-1}\Vert }^2}\mathrm{and}{\theta }_k=\frac{g_k^T{g}_{k-1}-{\Vert g_{k-1}\Vert }^2}{{\Vert g_{k-1}\Vert }^2}$$

The idea of the extension arises from the recent literature reviews, for instance in [16,17,18,19,20], which seek to improve the performance and effectiveness of the existing methods. The proposed directions with the exact line search procedure were implemented in the algorithm as follows.

Algorithm 1: Steepest Descent Method.

Step 0: Given a starting or initial point $x_0$, set $k=0$. Step 1: Determine the direction, $d_k$ using (5). Step 2: Evaluate step length or step size, ${\lambda }_k$ using exact line search as in (2). Step 3: Update new point, $x_{k+1}\leftarrow x_k+{\lambda }_k{d}_k$ for $k\leftarrow k+1$. If $\Vert g_k\Vert \le \epsilon$, then, stop, else go to Step 1.

3.1. Convergence Analysis

This section indicates the theoretical prove that (5) holds the convergence analysis both in sufficient descent directions and global convergence properties.

3.1.1. Sufficient Descent Conditions

Let sequence $\left\{d_k\right\}$ and $\left\{x_k\right\}$ be generated by (5) and (1), then

$$g_k^T{d}_k\le -{\Vert g_k\Vert }^2 \mathrm{for} k\ge 0.$$

(6)

Theorem 1. 

Consider the three-term search direction given by (5) with the TTSD1 as scaled parameters and the step size determined by the exact procedure (2). Then condition (6) holds for all$k\ge 0$.

Proof. 

Obviously, if $k=0$, then the conclusion is true.

Then, to show that for $k\ge 1$, condition (6) will also hold true.

Multiply (5) by $g_k$ and by noting that $g_k^T{d}_k=0$ for exact line search procedure, we will get

$$\begin{array}{cc}\hfill g_k^T{d}_k& =-{\Vert g_k\Vert }^2-g_k^T\left({\beta }_k{g}_{k-1}-{\theta }_k{y}_{k-1}\right)\hfill \\ \hfill & =-{\Vert g_k\Vert }^2-\frac{{\Vert g_k\Vert }^2}{{\Vert g_{k-1}\Vert }^2}{g}_k^T{g}_{k-1}+\frac{g_k^T{g}_{k-1}}{{\Vert g_{k-1}\Vert }^2}{g}_k^T\left(g_k-g_{k-1}\right)\hfill \\ \hfill & =-{\Vert g_k\Vert }^2-\frac{{\Vert g_k^T{g}_{k-1}\Vert }^2}{{\Vert g_{k-1}\Vert }^2}\hfill \\ \hfill & \le -{\Vert g_k\Vert }^2\hfill \end{array}$$

Therefore, condition (6) holds and thus the proof is complete, which implies that $d_k$ is a sufficient descent direction.□

3.1.2. Global Convergence

The following assumptions and lemma are needed in the analysis of the global convergence of SD methods.

Assumption  1.

The level set $\mathsf{\Omega }=\left\{x\in R^n|f(x)\le f\left(x_0\right)\right\}$ is bounded where $x_0$ is the initial point.

In some neighborhoods $N$ of $\mathsf{\Omega }$, the objective function is continuously differentiable, and its gradient is Lipchitz continuous, namely, there exists a constant $l>0$ such that $\Vert g(x)-g(y)\Vert \le l\Vert x-y\Vert$ for any $x,y\in N$.

These assumptions yield the following Lemma 1.

Lemma 1.

Suppose that Assumption 1 holds true. Let $x_k$ be generated by Algorithm 1, $d_k$ satisfies (6) and ${\lambda }_k$ satisfies exact minimization rule, then there exists a positive constant $h$ such that

$${\lambda }_k\ge h\frac{{\Vert g_k\Vert }^2}{{\Vert d_k\Vert }^2}$$

and one can also have,

$${\displaystyle \sum _{k=0}^{\infty }\frac{{\Vert g_k\Vert }^4}{{\Vert d_k\Vert }^2}}<\infty$$

This property is known as Zoutendijk condition. Details of this condition are given in [21].

Theorem 2.

Assume that Assumption 1 holds true. Consider $x_k$ generated by Algorithm 1 above, ${\lambda }_k$ is calculated using exact line search and possesses the sufficient descent condition. Then,

$$\underset{k\to \infty }{\lim }\Vert g_k\Vert =0 \mathrm{or} {\displaystyle \sum _{k=0}^{\infty }\frac{{\left(g_k^T{d}_k\right)}^2}{{\Vert d_k\Vert }^2}}<\infty$$

Proof. 

The proof is done using a contradiction rule. By assuming that Theorem 2 is not true, that is, $\underset{k\to \infty }{\lim }\Vert g_k\Vert \ne 0$. Then, there exists a positive constant ${\delta }_1$, such that $\Vert g_k\Vert \ge {\delta }_1$ for all value of $k$. From Assumption 1, we know that there exists a positive constant ${\delta }_2$ such that $\Vert g_k\Vert \le {\delta }_2$ for all values of $k$. From (5) and using the first scaled parameters (TTSD1), we have

$$\begin{array}{cc}\hfill \Vert d_k\Vert & \le \Vert g_k\Vert +\left|{\beta }_k\right|\Vert g_{k-1}\Vert +\left|{\theta }_k\right|\Vert y_{k-1}\Vert \hfill \\ \hfill & \le \Vert g_k\Vert +\frac{{\Vert g_k\Vert }^2}{{\Vert g_{k-1}\Vert }^2}\Vert g_{k-1}\Vert +\frac{\Vert g_k\Vert \Vert g_{k-1}\Vert }{{\Vert g_{k-1}\Vert }^2}\left(\Vert g_k\Vert +\Vert g_{k-1}\Vert \right)\hfill \\ \hfill & \le 2\Vert g_k\Vert +2\frac{{\Vert g_k\Vert }^2}{\Vert g_{k-1}\Vert }\hfill \\ \hfill & \le 2{\delta }_2+2\frac{{\delta }_2^2}{{\delta }_1}\hfill \\ \hfill & \triangleq M_1 \mathrm{where} M_1=2{\delta }_2+2\frac{{\delta }_2^2}{{\delta }_1}\hfill \end{array}$$

The above inequality implies

$${\displaystyle \sum _{k=0}^{\infty }\frac{{\Vert g_k\Vert }^4}{{\Vert d_k\Vert }^2}}\ge {\displaystyle \sum _{k=0}^{\infty }\frac{{\delta }_1^4}{M_1^2}}$$

(7)

Thus, from (7), it follows that

$${\displaystyle \sum _{k=0}^{\infty }\frac{{\Vert g_k\Vert }^4}{{\Vert d_k\Vert }^2}}=\infty$$

which contradicts Zoutendijk condition in Lemma 1.

Therefore,

$${\displaystyle \sum _{k=0}^{\infty }\frac{{\left(g_k^T{d}_k\right)}^2}{{\Vert d_k\Vert }^2}}<\infty$$

Hence, the proof is complete. □

Remark 1.

The sufficient property and global convergence for TTSD2 can also be proven similar to the proof of Theorem 1 and 2.

4. Numerical Experiments

This section examines the feasibility and effectiveness of Algorithm 1 with the use of (4) and (5) as the search direction in Step 3 under the exact line search rules by implementing the performance profile introduced by [22] as a tool for comparison. The test problems with the sources are listed in

Table 1. List of test functions.

Number	Functions	Initial Points
F1	Extended White and Holst [23]	(0,0,…,0), (2,2,…,2), (5,5,…,5)
F2	Extended Rosenbrock [24]	(0,0,…,0), (2,2,…,2), (5,5,…,5)
F3	Extended Freudenstein and Roth [24]	(0.5,0.5,…,0.5), (4,4,…,4), (5,5,…,5)
F4	Extended Beale [25]	(0,0,…,0), (2.5,2.5,…,2.5), (5,5,…,5)
F5	Raydan 1 [23]	(1,1,…,1), (20,20,…,20), (5,5,…,5)
F6	Extended Tridiagonal 1 [23]	(2,2,…,2), (3.5,3.5,…,3.5), (7,7,…,7)
F7	Diagonal 4 [23]	(1,1,…,1), (5,5,…,5), (10,10,…,10)
F8	Extended Himmelblau [25]	(1,1,…,1), (5,5,…,5), (15,15,…,15)
F9	Fletcher [23]	(0,0,…,0), (2,2,…,2), (7,7,…,7)
F10	Nonscomp [23]	(3,3,…,3),(10,10,…,10),(15,15,…,15)
F11	Extended Denschnb [23]	(1,1,…,1), (5,5,…,5), (15,15,…,15)
F12	Shallow [26]	(−2,−2,…,−2), (0,0,…,0), (5,5,…,5)
F13	Generalized Quartic [23]	(1,1,…,1), (4,4,…,4), (−1,−1,…,−1)
F14	Power [23]	(−3,−3,…,−3), (1,1,…,1), (5,5,…,5)
F15	Quadratic 1 [23]	(−3,−3,…,−3), (1,1,…,1), (10,10,…,10)
F16	Extended Sum Squares [27]	(2,2,…,2),(10,10,…,10),(−15,−15,…,−5)
F17	Extended Quadratic Penalty 1 [23]	(1,1,…,1), (10,10,…,10), (15,15,…,15)
F18	Extended Penalty [23]	(1,1,…,1), (5,5,…,5), (10,10,…,10)
F19	Leon [28]	(1,1,…,1), (5,5,…,5), (10,10,…,10)
F20	Extended Quadratic Penalty 2 [23]	(5,5,…,5), (10,10,…,10), (15,15,…,15)
F21	Maratos [23]	(1.1,1.1,…,1.1), (5,5,…,5), (10,10,…,10)
F22	Three Hump [29]	(3,3), (20,20), (50,50)
F23	Six Hump [29]	(10,10), (15,15), (20,20)
F24	Booth [25]	(3,3), (20,20), (50,50)
F25	Trecanni [30]	(−5,−5), (20,20), (50,50)
F26	Zettl [25]	(−10,−10), (20,20), (50,50)

. The codes were written in MATLAB 2017a.

For the purpose of comparison, the methods were evaluated over the same set of test problems (see Table 1). The total number of test problems was twenty-six with three different initial points ranging from 2 to 5000 number of variables. The results were divided into two groups, which in the first group was the comparison between the proposed directions with standard and previous SD methods, [8,9] while in the second group the numerical results were compared with another three-term iterative method introduced by [11] using exact line search procedures. Numerical results were compared based on the number of iterations and CPU times evaluated. In the experiments, the termination condition is $\Vert g_k\Vert \le {10}^{-5}$. We also forced the routine to stop if the total number of iteration exceeded 10,000.

For the methods being analyzed, a performance profile introduced by [22] was implemented to compare the performance of the set solvers S on a test set of problems P. Assuming $n_s$ as number of solvers and $n_p$ as number of problems, they defined $t_{p,s}=\mathrm{computing} \mathrm{time} (\mathrm{number} \mathrm{of} \mathrm{iterations} \mathrm{or} \mathrm{others}) \mathrm{needed} \mathrm{to} \mathrm{solve} \mathrm{problem} p \mathrm{by} \mathrm{solver} s$.

The performance ratio used to compare the performance by solver s with the best performance by any solver on problem p which they defined as

$$r_{p,s}=\frac{t_{p,s}}{\min \left\{t_{p,s}:s\in S\right\}}$$

In order to get the overall evaluations of the solver’s performance, they definedas ρ_s(t) as a probability for a solver $s\in S$ that $r_{p,s}$ was within a factor $t\in R$ of the best possible ration. The probability is described as

$${\rho }_s(t)=\frac{1}{n_p}size\left\{p\in P:r_{p,s}\le t\right\}$$

in which the function ${\rho }_s$ was the cumulative distribution function for the performance ratio. The performance profile ${\rho }_s:R\in \left[0,1\right]$ was for a solver was piecewise non-decreasing and continuous from the right at each breakpoint. Generally, the higher value of ${\rho }_s(t)$ or in other words, the solver whose performance profile plot is on the top right will win the rest of the solvers or represents the best solver.

Figure 1 show the comparison of the proposed method with the standard SD, ZMRI and RRM methods. In Figure 2 in order to emphasize the proposed search direction from the direction in [11] abbreviated as WH, it might call the present formula as first and second three-term SD methods, TTSD1 and TTSD2, respectively. The performance for all the methods, referring to the number of iterations evaluated and central processing unit (CPU) time, respectively, are displayed.

From the above figures, the TTSD1 method outperforms the other methods in both the number of iterations and CPU time evaluations. This can be seen from the left side of Figure 1 and Figure 2 in which TTSD1 is the fastest method in solving all of the test problems and from the right side of the figures, this method also gives the highest percentage of successfully solved test problems compared to other methods. The probability of all the solvers or the methods involved was not approaching 1 which means that they are not able to solve all of the problems tested. The percentage of the successful problems solved by each solver is tabularized in

Table 2. CPU time (in seconds) per single iteration and successful percentage in solving all the functions using the exact line search.

Methods	Total Number of Iterations	Total Number of Cpu Times (s)	Cpu Time Per Iteration (s)	Successful Functions Solved (%)
SDC	329,978	2638.52	0.007996	74.45
ZMRI	106,316	493.10	0.004638	75.18
RRM	155,271	1412.42	0.009096	72.02
WH	116,822	890.50	0.007623	82.48
TTSD1	70,520	376.43	0.005338	81.02
TTSD2	73,981	430.40	0.005818	82.97

. Table 2 also presents the CPU time per single iteration based on the evaluation of the total iterations and total CPU times. Although the performance of other methods seems to be much better than the proposed method, TTSD1 and TTSD2 can be considered as the superior method since it can solve 81.02% and 82.97% of the functions tested.

5. Implementation in the Regression Model

In modern times, optimal mathematical models have become common resources for researchers, for instance, in the construction industry, these tools are used to find a solution to minimize costs and maximize profits [31]. Steepest descent method is said to have various applications mostly in finance, network analysis and physics as it is easy to use. One of the most frequent employment of this method is in regression analysis. This paper aims to investigate the use of the proposed direction in describing the relationship between fin dorsal length and the total length of silky shark. The data were collected by [32] from March 2018 to February 2019 at Tanjung Luar Fish Landing Post, West Nusa Tenggara. The study was carried out to set the minimum size of fin products for international trade and the author also pointed out that this data can be used by the fisheries authority to determine the allowed minimum size of silky shark fins for export.

Figure 3 shows the linear approximation between the total length and the length of a dorsal fin of silky shark as $y=0.125610046x-0.018027898$. In order to measure the model performance, the coefficient of determination, $R^2$, has been calculated as a standard metric for model errors and it showed that the value of $R^2$ is close to 1, means there is a strong relationship between the total length of silky shark with the length of its dorsal fin. The total length of the silky shark was measured from the anterior tip of the snout to the posterior part of the caudal fin while the dorsal fin length measured from the fin base to the tip of the fin as shown in Figure 4.

The linear regression analysis was implemented by using the dorsal fin length as dependent variables $y$ and the total length of a silky shark as an independent variable $x$ with a model is indicated as

$$y=a_0+a_{1}x.$$

In order to estimate the above linear regression equation, the least square method was conducted by assuming the estimators are the values of parameters $a={\left(a_0,a_1\right)}^T$ which minimize the objective function as follows:

$$S=\underset{a\in R^2}{\min }f(a)={\displaystyle \sum _{i=1}^n{\left[y_i-\left(a_0+a_{1}x\right)\right]}^2}$$

(8)

The sum squares of $S$ can be minimized by utilizing the concept of calculus, differentiating (8) with respect to all the parameters involved. The equations can be written in a matrix form and lead to the system of the linear equation. By using the inversion of the matrix method to solve the system of linear equation, the solution is derived as

$$y=-0.018027898+0.125610064x.$$

Another method to find the solution of a system of linear equation is by using the numerical method. In this context, the proposed three-term method is implemented as a numerical method to solve the system as a comparison with the aforementioned inversion of the matrix. To test the efficiency of the proposed method TTSD1 and TTSD2,

Table 3. Summary of results.

Methods	Parameters		Number of Iterations	CPU Time (s)	Sum of Relative Errors
	a0	a1
Direct Inverse	−0.018027898	0.125610046	−	−	0.900514941
TTSD1	−0.01802868	0.125610507	3	0.1257256	0.900507129
TTSD2	−0.018027192	0.125609646	11	0.1104726	0.900523836
WH	−0.018025098	0.125608377	65	0.1682789	0.900540824

gives an overview of the estimations model coefficients using an inverse method, TTSD methods and also WH method with the number of iterations (with initial point is $\left(0,0\right)$).

The accuracy and performance of these methods are measured by the sum of relative errors by using the total of the differences between the approximation and the exact values of the data. The sum of relative errors are tabulated in Table 3 where the equation of the relative errors is defined as

$$\mathit{Relative} \mathit{Error}=\frac{\left|\mathit{Exact} \mathit{Value} - \mathit{Approximate} \mathit{Value}\right|}{\left|\mathit{Exact} \mathit{Value}\right|}$$

where the exact value gained from the actual data and the approximate value is the value obtained by each method involved. From Table 3, it can be observed that TTSD1 has the least value of errors followed by the inversion matrix method and TTSD2 which implies that these two methods are comparable with the direct inverse method.

6. Conclusions and Future Recommendations

The main objective of this paper is to propose a three-term SD method also known as the iterative method with two different scaled parameters. The effectiveness of the method, TTSD1 and TTSD2, were tested by comparing with the previous SD (standard, ZMRI and RRM) and three-term method presented in [13], named the WH method, using the same set of test problems under exact line search algorithms. The proposed method possesses sufficient descent and global convergence properties. Through several tests, the method TTSD1 and TTSD2 really outperform the previous SD and other three-term iterative methods. The reliability of TTSD1 and TTSD2 was found to be consistent with the results obtained by the direct inverse method for the implementation in the regression analysis. This finding shows that the methods are comparable and applicable. There is abundant room for further research on the SD method. In the future, we intend to test this new TTSD1 and TTSD2 using the inexact line search.