Regularization Total Least Squares and Randomized Algorithms

Abstract

In order to achieve an effective approximation solution for solving discrete ill-conditioned problems, Golub, Hansen, and O’Leary used Tikhonov regularization and the total least squares (TRTLS) method, where the bidiagonal technique is considered to deal with computational aspects. In this paper, the generalized singular value decomposition (GSVD) technique is used for computational aspects, and then Tikhonov regularized total least squares based on the generalized singular value decomposition (GTRTLS) algorithm is proposed, whose time complexity is better than TRTLS. For medium- and large-scale problems, the randomized GSVD method is adopted to establish the randomized GTRTLS (RGTRTLS) algorithm, which reduced the storage requirement, and accelerated the convergence speed of the GTRTLS algorithm.

1. Introduction

In practical problems, many discrete ill-conditioned problems arising from many different fields of physics and engineering can be reduced to solving linear equations in the form of $Ax\approx b$. The methods used commonly are least squares (LS) [1] and total least squares (TLS) [2,3]. However, these kinds of problems are often ill-conditioned, such as the first kind of integral equations [4,5]. In order to reduce the serious instability caused by the problems themselves, regularization treatment [6,7,8,9,10,11] becomes an effective method, that is, replacing the original ill-conditioned problem with an adjoining well-conditioned one, whose solution is called a regularized solution to approximate the true one. We know that Tikhonov regularization is one of the common methods, which is widely used in the industrial field [6]. For example, Tikhonov regularized TLS (TRTLS) proposed by Golub, Hansen, and O’Leary can be used to approach the true solution. During the process, the bidiagonalization technique is used. It is shown that the ideal approximation solution cannot be obtained by the truncation singular value method in some practical problems. The total least squares problem with the general Tikhonov regularization (TRTLS) is a non-convex optimization problem with local non-global minimizers. Xia [12] proposed an efficient branch-and-bound algorithm (algorithm BTD) for solving TRTLS problems guaranteed to find a global$ϵ$-approximation solution in most $O(1/ϵ)$ iterations, and the computational effort in each iteration is $O\left(n^{3}log\right(1/ϵ)$. Beamforming is one of the most important techniques for enhancing the quality of signal in array sensor signal processing, and the performance of a beamformer is usually related to the design of array configuration and beamformer weight. In [13], Chen first proposed a design model for a proximal sparse beamformer, which obtains sparse and robust filter coefficients by solving the composite optimization problem. The objective function of the model is the sum of the least squares term, the approximate term, and the ${\mathcal{l}}_1$-regularization term.

Hansen often uses generalized singular value decomposition (GSVD) to analyze regularization methods [14]. However, using the GSVD method to solve large-scale discrete ill-conditioned problems requires a large amount of computation and memory requirement. For this kind of problem, Martin and Reichel [9] proposed a method to find the corresponding truncated regularization (TR) solution by using low-rank partial singular value decomposition. In order to improve the time complexity, this paper uses GSVD technology to deal with Tikhonov regularization TLS and establishes Tikhonov regularization TLS based on the GSVD (GTRTLS) algorithm. At the same time, for medium- and large-scale problems, in order to reduce the storage requirements and accelerate the speed of GSVD, the randomized GSVD method [15,16] is used, and then we obtain the randomized GTRTLS (RGTRTLS) algorithm. For the randomized algorithms of large-scale matrix decompositions and their application to ill-conditioned problems, one can see [17,18,19,20] for examples and details.

Our main contribution is to use GSVD technology to deal with Tikhonov regularization TLS (GTRTLS) and to adopt the randomized techniques of [15,16] to implement the GTRTLS procedure in the regularization. The randomized GSVD requires much less storage and computational time than the classical schemes. Numerical examples show the effectiveness and superiority of our algorithms.

This paper is organized as follows: Section 2 describes our technique of combining Tikhonov regularized TLS and GSVD. Section 3 contains our randomized algorithms, and their error analyses for randomized algorithms are in Section 4. The improvement in time and memory requirements is illustrated with numerical examples in Section 5. Section 6 concludes this paper.

2. Tikhonov Regularization TLS and GSVD

The regularized TLS problem can be expressed as

$$\min {‖\left(A,b\right)-\left(\overline{A},\overline{b}\right)‖}_F \mathrm{s}.\mathrm{t}. \overline{A}x=\overline{b}, {‖Lx‖}_2\le \delta$$

(1)

where $\mathsf{\delta }$ is a positive constant. Typical examples of the matrix L are the first derivative approximation ${\mathrm{L}}_1$ and the second derivative approximation ${\mathrm{L}}_2$, which are as follows (see [14], Equation (1.2), and [21], Equation (4.57)):

$$L_1={\left(\begin{array}{cccc}1& -1& & \\ & \ddots & \ddots & \\ & & 1& -1\end{array}\right)}_{(n-1)\times n}, {\mathrm{L}}_2={\left(\begin{array}{ccccc}-1& 2& -1& & \\ & \ddots & \ddots & \ddots & \\ & & -1& 2& -1\end{array}\right)}_{(n-2)\times n}.$$

More precisely, derivative-based finite-difference methods $L_1$ and $L_2$ are approximations of the first and second derivative operators on a uniform grid, where the scaling factor is ignored.

The corresponding Lagrange multiplier formulation is

$$\mathcal{L}\left(\overline{A},x,\mu \right)={‖\left(A,b\right)-\left(\overline{A},\overline{b}\right)‖}_F^2+\mu ({‖Lx‖}_2^2-{\delta }^2)$$

(2)

where $\mu$ is the Lagrange multiplier.

To ensure that the TRTLS problem (1) has a unique solution, throughout this paper, we assume that

$${\sigma }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(\mathrm{AK}\right)>{\sigma }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(\right(\mathrm{A}\mathrm{K},\mathrm{b}\left)\right),$$

(3)

where $K\in {\mathbb{R}}^{n\times s}$ is a matrix whose columns form an orthonormal basis of the null-space of the regularization matrix L, and ${\sigma }_{min}$ denotes the minimal singular value of its argument [4].

A popular approach to overcoming numerical instability is Tikhonov regularization TLS [4]. It can be seen that the regularized total least squares solution can be obtained from the following theorem (see reference [7]):

Theorem 1

([7]). With the inequality constraint replaced by equality, the TRTLS solution x to (1) is a solution to the problem

$$\left(A^{T}A+{\lambda }_I{I}_n+{\lambda }_L{L}^{T}L\right)x=A^{T}b,$$

(4)

where the parameters ${\lambda }_I$  and  ${\lambda }_L$  are given by

$${\lambda }_I=-\frac{{‖b-Ax‖}_2^2}{1+{‖x‖}_2^2},$$

$${\lambda }_L=\mu ( 1+{‖x‖}_2^2),$$

and where $\mu$ is the Lagrange multiplier in (2). ${\lambda }_I and {\lambda }_L$  are related by

$${\mathsf{\lambda }}_{\mathrm{L}}{\mathsf{\delta }}^2={\mathrm{b}}^{\mathrm{T}}\left(\mathrm{b}-\mathrm{A}\mathrm{x}\right)+{\mathsf{\lambda }}_{\mathrm{I}}.$$

Moreover, the TLS residual satisfies 

$${‖(\mathrm{A},\mathrm{b})-(\overline{\mathrm{A}},\overline{\mathrm{b}})‖}_{\mathrm{F}}^2=-{\mathsf{\lambda }}_{\mathrm{I}}.$$

For problem (1), we have the following assumptions:

$$\mathrm{A}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{n}}, \mathrm{L}\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{n}}, \mathrm{m}\ge \mathrm{n}\ge \mathrm{p}, \mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}\left(\mathrm{L}\right)=\mathrm{p},$$

$$\mathcal{N}\left(\mathrm{A}\right)\bigcap \mathcal{N}\left(\mathrm{L}\right)=\left\{0\right\}⟺\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}\left(\left(\begin{array}{c}\mathrm{A}\\ \mathrm{L}\end{array}\right)\right)=\mathrm{p}$$

(5)

According to the literature [1,12,22], the GSVD of matrix for $\{\mathrm{A},\mathrm{L}\}$ is

$$\mathrm{A}=\mathrm{U}\left(\begin{array}{c}\mathsf{\Sigma }\\ 0\end{array}\right){\mathrm{X}}^{-1},\mathsf{\Sigma }=\left(\begin{array}{cc}{\mathsf{\Sigma }}_{\mathrm{p}}& \\ & {\mathrm{I}}_{\mathrm{n}-\mathrm{p}}\end{array}\right), \mathrm{L}=\mathrm{V}\left(\begin{array}{cc}\mathrm{M}& 0\end{array}\right){\mathrm{X}}^{-1},$$

(6)

where $\mathrm{U}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{m}}$ and $\mathrm{V}\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{p}}$ are orthonormal matrices, $\mathrm{X}$ is an invertible matrix. The matrices ${\mathsf{\Sigma }}_p$, $\mathrm{M}$ are diag$\{{\sigma }_{1,}{\sigma }_2,\cdots ,{\sigma }_p\}$ and diag$\{{\mu }_{1,}{\mu }_2,\cdots ,{\mu }_p\}$, respectively, satisfying $0\le {\sigma }_1\le {\sigma }_2\le \cdots \le {\sigma }_p\le 1$, $1{\ge \mu }_1{\ge \mu }_2\ge \cdots \ge {\mu }_p\ge 0$, ${{\sigma }_i}^2+{{\mu }_i}^2=1,i=1,2,\cdots ,p.$

It can be seen that (4) is equivalent to the augmented system

$$\left(\begin{array}{ccc}{\mathrm{I}}_{\mathrm{m}}& 0& \mathrm{A}\\ 0& {\mathrm{I}}_{\mathrm{p}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\mathrm{L}\\ {\mathrm{A}}^{\mathrm{T}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}{\mathrm{L}}^{\mathrm{T}}& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}\mathrm{r}\\ \mathrm{s}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}\mathrm{b}\\ 0\\ 0\end{array}\right).$$

(7)

In order to improve the time complexity, this paper uses GSVD technology to deal with Tikhonov regularization TLS. In the first step, we reduce the GSVD of $\{\mathrm{A},\mathrm{L}\}$ to (7)

$$\left(\begin{array}{ccc}{\mathrm{I}}_{\mathrm{m}}& 0& \left(\begin{array}{c}\mathsf{\Sigma }\\ 0\end{array}\right){\mathrm{X}}^{-1}\\ 0& {\mathrm{I}}_{\mathrm{p}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\left(\begin{array}{cc}\mathrm{M}& 0\end{array}\right){\mathrm{X}}^{-1}\\ {\mathrm{X}}^{-\mathrm{T}\left(\begin{array}{cc}{\mathsf{\Sigma }}^{\mathrm{T}}& 0\end{array}\right)}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}{\mathrm{X}}^{-\mathrm{T}}\left(\begin{array}{c}{\mathrm{M}}^{\mathrm{T}}\\ 0\end{array}\right)& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}{\mathrm{U}}^{\mathrm{T}}\mathrm{r}\\ {\mathrm{V}}^{\mathrm{T}}\mathrm{s}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\\ 0\\ 0\end{array}\right)$$

(8)

Let $\mathrm{U}=\left(\begin{array}{cc}{\mathrm{U}}_1,& {\mathrm{U}}_2\end{array}\right)$; then, we have

$$\left(\begin{array}{ccc}{\mathrm{I}}_{\mathrm{n}}& 0& {\mathsf{\Sigma }\mathrm{X}}^{-1}\\ 0& {\mathrm{I}}_{\mathrm{p}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\left(\begin{array}{cc}\mathrm{M}& 0\end{array}\right){\mathrm{X}}^{-1}\\ {\mathrm{X}}^{-\mathrm{T}}{\mathsf{\Sigma }}^{\mathrm{T}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}{\mathrm{X}}^{-\mathrm{T}}\left(\begin{array}{c}{\mathrm{M}}^{\mathrm{T}}\\ 0\end{array}\right)& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}{{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{r}\\ {\mathrm{V}}^{\mathrm{T}}\mathrm{s}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}{{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{b}\\ 0\\ 0\end{array}\right)$$

(9)

or

$$\left({\mathsf{\Sigma }}^{\mathrm{T}}\mathsf{\Sigma }+{\mathsf{\lambda }}_{\mathrm{I}}{\mathrm{X}}^{\mathrm{T}}\mathrm{X}+{\mathsf{\lambda }}_{\mathrm{L}}\left(\begin{array}{cc}{\mathrm{M}}^{\mathrm{T}}\mathrm{M}& 0\\ 0& 0\end{array}\right)\right){\mathrm{X}}^{-1}\mathrm{x}={{\mathsf{\Sigma }}^{\mathrm{T}}{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{b}.$$

(10)

In the second step, using the Elden algorithm [4], only p steps of Givens transformation are needed to eliminate the ${{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\left(\begin{array}{cc}\mathrm{M}& 0\end{array}\right)$, which can be expressed as

$$\left(\begin{array}{c}\mathsf{\Sigma }\\ {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\left(\begin{array}{cc}\mathrm{M}& 0\end{array}\right)\end{array}\right)=\mathrm{G}\left(\begin{array}{c}\widehat{\mathsf{\Sigma }}\\ 0\end{array}\right)=\left(\begin{array}{cc}{\mathrm{G}}_{11}& {\mathrm{G}}_{12}\\ {\mathrm{G}}_{21}& {\mathrm{G}}_{22}\end{array}\right)\left(\begin{array}{c}\widehat{\mathsf{\Sigma }}\\ 0\end{array}\right).$$

When G is applied to the augmented system (9), we have

$$\left(\begin{array}{ccc}{\mathrm{I}}_{\mathrm{n}}& 0& {\widehat{\mathsf{\Sigma }}\mathrm{X}}^{-1}\\ 0& {\mathrm{I}}_{\mathrm{p}}& 0\\ {\mathrm{X}}^{-\mathrm{T}}{\widehat{\mathsf{\Sigma }}}^{\mathrm{T}}& 0& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}\widehat{\mathrm{r}}\\ \widehat{\mathrm{s}}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}{{\mathrm{G}}_{11}}^{\mathrm{T}}{{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{b}\\ {{\mathrm{G}}_{12}}^{\mathrm{T}}{{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{b}\\ 0\end{array}\right).$$

Since the solution of $\widehat{\mathrm{s}}$ can be obtained from the above formula, only the following system can be considered:

$$\left(\begin{array}{cc}{\mathrm{I}}_{\mathrm{n}}& {\widehat{\mathsf{\Sigma }}\mathrm{X}}^{-1}\\ {\mathrm{X}}^{-\mathrm{T}}{\widehat{\mathsf{\Sigma }}}^{\mathrm{T}}& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}\widehat{\mathrm{r}}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}{{\mathrm{G}}_{11}}^{\mathrm{T}}{{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{b}\\ 0\end{array}\right).$$

In the third step, ${\widehat{\mathsf{\Sigma }}\mathrm{X}}^{-1}$ is reduced to $n\times n$ bidiagonal matrix $B$ by orthogonal transformation, such that $H^T{\widehat{\mathsf{\Sigma }}\mathrm{X}}^{-1}K=B$,

$$\left(\begin{array}{cc}{\mathrm{I}}_{\mathrm{n}}& B\\ {\mathrm{B}}^{\mathrm{T}}& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}{H}^T\widehat{\mathrm{r}}\\ K^T\mathrm{x}\end{array}\right)=\left(\begin{array}{c}{{H^T\mathrm{G}}_{11}}^{\mathrm{T}}{{\mathrm{U}}_1}^{\mathrm{T}}\mathrm{b}\\ 0\end{array}\right).$$

Finally, through a series of Givens transformations, the above system, whose coefficient matrix can be transformed into a $2n\times 2n$ symmetric indefinite tridiagonal matrix, can be solved by Gaussian partial principal component selection strategy.

To sum up, we call the above algorithm a Tikhonov regularized total least squares algorithm using GSVD technology. It is called Tikhonov regularized total least squares based on generalized singular value decomposition (GTRTLS algorithm for short).

Remark 1.

In order to overcome the ill-posedness, we can discard the element close to 0 of item $\Sigma$ in GSVD, that is, truncated GSVD (TGSVD)

$${\mathrm{A}}_{\mathrm{k}}=\mathrm{U}\left(\begin{array}{cc}{{\mathsf{\Sigma }}_{\mathrm{p}}}^{(\mathrm{n}-\mathrm{k})}& \\ & {\mathrm{I}}_{\mathrm{n}-\mathrm{p}}\end{array}\right){\mathrm{X}}^{-1}=\mathrm{U}\left(\begin{array}{cc}\begin{array}{cc}0& \\ & {\sigma }_{n-k+1}\end{array}& \\ & \begin{array}{ccc}\ddots & & \\ & {\sigma }_p& \\ & & {\mathrm{I}}_{\mathrm{n}-\mathrm{p}}\end{array}\end{array}\right){\mathrm{X}}^{-1}$$

and $L$ (see Equation (6)), where  ${{\Sigma }_p}^{(n-k)} (n-p\le k\le n)$ equals ${\Sigma }_p$ with the smallest$n-k$ ${\sigma }_i$’s being replaced by zeros. In TGSVD, the main information of the original system is retained by choosing the appropriate parameter k, and then the truncated system is obtained by the truncation regularization method. In other words, we combine truncated GSVD and TR to achieve a better regularization effect, which is called TGTRTLS; the expression is as follows:

$$\left({{\mathrm{A}}_{\mathrm{k}}}^{\mathrm{T}}{\mathrm{A}}_{\mathrm{k}}+{\mathsf{\lambda }}_{\mathrm{I}}{\mathrm{I}}_{\mathrm{n}}+{\mathsf{\lambda }}_{\mathrm{L}}{\mathrm{L}}^{\mathrm{T}}\mathrm{L}\right){\mathrm{x}}_{\mathrm{k},\mathsf{\lambda }}={{\mathrm{A}}_{\mathrm{k}}}^{\mathrm{T}}\mathrm{b}.$$

Remark 2.

According to Theorem 1, combined with Formula (10), the values of parameters ${\lambda }_I$ and ${\lambda }_L$ can be given more effectively. Statistical aspects of a negative regularization parameter in Tikhonov’s method are discussed in [7].

3. Randomized GTRTLS Algorithms

In recent years, there have been many research results of randomized algorithms [15,16]. In the truncated case, the randomized algorithm can take the subspace as a random sample and capture most of the information of the matrix, that is, a large-scale problem can be randomly projected into a smaller subspace and also contain its main information, and then some regularization methods are used to solve the small-scale problem. In particular, for severe ill-conditioned problems, we find that GSVD combined with the randomized algorithm is more effective than the classical GSVD method. The general idea is as follows: First of all, with high probability, one can select an orthonormal matrix $\mathrm{Q}\in {\mathbb{R}}^{\mathrm{m}\times \left(\mathrm{k}+\mathrm{s}\right)}$ such that $‖\mathrm{A}-\mathrm{Q}{\mathrm{Q}}^{\mathrm{T}}\mathrm{A}‖\le \mathrm{c}{\overline{\mathsf{\sigma }}}_{\mathrm{k}+1}$, where ${\overline{\mathsf{\sigma }}}_{\mathrm{k}+1}$ is the (k + 1)-th largest singular value of A, and c is a constant which depends on k and the oversampling parameter s. It satisfies that $\mathcal{R}\left({\mathrm{A}}^{\mathrm{T}}\mathrm{Q}\right)\subseteq \mathcal{R}\left({\mathrm{A}}^{\mathrm{T}}\right)$; here, $\mathcal{R}\left({\mathrm{A}}^{\mathrm{T}}\mathrm{Q}\right)$ is the approximate subspace spanned by the dominant right singular vectors of $\mathrm{A}$. Next, a matrix ${({\left({\mathrm{Q}}^{\mathrm{T}}\mathrm{A}\right)}^{\mathrm{T}},{\mathrm{L}}^{\mathrm{T}})}^{\mathrm{T}}$ with a small scaled is obtained which can be used to calculate the GSVD of ${({\mathrm{A}}^{\mathrm{T}},{\mathrm{L}}^{\mathrm{T}})}^{\mathrm{T}}$, approximately

$$\left(\begin{array}{c}\mathrm{A}\\ \mathrm{L}\end{array}\right)\approx \left(\begin{array}{c}\mathrm{Q}{\mathrm{Q}}^{\mathrm{T}}\mathrm{A}\\ \mathrm{L}\end{array}\right)=\left(\begin{array}{cc}\mathrm{U}& \\ & \mathrm{V}\end{array}\right)\left(\begin{array}{c}\mathrm{C}\\ \mathrm{S}\end{array}\right)\mathrm{Z}$$

where $U\in {\mathbb{R}}^{m\times l}$ and $V\in {\mathbb{R}}^{p\times p}$ are orthonormal, $Z=X^{-1}\in {\mathbb{R}}^{n\times n}$ is nonsingular, and $C\in {\mathbb{R}}^{l\times n}$ and $S\in {\mathbb{R}}^{p\times n}$ are rectangular diagonal matrices. Randomized sampling can be used to identify a subspace that captures most of the action of a matrix [15]. It provides us with an efficient way for truncation. A large-scale problem is projected randomly to a smaller subspace that contains the main information; then, the resulting small-scale problem can be solved by some regularization methods. Especially for severely ill-posed problems, randomized algorithms are much more efficient than the classical GSVD. So, the advantage of this algorithm is obvious when $m\gg n$. The detailed implementation process is shown in reference [16]. For the convenience of reading, we describe it as follows (Algorithm 1):

Algorithm 1. Randomized GSVD

Input: $\mathrm{A}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{n}}$, $\mathrm{L}\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{n}}$, where $\mathrm{n}-\mathrm{p}<l<min\{m,n\}$. Output: Orthonormal $\mathrm{U}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{l}}$ and $\mathrm{V}\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{p}}$, rectangular diagonal $\mathrm{C}\in {\mathbb{R}}^{\mathrm{l}\times \mathrm{n}}$ and $\mathrm{S}\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{n}}$ and nonsingular $\mathrm{Z}={\mathrm{X}}^{-1}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{n}}$.    1: Generate an $\mathrm{n}\times \mathrm{l}$ Gaussian random matrix $\mathsf{\Omega }$;    2: Form the $\mathrm{m}\times \mathrm{l}$ matrix $\mathrm{Y}=\mathrm{A}\mathsf{\Omega }$;    3: Compute the $\mathrm{m}\times \mathrm{l}$ orthonormal matrix $\mathrm{Q}$ via the QR factorization $\mathrm{Y}=\mathrm{Q}\mathrm{R}$;    4: Form the $\mathrm{l}\times \mathrm{n}$ matrix $\mathrm{B}={\mathrm{Q}}^{\mathrm{T}}\mathrm{A}$;    5: Compute the GSVD of $\{\mathrm{B},\mathrm{L}\}$ in (11): $\left(\begin{array}{c}\mathrm{B}\\ \mathrm{L}\end{array}\right)=\left(\begin{array}{cc}\mathrm{W}& \\ & \mathrm{V}\end{array}\right)\left(\begin{array}{c}\mathrm{C}\\ \mathrm{S}\end{array}\right){\mathrm{X}}^{-1}$;    6: Form the matrix $\mathrm{U}\in {\mathbb{R}}^{\mathrm{m}\times \mathrm{l}},\mathrm{U}=\mathrm{Q}\mathrm{W}$ and denote $\mathrm{Z}={\mathrm{X}}^{-1}$ in (12).

Now, we use randomized GSVD technology to deal with Tikhonov regularization TLS. In the first step, the approximate augmented system of augmented system (7) can be obtained by using randomized GSVD

$$\left(\begin{array}{ccc}{\mathrm{I}}_{\mathrm{m}}& 0& \mathrm{Q}{\mathrm{Q}}^{\mathrm{T}}\mathrm{A}\\ 0& {\mathrm{I}}_{\mathrm{p}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\mathrm{L}\\ {\mathrm{A}}^{\mathrm{T}}\mathrm{Q}{\mathrm{Q}}^{\mathrm{T}}& {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}{\mathrm{L}}^{\mathrm{T}}& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}\mathrm{r}\\ \mathrm{s}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}\mathrm{b}\\ 0\\ 0\end{array}\right),$$

(11)

and we have

$$\left(\begin{array}{ccc}{I}_l& 0& C{X}^{-1}\\ 0& I_p& {{\lambda }_L}^{1/2}S{X}^{-1}\\ X^{-T}{C}^T& {{\lambda }_L}^{1/2}{X}^{-T}{S}^T& {-{\lambda }_{I}I}_n\end{array}\right)\left(\begin{array}{c}{U}^{T}r\\ V^{T}s\\ x\end{array}\right)=\left(\begin{array}{c}{U}^{T}b\\ 0\\ 0\end{array}\right)$$

(12)

In the second step, we use Givens transformation to eliminate ${{\lambda }_L}^{1/2}\mathrm{S}$, which can be expressed as

$$\left(\begin{array}{c}\mathrm{C}\\ {{\mathsf{\lambda }}_{\mathrm{L}}}^{1/2}\mathrm{S}\end{array}\right)=\overline{\mathrm{G}}\left(\begin{array}{c}\overline{\mathsf{\Sigma }}\\ 0\end{array}\right)=\left(\begin{array}{ccc}{\overline{\mathrm{G}}}_{11}& {\overline{\mathrm{G}}}_{12}& {\overline{\mathrm{G}}}_{13}\\ {\overline{\mathrm{G}}}_{21}& {\overline{\mathrm{G}}}_{22}& {\overline{\mathrm{G}}}_{23}\\ {\overline{\mathrm{G}}}_{31}& {\overline{\mathrm{G}}}_{32}& {\overline{\mathrm{G}}}_{33}\end{array}\right)\left(\begin{array}{c}\overline{\mathsf{\Sigma }}\\ 0\end{array}\right),$$

where ${\overline{\mathrm{G}}}_{11}\in {\mathbb{R}}^{\mathrm{l}\times \mathrm{l}}$, ${\overline{\mathrm{G}}}_{22}\in {\mathbb{R}}^{(\mathrm{n}-\mathrm{l})\times (\mathrm{n}-\mathrm{l})}$, ${\overline{\mathrm{G}}}_{33}\in {\mathbb{R}}^{(\mathrm{l}+\mathrm{p}-\mathrm{n})\times (\mathrm{l}+\mathrm{p}-\mathrm{n})}$, $\overline{\mathsf{\Sigma }}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{n}}.$

Let $\mathrm{V}=\left(\begin{array}{cc}{\mathrm{V}}_1& {\mathrm{V}}_2\end{array}\right)$, ${\mathrm{V}}_1{\in \mathbb{R}}^{\mathrm{p}\times (\mathrm{n}-\mathrm{l})}$, ${\mathrm{V}}_2{\in \mathbb{R}}^{\mathrm{p}\times (\mathrm{l}+\mathrm{p}-\mathrm{n})}$; when $\overline{\mathrm{G}}$ is applied to the augmentation system, we can get

$$\left(\begin{array}{ccc}{\mathrm{I}}_{\mathrm{n}}& 0& \overline{\mathsf{\Sigma }}{\mathrm{X}}^{-1}\\ 0& {\mathrm{I}}_{\mathrm{l}+\mathrm{p}-\mathrm{n}}& 0\\ {\mathrm{X}}^{-\mathrm{T}}{\overline{\mathsf{\Sigma }}}^{\mathrm{T}}& 0& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}\left(\begin{array}{c}{\mathrm{U}}^{\mathrm{T}}\mathrm{r}\\ {{\mathrm{V}}_1}^{\mathrm{T}}\mathrm{s}\end{array}\right)\\ {{\mathrm{V}}_2}^{\mathrm{T}}\mathrm{s}\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}\left(\begin{array}{c}{{\overline{\mathrm{G}}}_{11}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\\ {{\overline{\mathrm{G}}}_{12}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\end{array}\right)\\ {{\overline{\mathrm{G}}}_{13}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\\ 0\end{array}\right)$$

(13)

The solution of ${{\mathrm{V}}_2}^{\mathrm{T}}\mathrm{s}$ can be obtained from the above equation, so only the following system can be considered:

$$\left(\begin{array}{cc}{\mathrm{I}}_{\mathrm{n}}& {\overline{\mathsf{\Sigma }}\mathrm{X}}^{-1}\\ {\mathrm{X}}^{-\mathrm{T}}{\overline{\mathsf{\Sigma }}}^{\mathrm{T}}& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}\left(\begin{array}{c}{\mathrm{U}}^{\mathrm{T}}\mathrm{r}\\ {{\mathrm{V}}_1}^{\mathrm{T}}\mathrm{s}\end{array}\right)\\ \mathrm{x}\end{array}\right)=\left(\begin{array}{c}\left(\begin{array}{c}{{\overline{\mathrm{G}}}_{11}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\\ {{\overline{\mathrm{G}}}_{12}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\end{array}\right)\\ 0\end{array}\right),$$

or

$$\left({\overline{\mathsf{\Sigma }}}^{\mathrm{T}}\overline{\mathsf{\Sigma }}+{\mathsf{\lambda }}_{\mathrm{I}}{\mathrm{X}}^{\mathrm{T}}\mathrm{X}\right){\mathrm{X}}^{-1}\mathrm{x}={\overline{\mathsf{\Sigma }}}^{\mathrm{T}}{{\overline{\mathrm{G}}}_{11}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}.$$

In the third step, ${\overline{\mathsf{\Sigma }}\mathrm{X}}^{-1}$ is reduced to bidiagonal matrix $\overline{B}$ by orthogonal transformation such that ${\overline{H}}^T{\overline{\mathsf{\Sigma }}\mathrm{X}}^{-1}\overline{K}=\overline{B}$,

$$\left(\begin{array}{cc}{\mathrm{I}}_{\mathrm{n}}& \overline{B}\\ {\overline{\mathrm{B}}}^{\mathrm{T}}& {-{\mathsf{\lambda }}_{\mathrm{I}}\mathrm{I}}_{\mathrm{n}}\end{array}\right)\left(\begin{array}{c}{\overline{H}}^T\widehat{\mathrm{r}}\\ {\overline{K}}^T\mathrm{x}\end{array}\right)=\left(\begin{array}{c}{{{\overline{H}}^T\mathrm{G}}_{11}}^{\mathrm{T}}{\mathrm{U}}^{\mathrm{T}}\mathrm{b}\\ 0\end{array}\right).$$

Finally, through a series of Givens transformations, the above system, whose coefficient matrix can be transformed into a $2n\times 2n$ symmetric indefinite tridiagonal matrix, can be solved by Gaussian partial principal component selection strategy.

To sum up, we call the above algorithm a Tikhonov regularized total least squares algorithm using randomized GSVD technology. It is called Tikhonov regularized total least squares based on randomized generalized singular value decomposition (RGTRTLS algorithm for short).

4. Error Analysis for Randomized Algorithms

First, we would like to review an important result of [16] regarding randomization algorithms.

Lemma 1

(see [15], Corollary 10.9). Suppose that $A\in {\mathbb{R}}^{m\times n}$ has the singular values ${\tilde{\sigma }}_1\ge {\tilde{\sigma }}_2\ge \cdots \ge {\tilde{\sigma }}_n$. Let $G$ be an $n\times ($ k+s) standard Gaussian matrix with k + s ≤ min{m, n} and s ≥ 4, and let Q be an orthonormal basis for the range of the sampled matrix AG. Then,

$$‖A-Q{Q}^{T}A‖\le \left(1+6\sqrt{(k+s)logs}\right){\tilde{\sigma }}_{k+1}+3\sqrt{(k+s)}\sqrt{\sum _{j>k}{\tilde{\sigma }}_j^2}$$

(14)

with a probability that is not less than $1-3s^{-s}$.

Next, a basic theory of perturbation analysis for TRTLS problems is needed.

Theorem 2

([10]). Consider the TRTLS problem (1) and assume that the genericity condition ${\sigma }_n$(A)$>{\sigma }_{n+1}\left(\right(A,b)$) holds. If ${‖(\delta A, \delta b)‖}_F$ is sufficiently small, then we find that

$$\frac{{‖\delta x‖}_2}{{‖x‖}_2}\le {\kappa }_A\frac{{‖\delta A‖}_2}{{‖A‖}_2}+{\kappa }_b\frac{{‖\delta b‖}_2}{{‖b‖}_2},$$

(15)

where

$${\kappa }_A=\frac{{‖A‖}_2}{{‖x‖}_2}\left({‖r‖}_2{‖{\mathcal{Z}}^{-1}\mathcal{K}‖}_2+{‖x‖}_2{‖{\mathcal{Z}}^{-1}\mathcal{K}{\mathrm{A}}^{\mathrm{T}}‖}_2\right), {\kappa }_b=\frac{{‖b‖}_2}{{‖x‖}_2}\left({‖{\mathcal{Z}}^{-1}\mathcal{K}{\mathrm{A}}^{\mathrm{T}}‖}_2\right),$$

with

$$\mathrm{r}=\mathrm{b}-\mathrm{A}\mathrm{x}, {\mathcal{Z}}^{-1}={\left({\mathrm{A}}^{\mathrm{T}}\mathrm{A}+{\mathsf{\lambda }}_{\mathrm{I}}{\mathrm{I}}_{\mathrm{n}}+{\mathsf{\lambda }}_{\mathrm{L}}{\mathrm{L}}^{\mathrm{T}}\mathrm{L}\right)}^{-1}, \mathcal{K}={\mathrm{I}}_{\mathrm{n}}+\frac{{\mathrm{L}}^{\mathrm{T}}\mathrm{L}\mathrm{x}{\mathrm{x}}^{\mathrm{T}}{\mathrm{L}}^{\mathrm{T}}\mathrm{L}{\mathcal{Z}}^{-1}}{{{\mathrm{x}}^{\mathrm{T}}\mathrm{L}}^{\mathrm{T}}\mathrm{L}{\mathcal{Z}}^{-1}{\mathrm{L}}^{\mathrm{T}}\mathrm{L}\mathrm{x}}.$$

Next, Lemma 1 is applied to the regularization system (1). Since the randomized system (11) can be seen as its perturbation, the following theorem is obtained from Theorem 2.

Theorem 3.

Let ${\tilde{\sigma }}_1\ge {\tilde{\sigma }}_2\ge \cdots \ge {\tilde{\sigma }}_n$ be the singular values of matrix A, and $\alpha =c/{‖A‖}_2$, c=$\left(1+6\sqrt{(k+s)logs}\right)+3\sqrt{(k+s)(n-k)}$, with the matrix  ${(A^T,L^T)}^T$as in (6). Suppose that Algorithm 1 is executed using the Gaussian matrix G to achieve GSVD approximation of matrix pairs  ${(A^T,L^T)}^T$. Assumption (5) is satisfied.  ${ x}_{trtls}$  is the solution of (1), and  ${ x}_{gtrtls}$  is the minimum two-norm solution of the problem  (11),  $\delta x={ x}_{trtls}$-${ x}_{gtrtls}$. Then we have

$$\frac{{‖\delta x‖}_2}{{‖{\mathrm{x}}_{\mathrm{t}\mathrm{r}\mathrm{t}\mathrm{l}\mathrm{s}}‖}_2}\le \alpha \left(\frac{{‖A‖}_2}{{‖{\mathrm{x}}_{\mathrm{t}\mathrm{r}\mathrm{t}\mathrm{l}\mathrm{s}}‖}_2}{‖r‖}_2+\frac{{‖b‖}_2+{‖A‖}_2{‖{\mathrm{x}}_{\mathrm{t}\mathrm{r}\mathrm{t}\mathrm{l}\mathrm{s}}‖}_2}{{‖{\mathrm{x}}_{\mathrm{t}\mathrm{r}\mathrm{t}\mathrm{l}\mathrm{s}}‖}_2}{‖{\mathcal{Z}}^{-1}\mathcal{K}{\mathrm{A}}^{\mathrm{T}}‖}_2\right){\tilde{\mathsf{\sigma }}}_{\mathrm{k}+1}+\mathcal{O}({\tilde{\mathsf{\sigma }}}_{\mathrm{k}+1})$$

with a probability greater than  $1-3s^{-s}$.

5. Numerical Examples

In this part, we illustrate the effectiveness and superiority of our methods through specific examples. We use the regularization tool package to perform the calculation on MATLAB R2016a.

Example 1.

The test problem is obtained by executing the function ilaplace (n, 2). The matrix A and the exact solution x are given such that ${‖A‖}_F={‖Ax‖}_2=1$, and the perturbed right-hand side is generated as $b=\left(A+\sigma {‖E‖}_F^{-1}E\right)x+\sigma {‖e‖}_2^{-1}e$, where the perturbations E and e are formed by a normal distribution with zero mean and unit standard deviation. L is the first derivative operator. The dimensions are $m=n=39$. Noise levels are taken as $\sigma =0.001$,  $\sigma =0.01$,  $\sigma =0.1 \sigma =1$.

We see that for small values of $\sigma$ and for the same value of ${\mathsf{\lambda }}_{\mathrm{L}}$, the three methods result in almost identical minimum relative errors. However, for a larger value of $\sigma$, the minimum relative errors of the GTRTLS method and the RGTRTLS method are significantly smaller than that of the TRTLS method, and they occur at smaller values of ${\mathsf{\lambda }}_{\mathrm{L}}$ as shown in

Table 1. Relative-error table of TRTLS method, GTRTLS method, and RGTRTLS method.

n = 39	$\mathit{\sigma }$	TRTLS	GTRTLS	RGTRTLS
ilaplace	0.001	0.1076	0.1043	0.1021
	0.01	0.1117	0.1078	0.1070
	0.1	0.1510	0.1240	0.1125
	1	0.2159	0.1719	0.1409

and Figure 1. So, the potential advantages of the GTRTLS method and the RGTRTLS method are shown.

We find that the calculation time of the RGTRTLS method is less than that of the GTRTLS method, and the GTRTLS method is less than that of the TRTLS method, as shown in

Table 2. Time comparison table of TRTLS method, GTRTLS method, and RGTRTLS method.

n = 39	$\mathit{\sigma }$	TRTLS	GTRTLS	RGTRTLS
ilaplace	0.001	0.006832	0.006000	0.004998
	0.01	0.005888	0.005001	0.004012
	0.1	0.005748	0.005003	0.003999
	1	0.005699	0.004999	0.004005

.

Example 2.

The test examples heat, phillips, baart, and shaw are from Hansen Tools [22]. Suppose$\mathit{\sigma }$is the relative noise level. We define

$$\overline{b}=\mathrm{b}+\mathsf{\sigma }‖\mathrm{b}‖\frac{\tau }{‖\tau ‖}$$

where  $\tau =2$  r  and  (m, 1)−1. It is easy to verify that  $‖\overline{b}-b‖$/$‖b‖=\sigma$. We set  $\sigma$  = 0.001  and the size n = 1024 in the tests. The matrix  $L$  is  $L_1$  , and the regularization parameter  ${\lambda }_L$  and  ${\lambda }_I$  are selected by  Remark 2.

For a better understanding of the tables below, we list the notation here:

• $x$ is the true solution of the TLS problem (1).
• $x_{gsvd}$ is the solution of (1) by classical GSVD.
• $x_{rgsvd}$ is the approximate regularized TLS solution in (11) by randomized algorithms.
• $E_{gsvd}=‖\mathrm{x}-x_{gsvd}‖/‖\mathrm{x}‖$, $E_{rgsvd}=‖\mathrm{x}-x_{rgsvd}‖/‖\mathrm{x}‖$ are the relative errors.
• The execution time (in seconds) of “GTRTLS” and “RGTRTLS” are $t_{gsvd}$ and $t_{rgsvd}$, respectively.

For n = 1024, the corresponding errors and time are shown in

Table 3. The comparison of GTRTLS method and RGTRTLS method.

n = 1024	heat	phillips	shaw	baart
$E_{gsvd}$	0.0565	0.0011	0.0258	0.0874
$E_{rgsvd}$	0.062	0.0041	0.0253	0.0896
$t_{gsvd}$	18.35	14.315	19.443	16.318
$t_{rgsvd}$	1.7946	1.2159	1.8324	1.5146

, and the performance is shown in Figure 2.

We apply the GTRTLS algorithm and the RGTRTLS algorithm to Example 2 and compare the errors and execution times. The randomized approach in Algorithm 1 still shows good performance in Table 3 and is competitive compared with the classical GSVD, judging from the errors $E_{gsvd}$ and $E_{rgsvd}$ and the execution times $t_{gsvd}$ and $t_{1gsvd}$.

We cannot solve large-scale or more complex ill-conditioned problems, such as n = 4096, using classical SVD or GSVD due to the high memory requirements. So, one can use preconditioned techniques first, and then use our method for computation.

6. Conclusions

In this paper, the generalized singular value decomposition technique is used to deal with Tikhonov regularized total least squares problems to approximate the true regularized TLS solutions, and the GTRTLS algorithm is proposed. The time complexity of the GTRTLS algorithm is better than TRTLS proposed by Golub, Hansen, and O’Leary. For medium- and large-scale problems, in order to reduce the storage requirements and accelerate the speed of GSVD, this paper adopts the random GSVD method and obtains the RGTRTLS algorithm. Numerical examples show that our algorithm has obvious effectiveness and superiority.