Acceleration of Approximate Matrix Multiplications on GPUs

Abstract

Matrix multiplication is important in various information-processing applications, including the computation of eigenvalues and eigenvectors, and in combinatorial optimization algorithms. Therefore, reducing the computation time of matrix products is essential to speed up scientific and practical calculations. Several approaches have been proposed to speed up this process, including GPUs, fast matrix multiplication libraries, custom hardware, and efficient approximate matrix multiplication (AMM) algorithms. However, research to date has yet to focus on accelerating AMMs for general matrices on GPUs, despite the potential of GPUs to perform fast and accurate matrix product calculations. In this paper, we propose a method for improving Monte Carlo AMMs. We also give an analytical solution for the optimal values of the hyperparameters in the proposed method. The proposed method improves the approximation of the matrix product without increasing the computation time compared to the conventional AMMs. It is also designed to work well with parallel operations on GPUs and can be incorporated into various algorithms. Finally, the proposed method is applied to a power method used for eigenvalue computation. We demonstrate that, on an NVIDIA A100 GPU, the computation time can be halved compared to the conventional power method using cuBLAS.

1. Introduction

Computing matrix products is a fundamental and essential operation in various information-processing tasks [1,2,3]. For example, one can obtain the largest eigenvalue and eigenvector of a positive definite matrix $\mathit{A}$ by repeating the following calculations:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\mathit{y}\hfill & \leftarrow \mathit{A}\mathit{x}\hfill \\ \mathit{x}\hfill & \leftarrow {\displaystyle \frac{\mathit{y}}{\parallel \mathit{y}\parallel }}\hfill \end{array}\right..\end{array}$$

(1)

In the power method, the computation of the matrix product is responsible for most of the computational burden. The same applies to optimization algorithms [4,5,6]. Therefore, it is crucial to reduce the computation time of matrix products to speed up various scientific and practical calculations.

A well-known method for accelerating matrix multiplication is to use GPUs. Other research has focused on implementing fast matrix multiplication libraries [7,8,9], designing custom hardware to accelerate specific classes of matrix multiplication [10,11,12], accelerating distributed matrix multiplication [13,14,15], designing exact and low-computation-order algorithms based on the divide-and-conquer method [16,17,18], and designing efficient approximate matrix multiplication (AMM) algorithms [19,20,21]. These areas are still active research fields. Recently, an AI-powered method that utilizes reinforcement learning to discover matrix arithmetic algorithms that are computationally less expensive than existing methods has been reported [22], building upon a previous study by Strassen [16].

For practical purposes, in some cases, the exact result of the matrix product may be optional. In this paper, we propose a method for such a case. It is known that when only an approximate result of the matrix product is required, the AMM methods can significantly reduce the computation time compared to other matrix acceleration approaches. MADDNESS, proposed in [20], provides high-speed approximate results. The paper demonstrated that MADDNESS, when executed on a CPU, can reduce the computation time by a factor of 100 compared to the exact matrix product while retaining sufficient computational accuracy to enable neural network inference. Allowing approximate calculations enables MADDNESS to achieve an overwhelming speed-up with other matrix optimization approaches. However, GPU computation can typically perform matrix products two orders of magnitude faster than CPU computation. Therefore, MADDNESS is practical for running neural networks in environments like the Internet of Things (IoT), where power consumption and installation space limitations make GPUs infeasible. On the other hand, if GPUs are available and there are no such limitations, obtaining the exact result by GPU computation rather than an approximate result by MADDNESS on a CPU is preferable.

A significant body of research proposing custom hardware exists to improve matrix multiplication performance. Hardware research often focuses on reducing energy consumption [23,24,25,26,27,28]. Existing hardware research has introduced various methods to reduce the energy consumption of artificial intelligence (AI) algorithms while maintaining acceptable accuracy. One approach involves approximate computing, which trades computational accuracy for improved energy efficiency. This strategy is particularly effective for AI algorithms that exhibit inherent error tolerance. Several papers have proposed using approximate multipliers, simplifying multiplication operations, and reducing energy consumption. Another approach involves using logarithmic multipliers, simplifying multiplication operations while sacrificing some degree of accuracy. This research significantly advances energy-efficient computing techniques tailored to AI algorithms, a central and pertinent topic within computer science. Our proposed method pertains to the approximate matrix multiplication algorithm and does not result in a reduction in the power consumption of a single FMA operation. However, it can potentially reduce the overall computation time of the application, thus making a valuable contribution to energy conservation.

To the best of our knowledge, no prior research other than the method for sparse matrices [29] has endeavored to accelerate approximate computations on GPUs, despite their capability for fast and precise computation of matrix products. Computations that can be efficiently accelerated by GPUs typically involve a large number of parallel executions of simple operations with regular memory access patterns. MADDNESS proposes an alternative approach to matrix multiplication by using hashing trees for table look-ups, which is considered unsuitable for GPU implementation. In addition, FD-AMM [30] incorporates the frequent direction algorithm [31] into AMM. However, the previous study [30] has reported that FD-AMM requires an SVD iteration to generate a sketch matrix whenever one of the multiplication matrices changes, resulting in longer computation times than existing methods. Thus, traditional AMM algorithms, which lack design considerations for parallel processing efficiency on GPUs, cannot guarantee the expected acceleration simply by implementing them on GPUs. Nevertheless, given that GPUs are already employed to expedite matrix products in numerous applications, it is also advantageous to leverage such devices for approximate calculations. Accordingly, we identified a known Monte Carlo AMM that can exploit the parallel computing capabilities of GPUs. Additionally, we devised an algorithm to enhance the approximation performance of this extant method while minimizing the increase in computational complexity. Our numerical experiments demonstrate that the proposed approach can expedite the computation of eigenvalues, which is crucial in both scientific and engineering domains.

2. Proposed Methods

2.1. Theory of Monte Carlo AMM

In this section, we present the theory behind the Monte Carlo method [32] used to compute the approximate matrix multiplication (AMM) of matrices $\mathit{A}\in {\mathbb{R}}^{m\times k}$ and $\mathit{B}\in {\mathbb{R}}^{k\times n}$. We denote the lth column and row of a real matrix $\mathit{M}$ by ${\mathit{M}}^{\left(l\right)}$ and ${\mathit{M}}_{\left(l\right)}$, respectively. The Frobenius norm of the matrix $\mathit{M}$ is denoted by $\parallel \mathit{M}\parallel$, defined as the square root of the sum of the squares of all elements in the matrix. Furthermore, the Euclidean norm of vector $\mathit{v}$ is denoted as $\left|\mathit{v}\right|$.

Algorithm 1 presents the pseudocode for the Monte Carlo AMM. It is a randomized algorithm that returns an approximate solution $\mathit{Z}$ that is close to the exact solution $\mathit{AB}$ with high probability. The expected square Frobenius error between $\mathit{AB}$ and $\mathit{Z}$ is evaluated using the following equation:

$$\begin{array}{c}\hfill \mathbb{E}\left[{\parallel \mathit{AB}-\mathit{Z}\parallel }^2\right]=\frac{1}{c}\left\{\sum _{l=1}^k\frac{{\left|{\mathit{A}}^{\left(l\right)}\right|}^2{\left|{\mathit{B}}_{\left(l\right)}\right|}^2}{p_l}-{\parallel \mathit{AB}\parallel }^2\right\}.\end{array}$$

(2)

Algorithm 1:  Monte Carlo AMM [32]

Require:  $\mathit{A}\in {\mathbb{R}}^{m\times k}$ and $\mathit{B}\in {\mathbb{R}}^{k\times n}$, and $c\in \mathbb{N}$

Ensure: $\mathit{Z}\in {\mathbb{R}}^{m\times n}$ such that $\parallel \mathit{AB}-\mathit{Z}\parallel$ is small

1:   $\mathit{Z}\leftarrow \mathit{O}$

2:  Set $p_1,\dots ,p_k$ such that ${\sum }_{l=1}^k{p}_l=1$

3:  for $t=1,\dots ,c$  do

4:        Pick $i_t$ with $\mathbb{P}[i_t=l]=p_l$

5:        $\mathit{Z}\leftarrow \mathit{Z}+{\displaystyle \frac{{\mathit{A}}^{\left(i_t\right)}{\mathit{B}}_{\left(i_t\right)}}{c{p}_{i_t}}}$

6:  end for

It has been proven that this equation is minimized when the probability $p_l$ of sampling the lth column/row is set to its optimal value, defined as

$$\begin{array}{c}\hfill p_l^{\mathrm{opt}}:=\frac{\left|{\mathit{A}}^{\left(l\right)}\right|\left|{\mathit{B}}_{\left(l\right)}\right|}{{\displaystyle \sum _{l^{\prime }=1}^k\left|{\mathit{A}}^{\left(l^{\prime }\right)}\right|\left|{\mathit{B}}_{\left(l^{\prime }\right)}\right|}}.\end{array}$$

(3)

In this case, the error is given by

$$\begin{array}{c}\hfill {\left.\mathbb{E}\left[{\parallel \mathit{AB}-\mathit{Z}\parallel }^2\right]\right|}_{p_l=p_l^{\mathrm{opt}}}=\frac{1}{c}\left\{{\left(\sum _{l=1}^k\left|{\mathit{A}}^{\left(l\right)}\right|\left|{\mathit{B}}_{\left(l\right)}\right|\right)}^2-{\parallel \mathit{AB}\parallel }^2\right\}.\end{array}$$

(4)

The statement that $\mathit{AB}={\sum }_{l=1}^k{\mathit{A}}^{\left(l\right)}{\mathit{B}}_{\left(l\right)}$ and Equation (3) imply that Algorithm 1 with the optimal Monte Carlo sampling is designed to assign a higher probability to columns/rows with larger Frobenius norms during sampling and to compute their product as an approximate solution.

2.2. AMM with Preprocessing: Proposed Methods

A straightforward method to reduce the expected squared error is to increase the number of sampling iterations c. The error decreases inversely with the number of iterations. However, increasing c results in a longer computation time for the Monte Carlo AMM. Therefore, in this paper, we propose a method to reduce the expected squared error without significantly increasing the computation time by further exploiting the information used in the conventional AMM.

Let ${\alpha }_i$ and ${\beta }_i$ be the mean of ${\mathit{A}}^{\left(i\right)}$ and ${\mathit{B}}_{\left(i\right)}$, respectively. We define the following two matrices

$$\begin{array}{c}\hfill \mathit{X}:=\mathit{A}-{\mathit{e}}_m{\left(\mathit{\alpha }+\mathit{x}\right)}^{\top },\mathit{Y}:=\mathit{B}-\left(\mathit{\beta }+\mathit{y}\right){\mathit{e}}_n^{\top },\end{array}$$

(5)

where $\mathit{\alpha }:=[{\alpha }_1,\dots ,{\alpha }_k{]}^{\top }$, $\mathit{\beta }:=[{\beta }_1,\dots ,{\beta }_k{]}^{\top }$, $\mathit{x}:=[x_1,\dots ,x_k{]}^{\top }$, $\mathit{y}:=[y_1,\dots ,y_k{]}^{\top }$, and ${\mathit{e}}_l$ is an l-dimensional vector whose elements are all one. These mean that we obtain $\mathit{X}$ and $\mathit{Y}$ by subtracting a constant value ${\alpha }_i+x_i$ from the ith column of $\mathit{A}$ and ${\beta }_i+y_i$ from the ith row of $\mathit{B}$. Then,

$$\begin{array}{cc}\hfill \mathit{AB}& =\left(\mathit{X}+{\mathit{e}}_m{\left(\mathit{\alpha }+\mathit{x}\right)}^{\top }\right)\left(\mathit{Y}+\left(\mathit{\beta }+\mathit{y}\right){\mathit{e}}_n^{\top }\right)\hfill \\ & \hfill =\mathit{XY}+\left(\mathit{X}\left(\mathit{\beta }+\mathit{y}\right)\right){\mathit{e}}_n^{\top }+{\mathit{e}}_m\left({\left(\mathit{\alpha }+\mathit{x}\right)}^{\top }\mathit{Y}\right)+{\mathit{e}}_m\left({\left(\mathit{\alpha }+\mathit{x}\right)}^{\top }\left(\mathit{\beta }+\mathit{y}\right)\right){\mathit{e}}_n^{\top }.\end{array}$$

(6)

The second, third, and fourth terms in Equation (6) can be computed using the matrix–vector and inner products instead of matrix–matrix products. By leveraging this fact, these terms can be accurately calculated with fewer computations for given matrices $\mathit{A}$, $\mathit{B}$, $\mathit{x}$, and $\mathit{y}$. Consequently, we utilize this approach to evaluate an AMM of matrices $\mathit{X}$ and $\mathit{Y}$ instead of directly using $\mathit{A}$ and $\mathit{B}$. Then, the expected error of this approximation relies solely on the error of $\mathit{XY}$. To simplify the notation, we define ${\mathit{A}}^{\prime }:=\mathit{A}-{\mathit{e}}_m{\mathit{\alpha }}^{\top }$ and ${\mathit{B}}^{\prime }:=\mathit{B}-\mathit{\beta }{\mathit{e}}_n^{\top }$. Notably, the matrices ${\mathit{A}}^{\prime }$ and ${\mathit{B}}^{\prime }$ are constant, and the crucial observation is that all column sums of ${\mathit{A}}^{\prime }$ and row sums of ${\mathit{B}}^{\prime }$ are equal to zero.

Algorithm 2 presents the pseudocode of our proposed method. The only differences between the proposed and conventional method lie in the second and third lines. The second line solely involves matrix–vector multiplication and inner product computation, which are less computationally intensive than matrix multiplication. Furthermore, the third line allows parallel execution for each pair of i and j. Consequently, the proposed method can be regarded as integrating the less computationally demanding preprocessing employed by the existing Monte Carlo AMM. This algorithm represents the calculation of an approximate solution to $\mathit{AB}$, as shown by the following calculation.

$$\begin{array}{cc}\hfill \mathit{AB}& ={\mathit{A}}^{\prime }{\mathit{B}}^{\prime }+\left({\mathit{A}}^{\prime }\mathit{\beta }\right){\mathit{e}}_n^{\top }+{\mathit{e}}_m\left({\mathit{\alpha }}^{\top }{\mathit{B}}^{\prime }\right)+{\mathit{e}}_m\left({\mathbf{\alpha }}^{\top }\mathit{\beta }\right){\mathit{e}}_n^{\top }\hfill \\ \hfill & ={\mathit{A}}^{\prime }{\mathit{B}}^{\prime }+\left(\mathit{A}\mathbf{\beta }\right){\mathit{e}}_n^{\top }+{\mathit{e}}_m\left({\mathbf{\alpha }}^{\top }\mathit{B}\right)-{\mathit{e}}_m\left({\mathbf{\alpha }}^{\top }\mathbf{\beta }\right){\mathit{e}}_n^{\top }\hfill \end{array}$$

(7)

Algorithm 2: Proposed method: Monte Carlo AMM with preprocessing

Require:  $\mathit{A}\in {\mathbb{R}}^{m\times k}$ and $\mathit{B}\in {\mathbb{R}}^{k\times n}$, and $c\in \mathbb{N}$

Ensure:  $\mathit{Z}\in {\mathbb{R}}^{m\times n}$, such that $\parallel \mathit{AB}-\mathit{Z}\parallel$ is small

1:  $\mathit{Z}\leftarrow \mathit{O}$

2:  ${\mathit{v}}_{\mathrm{c}}\leftarrow \mathit{A}\mathit{\beta }$, ${\mathit{v}}_{\mathrm{r}}\leftarrow {\mathbf{\alpha }}^{\top }\mathit{B}$, $w\leftarrow {\mathbf{\alpha }}^{\top }\mathit{\beta }$

3:  $Z_{ij}\leftarrow Z_{ij}-w+{\left[{\mathit{v}}_{\mathrm{c}}\right]}_i+{\left[{\mathit{v}}_{\mathrm{r}}\right]}_j$ for all $1\le i\le m$ and $1\le j\le n$

4:  Set $p_1,\dots ,p_k$ such that ${\sum }_{l=1}^k{p}_l=1$

5:  for  $t=1,\dots ,c$  do

6:        Pick $i_t$ with $\mathbb{P}[i_t=l]=p_l$

7:       $\mathit{Z}\leftarrow \mathit{Z}+{\displaystyle \frac{{\mathit{A}}^{\prime \left(i_t\right)}{\mathit{B}}_{\left(i_t\right)}^{\prime }}{c{p}_{i_t}}}$

8:  end for

Equation (7) can be regarded as the case of $\mathit{x}=\mathit{y}=\mathbf{0}$ in Equation (6). Remarkably, we can prove theoretically that the expected error by Algorithm 2 is equal to or less than the error by Algorithm 1. Considering the discussion on Equation (4), we see that the expected squared error for computing $\mathit{AB}$ using Equation (6) and the combination of AMM with exact computations is proportional to   

$$\begin{array}{cc}E:& ={\left(\sum _{l=1}^k\left|{\mathit{X}}^{\left(l\right)}\right|\left|{\mathit{Y}}_{\left(l\right)}^{\top }\right|\right)}^2-{∥\mathit{XY}∥}^2\hfill \\ & ={\left(\sum _{l=1}^k\left|{\mathit{A}}^{\prime \left(l\right)}-x_l{\mathit{e}}_m\right|\left|{\mathit{B}}_{\left(l\right)}^{\prime \top }-y_l{\mathit{e}}_n\right|\right)}^2-{∥\left({\mathit{A}}^{\prime }-{\mathit{e}}_m{\mathit{x}}^{\top }\right)\left({\mathit{B}}^{\prime }-\mathit{y}{\mathit{e}}_n^{\top }\right)∥}^2.\hfill \end{array}$$

(8)

The expected squared error can be represented by $E/c$. Hereafter, our objective is to establish the superior accuracy of the proposed method compared to the original Monte Carlo AMM. To achieve this, we aim to demonstrate that, through the presentation of several theorems, the global minimum of E occurs at $\mathit{x}=\mathit{y}=\mathbf{0}$.

Theorem 1.

Equation (8) is minimized at $x_i=0$ if all elements in ${\mathit{A}}^{\prime \left(i\right)}$ are zero.

Proof. 

Suppose that all elements of ${\mathit{A}}^{\prime \left(i\right)}$ are zero for a certain i. Let $\mathit{z}:=[z_1,\dots ,z_k{]}^{\top }$ be the vector, such that

$$\begin{array}{c}\hfill z_l=\left\{\begin{array}{cc}{x}_l\hfill & (l\ne i)\hfill \\ 0\hfill & (l=i)\hfill \end{array}\right..\end{array}$$

(9)

For simplicity, we introduce the following notations.

$$\begin{array}{cc}\hfill f_1(\mathit{x},\mathit{y})& :={\left(\sum _{l=1}^k\left|{\mathit{A}}^{\prime \left(l\right)}-x_l{\mathit{e}}_m\right|\left|{\mathit{B}}_{\left(l\right)}^{\prime \top }-y_l{\mathit{e}}_n\right|\right)}^2\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill f_2(\mathit{x},\mathit{y})& :=-{∥\left({\mathit{A}}^{\prime }-{\mathit{e}}_m{\mathit{x}}^{\top }\right)\left({\mathit{B}}^{\prime }-\mathit{y}{\mathit{e}}_n^{\top }\right)∥}^2\hfill \end{array}$$

(11)

Since we have $\left|{\mathit{A}}^{\prime \left(i\right)}-z_i{\mathit{e}}_m\right|=0$, the following equalities hold.

$$\begin{array}{cc}\hfill f_1(\mathit{x},\mathit{y})-f_1(\mathit{z},\mathit{y})& ={\left(\sum _l\left|{\mathit{X}}^{\left(l\right)}\right|\left|{\mathit{Y}}_{\left(l\right)}^{\top }\right|\right)}^2-{\left(\sum _{l\ne i}\left|{\mathit{X}}^{\left(l\right)}\right|\left|{\mathit{Y}}_{\left(l\right)}^{\top }\right|\right)}^2\hfill \\ \hfill & ={\left|{\mathit{X}}^{\left(i\right)}\right|}^2{\left|{\mathit{Y}}_{\left(i\right)}^{\top }\right|}^2+2\left|{\mathit{X}}^{\left(i\right)}\right|\left|{\mathit{Y}}_{\left(i\right)}^{\top }\right|\sum _{l\ne i}\left|{\mathit{X}}^{\left(l\right)}\right|\left|{\mathit{Y}}_{\left(l\right)}^{\top }\right|\hfill \end{array}$$

(12)

In addition, we have

$$\begin{array}{cc}\hfill f_2(\mathit{x},\mathit{y})-f_2(\mathit{z},\mathit{y})=& {∥\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)}∥}^2-{∥\sum _l{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)}∥}^2\hfill \\ \hfill =& {∥\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)}∥}^2-\left({∥\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)}∥}^2+{∥{\mathit{X}}^{\left(i\right)}{\mathit{Y}}_{\left(i\right)}∥}^2+2〈\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)},{\mathit{X}}^{\left(i\right)}{\mathit{Y}}_{\left(i\right)}〉\right)\hfill \\ \hfill =& -{∥{\mathit{X}}^{\left(i\right)}{\mathit{Y}}_{\left(i\right)}∥}^2-2〈\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)},{\mathit{X}}^{\left(i\right)}{\mathit{Y}}_{\left(i\right)}〉,\hfill \end{array}$$

(13)

where $\langle ·,·\rangle$ denotes the Frobenius inner product.

We can easily see that ${\left|\mathit{v}\right|}^2{\left|\mathit{w}\right|}^2={∥\mathit{v}{\mathit{w}}^{\top }∥}^2$ holds for vectors $\mathit{v}$ and $\mathit{w}$. Combining this equality, the Cauchy–Schwarz inequality, and the triangle inequality, we obtain

$$\begin{array}{cc}\hfill 〈\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)},{\mathit{X}}^{\left(i\right)}{\mathit{Y}}_{\left(i\right)}〉& \le ∥\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)}∥∥{\mathit{X}}^{\left(i\right)}{\mathit{Y}}_{\left(i\right)}∥\hfill \\ \hfill & =\left|{\mathit{X}}^{\left(i\right)}\right|\left|{\mathit{Y}}_{\left(i\right)}\right|∥\sum _{l\ne i}{\mathit{X}}^{\left(l\right)}{\mathit{Y}}_{\left(l\right)}∥\le \left|{\mathit{X}}^{\left(i\right)}\right|\left|{\mathit{Y}}_{\left(i\right)}\right|\sum _{l\ne i}\left|{\mathit{X}}^{\left(l\right)}\right|\left|{\mathit{Y}}_{\left(l\right)}\right|.\hfill \end{array}$$

(14)

From Equation (12), Equation (13), and Equation (14), we have

$$\begin{array}{c}\hfill \left\{f_1(\mathit{x},\mathit{y})+f_2(\mathit{x},\mathit{y})\right\}-\left\{f_1(\mathit{z},\mathit{y})+f_2(\mathit{z},\mathit{y})\right\}\ge 0.\end{array}$$

(15)

This inequality means that Equation (8) is minimized at $x_i=0$ if all elements of ${\mathit{A}}^{\prime \left(i\right)}$ are zero.    □

If all the elements in ${\mathit{A}}^{\left(i\right)}$ are identical, then according to the definitions of $\mathit{A}$ and ${\mathit{A}}^{\prime }$, all elements in ${\mathit{A}}^{\prime \left(i\right)}$ are also identical. Hence, applying Theorem 1, we can deduce that the value of $x_i$ yields the minimum value of E is 0, which implies ${\mathit{A}}^{\prime \left(i\right)}-x_i{\mathit{e}}_m=\mathbf{0}$. Combining this equation with Equation (8), we can obtain the values of $x_l$$(l\ne i)$, which minimize E by extracting the submatrices, respectively, after removing the ith column from ${\mathit{A}}^{\prime }$ and the ith row from ${\mathit{B}}^{\prime }$.

Therefore, to minimize E, we can eliminate the ith column of ${\mathit{A}}^{\prime }$ and the ith row of ${\mathit{B}}^{\prime }$ if all of their elements are identical. Subsequently, we can minimize the error E using the resulting submatrix, which can be obtained by removing identical elements, as presented in Equation (8). Hereafter, we denote the standard deviation of the elements in ${\mathit{A}}^{\prime \left(i\right)}$ and ${\mathit{B}}_{\left(i\right)}^{\prime }$ by ${\sigma }_i$ and ${\delta }_i$, respectively, with the assumption that ${\sigma }_i>0$ and ${\delta }_i>0$ for all i.

Theorem 2.

The following equations hold for any i.

$$\begin{array}{c}\hfill {\left.{\displaystyle \frac{\partial E}{\partial x_i}}\right|}_{\begin{array}{c}\mathit{x}=\mathbf{0}\\ \mathit{y}=\mathbf{0}\end{array}}=0,{\left.{\displaystyle \frac{\partial E}{\partial y_i}}\right|}_{\begin{array}{c}\mathit{x}=\mathbf{0}\\ \mathit{y}=\mathbf{0}\end{array}}=0.\end{array}$$

(16)

Proof. 

Let ${\left[\mathit{M}\right]}_{i,j}$ be the ith row and jth column element of matrix $\mathit{M}$. For matrices $\mathit{A}\in {\mathbb{R}}^{m\times k}$ and $\mathit{B}\in {\mathbb{R}}^{k\times n}$, we have the following equalities:

$$\begin{array}{cc}\hfill {\parallel \mathit{AB}\parallel }^2& =\sum _{i=1}^m\sum _{j=1}^n{\left(\sum _{l=1}^k{\left[\mathit{A}\right]}_{i,l}{\left[\mathit{B}\right]}_{l,j}\right)}^2\hfill \\ \hfill & =\sum _{i=1}^m\sum _{j=1}^n\sum _{l_1,l_2}{\left[\mathit{A}\right]}_{i,l_1}{\left[\mathit{A}\right]}_{i,l_2}{\left[\mathit{B}\right]}_{l_1,j}{\left[\mathit{B}\right]}_{l_2,j}\hfill \\ \hfill & =\sum _{l_1,l_2}\sum _{i=1}^m\sum _{j=1}^n{\left[\mathit{A}\right]}_{i,l_1}{\left[\mathit{A}\right]}_{i,l_2}{\left[\mathit{B}\right]}_{l_1,j}{\left[\mathit{B}\right]}_{l_2,j}\hfill \\ \hfill & =\sum _{l_1,l_2}\left(\sum _{i=1}^m{\left[\mathit{A}\right]}_{i,l_1}{\left[\mathit{A}\right]}_{i,l_2}\right)\left(\sum _{j=1}^n{\left[\mathit{B}\right]}_{l_1,j}{\left[\mathit{B}\right]}_{l_2,j}\right)=\sum _{l_1,l_2}\left({\mathit{A}}^{\left(l_1\right)\top }{\mathit{A}}^{\left(l_2\right)}\right)\left({\mathit{B}}_{\left(l_1\right)}{\mathit{B}}_{\left(l_2\right)}^{\top }\right)\hfill \end{array}$$

(17)

Using Equation (17), we can reformulate a term in Equation (8) as

$$\begin{array}{c}\hfill {∥\left({\mathit{A}}^{\prime }-{\mathit{e}}_m{\mathit{x}}^{\top }\right)\left({\mathit{B}}^{\prime }-\mathit{y}{\mathit{e}}_n^{\top }\right)∥}^2=\sum _{l_1,l_2}\left\{{\left({\mathit{A}}^{\prime \left(l_1\right)}-x_{l_1}{\mathit{e}}_m\right)}^{\top }\left({\mathit{A}}^{\prime \left(l_2\right)}-x_{l_2}{\mathit{e}}_m\right)\right\}\left\{\left({\mathit{B}}_{\left(l_1\right)}^{\prime }-y_{l_1}{\mathit{e}}_n^{\top }\right)\left({\mathit{B}}_{\left(l_2\right)}^{\prime \top }-y_{l_2}{\mathit{e}}_n\right)\right\}.\end{array}$$

(18)

Since the sum of each column of ${\mathit{A}}^{\prime }$ and each row of ${\mathit{B}}^{\prime }$ is zero, ${\mathit{A}}^{\prime \left(l\right)\top }{\mathit{e}}_m=0$ and ${\mathit{B}}_{\left(l\right)}^{\prime }{\mathit{e}}_n=0$ are valid for any l. Applying this to the equality above, we have

$$\begin{array}{cc}\hfill {∥\left({\mathit{A}}^{\prime }-{\mathit{e}}_m{\mathit{x}}^{\top }\right)\left({\mathit{B}}^{\prime }-\mathit{y}{\mathit{e}}_n^{\top }\right)∥}^2& =\sum _{l_1,l_2}\left({\mathit{A}}^{\prime \left(l_1\right)\top }{\mathit{A}}^{\prime \left(l_2\right)}+m{x}_{l_1}{x}_{l_2}\right)\left({\mathit{B}}_{\left(l_1\right)}^{\prime }{\mathit{B}}^{\prime \top }{}_{\left(l_2\right)}+n{y}_{l_1}{y}_{l_2}\right)\hfill \\ \hfill & =\parallel {\mathit{A}}^{\prime }{\mathit{B}}^{\prime }{\parallel }^2+{∥{\mathit{A}}^{\prime }\mathit{y}{\mathit{e}}_n^{\top }∥}^2+{∥{\mathit{B}}^{\prime \top }\mathit{x}{\mathit{e}}_m^{\top }∥}^2+{∥{\mathit{e}}_m{\mathit{x}}^{\top }\mathit{y}{\mathit{e}}_n^{\top }∥}^2\hfill \\ \hfill & =\parallel {\mathit{A}}^{\prime }{\mathit{B}}^{\prime }{\parallel }^2+n{\left|{\mathit{A}}^{\prime }\mathit{y}\right|}^2+m{\left|{\mathit{B}}^{\prime \top }\mathit{x}\right|}^2+mn{\left({\mathit{x}}^{\top }\mathit{y}\right)}^2.\hfill \end{array}$$

(19)

Since the average of elements in ${\mathit{A}}^{\prime \left(i\right)}$ is zero, we have ${\sigma }_i^2=\left({\sum }_l{\left[{\mathit{A}}^{\prime \left(i\right)}\right]}_l^2\right)/m$, where ${\left[\mathit{v}\right]}_i$ denotes the ith element of vector $\mathit{v}$. From this equality, we obtain

$$\begin{array}{c}\hfill \left|{\mathit{A}}^{\prime \left(i\right)}-x_i{\mathit{e}}_m\right|=\sqrt{{\sum }_{l=1}^m{\left({\left[{\mathit{A}}^{\prime \left(i\right)}\right]}_l-x_i\right)}^2}=\sqrt{{\sum }_{l=1}^m\left({\left[{\mathit{A}}^{\prime \left(i\right)}\right]}_l^2+x_i^2\right)}=\sqrt{m\left({\sigma }_i^2+x_i^2\right)}.\end{array}$$

(20)

In the same way, we have

$$\begin{array}{c}\hfill \left|{\mathit{B}}_{\left(i\right)}^{\prime \top }-y_i{\mathit{e}}_n\right|=\sqrt{n\left({\delta }_i^2+y_i^2\right)}.\end{array}$$

(21)

Substituting Equation (19), Equation (20), and Equation (21) into Equation (8), we obtain

$$\begin{array}{c}\hfill E=mn{\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)}^2-{\parallel {\mathit{A}}^{\prime }{\mathit{B}}^{\prime }\parallel }^2-n{\left|{\mathit{A}}^{\prime }\mathit{y}\right|}^2-m{\left|{\mathit{B}}^{\prime \top }\mathit{x}\right|}^2-mn{\left({\mathit{x}}^{\top }\mathit{y}\right)}^2.\end{array}$$

(22)

Because we have ${\left|{\mathit{B}}^{\prime \top }\mathit{x}\right|}^2={\mathit{x}}^{\top }{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\mathit{x}={\sum }_{l_1,l_2}{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{l_1,l_2}{x}_{l_1}{x}_{l_2}$, the following holds.

$$\begin{array}{c}\hfill \frac{\partial {\left|{\mathit{B}}^{\prime \top }\mathit{x}\right|}^2}{\partial x_i}=2\sum _j{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,j}{x}_j=2{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\mathit{x}\right]}_i\end{array}$$

(23)

Thus,

$$\begin{array}{cc}\hfill \frac{\partial E}{\partial x_i}& =2mn{x}_i\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}}-2m{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\mathit{x}\right]}_i-2mn{y}_i{\mathit{x}}^{\top }\mathit{y}.\hfill \end{array}$$

(24)

In the same way, we have

$$\begin{array}{cc}\hfill \frac{\partial E}{\partial y_i}& =2mn{y}_i\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)\sqrt{\frac{{\sigma }_i^2+x_i^2}{{\delta }_i^2+y_i^2}}-2n{\left[{\mathit{A}}^{\prime \top }{\mathit{A}}^{\prime }\mathit{y}\right]}_i-2mn{x}_i{\mathit{x}}^{\top }\mathit{y}.\hfill \end{array}$$

(25)

By combining Equation (24) with Equation (25), we can derive Equation (16).    □

Theorem 3.

The largest eigenvalue of $\mathit{M}{\mathit{M}}^{\top }$ is less than or equal to ${∥\mathit{M}∥}^2$ for any real nonzero matrix $\mathit{M}$. The equality holds if the rank of $\mathit{M}$ is one.

Proof. 

Let ${\lambda }_i$ be the ith largest eigenvalue of $\mathit{M}{\mathit{M}}^{\top }$. Since $\mathit{M}{\mathit{M}}^{\top }$ is positive semidefinite, all eigenvalues of $\mathit{M}{\mathit{M}}^{\top }$ are non-negative, and we have ${∥\mathit{M}∥}^2=\mathrm{Tr}\left(\mathit{M}{\mathit{M}}^{\top }\right)={\sum }_i{\lambda }_i\ge {\lambda }_1$. Therefore, the largest eigenvalue of $\mathit{M}{\mathit{M}}^{\top }$ is less than or equal to ${∥\mathit{M}∥}^2$.

The equality holds if and only if all eigenvalues other than the largest one are zero, which means that the rank of $\mathit{M}{\mathit{M}}^{\top }$ is one. Hereafter, we assume that the rank of $\mathit{M}{\mathit{M}}^{\top }$ is one. Let r be the rank of $\mathit{M}\in {\mathbb{R}}^{m\times n}$. We then have the singular value decomposition (SVD) of $\mathit{M}$ as $\mathit{M}\mathit{U}\Sigma {\mathit{V}}^{\top }$, where $\mathit{U}\in {\mathbb{R}}^{m\times r},\Sigma \in {\mathbb{R}}^{r\times r}$, and $\mathit{V}\in {\mathbb{R}}^{n\times r}$. Because $\mathit{V}$ is an orthogonal matrix and $\Sigma$ is a diagonal matrix, $\mathit{M}{\mathit{M}}^{\top }=\mathit{U}{\Sigma }^2{\mathit{U}}^{\top }$ holds. This is nothing but the SVD of $\mathit{M}{\mathit{M}}^{\top }$, and it implies that the rank of $\mathit{M}{\mathit{M}}^{\top }$ is r as well. Since the rank of $\mathit{M}{\mathit{M}}^{\top }$ is one, we have $r=1$; thus, the rank of $\mathit{M}$ is one.    □

Theorem 4.

The following equation holds for any combination of i and j.

$$\begin{array}{c}\hfill {\left.{\displaystyle \frac{{\partial }^{2}E}{\partial x_i\partial y_j}}\right|}_{\begin{array}{c}\mathit{x}=\mathbf{0}\\ \mathit{y}=\mathbf{0}\end{array}}=\mathbf{0}.\end{array}$$

(26)

Proof. 

From Equation (24), we have

$$\begin{array}{cc}\hfill \frac{{\partial }^{2}E}{\partial x_i\partial y_i}& =\frac{2mn{x}_i{y}_i}{\sqrt{\left({\sigma }_i^2+x_i^2\right)\left({\delta }_i^2+y_i^2\right)}}\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)-2mn{\mathit{x}}^{\top }\mathit{y}.\hfill \end{array}$$

(27)

Furthermore,

$$\begin{array}{cc}\hfill \frac{{\partial }^{2}E}{\partial x_i\partial y_j}& =2mn{x}_i{y}_j\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}·\frac{{\sigma }_j^2+x_j^2}{{\delta }_j^2+y_j^2}}-2mn{x}_j{y}_i\hfill \end{array}$$

(28)

holds for any $i\ne j$. Therefore, we have Equation (26).    □

Theorem 5.

The Hessian matrices of E, regarding $\mathit{x}$ and $\mathit{y}$, and ${\nabla }_{xx}^{2}E$ and ${\nabla }_{yy}^{2}E$, are positive semidefinite at $\mathit{x}=\mathit{y}=\mathbf{0}$.

Proof. 

We define $f\left(x_i\right){\sum }_{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}$ for simplicity. Then, we have

$$\begin{array}{cc}\hfill \frac{\partial }{\partial x_i}\left\{x_{i}f\left(x_i\right)\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}}\right\}& =f\left(x_i\right)\frac{\partial }{\partial x_i}\left(x_i\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}}\right)+x_i\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}}\frac{\partial f\left(x_i\right)}{\partial x_i}\hfill \\ \hfill & =f\left(x_i\right)\frac{{\sigma }_i^2\sqrt{{\delta }_i^2+y_i^2}}{{\left({\sigma }_i^2+x_i^2\right)}^{\frac{3}{2}}}+\frac{x_i^2({\delta }_i^2+y_i^2)}{{\sigma }_i^2+x_i^2}.\hfill \end{array}$$

(29)

Moreover, the following equalities hold.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\displaystyle \frac{\partial }{\partial x_i}{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\mathit{x}\right]}_i=\frac{\partial }{\partial x_i}\sum _j{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,j}{x}_j={\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,i}}\hfill \\ {\displaystyle \frac{\partial }{\partial x_i}\left(y_i{\mathit{x}}^{\top }\mathit{y}\right)=y_i^2}\hfill \end{array}\right.\end{array}$$

(30)

Substituting these equalities into Equation (24), we can simplify ${\partial }^{2}E/\partial x_i^2$ as

$$\begin{array}{c}\hfill \frac{2mn{\sigma }_i^2\sqrt{{\delta }_i^2+y_i^2}}{{\left({\sigma }_i^2+x_i^2\right)}^{\frac{3}{2}}}\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)+\frac{2mn{x}_i^2\left({\delta }_i^2+y_i^2\right)}{{\sigma }_i^2+x_i^2}-2m{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,i}-2mn{y}_i^2.\end{array}$$

(31)

Since ${\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,i}$ is equal to the square of the Euclidean length of ${\mathit{B}}_{\left(i\right)}^{\prime }$, we have ${\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,i}=n{\delta }_i^2$. Substituting this into Equation (31), we obtain

$$\begin{array}{c}\hfill \frac{{\partial }^{2}E}{\partial x_i^2}=\frac{2mn{\sigma }_i^2}{{\sigma }_i^2+x_i^2}\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}}\sum _{l\ne i}\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}.\end{array}$$

(32)

This equation implies that

$$\begin{array}{c}\hfill {\left.\frac{{\partial }^{2}E}{\partial x_i^2}\right|}_{\begin{array}{c}\mathit{x}=\mathbf{0}\\ \mathit{y}=\mathbf{0}\end{array}}=\frac{2mn{\delta }_i}{{\sigma }_i}\sum _{l\ne i}{\sigma }_l{\delta }_l.\end{array}$$

(33)

For $i\ne j$, we can simplify ${\partial }^{2}E/\partial x_i\partial x_j$ as

$$\begin{array}{c}\hfill 2mn\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}·\frac{{\delta }_j^2+y_j^2}{{\sigma }_j^2+x_j^2}}{x}_i{x}_j-2m{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,j}-2mn{y}_i{y}_j.\end{array}$$

(34)

This equation implies that

$$\begin{array}{c}\hfill {\left.{\displaystyle \frac{{\partial }^{2}E}{\partial x_i\partial x_j}}\right|}_{\begin{array}{c}\mathit{x}=\mathbf{0}\\ \mathit{y}=\mathbf{0}\end{array}}=-2m{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,j}.\end{array}$$

(35)

Combining Equations (33) and (35), we can express the Hessian matrix of E regarding $\mathit{x}$ at $\mathit{x}=\mathit{y}=\mathbf{0}$ as follows:

$$\begin{array}{cc}\hfill {\left.{\nabla }_{xx}^{2}E\right|}_{\begin{array}{c}\mathit{x}=\mathbf{0}\\ \mathit{y}=\mathbf{0}\end{array}}:=\left[\begin{array}{cccc}\frac{{\partial }^{2}E}{\partial x_1^2}& \frac{{\partial }^{2}E}{\partial x_1\partial x_2}& \dots & \frac{{\partial }^{2}E}{\partial x_1\partial x_k}\\ \frac{{\partial }^{2}E}{\partial x_1\partial x_2}& \frac{{\partial }^{2}E}{\partial x_2^2}& \dots & \frac{{\partial }^{2}E}{\partial x_2\partial x_k}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}E}{\partial x_1\partial x_k}& \frac{{\partial }^{2}E}{\partial x_2\partial x_k}& \dots & \frac{{\partial }^{2}E}{\partial x_k^2}\end{array}\right]& =2m\left\{n\left(\sum _{l=1}^k{\sigma }_l{\delta }_l\right)\left[\begin{array}{ccc}\frac{{\delta }_1}{{\sigma }_1}& & \mathit{O}\\ & \ddots & \\ \hfill \mathit{O}& \hfill & \hfill \frac{{\delta }_k}{{\sigma }_k}\end{array}\right]-{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right\}\hfill \end{array}$$

(36)

Let $\mathit{Z}$ be a diagonal and regular matrix $\mathit{Z}\mathrm{diag}(\sqrt{{\delta }_1/{\sigma }_1},\dots ,\sqrt{{\delta }_k/{\sigma }_k})$. From Theorem 3, the largest eigenvalue of ${\mathit{Z}}^{-1}{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }{\left({\mathit{Z}}^{-1}\right)}^{\top }$ is less than or equal to ${∥{\mathit{Z}}^{-1}{\mathit{B}}^{\prime }∥}^2={\sum }_{l=1}^k{∥\sqrt{{\sigma }_l/{\delta }_l}{\mathit{B}}_{\left(l\right)}^{\prime }∥}^2=n{\sum }_{l=1}^k{\sigma }_l{\delta }_l.$ Thus,

$$\begin{array}{c}\hfill n\left(\sum _{l=1}^k{\sigma }_l{\delta }_l\right)\mathit{I}-{\mathit{Z}}^{-1}{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }{\left({\mathit{Z}}^{-1}\right)}^{\top }⪰\mathit{O},\end{array}$$

(37)

where $\mathit{I}$ is an identity matrix. Therefore, $n\mathit{Z}\left({\sum }_{l=1}^k{\sigma }_l{\delta }_l\right)\mathit{Z}-{\mathit{B}}^{\prime }{\mathit{B}}^{\top }⪰\mathit{O}$ holds, which shows that ${\nabla }_{\mathit{xx}}^{2}E$ is positive semidefinite at $\mathit{x}=\mathit{y}=\mathbf{0}$. In the same way, we can prove that ${\nabla }_{\mathit{yy}}^{2}E$ is positive semidefinite at $\mathit{x}=\mathit{y}=\mathbf{0}$.    □

Theorem 6.

The function E is locally optimal at $\mathit{x}=\mathit{y}=\mathbf{0}$.

Proof. 

The Hessian matrix of E can be expressed as

$$\begin{array}{c}\hfill {\nabla }^{2}E:=\left[\begin{array}{cc}{\nabla }_{\mathit{xx}}^{2}E& {\nabla }_{\mathit{xy}}^{2}E\\ {\nabla }_{\mathit{yx}}^{2}E& {\nabla }_{\mathit{yy}}^{2}E\end{array}\right].\end{array}$$

(38)

Theorem 4 provides ${\nabla }_{\mathit{xy}}^{2}E={\nabla }_{\mathit{yx}}^{2}E=\mathit{O}$ at $\mathit{x}=\mathit{y}=\mathbf{0}$. Furthermore, Theorem 5 claims that ${\nabla }_{\mathit{xx}}^{2}E$ and ${\nabla }_{\mathit{yy}}^{2}E$ are positive semidefinite at $\mathit{x}=\mathit{y}=\mathbf{0}$, which means ${\nabla }^{2}E$ is also positive semidefinite at $\mathit{x}=\mathit{y}=\mathbf{0}$. Based on the positive semidefinite property and Theorem 2, it is apparent that the point $\mathit{x}=\mathit{y}=\mathbf{0}$ serves as a locally optimal solution for the minimization of E.    □

The preceding discussion demonstrates that the point $\mathit{x}=\mathit{y}=\mathbf{0}$ constitutes a local optimum of E. Hereafter, we establish that, in almost all cases, this point is a global optimum because E has only one local optimum, and, in other cases, the point is globally optimal as well because all local optima are connected.

Theorem 7.

Let $\mathit{M}$ be a rank-1 matrix and $\mathit{v}$ be a vector satisfying $\mathit{M}{\mathit{M}}^{\top }=\mathit{v}{\mathit{v}}^{\top }$. The vector $\mathit{v}$ is the only eigenvector of $\mathit{M}{\mathit{M}}^{\top }$. The matrix ${∥\mathit{M}∥}^2\mathit{I}-\mathit{M}{\mathit{M}}^{\top }$ has an eigenvalue of 0, and only a scaled version of $\mathit{v}$ is its eigenvector.

Proof. 

As ${∥\mathit{M}∥}^2=\mathrm{Tr}\left(\mathit{M}{\mathit{M}}^{\top }\right)=\mathrm{Tr}\left(\mathit{v}{\mathit{v}}^{\top }\right)={\left|\mathit{v}\right|}^2$ holds, we have $\left({∥\mathit{M}∥}^2\mathit{I}-\mathit{M}{\mathit{M}}^{\top }\right)\mathit{v}={\left|\mathit{v}\right|}^2\mathit{v}-\mathit{v}{\mathit{v}}^{\top }\mathit{v}=\mathbf{0}$. Since $\mathit{M}$ is a rank-1 matrix, $\mathit{M}{\mathit{M}}^{\top }$ is a rank-1 matrix as well and has only one eigenvalue. Combining this and ${∥\mathit{M}∥}^2\mathit{v}=\mathit{M}{\mathit{M}}^{\top }\mathit{v}$, we can see that $\mathit{v}$ is the unique eigenvector of $\mathit{M}{\mathit{M}}^{\top }$. Moreover, based on the equality above, the matrix ${∥\mathit{M}∥}^2\mathit{I}-\mathit{M}{\mathit{M}}^{\top }$ has an eigenvalue of 0, and only a scaled version of $\mathit{v}$ becomes its eigenvector.    □

Theorem 8.

Let $\mathit{M}$ be a rank-1 matrix, and $\mathit{v}$ be a vector satisfying $\mathit{M}{\mathit{M}}^{\top }=\mathit{v}{\mathit{v}}^{\top }$. Let $\mathit{Z}$ be a regular and diagonal matrix. The vectors $\mathit{x}$ that satisfy ${\mathit{x}}^{\top }\left({∥{\mathit{Z}}^{-1}\mathit{M}∥}^2{\mathit{Z}}^2-\mathit{M}{\mathit{M}}^{\top }\right)\mathit{x}=0$ are scaled versions of ${\mathit{Z}}^{-2}\mathit{v}$.

Proof. 

Since $\mathit{M}$ is a rank-1 matrix and $\mathit{Z}$ is regular, ${\mathit{Z}}^{-1}\mathit{M}$ is a rank-1 matrix as well. Let $\mathit{Q}$ be ${∥{\mathit{Z}}^{-1}\mathit{M}∥}^2\mathit{I}-\left({\mathit{Z}}^{-1}\mathit{M}\right){\left({\mathit{Z}}^{-1}\mathit{M}\right)}^{\top }$, and $\tilde{\mathit{v}}$ be a vector satisfying $\tilde{\mathit{v}}{\tilde{\mathit{v}}}^{\top }=\left({\mathit{Z}}^{-1}\mathit{M}\right){\left({\mathit{Z}}^{-1}\mathit{M}\right)}^{\top }$. Theorem 7 implies that $\mathit{Q}$ has an eigenvalue of 0 and only a scaled version of $\tilde{\mathit{v}}$ becomes its eigenvector. Because $\left(\mathit{Z}\tilde{\mathit{v}}\right){\left(\mathit{Z}\tilde{\mathit{v}}\right)}^{\top }=\mathit{M}{\mathit{M}}^{\top }$ holds, we have $\mathit{v}=\mathit{Z}\tilde{\mathit{v}}$. Thus, a scaled version of ${\mathit{Z}}^{-1}\mathit{v}$ is an eigenvector of $\mathit{Q}$ corresponding to an eigenvalue of 0. Therefore, the following equality holds for any real value of c.

$$\begin{array}{c}\hfill \mathit{Q}\left(c{\mathit{Z}}^{-1}\mathit{v}\right)=\mathbf{0}\end{array}$$

(39)

We can easily see that $\mathit{ZQZ}\left(c{\mathit{Z}}^{-2}\mathit{v}\right)=\mathbf{0}$ holds, and it implies that the matrix $\mathit{ZQZ}$ has the eigenvalue of 0 and only a scaled version of ${\mathit{Z}}^{-2}\mathit{v}$ is one of its eigenvectors. Moreover, we have the following equalities:

$$\begin{array}{c}\hfill \mathit{ZQZ}=\mathit{Z}\left({∥{\mathit{Z}}^{-1}\mathit{M}∥}^2\mathit{I}-{\mathit{Z}}^{-1}\mathit{M}{\mathit{M}}^{\top }{\left({\mathit{Z}}^{-1}\right)}^{\top }\right)\mathit{Z}={∥{\mathit{Z}}^{-1}\mathit{M}∥}^2{\mathit{Z}}^2-\mathit{M}{\mathit{M}}^{\top }\end{array}$$

(40)

Therefore, the matrix ${∥{\mathit{Z}}^{-1}\mathit{M}∥}^2{\mathit{Z}}^2-\mathit{M}{\mathit{M}}^{\top }$ has an eigenvalue of 0, and a scaled version of ${\mathit{Z}}^{-2}\mathit{v}$ is one of its eigenvectors.

Theorem 3 claims that $\mathit{Q}$ is positive semidefinite, and it implies that $\mathit{ZQZ}$ is positive semidefinite as well. Thus, the vector $\mathit{x}$ satisfying ${\mathit{x}}^{\top }\left({∥{\mathit{Z}}^{-1}\mathit{M}∥}^2{\mathit{Z}}^2-\mathit{M}{\mathit{M}}^{\top }\right)\mathit{x}=0$ is zero vector or a scaled version of the eigenvector corresponding to the eigenvalue of 0. Therefore, $\mathit{x}$ are scaled versions of ${\mathit{Z}}^{-2}\mathit{v}$.    □

Theorem 9.

The function E is globally optimal at  x = y = 0.

Proof. 

From Equation (24), we obtain ∇_xE as

$$\begin{array}{c}\hfill 2mn\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)\left[\begin{array}{ccc}\sqrt{\frac{{\delta }_1^2+y_1^2}{{\sigma }_1^2+x_1^2}}& & \mathit{O}\\ & \ddots & \\ \mathit{O}& & \sqrt{\frac{{\delta }_k^2+y_k^2}{{\sigma }_k^2+x_k^2}}\end{array}\right]\mathit{x}-2m{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\mathit{x}-2mn\left({\mathit{x}}^{\top }\mathit{y}\right)\mathit{y}.\end{array}$$

(41)

This implies that

$$\begin{array}{cc}\hfill \frac{{\mathit{x}}^{\top }{\nabla }_{\mathit{x}}E}{2m}& ={\mathit{x}}^{\top }\left\{n\left(\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)\left[\begin{array}{ccc}\sqrt{\frac{{\delta }_1^2+y_1^2}{{\sigma }_1^2+x_1^2}}& & \mathit{O}\\ & \ddots & \\ \mathit{O}& & \sqrt{\frac{{\delta }_k^2+y_k^2}{{\sigma }_k^2+x_k^2}}\end{array}\right]-\left({\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }+n\mathit{y}{\mathit{y}}^{\top }\right)\right\}\mathit{x}.\hfill \end{array}$$

(42)

We study the conditions that $\mathit{x}$ must satisfy in the local solution of E. In that case, ${\mathit{x}}^{\top }{\nabla }_{\mathit{x}}E$ must be zero due to ${\nabla }_{\mathit{x}}E=\mathbf{0}$.

Let ${\mathit{B}}^{\prime \prime }$ be ${\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime },\sqrt{n}\mathit{y}\in {\mathbb{R}}^{k\times (m+1)}$ and ${\mathit{Z}}^{\prime }$ be a diagonal and regular matrix

$$\begin{array}{c}\hfill {\mathit{Z}}^{\prime }:=\mathrm{diag}\left({\left(\frac{{\delta }_1^2+y_1^2}{{\sigma }_1^2+x_1^2}\right)}^{\frac{1}{4}},\dots ,{\left(\frac{{\delta }_k^2+y_k^2}{{\sigma }_k^2+x_k^2}\right)}^{\frac{1}{4}}\right).\end{array}$$

(43)

From Theorem 3, the largest eigenvalue of ${\mathit{Z}}^{\prime -1}{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }{\left({\mathit{Z}}^{\prime -1}\right)}^{\top }$ is less than or equal to

$$\begin{array}{cc}\hfill {∥{\mathit{Z}}^{\prime -1}{\mathit{B}}^{\prime \prime }∥}^2& =\sum _{l=1}^k{\left|{\left(\frac{{\sigma }_l^2+x_l^2}{{\delta }_l^2+y_l^2}\right)}^{\frac{1}{4}}{\mathit{B}}_{\left(l\right)}^{\prime \prime }\right|}^2\hfill \\ \hfill & =\sum _{l=1}^k\sqrt{\frac{{\sigma }_l^2+x_l^2}{{\delta }_l^2+y_l^2}}{\left|\left[\begin{array}{c}{\mathit{B}}_{\left(l\right)}^{\prime \top }\\ \sqrt{n}{y}_l\end{array}\right]\right|}^2\hfill \\ \hfill & =\sum _{l=1}^k\sqrt{\frac{{\sigma }_l^2+x_l^2}{{\delta }_l^2+y_l^2}}n\left({\delta }_l^2+y_l^2\right)=n\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}.\hfill \end{array}$$

(44)

Thus, we have

$$\begin{array}{c}\hfill n\sum _{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}{\mathit{I}}_k-{\mathit{Z}}^{\prime -1}{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }{\left({\mathit{Z}}^{\prime -1}\right)}^{\top }⪰\mathit{O}.\end{array}$$

(45)

In particular, "≻" holds if $\mathrm{rank}\left({\mathit{B}}^{\prime \prime }\right)>1$.

(i)

$\mathrm{rank}\left({\mathit{B}}^{\prime }\right)>1$, or $\mathrm{rank}\left({\mathit{B}}^{\prime }\right)=1$ and $\mathit{y}$ is not a scaled version of a column vector of ${\mathit{B}}^{\prime }$

Because the rank of ${\mathit{B}}^{\prime \prime }$ is greater than 1, we have $n{\sum }_{l=1}^k\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}{\mathit{Z}}^{\prime }{\mathit{Z}}^{\prime \top }-{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }\succ \mathit{O}$. Combining this inequality and Equation (42), we see that ${\mathit{x}}^{\top }{\nabla }_{\mathit{x}}E=0$ holds at $\mathit{x}=\mathbf{0}$ only. Therefore, if $\mathit{x}$ is a local minimum of E, then ${\nabla }_{\mathit{x}}E=\mathbf{0}$ holds and we have $\mathit{x}=\mathbf{0}$. Since a local minimum is limited to $\mathit{x}=\mathbf{0}$, it also becomes the global minimum.

(ii)

$\mathrm{rank}\left({\mathit{B}}^{\prime }\right)=1$ and $\mathit{y}$ is a scaled version of a column vector of ${\mathit{B}}^{\prime }$

All rows of ${\mathit{B}}^{\prime }$ are dependent due to $\mathrm{rank}\left({\mathit{B}}^{\prime }\right)=1$. As $\mathit{y}$ is a scaled version of a column vector of ${\mathit{B}}^{\prime }$, we can express ${\mathit{B}}^{\prime }$ as ${\mathit{B}}^{\prime }={y_1\mathit{u},\dots ,y_k\mathit{u}}^{\top }$ by introducing a vector $\mathit{u}$. By definition, $y_i$ is not always zero.

Substituting Equation (44) into Equation (42), we have

$$\begin{array}{c}\hfill \frac{{\mathit{x}}^{\top }{\nabla }_{\mathit{x}}E}{2m}={\mathit{x}}^{\top }\left(∥{\mathit{Z}}^{-1}{\mathit{B}}^{\prime \prime }∥{\mathit{Z}}^{\prime 2}-{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }\right)\mathit{x}.\end{array}$$

(46)

The right-hand side of Equation (46) is zero due to ${\mathit{x}}^{\top }{\nabla }_{\mathit{x}}E=0$. Theorem 8 claims that the vectors $\mathit{x}$ satisfying ${\mathit{x}}^{\top }\left(∥{\mathit{Z}}^{-1}{\mathit{B}}^{\prime \prime }∥{\mathit{Z}}^{\prime 2}-{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }\right)\mathit{x}=0$ are scaled versions of ${\mathit{Z}}^{\prime -2}\mathit{v}$, where $\mathit{v}{\mathit{v}}^{\top }={\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }$. The equality ${\mathit{B}}^{\prime \prime }=\mathit{y}{\mathit{u}}^{\top },\sqrt{n}$ gives us ${\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }=\left({\left|\mathit{u}\right|}^2+n\right)\mathit{y}{\mathit{y}}^{\top }$, and it means $\mathit{v}=\sqrt{{\left|\mathit{u}\right|}^2+n}\mathit{y}$. Thus,

$$\begin{array}{c}\hfill x_i=c\frac{\sqrt{{\sigma }_i^2+x_i^2}}{\sqrt{{\delta }_i^2+y_i^2}}\sqrt{{\left|\mathit{u}\right|}^2+n}{y}_i=c\frac{\sqrt{{\sigma }_i^2+x_i^2}}{\sqrt{y_i^2{\left|\mathit{u}\right|}^2+y_i^2}}\sqrt{{\left|\mathit{u}\right|}^2+n}{y}_i=c^{\prime }\sqrt{{\sigma }_i^2+x_i^2}\end{array}$$

(47)

must be satisfied with a certain real value of c, where $c^{\prime }:=c\sqrt{({\left|\mathit{u}\right|}^2+n)/{\left(\right|\mathit{u}|}^2+1)}$. Solving this equation, we have $x_i=\pm c^{\prime }{\sigma }_i/\sqrt{1-c^{\prime 2}}$. This means that if $\mathit{x}$ is a scaled version of vector ${[{\sigma }_1,\dots ,{\sigma }_k]}^{\top }$ then we have ${\nabla }_{\mathit{x}}E=\mathbf{0}$. Because $\mathit{x}$ can be $\mathbf{0}$, points satisfying ${\nabla }_{\mathit{x}}E=\mathbf{0}$ have the same value of E at $\mathit{x}=\mathbf{0}$.

We show that ${\nabla }_{\mathit{xx}}^{2}E⪰\mathit{O}$ if $\mathit{x}$ is a scaled version of vector ${[{\sigma }_1,\dots ,{\sigma }_k]}^{\top }$. Substituting ${\sigma }_i^2/({\sigma }_i^2+x_i^2)=1-c^{\prime 2}$ into Equation (32), we have

$$\begin{array}{cc}\hfill \frac{1}{2mn}\frac{{\partial }^{2}E}{\partial x_i^2}& =\left(1-c^{\prime 2}\right)\left(\sqrt{\frac{{\delta }_i^2+y_i^2}{{\sigma }_i^2+x_i^2}}\sum _l\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}-\left({\delta }_i^2+y_i^2\right)\right).\hfill \end{array}$$

(48)

We also obtain the following equality for $i\ne j$ by applying $x_i^2/({\sigma }_i^2+x_i^2)=c^{\prime 2}$ into Equation (34).

$$\begin{array}{cc}\hfill \frac{1}{2mn}\frac{{\partial }^{2}E}{\partial x_i\partial x_j}& =c^{\prime 2}\sqrt{\left({\delta }_i^2+y_i^2\right)\left({\delta }_j^2+y_j^2\right)}-\frac{{\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,j}+n{y}_i{y}_j}{n}.\hfill \end{array}$$

(49)

Summarizing Equations (48), (49), and the equality ${\left[{\mathit{B}}^{\prime }{\mathit{B}}^{\prime \top }\right]}_{i,i}+n{y}_i^2=n\left({\delta }_i^2+y_i^2\right)$, we can simplify ${\nabla }_{\mathit{xx}}^{2}E$ as follows:

$$\begin{array}{cc}\hfill \frac{{\nabla }_{\mathit{xx}}^{2}E}{2mn}=& \left(1-c^{\prime 2}\right)\left\{\left(\sum _l\sqrt{\left({\sigma }_l^2+x_l^2\right)\left({\delta }_l^2+y_l^2\right)}\right)\left[\begin{array}{ccc}\sqrt{\frac{{\delta }_1^2+y_1^2}{{\sigma }_1^2+x_1^2}}& & \mathit{O}\\ & \ddots & \\ \mathit{O}& & \sqrt{\frac{{\delta }_k^2+y_k^2}{{\sigma }_k^2+x_k^2}}\end{array}\right]-\frac{{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }}{n}\right\}\hfill \\ \hfill & +c^{\prime 2}\left\{\left[\begin{array}{c}\sqrt{{\delta }_1^2+y_1^2}\\ ⋮\\ \sqrt{{\delta }_k^2+y_k^2}\end{array}\right]\left[\begin{array}{ccc}\sqrt{{\delta }_1^2+y_1^2}& \cdots & \sqrt{{\delta }_k^2+y_k^2}\end{array}\right]-\frac{{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }}{n}\right\}\hfill \end{array}$$

(50)

The assumption on ${\mathit{B}}^{\prime }$ and definition of ${\delta }_i$ attain $\sqrt{{\delta }_i^2+y_i^2}=y_i\sqrt{{\left|\mathit{u}\right|}^2+1}$ for all i. Thus, we have

$$\begin{array}{cc}\hfill \left[\begin{array}{c}\sqrt{{\delta }_1^2+y_1^2}\\ ⋮\\ \sqrt{{\delta }_k^2+y_k^2}\end{array}\right]\left[\begin{array}{ccc}\sqrt{{\delta }_1^2+y_1^2}& \cdots & \sqrt{{\delta }_k^2+y_k^2}\end{array}\right]-\frac{{\mathit{B}}^{\prime \prime }{\mathit{B}}^{\prime \prime \top }}{n}& =\left(1-\frac{1}{n}\right){\left|\mathit{u}\right|}^2\mathit{y}{\mathit{y}}^{\top }⪰\mathit{O}.\hfill \end{array}$$

(51)

The discussion regarding the positive semidefiniteness of Equation (36) also demonstrates that the first term on the right-hand side of Equation (50) is positive semidefinite. Therefore, ${\nabla }_{\mathit{xx}}^{2}E⪰\mathit{O}$ holds if $\mathit{x}$ is a scaled version of vector ${[{\sigma }_1,\cdots ,{\sigma }_k]}^{\top }$.

Through the discussion on the conditions under which $\mathit{x}$ must satisfy in the local solution of E when $\mathit{y}$ is a constant vector, the following revelations emerge:

• If $\mathit{x}$ is a local solution, it must be a scaled version of vector ${[{\sigma }_1,\cdots ,{\sigma }_k]}^{\top }$.
• All local solutions are continuous and have the same value of E.
• ${\nabla }_{\mathit{xx}}^{2}E⪰\mathit{O}$ holds if $\mathit{x}$ is a scaled version of vector ${[{\sigma }_1,\cdots ,{\sigma }_k]}^{\top }$.

In other words, the set of local solutions is globally minimal, and $\mathit{x}=\mathbf{0}$ also becomes a global minimum.

Both discussions (i) and (ii) yield the conclusion that $\mathit{x}=\mathbf{0}$ is valid when E is minimized under the condition of a constant $\mathit{y}$. Similar observations can be made regarding investigating the conditions for $\mathit{y}$ in the local solution of E. Hence, the function E attains global optimality at $\mathit{x}=\mathit{y}=\mathbf{0}$.    □

3. Results

3.1. Assessing the Accuracy of Approximate Matrix Multiplications

We evaluate the approximation performance of the proposed method for matrix multiplication. First, we set the size of the matrices to $m=n=k=2^{15}$. Then, we randomly determine each element of the matrices independently of a uniform distribution between 0 and 1.

Figure 1A evaluates the speedup ratio of the computation time and the approximation performance obtained by applying the conventional and proposed methods for AMM. The horizontal axis compares the computation time of the approximate calculation with that of the exact matrix product calculation. In the experiments of this chapter, we implemented the CUDA program for AMM using CUDA 12.0, which was executed on an NVIDIA A100 GPU. The exact matrix product was computed using an API incorporated into cuBLAS (NVIDIA’s official matrix library). The numerical precision was set to a single-precision floating point. The vertical axis in this figure represents the median relative accuracy between an approximate matrix $\mathit{Z}$ and the exact solution $\mathit{AB}$, calculated by executing each method 100 times. The relative accuracy is defined as

$$\begin{array}{c}\hfill 1-\frac{\parallel \mathit{AB}-\mathit{Z}\parallel }{\parallel \mathit{AB}\parallel }.\end{array}$$

(52)

Figure 1A demonstrates that the proposed method achieves higher acceleration while maintaining relative accuracy. For instance, at a relative accuracy of 0.95, the acceleration rate is 87.2 times faster for the conventional method and 443.2 times faster for the proposed method. Although the relative accuracy required depends on the application, the proposed method outperforms the traditional method at all levels of relative accuracy.

Figure 1B displays a graph showing the speedup ratio trend achieving a relative accuracy of 0.95 when the matrix size is changed from $2^{10}$ to $2^{15}$. This demonstrates that the proposed method can achieve higher speedup ratios than the conventional method regardless of the matrix size. Furthermore, the difference in the speedup ratio expands as the matrix size increases. If the matrix size is small, the exact product does not generally become the bottleneck of the entire calculation. The approximate product is required mainly for large-scale computations. The property of the proposed method (that the advantage becomes more pronounced as the matrix size increases) is desirable.

3.2. Application to a Practical Problem: Computing Eigenvalues and Eigenvectors

Next, we evaluate the performance of the proposed methodology in practical applications. We focus on the computation of eigenvalues, particularly the largest eigenvalue. Several algorithms have been proposed for computing the largest eigenvalue, including power iteration, inverse power iteration [33], accelerated stochastic power iteration [34], and the LOBPCG method [35]. This paper applies the proposed AMM to the power iteration, which forms the backbone of many existing techniques.

We assume for simplicity that the matrix $\mathit{A}\in {\mathbb{R}}^{m\times m}$ is symmetric. Then, all eigenvalues of $\mathit{A}$ are real. See Algorithm 3. For any square matrix $\mathit{A}$, let $\lambda$ be an eigenvalue of $\mathit{A}$. Then, ${\mathit{A}}^2$ has an eigenvalue of ${\lambda }^2$. Therefore, we substitute $\mathit{A}$ with ${\mathit{A}}^2$ in Algorithm 3.

Algorithm 3:  Power method

Require:  $\mathit{A}\in {\mathbb{R}}^{m\times m}$, $\mathit{x}\ne \mathbf{0}$ : m-dimensional randomized unit vector.

Ensure:  $\lambda$: the eigenvalue of $\mathit{A}$ with the largest absolute value

1:  for while not converged do

2:    $\mathit{y}\leftarrow \mathit{Ax}$

3:    $\lambda \leftarrow {\mathit{x}}^{\top }\mathit{y}$

4:    $\mathit{x}\leftarrow \mathit{y}/\parallel \mathit{y}\parallel$

5:  end for

6:  Return $\lambda$

In this case, the second line is expressed as $\mathit{y}\leftarrow {\mathit{A}}^2\mathit{x}$, and we apply the proposed AMM to this step. From Equation (7), we have

$$\begin{array}{cc}\hfill {\mathit{A}}^2& =\mathit{A}{\mathit{A}}^{\top }={\mathit{A}}^{\prime }{\mathit{A}}^{\prime \top }+\left(\mathit{A}\mathbf{\alpha }\right){\mathit{e}}_m^{\top }+{\mathit{e}}_m{\left(\mathit{A}\mathbf{\alpha }\right)}^{\top }-{\mathit{e}}_m\left({\mathbf{\alpha }}^{\top }\mathbf{\alpha }\right){\mathit{e}}_m^{\top }.\hfill \end{array}$$

(53)

We execute the calculation of ${\mathit{A}}^{\prime }{\mathit{A}}^{\prime \top }$ in Equation (53) using Algorithm 2. By incorporating this AMM into Algorithm 3, we obtain Algorithm 4, a power iteration algorithm that utilizes the approximation. The fourth to eighth lines of Algorithm 4 correspond to the Monte Carlo AMM with the proposed method.

Algorithm 4: Power method with the proposed Monte Carlo AMM

Require: $\mathit{A}$:  $m\times m$ symmetric positive-semidefinite matrix, $\mathit{x}\ne \mathbf{0}$: m-dimensional randomized unit vector, $c\in \mathbb{N}$

Ensure:  $\lambda$: the largest absolute eigenvalue of $\mathit{A}$

1:  $\mathit{z}\leftarrow \mathit{A}\mathbf{\alpha }$, $\mathit{v}\leftarrow \mathit{z}-{\left|\mathbf{\alpha }\right|}^2{\mathit{e}}_m$

2:  Set $p_1,\dots ,p_m$, such that ${\sum }_{l=1}^m{p}_l=1$

3:  for while not converged do

4:    $\mathit{y}\leftarrow \left({\mathit{e}}_m^{\top }\mathit{x}\right)\mathit{v}+\left({\mathit{z}}^{\top }\mathit{x}\right){\mathit{e}}_m$

5:    for $t=1,\dots ,c$ do

6:        Pick $i_t$ with $\mathbb{P}[i_t=l]=p_l$

7:        $\mathit{y}\leftarrow \mathit{y}+{\displaystyle \frac{{\mathit{A}}^{\prime \left(i_t\right)}{\mathit{A}}_{\left(i_t\right)}^{\prime \top }\mathit{x}}{c{p}_{i_t}}}$

8:    end for

9:    $\lambda \leftarrow {\mathit{x}}^{\top }\mathit{y}$

10:    $\mathit{x}\leftarrow \mathit{y}/\parallel \mathit{y}\parallel$

11:  end for

12:  Return $\sqrt{\lambda }$

In this section, we assess the performance of the proposed methodology by employing the principal component analysis (PCA) [36] as a representative example, which serves as one of the applications for computing eigenvalues. The PCA is a method employed to examine and reduce the dimensionality of the dataset. It involves identifying the principal components and linear combinations of the original features that effectively capture the most substantial variations within the data. The eigenvalues and eigenvectors of a covariance matrix hold a pivotal role in PCA. When presented with a dataset comprising p features, the initial step in PCA involves the computation of the covariance matrix. This matrix assumes a symmetric structure with dimensions of p-by-p, where each element denotes the covariance between two specific features. The quest for the most dominant principal components is tantamount to identifying the largest eigenvector within the covariance matrix. Hence, within this section, we focus on image analysis, an area in which PCA finds frequent application. Specifically, we have obtained the largest eigenvalues and corresponding eigenvectors from a matrix with a size of $11750\times 11750$, which is acquired via the eigenface algorithm [37] on the LFW dataset [38].

Figure 2A illustrates the variation in convergence over time when CUDA programs of the power method (PM) are run on an NVIDIA A100 GPU. Both figures share a common horizontal axis representing the computation time.

The vertical axis in Figure 2A represents the relative error between the largest and estimated eigenvalue during iterations. In contrast, the vertical axis in Figure 2B represents the deviation between the eigenvector corresponding to the largest eigenvalue and the estimated eigenvector during iterations. Incorporating the proposed AMM into the power method requires the execution of the first line of Algorithm 4, resulting in a longer duration until the completion of the initial iteration compared to the conventional method incorporating AMM in the power method. However, since this computation only needs to be performed once, its impact on the overall computation time is limited. Compared to the power method with the conventional AMM, the method with the proposed AMM demonstrates favorable convergence in both indicators. The power method with the proposed AMM cannot achieve the same level of convergence as the one employing exact matrix multiplication. This is because the expected variance from the AMM breaks the condition revealed in previous studies on the noisy power method [39].

To solve this problem, we propose a variance reduction method that performs exact matrix multiplication under some criteria. When obtaining the approximate matrix product of the matrices $\mathit{A}$ and $\mathit{B}+\tilde{\mathit{B}}$, if $\mathit{C}=\mathit{AB}$ is known, we can compute the approximate solution $\tilde{\mathit{C}}\approx \mathit{A}\tilde{\mathit{B}}$ and provide $\mathit{C}+\tilde{\mathit{C}}$ as an approximate solution. As mentioned in Section 2, the expected error will decrease if the Frobenius norm of $\tilde{\mathit{B}}$ is small. The estimated eigenvector converges gradually through the power method. Therefore, computing the exact matrix product only sometimes and minimizing the expected error of the AMM are expected to be effective. Based on this motivation, we present Algorithm 5. It uses the proposed AMM while performing exact matrix multiplication only for the steps that satisfy the given conditions.

Algorithm 5: Variance reduction power method with the proposed Monte Carlo AMM

Require: $\mathit{A}$:  $m\times m$ symmetric positive-semidefinite matrix, $\mathit{x}\ne \mathbf{0}$: m-dimensional randomized unit vector, $c\in \mathbb{N}$

Ensure:  $\lambda$: the largest absolute eigenvalue of $\mathit{A}$

1:  $\mathit{z}\leftarrow \mathit{A}\mathbf{\alpha }$, $\mathit{v}\leftarrow \mathit{z}-{\left|\mathbf{\alpha }\right|}^2{\mathit{e}}_m$, $\overline{\mathit{x}}\leftarrow \mathbf{0}$, $\overline{\mathit{y}}\leftarrow \mathbf{0}$

2:  Set $p_1,\dots ,p_m$, such that ${\sum }_{l=1}^m{p}_l=1$

3:  for while not converged do

4:    if criteria are met then

5:        $\mathit{y}\leftarrow \overline{\mathit{y}}+\left({\mathit{e}}_m^{\top }(\mathit{x}-\overline{\mathit{x}})\right)\mathit{v}+\left({\mathit{z}}^{\top }(\mathit{x}-\overline{\mathit{x}})\right){\mathit{e}}_m$

6:        for $t=1,\dots ,c$ do

7:              Pick $i_t$ with $\mathbb{P}[i_t=l]=p_l$

8:           $\mathit{y}\leftarrow \mathit{y}+{\displaystyle \frac{{\mathit{A}}^{\prime \left(i_t\right)}{\mathit{A}}_{\left(i_t\right)}^{\prime \top }(\mathit{x}-\overline{\mathit{x}})}{c{p}_{i_t}}}$

9:           end for

10:    else

11:        $\mathit{y}\leftarrow {\mathit{A}}^2\mathit{x}$, $\overline{\mathit{x}}\leftarrow \mathit{x}$, $\overline{\mathit{y}}\leftarrow \mathit{y}$

12:    end if

13:    $\lambda \leftarrow {\mathit{x}}^{\top }\mathit{y}$

14:    $\mathit{x}\leftarrow \mathit{y}/\parallel \mathit{y}\parallel$

15:  end for

16:  Return $\sqrt{\lambda }$

Various conditions can be considered for performing exact matrix multiplication. For instance, we can employ a static strategy where an exact computation is performed every few steps. Alternatively, a dynamic strategy is also promising, where convergence stagnation due to approximation errors is detected, and an exact computation is executed. In this section, we adopt a hybrid approach that combines an AMM with the exact computation based on this dynamic strategy. To assess the convergence of eigenvector calculations, we adopt the following rules:

1.

If the dot product between the current and previous vectors of $\mathit{x}$ exceeds a threshold value $\theta$, then we execute the exact matrix product at the next step.

2.

Even if the initial condition is met, the subsequent calculation employs an AMM if the exact matrix multiplication was previously executed.

The experimental results are presented in Figure 3. We set the number of samplings c to 512. We executed each method 1000 times while changing random seeds. In this figure, the markers represent the median, while the shaded region represents the interquartile range from the first to the third quartiles. The top row corresponds to $\theta =1-{10}^{-4}$, while the bottom row corresponds to $\theta =1-{10}^{-3}$. The left panel depicts the relative error between the true largest eigenvalue ${\lambda }_{max}$ and the estimated largest eigenvalue ${\tilde{\lambda }}_{max}$. The right panel illustrates the discrepancy between the eigenvectors corresponding to the largest eigenvalue, denoted as ${\mathit{v}}_{max}$, and the estimated eigenvectors denoted as ${\tilde{\mathit{v}}}_{max}$. The threshold is a hyperparameter, and it is not clear how to determine an appropriate value for it. Therefore, the method should exhibit stable convergence for various threshold values. For $\theta =1-{10}^{-4}$, the power method with the conventional AMM fails to converge. In contrast, the power method with the proposed AMM demonstrates convergence performance comparable to the power method employing exact matrix multiplication while achieving a reduced computation time. When the threshold is changed to $\theta =1-{10}^{-3}$, the steps involving exact matrix multiplication increase. Consequently, even the power method with the conventional AMM exhibits good convergence. However, the increased frequency of exact matrix multiplications requires more computation time than the power method with the proposed AMM. Thus, the power method with the proposed AMM demonstrates more robust convergence than the one utilizing only exact matrix multiplication. Moreover, it is faster than the power method using only exact matrix multiplication. For instance, when $\theta =1-{10}^{-3}$, the computation time until reaching $1-{\left({\mathit{v}}_{max}^{\top }{\tilde{\mathit{v}}}_{max}\right)}^2={10}^{-11}$ is reduced by about 40%.

4. Discussion

One of the challenges of the proposed method is that its effectiveness diminishes when the elements of matrix $\mathit{A}$’s columns, representing $\mathbf{\alpha }$, and the elements of matrix $\mathit{B}$’s rows, representing $\mathbf{\beta }$, are close to zero. In practical applications and datasets, matrices often exhibit element distributions with specific tendencies or possess the property of being positive semidefinite, thereby deviating from the scenarios where the proposed method is less effective. However, there are cases in specific applications where these conditions apply, requiring alternative methods to improve the approximation accuracy of the matrices.

In this paper, we applied AMM to the simple power method and computed the largest eigenvalue. By incorporating QR decomposition into the iterations of the power method, we can obtain the first several singular eigenvectors [39]. Furthermore, by extending the application of AMM to other variants of the power method discussed in this paper, we anticipate accelerating the computation of various eigenvalue calculations. As mentioned in the previous section, some prior studies on the noisy power method analyze the convergence behavior when noise is introduced in each iteration of the power method. Although we determined the threshold $\theta$ in this paper, it is possible to analytically or dynamically adjust $\theta$ based on such prior research. This remains a challenge for future investigation.

Besides the power method, least square minimization can also be considered among the applications of AMM. Previous research has explored accelerating the analysis of large-scale single nucleotide polymorphisms by utilizing approximate calculations to reduce computational complexity, as the use of pseudo-inverse matrices for obtaining approximate solutions incur high computational costs [19]. By applying our proposed concept to randomized numerical linear algebra (RandNLA), including these applications, there is potential to achieve faster computations and obtain higher-quality approximate solutions. Therefore, applying our method to various scientific and technological computations will be important.