A Jacobi–Davidson Method for Large Scale Canonical Correlation Analysis

Abstract

In the large scale canonical correlation analysis arising from multi-view learning applications, one needs to compute canonical weight vectors corresponding to a few of largest canonical correlations. For such a task, we propose a Jacobi–Davidson type algorithm to calculate canonical weight vectors by transforming it into the so-called canonical correlation generalized eigenvalue problem. Convergence results are established and reveal the accuracy of the approximate canonical weight vectors. Numerical examples are presented to support the effectiveness of the proposed method.

1. Introduction

Canonical correlation analysis (CCA) is one of the most representative two-view multivariate statistical techniques and has been applied to a wide range of machine learning problems including classification, retrieval, regression and clustering [1,2,3]. It seeks a pair of linear transformations for two view high-dimensional features such that the associated low-dimensional projections are maximally correlated. Denote the data matrices $S_a\in {\mathbb{R}}^{m\times d}$ and $S_b\in {\mathbb{R}}^{n\times d}$ from two data sets with m and n features, respectively, where d is the number of samples. Without loss of generality, we assume $S_a$ and $S_b$ are centered, i.e., $S_a{\mathbf{1}}_d=0$ and $S_b{\mathbf{1}}_d=0$ where ${\mathbf{1}}_d\in {\mathbb{R}}^d$ is the vector of all ones, otherwise, we can preprocess $S_a$ and $S_b$ as $S_a\leftarrow S_a-\frac{1}{d}(S_a{\mathbf{1}}_d){\mathbf{1}}_d^{\mathrm{T}}$ and $S_b\leftarrow S_b-\frac{1}{d}(S_b{\mathbf{1}}_d){\mathbf{1}}_d^{\mathrm{T}}$, respectively. CCA aims to find a pair of canonical weight vectors $x\in {\mathbb{R}}^m$ and $y\in {\mathbb{R}}^n$ that maximize the canonical correlation

$$max{x}^{\mathrm{T}}Cy,\mathrm{subject} \mathrm{to}{x}^{\mathrm{T}}Ax=1\mathrm{and}{y}^{\mathrm{T}}By=1,$$

(1)

where

$$A=S_a{S}_a^{\mathrm{T}}\in {\mathbb{R}}^{m\times m},B=S_b{S}_b^{\mathrm{T}}\in {\mathbb{R}}^{n\times n}\mathrm{and}C=S_a{S}_b^{\mathrm{T}}\in {\mathbb{R}}^{m\times n},$$

(2)

and then projects the high-dimensional data $S_a$ and $S_b$ onto low-dimensional subspaces spanned by x and y, respectively, to achieve the purpose of dimensionality reduction. In most cases [1,4,5], only one pair of canonical weight vectors is not enough since it means the dimension of low-dimensional subspaces is just one. When a set of canonical weight vectors are required, the single-vector CCA (1) has been extended to obtain the pair of canonical weight matrices $X\in {\mathbb{R}}^{m\times k}$ and $Y\in {\mathbb{R}}^{n\times k}$ by solving the optimization problem

$$maxtrace(X^{\mathrm{T}}CY),\mathrm{subject} \mathrm{to}{X}^{\mathrm{T}}AX=I_k\mathrm{and}{Y}^{\mathrm{T}}BY=I_k.$$

(3)

Usually, both A and B are symmetric positive definite. However, there are cases, such as the under-sampled problem [6], that A and B may be semi-definite. In such a case, some regular techniques [7,8,9,10] by adding a multiple of the identity matrix to them are applied to find the optimal solution of   

$$maxtrace(X^{\mathrm{T}}CY),\mathrm{subject} \mathrm{to}{X}^{\mathrm{T}}(A+{\kappa }_a{I}_m)X=I_k\mathrm{and}{Y}^{\mathrm{T}}(B+{\kappa }_b{I}_n)Y=I_k,$$

where ${\kappa }_a$ and ${\kappa }_b$ are called regularization parameters and they usually are chosen to maximize the cross-validation score [11]. In other words, A and B are replaced by $A+{\kappa }_a{I}_m$ and $B+{\kappa }_b{I}_n$ to keep the invertible of A and B, respectively. Hence, in this paper, by default we assume A and B are both positive definite and $m\ge n$ unless explicitly stated otherwise.

As shown in [4], the optimization problem (3) can be equivalently transformed into solving the following generalized eigenvalue problem

$$Kz:=\left[\begin{array}{cc}0& C\\ C^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\lambda \left[\begin{array}{cc}A& 0\\ 0& B\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\lambda Mz,$$

(4)

where the positive definiteness of the matrices A and B implies M being positive definite. The generalized eigenvalue problem (4) is referred as the Canonical Correlation Generalized Eigenvalue Problem (CCGEP) in this paper. Define $J:=\left[\begin{array}{cc}{R}_{\mathrm{A}}^{-1}& 0\\ 0& R_{\mathrm{B}}^{-1}\end{array}\right]$ where

$$A=R_{\mathrm{A}}^{\mathrm{T}}{R}_{\mathrm{A}}\mathrm{and}B=R_{\mathrm{B}}^{\mathrm{T}}{R}_{\mathrm{B}}$$

(5)

are their Cholesky decomposition. It is easy to verify that

$$J^{\mathrm{T}}\left[\begin{array}{cc}0& C\\ C^{\mathrm{T}}& 0\end{array}\right]J{J}^{-1}\left[\begin{array}{c}x\\ y\end{array}\right]=\lambda J^{\mathrm{T}}\left[\begin{array}{cc}A& 0\\ 0& B\end{array}\right]J{J}^{-1}\left[\begin{array}{c}x\\ y\end{array}\right]$$

gives rise to

$$\left[\begin{array}{cc}0& \tilde{C}\\ {\tilde{C}}^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{c}p\\ q\end{array}\right]=\lambda \left[\begin{array}{c}p\\ q\end{array}\right],$$

(6)

where

$$\tilde{C}=R_{\mathrm{A}}^{-\mathrm{T}}C{R}_{\mathrm{B}}^{-1},p=R_{\mathrm{A}}x\mathrm{and}q=R_{\mathrm{B}}y,$$

(7)

and it implies

$$\tilde{C}q=\lambda p\mathrm{and}{\tilde{C}}^{\mathrm{T}}p=\lambda q.$$

It means that the eigenpairs of (4) can be obtained by computing the singular values and the associated left and right singular vectors of $\tilde{C}$. This method works well when the sample size d and feature dimension m and n are of moderate size but it will be very slow and numerically unstable for large-scale datasets which are ubiquitous in the age of “Big Data” [12]. For large-scale datasets, the equivalence between (4) and (6) makes it possible for us to simply adapt the subspace type algorithms for calculating the partial singular values decomposition, such as Lanczos type algorithms [13,14] and Davidson type algorithms [15,16], and then translate them for CCGEP (4). However, in practice, the decompositions of the sample covariance matrices A and B are usually unavailable in large scale matrix cases. The reason is that the decomposition is too expensive to compute explicitly for large scale problems, and may destroy the sparsity and some structural information. Furthermore, sometime sample covariance matrices A and B should never be explicitly formed, such as in online learning systems.

Meanwhile, in [17], it is suggested to solve CCGEP (4) by considering the large scale symmetric positive definite pencil $\{K,M\}$. Some subspace type numerical algorithms also have been generalized to computing partial eigenpairs of $\{K,M\}$, see [18,19]. However, these generic algorithms do not make use of the special structure in (4), and they usually are less efficient than custom-made algorithms. Therefore, existing algorithms either can not avoid the covariance matrices decomposition, or do not consider the structure of CCGEP.

In this paper, we will focus on the Jacobi–Davidson type subspace method for canonical correlation analysis. The idea of Jacobi–Davidson algorithm proposed in [20] is Jacobi’s approach combined with Davidson type subspace method. Its essence is to construct a correction for a given eigenvector approximation in a subspace orthogonal to the given approximation. The correction in a given subspace is extracted in a Davidson manner, and then the expansion of the subspace is done by solving its correction equation. Due to the significant improvement in convergence, the Jacobi–Davidson has been one of the most powerful algorithms in the matrix eigenvalue problem, and is almost generalized to all fields of matrix computation. For example, in [15,21], Hochstenbach presented Jacobi–Davidson methods for singular value problems and generalized singular value problems, respectively. In [22,23], Jacobi–Davidson methods are developed to solve the nonlinear and two-parameter eigenvalue problems, respectively. Some other recent work on Jacobi–Davidson methods can be found in [24,25,26,27,28,29]. Motivated by these facts, we will continue the effort by extending the Jacobi–Davidson variant to canonical correlation analysis. The main contribution is that the algorithm directly tackles CCGEP (4) without involving the large matrix decomposition, and does take advantage of the special structure of K and M, while the significance of transforming (4) into (6) lies only in our theoretical developments.

The rest of this paper is organized as follows. Section 2 collects some notations and a basic result for CCGEP that are essential to our later development. Our main algorithm is given and analyzed in detail in Section 3. We present some numerical examples in Section 4 to show the behaviors of our proposed algorithm and to support our analysis. Finally, conclusions are made in Section 5.

2. Preliminaries

Throughout this paper, ${\mathbb{R}}^{m\times n}$ is the set of all $m\times n$ real matrices, ${\mathbb{R}}^m={\mathbb{R}}^{m\times 1}$, and $\mathbb{R}={\mathbb{R}}^1$. $I_n$ is the $n\times n$ identity matrix. The superscript “${·}^{\mathrm{T}}$” takes transpose only, and ${\parallel ·\parallel }_1$ denotes the ${\ell }_1$-norm of a vector or matrix. For any matrix $N\in {\mathbb{R}}^{m\times n}$ with $m\ge n$, ${\sigma }_i(N)$ for $i=1,\cdots ,n$ is used to denote the singular values of N in descending order.

For vectors $x,y\in {\mathbb{R}}^n$, the usual inner product and its induced norm are conveniently defined by

$$\langle x,y\rangle :=y^{\mathrm{T}}x,{\parallel x\parallel }_2:=\sqrt{\langle x,x\rangle }.$$

With them, the usual acute angle $\angle (x,y)$ between x and y can then be defined by

$$\angle (x,y):=arccos\frac{|\langle x,y\rangle |}{{\parallel x\parallel }_2{\parallel y\parallel }_2}.$$

Similarly, given any symmetric positive definite $W\in {\mathbb{R}}^{n\times n}$, the W-inner product and its induced W-norm are defined by

$${\langle x,y\rangle }_W:=y^{\mathrm{T}}Wx,{\parallel x\parallel }_W:={\sqrt{\langle x,x\rangle }}_W.$$

If ${\langle x,y\rangle }_W=0$, then we say $x{\perp }_{W}y$ or $y{\perp }_{W}x$. The W-acute angle ${\angle }_W(x,y)$ between x and y can then be defined by

$${\angle }_W(x,y):=arccos\frac{{|\langle x,y\rangle }_W|}{{\parallel x\parallel }_W{\parallel y\parallel }_W}.$$

Let the singular value decomposition of $\tilde{C}$ be $\tilde{C}=P\Lambda Q^{\mathrm{T}}$ where $P=[p_1,\cdots ,p_m]\in {\mathbb{R}}^{m\times m}$ and $Q=[q_1,\cdots ,q_n]\in {\mathbb{R}}^{n\times n}$ are orthonormal, i.e., $P^{\mathrm{T}}P=I_m$ and $Q^{\mathrm{T}}Q=I_n$, and $\Lambda =diag({\lambda }_1,\cdots ,{\lambda }_n)\in {\mathbb{R}}^{m\times n}$ with ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_n\ge 0$ is a leading diagonal matrix. The singular value decomposition of $\tilde{C}$ closely relates to the eigendecomposition of the following symmetric matrix  [30] (p. 32):

$$\left[\begin{array}{cc}0& \tilde{C}\\ {\tilde{C}}^{\mathrm{T}}& 0\end{array}\right],$$

(8)

whose eigenvalues are $\pm {\lambda }_i$ for $i=1,\cdots ,n$ plus $m-n$ zeros, i.e.,

$$-{\lambda }_1\le \cdots \le -{\lambda }_n\le \underset{m-n}{\underbrace{0\le \cdots \le 0}}\le {\lambda }_n\le \cdots \le {\lambda }_1,$$

(9)

with associated eigenvectors are

$$\left[\begin{array}{c}{p}_i\\ \pm q_i\end{array}\right],i=1,2,\cdots ,n,\mathrm{and}\left[\begin{array}{c}{p}_i\\ 0\end{array}\right],i=n+1,\cdots ,m,$$

respectively. The equivalence between (4) and (6) leads that the eigenvalues of CCGEP (4) are $\pm {\lambda }_i$ for $i=1,\cdots ,n$ plus $m-n$ zeros, and the corresponding eigenvectors are

$$\left[\begin{array}{c}{x}_i\\ \pm y_i\end{array}\right],i=1,2,\cdots ,n,\mathrm{and}\left[\begin{array}{c}{x}_i\\ 0\end{array}\right],i=n+1,\cdots ,m,$$

respectively, where

$$x_i=R_{\mathrm{A}}^{-1}{p}_i\mathrm{and}{y}_i=R_{\mathrm{B}}^{-1}{q}_i.$$

(10)

Let $X=[x_1,\cdots ,x_m]$ and $Y=[y_1,\cdots ,y_n]$. Then, the A- and B-orthonormal constraints of X and Y, respectively, i.e.,

$$X^{\mathrm{T}}AX=I_m\mathrm{and}{Y}^{\mathrm{T}}BY=I_n$$

(11)

are followed by $P^{\mathrm{T}}P=I_m$ and $Q^{\mathrm{T}}Q=I_n$. Here, the first few $x_i$ and $y_i$ for $i=1,2,\cdots ,k$ with $k<n$ are wanted canonical correlation weight vectors. Furthermore, their corresponding eigenvalues satisfy the following maximization principle which is critical to our later developments. For the proof see Appendix A.1.

Theorem 1.

The following equality holds for any $U\in {\mathbb{R}}^{m\times k}$ and $V\in {\mathbb{R}}^{n\times \ell }$

$$\sum _{i=1}^{min\{k,\ell \}}{\lambda }_i=\underset{U^{\mathrm{T}}AU=I_k,V^{\mathrm{T}}BV=I_{\ell }}{max}\sum _{i=1}^{min\{k,\ell \}}{\sigma }_i\left(U^{\mathrm{T}}CV\right),$$

(12)

where A, B and C are defined in (2) and ${\lambda }_i$ defined in (9), and ${\sigma }_i(U^{\mathrm{T}}CV)$ for $1\le i\le min\{k,\ell \}$ are the singular values of $U^{\mathrm{T}}CV$.

3. The Main Algorithm

The idea of the Jacobi–Davidson method [20] is to construct iteratively approximations of certain eigenpairs. It uses solving a correction equation to expand the search subspace, and finds approximate eigenpairs as best approximations in such search subspace.

3.1. Subspace Extraction

Let $\mathcal{X}\subseteq {\mathbb{R}}^m$ and $\mathcal{Y}\subseteq {\mathbb{R}}^n$ with $dim(\mathcal{X})=k$ and $dim(\mathcal{Y})=\ell$, respectively. As stated in [31], we call $\{\mathcal{X},\mathcal{Y}\}$ a pair of defalting subspaces of CCGEP (4) if

$$C\mathcal{Y}\subseteq A\mathcal{X}\mathrm{and}{C}^{\mathrm{T}}\mathcal{X}\subseteq B\mathcal{Y}.$$

(13)

Let $X\in {\mathbb{R}}^{m\times k}$ and $Y\in {\mathbb{R}}^{n\times \ell }$ be A- and B-orthonormal basis matrices of the subspaces $\mathcal{X}$ and $\mathcal{Y}$, respectively, i.e.,

$$X^{\mathrm{T}}AX=I_k\mathrm{and}{Y}^{\mathrm{T}}BY=I_{\ell }.$$

The equality (13) implies that there exist $D_{\mathrm{A}}\in {\mathbb{R}}^{k\times \ell }$ and $D_{\mathrm{B}}\in {\mathbb{R}}^{\ell \times k}$ [32] (Equation (2.11)) such that

$$CY=AX{D}_{\mathrm{A}}\mathrm{and}{C}^{\mathrm{T}}X=BY{D}_{\mathrm{B}}.$$

(14)

They imply $D_{\mathrm{A}}=X^{\mathrm{T}}CY={(Y^{\mathrm{T}}{C}^{\mathrm{T}}X)}^{\mathrm{T}}=D_{\mathrm{B}}^{\mathrm{T}}$. So (14) is equivalent to

$$CY=AX{D}_{\mathrm{A}}\mathrm{and}{C}^{\mathrm{T}}X=BY{D}_{\mathrm{A}}^{\mathrm{T}}.$$

Now if $(\lambda ,\widehat{x},\widehat{y})$ is a singular triplet of $D_{\mathrm{A}}$, then $(\lambda ,z)$ gives an eigenpair of (4), where $z={[x^{\mathrm{T}},y^{\mathrm{T}}]}^{\mathrm{T}}$, $x=X\widehat{x}$ and $y=Y\widehat{y}$. This is because

$$Cy=C(Y\widehat{y})=(CY)\widehat{y}=AX{D}_{\mathrm{A}}\widehat{y}=\lambda AX\widehat{x}=\lambda Ax.$$

and similarly $C^{\mathrm{T}}x=\lambda By$. That means

$$\left[\begin{array}{cc}0& C\\ C^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\lambda \left[\begin{array}{cc}A& 0\\ 0& B\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right].$$

Hence, we have shown that a pair of deflating subspaces $\{\mathcal{X},\mathcal{Y}\}$ can be used to recover those eigenpairs associated with the pair of deflating subspaces of CCGEP (4). In practice, pairs of exact deflating subspaces are usually not available, and one usually use Lanczos type methods [14] or Davidson type methods [15] to generate approximate ones, such as Krylov subspaces in Lanczos method [33]. Next, we will discuss how to extract best approximate eigenpairs from a given pair of approximate deflating subspaces.

In what follows, we consider the simple case: $k=\ell$. Suppose $\{\mathcal{U},\mathcal{V}\}$ is an approximation of a pair of deflating subspaces $\{\mathcal{X},\mathcal{Y}\}$ with $dim(\mathcal{U})=dim(\mathcal{V})=k$. Let $U\in {\mathbb{R}}^{m\times k}$ and $V\in {\mathbb{R}}^{n\times k}$ be the A- and B-orthonormal basis matrices of the subspaces $\mathcal{U}$ and $\mathcal{V}$, respectively. Denote ${\theta }_i$, $i=1,2,\cdots ,k$ the singular values of $U^{\mathrm{T}}CV$ in descending order with associated left and right singular vectors ${\widehat{u}}_i$ and ${\widehat{v}}_i$, respectively, i.e.,

$$(U^{\mathrm{T}}CV){\widehat{v}}_i={\theta }_i{\widehat{u}}_i\mathrm{and}{(U^{\mathrm{T}}CV)}^{\mathrm{T}}{\widehat{u}}_i={\theta }_i{\widehat{v}}_i,\mathrm{for}1\le i\le k.$$

Even though U and V as A- and B-orthonormal basis matrices are not unique, these ${\theta }_i$ are. Motivated by the maximization principle in Theorem 1, we would seek the best approximations associated with the pair of approximate deflating subspaces $\{\mathcal{U},\mathcal{V}\}$ to the eigenpairs $({\lambda }_i,z_i)$ ($1\le i\le j\le k$) in the sense of

$$max\sum _{i=1}^j{\sigma }_i({\tilde{U}}_j^{\mathrm{T}}C{\tilde{V}}_j).$$

(15)

for any ${\tilde{U}}_j\in {\mathbb{R}}^{m\times j}$ and ${\tilde{V}}_j\in {\mathbb{R}}^{n\times j}$ satisfying $span({\tilde{U}}_j)\subseteq \mathcal{U}$, $span({\tilde{V}}_j)\subseteq \mathcal{V}$ and ${\tilde{U}}_j^{\mathrm{T}}A{\tilde{U}}_j={\tilde{V}}_j^{\mathrm{T}}B{\tilde{V}}_j=I_j$. We claim that the quantity in (15) is given by ${\sum }_{i=1}^j{\theta }_i$. To see this, we notice that any ${\tilde{U}}_j$ and ${\tilde{V}}_j$ in (15) can be written as

$${\tilde{U}}_j=U{\widehat{U}}_j\mathrm{and}{\tilde{V}}_j=V{\widehat{V}}_j$$

for some ${\widehat{U}}_j\in {\mathbb{R}}^{k\times j}$ and ${\widehat{V}}_j\in {\mathbb{R}}^{k\times j}$ with ${\widehat{U}}_j^{\mathrm{T}}{\widehat{U}}_j={\widehat{V}}_j^{\mathrm{T}}{\widehat{V}}_j=I_j$, and vice versa. Therefore the quantity in (15) is equal to

$$\underset{{\widehat{U}}_j^{\mathrm{T}}{\widehat{U}}_j={\widehat{V}}_j^{\mathrm{T}}{\widehat{V}}_j=I_j}{max}\sum _{i=1}^j{\sigma }_i({\widehat{U}}^{\mathrm{T}}{U}^{\mathrm{T}}CV\widehat{V}),$$

which is ${\sum }_{i=1}^j{\theta }_i$ by the proposition of the singular value decomposition of $U^{\mathrm{T}}CV$ [30]. The maximum is attended at ${\widehat{U}}_j=[{\widehat{u}}_1,{\widehat{u}}_2,\dots ,{\widehat{u}}_j]$ and ${\widehat{V}}_j=[{\widehat{v}}_1,{\widehat{v}}_2,\dots ,{\widehat{v}}_j]$. Therefore naturally, the best approximations to $({\lambda }_i,z_i)$ ($1\le i\le j$) in the sense of (15) are given by

$$({\theta }_i,{\tilde{z}}_i),\mathrm{where}{\tilde{z}}_i=\left[\begin{array}{c}{\tilde{x}}_i\\ {\tilde{y}}_i\end{array}\right],{\tilde{x}}_i=U{\widehat{u}}_i\mathrm{and}{\tilde{y}}_i=V{\widehat{v}}_i.$$

(16)

Define the residual

$$r_i:=K{\tilde{z}}_i-{\theta }_{i}M{\tilde{z}}_i=\left[\begin{array}{cc}0& C\\ C^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{c}{\tilde{x}}_i\\ {\tilde{y}}_i\end{array}\right]-{\theta }_i\left[\begin{array}{cc}A& 0\\ 0& B\end{array}\right]\left[\begin{array}{c}{\tilde{x}}_i\\ {\tilde{y}}_i\end{array}\right]=\left[\begin{array}{c}{r}_a^{(i)}\\ r_b^{(i)}\end{array}\right],$$

(17)

where K and M defined in (4), $r_a^{(i)}=C{\tilde{y}}_i-{\theta }_{i}A{\tilde{x}}_i$ and $r_b^{(i)}=C^{\mathrm{T}}{\tilde{x}}_i-{\theta }_{i}B{\tilde{y}}_i$. It is noted that

$$U^{\mathrm{T}}{r}_a^{(i)}=U^{\mathrm{T}}(C{\tilde{y}}_i-{\theta }_{i}A{\tilde{x}}_i)=U^{\mathrm{T}}CV{\widehat{v}}_i-{\theta }_i{U}^{\mathrm{T}}AU{\widehat{u}}_i=0$$

and similarly $V^{\mathrm{T}}{r}_b^{(i)}$= 0. We summarize what we do in this subsection in the following theorem.

Theorem 2.

Suppose $\{\mathcal{U},\mathcal{V}\}$ is a pair of approximate deflating subspaces with $dim(\mathcal{U})=dim(\mathcal{V})=k$. Let $U\in {\mathbb{R}}^{m\times k}$ and $V\in {\mathbb{R}}^{n\times k}$ be the A- and B-orthonormal basis matrices of the subspaces $\mathcal{U}$ and $\mathcal{V}$, respectively. Denote ${\theta }_i$, $i=1,2,\cdots ,k$ the singular values of $U^{\mathrm{T}}CV$ in descending order. Then, for any $j\le k$,

$$\sum _{i=1}^j{\theta }_i=\underset{\genfrac{}{}{0pt}{}{span({\tilde{U}}_j)\subseteq \mathcal{U},span({\tilde{V}}_j)\subseteq \mathcal{V}}{{\tilde{U}}_j^{\mathrm{T}}A{\tilde{U}}_j={\tilde{V}}_j^{\mathrm{T}}B{\tilde{V}}_j=I_j}}{max}\sum _{i=1}^j{\sigma }_i({\tilde{U}}_j^{\mathrm{T}}C{\tilde{V}}_j),$$

the best approximations to the eigenpairs $({\lambda }_i,z_i)$ ($1\le i\le j$) in the sense of (15) are $({\theta }_i,{\tilde{z}}_i)$ ($1\le i\le j$) given by (16), and the associated residuals defined in (17) admit $r_a^{(i)}\perp \mathcal{U}$ and $r_b^{(i)}\perp \mathcal{V}$.

3.2. Correction Equation

In this subsection, we turn to construct a correction equation for a given eigenpair approximation. Suppose $(\theta ,{[{\tilde{x}}^{\mathrm{T}},{\tilde{y}}^{\mathrm{T}}]}^{\mathrm{T}})$ with ${\tilde{x}}^{\mathrm{T}}A\tilde{x}={\tilde{y}}^{\mathrm{T}}B\tilde{y}=1$ is an approximation of the eigenpair $(\lambda ,{[x^{\mathrm{T}},y^{\mathrm{T}}]}^{\mathrm{T}})$ of CCGEP (4), and ${[r_a^{\mathrm{T}},r_b^{\mathrm{T}}]}^{\mathrm{T}}$ is the associated residual. We seek A- and B-orthogonal modifications of $\tilde{x}$ and $\tilde{y}$, respectively, such that

$$\left[\begin{array}{cc}0& C\\ C^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{c}\tilde{x}+s\\ \tilde{y}+t\end{array}\right]=\lambda \left[\begin{array}{cc}A& 0\\ 0& B\end{array}\right]\left[\begin{array}{c}\tilde{x}+s\\ \tilde{y}+t\end{array}\right],$$

(18)

where $s{\perp }_A\tilde{x}$ and $t{\perp }_B\tilde{y}$. Then, by (18), we have

$$\left[\begin{array}{cc}-\lambda A& C\\ C^{\mathrm{T}}& -\lambda B\end{array}\right]\left[\begin{array}{c}s\\ t\end{array}\right]=-\left[\begin{array}{c}{r}_a\\ r_b\end{array}\right]+(\lambda -\theta )\left[\begin{array}{cc}A& 0\\ 0& B\end{array}\right]\left[\begin{array}{c}\tilde{x}\\ \tilde{y}\end{array}\right].$$

(19)

Notice that $r_a\perp \tilde{x}$ and $r_b\perp \tilde{y}$ by Theorem 2, which gives rise to

$$\left[\begin{array}{cc}{I}_m-A\tilde{x}{\tilde{x}}^{\mathrm{T}}& 0\\ 0& I_n-B\tilde{y}{\tilde{y}}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{c}{r}_a\\ r_b\end{array}\right]=\left[\begin{array}{c}{r}_a\\ r_b\end{array}\right],\left[\begin{array}{cc}{I}_m-A\tilde{x}{\tilde{x}}^{\mathrm{T}}& 0\\ 0& I_n-B\tilde{y}{\tilde{y}}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{c}A\tilde{x}\\ B\tilde{y}\end{array}\right]=0,$$

and

$$\left[\begin{array}{cc}{I}_m-A\tilde{x}{\tilde{x}}^{\mathrm{T}}& 0\\ 0& I_n-B\tilde{y}{\tilde{y}}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}-\lambda A& C\\ C^{\mathrm{T}}& -\lambda B\end{array}\right]\left[\begin{array}{c}s\\ t\end{array}\right]=-\left[\begin{array}{c}{r}_a\\ r_b\end{array}\right].$$

(20)

Because $s{\perp }_A\tilde{x}$ and $t{\perp }_B\tilde{y}$, Equation (20) is rewritten as

$$\left[\begin{array}{cc}{I}_m-A\tilde{x}{\tilde{x}}^{\mathrm{T}}& 0\\ 0& I_n-B\tilde{y}{\tilde{y}}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}-\lambda A& C\\ C^{\mathrm{T}}& -\lambda B\end{array}\right]\left[\begin{array}{cc}{I}_m-\tilde{x}{\tilde{x}}^{\mathrm{T}}A& 0\\ 0& I_n-\tilde{y}{\tilde{y}}^{\mathrm{T}}B\end{array}\right]\left[\begin{array}{c}s\\ t\end{array}\right]=-\left[\begin{array}{c}{r}_a\\ r_b\end{array}\right].$$

(21)

However, we do not know $\lambda$ here. It is natural that we use $\theta$ to replace $\lambda$ in (21) to get the final correction equation, i.e.,

$$\left[\begin{array}{cc}{I}_m-A\tilde{x}{\tilde{x}}^{\mathrm{T}}& 0\\ 0& I_n-B\tilde{y}{\tilde{y}}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}-\theta A& C\\ C^{\mathrm{T}}& -\theta B\end{array}\right]\left[\begin{array}{cc}{I}_m-\tilde{x}{\tilde{x}}^{\mathrm{T}}A& 0\\ 0& I_n-\tilde{y}{\tilde{y}}^{\mathrm{T}}B\end{array}\right]\left[\begin{array}{c}s\\ t\end{array}\right]=-\left[\begin{array}{c}{r}_a\\ r_b\end{array}\right].$$

(22)

We summarize what we have so far into Algorithm 1, and make a few comments on Algorithm 1.

(1)

At step 2, A- and B-orthogonality procedures are applied to make sure $U^{\mathrm{T}}A\tilde{t}=0$ and $V^{\mathrm{T}}B\tilde{s}=0$.

(2)

At step 7, in most cases, the correct equation is not necessity to solve exactly. Some steps of iterative methods for symmetric linear systems, such as linear conjugate gradient method (CG) [34] or the minimum residual method (MINRES) [35], are sufficient. Usually, more steps in solving the correction equation lead to fewer outer iterations. This will be shown in numerical examples.

(3)

For the convergence test, we use the relative residual norms

$$\eta ({\theta }_i,{\tilde{z}}_i):=\frac{\parallel r_a^{(i)}{\parallel }_1+{\parallel r_b^{(i)}\parallel }_1}{{(\parallel C\parallel }_1+{\theta }_i{\parallel A\parallel }_1)\parallel {\tilde{x}}_i{\parallel }_1+{(\parallel C\parallel }_1+{\theta }_i{\parallel B\parallel }_1)\parallel {\tilde{y}}_i{\parallel }_1}$$

(23)

to determine if the approximate eigenparis $({\theta }_i,{\tilde{z}}_i)$ has converged to a desired accuracy. In addition, in the practical implementation, once one or several of approximate eigenpairs converge to a preset accuracy, they should be deflated so that they will not be re-computed in the following iterations. Suppose ${\lambda }_i$ for $1\le i\le j$, $X_j=[x_1,\cdots ,x_j]$ and $Y_j=[y_1,\cdots ,y_j]$ have been computed where $j\le k$. We can consider the generalized eigenvalue problem

$$\tilde{K}z=\lambda Mz,$$

(24)

where

$$\begin{array}{c}\hfill \tilde{K}=\left[\begin{array}{cc}{I}_m-A{X}_j{X}_j^{\mathrm{T}}& 0\\ 0& I_n-B{Y}_j{Y}_j^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}0& C\\ C^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{cc}{I}_m-X_j{X}_j^{\mathrm{T}}A& 0\\ 0& I_n-Y_j{Y}_j^{\mathrm{T}}B\end{array}\right].\end{array}$$

(25)

By (11), it is clear that the eigenvalues of (24) consist of two groups. Those eigenvalues associated with the eigenvectors ${[x_1^{\mathrm{T}},y_1^{\mathrm{T}}]}^{\mathrm{T}},\cdots ,{[x_j^{\mathrm{T}},y_j^{\mathrm{T}}]}^{\mathrm{T}},{[x_1^{\mathrm{T}},-y_1^{\mathrm{T}}]}^{\mathrm{T}},\cdots ,{[x_j^{\mathrm{T}},-y_j^{\mathrm{T}}]}^{\mathrm{T}}$ are shifted to zero and the others remain unchanged. Furthermore, for the correction equation, we find s and t subject to additional A- and B-orthogonality constrains for s and t against $X_j$ and $Y_j$, respectively. By a similar derivation of (22), the correction equation after deflation becomes

$$\begin{array}{cc}\hfill & \left[\begin{array}{cc}{I}_m-A\tilde{x}{\tilde{x}}^{\mathrm{T}}& 0\\ 0& I_n-B\tilde{y}{\tilde{y}}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}-{\theta }_{1}A& \widehat{C}\\ {\widehat{C}}^{\mathrm{T}}& -{\theta }_{1}B\end{array}\right]\left[\begin{array}{cc}{I}_m-\tilde{x}{\tilde{x}}^{\mathrm{T}}A& 0\\ 0& I_n-\tilde{y}{\tilde{y}}^{\mathrm{T}}B\end{array}\right]\left[\begin{array}{c}s\\ t\end{array}\right]=-\left[\begin{array}{c}{r}_a^{(1)}\\ r_b^{(1)}\end{array}\right],\hfill \end{array}$$

(26)

where $\widehat{C}=(I_m-A{X}_j{X}_j^{\mathrm{T}})C(I_n-Y_j{Y}_j^{\mathrm{T}}B)$. Notice that $s{\perp }_A{X}_j$ and $t{\perp }_B{Y}_j$ mean $U{\perp }_A{X}_j$ and $V{\perp }_B{Y}_j$ in Algorithm 1, respectively. It follows that $U^{\mathrm{T}}\widehat{C}V=U^{\mathrm{T}}CV$.

(4)

At step 5, LAPACK’s routine xGESVD can be used to solve the singular value problem of $U^{\mathrm{T}}CV$ because of its small size, where $U^{\mathrm{T}}CV$ takes the following form:

This form is preserved in the algorithm during refining the basis U and V at step 8. The new basis matrices $U\widehat{U}$ and $V\widehat{V}$ are reassigned to U and V, respectively. Although a few extra costs are incurred, this refinement is necessary in order to have faster convergence for eigenvectors as stated in [36,37]. Furthermore, the restart is easily executed by keeping the first $s_{min}$ columns of U and V when the dimension of the subspaces $span\left\{U\right\}$ and $span\left\{V\right\}$ exceeds $s_{max}$. The restart technique appears at step 8 to keep the size of U, V and $U^{\mathrm{T}}CV$ small. There are many ways to specify $s_{max}$ and $s_{min}$. In our numerical examples, we just simply take $s_{max}=3k$ and $s_{min}=k$.

Algorithm 1 Jacobi–Davidson method for canonical correlation analysis (JDCCA)

Input: Initial vectors $u_0$, $v_0$, $s=u_0$, $t=v_0$ and $V=U=\left[\right]$. Output: Converged canonical weight vectors ${\tilde{x}}_i$ and ${\tilde{y}}_i$ for $i=1,\cdots ,k$. 1: for$iter=1,2,\cdots$,until convergence do 2:  A- and B-orthogonal s and t against U and V, respectively, to obtain $\tilde{s}$ and $\tilde{t}$. 3:  Compute $u=\tilde{s}/{\parallel \tilde{s}\parallel }_A$ and $v=\tilde{t}/{\parallel \tilde{t}\parallel }_B$. Let $U=[U,u]$ and $V=[V,v]$. 4:  Update the corresponding column and row of $U^{\mathrm{T}}CV$. 5:  Compute the singular value decomposition of $U^{\mathrm{T}}CV$, i.e., $U^{\mathrm{T}}CV=\widehat{U}\mathsf{\Theta }{\widehat{V}}^{\mathrm{T}}$. 6:  Compute the wanted approximate eigenpairs $({\theta }_i,{[{\tilde{x}}_i^{\mathrm{T}},{\tilde{y}}_i^{\mathrm{T}}]}^{\mathrm{T}})$ by (16) and the corresponding residuals $r_a^{(i)}$ and $r_b^{(i)}$. 7:  Solve $$\left[\begin{array}{cc}{I}_m-A{\tilde{x}}_1{\tilde{x}}_1^{\mathrm{T}}& 0\\ 0& I_n-B{\tilde{y}}_1{\tilde{y}}_1^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}-{\theta }_{1}A& C\\ C^{\mathrm{T}}& -{\theta }_{1}B\end{array}\right]\left[\begin{array}{cc}{I}_m-{\tilde{x}}_1{\tilde{x}}_1^{\mathrm{T}}A& 0\\ 0& I_n-{\tilde{y}}_1{\tilde{y}}_1^{\mathrm{T}}B\end{array}\right]\left[\begin{array}{c}t\\ s\end{array}\right]=-\left[\begin{array}{c}{r}_a^{(1)}\\ r_b^{(1)}\end{array}\right].$$ 8:  Update $U=U\widehat{U}$ and $V=V\widehat{V}$. If the dimension of U and V exceeds $s_{max}$, then replace U and V with $U_{(1:s_{min})}$ and $V_{(1:s_{min})}$ respectively. 9: end for

3.3. Convergence

The convergence theories on the Jacobi–Davidson method for the eigenvalue and singular value problem are given in [15,38], respectively. Here we prove a similar convergence result for the Jacobi–Davidson method of CCGEP based on the following lemma. Specifically, if we solve the correction Equation (22) exactly, and then $\tilde{x}$ and $\tilde{y}$ are close enough to x and y, respectively, it can be hoped that the approximate eigenvectors converge cubically. For the proof see Appendix A.2 and Appendix A.3.

Lemma 1.

Let λ be a simple eigenvalue of CCGEP (4) with the corresponding eigenvector ${[x^{\mathrm{T}},y^{\mathrm{T}}]}^{\mathrm{T}}$. Then the matrix

$$G:=\left[\begin{array}{cc}{I}_m-Ax{x}^{\mathrm{T}}& 0\\ 0& I_n-By{y}^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}-\lambda A& C\\ C^{\mathrm{T}}& -\lambda B\end{array}\right]\left[\begin{array}{cc}{I}_m-x{x}^{\mathrm{T}}A& 0\\ 0& I_n-y{y}^{\mathrm{T}}B\end{array}\right]$$

is a bijection from $span{(x)}^{{\perp }_A}\times span{(y)}^{{\perp }_B}$ onto itself, where $span{(x)}^{{\perp }_A}$ and $span{(y)}^{{\perp }_B}$ are A- and B-orthogonal complementary spaces of $span(x)$ and $span(y)$, respectively.

Theorem 3.

Assume the condition of Lemma 1, $sin{\angle }_A(x,\tilde{x})=\mathcal{O}(\epsilon )$ and $sin{\angle }_B(y,\tilde{y})=\mathcal{O}(\epsilon )$. Let ${[s^{\mathrm{T}},t^{\mathrm{T}}]}^{\mathrm{T}}$ be the exact solution of the correction Equation (22). Then,

$$\left|sin{\angle }_A(x,\tilde{x}+s)\right|=\mathcal{O}({\epsilon }^3)and\left|sin{\angle }_B(y,\tilde{y}+t)\right|=\mathcal{O}({\epsilon }^3).$$

(27)

4. Numerical Examples

In this section, we present some numerical examples to illustrate the effectiveness of Algorithm 1. Our goal is to compute the first few canonical weight vectors. A computed approximate eigenpair $({\theta }_i,{\tilde{z}}_i)$ is considered converged when its relative residual norm

$$\eta ({\theta }_i,{\tilde{z}}_i)=\frac{\parallel r_a^{(i)}{\parallel }_1+{\parallel r_b^{(i)}\parallel }_1}{{(\parallel C\parallel }_1+{\theta }_i{\parallel A\parallel }_1)\parallel {\tilde{x}}_i{\parallel }_1+{(\parallel C\parallel }_1+{\theta }_i{\parallel B\parallel }_1)\parallel {\tilde{y}}_i{\parallel }_1}\le {10}^{-8}.$$

(28)

All the experiments in this paper are executed on a Ubuntu 12.04 (64 bit) Desktop-Intel(R) Core(TM) i7-6700 [email protected] GHz, 32 GB of RAM using MATLAB 2010a with machine epsilon $2.22\times {10}^{-16}$ in double-precision floating point arithmetic.

Example 1.

We first examine Theorem 3 by using two pairs of data matrices $S_a$ and $S_b$ which come from a publicly available handwritten numerals dataset (https://archive.ics.uci.edu/ml/datasets/Multiple+Features). It consists of features handwritten numerals (‘0’–‘9’) and each digit has 200 patterns. Each pattern is represented by six different feature sets, i.e., Fou, Fac, Kar, Pix, Zer and Mor. Two pairs of feature sets Fou-Zer and Pix-Fou are chosen for $S_a$ and $S_b$, respectively, such that $S_a\in {\mathbb{R}}^{76\times d}$ and $S_b\in {\mathbb{R}}^{47\times d}$ in Fou-Zer, and $S_a\in {\mathbb{R}}^{240\times d}$ and $S_b\in {\mathbb{R}}^{76\times d}$ in Pix-Fou with $d=2000$. To make the numerical example repeatable, the initial vectors are set to be

$$u_0=x_1+{10}^{-3}\times \mathsf{ones}(m,1)and{v}_0=y_1+{10}^{-3}\times \mathsf{ones}(n,1)$$

where m and n are the dimension of $S_a$ and $S_b$, respectively, $\mathsf{ones}$ is MATLAB built-in function, and ${[x_1^{\mathrm{T}},y_1^{\mathrm{T}}]}^{\mathrm{T}}$ is computed by MATLAB’s function eig on (4) and considered to be the “exact” eigenvector for testing purposes. The corrected Equation (22) in Algorithm 1 is solved by direct methods, such as Gaussian elimination, and the solution ${[s^{\mathrm{T}},t^{\mathrm{T}}]}^{\mathrm{T}}$ by such methods is regarded as “exactly” in this example. Figure 1 plots $sin{\angle }_A(x_1,{\tilde{x}}_1)$ and $sin{\angle }_B(y_1,{\tilde{y}}_1)$ in the first three iterations of Algorithm 1 for computing first canonical weight vector of Fou-Zer and Pix-Fou. It is clearly shown by Figure 1 that the convergence of Algorithm 1 is very fast when the initial vectors are enough close to the exact vectors, and the cubical convergence of Algorithm 1 appears in the third iteration.

Example 2.

As stated in Algorithm 1, the implementation of JDCCA involves solving the correction Equation (22) in every step. Direct solvers mentioned in Example 1 referring to $\mathcal{O}\left({(m+n)}^3\right)$ operations are prohibitively expensive in solving large-scale sparse linear systems. In such a case, iterative methods, such as MINRES method which is simply GMRES [39] applied to symmetric linear systems, are usually preferred. In this example, we report the effect of the number of steps in the solution of the correction equation, denoted by $n_g$, on the total number of matrix-vector products (denoted by “#mvp”), outer iteration number (denoted by “#iter”), and CPU time in seconds for Algorithm 1 to compute the first 10 canonical weight vectors of the test problems appeared in

Table 1. Test problems.

Problems	ORL	FERET	Yale
m	10,304	6400	10,000
n	10,304	6400	10,000
d	200	600	75

. Table 1 presents three face datasets, i.e., ORL (https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html,) FERET (http://www.nist.gov/itl/iad/ig/colorferet.cfm) and Yale (https://computervisiononline.com/dataset/1105138686) datasets. The ORL database contains 400 face images of 40 distinct persons. For each individual, there are 10 different gray scale images with $92\times 112$ pixels. These images are collected by volunteers at different times, different lighting and different facial expressions (blinking or closed eyes, smiling or no-smiling, wearing glasses or no-glasses). In order to apply CCA, as numerical experiments in [40], the ORL dataset is partitioned into two groups. We select the first five images per individual as the first view to generate the data matrix $S_a$, while the remaining for $S_b$. Similarly, we get data matrices $S_a$ and $S_b$ for the FERET and Yale datasets. The numbers of row and column of $S_a$ and $S_b$ are detailed in Table 1.

In this example, we set the initial vectors $u_0=\mathsf{ones}(m,1)$ and $v_0=\mathsf{ones}(n,1)$ with $s_{max}=30$ and $s_{min}=10$ for restarting and simply take regularization parameter ${\kappa }_a={\kappa }_b={10}^{-4}$. We let MINRES steps $n_g$ vary from to 5 to 40, and collect the numerical results in Figure 2. As expected, the number of total outer iterations decreases as $n_g$ increases, while the total number of matrix-vector products does not change monotonically with $n_g$. It depends on the degree of reduction of outer iterations by the increasing of $n_g$. In addition, it is shown by Figure 2 that the total #mvp is not a unique deciding factor on the total CPU time. When $n_g$ is larger, the significantly reduced #iter leads to smaller total CPU time. For these three test examples, the MINRES steps $n_g$ around 15 to 25 appear to be cost-effective, further increasing $n_g$ over 40 usually does not have significant effect. The least efficient case is when $n_g$ is too small.

Example 3.

In this example, we compare Algorithm 1, i.e., JDCCA, with Jacobi–Davidson QZ type method [41] (JDQZ) for the large scale symmetric positive definite pencil $\{K,M\}$ defined in (4) to compute the first 10 canonical weight vectors of the test problems appeared in Table 1 with MINRES steps $n_g=20$. We take $u_0=\mathsf{ones}(m,1)$ and $v_0=\mathsf{ones}(n,1)$ in Algorithm 1 and the initial vector $\mathsf{ones}(m+n,1)$ for the JDQZ algorithm, and compute the same relative residual norms $\eta ({\theta }_i,{\tilde{z}}_i)$. The corresponding numerical results are plotted in Figure 3. For these three test problems, it is suggested by Figure 3 that Algorithm 1 always outperforms the JDQZ algorithm. Other experiments that we tested with different test problems and MINRES steps not reported here also illustrate our points.

5. Conclusions

To analyze the correlations between two data sets, several numerical algorithms have been available to find the canonical correlations and the associated canonical weight vectors; however, there is very little discussion of the large scale sparse and structured matrix cases in the literature. In this paper, a Jacobi–Davidson type method, i.e., Algorithm 1, is presented for large scale canonical correlation analysis by computing a small portion of eigenpairs of the canonical correlation generalized eigenvalue problem (4). The theoretical result is established in Theorem 3 to demonstrate that the cubic convergence of the approximate eigenvector if the correction equation is solved exactly and the approximate eigenvector of the previous step is close enough to the exact one. The corresponding numerical results are presented to confirm the effectiveness of asymptotic convergence rate provided by Theorem 3, and to demonstrate that Algorithm 1 performs far superior to the JDQZ method for the large scale symmetric positive definite pencil $\{K,M\}$.

Notice that the main computational tasks in every iteration of Algorithm 1 consist of solving the correction Equation (22). In our numerical example, we only focus on the plain version of MINRES, i.e., without considering any preconditioner. However, it is not hard to notice that incorporating a preconditioner presents no difficulty and can promote the numerical performance if the preconditioner is available. In addition, from the point of view that multi-set canonical correlation analysis (MCCA) [42] proposed to analyze linear relationships among more than two data sets can be equivalently transformed to the following generalized eigenvalue problem

$$\left[\begin{array}{cccc}0& C_{12}& \cdots & C_{1t}\\ C_{21}& 0& \cdots & C_{2t}\\ ⋮& ⋮& ⋮& ⋮\\ C_{t1}& C_{t2}& \cdots & 0\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_t\end{array}\right]=\lambda \left[\begin{array}{cccc}{C}_{11}& 0& \cdots & 0\\ 0& C_{22}& \cdots & 0\\ ⋮& ⋮& ⋮& ⋮\\ 0& 0& \cdots & C_{tt}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_t\end{array}\right],$$

where $C_{ij}=S_i{S}_j^{\mathrm{T}}$ and $S_i$ is the data matrix, the development of efficient Jacobi–Davidson methods for treating such large scale MCCA will be a subject of our future study.