A Parallel Two-Stage Iteration Method for Solving Continuous Sylvester Equations

Abstract

In this paper we propose a parallel two-stage iteration algorithm for solving large-scale continuous Sylvester equations. By splitting the coefficient matrices, the original linear system is transformed into a symmetric linear system which is then solved by using the SYMMLQ algorithm. In order to improve the relative parallel efficiency, an adjusting strategy is explored during the iteration calculation of the SYMMLQ algorithm to decrease the degree of the reduce-operator from two to one communications at each step. Moreover, the convergence of the iteration scheme is discussed, and finally numerical results are reported showing that the proposed method is an efficient and robust algorithm for this class of continuous Sylvester equations on a parallel machine.

1. Introduction

The solution of the continuous Sylvester equation

$$AX+XB=F$$

(1)

with large sparse matrices $A\in {\mathbb{R}}^{n\times n}$ , $B\in {\mathbb{R}}^{n\times n}$ , $X\in {\mathbb{R}}^{n\times n}$ , $F\in {\mathbb{R}}^{n\times n}$ and with A, B positive definite is a common task in numerical linear algebra. It arises in many scientific computing and engineering applications, such as control theory [1,2], neural networks, model reduction [3], image processing [4], and so on. Therefore, the problem has remained an active area of research. In this context, recent methodological advances have been thoroughly discussed in many papers [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]. Iterative methods for solving linear or nonlinear equations have seen constant improvement in recent years to reduce the computational time; for example, two multi-step derivative-free iterative methods [5]: block Jacobi two stage method [6] and SYMMLQ algorithm [7,8,9]. In addition, widely-used direct methods are, for instance, the Bartels–Stewart [10] and the Hessenberg–Schur method [11]. The main idea is to transform A and B into triangular or Hessenberg form [21] by an orthogonal similarity transformation and then to solve the resulting system of linear equations directly by a back-substitution process. However, this method is not applicable in large-scale problems due to the prohibitive computational issue. In order to overcome this limitation, fast iterative methods have been developed such as the Smith method [12], the alternating direction implicit (ADI) method [13], gradient-based methods [14,15], and the Krylov subspace-based algorithm [7,16,17]. At present, the conjugate gradient (CG) method [7] and the preconditioned conjugate gradient method [18] are popularly used with the advantages of small storage and suitability for parallel computing. Typically, the SYMMLQ algorithm [7,8,9] is quite efficient in the case of symmetric coefficient matrices, as it has tremendous advantages in small storage capacity and stable computations. However, it is not a good option for multi-computer systems due to the high cost of global communication. For asymmetric coefficient matrices, a modified conjugate gradient method (MCG) is useful. However, its convergence speed is slow [22,23].

Another type of iteration based on splitting methods allows us to better utilize standard methodologies. For instance, Bai et al. [24] proposed Hermitian and skew-Hermitian splitting (HSS) iteration methods for solving systems of linear equations with non-Hermitian positive definite form. This has been studied widely and generalized in [25,26,27,28]. Recently, a Hermitian and skew-Hermitian splitting (HSS) iteration method for solving large sparse continuous Sylvester equations with non-Hermitian and positive definite/semidefinite matrices was discussed in [29]. Wang et al. [30] presented a positive-definite and skew-Hermitian splitting (PSS) iteration method, and in [31] Zhou et al. applied the modified Hermitian and skew-Hermitian splitting (MHSS) iteration method to solve the continuous Sylvester equation. Zheng and Ma in [32] applied the idea of the normal and skew-Hermitian splitting (NSS) iteration method to continuous Sylvester equations.

However, these iteration methods have the common difficulty that there is no accurate formula to determine the positive value of the corresponding parameter in the iteration scheme. In many articles, a large amount of work has been done to address this issue. Unfortunately this estimation methodology is still not fully resolved in practical applications. In addition, their implementations need to solve two continuous Sylvester equations, which results in great additional computational cost.

All of these brought about the need for the development and validation of an efficient parallel algorithm. In this paper we have proposed a parallel algorithm of two-stage iteration for solving large-scale continuous Sylvester equations with the combination of the HSS iteration method and the SYMMLQ algorithm. The main idea is to split the coefficient matrices into a symmetric and an anti-symmetric matrix, respectively. Then, the original equations are transferred into symmetric matrix equations which are solved by the SYMMLQ algorithm. Furthermore, we focus on the improvement of the parallel efficiency of the SYMMLQ algorithm by adjusting the calculation steps.

The remainder of this paper is organized as follows. In Section 2, a description of the two-stage iteration method is presented based on a splitting method and the SYMMLQ algorithm for solving the continuous Sylvester Equation (1). Then the parallel implementation of the algorithm is given in Section 3 . Its convergence analysis and numerical examples are mentioned in Section 4 and Section 5, respectively. We end with conclusions.

Notation in this paper: $A^{\mathrm{T}}$ denotes the transpose of matrix A; inner product using $[A,B]=tr(A^{\mathrm{T}}B)$; matrix norm of A induced by $\parallel A\parallel =\sqrt{[A,A]}=\sqrt{tr(A^{\mathrm{T}}A)}$ and $\rho (A)$ is the spectral radius of the matrix A. For the matrix $X={(x_1,x_2,\cdots ,x_n)}^{\mathrm{T}}\in {\mathbb{R}}^{n\times n}$, $\overline{vec}(X)$ denotes the $\overline{vec}$ operator defined as $\overline{vec}(X)={(x_1^{\mathrm{T}},x_2^{\mathrm{T}},\cdots ,x_n^{\mathrm{T}})}^{\mathrm{T}}\in {\mathbb{R}}^{n^2}$.

2. Description of the Two-Stage Iteration Method

The two-stage iteration method consisting of the outer and inner iterations is discussed in this section. The outer iteration is performed by splitting the coefficient matrices, while the inner iteration is computed via the SYMMLQ algorithm.

2.1. Outer Iteration Scheme

We can split A and B into symmetric and antisymmetric parts:

$$A=M_1-N_1,B=M_2-N_2$$

(2)

where $M_1$ and $M_2$ are symmetric parts and $N_1$ and $N_2$ are antisymmetric parts. They can be rewritten as

$$\begin{array}{c}{M}_1=\frac{1}{2}(A+A^{\mathrm{T}}),N_1=\frac{1}{2}(A^{\mathrm{T}}-A),\\ M_2=\frac{1}{2}(B+B^{\mathrm{T}}),N_2=\frac{1}{2}(B^{\mathrm{T}}-B).\end{array}$$

More details can be found in [24,28,29,31].

Then, the continuous Sylvester Equation (1) can be written as the following matrix equation:

$$M_{1}X+X{M}_2=N_{1}X+X{N}_2+F.$$

(3)

Given an initial guess $X^{(0)}\in {\mathbb{R}}^{n\times n}$, because the initial guess has an effect on the convergence speed of the algorithm, it has little influence on the calculation results. For convenience, the initial guess is taken as $X^{(0)}=O$ in the numerical examples. For $k=0,1,2,\cdots$, we use the following iteration scheme until $\left\{X^{(k)}\right\}$ converges:

$$M_1{X}^{(k+1)}+X^{(k+1)}{M}_2=N_1{X}^{(k)}+X^{(k)}{N}_2+F.$$

(4)

Let ${\tilde{F}}^{(k)}=N_1{X}^{(k)}+X^{(k)}{N}_2+F$. Then, the outer iteration can be expressed as

$$M_1{X}^{(k+1)}+X^{(k+1)}{M}_2={\tilde{F}}^{(k)}.$$

(5)

We need to solve Equation (5) at each step of the iteration method so as to form the two-stage iteration method. Equation (5) is computed by the SYMMLQ algorithm, since the new coefficient matrices are symmetric.

2.2. Inner Iteration Scheme Based on the SYMMLQ Algorithm

Equation (5) is changed into the following linear system by using the $\overline{vec}$ operator

$$H{x}^{(k+1)}={\tilde{f}}^{(k)},$$

(6)

where $H=A\otimes I+I\otimes B^{\mathrm{T}}$, the vectors $x^{(k+1)}$ and $\tilde{f}$ contain the concatenated columns of the matrices $X^{(k+1)}$ and ${\tilde{F}}^{(k)}$, respectively, with ⊗ being the Kronecker product symbol.

Then, the SYMMLQ algorithm is proposed to solve the kth step iteration equation of the outer iteration scheme. For more details, we can refer to [8,9]. The description of the corresponding SYMMLQ algorithm is given roughly in the following manner.

Let $y=x^{(k+1)}$ and $b={\tilde{f}}^{(k)}$. Then, Equation (6) can be written equivalently as

$$Hy=b.$$

(7)

Then, transform H into an $i\times i$ symmetric tridiagonal matrix $T_i$ by the Lanczos orthogonalization process. The coefficient matrix H is potentially unstable, and consequently $T_i$ is not positive definite. So, the LQ decomposition ( Triangular Orthogonal decomposition, where L is a lower triangular matrix, Q is an orthogonal matrix ) is used to transform $T_i$ into an $i\times i$ lower triangular matrix $L_i$, seen in the following flow chart: The SYMMLQ algorithm:

$$H\underset{orthogonalization}{\overset{Lanczos}{\to }}{T}_i\underset{decomposition}{\overset{LQ}{\to }}{L}_i{P}_i.$$

(8)

where $T_i=L_i{P}_i$, and

$$\begin{array}{cc}\hfill T_i=\left(\begin{array}{ccccc}{a}_1& b_1& & & \\ b_1& a_2& b_2& & \\ & b_2& a_3& \ddots & \\ & & \ddots & \ddots & b_{i-1}\\ & & & b_{i-1}& a_i\end{array}\right),& L_i=\left(\begin{array}{ccccc}{r}_1& & & & \\ {\delta }_2& r_2& & & \\ {\epsilon }_3& {\delta }_3& r_3& & \\ & \ddots & \ddots & \ddots & \\ & & {\epsilon }_i& {\delta }_i& r_i\end{array}\right),\hfill \end{array}$$

$$P_i=P_{i-1}\left(\begin{array}{ccccc}1& & & & \\ & 1& & & \\ & & \ddots & & \\ & & & c_{i-1}& s_{i-1}\\ & & & -s_{i-1}& c_{i-1}\end{array}\right).$$

According to the above observation, we can establish the following stationary fixed-point iteration form for Equation (7):

$$y^{(i+1)}=y^{(i)}+Q_i{z}_i$$

(9)

where $Q=(q_1,q_2,\cdots ,q_i)$ is an $n\times i$ matrix and $q_1,q_2,\cdots ,q_i$ are orthonormal vectors which are computed by the Lanczos orthogonalization process. Here $z_i$ satisfies the following equation:

$$T_i{z}_i={\parallel r^{(0)}\parallel }_2{e}_1,$$

(10)

where $r^{(0)}=b-H{y}^{(0)}$, $y^{(0)}$ is a given initial vector, and $e_1={(1,0,\cdots ,0)}^{\mathrm{T}}$ is an i-dimensional unit vector. More details can be found in the literature [33].

3. Parallel Implementation of the Two-Stage Iteration Method

In this section we discuss the parallel implementation including data storage, and implementation of outer iteration and inner iteration.

3.1. Data Storage

For convenience, let p be the number of processors, $p_i(i=1,2,\cdots ,p)$ represent ith processor, and l is an integer in $n=pl$.

Mark

$$\begin{array}{c}A={(A_1^{\mathrm{T}},A_2^{\mathrm{T}},\cdots ,A_p^{\mathrm{T}})}^{\mathrm{T}},B={(B_1^{\mathrm{T}},B_2^{\mathrm{T}},\cdots ,B_p^{\mathrm{T}})}^{\mathrm{T}},F={(F_1^{\mathrm{T}},F_2^{\mathrm{T}},\cdots ,F_p^{\mathrm{T}})}^{\mathrm{T}},\hfill \\ X^{(0)}={({(X_1^{(0)})}^{\mathrm{T}},{(X_2^{(0)})}^{\mathrm{T}},\cdots ,{(X_p^{(0)})}^{\mathrm{T}})}^{\mathrm{T}},A^{\mathrm{T}}={({\tilde{A}}_1^{\mathrm{T}},{\tilde{A}}_2^{\mathrm{T}},\cdots ,{\tilde{A}}_p^{\mathrm{T}})}^{\mathrm{T}},\hfill \\ B^{\mathrm{T}}={({\tilde{B}}_1^{\mathrm{T}},{\tilde{B}}_2^{\mathrm{T}},\cdots ,{\tilde{B}}_p^{\mathrm{T}})}^{\mathrm{T}},\tilde{F}={({\tilde{F}}_1^{\mathrm{T}},{\tilde{F}}_2^{\mathrm{T}},\cdots ,{\tilde{F}}_p^{\mathrm{T}})}^{\mathrm{T}},\hfill \\ M_1={({(M_{1,1})}^{\mathrm{T}},{(M_{1,2})}^{\mathrm{T}},\cdots ,{(M_{1,p})}^{\mathrm{T}})}^{\mathrm{T}},M_2={({(M_{2,1})}^{\mathrm{T}},{(M_{2,2})}^{\mathrm{T}},\cdots ,{(M_{2,p})}^{\mathrm{T}})}^{\mathrm{T}},\hfill \\ N_1={({(N_{1,1})}^{\mathrm{T}},{(N_{1,2})}^{\mathrm{T}},\cdots ,{(N_{1,p})}^{\mathrm{T}})}^{\mathrm{T}},N_2={({(N_{2,1})}^{\mathrm{T}},{(N_{2,2})}^{\mathrm{T}},\cdots ,{(N_{2,p})}^{\mathrm{T}})}^{\mathrm{T}}\hfill \end{array}$$

where $A_i$, $B_i$, $F_i$, $X_i^{(0)}$, ${\tilde{A}}_i$, ${\tilde{B}}_i$, ${\tilde{F}}_i$, $M_{1,i}$, $M_{2,i}$, $N_{1,i}$, $N_{2,i}$ $(i=1,2,\cdots ,p)$ are $l\times n$ sub-block matrices. These are saved in row storage. Then, store $A_i$, $B_i$, $F_i$, $X_i^{(0)}$, ${\tilde{A}}_i$, ${\tilde{B}}_i$ on the processor $p_i(i=1,2,\cdots ,p)$.

Note: 

Due to the storage method, we chose the way of matrix multiplication with block row–row matrices in parallel computing process. Detailed descriptions of parallel computing Matrix multiplication are found in References [5,23,34].

3.2. Parallel Implementation of Outer Iteration Method

(1) Splitting process: Processor $p_i(i=1,2,\cdots ,p)$ computes

$$M_{1,i}=(A_i+{\tilde{A}}_i)/2,N_{1,i}=({\tilde{A}}_i-A_i)/2,M_{2,i}=(B_i+{\tilde{B}}_i)/2,N_{2,i}=({\tilde{B}}_i-B_i)/2.$$

(2) Cycle processor:

Step 1. Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\Delta X_i^{(k)}=X_i^{(k)}-X_i^{(k-1)}\mathrm{and}[\Delta X_i^{(k)},\Delta X_i^{(k)}],$$

get $[\Delta X_i^{(k)},\Delta X_i^{(k)}]$ after all-reduce. If $\parallel \Delta X^{(k)}\parallel <\epsilon$ stop; otherwise, turn to step 2.

Step 2. Compute ${\tilde{F}}_i=N_{1,i}{X}^{(k)}+X_i^{(k)}{N}_2+F_i$ in each processor.

Step 3. Use the improved parallel process of SYMMLQ algorithm to solve the new equation

$$M_1{X}^{(k+1)}+X^{(k+1)}{M}_2=\tilde{F}.$$

(11)

This step, which improves the parallel efficiency and reduces the parallel time by reducing the frequency of communication, plays an important role in the whole parallel implementation of the two-stage iteration method, and the detailed implementation can be seen in Section 3.3 and Section 3.4.

Step 4. Let $k:=k+1$, turn to Step 1.

3.3. Parallel Implementation of Inner Iteration Scheme

(1) Compute process:

① Processor $p_i(i=1,2,\cdots ,p)$ computes

$$R_i^{(0)}=F_i-(A_i{X}^{(0)}+X_i^{(i)}B)\mathrm{and}[R_i^{(0)},R_i^{(0)}],$$

obtain $[R_i^{(0)},R_i^{(0)}]$ after all-reduce, then compute

$$Q_i^{(1)}=R_i^{(0)}/\parallel R^{(0)}\parallel .$$

② Compute

$$G_i^{(1)}=A_i{Q}^{(1)}+Q_i^{(1)}B\mathrm{and}[G_i^{(1)},Q_i^{(1)}],$$

obtain $a_1=[G_i^{(1)},Q_i^{(1)}]$ after all-reduce. Compute

$$H_i^{(1)}=G_i^{(1)}-a_1{Q}_i^{(1)}\mathrm{and}[H_i^{(1)},H_i^{(1)}],$$

get $[H^{(1)},H^{(1)}]$ after all-reduce and $b_1=\sqrt{[H^{(1)},H^{(1)}}$, then compute ${\tilde{r}}_1$,${\tilde{\mathsf{\xi }}}_1$,${\tilde{W}}_i^{(1)}$ and $X_i^{(1)}=X_i^{(0)}+{\tilde{\mathsf{\xi }}}_1{\tilde{W}}_i^{(1)}$ in each processor.

③ Processor $p_i(i=1,2,\cdots ,p)$ computes

$$Q_i^{(2)}=H_i^{(1)}/b_1\mathrm{and}{G}_i^{(2)}=A_i{Q}^{(2)}+Q_i^{(2)}B,$$

and computes the inner product $[G_i^{(2)},Q_i^{(2)}]$, get $a_2=[G^{(2)},Q^{(2)}]$ after all-reduce. Compute

$$H_i^{(2)}=G_i^{(2)}-a_2{Q}_i^{(2)}-b_1{Q}_i^{(1)}\mathrm{and}[H_i^{(2)},H_i^{(2)}],$$

get $[H^{(2)},H^{(2)}]$ after all-reduce, compute $b_2=\sqrt{[H^{(2)},H^{(2)}]}$.

④ Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\begin{array}{c}{c}_1=\frac{a_1}{{(a_1^2+b_1^2)}^{\frac{1}{2}}},s_1=\frac{b_1}{{(a_1^2+b_1^2)}^{\frac{1}{2}}},{\delta }_2=b_1{c}_1+a_2{s}_1,{\epsilon }_3=b_2{s}_1,\hfill \\ r_1={(a_1^2+b_1^2)}^{\frac{1}{2}},{\mathsf{\xi }}_1=\parallel r_0\parallel /r_1,{\tilde{r}}_2=b_1{s}_1-a_2{c}_1,r_2={({\tilde{r}}_2^2+b_2^2)}^{\frac{1}{2}},{\tilde{\delta }}_3=-b_2{c}_1,\hfill \end{array}$$

if ${\tilde{r}}_2<{10}^{-15}$ stop, otherwise compute ${\tilde{\mathsf{\xi }}}_2=-{\delta }_2{\mathsf{\xi }}_1/{\tilde{r}}_2$.

⑤ Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\begin{array}{c}{\mathsf{\xi }}_2=-\frac{{\delta }_2{\mathsf{\xi }}_1}{r_2},W_i^{(1)}=c_1{\tilde{W}}_i^{(1)}+s_1{Q}_i^{(2)},{\tilde{W}}_i^{(2)}=s_1{\tilde{W}}_i^{(1)}-c_1{Q}_i^{(2)},\hfill \\ Y_i^{(1)}=X_i^{(0)}+{\mathsf{\xi }}_1{W}_i^{(1)},X_i^{(2)}=Y_i^{(1)}+{\tilde{\mathsf{\xi }}}_2{\tilde{W}}_i^{(2)}.\hfill \end{array}$$

(2) Cycle process:

Step 1. Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\parallel R^{(k-1)}\parallel =|b_{k-1}(s_{k-2}{\mathsf{\xi }}_{k-2}-c_{k-2}{\tilde{\mathsf{\xi }}}_{k-1})|,$$

if $\parallel R^{(k-1)}\parallel <\epsilon$ stop, otherwise, turn to Step 2.

Step 2. Processor $p_i(i=1,2,\cdots ,p)$ computes

$$Q_i^{(k)}=H_i^{(k-1)}/b_{k-1}\mathrm{and}{G}_i^{(k)}=A_i{Q}^{(k)}+Q_i^{(k)}B,$$

then compute $[G_i^{(k)},Q_i^{(k)}]$, obtain $a_k=[G^{(k)},Q^{(k)}]$, after all-reduce. Compute

$$H_i^{(k)}=G_i^{(k)}-a_k{Q}_i^{(k)}-b_{k-1}{Q}_i^{(k-1)}\mathrm{and}[H_i^{(k)},H_i^{(k)}],$$

Obtain $[H^{(k)},H^{(k)}]$ after all-reduce, compute $b_k=\sqrt{[H^{(k)},H^{(k)}]}$.

Step 3. Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\begin{array}{c}{c}_{k-1}=\frac{{\tilde{r}}_{k-1}}{{({\tilde{r}}_{k-1}^2+b_{k-1}^2)}^{\frac{1}{2}}},s_{k-1}=\frac{b_{k-1}}{{({\tilde{r}}_{k-1}^2+b_{k-1}^2)}^{\frac{1}{2}}},{\epsilon }_{k+1}=b_k{s}_{k-1},\hfill \\ {\delta }_k={\tilde{\delta }}_k{c}_{k-1}+a_k{s}_{k-1},{\tilde{r}}_k={\tilde{\delta }}_k{s}_{k-1}-a_k{c}_{k-1},r_k={({\tilde{r}}_k^2+b_k^2)}^{\frac{1}{2}},{\tilde{\delta }}_{k+1}=-b_k{c}_{k-1},\hfill \end{array}$$

if ${\tilde{r}}_k<{10}^{-15}$ stop; otherwise, compute ${\tilde{\mathsf{\xi }}}_k=({\epsilon }_k{\mathsf{\xi }}_{k-2}+{\delta }_k{\mathsf{\xi }}_{k-1})/{\tilde{r}}_k$.

Step 4. Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\begin{array}{c}{\mathsf{\xi }}_k=-({\epsilon }_k{\mathsf{\xi }}_{k-2}+{\delta }_k{\mathsf{\xi }}_{k-1})/r_k,W_i^{k-1}=c_{k-1}{\tilde{W}}_i^{(k-1)}+s_{k-1}{Q}_i^{(k)},\hfill \\ {\tilde{W}}_i^{(k)}=s_{k-1}{\tilde{W}}_i^{k-1}-c_{k-1}{Q}_i(k),Y_i^{(k-1)}=Y_i^{(k-2)}+{\mathsf{\xi }}_{k-1}{W}_i^{(k-1)},\hfill \\ X_i^{(k)}=Y_i^{(k-1)}+{\tilde{\mathsf{\xi }}}_k{\tilde{W}}_i^{(k)}.\hfill \end{array}$$

Step 5 Let $k:=k+1$, turn to Step 1.

3.4. Improved Parallel Implementation of the SYMMLQ Algorithm

Clearly, when computing $a_k$, $b_k$ in each step of the inner iteration, all processors need to apply the reduce operator twice in the parallel implementation of the SYMMLQ algorithm in Section 3.3. Therefore, we should rearrange Step 2 in the cycle process, while the remaining steps remain the same. The detailed parallel process of the algorithm can be expressed as follows.

Processor $p_i(i=1,2,\cdots ,p)$ computes

$$\begin{array}{c}{Q}_i^{(k)}=(H_i^{(k-1)}-a_{k-1}{Q}_i^{(k-1)})/b_{k-1},G_i^{(k)}=A_i{Q}^{(k)}+Q_i^{(k)}B,\\ H_i^{(k)}=G_i^{(k)}-b_{k-1}{Q}_i^{(k-1)}\mathrm{and}{L}_i^{(k)}=H_i^{(k)}-a_{k-1}{Q}_i^{(k)},\end{array}$$

then compute $[G_i^{(k)},Q_i^{(k)}]$ and $[L_i^{(k)},L_i^{(k)}]$, get $a_k=[G^{(k)},Q^{(k)}]$ and $e_k=[L^{(k)},L^{(k)}]$ after one all-reduce, compute $b_k=\sqrt{e_k-{(a_k-a_{k-1})}^2}$.

In this way, computing $a_k$, $e_k$ only needs to all-reduce once, so as to reduce the frequency of communication and thus reduce the parallel time. Eventually, we obtain an improved parallel implementation of the SYMMLQ algorithm.

4. Convergence Analysis

The convergence property above two-stage iteration method is mentioned here.

Lemma 1

(see [35]). Let H be a positive definite matrix, and $H=M-N$ be a splitting, with $M=(H+H^{\mathrm{T}})/2$, and $N=(H^{\mathrm{T}}-H)/2$. Then, $\rho (M^{-1}N)<1$ if for all $y\in C^n$, it holds that $y^{\mathrm{H}}My>|y^{\mathrm{H}}Ny|$.

Based on the above lemma, we obtain the following conclusion.

Theorem 1.

Assume the positive definite matrix $H\in R^{n\times n}$ is split according to Lemma 1. If for all $y\in C^n$, we have that $|Re(y^{\mathrm{H}}Hy)|>|Im(y^{\mathrm{H}}Hy)|$, then $\rho (M^{-1}N)<1$ holds.

Proof of Theorem 1.

For all $y\in C^n$, we can get

$$y^{\mathrm{H}}My=(y^{\mathrm{H}}Hy+y^{\mathrm{H}}{H}^{\mathrm{H}}y)/2=Re(y^{\mathrm{H}}Hy)$$

(12)

and

$$y^{\mathrm{H}}Ny=(-y^{\mathrm{H}}Hy+y^{\mathrm{H}}{H}^{\mathrm{H}}y)/2=Im(y^{\mathrm{H}}Hy).$$

(13)

According to the Lemma 1, we can obtain $|Re(y^{\mathrm{H}}Hy)|>|Im(y^{\mathrm{H}}Hy)|$. Therefore $\rho (M^{-1}N)<1$. ☐

The above theorem is generalized to the continuous Sylvester equation. The convergence theorem of the matrix equation can be obtained as follows:

Theorem 2.

Let $A,B\in R^{n\times n}$ be positive definite matrices in the Sylvester Equation (1) and suppose that they are split as in the two-stage iterative format (5). If for all $Y\in C^{n\times n}$, we have

$$|Re[Y,AY+YB]|>|Im[Y,AY+YB]|,$$

(14)

then

$$\rho ({(M_1\otimes I+I\otimes M_2^{\mathrm{T}})}^{-1}(N_1\otimes I+I\otimes N_2^{\mathrm{T}}))<1$$

(15)

holds.

Proof of Theorem 2.

By using the Kronecker product, we can transform Equation (1) into the matrix-vector form:

$$(A\otimes I+I\otimes B^{\mathrm{T}})x=f,$$

(16)

where $x=\overline{vec}(X)$, $f=\overline{vec}(F)$. When the coefficient matrices A and B are positive definite matrices, we split them as in Section 2.1. Then, Equation (3) can be rewritten equivalently as

$$(M_1\otimes I+I\otimes M_2^{\mathrm{T}})x=(N_1\otimes I+I\otimes N_2^{\mathrm{T}})x+f.$$

(17)

According to Theorem 1, we yield

$$|Re(y^{\mathrm{H}}(A\otimes I+I\otimes B^{\mathrm{T}})y)|>|Im(y^{\mathrm{H}}(A\otimes I+I\otimes B^{\mathrm{T}})y)|.$$

(18)

On the other hand, it holds that

$$y^{\mathrm{H}}(A\otimes I+I\otimes B^{\mathrm{T}})y=tr(Y^{\mathrm{H}}(AY+YB))=[Y,AY+YB].$$

(19)

Therefore,

$$|Re[Y,AY+YB]|>|Im[Y,AY+YB]|.$$

(20)

 ☐

5. Numerical Examples

In order to illustrate the performance of the two-stage iteration (TS iteration) method, some examples were performed in Matlab on an Intel dual core processor (1.00 GHz, 2 GB RAM) and the parallel machine Lenovo Shen-teng 1800 cluster. All iterations were started from a zero matrix and terminated when the current iterate satisfied $\parallel R^{(k)}\parallel <{10}^{-6}$, where $R^{(k)}=F-(A{X}^{(k)}+X^{(k)}B)$ is the residual of the kth iteration.

Here we compare the TS iteration method with the HSS iteration method proposed in [29].

Notation:

T	the computational time in seconds
ITs	the number of iteration steps
p	the total number of processors
S	speedup ratio
$E_1$	parallel efficiency
$\Delta$	error $\parallel R^{(k)}\parallel$

Example 1.

Consider the continuous Sylvester Equation (1) with $m=n$ and the matrices

$$A=B=M+2rN+\frac{100}{{(n+1)}^2}I,$$

where I is the identity matrix, and $M,N\in R^{n\times n}$ are the tridiagonal matrices given by

$$M=tridiag(-1,-2,-1)andN=tridiag(0.5,0,-0.5).$$

The goal in this test is to compare the iteration steps and the computational time by using TS iteration method, HSS iteration method, and MCG method with $r=0.01$, $r=0.1$, and $r=1.0$. The numerical results are listed in

Table 1. Comparison of the the number of iteration steps (ITs) and the computational time in seconds (T) when choosing $r=0.01$. HSS: Hermitian and skew-Hermitian splitting; MCG: modified conjugate gradient method; TS: two-stage iteration.

n	TS		HSS		MCG
	ITs	T	ITs	T	ITs	T
32	4	0.030	41	0.053	237	0.036
64	20	0.070	111	0.378	943	0.356
128	130	1.411	270	5.995	4091	9.526

,

Table 2. Comparison of ITs and T when choosing $r=0.1$.

n	TS		HSS		MCG
	ITs	T	ITs	T	ITs	T
32	10	0.020	55	0.055	250	0.040
64	27	0.159	96	0.464	1006	0.399
128	149	2.593	205	7.996	3864	9.010

and

Table 3. Comparison of ITs and T when choosing $r=1.0$.

n	TS		HSS		MCG
	ITs	T	ITs	T	ITs	T
32	31	0.043	48	0.061	183	0.059
64	54	0.395	125	0.443	254	0.415
128	158	5.796	247	8.872	458	8.236

, respectively. The optimal parameters ${\alpha }_{exp}(\mathrm{with}{\beta }_{exp}={\alpha }_{exp})$ for the HSS iteration method are given in

Table 4. The optimal values ${\alpha }_{exp}$ for HSS.

n	HSS
	$\mathbf{r}=\mathbf{0}.\mathbf{01}$	$\mathbf{r}=\mathbf{0}.\mathbf{1}$	$\mathbf{r}=\mathbf{1}.\mathbf{0}$
32	0.40	0.40	0.95
64	0.17	0.23	0.81
128	0.09	0.13	0.62

proposed in [29].

From the above tables, we see that both iteration steps and computational time by the TS method are much less than those by the HSS and MCG in all cases. The comparison between MCG and HSS is not straight-forward. Furthermore, for the above tables we observe that in some cases the number of iteration steps of MCG is larger than that of HSS, while on the contrary, for the computational time it mainly depends on the computational time of each iteration step.

Example 2.

Consider the elliptic partial differential equation

$$\frac{{\partial }^{2}u}{\partial x^2}+\frac{{\partial }^{2}u}{\partial y^2}+\sin 2\pi x\frac{\partial u}{\partial x}+\sin 2\pi y\frac{\partial u}{\partial y}+u=0,0\le x,y\le 1$$

with boundary condition

$$\left\{\begin{array}{c}\hfill {u|}_{x=0}{=u|}_{x=1}=10+\cos \pi y\\ \hfill {u|}_{y=0}{=u|}_{y=1}=10+\cos \pi x\end{array}.\right.$$

Let the step size be $h=1/101$ and $h=1/1001$. That means that the size of the linear system is $100\times 100$ and $1000\times 1000$, respectively. The equation is discretized by using five-point difference format, and then is transformed into a Sylvester equation. The numerical results are shown in

Table 5. Numerical results of Example 2 with $h=1/101$.

	TS				MCG
p	1	2	4	6	1	2	4	6
T	140.25	77.62	39.83	30.45	297.47	160.89	84.29	63.56
ITs	1067	1067	1067	1067	4992	4992	4992	4992
S		1.81	3.52	4.61		1.85	3.53	4.68
$E_1$	1.00	0.90	0.88	0.77	0.47	0.44	0.42	0.37
$\Delta$	9.66 × 10${}^{-7}$	9.66 × 10${}^{-7}$	9.66 × 10${}^{-7}$	9.14 × 10${}^{-7}$	9.01 × 10${}^{-7}$	9.13 × 10${}^{-7}$	9.11 × 10${}^{-7}$	9.80 × 10${}^{-7}$

and

Table 6. Numerical results of Example 2 with $h=1/1001$.

	TS				MCG
p	20	25	30	35	20	25	30	35
T	641.12	569.88	491.28	463.74	1563.70	1374.68	1184.62	1116.93
ITs	7194	7194	7194	7194	91,332	91,332	91,332	91,336
S		22.50	26.10	27.65		22.75	26.40	28.00
$E_1$	1.00	0.90	0.87	0.79	0.16	0.37	0.36	0.33
$\Delta$	9.72 × 10${}^{-7}$	9.72 × 10${}^{-7}$	9.72 × 10${}^{-7}$	9.72 × 10${}^{-7}$	9.83 × 10${}^{-7}$	9.83 × 10${}^{-7}$	9.93 × 10${}^{-7}$	9.54 × 10${}^{-7}$

. This numerical experiment is performed on the parallel machine Lenovo Shen-teng 1800 cluster. Here we focus on comparing the parallel performance with the TS iteration method and the MCG method.

From the results Table 5 and Table 6, we observe that both iteration steps and computational time using TS are much less than those with the MCG. Furthermore, parallel efficiency with the TS method is higher than with the MCG. In addition, the advantages of the TS method increase over the MCG method with increasing scale-size of equations from ${10}^4$ ($n=100$) to ${10}^6$ ($n=1000$).

Example 3.

Let $A={(a_{ij})}_{n\times n}$ and $B={(b_{ij})}_{n\times n}$ where

$$a_{ij}=\{\begin{array}{cc}-n+\sin (i+j),\hfill & i=j\hfill \\ i,\hfill & others\hfill \end{array},b_{ij}=\{\begin{array}{cc}3+2\sin (i+j),\hfill & i=j\hfill \\ \cos (i+j),\hfill & j-i=1.\hfill \\ \sin (i+j),\hfill & i-j=1\hfill \end{array}$$

In the Sylvester matrix equation $AX+XB=F$, let $n=1000$ and F is an any given matrix. The numerical results are listed in

Table 7. Numerical results of Example 3.

	TS
p	16	20	24	28
T	2480.19	2351.59	2296.47	2241.46
ITs	509	509	509	509
S		16.87	17.28	17.92
$E_1$	1.00	0.84	0.72	0.64
$\Delta$	8.83 × 10${}^{-7}$	8.83 × 10${}^{-7}$	8.83 × 10${}^{-7}$	8.83 × 10${}^{-7}$

.

From Table 7 we observe that the two-stage iteration method is still efficient in the case that the coefficient matrices are indefinite matrices. This indicates that the condition for the convergence is only a sufficient condition in Theorem 1.

6. Conclusions

In this paper we have proposed a two-stage iteration parallel method for solving the continuous Sylvester equations. The outer iteration scheme is based on the coefficient matrices’ splitting. Furthermore, an inner iteration scheme is obtained using the SYMMLQ algorithm. Its parallel implementation and its improved strategy have been explained in detail. Moreover, we have proved the convergence of the proposed iteration method under certain conditions. Numerical results show that the new proposed algorithm is better than both MCG and HSS methods with regard to computational storage, computational time, and iteration steps. Crucially, these advantages become more significant for large-scale systems of continuous Sylvester equations.