3.3.1. Subspace Separation

Under the above assumption, theoretically, the covariance matrix of the array output is given by

$$\mathbf{R} = \mathbb{E}\left\{ \mathbf{B} \mathbf{B}^H \right\} = \mathbf{C} \mathbf{R}\_s \mathbf{C}^H + 6\sigma^2 \mathbf{I}\_{2M+2\nu} \tag{26}$$

where E{·} stands for the mathematical expectation operator, *σ*<sup>2</sup> is the noise power on each vector antenna, **R***S* = <sup>E</sup>**S**(*t*)**S***<sup>H</sup>*(*t*).

Since the geometric product is non-commutativity, the Eigenvalue Decomposition (ED) is different from the conventional real methods but similar to the quaternion case. In other words, there are two possible eigenvalues, namely the left and the right eigenvalue for G3 matrix. In the proposed algorithm, the right eigenvalue is selected because the right ED of G3 matrix can be converted to the right ED of its CRM [20].

The ED of **R** is denoted by

$$\mathbf{R} = \mathbf{U}\_{\rm s} \boldsymbol{\Sigma}\_{\rm s} \mathbf{U}\_{\rm s}^{H} + \mathbf{U}\_{\rm n} \boldsymbol{\Sigma}\_{\rm n} \mathbf{U}\_{\rm n}^{H}.\tag{27}$$

According to the principle of subspace separation, **U***s* is the signal subspace corresponding to *K* larger eigenvalues, and **Σ***s* is a diagonal matrix composed of *K* larger eigenvalues. In addition, **U***n* is orthogonal to **U***s* and it is the noise subspace corresponding to the remaining 4(*M* + 1) − *K* small eigenvalues. Similarly, **Σ***n* is a diagonal matrix composed of the remaining small eigenvalues.

In the actual processing, the received signal is usually sampled. So, for a certain number of snapshots *N*, (26) and (27) can be rewritten as

$$\begin{split} \hat{\mathbf{R}} &= \frac{1}{N} \sum\_{i=1}^{N} \mathbf{B}(t\_i) \mathbf{B}^H(t\_i), \\ \hat{\mathbf{R}} &= \hat{\mathbf{U}}\_s \hat{\mathbf{S}}\_s \hat{\mathbf{U}}\_s^H + \hat{\mathbf{U}}\_n \hat{\mathbf{S}}\_n \hat{\mathbf{U}}\_n^H. \end{split} \tag{28}$$

Because the space formed by the eigenvectors corresponding to the larger eigenvalues is the same as the space formed by the steering multi-vectors of the incident signals, that is, span{**<sup>U</sup>***s*} = span{**C**}, there exists a unique non-singular matrix **T**, which satisfies

$$\mathbf{U}\_{\mathbb{S}} = \mathbf{CT}.\tag{29}$$

The rotational invariance relations exist among three subarrays, but **U***s* is the signal subspace of the whole array; therefore, after obtaining **U***<sup>s</sup>*, the signal subspace of three subarrays must be separated. By the arrangemen<sup>t</sup> of sensor array, we find that the signal subspace of three subarrays can be calculated by

$$\begin{aligned} \mathbf{U}\_{s1} &= \mathbf{K}\_1 \mathbf{U}\_s = \mathbf{C} \mathbf{T}\_\prime \\ \mathbf{U}\_{s2} &= \mathbf{K}\_2 \mathbf{U}\_s = \mathbf{C} \mathbf{F} \mathbf{T}\_\prime \\ \mathbf{U}\_{s3} &= \mathbf{K}\_3 \mathbf{U}\_s = \mathbf{C} \mathbf{G} \mathbf{T}\_\prime \end{aligned} \tag{30}$$

where **U***s*1, **U***s*<sup>2</sup> and **U***s*<sup>3</sup> are signal subspaces of subarray one, subarray two and subarray three, respectively.

$$\begin{aligned} \mathbf{K}\_1 &= \begin{bmatrix} \mathbf{I}\_M & \mathbf{0}\_{M \times (M+2)} \end{bmatrix}\_{M \times (2M+2)'} \\ \mathbf{K}\_2 &= \begin{bmatrix} \mathbf{0}\_{M \times 1} & \mathbf{I}\_M & \mathbf{0}\_{M \times (M+1)} \end{bmatrix}\_{M \times (2M+2)'} \\ \mathbf{K}\_3 &= \begin{bmatrix} \mathbf{0}\_{M \times (M+1)} & \mathbf{I}\_M & \mathbf{0}\_{M \times 1} \end{bmatrix}\_{M \times (2M+2)'} \end{aligned} \tag{31}$$
