Transceiver Optimization for Multiuser Multiple-Input Multiple-Output Full-Duplex Amplify-and-Forward Relay Downlink Communications

Abstract

This paper considers the transceiver design in a multiuser multiple-input multiple-output (MIMO) full-duplex (FD) amplify-and-forward (AF) relay downlink communication system, where users simultaneously transmit data via an FD relay node. The design incorporates an imperfect loop interference (LI) cancellation which results in a residual LI. Linear precoders are employed at the sources and relay, and minimum mean-squared-error (MMSE) combiners are employed at the destinations to mitigate the effect of the residual LI. The corresponding design problem is highly nonconvex, so a closed-form solution is intractable. Thus, an iterative method is developed to solve this optimization problem. Simulation results are presented which show that the proposed iterative algorithm provides better performance than the corresponding half-duplex (HD) solution in terms of the achievable rate under residual LI.

1. Introduction

Multiple-input multiple-output (MIMO) relay communication systems have been the subject of considerable research due to their ability to improve both achievable rates and coverage [1,2,3,4,5,6]. A relay is used between the source and destination, so the signal is transmitted from the source to the relay and then from the relay to the destination. The relay node can employ either the decode-and-forward (DF) or amplify-and-forward (AF) protocols. AF amplifies the received signal and forwards it to the destination, so it has a lower complexity than the DF protocol [7].

Precoding is a well-known technique for interference mitigation. A joint precoding optimization for a multiuser relay downlink system was investigated in [1]. The sum capacity was maximized by using quadratic programming, but multiple antennas were employed only at the relay. The performance can be improved by using multiple antennas at the source and destination. A transceiver design for a multiuser non-regenerative MIMO relay system with multiple antennas at both the source and destination was investigated in [2,3,4,5,6]. However, a half-duplex (HD) relay was employed, so transmission from the source to destination requires an extra time slot which limits the potential capacity.

In the past decade, full-duplex (FD) relay systems have attracted attention because data transmission from a source to a destination can be completed in one time slot [7,8,9,10,11]. Thus, FD MIMO relaying can increase capacity compared to HD systems [9], but loop interference (LI) is a critical issue because the relay transmits and receives simultaneously. In general, the LI is much larger than the channel noise and so can significantly degrade performance. As a consequence, several LI cancellation techniques have been developed [10]. Temporal cancellation methods such as antenna isolation and analog/digital precancellation have been shown to be effective. However, it is impossible to cancel the LI completely, and the residual LI can still be larger than the noise [12].

Estimating channel state information (CSI) is challenging due to the dynamic nature of wireless environments and the complexity of modern communication systems. However, current estimation algorithms have been shown to provide good accuracy, particularly in cellular systems [13]. Thus, it can be assumed that CSI is available in the proposed system. In [14], deep neural networks were considered for channel estimation with multiuser precoding in the downlink of a frequency-division duplex massive MIMO system. In [15], an efficient transceiver design was examined for full-duplex (FD) multiuser massive MIMO systems operating in millimeter wave frequencies. An iterative solution was developed for the proposed digital MMSE transceiver beamforming. Secrecy in cell-free massive MIMO systems was proposed in [16] through the joint optimization of uplink power control, downlink beamforming, and duplex mode selection. Artificial noise was incorporated, and an iterative method was used to solve the associated nonconcave–convex approximation. In [17], beamforming was investigated for in-band FD multi-cell multiuser MIMO networks. Hardware impairments, channel uncertainty, and limited channel state information (CSI) were considered in the MMSE beamforming design to improve system performance under global and local CSI. However, none of the approaches in the literature considered the practical limitation of residual loop interference due to imperfect CSI cancellation. This interference should be considered in developing robust and effective wireless communication systems. The impact of residual self-interference (SI) and loop interference (LI) in a full-duplex space shift keying communication system was investigated in [18]. Thus, the residual LI due to imperfect interference cancellation is considered in this paper.

Unlike the uplink communications in [19], this paper considers a MIMO FD relaying downlink communication system with multiple source–destination pairs. A transceiver design for the sources and relay, and a linear combiner design at the destinations, are investigated considering the residual LI with the goal of minimizing the MSE of the received signals at the destinations. An iterative algorithm is developed to update the source and relay transceiver matrices and the linear combining matrix at the destinations.

The remainder of this paper is organized as follows. Section 2 presents the system model, and the optimization problem is solved in Section 3. Results are presented in Section 4, which demonstrate the effectiveness of this solution, and some conclusions are given in Section 5.

Notation: Bold uppercase, bold lowercase, and normal letters denote matrices, vectors, and scalars, respectively. $\mathrm{vec}(·)$ denotes matrix vectorization, ⊗ denotes the matrix Kronecker product, $\mathrm{tr}\{·\}$ is the trace of a matrix, and ${\mathbf{I}}_N$ is the $N\times N$ identity matrix.

2. System Model

Consider a MIMO full-duplex (FD) relay system with K source–destination pairs communicating simultaneously with the aid of a relay. The source-to-destination links are assumed to be inconsequential due to the large-scale fading and the large distances between them.

Figure 1 presents the system model with the kth source–destination pairs equipped with $N_{s_k}$ and $N_{d_k}$ antennas, respectively. The FD relay has $N_r$ and $N_t$ antennas to simultaneously receive and transmit signals, respectively. Therefore, communication between the source–destination pairs is accomplished in one time slot compared to an HD system that requires two time slots.

Let ${\mathbf{s}}_k[n]$ represent the length d signal vector at time n for the kth source. A linear transceiver matrix ${\mathbf{\Theta }}_k[n]$ is applied to ${\mathbf{s}}_k[n]$ before transmission. The received signal at the relay can then be expressed as

$${\mathbf{y}}_r[n]=\sum _{k=1}^K{\mathbf{H}}_k[n]{\mathbf{\Theta }}_k[n]{\mathbf{s}}_k[n]+{\mathbf{R}}_{LI}[n]\mathbf{t}[n]+{\mathbf{n}}_r[n],$$

(1)

where ${\mathbf{H}}_k[n]\in {\mathbb{C}}^{N_r\times N_{s_k}}$ is the channel between source k and the relay node, ${\mathbf{R}}_{LI}[n]\in {\mathbb{C}}^{N_r\times N_t}$ is the loop interference (LI) channels, and ${\mathbf{n}}_r[n]\in {\mathbb{C}}^{N_r\times 1}$ is an independent and identically distributed (i.i.d.) additive white Gaussian noise (AWGN) matrix. As the received signal ${\mathbf{y}}_r[n]$ is corrupted by residual LI, from [12], it can be written as

$${\mathbf{y}}_r[n]=\sum _{k=1}^K{\mathbf{H}}_k[n]{\mathbf{\Theta }}_k[n]{\mathbf{s}}_k[n]+{\mathbf{R}}_{LI}[n]\mathrm{\Delta }\mathbf{t}[n]+{\mathbf{n}}_r[n],$$

(2)

where $\mathrm{\Delta }\mathbf{t}[n]=\mathrm{t}[n]-\tilde{\mathbf{t}}[n]$, and ${\mathbf{R}}_{LI}[n]\mathrm{\Delta }\mathbf{t}[n]$ is the residual LI. At time $n+1$, the relay amplifies the received signal with a relay transceiver matrix $\mathbf{\Gamma }[n+1]\in {\mathbb{C}}^{N_t\times N_r}$ and forwards the resulting signal to the destinations. The received signal at the kth destination is then

$$\begin{array}{c}\hfill {\mathbf{y}}_k[n+1]={\mathbf{G}}_k[n+1]\mathbf{\Gamma }[n+1]{\mathbf{y}}_r[n]+{\mathbf{n}}_{d_k}[n+1],\end{array}$$

(3)

where ${\mathbf{G}}_k[n+1]\in {\mathbb{C}}^{N_i\times N_t}$ is the channel matrix between the relay and destination k, and ${\mathbf{n}}_{d_k}[n+1]\in {\mathbb{C}}^{N_k\times 1}$ is an i.i.d. AWGN vector with a zero mean and unit variance. As in [6,7,12], channel state information (CSI) is assumed to be available at all nodes. In addition, channel variations during the transceiver matrix update interval are assumed to be relatively small [19] and so do not influence the transceiver design [12]. Thus, the received signal at the kth destination can be expressed as

$${\mathbf{y}}_k={\mathbf{G}}_k\mathbf{\Gamma }\sum _{k=1}^K{\mathbf{H}}_k{\mathbf{\Theta }}_k{\mathbf{s}}_k+{\mathbf{n}}_k,$$

(4)

where

$${\mathbf{n}}_k={\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{R}}_{LI}\mathrm{\Delta }\mathbf{t}+{\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{n}}_r+{\mathbf{n}}_{d_k}.$$

(5)

As in [18,20], $\mathrm{\Delta }\mathbf{t}$ is modeled as white Gaussian noise which is independent of ${\mathbf{n}}_r$ and ${\mathbf{n}}_{d_k}$, and has variance ${\sigma }_t^2$. Consequently, the covariance matrix of ${\mathbf{n}}_k$ is

$$\begin{array}{c}{\mathbf{C}}_{n_k}=\mathbb{E}[{\mathbf{n}}_k{\mathbf{n}}_k^H].\end{array}$$

(6)

A linear combiner ${\mathbf{M}}_k\in {\mathbb{C}}^{N_k\times N_b}$ is used at the kth destination to obtain the signal estimate

$${\widehat{\mathbf{s}}}_k={\mathbf{M}}_k^H{\mathbf{y}}_k.$$

(7)

The mean square error (MSE) at the kth user is defined as

$$\begin{array}{rr}\hfill {\mathrm{MSE}}_k& =\mathbb{E}[({\widehat{\mathbf{s}}}_k-{\mathbf{s}}_k){({\widehat{\mathbf{s}}}_k-{\mathbf{s}}_k)}^H]\hfill \\ \hfill & =({\mathbf{M}}_k^H{\mathbf{L}}_k-{\mathbf{I}}_d){({\mathbf{M}}_k^H{\mathbf{L}}_k-{\mathbf{I}}_d)}^H+{\mathbf{M}}_k^H{\mathbf{C}}_{n_k}{\mathbf{M}}_k\hfill \\ \hfill & +{\mathbf{M}}_k^H{\mathbf{E}}_{n_k}{\mathbf{M}}_k,\hfill \end{array}$$

(8)

where ${\mathbf{L}}_k={\mathbf{G}}_k\mathbf{\Gamma }{\overline{\mathbf{H}}}_k$ and ${\mathbf{E}}_{n_k}={\mathbf{G}}_k\mathbf{\Gamma }{\sum }_{m=1,m\ne k}^K{\overline{\mathbf{H}}}_m{\overline{\mathbf{H}}}_m^H{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H,k=1,\dots ,K$. Optimization is employed to obtain the linear transceiver matrices ${\mathbf{\Theta }}_k$ and $\mathbf{\Gamma }$ at the source and relay, and to obtain the linear combiners ${\mathbf{M}}_k$ at the destinations. The minimization of the sum mean squared error (SMSE) of the received signals at the destinations can be expressed as

$$\begin{array}{}\mathrm{(9a)}& \hfill \underset{{\mathbf{\Theta }}_k,\mathbf{\Gamma },{\mathbf{M}}_k}{min}& \mathrm{SMSE}\hfill \mathrm{(9b)}& \hfill s.t.& \mathrm{tr}(\mathbf{\Gamma }({\mathbf{y}}_r{\mathbf{y}}_r^H){\mathbf{\Gamma }}^H)\le P_r\hfill \mathrm{(9c)}& \hfill & \mathrm{tr}({\mathbf{\Theta }}_k{\mathbf{\Theta }}_k^H)\le P_{s_k},k=1,\dots ,K,\hfill \end{array}$$

where $\mathrm{SMSE}={\sum }_{k=1}^K\mathrm{tr}({\mathrm{MSE}}_k)$, and $P_{s_k}>0$ and $P_r>0$ are the power constraints at the kth source and relay, respectively.

3. Optimization Problem Solution

The problem in (9) is highly nonconvex which makes obtaining a global optimal solution intractable, so an iterative algorithm is presented. This solution is based on an alternating optimization that updates ${\mathbf{\Theta }}_k$, $\mathbf{\Gamma }$ and ${\mathbf{M}}_k$ individually with the other parameters kept fixed to solve the three convex subproblems.

First, given ${\mathbf{\Theta }}_k$ and $\mathbf{\Gamma }$, the optimal combiner at the kth destination ${\mathbf{M}}_k$ is obtained by solving the unconstrained convex problem for ${\mathbf{M}}_k$, which is independent of the constraints in (9b) and (9c). The optimal solution can then be obtained by taking the derivative of (8) with respect to ${\mathbf{M}}_k$ and setting it to zero. Solving $\frac{\partial }{\partial {\mathbf{M}}_k}\mathrm{tr}({\mathrm{MSE}}_k)=0$ gives

$${\mathbf{M}}_k={\mathbf{C}}_{{\mu }_k}^{-1}{\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{H}}_k{\mathbf{\Theta }}_k,$$

(10)

where

$$\begin{array}{c}{\mathbf{C}}_{{\mu }_k}={({\mathbf{G}}_k\mathbf{\Gamma }\mathbf{H}{\mathbf{\Theta }}_k{\mathbf{\Theta }}_k^H{\mathbf{H}}_k^H{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H+{\mathbf{C}}_{n_k}+{\mathbf{E}}_{n_k})}^{-1}{\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{H}}_k{\mathbf{\Theta }}_k,\hfill \end{array}$$

which is known as the Wiener filter.

Second, with ${\mathbf{M}}_k$ from (10) and given ${\mathbf{\Theta }}_k$, $\mathbf{\Gamma }$ can be obtained by solving the following problem. The objective function in (9) can be expressed as

$$\begin{array}{cc}\hfill \mathrm{SMSE}=& \sum _{k=1}^K\mathrm{tr}(({\mathbf{M}}_k^H{\mathbf{G}}_k\mathbf{\Gamma }\mathbf{H}{\mathbf{\Theta }}_k-{\mathbf{I}}_{N_k})({\mathbf{M}}_k^H{\mathbf{G}}_k\mathbf{\Gamma }\mathbf{H}{\mathbf{\Theta }}_k\hfill \\ \hfill & -{\mathbf{I}}_{N_k}{)}^H+{\mathbf{M}}_k^H({\sigma }_t^2{\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H+\hfill \\ \hfill & {\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H+{\mathbf{I}}_{N_k}){\mathbf{M}}_k+{\mathbf{M}}_k^H{\mathbf{G}}_k\mathbf{\Gamma }\times \hfill \\ \hfill & \sum _{m=1,m\ne k}^K\mathbf{H}{\mathbf{\Theta }}_m{\mathbf{\Theta }}_m^H{\mathbf{H}}^H{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H{\mathbf{M}}_k).\hfill \end{array}$$

(11)

Note that since ${\mathbf{\Theta }}_k$ is known, the constraint in (9c) is eliminated. The optimization problem then becomes

$$\begin{array}{cc}\hfill \underset{\mathbf{\Gamma }}{min}& \mathrm{SMSE}\hfill \\ \hfill s.t.& \mathrm{tr}(\mathbf{\Gamma }(\sum _{k=1}^K{\mathbf{H}}_k{\mathbf{\Theta }}_k{\mathbf{\Theta }}_k^H{\mathbf{H}}_k^H+{\sigma }_t^2{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H+{\mathbf{I}}_{N_r}){\mathbf{\Gamma }}^H)\le P_r.\hfill \end{array}$$

(12)

Consider the singular-value decomposition (SVD) of the equivalent source-to-relay and relay-to-destination channels:

$$\mathbf{H}=[{\mathbf{H}}_1{\mathbf{\Theta }}_1,\dots ,{\mathbf{H}}_K{\mathbf{\Theta }}_K]={\mathbf{U}}_h{\mathrm{\Lambda }}_h{\mathbf{V}}_h^H,$$

(13)

and

$$\mathbf{G}={[{\mathbf{G}}_1^T,\dots ,{\mathbf{G}}_K^T]}^T={\mathbf{U}}_g{\mathrm{\Lambda }}_g{\mathbf{V}}_g^H.$$

(14)

The optimal relay precoding matrix is similar to that in [7]:

$$\mathbf{\Gamma }={\mathbf{V}}_g\mathbf{A}{\mathbf{U}}_h^H,$$

(15)

and from (13) and (14), we have

$${\mathbf{H}}_k{\mathbf{\Theta }}_k={\mathbf{U}}_h{\mathrm{\Lambda }}_h{\mathbf{V}}_{h_k}^H,$$

(16)

and

$${\mathbf{G}}_k={\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g{\mathbf{V}}_g^H.$$

(17)

Note that

$${\mathbf{V}}_h={[{\mathbf{V}}_{h1}^T,\dots ,{\mathbf{V}}_{h_K}^T]}^T,$$

and

$${\mathbf{U}}_g={[{\mathbf{U}}_{g1}^T,\dots ,{\mathbf{U}}_{g_K}^T]}^T,$$

which have dimensions $d\times L_1$ and $N_i\times L_2$, respectively. Substituting (13)–(15) in (11) gives

$$\begin{array}{cc}\hfill \mathrm{SMSE}=& \sum _{k=1}^K\mathrm{tr}(({\mathbf{M}}_k^H{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g\mathbf{A}{\mathrm{\Lambda }}_h{\mathbf{V}}_{h_k}^H-{\mathbf{I}}_{N_k})\times \hfill \\ \hfill & {({\mathbf{M}}_k^H{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g\mathbf{A}{\mathrm{\Lambda }}_h{\mathbf{V}}_{h_k}^H-{\mathbf{I}}_{N_k})}^H\hfill \\ \hfill & +{\mathbf{M}}_k^H({\sigma }_t^2{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g\mathbf{A}{\mathbf{U}}_h^H{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H{\mathbf{U}}_h{\mathbf{A}}^H{\mathrm{\Lambda }}_g{\mathbf{U}}_{g_k}^H\hfill \\ \hfill & +{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g\mathbf{A}{\mathbf{A}}^H{\mathrm{\Lambda }}_g{\mathbf{U}}_{g_k}^H+{\mathbf{I}}_{N_k}+{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g\mathbf{A}\hfill \\ \hfill & \times \sum _{m=1,m\ne k}^K{\mathrm{\Lambda }}_h{\mathbf{V}}_{h_m}^H{\mathbf{V}}_{h_m}{\mathrm{\Lambda }}_h{\mathbf{A}}^H{\mathrm{\Lambda }}_g{\mathbf{U}}_{g_k}^H){\mathbf{M}}_k).\hfill \end{array}$$

(18)

Using the matrix identities

$$\begin{array}{ccc}\mathrm{tr}({\mathbf{C}}^T\mathbf{D})& =& {((\mathbf{C}))}^H\mathrm{vec}(\mathbf{D}),\hfill \\ \mathrm{tr}({\mathbf{A}}^H\mathbf{B}\mathbf{A}\mathbf{C})& =& \mathrm{tr}{(\mathrm{vec}(\mathbf{A}))}^H({\mathbf{C}}^T\otimes \mathbf{B})\mathrm{vec}(\mathbf{A})\hfill \\ \mathrm{vec}\mathbf{A}\mathbf{B}\mathbf{C}& =& ({\mathbf{C}}^T\otimes \mathbf{A})\mathrm{vec}(\mathbf{B}),\hfill \end{array}$$

where $\mathrm{vec}(·)$ concatenates the columns of a matrix into a single vector, the SMSE in (11) can be expressed as a function of $\mathbf{a}=\mathrm{vec}(\mathbf{A})$:

$$\begin{array}{cc}\hfill \mathrm{SMSE}=& \sum _{k=1}^K{({\mathbf{P}}_k\mathbf{a}-\mathrm{vec}({\mathbf{I}}_{N_k}))}^H({\mathbf{P}}_k\mathbf{a}-\mathrm{vec}({\mathbf{I}}_{N_k}))\hfill \\ \hfill +& {\mathbf{a}}^H{\mathbf{Q}}_k\mathbf{a}+{\mathbf{a}}^H{\mathbf{S}}_k\mathbf{a}+{\mathbf{a}}^H{\mathbf{R}}_k\mathbf{a}+{\mathbf{t}}_1\hfill \end{array}$$

(19)

where

$${\mathbf{t}}_1=\sum _{k=1}^K\mathrm{tr}({\mathbf{M}}_k^H{\mathbf{M}}_k),$$

does not depend on $\mathbf{a}$ and

$${\mathbf{P}}_k={({\mathrm{\Lambda }}_h{\mathbf{V}}_{h_k}^H)}^T\otimes ({\mathbf{M}}_k^H{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g),$$

(20)

$${\mathbf{Q}}_k={\sigma }_t^2({({\mathbf{U}}_h^H{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H{\mathbf{U}}_h)}^T\otimes {\mathrm{\Lambda }}_g{\mathbf{U}}_{g_k}^H{\mathbf{M}}_k{\mathbf{M}}_k^H{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g),$$

(21)

$${\mathbf{S}}_k={\mathbf{I}}_L\otimes ({\mathrm{\Lambda }}_g{\mathbf{U}}_{g_k}^H{\mathbf{M}}_k{\mathbf{M}}_k^H{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g),$$

(22)

$${\mathbf{R}}_k={(\sum _{m=1,m\ne k}^K{\mathrm{\Lambda }}_h{\mathbf{V}}_{h_m}^H{\mathbf{V}}_{h_m}{\mathrm{\Lambda }}_h)}^T\otimes ({\mathrm{\Lambda }}_g{\mathbf{U}}_{g_k}^H{\mathbf{M}}_k{\mathbf{M}}_k^H.{\mathbf{U}}_{g_k}{\mathrm{\Lambda }}_g).$$

(23)

The relay transmit power constraint in (12) can be written as

$$\begin{array}{cc}\hfill \mathrm{tr}(\mathbf{\Gamma }& (\sum _{k=1}^K{\mathbf{H}}_k{\mathbf{\Theta }}_k{\mathbf{\Theta }}_k^H{\mathbf{H}}_k^H+{\sigma }_t^2{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H+{\mathbf{I}}_{N_r}){\mathbf{\Gamma }}^H)\hfill \\ \hfill & =\mathrm{tr}(\mathbf{A}\times ({\mathrm{\Lambda }}_h^2+{\sigma }_t^2{\mathbf{U}}_h^H{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H{\mathbf{U}}_h+{\mathbf{I}}_{L_1}){\mathbf{A}}^H).\hfill \end{array}$$

(24)

Defining $\mathbf{D}=({\mathrm{\Lambda }}_h^2+{\sigma }_t^2{\mathbf{U}}_h^H{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H{\mathbf{U}}_h+{\mathbf{I}}_{L_1})\otimes {\mathbf{I}}_{L_1}$, (24) can be written as

$${\mathbf{a}}^H\mathbf{D}\mathbf{a}\le P_r.$$

(25)

The original relay optimization problem is then given by

$$\begin{array}{cc}\hfill \underset{\mathbf{A}}{min}& \mathrm{SMSE}\hfill \\ \hfill s.t.& {\mathbf{a}}^H\mathbf{D}\mathbf{a}\le P_r\hfill \end{array}$$

(26)

This is a quadratically constrained quadratic programming (QCQP) problem which is convex and so can be efficiently solved using the interior point method. For example, the CVX toolbox for disciplined convex programming [21] can be employed.

Third, the kth source precoding matrix ${\mathbf{\Theta }}_k$ can be obtained using ${\mathbf{M}}_k$ and $\mathbf{\Gamma }$ given above. The corresponding optimization problem can be formulated as the following QCQP problem:

$$\begin{array}{cc}\hfill \mathrm{SMSE}=& \sum _{k=1}^K\mathrm{tr}(({\overline{\mathbf{G}}}_k\mathbf{\Gamma }{\mathbf{H}}_k{\mathbf{\Theta }}_k-{\mathbf{I}}_{N_k}){({\overline{\mathbf{G}}}_k\mathbf{\Gamma }{\mathbf{H}}_k{\mathbf{\Theta }}_k-{\mathbf{I}}_{N_k})}^H\hfill \\ \hfill & +{\overline{\mathbf{G}}}_k\mathbf{\Gamma }\sum _{m=1,m\ne k}^K{\mathbf{H}}_k{\mathbf{\Theta }}_k{\mathbf{\Theta }}_k^H{\mathbf{H}}_k^H{\mathbf{\Gamma }}^H{\overline{\mathbf{G}}}_k^H)+{\mathbf{t}}_2,\hfill \end{array}$$

(27)

where ${\overline{\mathbf{G}}}_k={\mathbf{M}}_k^H{\mathbf{G}}_k$ and ${\mathbf{t}}_2={\sum }_{k=1}^K\mathrm{tr}({\mathbf{M}}_k^H{\mathbf{C}}_k{\mathbf{M}}_k)$ can be ignored in the optimization as they do not depend on ${\mathbf{\Theta }}_k$. Using the matrix identities, (27) can be written as a function of ${\theta }_k=\mathrm{vec}({\mathbf{\Theta }}_k)$ which gives

$$\begin{array}{cc}\hfill \mathrm{SMSE}=& \sum _{k=1}^K[{({\mathbf{S}}_k{\theta }_k-\mathrm{vec}({\mathbf{I}}_{N_k}))}^H({\mathbf{S}}_k{\theta }_k-\mathrm{vec}({\mathbf{I}}_{N_k}))\hfill \\ \hfill & +\sum _{m=1,m\ne k}^K{\theta }_m^H({\mathbf{I}}_d\otimes {\mathbf{H}}_m^H{\mathbf{\Gamma }}^H{\overline{\mathbf{G}}}_k^H{\overline{\mathbf{G}}}_k\mathbf{\Gamma }{\mathbf{H}}_m){\theta }_m]\hfill \\ \hfill & +{\mathbf{t}}_2\hfill \\ \hfill =& \sum _{k=1}^K[{({\mathbf{S}}_k{\theta }_k-\mathrm{vec}({\mathbf{I}}_{N_k}))}^H({\mathbf{S}}_k{\theta }_k-\mathrm{vec}({\mathbf{I}}_{N_k}))\hfill \\ \hfill & +{\theta }_k^H{\mathbf{T}}_k{\theta }_k]+{\mathbf{t}}_2,\hfill \end{array}$$

(28)

where

$${\mathbf{S}}_k={\mathbf{I}}_{N_k}\otimes ({\overline{\mathbf{G}}}_k\mathbf{\Gamma }\mathbf{H}),$$

$${\mathbf{T}}_k={\mathbf{I}}_{N_k}\otimes \sum _{m=1,m\ne k}^K({\mathbf{H}}^H{\mathbf{\Gamma }}^H{\overline{\mathbf{G}}}_k^H{\overline{\mathbf{G}}}_k\mathbf{\Gamma }\mathbf{H}),$$

and

$${\mathbf{t}}_2=\sum _{k=1}^K\mathrm{tr}({\mathbf{M}}_k^H{\mathbf{C}}_{n_k}{\mathbf{M}}_k),$$

which is independent of ${\mathbf{\Theta }}_k$ and so can be ignored. Introducing $\mathbf{T}=\mathrm{bd}({\mathbf{T}}_1,\dots ,{\mathbf{T}}_K)$, where $\mathrm{bd}(·)$ denotes a block-diagonal matrix, and ${\overline{\mathbf{S}}}_k=[{\mathbf{S}}_{k1},\dots ,{\mathbf{S}}_{kK}]$, where ${\mathbf{S}}_{kk}={\mathbf{S}}_k$ and ${\mathbf{S}}_{jk}=0,j\ne k$, (27) can be written as a function of $\theta ={[{\theta }_1^T,{\theta }_1^T,\dots ,{\theta }_K^T]}^T$, which gives

$${\mathrm{\Phi }}_1(\theta )=\sum _{k=1}^K{({\overline{\mathbf{S}}}_k\theta -\mathrm{vec}({\mathbf{I}}_{N_k}))}^H({\overline{\mathbf{S}}}_k\theta -\mathrm{vec}({\mathbf{I}}_{N_k}))+{\theta }^H\mathbf{T}\theta$$

(29)

Now, defining

$${\mathbf{E}}_j={\mathbf{I}}_{N_k}\otimes ({\mathbf{H}}_j^H{\mathbf{\Gamma }}^H\mathbf{\Gamma }{\mathbf{H}}_j),$$

$$\mathbf{E}=\mathrm{bd}({\mathbf{E}}_1,{\mathbf{E}}_2,\dots ,{\mathbf{E}}_K),$$

and

$$\overline{\mathbf{E}}=\mathrm{bd}({\overline{\mathbf{E}}}_{i1},{\overline{\mathbf{E}}}_{i2},\dots ,{\overline{\mathbf{E}}}_{iK}),$$

where ${\overline{\mathbf{E}}}_{iK}={\mathbf{I}}_{d{N}_s}$ and ${\overline{\mathbf{E}}}_{ij}=0,i\ne j$, the optimal $\theta$ can be obtained by solving the following problem

$$\begin{array}{cc}\hfill \underset{\theta }{min}& {\mathrm{\Phi }}_1(\theta )\hfill \\ \hfill s.t.& \sum _{i=1}^K{\theta }^H{\overline{\mathbf{E}}}_i\theta \le P_s\hfill \\ \hfill & {\theta }^H\mathbf{E}\theta \le P_r-{\sigma }_r^2\mathrm{tr}(\mathbf{\Gamma }({\sigma }_t^2{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H+{\mathbf{I}}_{N_r}){\mathbf{\Gamma }}^H).\hfill \end{array}$$

(30)

This is a QCQP problem which can be solved using the CVX toolbox for disciplined convex programming.

The symbols employed in this work are summarized in

Table 1. Symbol descriptions.

Symbol	Description
${\mathbf{s}}_k$	Signal vector for the kth source
${\mathbf{\Theta }}_k$	Linear transceiver matrix at the source for the kth user
${\mathbf{H}}_k$	Channel between source k and the relay node
${\mathbf{R}}_{LI}$	Loop interference (LI) channels
${\mathbf{n}}_r$	i.i.d. additive white Gaussian noise (AWGN) matrix
${\mathbf{M}}_k$	Linear combiner at the destination
$\mathbf{\Gamma }$	Transceiver matrix at the relay for the kth user
${\mathbf{M}}_k$	Linear combiners at the destinations for the kth user
R	Achievable rate

. The proposed iterative algorithm is given in Algorithm 1. The convergence of this algorithm is guaranteed as follows. As the three subproblems are convex, each update of ${\mathbf{\Theta }}_k$, $\mathbf{\Gamma }$, and ${\mathbf{M}}_k$ may decrease or at least not increase the value of the objective function. Thus, the iterative algorithm will converge to an optimum solution.

Algorithm 1: Iterative Design of ${\mathbf{\Theta }}_k$, $\mathbf{\Gamma }$, and ${\mathbf{M}}_k$.

1: Initialize the algorithm with ${\mathbf{\Theta }}^{(0)}=\sqrt{\frac{P_s}{L}}{\mathbf{I}}_L$ and ${\mathbf{\Gamma }}^{(0)}=\sqrt{\frac{P_r}{\mathrm{tr}{({\mathbf{H}}_{SR}{\mathbf{\Theta }}_k^{(0)}{({\mathbf{H}}_{SR}{\mathbf{\Theta }}_k^{(0)})}^H+{\sigma }_t^2{\mathbf{R}}_{LI}{\mathbf{R}}_{LI})}^H+{\mathbf{I}}_{N_r})}}{\mathbf{I}}_{N_r}$, and set $n=0$.    2: Update ${\mathbf{M}}_k^{(n)}$ using ${\mathbf{\Gamma }}^{(n)}$ and ${\mathbf{\Theta }}_k^{(n)}$ using (10).    3: Update ${\mathbf{\Gamma }}^{(n+1)}$ using ${\mathbf{M}}_k^{(n)}$ and ${\mathbf{\Theta }}_k^{(n)}$, $k=1,\dots ,K$, in (15).    4: Update ${\mathbf{\Theta }}_k^{(n+1)}$ using ${\mathbf{M}}_k^{(n)}$ and ${\mathbf{\Gamma }}^{(n+1)}$ by solving the problem in (30).    5: If $({\mathrm{SMSE}}^{(n)}-{\mathrm{SMSE}}^{(n+1)})/{\mathrm{SMSE}}^{(n)}>ϵ$, go to step 2.    6: End

4. Numerical Results

This section presents the evaluation of the performance of the proposed multiuser MIMO FD relay downlink algorithm using numerical simulation. The CVX toolbox was employed, which is a Matlab-based software package for solving convex optimization problems [21]. For simplicity, we considered a system with two sources and two destinations. The extension to more than two source–destination pairs is straightforward. We assumed that the sources and destinations were each equipped with two antennas, and the relay was equipped with four receive antennas and four transmit antennas. The simulation parameters are given in

Table 2. Simulation parameters.

Parameter	Value
Downlink/uplink users	2
Relay antennas	2 × 2
Downlink/uplink user antennas	2 × 2
LI levels	0 dB, 5 dB, 10 dB
No. of independent channel realizations	1000
Source–relay and relay–source SNR	0 dB, 5 dB, 10 dB, 15 dB, 20 dB, 25 dB, 30 dB

.

As in the related literature, flat-fading MIMO channels were considered. It was assumed that the elements of ${\mathbf{H}}_k$, ${\mathbf{G}}_k$, and ${\mathbf{R}}_{LI}$ were i.i.d. complex Gaussian random variables with a zero mean and unit variance. Moreover, all noise terms were i.i.d. complex, circularly symmetric Gaussian random variables with a zero mean and unit variance. According to [9], the residual LI can be 0 dB to 20 dB larger than the channel noise. Therefore, LI levels of 0 dB, 5 dB and 10 dB were employed here. In all cases, the results are given for an average of 1000 independent channel realizations. Note that the HD system mentioned in this paper is the same as the proposed FD system except that the residual LI term is zero and the achievable rate of the HD system is half of the FD system since two time slots are required for transmission between a source and destination. Thus, the same optimization procedure is employed.

Figure 2 presents the SMSE for the HD and FD systems with different levels of residual LI. The signal-to-noise ration (SNR) between the sources and relay varies from 0 dB to 30 dB, and the SNR between the relay and destinations is fixed at 30 dB. These results indicate that the HD system provides the best SMSE as the FD performance is degraded as the LI increases.

The achievable rate for the system in (5) can be obtained using an approach similar to that in [8], and can be written as

$$R=\sum _{k=1}^K{R}_k,$$

(31)

where

$$\begin{array}{cc}\hfill R_k& ={log}_{2}det\left[{\mathbf{I}}_{N_d}\right.+({\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{H}}_k{\mathbf{\Theta }}_k){({\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{H}}_k{\mathbf{\Theta }}_k)}^H\hfill \\ \hfill & \times {({\sigma }_t^2{\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{R}}_{LI}{\mathbf{R}}_{LI}^H{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H+{\mathbf{G}}_k\mathbf{\Gamma }{\mathbf{\Gamma }}^H{\mathbf{G}}_k^H+{\mathbf{I}}_{N_d})}^{-1}].\hfill \end{array}$$

(32)

Figure 3 gives the achievable rate for the HD and FD systems. In this figure, the SNR between the sources and relay varies from 0 dB to 30 dB while the SNR between the relay and destinations is fixed at 30 dB. The HD system corresponds to the case when the residual LI is zero and two time slots are required for transmission from sources to destinations. Therefore, the FD achievable rate is twice the HD achievable rate if the LI is cancelled completely. Further, the achievable rate with the FD system is higher when the residual LI is 0 dB to 10 dB. The achievable rate of the FD system is degraded as the residual LI increases in the high-SNR region since the residual LI is greater than the multiuser interference. In the low-SNR region, the multiuser interference dominates the residual LI, so the effect of the residual LI is negligible.

Figure 4 and Figure 5 present the SMSE and achievable rate with a fixed SNR of 30 dB between the sources and relay and an SNR between the relay and destinations from 0 dB to 30 dB. The SMSE in Figure 4 is better than in Figure 2 in the low-SNR region because a higher transmit power results in more residual LI. Figure 5 shows that the achievable rate of the FD system is always higher than that of the HD system for residual LI levels up to 10 dB. These results confirm that the proposed approach is effective in addressing the effects of the residual loop interference due to imperfect LI cancellation in full-duplex systems. Further, this solution is superior to conventional half-duplex communications.

Table 3. Percentage improvement in achievable rate.

Residual LI	${\mathbf{SNR}}_{\mathit{r}-\mathit{d}}$ = 30 dB with ${\mathbf{SNR}}_{\mathit{s}-\mathit{r}}$ = 27 dB	${\mathbf{SNR}}_{\mathit{r}-\mathit{d}}$ = 27 dB with ${\mathbf{SNR}}_{\mathit{s}-\mathit{r}}$ = 30 dB
FD No LI	$100\%$	$100\%$
FD LI = 0 dB	$76\%$	$86\%$
FD LI = 5 dB	$36\%$	$58\%$
FD LI = 10 dB	$4\%$	$9\%$

gives the percentage improvement in achievable rate with the proposed system over the conventional half-duplex system for different levels of residual LI and SNRs. These results demonstrate the robustness of the proposed design as the improvements exceed 75% when the residual LI is low. Further, they confirm the effectiveness in optimizing performance in practical scenarios.

5. Conclusions

This paper considers a transceiver design for the downlink of a multiuser non-regenerative MIMO full-duplex (FD) relay system with residual loop interference (LI). The source and relay transceiver matrices and destination combining matrix are optimized to minimize the sum mean squared error (SMSE). The original nonconvex problem is converted to three convex subproblems, and an iterative algorithm is developed to optimize the three matrices. Results are presented which demonstrate that the proposed iterative method outperforms the corresponding half-duplex (HD) relay system in terms of the SMSE and achievable rate. The effectiveness of the proposed solution was illustrated via simulation. Future research can consider interference cancellation techniques, integrating machine learning for dynamic system optimization, and next-generation wireless technologies. Internet of things (IoT) and mobile communication applications can also be examined considering the adaptability and scalability of the proposed approach and to further confirm its impact.