Closed-Form Expression for the Symbol Error Probability in Full-Duplex Spatial Modulation Relay System and Its Application in Optimal Power Allocation

Abstract

Full-duplex (FD) communication and spatial modulation (SM) are two promising techniques to achieve high spectral efficiency. Recent works in the literature have investigated the possibility of combining the FD mode with SM in the relay system to benefit their advantages. In this paper, we analyze the performance of the FD-SM decode-and-forward (DF) relay system and derive the closed-form expression for the symbol error probability (SEP). To tackle the residual self-interference (RSI) due to the FD mode at the relay, we propose a simple yet effective power allocation algorithm to compensate for the RSI impact and improve the system SEP performance. Both numerical and simulation results confirm the accuracy of the derived SEP expression and the efficacy of the proposed optimal power allocation.

1. Introduction

Full-duplex (FD) communication and spatial modulation (SM) are two promising techniques to increase the spectral efficiency of wireless systems [1,2,3]. Theoretically, an FD communication system can double the spectral efficiency as its transceivers can transmit and receive signals at the same time and on the same frequency [4,5]. Besides increasing the channel capacity, the FD communication systems can also reduce the end-to-end and feedback delay, improve network security and solve the hidden terminal problem. Therefore, the FD mode has found its applications in various wireless systems such as sensor networks, massive MIMO, relaying networks and possibly future wireless networks such as the fifth-generation (5G) and beyond. Unfortunately, the residual self-interference (RSI) due to imperfect self-interference cancellation (SIC) limits the capacity and performance of the FD communication systems [6]. In the literature, numerous solutions such as relay selection scheme and adaptive transmission [7,8] and optimal power allocation [9,10,11] were proposed to reduce the impact of the RSI and improve the performance of the FD relay communication system. These solutions significantly improved the capacity, outage performance and energy efficiency of the single-input single-output (SISO) systems.

Meanwhile, the multiple-input multiple-output (MIMO) transmission is another solution that can also achieve high capacity and better performance over the SISO systems [12]. However, the hardware of the MIMO system is more complex, that is, more radio frequency (RF) chains, as both transmitter and receiver user multiple antennas. The MIMO receiver also requires a high-complexity detector to deal with the inter-channel interference (ICI) due to simultaneous transmissions on the same frequency from different antennas. In that context, SM can tackle these issues correctly as it activates only one transmit antenna for transmission and requires only low-complexity iterative maximal-ratio combining (MRC) at the receiver for signal estimation [12,13,14,15]. Therefore, the combined SM and FD system achieves higher spectral efficiency while reducing the complexity requirement at the transceiver, particularly of the relay node.

In the literature, using FD relay in SM systems has attracted great interest because this integrated FD-SM system has the advantages of both spectral efficiency improvement and performance enhancement. Numerous works focused on analyzing the system performance in terms of the outage probability ($\mathrm{OP}$) [2], bit error rate ($\mathrm{BER}$) [3], average symbol error probability ($\mathrm{SEP}$), ergodic capacity [16] and average $\mathrm{BER}$ [17,18]. Specifically, in Reference [2], the lower and upper bounds of the $\mathrm{OP}$ of the SM-MIMO system with decode-and-forward (DF) FD relay were derived over cascaded $\alpha -\mu$ fading channels. It also demonstrated that the RSI had a strong impact on the $\mathrm{OP}$ performance of the system. The work in Reference [3] considered the SM-MIMO system with amplify-and-forward (AF) FD/half-duplex (HD) relaying. It successfully derived a new unified tight upper-bound for the system $\mathrm{BER}$. The results of the paper indicate that the SM-MIMO-FD relay system can improve the $\mathrm{BER}$ and the spectral efficiency if suitable SIC techniques are applied. Under the same assumption of the RSI, References [16,17,18] investigated the SM-MIMO-FD relay systems which can exploit the benefits of the FD transmission mode. The approximate expressions of $\mathrm{SEP}$ ([16]) and $\mathrm{BER}$ ([17,18]) were also derived for performance evaluation.

Although the previous works conducted various performance analyses, their results were limited to either upper and lower bounds or approximate expressions but not the exact closed-form expressions of SEP and BER. Therefore, it is required to have exact mathematical expressions for the performance evaluation rather than the upper bound or approximate ones for better understanding the system behaviors. Moreover, since the FD mode significantly degrades the system performance, besides effective SIC techniques, there should be other solutions such as power allocation to compensate for this degradation. Motivated by the previous works, in this paper, we aim to derive the exact closed-form expression of the SEP of the SM-FD relay system with DF protocol applied at the relay to enhance the system performance. Based on the derived expression of SEP, we can determine the optimal power allocation for the FD relay to reduce the RSI impact on the $\mathrm{SEP}$ performance. So far, this is the first work that successfully derives the exact closed-form expression of SEP and use it for optimizing the power allocation for SM-MIMO-FD relay systems. The main contributions of this paper can be summarized as follows:

• We analyze the SM-MIMO-FD relay system where SM is used at the source and the relay nodes under the impact of the RSI caused by the imperfect SIC. Unlike previous works, we derive the exact closed-form expression of $\mathrm{SEP}$ for the system over the Rayleigh fading channel.
• We propose an algorithm to calculate the optimal transmission power of the FD relay. Based on this algorithm, we obtain the optimal power allocation for the considered system. The proposed optimal transmission power algorithm significantly improves the $\mathrm{SEP}$ performance, especially in the low $\mathrm{SNR}$ region. Additionally, using the derived expression of $\mathrm{SEP}$, we can also examine the influences of the number of transmitting/receiving antennas and the RSI on the system performance in the case with and without optimal power allocation.

The rest of this paper is organized as follows. Section 2 presents the system model. Section 3 provides the detailed derivations of the closed-form expression of $\mathrm{SEP}$. The optimal power allocation algorithm for the FD relay is developed in Section 3. Section 4 presents the analytical and simulation results for performance evaluation. Finally, Section 5 concludes the paper.

2. System Model

We consider a single-user single-carrier SM-MIMO-FD relay system with a source node $\mathrm{S}$, which transmits its signal to a destination node $\mathrm{D}$ via a relay node $\mathrm{R}$ as shown in Figure 1. Particularly, $\mathrm{S}$ and $\mathrm{D}$ operate in the half-duplex (HD) mode with $N_{\mathrm{S}}$ transmitting antennas and $N_{\mathrm{D}}$ receiving antennas, respectively. The relay node operates in the FD mode with $N_{\mathrm{tR}}$ transmitting antennas and $N_{\mathrm{rR}}$ receiving  antennas.

At time slot t, the received signal at $\mathrm{R}$ can be calculated as

$$\begin{array}{c}\hfill {\mathbf{y}}_{\mathrm{R}}\left(t\right)=\sqrt{P_{\mathrm{S}}}{\mathbf{h}}_i^{\mathrm{R}}{x}_{\mathrm{S}}\left(t\right)+\sqrt{P_{\mathrm{R}}}{\mathbf{h}}_j^{\mathrm{R}}{x}_{\mathrm{R}}\left(t\right)+{\mathbf{z}}_{\mathrm{R}}\left(t\right),\end{array}$$

(1)

where ${\mathbf{h}}_i^{\mathrm{R}}$ and ${\mathbf{h}}_j^{\mathrm{R}}$ are respectively the channel vector from the ith active antenna of $\mathrm{S}$ to $N_{\mathrm{rR}}$ receiving antennas of $\mathrm{R}$ and from the jth active antenna of $\mathrm{R}$ to all reception antennas of $\mathrm{R}$. These channels are assumed to undergo flat Rayleigh fading, which can be modeled by independent and identically distributed complex Gaussian random variables with zero mean and unit variance. $x_{\mathrm{S}}$ and $x_{\mathrm{R}}$ are the transmitted signals at the ith antenna of $\mathrm{S}$ and the jth antenna of $\mathrm{R}$, respectively; $P_{\mathrm{S}}$ and $P_{\mathrm{R}}$ are the average transmission power at $\mathrm{S}$ and $\mathrm{R}$, respectively; ${\mathbf{z}}_{\mathrm{R}}$ is the noise vector whose elements follow a complex Gaussian distribution with variance ${\sigma }^2$.

At the FD relay, we assume that the transmitting and receiving antennas are both directional, thus there will be no direct link which causes the self-interference (SI) from the transmit to the receive antenna. This SI is mainly due to reflections caused by multipath propagation. We also assume that the system can use all SIC techniques in the three domains, that is, propagation, analog and digital, to remove the SI [19,20]. Specifically, $\mathrm{R}$ can use all available isolation techniques to suppress SI. It can use the cross-polarization transmission to isolate the transmitting and receiving antennas [19,21]. In the analog domain, thanks to the SI awareness of analog circuits, the transmitted signal via the transmit antenna is collected and then subtracted from the received signal. The RSI is then converted to the digital domain for further SIC via digital signal processing. As $\mathrm{R}$ knows its transmitted signal, it can subtract the SI from the received signal by using SI channel estimation [20,22]. Thanks to all these SIC techniques, the relay node can achieve up to 110 dB SI suppression [23]. Moreover, since the SI is canceled from the received signal in the analog and digital domain by reconstructing the SI signal, the RSI is in effect the resulted errors due to the imperfect reconstruction or more correctly, the imperfect SI channel estimation. Moreover, as the digital-domain cancellation is done after a quantization operation, RSI at the relay ${\mathbf{r}}_{\mathrm{SI}}$ can be modeled using complex Gaussian random variable [4,17,20,23] with zero mean and variance of ${\sigma }_{\mathrm{RSI}}^2$, that is ${\sigma }_{\mathrm{RSI}}^2=\tilde{\Omega }{P}_{\mathrm{R}}$ where $\tilde{\Omega }$ denotes the SIC capability at the relay. Therefore, the received signal at $\mathrm{R}$ can be rewritten from (1) as

$$\begin{array}{c}\hfill {\mathbf{y}}_{\mathrm{R}}\left(t\right)=\sqrt{P_{\mathrm{S}}}{\mathbf{h}}_i^{\mathrm{R}}{x}_{\mathrm{S}}\left(t\right)+{\mathbf{r}}_{\mathrm{SI}}\left(t\right)+{\mathbf{z}}_{\mathrm{R}}\left(t\right),\end{array}$$

(2)

and the received signal at the destination $\mathrm{D}$ is then given by

$$\begin{array}{c}\hfill {\mathbf{y}}_{\mathrm{D}}\left(t\right)=\sqrt{P_{\mathrm{R}}}{\mathbf{h}}_j^{\mathrm{D}}{x}_{\mathrm{R}}\left(t\right)+{\mathbf{z}}_{\mathrm{D}}\left(t\right),\end{array}$$

(3)

where ${\mathbf{h}}_j^{\mathrm{D}}$ is the channel vector from the jth antenna of $\mathrm{R}$ to $N_{\mathrm{D}}$ receiving antennas of $\mathrm{D}$; ${\mathbf{z}}_{\mathrm{D}}$ is the noise vector at $\mathrm{D}$. Both $\mathrm{S}$ and $\mathrm{R}$ are assumed to have the same $N_t$ transmitting antennas for the same expected spectral efficiency. In the SM system, to estimate the transmitted bits, the receiver needs to use joint ML detection for both the activated transmit antenna and the M-ary modulated symbols. This joint detector is computationally complex, especially for the SM system with a large number of antennas. Near-ML low-complexity detectors such as those summarized in Reference [24] or that for the index modulation in the frequency domain [25] are more favorable for practical implementation. In this paper, as we are interested in analyzing the effect of the RSI due to the FD mode on the system performance, we assume that the receivers of both $\mathrm{R}$ and $\mathrm{D}$ can perfectly estimate the transmitted antenna index of the respective transmitters for the ML detection.

From (2) and (3), we can calculate the instantaneous signal-to-interference-plus-noise-ratios ($\mathrm{SINR}$s) of $\mathrm{S}-\mathrm{R}$ and $\mathrm{R}-\mathrm{D}$ links as follows

$$\begin{array}{c}\hfill {\gamma }_{\mathrm{R}}=\frac{P_{\mathrm{S}}{\parallel {\mathbf{h}}_i^{\mathrm{R}}\parallel }^2}{{\sigma }_{\mathrm{RSI}}^2+{\sigma }^2}={\parallel {\mathbf{h}}_i^{\mathrm{R}}\parallel }^2{\overline{\gamma }}_{\mathrm{R}},\end{array}$$

(4)

$$\begin{array}{c}\hfill {\gamma }_{\mathrm{D}}=\frac{P_{\mathrm{R}}{\parallel {\mathbf{h}}_j^{\mathrm{D}}\parallel }^2}{{\sigma }^2}={\parallel {\mathbf{h}}_j^{\mathrm{D}}\parallel }^2{\overline{\gamma }}_{\mathrm{D}},\end{array}$$

(5)

where ${\overline{\gamma }}_{\mathrm{R}}=\frac{P_{\mathrm{S}}}{{\sigma }_{\mathrm{RSI}}^2+{\sigma }^2}$ and ${\overline{\gamma }}_{\mathrm{D}}=\frac{P_{\mathrm{R}}}{{\sigma }^2}$ denote the average $\mathrm{SINR}$ at $\mathrm{R}$ and the average signal-to-noise-ratio ($\mathrm{SNR}$) at $\mathrm{D}$, respectively.

Since the relay node uses the DF protocol, the instantaneous end-to-end $\mathrm{SINR}$ of the considered system is defined as

$$\begin{array}{c}\hfill {\gamma }_{\mathrm{e}2\mathrm{e}}=min({\gamma }_{\mathrm{R}},{\gamma }_{\mathrm{D}}).\end{array}$$

(6)

where ${\gamma }_{\mathrm{R}}$ and ${\gamma }_{\mathrm{D}}$ are respectively the instantaneous $\mathrm{SINR}$s at $\mathrm{R}$ and $\mathrm{D}$.

3. Optimal Power Allocation for FD Mode

To find the optimal power allocation for the FD relay, we first calculate the $\mathrm{SEP}$ of the considered system using the definition given in Reference [26] as follows

$$\begin{array}{c}\hfill \mathrm{SEP}=a\mathbb{E}\left\{Q\left(\sqrt{b{\gamma }_{\mathrm{e}2\mathrm{e}}}\right)\right\}=\frac{a}{\sqrt{2\pi }}\underset{0}{\overset{\infty }{\int }}{F}_{{\gamma }_{\mathrm{e}2e}}\left(\frac{t^2}{b}\right)e^{-\frac{t^2}{2}}dt,\end{array}$$

(7)

where a and b are constants whose values depend on the modulation types, for example, $a=1,b=2$ for the binary phase-shift keying (BPSK) modulation [26]. The values of a and b are determined using Table 6.1 of Reference [26]; $Q\left(x\right)$ is the Gaussian function; ${\gamma }_{\mathrm{e}2\mathrm{e}}$ is the instantaneous end-to-end SINR of the considered system which is determined in (6); $F_{{\gamma }_{\mathrm{e}2e}}(.)$ is cumulative distribution function (CDF) of ${\gamma }_{\mathrm{e}2\mathrm{e}}$ [21,27]. After some mathematical manipulations, (7) becomes

$$\begin{array}{c}\hfill \mathrm{SEP}=\frac{a\sqrt{b}}{2\sqrt{2\pi }}\underset{0}{\overset{\infty }{\int }}\frac{e^{-bx/2}}{\sqrt{x}}{F}_{{\gamma }_{\mathrm{e}2e}}\left(x\right)dx.\end{array}$$

(8)

To obtain the closed-form expression of (8), we calculate the CDF, $F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)$, of the probability that the instantaneous end-to-end $\mathrm{SINR}$ falls below a defined threshold. Mathematically, $F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)$ is expressed as

$$\begin{array}{cc}\hfill F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)& =\mathrm{Pr}\left\{{log}_2\left(N_t\right)+{log}_2(1+{\gamma }_{\mathrm{e}2\mathrm{e}})<\mathcal{R}\right\}\hfill \\ \hfill & =\mathrm{Pr}\left\{{\gamma }_{\mathrm{e}2\mathrm{e}}<2^{\mathcal{R}-{log}_2\left(N_t\right)}-1\right\},\hfill \end{array}$$

(9)

where $\mathcal{R}$ is the minimum data transmission rate of the considered system; the term ${log}_2\left(N_t\right)$ denotes the number of bits which is used for activating the transmit antenna at the transmitters of ($\mathrm{S}$ or $\mathrm{R}$).

Using the probability law of two independent variables $\mathcal{A}$ and $\mathcal{B}$ [28], that is $\mathrm{Pr}\{\mathcal{A}\cup \mathcal{B}\}=\mathrm{Pr}\left\{\mathcal{A}\right\}+\mathrm{Pr}\left\{\mathcal{B}\right\}-\mathrm{Pr}\left\{\mathcal{A}\right\}\mathrm{Pr}\left\{\mathcal{B}\right\}$, we have

$$\begin{array}{cc}\hfill F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)=& \mathrm{Pr}\left\{{\gamma }_{\mathrm{e}2\mathrm{e}}<2^{\mathcal{R}-{log}_2\left(N_t\right)}-1\right\}=\mathrm{Pr}\{{\gamma }_{\mathrm{e}2\mathrm{e}}<x\}\hfill \\ \hfill =& \mathrm{Pr}\{{\gamma }_{\mathrm{R}}<x\}+\mathrm{Pr}\{{\gamma }_{\mathrm{D}}<x\}\hfill \\ \hfill & -\mathrm{Pr}\{{\gamma }_{\mathrm{R}}<x\}\mathrm{Pr}\{{\gamma }_{\mathrm{D}}<x\},\hfill \end{array}$$

(10)

where $x=2^{\mathcal{R}-{log}_2\left(N_t\right)}-1$.

To calculate $F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)$ in (10), we first start with the CDF and probability distribution function (PDF) of the channel gain which follows Rayleigh fading distribution, that is,

$$\begin{array}{cc}\hfill F_{{\left|h\right|}^2}\left(x\right)& ={\mathrm{Pr}\left\{\right|h|}^2<x\}=1-\exp \left(-\frac{x}{\Omega }\right),x⩾0,\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill f_{{\left|h\right|}^2}\left(x\right)& =\frac{1}{\Omega }\exp \left(-\frac{x}{\Omega }\right),x⩾0,\hfill \end{array}$$

(12)

where $\Omega =\mathbb{E}\left\{\right|h{|}^2\}$ is the average channel gain; $\mathbb{E}$ denotes the expectation operator. In this paper, for the ease of presentation, we choose $\Omega =1$ for all channel gains.

Then, we apply (11) and (12) to compute the probability in (10) as

$$\begin{array}{cc}\hfill \mathrm{Pr}\{{\gamma }_{\mathrm{R}}<x\}& =\mathrm{Pr}\left\{\parallel {\mathbf{h}}_i^{\mathrm{R}}{\parallel }^2{\overline{\gamma }}_{\mathrm{R}}<x\right\}\hfill \\ \hfill & =\mathrm{Pr}\left\{\parallel {\mathbf{h}}_i^{\mathrm{R}}{\parallel }^2<\frac{x}{{\overline{\gamma }}_{\mathrm{R}}}\right\}.\hfill \end{array}$$

(13)

Based on the CDF of the summation of channel gains [20], this probability is calculated as

$$\begin{array}{c}\hfill \mathrm{Pr}\{{\gamma }_{\mathrm{R}}<x\}=1-e^{-\frac{x}{{\overline{\gamma }}_{\mathrm{R}}}}\sum _{i=0}^{N_{\mathrm{rR}}-1}\frac{1}{i!}\frac{x^i}{{\overline{\gamma }}_{\mathrm{R}}^i}.\end{array}$$

(14)

Using similar calculations, we can obtain $\mathrm{Pr}\{{\gamma }_{\mathrm{D}}<x\}$ as

$$\begin{array}{c}\hfill \mathrm{Pr}\{{\gamma }_{\mathrm{D}}<x\}=1-e^{-\frac{x}{{\overline{\gamma }}_{\mathrm{D}}}}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{1}{j!}\frac{x^j}{{\overline{\gamma }}_{\mathrm{D}}^j}.\end{array}$$

(15)

Finally, $F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)$ in (10) can be given by

$$\begin{array}{c}\hfill F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)=1-e^{-\frac{x}{{\overline{\gamma }}_{\mathrm{R}}}-\frac{x}{{\overline{\gamma }}_{\mathrm{D}}}}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{1}{i!j!}\frac{x^{i+j}}{{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j}.\end{array}$$

(16)

Remark: As shown in (16), the active antenna at the transmitter is hidden in the variable x because $x=2^{\mathcal{R}-{log}_2\left(N_t\right)}-1$. Moreover, under the assumption that the receiver can estimate the index of the transmitter’s activated antenna, it can successfully decode the transmitted bits used to modulate the active antenna. Substituting $F_{{\gamma }_{\mathrm{e}2e}}\left(x\right)$ in (16) into (8), we obtain the closed-form expression of $\mathrm{SEP}$ as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathrm{SEP}=& \frac{a\sqrt{b}}{2\sqrt{2\pi }}\left[\underset{0}{\overset{\infty }{\int }}\frac{e^{-bx/2}}{\sqrt{x}}dx-\underset{0}{\overset{\infty }{\int }}\frac{e^{-bx/2}}{\sqrt{x}}{e}^{-\frac{x}{{\overline{\gamma }}_{\mathrm{R}}}-\frac{x}{{\overline{\gamma }}_{\mathrm{D}}}}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{1}{i!j!}\frac{x^{i+j}}{{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j}dx\right]\hfill \\ \hfill =& \frac{a}{2}-\frac{a\sqrt{b}}{2\sqrt{2\pi }}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{\Gamma (i+j+\frac{1}{2})}{i!j!{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}·\hfill \end{array}\end{array}$$

(17)

It is worth noting that we have used equations ([29] (3.361.2)) and ([29] (3.381.4))to solve the first and second integrals in (17), respectively.

For the purpose of improving system performance and reducing the impact of the RSI in FD mode, we can calculate the optimal transmission power of $\mathrm{R}$ to minimize the system $\mathrm{SEP}$. The optimal transmission power of $\mathrm{R}$ for minimizing the system SEP, denoted by $P_{\mathrm{R}}^{*}$, is defined as

$$\begin{array}{c}\hfill P_{\mathrm{R}}^{*}=arg\underset{P_{\mathrm{R}}}{min}\mathrm{SEP}.\end{array}$$

(18)

To explicitly determine the $\mathrm{minSEP}$ in (18), we begin with two terms in (17), that is, ${\mathrm{SEP}}_1$ and ${\mathrm{SEP}}_2$, as follows

$$\begin{array}{c}\hfill \mathrm{SEP}=\underset{{\mathrm{SEP}}_1}{\underbrace{\frac{a}{2}}}-\underset{{\mathrm{SEP}}_2}{\underbrace{\frac{a\sqrt{b}}{2\sqrt{2\pi }}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{\Gamma (i+j+\frac{1}{2})}{i!j!{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}}},\end{array}$$

(19)

where

$$\begin{array}{c}\hfill {\mathrm{SEP}}_1=\frac{a}{2},\end{array}$$

(20)

and

$$\begin{array}{c}\hfill {\mathrm{SEP}}_2=\frac{a\sqrt{b}}{2\sqrt{2\pi }}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{\Gamma (i+j+\frac{1}{2})}{i!j!{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}.\end{array}$$

(21)

As mentioned in Section 3, after (7), since a and b are constants, ${\mathrm{SEP}}_1=\frac{a}{2}$ is also constant. Therefore, we have

$$\begin{array}{c}\hfill min\mathrm{SEP}=min({\mathrm{SEP}}_1-{\mathrm{SEP}}_2)={\mathrm{SEP}}_1-max{\mathrm{SEP}}_2=max{\mathrm{SEP}}_2.\end{array}$$

(22)

Next, (22) can be rewritten as

$$\begin{array}{c}\hfill min\mathrm{SEP}=max{\mathrm{SEP}}_2=max\left(\frac{a\sqrt{b}}{2\sqrt{2\pi }}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{\Gamma (i+j+\frac{1}{2})}{i!j!{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}\right),\end{array}$$

(23)

and (23) can be presented as

$$\begin{array}{c}\hfill min\mathrm{SEP}=max\left(\frac{a\sqrt{b}}{2\sqrt{2\pi }}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{\Gamma (i+j+\frac{1}{2})}{i!j!}\times \frac{1}{{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}\right).\end{array}$$

(24)

It is noted that, in (24), a, b, $N_{\mathrm{rR}}$ and $N_{\mathrm{D}}$ are constants; i and j are antenna indices which do not depend on the transmission power of $\mathrm{S}$ and $\mathrm{R}$. Moreover, $\Gamma (i+j+\frac{1}{2})$ is also a constant for the certain values of i and j. Therefore, (24) is maximized when

$$\begin{array}{c}\hfill \frac{1}{{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}\end{array}$$

(25)

is maximized or

$$\begin{array}{c}\hfill {\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}\end{array}$$

(26)

is minimized.

In summary, the $min\mathrm{SEP}$ in (18) is given by

$$\begin{array}{cc}\hfill min\mathrm{SEP}& =min\left(\frac{a}{2}-\frac{a\sqrt{b}}{2\sqrt{2\pi }}\sum _{i=0}^{N_{\mathrm{rR}}-1}\sum _{j=0}^{N_{\mathrm{D}}-1}\frac{\Gamma (i+j+\frac{1}{2})}{i!j!{\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}}\right)\hfill \\ \hfill & =min\left({\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}\right).\hfill \end{array}$$

(27)

Denote $P_{\mathrm{R}}=\alpha P_{\mathrm{S}}$ and $f\left(\alpha \right)={\overline{\gamma }}_{\mathrm{R}}^i{\overline{\gamma }}_{\mathrm{D}}^j{(\frac{1}{{\overline{\gamma }}_{\mathrm{R}}}+\frac{1}{{\overline{\gamma }}_{\mathrm{D}}}+\frac{b}{2})}^{i+j+\frac{1}{2}}$. Now we need to find ${\alpha }^{*}$, which is the optimal value of $\alpha$. Then for this given ${\alpha }^{*}$, we can obtain $P_{\mathrm{R}}^{*}$. The procedure for obtaining ${\alpha }^{*}$ and $P_{\mathrm{R}}^{*}$ is summarized in the following Algorithm 1.

Algorithm 1 Calculation of optimal ${\alpha }^{*}$ and $P_{\mathrm{R}}^{*}$

1: Solve $\frac{\partial f\left(\alpha \right)}{\partial \alpha }=0$ for $\alpha ={\alpha }_0$;   2: if$\left\{\begin{array}{c}{\alpha }_0>0\hfill \\ \frac{\partial f\left(\alpha \right)}{\partial \alpha }<0\mathrm{for}\alpha <{\alpha }_0\\ \frac{\partial f\left(\alpha \right)}{\partial \alpha }>0\mathrm{for}\alpha >{\alpha }_0\end{array}\right.$   3: then   4:   Output optimal value of $\alpha$   ${\alpha }^{*}={\alpha }_0$;   thus   $P_{\mathrm{R}}^{*}={\alpha }^{*}{P}_{\mathrm{S}}$   5: else 6:     Output optimal value ${\alpha }^{*}=\varnothing$;   7: end

We will explain step-by-step the process of Algorithm 1 as follows.

• Step 1: We take the derivative of $\frac{\partial f\left(\alpha \right)}{\partial \alpha }$ with respect to $\alpha$ and solve $\frac{\partial f\left(\alpha \right)}{\partial \alpha }=0$ to obtain the stationary point ${\alpha }_0$. Specifically, after some basic algebra calculations, we obtain the following equation

  $$\begin{array}{c}\hfill f^{\prime }\left(\alpha \right)\le 2\tilde{\Omega }{P}_{\mathrm{S}}{\alpha }^2-2b{P}_{\mathrm{S}}\alpha -9.\end{array}$$

  (28)
  Then, ${\alpha }_0$ can be calculated as

  $$\begin{array}{c}\hfill {\alpha }_0=\frac{b{P}_{\mathrm{S}}+\sqrt{P_{\mathrm{S}}(b^2{P}_{\mathrm{S}}+18\tilde{\Omega })}}{2\tilde{\Omega }{P}_{\mathrm{S}}}.\end{array}$$

  (29)
• Step 2: We check whether $\frac{\partial f\left(\alpha \right)}{\partial \alpha }$ is negative or positive in a specific interval to determine maximum or minimum point. If $\frac{\partial f\left(\alpha \right)}{\partial \alpha }$ is negative when $\alpha <{\alpha }_0$ and positive when $\alpha >{\alpha }_0$, an optimal transmission power $P_{\mathrm{R}}^{*}$ exits and it is given by

  $$\begin{array}{c}\hfill P_{\mathrm{R}}^{*}=\frac{b{P}_{\mathrm{S}}+\sqrt{P_{\mathrm{S}}(b^2{P}_{\mathrm{S}}+18\tilde{\Omega })}}{2\tilde{\Omega }}.\end{array}$$

  (30)
  It is worth noting that (30) is the bounded optimal transmission power at the FD relay node and it is often used for the systems with complex mathematical expressions.
• Step 3: Otherwise, if ${\alpha }^{*}=\varnothing$, depending on whether $\frac{\partial f\left(\alpha \right)}{\partial \alpha }$ is less or greater than 0, we can select an appropriate value of ${\alpha }^{*}$ to get $P_{\mathrm{R}}^{*}$.

4. Numerical Results

In this section, to validate the derived mathematical expressions in the previous sections, we provide analytical results together with the Monte-Carlo simulation results for comparison. For ease of presentation, both $\mathrm{S}$ and $\mathrm{R}$ use two transmitting antennas, that is $N_t=2$, while the number of receiving antennas $N_{\mathrm{rR}}$ and $N_{\mathrm{D}}$ are set to be equal and varies from 2 to 4 for performance evaluations. The SIC capability used for evaluation is $\tilde{\Omega }=\{-10,-5,0\}\mathrm{dB}$. For an LTE relay, the typical transmission power ranges from 23 dBm to 30 dBm. Taking 30 dBm for consideration, the RSI levels are ${\sigma }_{\mathrm{RSI}}^2=\{20,25,30\}$ dBm, respectively. In all figures, we define the average SNR for the case without optimization as follows: $\mathrm{SNR}=\frac{P_{\mathrm{S}}}{{\sigma }^2}=\frac{P_{\mathrm{R}}}{{\sigma }^2}$. In the case with optimization, the average SNR is defined by SNR at R, that is, $\mathrm{SNR}=\frac{P_{\mathrm{S}}}{{\sigma }^2}$. The analytical curves are plotted using Equation (17) while the markers refer to Monte-Carlo simulation results. The simulation results were obtained using ${10}^6$ channel realizations.

Figure 2 plots the SEP of the considered SM-MIMO-FD relay system versus the SNR in dB for two modulation schemes, that is, BPSK ($a=1$, $b=2$) and 4-QAM ($a=2$, $b=1$). We should remind that a and b are constants, which depend on the types modulation scheme. These values for each type of modulation are given in Table 6.1 of Reference [26]. As shown in Figure 2, similar patterns of SEPs can be observed in both BPSK and 4-QAM modulation schemes. However, the system with 4-QAM modulation has higher SEP than the system with BPSK modulation. Moreover, the benefit of optimization is also reduced as the modulation order increases. Therefore, although our analysis method can be applied for all modulation types, we use the BPSK modulation in the following figures to clearly show the advantage of our proposed optimization algorithm in reducing the SEP of the considered SM-MIMO-FD relay system.

Figure 3 compares the $\mathrm{SEP}$s of the SM-MIMO-FD relay system in two cases, that is $\alpha =1$ (without optimal power allocation, $P_{\mathrm{R}}=P_{\mathrm{S}}$) and $\alpha ={\alpha }^{*}$ (with optimal power allocation, $P_{\mathrm{R}}=P_{\mathrm{R}}^{*}$) in (30) for different numbers of receiving antennas of $\mathrm{R}$ and $\mathrm{D}$. Although we have used $N_{\mathrm{rR}}=N_{\mathrm{D}}=2,3,4$ to obtain this figure, it is worth noting that we can use different numbers of receiving antennas at R and D for numerical calculation using the closed-form expression of SEP. The modulation used for evaluation is BPSK with parameters $a=1,b=2$. The typical SIC capability of $\tilde{\Omega }=-10\mathrm{dB}$ is used for calculation. It is easy to see from the figure that the analytical results perfectly match the simulation ones. Although the $\mathrm{SEP}$s of both the cases with and without optimal power allocation suffer the same error floors in high $\mathrm{SNR}$ regime, the $\mathrm{SEP}$ with optimal power allocation is significantly lower than that without optimal power allocation in low $\mathrm{SNR}$ regime. For example, when $\mathrm{SNR}=8\mathrm{dB}$, the $\mathrm{SEP}$ in the case with ${\alpha }^{*}$ is approximately ten times less than the case without ${\alpha }^{*}$.

Figure 4 shows the impact of the RSI on the $\mathrm{SEP}$ of the system with and without optimal power allocation for different SIC capabilities, that is, $\tilde{\Omega }=-10,-5,0\mathrm{dB}$. When the RSI is small ($\tilde{\Omega }=-10\mathrm{dB}$), the difference in $\mathrm{SEP}$s in the cases with and without using ${\alpha }^{*}$ is large. However, when the RSI becomes larger ($\tilde{\Omega }=-5\mathrm{dB}$), the benefit of using ${\alpha }^{*}$ decreases. Therefore, it is necessary to combine SIC techniques with optimal power allocation to achieve the best performance for this system. Figure 3 and Figure 4 can be served as the guideline to determine when the optimal power allocation should be used. Specifically, for $\mathrm{SNR}<20\mathrm{dB}$, we use $P_{\mathrm{R}}^{*}$ to improve the system performance. For higher $\mathrm{SNR}$, that is $\mathrm{SNR}>20\mathrm{dB}$, we use $P_{\mathrm{R}}=P_{\mathrm{S}}$ to reduce the signal processing complexity at the FD relay. The advantage of the system with optimization is that it requires only $\mathrm{SNR}\ge 8\mathrm{dB}$ to achieve a reliable voice transmission.

5. Conclusions

In this paper, we have analyzed the performance of the SM-MIMO-FD DF relay system and derived the closed-form expression of the system $\mathrm{SEP}$. Understanding the importance of the optimal power allocation to the SM-MIMO-FD relay system, especially in the case of imperfect SIC, we have proposed an optimal power allocation algorithm for the FD relay to minimize the system $\mathrm{SEP}$. Both numerical and simulation results showed that the RSI has a substantial impact on the $\mathrm{SEP}$ performance. However, the $\mathrm{SEP}$ of the system with optimal power allocation is significantly lower compared with that of the system without optimal power allocation. This result confirms the effectiveness of using optimal power allocation for compensating the impact of the RSI in the SM-MIMO-FD DF relay system.