Outage Performance Analysis of Improper Gaussian Signaling for Two-User Downlink NOMA Systems with Imperfect Successive Interference Cancellation

Abstract

The improper Gaussian signaling (IGS) technique can improve the achievable rate of an interference-limited network by fully exploiting the second-order statistics of complex signaling. This paper addresses the outage performance analysis of a two-user downlink non-orthogonal multiple access (NOMA) system using the IGS technique in the presence of imperfect successive interference cancellation (SIC). The strong channel user (SU) adopts the IGS, while the weak channel user (WU) adopts the traditional proper Gaussian signaling (PGS). Considering a practical scenario where the transmitter has obtained the statistics of the channel coefficients instead of the instantaneous channel state information (CSI), the expressions of the achievable rates of both users under residual interference due to imperfect SIC are derived, together with their outage probabilities, subject to predetermined target rates and channel statistics. Given a fixed transmit power of the WU, both the transmit power and the degree of impropriety of the SU are optimized to minimize the outage probability subject to the outage constraint of the WU. Numerical results are provided to assess the benefits of the proposed IGS-based downlink NOMA system, which are consistent with the analysis.

1. Introduction

The fifth generation (5G) and beyond communication systems support numerous wireless devices and a variety of services. To meet the massive demand for data throughput, non-orthogonal multiple access (NOMA) has been proposed as a promising technology to improve the spectral efficiency of communication systems [1,2,3,4]. The existing NOMA schemes can be mainly divided into two categories: power domain multiplexing [5,6] and code domain multiplexing [7,8,9]. Power domain multiplexing means different users are allowed to transmit data using the same communication resources with different power levels according to their channel conditions, and the base station (BS) broadcasts the superposed signals for multiple users at different power levels. Typically, the BS allocates more power to the user with a weak channel strength. The weak channel user decodes its signal by treating the signal intended for the user with the strong channel strength as interference, and the strong channel user needs to first remove the signal for the weak channel user from the received signal through successive interference cancellation (SIC) [10] and then decode its own signal. Recently, several works have considered discrete signaling, which can be formed by practical modulation schemes without SIC [11,12]. The discrete signaling was adopted in a multi-user downlink NOMA system without SIC in [11], where the outage rates were very close to the outage capacity region given in [13]. Similarly, the discrete signaling without SIC achieves a rate close to Gaussian signaling with perfect SIC for the interference channel [12]. However, the assumption of Gaussian signaling is still common in the theoretical analysis of communication systems and is useful for its simplicity in mathematical manipulations, and SIC should be considered under the Gaussian signaling setting to characterize the capacity and achievable rate region of a NOMA system. In a real-world scenario, a perfect SIC is impractical because the interference cannot be canceled completely due to deep fading, imperfect decoding, channel estimation errors, and so on [10,14,15]. It has been discovered in [16] that the NOMA system performance converges to an error floor in the high signal-to-noise ratio (SNR) region with imperfect SIC and, thus, a zero diversity order is obtained. Therefore, it is of great interest to consider the detrimental impacts of residual interference due to the imperfect SIC on NOMA systems.

Mitigating interference in communication systems is a challenging research problem. It is well known that statistical signal characteristics significantly affect achievable rates and communication performance. Proper Gaussian signaling (PGS) is a common assumption in communication systems, where the real and imaginary parts of complex signals are uncorrelated zero-mean Gaussian random variables with equal power [17,18]. Most existing studies on communication systems have adopted the PGS assumption without any justification because it is known to achieve the maximum achievable rate in an additive white Gaussian noise channel [17]. However, by allowing the real and imaginary parts of the signals to be correlated or to have unequal powers, improper Gaussian signaling (IGS) can increase the achievable rates and enlarge the capacity regions in interference-limited scenarios, including interference channels [19,20], cognitive radio scenarios [21], relay networks [22,23], and so on. Fundamental studies on the second-order statistics of improper complex random vectors were performed in [24,25,26,27]. IGS was used to model the resultant signals from in-phase and quadrature imbalance in [28]. The usefulness of the IGS for interference alignment in wireless communications was investigated in [29]. In [30], maximally IGS was applied to a two-user single-input–single-output (SISO) interference channel to achieve a close-to-optimal sum rate in the interference-limited regime with strong interference and relatively weak noise. In [31], an achievable rate expression for a general multi-user multiple-input–multiple-output interference channel with IGS was derived, followed by the Pareto boundary characterization of the achievable rate region in a two-user SISO setting. The study was further extended to the multiple-input–single-output scenario in [32]. In [21], IGS was adopted to underlay cognitive radio networks to improve the achievable rate of the unlicensed secondary user. A similar cognitive radio scenario was studied in [33], where only statistical channel state information (CSI) was assumed to be available at the secondary user nodes, and IGS was shown to improve the outage performance of the secondary user while satisfying the quality-of-service (QoS) constraint of the primary user. The benefits of adopting IGS in a two-hop alternate relaying system were investigated in [22], where significant performance improvement can be achieved by IGS, especially when the first hop is a bottleneck.

Although research has been conducted to optimize IGS-based schemes in various wireless communication scenarios to improve the system throughput and reliability, the application of IGS in NOMA systems has just emerged. A two-user downlink NOMA system with imperfect SIC was considered in [34], where the closed-form expressions of the sum rates of both users using IGS were carried out. Assuming that the strong user and the weak user employ IGS and PGS, respectively, the degree of impropriety of the strong user was optimized to maximize the sum rate. The two-user NOMA system was further investigated in [35], where the strong user employs PGS, and the weak user adopts IGS. Optimization problems were formulated to maximize the rate of either the strong user or the weak user subject to its power and the impropriety degree of the weak user, assuming a fixed power and a predetermined QoS requirement of the other user. The sum rate was also maximized using a Q-learning-based algorithm. In [36], the outage probabilities and ergodic capacities were studied in a NOMA system with imperfect SIC, in which the strong user applies IGS, and a characterization of the system performance at high SNR was given. The fairness optimization problem was formulated to minimize the maximum outage probability of the two users by adjusting the power allocation factor and the circularity coefficient of the strong user. However, when different QoS constraints are imposed on the two users, other metrics should be considered.

In this paper, we investigate a downlink NOMA system with imperfect SIC, where a BS serves a strong channel user and a weak channel user. The strong user (SU) can employ either proper or improper signaling, while the weak user (WU) adopts proper signaling. Only the statistical CSI is assumed to be known at the BS, which is more practical than the knowledge of the instantaneous CSI assumed in [34,35]. We provide a solution to improve the performance of the SU while satisfying the QoS of the WU. The main contributions of this paper are summarized as follows:

• We derive the instantaneous rates of both users in the downlink NOMA system with imperfect SIC. Based on the rate expressions and the Rayleigh fading channel statistics, their outage probabilities are also derived in closed form.
• Unlike the previous attempt to optimize the fairness of both users [36], we consider a scenario where the two users have different requirements for data throughput and reliability. The performance optimization problem of the SU is formulated, where its power and degree of impropriety are adjusted to minimize its outage probability, subject to a predetermined target rate, while fulfilling the specific maximum outage probability constraint of the WU. The solution to the problem is derived, followed by an analysis of the condition for the IGS scheme to outperform the PGS one, which is mainly determined by the target rates of both users, the power of the signal intended for the WU, and the power ratio between two users.
• We investigate the outage probabilities of the SU in both PGS and IGS designs at extreme SNRs and find that the IGS can bring benefits at a high target rate of the WU and a large power ratio between the SU and WU. Through simulation, it is shown that the SU benefits from the IGS-based transmission strategy in a certain SNR range under certain conditions involving the target rates of both users and their power ratio.

The rest of this paper is organized as follows. Section 2 describes the two-user downlink NOMA system model. Section 3 derives the outage probabilities for both users in the NOMA system. In Section 4, the signal parameters of the SU are adjusted to minimize its outage probability under both PGS and IGS constraints while guaranteeing the QoS of the WU, along with a brief characterization of the system performance at low and high SNRs. Numerical experiments are then conducted to investigate the outage probabilities of both users in Section 5. Finally, we conclude the paper in Section 6.

The notations that will be used throughout the paper are listed and explained in

Table 1. Notations.

Notation	Explanation	Expression
$p_i$	Power of user i
$\kappa$	Impropriety degree of the signal of the strong user (SU)
${\sigma }_{n_i}^2$	Noise power at the receiver of user i
${\gamma }_i$, ${\gamma }_r$	Instantaneous channel-to-noise ratios (CNRs) of user i and the residual interference channel, respectively
$R_1$	Instantaneous rate of the SU to decode its own signal after successive interference cancellation (SIC)	(13)
$R_{12}$	Instantaneous rate to decode the signal of the weak user (WU) at the SU end	(11)
$R_2$	Instantaneous rate of the WU	(9)
${\overline{\gamma }}_i$	Average CNR of user i	$\mathbb{E}\left\{{\gamma }_i\right\}$
$\xi$	Level of residual interference after SIC	$\mathbb{E}\left\{{\gamma }_r\right\}/{\overline{\gamma }}_1$
$R_{0,i}$	Target rate of user i
${\Gamma }_i$	$2^{2R_{0,i}}-1$
$v_1$, $v_2$	Functions of $p_1$ and $\kappa$, used as parameters for optimization	(25) and (19)
$P_{\mathrm{out},i}$	Outage probability of user i	(28) and (21)
$P_{\mathrm{out},\mathrm{th}}$	Maximum outage threshold of the WU
$p_{1,\max }$	Maximum power budget of the SU
${\lambda }_m$	Maximum power ratio between SU and WU	$p_{1,\max }/p_2$
$v_{2,\mathrm{th}}$	Value of $v_2$ when $P_{\mathrm{out},2}=P_{\mathrm{out},\mathrm{th}}$	(31)
$v_2^{\mathrm{p}}\left(v_1\right)$	Value of $v_2$ when $\kappa =0$ (proper Gaussian signaling, PGS)	(36)
$v_2^{\mathrm{i}}\left(v_1\right)$	Value of $v_2$ when $\kappa =1$ (maximally improper Gaussian signaling, maximally IGS)	(36)
$v_2^{\mathrm{m}}\left(v_1\right)$	Value of $v_2$ when $p_1=p_{1,\max }$	(36)
$v_2^{\mathrm{L}}\left(v_1\right)$	Lower bound of a feasible $v_2$ given a $v_1$	(44)
$v_2^{\mathrm{U}}\left(v_1\right)$	Upper bound of a feasible $v_2$ given a $v_1$	(45)
$v_1^{\mathrm{c}}$, $v_2^{\mathrm{c}}$	Values of $v_1$ and $v_2$ when $v_1-v_2+1/{\Gamma }_2=0$, making $v_2^{\mathrm{p}}\left(v_1^{\mathrm{c}}\right)=v_2^{\mathrm{i}}\left(v_1^{\mathrm{c}}\right)=v_2^{\mathrm{m}}\left(v_1^{\mathrm{c}}\right)=v_2^{\mathrm{c}}$	(37)
$v_1^{\mathrm{pm}}$	Value of $v_1$ at $p_1=p_{1,\max }$ and $\kappa =0$, making $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)=v_2^{\mathrm{m}}\left(v_1^{\mathrm{pm}}\right)$	(39)
$v_1^{\mathrm{im}}$	Value of $v_1$ at $p_1=p_{1,\max }$ and $\kappa =1$, making $v_2^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)=v_2^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right)$	(46)
$v_e^{\mathrm{p}}$	Value of $v_1$ making $v_2^{\mathrm{p}}\left(v_1\right)=v_1$	(50)
$D^{\mathrm{p}}\left(v_1\right)$	A term of $d{P}_{\mathrm{out},1}(v_1,v_2^{\mathrm{p}}\left(v_1\right))/d{v}_1$, as an increasing function	(52)
$D^{\mathrm{i}}\left(v_1\right)$	A term of $d{P}_{\mathrm{out},1}(v_1,v_2^{\mathrm{i}}\left(v_1\right))/d{v}_1$	(63)
$D^{\mathrm{m}}\left(v_1\right)$	A term of $d{P}_{\mathrm{out},1}(v_1,v_2^{\mathrm{m}}\left(v_1\right))/d{v}_1$, as an increasing function	(59)
$v_1^{\mathrm{dp}}$	Extreme point of $v_2^{\mathrm{p}}(·)$; solution to $D^{\mathrm{p}}(·)=0$
$v_1^{\mathrm{di}}$	Extreme point of $v_2^{\mathrm{i}}(·)$; solution to $D^{\mathrm{i}}(·)=0$
$v_1^{\mathrm{dm}}$	Extreme point of $v_2^{\mathrm{m}}(·)$; solution to $D^{\mathrm{m}}(·)=0$
$v_{1,\mathrm{th}}^{\mathrm{p}}$	Solution to $v_2^{\mathrm{p}}(·)=v_{2,\mathrm{th}}$	(51)
$v_{1,\mathrm{th}}^{\mathrm{i}}$	Solution to $v_2^{\mathrm{i}}(·)=v_{2,\mathrm{th}}$	(56)
$v_{1,\mathrm{th}}^{\mathrm{m}}$	Solution to $v_2^{\mathrm{m}}(·)=v_{2,\mathrm{th}}$	(57)
$v_{1,\mathrm{th}}$	Solution to $v_2^{\mathrm{U}}(·)=v_{2,\mathrm{th}}$	(55)
$v_1^{\mathrm{op}}$	Optimal value of $v_1$ in PGS design; solution to the problem (48)	(53)
$v_1^{\mathrm{oi}}$	Optimal value of $v_1$ when $\kappa =1$	(62)
$v_1^{\mathrm{om}}$	Optimal value of $v_1$ when $p_1=p_{1,\max }$	(61)
$v_1^{★}$	Optimal value of $v_1$ in improper Gaussian signaling (IGS) design; solution to the problem (54)	(67)
$p_1^{\mathrm{op}}$	Optimal power of the SU in PGS design	$v_1^{\mathrm{op}}{p}_2(\sqrt{1+{\Gamma }_1}-1)$
$p_1^{★}$	Optimal power of the SU in IGS design	(68)
${\kappa }^{★}$	Optimal impropriety degree of the signal of the SU in the IGS design	(69)

.

2. System Model

2.1. Preliminaries for Improper Random Variables

For a zero-mean complex random variable x, its variance ${\sigma }_x^2$ and complementary variance ${\tilde{\sigma }}_x^2$ are, respectively, defined as [17]

$${\sigma }_x^2=\mathbb{E}\left\{{\left|x\right|}^2\right\},{\tilde{\sigma }}_x^2=\mathbb{E}\left\{x^2\right\},$$

(1)

where $\mathbb{E}\{·\}$ denotes the statistical expectation. It is easy to verify that ${\sigma }_x^2\ge 0$ and $|{\tilde{\sigma }}_x^2|\le {\sigma }_x^2$, which is the sufficient and necessary condition for the variance ${\sigma }_x^2$ and complementary variance ${\tilde{\sigma }}_x^2$ to be a valid pair. The impropriety degree, or circularity coefficient of x, is defined as

$${\kappa }_x=\frac{|{\tilde{\sigma }}_x^2|}{{\sigma }_x^2}.$$

(2)

Obviously, we have $0\le {\kappa }_x\le 1$. The random variable x is called proper if ${\tilde{\sigma }}_x^2=0$, i.e., ${\kappa }_x=0$; otherwise, it is called improper. Specifically, x is called maximally improper if ${\kappa }_x=1$. Furthermore, if x is a Gaussian variable, then the differential entropy of x is a function of both its variance and complementary variance, given by [18]

$$h\left(x\right)=\frac{1}{2}{log}_2\left(\pi e\left({\sigma }_x^4-{\left|{\tilde{\sigma }}_x^2\right|}^2\right)\right)=\frac{1}{2}{log}_2\left(\pi e{\sigma }_x^4\left(1-{\kappa }_x^2\right)\right).$$

(3)

The interested readers are referred to [18,25,37] for a comprehensive treatment of the topic.

2.2. System Description

Consider a downlink NOMA system where a single-antenna base station (BS) sends independent information to two single-antenna receivers, including a strong channel user (SU, user 1) and a weak channel user (WU, user 2), as shown in Figure 1. The channel condition of the SU is satisfactory, but that of the WU is significantly degraded.

The received signal of the SU and WU can, respectively, be expressed as

$$\begin{array}{cc}\hfill y_1=& \sqrt{p_2}{h}_1{x}_2+\underset{z_{12}}{\underbrace{\sqrt{p_1}{h}_1{x}_1+n_1}},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill y_2=& \sqrt{p_2}{h}_2{x}_2+\underset{z_2}{\underbrace{\sqrt{p_1}{h}_2{x}_1+n_2}},\hfill \end{array}$$

(5)

where $h_i$ denotes the channel coefficient from the BS to user $i,i=1,2$, modeled as a Rayleigh fading channel; $n_i$ represents the noise at the receiver end of user i, modeled as a zero-mean white proper Gaussian additive random variable with variance ${\sigma }_{n_i}^2$; $p_i$ is the allocated power of user i, and $x_i$ is the transmitted signal intended for user i with zero mean and unit variance. According to the NOMA principle, the WU should be allocated with a higher power than the SU, i.e., $p_1\le p_2$, so that the WU can decode its signal directly by treating the signal of the SU as interference, and the term $z_2$ in (5) denotes the interference-plus-noise component of the WU. As for the SU, it first decodes the signal of the WU and subtracts it from the received signal $y_1$ by performing SIC, and then detects its own signal. The term $z_{12}$ in (4) denotes the interference-plus-noise component to decode the signal of the WU at the SU end. In practice, the SIC detector at the SU is likely to be imperfect, so there is a residual interference component after decoding. The received signal of the SU after SIC can be modeled as [34]

$$y_{1,\mathrm{sic}}=\sqrt{p_1}{h}_1{x}_1+\underset{z_1}{\underbrace{\sqrt{p_2}{h}_r{x}_2+n_1}},$$

(6)

where $h_r$ represents the residual interference channel coefficient for the imperfect SIC at the SU, and $z_1$ is the interference-plus-noise component to decode its own signal after SIC.

By the maximum entropy theorem, the PGS can achieve the maximum achievable rate in an additive white Gaussian noise channel [17]. Since the channel condition of the WU has been severely degraded, but the channel condition of the SU makes its QoS requirement easy to fulfill, we assume that the WU uses PGS, and the SU can adopt either PGS or IGS according to its power constraints. Therefore, $x_1$ in (4)–(6) is an improper Gaussian signal with its impropriety degree denoted as $\kappa$, and $x_2$ is proper. Since both $x_1$ and $x_2$ have unit variance, i.e., ${\sigma }_{x_1}^2={\sigma }_{x_2}^2=1$, their complementary variances obey $|{\tilde{\sigma }}_{x_1}^2|=\kappa$ and ${\tilde{\sigma }}_{x_2}^2=0$, respectively. Based on the analysis in [31], for a received signal y and an interference-plus-noise signal z, the achievable rate is given by

$$R=h\left(y\right)-h\left(z\right)=\frac{1}{2}{log}_2\frac{{\sigma }_y^4-{\left|{\tilde{\sigma }}_y^2\right|}^2}{{\sigma }_z^4-{\left|{\tilde{\sigma }}_z^2\right|}^2}.$$

(7)

According to (5), the variances and complementary variances of $y_2$ and $z_2$ can be, respectively, given by

$$\begin{array}{cc}\hfill {\sigma }_{y_2}^2& =p_2{\left|h_2\right|}^2+p_1{\left|h_2\right|}^2+{\sigma }_{n_2}^2,\hfill \\ \hfill {\sigma }_{z_2}^2& =p_1{\left|h_2\right|}^2+{\sigma }_{n_2}^2,\hfill \\ \hfill \left|{\tilde{\sigma }}_{y_2}^2\right|& =\left|{\tilde{\sigma }}_{z_2}^2\right|=p_1{\left|h_2\right|}^2\kappa .\hfill \end{array}$$

(8)

Upon substituting (8) into (7), the achievable rate of the WU can be given by

$$R_2=h\left(y_2\right)-h\left(z_2\right)=\frac{1}{2}{log}_2\left(1+\frac{p_2^2{\gamma }_2^2+2p_2{\gamma }_2\left(p_1{\gamma }_2+1\right)}{{\left(p_1{\gamma }_2+1\right)}^2-p_1^2{\gamma }_2^2{\kappa }^2}\right),$$

(9)

where ${\gamma }_2={\left|h_2\right|}^2/{\sigma }_{n_2}^2$ is the instantaneous channel-to-noise ratio (CNR) of the WU. Similarly, according to (4), the variances and complementary variances of $y_1$ and $z_{12}$ can, respectively, be given by

$$\begin{array}{cc}\hfill {\sigma }_{y_1}^2& =p_2{\left|h_1\right|}^2+p_1{\left|h_1\right|}^2+{\sigma }_{n_1}^2,\hfill \\ \hfill {\sigma }_{z_{12}}^2& =p_1{\left|h_1\right|}^2+{\sigma }_{n_1}^2,\hfill \\ \hfill \left|{\tilde{\sigma }}_{y_1}^2\right|& =\left|{\tilde{\sigma }}_{z_{12}}^2\right|=p_1{\left|h_1\right|}^2\kappa ,\hfill \end{array}$$

(10)

and the achievable rate to decode the signal of the WU at the SU end is given by

$$R_{12}=h\left(y_1\right)-h\left(z_{12}\right)=\frac{1}{2}{log}_2\left(1+\frac{p_2^2{\gamma }_1^2+2p_2{\gamma }_1\left(p_1{\gamma }_1+1\right)}{{\left(p_1{\gamma }_1+1\right)}^2-p_1^2{\gamma }_1^2{\kappa }^2}\right),$$

(11)

where ${\gamma }_1={\left|h_1\right|}^2/{\sigma }_{n_1}^2$ is the instantaneous CNR of the SU. It can also be obtained from (6) that

$$\begin{array}{cc}\hfill {\sigma }_{y_{1,\mathrm{sic}}}^2& =p_1{\left|h_1\right|}^2+p_2{\left|h_r\right|}^2+{\sigma }_{n_1}^2,\hfill \\ \hfill {\sigma }_{z_1}^2& =p_2{\left|h_r\right|}^2+{\sigma }_{n_1}^2,\hfill \\ \hfill \left|{\tilde{\sigma }}_{y_{1,\mathrm{sic}}}^2\right|& =p_1{\left|h_1\right|}^2\kappa ,\hfill \\ \hfill {\tilde{\sigma }}_{z_1}^2& =0,\hfill \end{array}$$

(12)

and the achievable rate of the SU when decoding its own signal after SIC is given by

$$R_1=h\left(y_{1,\mathrm{sic}}\right)-h\left(z_1\right)=\frac{1}{2}{log}_2\left(1+\frac{p_1^2{\gamma }_1^2\left(1-{\kappa }^2\right)+2p_1{\gamma }_1\left(p_2{\gamma }_r+1\right)}{{\left(p_2{\gamma }_r+1\right)}^2}\right)$$

(13)

according to (10) and (7), where ${\gamma }_r={\left|h_r\right|}^2/{\sigma }_{n_1}^2$ is the instantaneous CNR of the residual interference channel.

3. Outage Probability Analysis

In practice, since the instantaneous CSI may be unavailable at the BS, owing to the lack of uplink feedback resources, only the statistical CSI of both links is available. The channel coefficients from the BS to user $i,i=1,2$, i.e., $h_i$, can be modeled as independent zero-mean proper Gaussian random variables; thus, the instantaneous CSI ${\gamma }_i={\left|h_i\right|}^2/{\sigma }_{n_i}^2$ are exponential random variables with means $\mathbb{E}\left\{{\gamma }_i\right\}={\sigma }_{h_i}^2/{\sigma }_{n_i}^2\triangleq {\overline{\gamma }}_i$. The cumulative distribution functions (CDF) and probability density functions of ${\gamma }_i$ are, respectively, given by

$$\begin{array}{cccc}& F_{{\gamma }_i}\left(\gamma \right)=Pr\left[{\gamma }_i<\gamma \right]=1-exp\left(-\frac{\gamma }{{\overline{\gamma }}_i}\right),\hfill & \hfill & \gamma >0,\hfill \\ \hfill & p_{{\gamma }_i}\left(\gamma \right)=\frac{d}{d\gamma }{F}_{{\gamma }_i}\left(\gamma \right)=\frac{1}{{\overline{\gamma }}_i}exp\left(-\frac{\gamma }{{\overline{\gamma }}_i}\right),\hfill & \hfill & \gamma >0,\hfill \end{array}$$

(14)

where $Pr[·]$ denotes the probability. The average CNR of the SU is greater than that of the WU, i.e., ${\overline{\gamma }}_1\ge {\overline{\gamma }}_2$. The residual interference channel CNR ${\gamma }_r$ can also be modeled as an exponential random variable with a mean of $\mathbb{E}\left\{{\gamma }_r\right\}={\sigma }_{h_r}^2/{\sigma }_{n_1}^2=\xi {\overline{\gamma }}_1$, where $\xi$ represents the level of residual interference [36,38,39,40,41]. The CDF of ${\gamma }_r$ is given by

$$F_{{\gamma }_r}\left({\gamma }_r\right)=Pr\left[{\gamma }_r<\gamma \right]=\left\{\begin{array}{cc}1-exp\left(-\frac{\gamma }{\xi {\overline{\gamma }}_1}\right),\hfill & \gamma >0,\hfill \\ 0,\hfill & \gamma \le 0.\hfill \end{array}\right.$$

(15)

The case $\xi =0$ refers to perfect SIC, and $\xi =1$ means fully imperfect SIC. In what follows, the overall outage probability of the proposed downlink NOMA system employing IGS is investigated to assess its error performance.

3.1. Weak User Outage Probability

Let $R_{0,i}$ be the target rate of user $i,i=1,2$. The outage probability of the WU, denoted by $P_{\mathrm{out},2}$, is defined as

$$P_{\mathrm{out},2}=Pr\left[R_2<R_{0,2}\right].$$

(16)

Substituting (9) into (16), we have

$$\begin{array}{cc}\hfill P_{\mathrm{out},2}=& Pr\left[\frac{p_2^2{\gamma }_2^2+2p_2{\gamma }_2\left(p_1{\gamma }_2+1\right)}{{\left(p_1{\gamma }_2+1\right)}^2-p_1^2{\gamma }_2^2{\kappa }^2}<{\Gamma }_2\right]\hfill \\ \hfill =& Pr\left[\left({\gamma }_2^{-1}+p_2{\tilde{v}}_2\right)\left({\gamma }_2^{-1}-p_2{v}_2\right)>0\right],\hfill \end{array}$$

(17)

where ${\Gamma }_2=2^{2R_{0,2}}-1$,

$${\tilde{v}}_2=\frac{\sqrt{p_2^2\left(1+{\Gamma }_2\right)+{\Gamma }_2^2{p}_1^2{\kappa }^2}-p_2+{\Gamma }_2{p}_1}{p_2{\Gamma }_2}>0,$$

(18)

$$v_2=\frac{\sqrt{p_2^2\left(1+{\Gamma }_2\right)+{\Gamma }_2^2{p}_1^2{\kappa }^2}+p_2-{\Gamma }_2{p}_1}{p_2{\Gamma }_2}.$$

(19)

Since ${\gamma }_2^{-1}+p_2{\tilde{v}}_2>0$, if $v_2>0$, according to (14), the outage probability of the WU in (17) can be rewritten as

$$P_{\mathrm{out},2}=Pr\left[{\gamma }_2<\frac{1}{p_2{v}_2}\right]=1-exp\left(-\frac{1}{{\overline{\gamma }}_2{p}_2{v}_2}\right).$$

(20)

Otherwise, if $v_2\le 0$, then the inequality inside the probability in (17) holds as long as ${\gamma }_2>0$, to yield $P_{\mathrm{out},2}=1$. Overall, the outage probability of the WU is given by

$$P_{\mathrm{out},2}=\left\{\begin{array}{cc}1-exp\left(-\frac{1}{{\overline{\gamma }}_2{p}_2{v}_2}\right),\hfill & v_2>0,\hfill \\ 1,\hfill & v_2\le 0,\hfill \end{array}\right.$$

(21)

where $v_2$ is given in (19).

The rest of this paper only considers the case when $P_{\mathrm{out},2}<1$, or equivalently, $v_2>0$ holds, so that the outage does not always occur.

3.2. Strong User Outage Probability

An outage occurs at the SU end when the achievable rate to decode its signal is lower than its target rate $R_{0,1}$, or the rate to decode the signal of the WU is lower than the target rate $R_{0,2}$ of the WU. The outage probability of the SU is given by

$$P_{\mathrm{out},1}=Pr\left[R_{12}<R_{0,2}\cup R_1<R_{0,1}\right].$$

(22)

Upon substituting (11) and (13) into (22), we have

$$\begin{array}{cc}\hfill P_{\mathrm{out},1}=& Pr\left[\frac{p_2^2{\gamma }_1^2+2p_2{\gamma }_1\left(p_1{\gamma }_1+1\right)}{{\left(p_1{\gamma }_1+1\right)}^2-p_1^2{\gamma }_1^2{\kappa }^2}<{\Gamma }_2\right.\hfill \\ \hfill & \left.\cup \frac{p_1^2{\gamma }_1^2\left(1-{\kappa }^2\right)+2p_1{\gamma }_1\left(p_2{\gamma }_r+1\right)}{{\left(p_2{\gamma }_r+1\right)}^2}<{\Gamma }_1\right],\hfill \end{array}$$

(23)

where ${\Gamma }_1=2^{2R_{0,1}}-1$. Similar to the analysis in Section 3.1, when $v_2>0$, the first inequality in (23) holds if and only if ${\gamma }_1<1/\left(p_2{v}_2\right)$. Note that both ${\gamma }_1$ and ${\gamma }_r$ are in the second inequality in (23). Given a fixed ${\gamma }_1$, this inequality can be rewritten as

$${\gamma }_r>{\gamma }_{r,0}\left({\gamma }_1\right)\triangleq v_1{\gamma }_1-\frac{1}{p_2},$$

(24)

where

$$v_1\triangleq \frac{p_1\left[1+\sqrt{1+{\Gamma }_1\left(1-{\kappa }^2\right)}\right]}{p_2{\Gamma }_1}.$$

(25)

Therefore, $P_{\mathrm{out},1}$ is given by

$$\begin{array}{cc}\hfill P_{\mathrm{out},1}=& Pr\left[{\gamma }_1<\frac{1}{p_2{v}_2}\cup {\gamma }_r>{\gamma }_{r,0}\left({\gamma }_1\right)\right]\hfill \\ \hfill =& Pr\left[{\gamma }_1<\frac{1}{p_2{v}_2}\right]+Pr\left[{\gamma }_1\ge \frac{1}{p_2{v}_2}\cap {\gamma }_r>{\gamma }_{r,0}\left({\gamma }_1\right)\right]\hfill \\ \hfill =& F_{{\gamma }_1}\left(\frac{1}{p_2{v}_2}\right)+{\int }_{\frac{1}{p_2{v}_2}}^{\infty }Pr\left[{\gamma }_r>{\gamma }_{r,0}\left({\gamma }_1\right)\mid {\gamma }_1\right]p_{{\gamma }_1}\left({\gamma }_1\right)d{\gamma }_1\hfill \\ \hfill =& F_{{\gamma }_1}\left(\frac{1}{p_2{v}_2}\right)+\underset{E}{\underbrace{{\int }_{\frac{1}{p_2{v}_2}}^{\infty }\left[1-F_{{\gamma }_r}\left({\gamma }_{r,0}\left({\gamma }_1\right)\right)\right]p_{{\gamma }_1}\left({\gamma }_1\right)d{\gamma }_1}}.\hfill \end{array}$$

(26)

According to (14) and (15), function E in (26) can be rewritten as

$$E=\left\{\begin{array}{cc}\frac{1}{{\overline{\gamma }}_1}{\int }_{\frac{1}{p_2{v}_2}}^{\infty }exp\left(-\frac{{\gamma }_{r,0}\left({\gamma }_1\right)}{\xi {\overline{\gamma }}_1}-\frac{{\gamma }_1}{{\overline{\gamma }}_1}\right)d{\gamma }_1,\hfill & v_1>v_2,\hfill \\ F_{{\gamma }_1}\left(\frac{1}{p_2{v}_1}\right)-F_{{\gamma }_1}\left(\frac{1}{p_2{v}_2}\right)\hfill \\ +\frac{1}{{\overline{\gamma }}_1}{\int }_{\frac{1}{p_2{v}_1}}^{\infty }exp\left(-\frac{{\gamma }_{r,0}\left({\gamma }_1\right)}{\xi {\overline{\gamma }}_1}-\frac{{\gamma }_1}{{\overline{\gamma }}_1}\right)d{\gamma }_1,\hfill & v_1\le v_2.\hfill \end{array}\right.$$

(27)

Upon combining (14), (24), (26), and (27), it is not difficult to verify that the SU outage probability $P_{\mathrm{out},1}$ can be further expressed as

$$P_{\mathrm{out},1}=\left\{\begin{array}{cc}1-exp\left(-\frac{1}{{\overline{\gamma }}_1{p}_2{v}_2}\right)\left[1-{\displaystyle \frac{exp\left[-\frac{1}{p_2\xi {\overline{\gamma }}_1}\left(\frac{v_1}{v_2}-1\right)\right]}{\frac{v_1}{\xi }+1}}\right],\hfill & v_1>v_2,\hfill \\ 1-exp\left(-\frac{1}{{\overline{\gamma }}_1{p}_2{v}_1}\right)\left[1-{\displaystyle \frac{1}{\frac{v_1}{\xi }+1}}\right],\hfill & v_1\le v_2,\hfill \end{array}\right.$$

(28)

where $v_1$ and $v_2$ are functions of $p_1$, $p_2$, and $\kappa$, given in (25) and (19), respectively.

4. Optimal Signaling Design

We next study the outage performance optimization of the SU by jointly optimizing its power $p_1$ and impropriety degree $\kappa$, subject to a predetermined QoS of the WU represented by a maximum outage probability threshold $P_{\mathrm{out},\mathrm{th}}$ for a target rate $R_{0,2}$. The outage performance optimization problem can be formulated as

$$\begin{array}{cc}\hfill \underset{p_1,\kappa }{\min }& P_{\mathrm{out},1}\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathrm{subject}\mathrm{to}& P_{\mathrm{out},2}\le P_{\mathrm{out},\mathrm{th}},\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill & 0<p_1\le p_{1,\max },\hfill \end{array}$$

(29b)

$$\begin{array}{cc}\hfill & 0\le \kappa \le 1,\hfill \end{array}$$

(29c)

$$\begin{array}{cc}\hfill & v_2>0,\hfill \end{array}$$

(29d)

where $p_{1,\max }$ represents the maximum power budget of the SU. According to our system model, the WU is allocated with more power than the SU, so $p_{1,\max }$ should not be greater than $p_2$. However, we will ignore this constraint while solving the optimization (29) in order to show that our proposed optimal signaling design still works when $p_1>p_2$. In the PGS design, the impropriety degree $\kappa$ of the SU is always zero, so its power $p_1$ should be optimized subject to (29a,b,d).

According to (28), $P_{\mathrm{out},1}$ can be viewed as a function of $v_1$ and $v_2$ with the following properties:

• Given a fixed $v_2$, $P_{\mathrm{out},1}$ decreases in $v_1$.
• Given a fixed $v_1$, $P_{\mathrm{out},1}$ decreases in $v_2$ when $v_2<v_1$ and remains static when $v_2\ge v_1$. Note that when $v_2<v_1$, the partial derivative of $P_{\mathrm{out},1}$ with respect to $v_2$ is

  $$\frac{\partial P_{\mathrm{out},1}}{\partial v_2}=-\frac{1}{{\overline{\gamma }}_1{p}_2{v}_2^2}exp\left(-\frac{1}{{\overline{\gamma }}_1{p}_2{v}_2}\right)\left[1-exp\left[-\frac{1}{p_2\xi {\overline{\gamma }}_1}\left(\frac{v_1}{v_2}-1\right)\right]\right]<0.$$

  (30)

Upon considering the expression of $P_{\mathrm{out},2}$ in (21) together, it is obvious that a greater $v_2$ may improve the outage performance of both the SU and the WU, and a greater $v_1$ can improve the outage performance of the SU while maintaining the outage probability of the WU.

Upon replacing $P_{\mathrm{out},2}$ in (29a) with the expression in (21), we have

$$v_2\ge v_{2,\mathrm{th}}\triangleq -\frac{1}{{\overline{\gamma }}_2{p}_{2}log\left(1-P_{\mathrm{out},\mathrm{th}}\right)}.$$

(31)

According to (25), we have $v_1>0$, and

$${\kappa }^2=1-\frac{p_2}{p_1}{v}_1\left(\frac{p_2{\Gamma }_1}{p_1}{v}_1-2\right).$$

(32)

Substituting (32) into (19) yields $p_1=\lambda (v_1,v_2)p_2$, where

$$\lambda \left(v_1,v_2\right)\triangleq \frac{{\Gamma }_1{v}_1^2+{\left(v_2-\frac{1}{{\Gamma }_2}\right)}^2-\frac{1+{\Gamma }_2}{{\Gamma }_2^2}}{2\left(v_1-v_2+\frac{1}{{\Gamma }_2}\right)}.$$

(33)

Therefore, $p_1$ and ${\kappa }^2$ can, respectively, be expressed as

$$\left\{\begin{array}{c}{p}_1=\lambda \left(v_1,v_2\right)p_2,\hfill \\ {\kappa }^2=1-\frac{v_1}{\lambda \left(v_1,v_2\right)}\left(\frac{{\Gamma }_1{v}_1}{\lambda \left(v_1,v_2\right)}-2\right).\hfill \end{array}\right.$$

(34)

Substituting (34) into conditions (29b,c) yields

$$\begin{array}{cc}\hfill {\kappa }^2\ge 0\Rightarrow & \lambda \left(v_1,v_2\right)\ge \left(\sqrt{1+{\Gamma }_1}-1\right)v_1,\hfill \\ \hfill \kappa \le 1\Rightarrow & \lambda \left(v_1,v_2\right)\le \frac{{\Gamma }_1{v}_1}{2},\hfill \\ \hfill p_1\le p_{1,\max }\Rightarrow & \lambda \left(v_1,v_2\right)\le {\lambda }_m,\hfill \end{array}$$

(35)

where ${\lambda }_m=p_{1,\max }/p_2$. To ease the analysis, denote $v_2^{\mathrm{p}}\left(v_1\right)$, $v_2^{\mathrm{i}}\left(v_1\right)$, and $v_2^{\mathrm{m}}\left(v_1\right)$ as the values of $v_2$ that make both sides of the inequalities in (35) equal, respectively, given by

$$\begin{array}{cc}\hfill v_2^{\mathrm{p}}\left(v_1\right)=& \frac{1}{\sqrt{1+{\Gamma }_2}-1}-\left(\sqrt{1+{\Gamma }_1}-1\right)v_1,\hfill \\ \hfill v_2^{\mathrm{i}}\left(v_1\right)=& \sqrt{\frac{1+{\Gamma }_2}{{\Gamma }_2^2}+\frac{{\Gamma }_1^2{v}_1^2}{4}}-\frac{{\Gamma }_1{v}_1}{2}+\frac{1}{{\Gamma }_2},\hfill \\ \hfill v_2^{\mathrm{m}}\left(v_1\right)=& \sqrt{{\lambda }_m^2+2{\lambda }_m{v}_1-{\Gamma }_1{v}_1^2+\frac{1+{\Gamma }_2}{{\Gamma }_2^2}}+\frac{1}{{\Gamma }_2}-{\lambda }_m.\hfill \end{array}$$

(36)

Both $v_2^{\mathrm{p}}\left(v_1\right)$ and $v_2^{\mathrm{i}}\left(v_1\right)$ decrease in $v_1$. The function $v_2^{\mathrm{m}}\left(v_1\right)$ increases in $v_1$ when $0<v_1\le {\lambda }_m/{\Gamma }_1$ and decreases in $v_1$ when $v_1\ge {\lambda }_m/{\Gamma }_1$. Next, the following cases are considered to simplify the conditions in (35):

• When $v_1-v_2+1/{\Gamma }_2=0$, the numerator on the right-hand side of (33) must be zero. After a few mathematical manipulations, we have

  $$\begin{array}{cc}\hfill v_1=& v_1^{\mathrm{c}}\triangleq \frac{\sqrt{1+{\Gamma }_2}}{{\Gamma }_2\sqrt{1+{\Gamma }_1}},\hfill \\ \hfill v_2=& v_2^{\mathrm{c}}\triangleq \frac{1}{{\Gamma }_2}\left(1+\frac{\sqrt{1+{\Gamma }_2}}{\sqrt{1+{\Gamma }_1}}\right),\hfill \end{array}$$

  (37)

  and

  $$p_1=\frac{v_1^{\mathrm{c}}{p}_2{\Gamma }_1}{1+\sqrt{1+{\Gamma }_1\left(1-{\kappa }^2\right)}},$$

  (38)

  where $\kappa$ can be assigned any value under the conditions in (29b,c). Note that $p_1$ given in (38) is increasing in $\kappa$, so that when $\kappa =0$, $p_1$ reaches its minimum $v_1^{\mathrm{c}}{p}_2(\sqrt{1+{\Gamma }_1}-1)$, which has to be no greater than $p_{1,\max }$, and thus $v_1^{\mathrm{c}}\le v_1^{\mathrm{pm}}$, in which

  $$v_1^{\mathrm{pm}}=\frac{{\lambda }_m}{\sqrt{1+{\Gamma }_1}-1}$$

  (39)

  is the value of $v_1$ at $p_1=p_{1,\max }$ and $\kappa =0$, to give $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)=v_2^{\mathrm{m}}\left(v_1^{\mathrm{pm}}\right)$. We prefer to let $\kappa =0$ in order to achieve the lowest power of the signal of the SU. Moreover, it is not difficult to verify that

  $$v_2^{\mathrm{c}}=v_2^{\mathrm{p}}\left(v_1^{\mathrm{c}}\right)=v_2^{\mathrm{i}}\left(v_1^{\mathrm{c}}\right)=v_2^{\mathrm{m}}\left(v_1^{\mathrm{c}}\right).$$

  (40)
• When $v_1-v_2+1/{\Gamma }_2<0$, according to (35), we have

  $$\begin{array}{cc}\hfill {\kappa }^2\ge 0\Rightarrow & v_2\le v_2^{\mathrm{p}}\left(v_1\right),\hfill \\ \hfill \kappa \le 1\Rightarrow & v_2\ge v_2^{\mathrm{i}}\left(v_1\right),\hfill \\ \hfill p_1\le p_{1,\max }\Rightarrow & v_2\ge v_2^{\mathrm{m}}\left(v_1\right).\hfill \end{array}$$

  (41)

  Both $v_2^{\mathrm{p}}(·)$ and $v_2^{\mathrm{i}}(·)$ are decreasing functions according to (36), and $v_1^{\mathrm{c}}-v_2^{\mathrm{c}}+1/{\Gamma }_2=0$ according to (37), to give $v_1<v_1^{\mathrm{c}}$ and $v_2>v_2^{\mathrm{c}}$. According to (36), the inequality $v_2^{\mathrm{p}}\left(v_1\right)>v_2^{\mathrm{i}}\left(v_1\right)$ holds as long as $0<v_1<v_1^{\mathrm{c}}$. Since the inequality $v_2^{\mathrm{p}}\left(v_1\right)\ge v_2^{\mathrm{m}}\left(v_1\right)$ can be equivalently written as $(v_1-v_1^{\mathrm{c}})(v_1-v_1^{\mathrm{pm}})\ge 0$, $v_1$ lies in the interval $(0,v_1^{\mathrm{pm}}]\cap (0,v_1^{\mathrm{c}})$.
• When $v_1-v_2+1/{\Gamma }_2>0$, according to (35), we have

  $$\begin{array}{cc}\hfill {\kappa }^2\ge 0\Rightarrow & v_2\ge v_2^{\mathrm{p}}\left(v_1\right),\hfill \\ \hfill \kappa \le 1\Rightarrow & v_2\le v_2^{\mathrm{i}}\left(v_1\right),\hfill \\ \hfill p_1\le p_{1,\max }\Rightarrow & v_2\le v_2^{\mathrm{m}}\left(v_1\right).\hfill \end{array}$$

  (42)

  Similar to the analysis in case 2, $v_1$ and $v_2$ satisfy $v_1>v_1^{\mathrm{c}}$ and $v_2<v_2^{\mathrm{c}}$, the inequality $v_2^{\mathrm{p}}\left(v_1\right)<v_2^{\mathrm{i}}\left(v_1\right)$ holds for any $v_1>v_1^{\mathrm{c}}$, and $v_2^{\mathrm{p}}\left(v_1\right)\le v_2^{\mathrm{m}}\left(v_1\right)$, as long as $0\le v_1\le v_1^{\mathrm{pm}}$. Hence, $v_1\in (v_1^{\mathrm{c}},v_1^{\mathrm{pm}}]$.

In summary, the conditions in (29) can be equivalently rewritten as

$$\left\{\begin{array}{c}{v}_2^{\mathrm{L}}\left(v_1\right)\le v_2\le v_2^{\mathrm{U}}\left(v_1\right),\hfill \\ v_2\ge v_{2,\mathrm{th}},\hfill \\ 0<v_1\le v_1^{\mathrm{pm}},\hfill \end{array}\right.$$

(43)

where $v_2^{\mathrm{L}}(·)$ and $v_2^{\mathrm{U}}(·)$ are, respectively, given by

$$v_2^{\mathrm{L}}\left(v_1\right)=\left\{\begin{array}{cc}\max \left\{v_2^{\mathrm{i}}\left(v_1\right),v_2^{\mathrm{m}}\left(v_1\right)\right\},\hfill & v_1\le v_1^{\mathrm{c}},\hfill \\ v_2^{\mathrm{p}}\left(v_1\right),\hfill & v_1>v_1^{\mathrm{c}},\hfill \end{array}\right.$$

(44)

$$\begin{array}{cc}\hfill v_2^{\mathrm{U}}\left(v_1\right)& =\left\{\begin{array}{cc}{v}_2^{\mathrm{p}}\left(v_1\right),\hfill & v_1\le v_1^{\mathrm{c}},\hfill \\ \min \left\{v_2^{\mathrm{i}}\left(v_1\right),v_2^{\mathrm{m}}\left(v_1\right)\right\},\hfill & v_1>v_1^{\mathrm{c}},\hfill \end{array}\right.\hfill \\ \hfill & =\left\{\begin{array}{cc}{v}_2^{\mathrm{p}}\left(v_1\right),\hfill & v_1\le v_1^{\mathrm{c}},\hfill \\ v_2^{\mathrm{i}}\left(v_1\right),\hfill & v_1^{\mathrm{c}}<v_1\le v_1^{\mathrm{im}},\hfill \\ v_2^{\mathrm{m}}\left(v_1\right),\hfill & v_1>\max \left\{v_1^{\mathrm{c}},v_1^{\mathrm{im}}\right\},\hfill \end{array}\right.\hfill \end{array}$$

(45)

and

$$v_1^{\mathrm{im}}=\frac{2{\lambda }_m}{{\Gamma }_1}$$

(46)

is the value of $v_1$ at $p_1=p_{1,\max }$ and $\kappa =1$, making $v_2^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)=v_2^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right)$. Since $v_2^{\mathrm{U}}\left(v_1\right)<v_2^{\mathrm{U}}\left(0\right)={(\sqrt{1+{\Gamma }_2}-1)}^{-1}$ for each $v_1>0$, $v_{2,\mathrm{th}}$ must be smaller than ${(\sqrt{1+{\Gamma }_2}-1)}^{-1}$, to give $p_2>-(\sqrt{1+{\Gamma }_2}-1)/[{\overline{\gamma }}_{2}log(1-P_{\mathrm{out},\mathrm{th}})]$. Figure 2 plots the feasible region of $v_1$ and $v_2$ for an exemplary NOMA scheme for $R_1=1$ bps/Hz, $R_2=1$ bps/Hz, ${\lambda }_m=0.75$, and $v_{2,\mathrm{th}}=0.4$, where the quantities $v_1^{\mathrm{c}}$, $v_1^{\mathrm{im}}$, $v_{1,\mathrm{th}}^{\mathrm{p}}$, $v_{1,\mathrm{th}}^{\mathrm{m}}$, $v_1^{\mathrm{pm}}$, $v_1^{\mathrm{zm}}$, and $v_1^{\mathrm{zp}}$ are the values of $v_1$ at the intersection points. See Table 1 for detailed explanations of these quantities.

Next, the following theorem provides a detailed characterization of the outage probability of the SU, $P_{\mathrm{out},1}$, which helps to find the optimal $v_1$ and $v_2$ to minimize it.

Figure 2. Feasible region of $v_1$ and $v_2$ with $R_1=1$ bps/Hz, $R_2=1$ bps/Hz, ${\lambda }_m=0.75$, and $v_{2,\mathrm{th}}=0.4$.

Figure 2. Feasible region of $v_1$ and $v_2$ with $R_1=1$ bps/Hz, $R_2=1$ bps/Hz, ${\lambda }_m=0.75$, and $v_{2,\mathrm{th}}=0.4$.

Theorem 1.

Given that $v_2$ is a monotonically decreasing function of $v_1$, and $(-\frac{d{v}_2}{d{v}_1})$ is a non-decreasing function of $v_1$. Let $v_1^{\mathrm{d}}=arg{\min }_{v_1}{P}_{\mathrm{out},1}$. Then $P_{\mathrm{out},1}$ monotonically decreases in $v_1$ when $v_1\le v_1^{\mathrm{d}}$ and increases when $v_1\ge v_1^{\mathrm{d}}$, and $v_1^{\mathrm{d}}>v_e>v_2\left(v_1^{\mathrm{d}}\right)$, where $v_e$ is the value of $v_1$, such that $v_2=v_1$.

Proof. 

The derivative of $P_{\mathrm{out},1}$ in (28) with respect to $v_1$ is given by

$$\begin{array}{cc}\hfill \frac{d{P}_{\mathrm{out},1}}{d{v}_1}=& \frac{1}{p_2{\overline{\gamma }}_1{v}_2}exp\left[-\frac{1}{{\overline{\gamma }}_1{p}_2\min \left(v_1,v_2\right)}-\frac{1}{p_2\xi {\overline{\gamma }}_1}\max \left(\frac{v_1}{v_2}-1,0\right)\right]\hfill \\ \hfill & \times \underset{D\left(v_1\right)}{\underbrace{\left\{\frac{1}{v_2}\left[exp\left[\frac{1}{p_2\xi {\overline{\gamma }}_1}\max \left(\frac{v_1}{v_2}-1,0\right)\right]-1\right]\left(-\frac{d{v}_2}{d{v}_1}\right)-\frac{\left(v_1+\xi \right)+p_2{\overline{\gamma }}_1\xi v_2}{{\left(v_1+\xi \right)}^2}\right\}}},\hfill \end{array}$$

(47)

where $D\left(v_1\right)$ is an increasing function of $v_1$ that is smaller than zero when $v_1\le v_e$ and goes to infinity when $v_1/v_2\to \infty$. Therefore, there is a unique $v_1^{\mathrm{d}}$ that makes $D\left(v_1^{\mathrm{d}}\right)=0$, and $v_1^{\mathrm{d}}>v_e$. When $v_1<v_1^{\mathrm{d}}$, we have $\frac{d{P}_{\mathrm{out},1}}{d{v}_1}<0$; thus, $P_{\mathrm{out},1}$ decreases in $v_1$. When $v_1>v_1^{\mathrm{d}}$, we have $\frac{d{P}_{\mathrm{out},1}}{d{v}_1}>0$ and, thus, $P_{\mathrm{out},1}$ increases in $v_1$. □

4.1. PGS Design

In the PGS design, the impropriety degree of the SU $\kappa \equiv 0$ and, thus, $v_2\equiv v_2^{\mathrm{p}}\left(v_1\right)$, where $v_1$ has to meet the constraints in (43). Note that the first condition in (43), i.e., $v_2^{\mathrm{L}}\left(v_1\right)\le v_2\le v_2^{\mathrm{U}}\left(v_1\right)$, holds as long as $0<v_1\le v_1^{\mathrm{pm}}$, so this condition is irrelevant to the range of $v_1$. Therefore, the original problem can be equivalently expressed as

$$\begin{array}{cc}\hfill \underset{v_1}{\min }& P_{\mathrm{out},1}\hfill \\ \hfill \mathrm{subject}\mathrm{to}& v_2^{\mathrm{p}}\left(v_1\right)\ge v_{2,\mathrm{th}},\hfill \\ \hfill & 0<v_1\le v_1^{\mathrm{pm}}.\hfill \end{array}$$

(48)

According to Theorem 1, since

$$-\frac{d{v}_2^{\mathrm{p}}\left(v_1\right)}{d{v}_1}=\sqrt{1+{\Gamma }_1}-1$$

(49)

is a constant, there exists a $v_1^{\mathrm{dp}}$, such that $P_{\mathrm{out},1}$ decreases in $v_1$ when $0<v_1\le v_1^{\mathrm{dp}}$ and increases in $v_1$ when $v_1\ge v_1^{\mathrm{dp}}$, and $v_1^{\mathrm{dp}}>v_e^{\mathrm{p}}$, where

$$v_e^{\mathrm{p}}=\frac{1}{\sqrt{1+{\Gamma }_1}\left(\sqrt{1+{\Gamma }_2}-1\right)}$$

(50)

is the value of $v_1$, making $v_2^{\mathrm{p}}\left(v_1\right)=v_1$. The substitution of $v_2^{\mathrm{p}}(·)$ in (36) into the first constraint in (48) yields

$$v_1\le v_{1,\mathrm{th}}^{\mathrm{p}}\triangleq \frac{1}{\sqrt{1+{\Gamma }_1}-1}\left(\frac{1}{\sqrt{1+{\Gamma }_2}-1}-v_{2,\mathrm{th}}\right).$$

(51)

Therefore, the optimal $v_1=\min \{v_1^{\mathrm{pm}},v_{1,\mathrm{th}}^{\mathrm{p}},v_1^{\mathrm{dp}}\}$. Note that by replacing $-\frac{d{v}_2}{d{v}_1}$ in $D(·)$ given in (47) with the expression in (49), we obtain

$$\begin{array}{cc}\hfill D^{\mathrm{p}}\left(v_1\right)\triangleq & \frac{\sqrt{1+{\Gamma }_1}-1}{v_2^{\mathrm{p}}\left(v_1\right)}\left[exp\left[\frac{1}{p_2\xi {\overline{\gamma }}_1}\max \left(\frac{v_1}{v_2^{\mathrm{p}}\left(v_1\right)}-1,0\right)\right]-1\right]\hfill \\ \hfill & -\frac{\left(v_1+\xi \right)+p_2{\overline{\gamma }}_1\xi v_2^{\mathrm{p}}\left(v_1\right)}{{\left(v_1+\xi \right)}^2},\hfill \end{array}$$

(52)

which is a monotonically increasing function, and $D^{\mathrm{p}}\left(v_1^{\mathrm{dp}}\right)=0$, so that the optimal $v_1$ is given by

$$v_1^{\mathrm{op}}=\left\{\begin{array}{cc}\min \left\{v_1^{\mathrm{pm}},v_{1,\mathrm{th}}^{\mathrm{p}}\right\},\hfill & D^{\mathrm{p}}\left(\min \left\{v_1^{\mathrm{pm}},v_{1,\mathrm{th}}^{\mathrm{p}}\right\}\right)\le 0,\hfill \\ v_1^{\mathrm{dp}},\hfill & D^{\mathrm{p}}\left(\min \left\{v_1^{\mathrm{pm}},v_{1,\mathrm{th}}^{\mathrm{p}}\right\}\right)>0,\hfill \end{array}\right.$$

(53)

where $v_1^{\mathrm{dp}}$ can be obtained with a bisection search, and the optimal transmit power of the SU for minimizing $P_{\mathrm{out},1}$ in the PGS design is given by $p_1^{\mathrm{op}}=v_1^{\mathrm{op}}{p}_2(\sqrt{1+{\Gamma }_1}-1)$.

4.2. IGS Design

When the SU adopts IGS, the quantities $v_1$ and $v_2$ are jointly optimized to minimize $P_{\mathrm{out},1}$ subject to the constraints given in (43). To solve the optimization problem (29) with two variables, we first represent one of the variables with the other one, so as to convert the original problem to a single-variable optimization. Since both $P_{\mathrm{out},1}$ and $P_{\mathrm{out},2}$ decrease in $v_2$, we first set $v_2=v_2^{\mathrm{U}}\left(v_1\right)$. Similar to the analysis in Section 4.1, the first condition in (43), i.e., $v_2^{\mathrm{L}}\left(v_1\right)\le v_2\le v_2^{\mathrm{U}}\left(v_1\right)$, is irrelevant to the range of $v_1$. Therefore, the original problem (29) can be simplified to

$$\begin{array}{cc}\hfill \underset{v_1}{\min }& P_{\mathrm{out},1}\hfill \\ \hfill \mathrm{subject}\mathrm{to}& v_2^{\mathrm{U}}\left(v_1\right)\ge v_{2,\mathrm{th}},0<v_1\le v_1^{\mathrm{pm}}.\hfill \end{array}$$

(54)

Since all three functions $v_2^{\mathrm{p}}(·)$, $v_2^{\mathrm{i}}(·)$, and $v_2^{\mathrm{m}}(·)$ defined in (36) are decreasing functions, $v_2^{\mathrm{U}}(·)$ given in (45) is also decreasing. Therefore, the conditions in (54) can be equivalently written as $0<v_1\le v_{1,\mathrm{th}}$, where

$$v_{1,\mathrm{th}}\triangleq \left\{\begin{array}{cc}{v}_{1,\mathrm{th}}^{\mathrm{p}},\hfill & v_{2,\mathrm{th}}\ge \max \left\{v_2^{\mathrm{c}},v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)\right\},\hfill \\ v_{1,\mathrm{th}}^{\mathrm{i}},\hfill & v_2^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)<v_{2,\mathrm{th}}<v_2^{\mathrm{c}},\hfill \\ v_{1,\mathrm{th}}^{\mathrm{m}},\hfill & v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)\le v_{2,\mathrm{th}}\le \min \left\{v_2^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right),v_2^{\mathrm{c}}\right\},\hfill \\ v_1^{\mathrm{pm}},\hfill & v_{2,\mathrm{th}}<v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right),\hfill \end{array}\right.$$

(55)

$v_{1,\mathrm{th}}^{\mathrm{i}}$ and $v_{1,\mathrm{th}}^{\mathrm{m}}$ are the solutions to $v_2^{\mathrm{i}}(·)=v_{2,\mathrm{th}}$ and $v_2^{\mathrm{m}}(·)=v_{2,\mathrm{th}}$, given by

$$v_{1,\mathrm{th}}^{\mathrm{i}}\triangleq {\displaystyle \frac{1+{\Gamma }_2-{\left({\Gamma }_2{v}_{2,\mathrm{th}}-1\right)}^2}{{\Gamma }_1{\Gamma }_2\left({\Gamma }_2{v}_{2,\mathrm{th}}-1\right)}}$$

(56)

and

$$v_{1,\mathrm{th}}^{\mathrm{m}}\triangleq {\displaystyle \frac{{\lambda }_m+\sqrt{{\lambda }_m^2-{\Gamma }_1\left[{\left(v_{2,\mathrm{th}}-\frac{1}{{\Gamma }_2}+{\lambda }_m\right)}^2-{\lambda }_m^2-\frac{1+{\Gamma }_2}{{\Gamma }_2^2}\right]}}{{\Gamma }_1}},$$

(57)

respectively.

Since $v_2^{\mathrm{U}}\left(v_1\right)$ given in (45) can be any of $v_2^{\mathrm{p}}\left(v_1\right)$, $v_2^{\mathrm{i}}\left(v_1\right)$, or $v_2^{\mathrm{m}}\left(v_1\right)$, three cases are studied in the sequel to determine the optimal $v_1$:

• $v_2^{\mathrm{U}}\left(v_1\right)=v_2^{\mathrm{p}}\left(v_1\right)$.
  In this case, we have $v_1\le v_1^{\mathrm{c}}$, and the IGS scheme reduces to the PGS scheme introduced in Section 4.1. Since $v_1^{\mathrm{c}}<v_2^{\mathrm{c}}=v_2^{\mathrm{p}}\left(v_1^{\mathrm{c}}\right)$, $v_e^{\mathrm{p}}=v_2^{\mathrm{p}}\left(v_e^{\mathrm{p}}\right)$, and $v_2^{\mathrm{p}}(·)$ is a decreasing function, we have $v_1^{\mathrm{c}}<v_e^{\mathrm{p}}<v_1^{\mathrm{dp}}$; thus, $P_{\mathrm{out},1}$ decreases in $v_1$ as $v_1\le v_1^{\mathrm{c}}$. If $v_{1,\mathrm{th}}\le v_1^{\mathrm{c}}$, then the optimal solution to (54) is given by $v_1^{★}=v_{1,\mathrm{th}}$, and the optimal power and impropriety degree of the SU are $p_1^{★}=p_1^{\mathrm{op}}=v_{1,\mathrm{th}}{p}_2(\sqrt{1+{\Gamma }_1}-1)$ and ${\kappa }^{★}=0$, respectively, which are the same as the results presented in Section 4.1. Otherwise, if $v_{1,\mathrm{th}}>v_1^{\mathrm{c}}$, then the optimal $v_1$ that minimizes the SU outage probability $P_{\mathrm{out},1}$ cannot be achieved in $(0,v_1^{\mathrm{c}}]$.
• $v_2^{\mathrm{U}}\left(v_1\right)=v_2^{\mathrm{m}}\left(v_1\right)$.
  In this case, we have $v_1\ge v_1^{\mathrm{c}}$ and $v_2^{\mathrm{m}}\left(v_1\right)\le v_2^{\mathrm{i}}\left(v_1\right)$, i.e., $\max (v_1^{\mathrm{c}},v_1^{\mathrm{im}})\le v_1\le v_{1,\mathrm{th}}$. Since

  $$\frac{d}{d{v}_1}\left(-\frac{d{v}_2^{\mathrm{m}}\left(v_1\right)}{d{v}_1}\right)=\frac{\left({\Gamma }_1+1\right){\lambda }_m^2+{\Gamma }_1\frac{1+{\Gamma }_2}{{\Gamma }_2^2}}{{\left({\lambda }_m^2+2{\lambda }_m{v}_1-{\Gamma }_1{v}_1^2+\frac{1+{\Gamma }_2}{{\Gamma }_2^2}\right)}^{3/2}}>0,$$

  (58)

  there exists a $v_1^{\mathrm{dm}}$, such that $P_{\mathrm{out},1}$ decreases in $v_1$ when $v_1\le v_1^{\mathrm{dm}}$ and increases when $v_1\ge v_1^{\mathrm{dm}}$ according to Theorem 1. Moreover, we have $v_1^{\mathrm{dm}}\ge v_2^{\mathrm{m}}\left(v_1^{\mathrm{dm}}\right)$ and $D^{\mathrm{m}}\left(v_1^{\mathrm{dm}}\right)=0$, where

  $$\begin{array}{cc}\hfill D^{\mathrm{m}}\left(v_1\right)\triangleq & \frac{1}{v_2^{\mathrm{m}}\left(v_1\right)}\left[exp\left[\frac{1}{p_2\xi {\overline{\gamma }}_1}\max \left(\frac{v_1}{v_2^{\mathrm{m}}\left(v_1\right)}-1,0\right)\right]-1\right]\hfill \\ \hfill & \times \left(-\frac{d{v}_2^{\mathrm{m}}\left(v_1\right)}{d{v}_1}\right)-\frac{\left(v_1+\xi \right)+p_2{\overline{\gamma }}_1\xi v_2^{\mathrm{m}}\left(v_1\right)}{{\left(v_1+\xi \right)}^2},\hfill \end{array}$$

  (59)

  $$-\frac{d{v}_2^{\mathrm{m}}\left(v_1\right)}{d{v}_1}=\frac{{\Gamma }_1{v}_1-{\lambda }_m}{\sqrt{{\lambda }_m^2+2{\lambda }_m{v}_1-{\Gamma }_1{v}_1^2+\frac{1+{\Gamma }_2}{{\Gamma }_2^2}}}.$$

  (60)

  Therefore, if $\max (v_1^{\mathrm{c}},v_1^{\mathrm{im}})\le v_{1,\mathrm{th}}$, then the optimal $v_1$ in $[\max (v_1^{\mathrm{c}},v_1^{\mathrm{im}}),v_{1,\mathrm{th}}]$ that minimizes $P_{\mathrm{out},1}$ can be expressed as

  $$v_1^{\mathrm{om}}=\left\{\begin{array}{cc}{v}_1^{\mathrm{im}},\hfill & D^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right)\ge 0,\hfill \\ v_{1,\mathrm{th}},\hfill & D^{\mathrm{m}}\left(v_{1,\mathrm{th}}\right)\le 0,\hfill \\ v_1^{\mathrm{dm}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

  (61)

  where $v_1^{\mathrm{dm}}$ can be obtained with the bisection search.
• $v_2^{\mathrm{U}}\left(v_1\right)=v_2^{\mathrm{i}}\left(v_1\right)$.
  In this case, we have $v_1^{\mathrm{c}}\le v_1\le \min (v_1^{\mathrm{im}},v_{1,\mathrm{th}})$. Unfortunately, the optimization is not simple compared with the previous cases, which have some interesting monotonic characteristics. However, inspired by the solution trend of the previous cases, $v_1$ can be given by

  $$v_1^{\mathrm{oi}}=\left\{\begin{array}{cc}\min \left(v_1^{\mathrm{im}},v_{1,\mathrm{th}}\right),\hfill & D^{\mathrm{i}}\left(\min \left(v_1^{\mathrm{im}},v_{1,\mathrm{th}}\right)\right)\le 0,\hfill \\ v_1^{\mathrm{di}},\hfill & D^{\mathrm{i}}\left(\min \left(v_1^{\mathrm{im}},v_{1,\mathrm{th}}\right)\right)>0,\hfill \end{array}\right.$$

  (62)

  where $v_1^{\mathrm{di}}$ is the solution to $D^{\mathrm{i}}\left(v_1\right)=0$ with $D^{\mathrm{i}}(·)$ given by

  $$\begin{array}{cc}\hfill D^{\mathrm{i}}\left(v_1\right)\triangleq & \frac{1}{v_2^{\mathrm{i}}\left(v_1\right)}\left[exp\left[\frac{1}{p_2\xi {\overline{\gamma }}_1}\max \left(\frac{v_1}{v_2^{\mathrm{i}}\left(v_1\right)}-1,0\right)\right]-1\right]\hfill \\ \hfill & \times \left(-\frac{d{v}_2^{\mathrm{i}}\left(v_1\right)}{d{v}_1}\right)-\frac{\left(v_1+\xi \right)+p_2{\overline{\gamma }}_1\xi v_2^{\mathrm{i}}\left(v_1\right)}{{\left(v_1+\xi \right)}^2},\hfill \end{array}$$

  (63)

  in which

  $$-\frac{d{v}_2^{\mathrm{i}}\left(v_1\right)}{d{v}_1}=\frac{{\Gamma }_1}{2}-\frac{{\Gamma }_1^2{v}_1}{2\sqrt{4\frac{1+{\Gamma }_2}{{\Gamma }_2^2}+{\Gamma }_1^2{v}_1^2}}.$$

  (64)
  Likewise, $v_1^{\mathrm{di}}$ can be obtained with the bisection search.

In summary, the solution to (54) can be given by

$$v_1^{★}=\left\{\begin{array}{cc}{v}_{1,\mathrm{th}},\hfill & v_{1,\mathrm{th}}\le v_1^{\mathrm{c}},\hfill \\ v_1^{\mathrm{om}},\hfill & v_1^{\mathrm{im}}\le v_1^{\mathrm{c}}<v_{1,\mathrm{th}},\hfill \\ v_1^{\mathrm{oi}},\hfill & v_1^{\mathrm{c}}<v_{1,\mathrm{th}}\le v_1^{\mathrm{im}},\hfill \\ arg{\min }_{v_1\in \left\{v_1^{\mathrm{om}},v_1^{\mathrm{oi}}\right\}}{P}_{\mathrm{out},1},\hfill & v_1^{\mathrm{c}}<v_1^{\mathrm{im}}<v_{1,\mathrm{th}}.\hfill \end{array}\right.$$

(65)

Consider the situation that $v_1^{\mathrm{c}}<v_1^{\mathrm{im}}<v_{1,\mathrm{th}}$. Since $P_{\mathrm{out},1}(v_1,v_2)$ is non-increasing in $v_2$, when $v_1>v_1^{\mathrm{im}}$, we have $v_2^{\mathrm{i}}\left(v_1\right)>v_2^{\mathrm{m}}\left(v_1\right)$ and $P_{\mathrm{out},1}(v_1,v_2^{\mathrm{i}}\left(v_1\right))\le P_{\mathrm{out},1}(v_1,v_2^{\mathrm{m}}\left(v_1\right))$, so when $v_1=v_1^{\mathrm{im}}$, we have $\frac{d}{d{v}_1}{P}_{\mathrm{out},1}(v_1,v_2^{\mathrm{i}}\left(v_1\right))\le \frac{d}{d{v}_1}{P}_{\mathrm{out},1}(v_1,v_2^{\mathrm{m}}\left(v_1\right))$ and, thus, $D^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)\le D^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right)$ according to (47), (59), and (63). By combining (61) and (62), the optimal $v_1$ can be given by

$$arg\underset{v_1\in \left\{v_1^{\mathrm{om}},v_1^{\mathrm{oi}}\right\}}{\min }{P}_{\mathrm{out},1}=\left\{\begin{array}{cc}{v}_1^{\mathrm{di}},\hfill & D^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)>0,\hfill \\ v_1^{\mathrm{im}},\hfill & D^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)\le 0\le D^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right),\hfill \\ v_{1,\mathrm{th}},\hfill & D^{\mathrm{m}}\left(v_{1,\mathrm{th}}\right)\le 0,\hfill \\ v_1^{\mathrm{dm}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(66)

By combining (65) and (66), the solution to (54) can be expressed as

$$v_1^{★}=\left\{\begin{array}{cc}{v}_{1,\mathrm{th}},\hfill & v_1^{\mathrm{im}}\le v_{1,\mathrm{th}}\mathrm{and}{D}^{\mathrm{m}}\left(v_{1,\mathrm{th}}\right)\le 0,\mathrm{or},v_1^{\mathrm{im}}>v_{1,\mathrm{th}}\mathrm{and}{D}^{\mathrm{i}}\left(v_{1,\mathrm{th}}\right)\le 0,\hfill \\ v_1^{\mathrm{im}},\hfill & v_1^{\mathrm{im}}\le v_{1,\mathrm{th}}\mathrm{and}{D}^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)\le 0\le D^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right),\hfill \\ v_1^{\mathrm{di}},\hfill & D^{\mathrm{i}}\left(\min \left(v_1^{\mathrm{im}},v_{1,\mathrm{th}}\right)\right)>0,\hfill \\ v_1^{\mathrm{dm}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(67)

where both $v_1^{\mathrm{di}}\in (v_1^{\mathrm{c}},\min (v_1^{\mathrm{im}},v_{1,\mathrm{th}})]$ and $v_1^{\mathrm{dm}}\in [\max (v_1^{\mathrm{im}},v_1^{\mathrm{c}}),v_{1,\mathrm{th}}]$ can be obtained with the bisection search. The corresponding power and impropriety degree of the SU, i.e., $p_1$ and $\kappa$, of the proposed IGS design for minimizing the outage probability of the SU, $P_{\mathrm{out},1}$, are given by

$$p_1^{★}=\left\{\begin{array}{cc}{v}_1^{★}{p}_2\left(\sqrt{1+{\Gamma }_1}-1\right),\hfill & v_1^{★}\le v_1^{\mathrm{c}},\hfill \\ v_1^{★}{p}_2{\Gamma }_1/2,\hfill & v_1^{\mathrm{c}}<v_1^{★}\le v_1^{\mathrm{im}},\hfill \\ p_{1,\max },\hfill & v_1^{★}>\max \left(v_1^{\mathrm{c}},v_1^{\mathrm{im}}\right)\hfill \end{array}\right.$$

(68)

and

$${\kappa }^{★}=\left\{\begin{array}{cc}0,\hfill & v_1^{★}\le v_1^{\mathrm{c}},\hfill \\ 1,\hfill & v_1^{\mathrm{c}}<v_1^{★}\le v_1^{\mathrm{im}},\hfill \\ \sqrt{1-\frac{v_1^{★}{p}_2}{p_{1,\max }}\left(\frac{{\Gamma }_1{v}_1^{★}{p}_2}{p_{1,\max }}-2\right)},\hfill & v_1^{★}>\max \left(v_1^{\mathrm{c}},v_1^{\mathrm{im}}\right),\hfill \end{array}\right.$$

(69)

respectively.

Based on the above analysis, the IGS design outperforms the PGS design if $v_2^{\mathrm{U}}\left(v_1^{★}\right)\ne v_2^{\mathrm{p}}\left(v_1^{★}\right)$, or equivalently, $v_1^{\mathrm{c}}<v_1^{★}<v_1^{\mathrm{pm}}$ according to (45). Since $v_1^{★}$ is given in (67), this condition can also be expressed as, $v_{1,\mathrm{th}}>v_1^{\mathrm{c}}$, and either $v_{1,\mathrm{th}}<v_1^{\mathrm{pm}}$ or $D^{\mathrm{m}}\left(v_1^{\mathrm{pm}}\right)>0$ is satisfied. According to (55), if $v_{2,\mathrm{th}}<v_2^{\mathrm{c}}$ and either $v_{2,\mathrm{th}}>v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)$ or $D^{\mathrm{m}}\left(v_1^{\mathrm{pm}}\right)>0$ is guaranteed, then the IGS design can achieve a lower outage probability of the SU than the PGS design.

4.3. Optimal Outage Performance in Extreme SNR

In what follows, the power ratio ${\lambda }_m=p_{1,\max }/p_2$ is assumed to be a constant in order to investigate how the SNR affects the optimality of our proposed IGS scheme. Note that all the expressions of $v_2^{\mathrm{p}}(·)$, $v_2^{\mathrm{i}}(·)$, and $v_2^{\mathrm{m}}(·)$ given in (36) are irrelevant to the value of $p_2$, so $v_1^{\mathrm{c}}$, $v_1^{\mathrm{pm}}$, and $v_1^{\mathrm{im}}$ all remain constant as $p_2$ increases. However, both $v_{2,\mathrm{th}}$ defined in (31) and $D\left(v_1\right)$ given in (47) monotonically decrease in $p_2$.

At low SNRs, $v_{2,\mathrm{th}}$ reaches a greater value. The PGS design achieves the optimal outage performance of the SU when the SNR is low enough so that $v_{2,\mathrm{th}}\ge v_2^{\mathrm{c}}$, or equivalently, $p_2\le -{[{\overline{\gamma }}_2{v}_2^{\mathrm{c}}log(1-P_{\mathrm{out},\mathrm{th}})]}^{-1}$. Therefore, the IGS design is unlikely to provide better outage performance than the traditional PGS design at a low SNR.

In contrast, at high SNRs, $v_{2,\mathrm{th}}$ reaches a smaller value and even approaches zero as $p_2\to \infty$. According to (55), $v_{1,\mathrm{th}}$ can be expressed as

$$\begin{array}{cc}\hfill v_{1,\mathrm{th}}=& \left\{\begin{array}{cc}{v}_{1,\mathrm{th}}^{\mathrm{m}},\hfill & v_{2,\mathrm{th}}\ge v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right),\hfill \\ v_1^{\mathrm{pm}},\hfill & v_{2,\mathrm{th}}<v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)\hfill \end{array}\right.\hfill \\ \hfill =& \min \left\{v_{1,\mathrm{th}}^{\mathrm{m}},v_1^{\mathrm{pm}}\right\}\hfill \end{array}$$

(70)

at a high SNR, which is greater than $v_1^{\mathrm{im}}$ since $v_2^{\mathrm{m}}(·)$ is decreasing and $v_{2,\mathrm{th}}=v_2^{\mathrm{m}}\left(v_{1,\mathrm{th}}^{\mathrm{m}}\right)<v_2^{\mathrm{m}}\left(v_1^{\mathrm{im}}\right)$. when $p_2$ is large enough so that both $D^{\mathrm{i}}\left(v_1^{\mathrm{im}}\right)$ and $D^{\mathrm{i}}\left(v_{1,\mathrm{th}}\right)$ are negative, according to (67), $v_1^{★}$ can be given by

$$v_1^{★}=\left\{\begin{array}{cc}{v}_{1,\mathrm{th}},\hfill & D^{\mathrm{m}}\left(v_{1,\mathrm{th}}\right)\le 0,\hfill \\ v_1^{\mathrm{dm}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(71)

Note that (71) has a similar form to the expression of $v_1^{\mathrm{op}}$ given in (53). According to (28), as $p_2$ approaches infinity, the outage probability of the SU converges to

$$\underset{p_2\to \infty }{\lim }{P}_{\mathrm{out},1}=\frac{\xi }{v_1+\xi }.$$

(72)

If $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)\le 0$, i.e., ${\lambda }_m\ge {\left(\sqrt{1+{\Gamma }_2}-1\right)}^{-1}$ according to (36) and (39), then $v_{2,\mathrm{th}}>0\ge v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)$, so the IGS design would continue to achieve a lower outage probability of the SU than the PGS design as $p_2$ approaches infinity.

In the PGS design, since $v_2^{\mathrm{p}}(·)$ is a decreasing function, and $v_{2,\mathrm{th}}=v_2^{\mathrm{p}}\left(v_{1,\mathrm{th}}^{\mathrm{p}}\right)>v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)$, we have $v_1^{\mathrm{pm}}>v_{1,\mathrm{th}}^{\mathrm{p}}$ and, thus, $\min \{v_1^{\mathrm{pm}},v_{1,\mathrm{th}}^{\mathrm{p}}\}=v_{1,\mathrm{th}}^{\mathrm{p}}\to v_1^{\mathrm{zp}}$ as $p_2\to \infty$, where $v_1^{\mathrm{zp}}$ is the solution to $v_2^{\mathrm{p}}(·)=0$, given by

$$v_1^{\mathrm{zp}}=\frac{1}{\left(\sqrt{1+{\Gamma }_1}-1\right)\left(\sqrt{1+{\Gamma }_2}-1\right)}.$$

(73)

Since $D^{\mathrm{p}}\left(v_1\right)$ approaches $-\infty$ as $p_2\to \infty$ and approaches ∞ as $v_1\to v_1^{\mathrm{zp}}$, the solution to $D^{\mathrm{p}}(·)=0$, i.e., $v_1^{\mathrm{dp}}$, approaches $v_1^{\mathrm{zp}}$ as $p_2\to \infty$. Therefore, $v_1^{\mathrm{op}}$ given in (53) approaches $v_1^{\mathrm{zp}}$ and, consequently, the outage probability of the SU, $P_{\mathrm{out},1}$, approaches $\xi /(v_1^{\mathrm{zp}}+\xi )$ as $p_2\to \infty$. In the IGS design, according to (70), the quantity $v_{1,\mathrm{th}}=v_{1,\mathrm{th}}^{\mathrm{m}}\to v_1^{\mathrm{zm}}$ as $p_2\to \infty$, where

$$v_1^{\mathrm{zm}}={\displaystyle \frac{{\lambda }_m+\sqrt{{\lambda }_m^2+{\Gamma }_1\left(2{\lambda }_m+1\right)/{\Gamma }_2}}{{\Gamma }_1}}$$

(74)

is the solution to $v_2^{\mathrm{m}}(·)=0$. Similarly, since $D^{\mathrm{m}}\left(v_1\right)$ approaches $-\infty$ as $p_2\to \infty$ and approaches ∞ as $v_1\to v_1^{\mathrm{zm}}$, according to (71), the quantity $v_1^{★}\to v_1^{\mathrm{zm}}$ and, thus, $P_{\mathrm{out},1}$ approaches $\xi /(v_1^{\mathrm{zm}}+\xi )$ as $p_2\to \infty$. When $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)<0$, we have $v_1^{\mathrm{zm}}>v_1^{\mathrm{zp}}$; therefore, the IGS design always yields an outage probability of the SU that is strictly smaller than that of the PGS design. This indicates that the IGS design outperforms the PGS design at a high SNR with a high target rate of the WU or a high maximum allowable power ratio between the SU and the WU.

Otherwise, if $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)>0$, i.e., ${\lambda }_m<{(\sqrt{1+{\Gamma }_2}-1)}^{-1}$, then a sufficiently high SNR would make $v_{2,\mathrm{th}}<v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)$, so $v_{1,\mathrm{th}}=v_1^{\mathrm{pm}}$ according to (70). Since $D^{\mathrm{m}}\left(v_1^{\mathrm{pm}}\right)\to -\infty$ as $p_2\to \infty$, the optimal $v_1$ is $v_1^{★}=v_1^{\mathrm{pm}}$, and, thus, the optimal outage performance of the SU is achieved by the PGS with the maximum power budget of the SU, and the IGS design reduces to the PGS one with $P_{\mathrm{out},1}$ approaching $\xi /(v_1^{\mathrm{pm}}+\xi )$ as the SNR increases. Therefore, the proposed IGS design outperforms the PGS design only in a certain SNR range.

5. Numerical Results

In order to evaluate the benefits of our designed IGS scheme for the downlink two-user NOMA system under imperfect SIC, numerical experiments were conducted in the MATLAB programming environment. Throughout the comparisons between PGS and IGS schemes, the effects of different system parameters on the benefits of the IGS-based system over the PGS-based system were highlighted. The experiments simulated both users’ outage probabilities for both the PGS- and IGS-based systems versus the SNR of the WU, assuming the different maximum power ratios between two users ${\lambda }_m$, levels of residual interference due to imperfect SIC $\xi$, and CNR ratios between two users ${\overline{\gamma }}_1/{\overline{\gamma }}_2$. The SNR of the WU was defined as $p_2/{\sigma }_{n_2}^2$. Unless otherwise specified, the simulation parameters were set according to

Table 2. Simulation parameters.

Parameter	Value	Parameter	Value
${\sigma }_{n_i}^2$	0 dBW	$P_{\mathrm{out},\mathrm{th}}$	0.05
${\overline{\gamma }}_1$	10 dB	$R_{0,1}$	1 bps/Hz
${\overline{\gamma }}_2$	0 dB	$R_{0,2}$	1.5 bps/Hz
$\xi$	0.01	${\lambda }_m$	0.5

. In the PGS scheme, both $v_1^{\mathrm{op}}$ and $v_2^{\mathrm{p}}\left(v_1^{\mathrm{op}}\right)$ were calculated according to (53) and (36) to obtain both users’ outage probabilities, given in (21) and (28). In the IGS scheme, we calculated $v_1^{★}$ according to (67) and $v_2^{\mathrm{U}}\left(v_1^{★}\right)$ according to (45), and obtained both users’ outage probabilities accordingly.

We first simulated the outage probabilities of both users for both the PGS- and IGS-based systems, assuming different maximum power ratios ${\lambda }_m$, given $R_1=R_2=1$ bps/Hz, as shown in Figure 3. The outage performance of the SU in both the PGS and IGS schemes improves with the increase in SNR. When the SNR of the WU is lower than 14.4 dB, both the PGS and IGS schemes at different ${\lambda }_m$ share the same outage probabilities of the WU. This is because $p_2$ is small enough, so that $v_{2,\mathrm{th}}$ is no smaller than both $v_2^{\mathrm{c}}$ and $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)$, making $v_1^{★}=v_1^{\mathrm{op}}\le v_{1,\mathrm{th}}<v_1^{\mathrm{pm}}$. Thus, the PGS scheme achieves the optimal outage performance with $p_1<p_{1,\max }={\lambda }_m{p}_2$, indicating that the minimum achievable $P_{\mathrm{out},1}$ is irrelevant to ${\lambda }_m$. As the SNR increases, the outage probability of the SU under the PGS scheme continues to decrease and eventually saturates at high SNR when $p_1$ reaches $p_{1,\max }$. In comparison, the IGS scheme achieves a lower outage probability of the SU for ${\lambda }_m\ge 0.75$, but the benefit of using IGS eventually decreases and even disappears as the SNR increases, i.e., the IGS scheme outperforms PGS only in a certain range of SNR, which is consistent with the analysis given at the end of Section 4.3. It can also be observed that the IGS scheme is more likely to outperform the PGS one with a larger ${\lambda }_m=p_{1,\max }/p_2$, i.e., a higher maximum power threshold of the SU. This result is consistent with the one in [36], in that improper signaling can achieve higher fairness with a higher power allocation factor of the SU. The outage probability of the WU is equal to its maximum threshold $P_{\mathrm{out},\mathrm{th}}$ at a low SNR. As the SNR increases, the WU achieves a lower outage probability while satisfying the optimal outage performance of the SU. When ${\lambda }_m<1$, since $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)>0$, according to Section 4.3, the optimal outage performance of the SU is achieved by the PGS design with the maximum power of the SU, $p_{1,\max }$, and $P_{\mathrm{out},2}$ is inversely proportional to SNR and identical in both the PGS and IGS designs. However, when ${\lambda }_m=1$, we have $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)=0$; therefore, $P_{\mathrm{out},2}$ decreases more slowly. The IGS-based system achieves a lower outage probability of the WU.

Next, we considered another pair of the target rates of both users, given by $R_1=1$ bps/Hz and $R_2=1.5$ bps/Hz. The outage probabilities of both users versus the SNR of the WU are shown in Figure 4. It is clear that for ${\lambda }_m=0.75$, the outage probabilities of the SU under PGS and IGS saturate at different values at high SNR. The IGS achieves a lower outage probability, and the gap between the IGS and PGS probabilities is about 0.005. This is because $v_2^{\mathrm{p}}\left(v_1^{\mathrm{pm}}\right)<0$ and, thus, $v_1^{★}>v_1^{\mathrm{op}}$ at a high SNR, which makes $P_{\mathrm{out},1}$ smaller according to (72). This is consistent with the benefits of IGS in interference-limited scenarios, which have been mentioned in various previous works, including [36]. However, since the outage probability is used as a metric in our design, only with a high target rate can IGS achieve such benefits.

We further investigated the outage probabilities of both users for both signaling schemes assuming different levels of residual interference $\xi$ after SIC, as shown in Figure 5. As $\xi$ increases, the outage probabilities of the SU increase due to stronger residual interference. Note that the minimum SNRs that allow IGS to achieve a better performance than the PGS scheme are the same for different $\xi$ since the values of $p_2$ that satisfy $v_{2,\mathrm{th}}=v_2^{\mathrm{c}}$ are the same. Figure 6 shows the outage probabilities of both users as functions of $\xi$, assuming different powers of the WU. It can be observed that $P_{\mathrm{out},1}$ is approximately linear in $\xi$ at a high SNR, which is consistent with (72).

Figure 4. The outage probabilities of both the SU and WU versus the SNR of the WU for different ${\lambda }_m$ when $R_1=1$ bps/Hz, $R_2=1.5$ bps/Hz.

Figure 4. The outage probabilities of both the SU and WU versus the SNR of the WU for different ${\lambda }_m$ when $R_1=1$ bps/Hz, $R_2=1.5$ bps/Hz.

Figure 7 shows the outage probabilities of both users assuming different CNRs of the SU, i.e., ${\overline{\gamma }}_1$. Since the CNR of the WU is 0 dB, ${\overline{\gamma }}_1$ can also be considered as the CNR ratio between the SU and the WU. As the CNR of the SU increases, the outage probability of the SU decreases due to its stronger channel strength. The outage probability of the WU is higher at a high SNR as ${\overline{\gamma }}_1$ increases. Similarly, the minimum SNR for IGS to outperform PGS is the same for different ${\overline{\gamma }}_1$. However, $P_{\mathrm{out},1}$ approaches the same value at a high SNR because ${\lim }_{p_2\to \infty }{P}_{\mathrm{out},1}$ given in (72) is irrelevant to ${\overline{\gamma }}_1$.

6. Conclusions

We investigated the outage performance of a two-user downlink NOMA system with a strong user (SU) and a weak user (WU) by adopting improper Gaussian signaling (IGS) and proper Gaussian signaling (PGS) schemes, respectively. With the statistical CSI available at the base station, we derived the closed-form outage probabilities of both users and provided the solution to the optimization by adjusting the power and the impropriety degree of the SU, subject to the outage probability constraints on the WU. We have shown that, whether the proposed IGS design can outperform the PGS design is mainly determined by the target rates of both users, the SNR of the WU, and the maximum allowable power ratio between the SU and the WU. The IGS is more likely to achieve a lower outage of the SU at a medium SNR than at a low SNR. The IGS design can also improve the outage performance of the SU at a fairly high SNR with a high target rate of the WU and a large power ratio between the SU and the WU. Numerical results on the downlink NOMA systems with both PGS and IGS support the analysis.

In future work, we shall consider the case where both users in the system employ IGS, and optimize the outage performance by jointly designing the powers and the impropriety degrees of both users, which can potentially achieve better outage performance than the current results, since in this work we assumed that the WU uses traditional PGS with a fixed signal power. Moreover, since we only investigated the SISO scenario to simplify the performance analysis in this work, we shall further extend our outage performance analysis of NOMA systems with more generalized multi-antenna settings, such as multiple-input–single-output (MISO) and multiple-input–multiple-output (MIMO), to enable a higher data rate and satisfy the high-throughput demand for the expansion of the Internet of Things. It is also of interest to explore the outage performance of IGS-based NOMA systems with advanced interference cancellation techniques, such as deep learning-based approaches or iterative interference cancellation.