Covert Communications in Active-RIS-Aided NOMA Systems: Element Grouping or Not?

Abstract

This paper investigates the impacts of element grouping on the covert communication performance of an active reconfigurable intelligent surface (ARIS)-aided uplink non-orthogonal multiple access (NOMA) system. Through element grouping, each element of the ARIS works in either the reflecting mode to reflect the information signal or the jamming mode to generate a jamming signal. Optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming jointly is necessary to maximize the covert communication rate. To tackle the unsolvable covert communication rate maximization problem, we decouple the original problem into three sub-problems of optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming, respectively. To tackle the mixed-integer non-linear programming for the element grouping, we introduce the arithmetic- and geometric-mean-based penalty term and apply the Dinkelbach transform to reformulate the optimization problem. Next, we propose an alternating optimization algorithm to optimize the system parameters. The numerical results demonstrate the effectiveness of the element grouping in improving the covert communication rate. However, the element grouping scheme achieves a lower covert communication rate performance compared to the scheme without element grouping, indicating that using element grouping for covert communications in the ARIS-aided uplink NOMA system is not the preferred option.

1. Introduction

Reconfigurable intelligent surfaces (RISs), which can be thought of as enablers to configure wireless propagation environments smartly, have been proposed and extensively developed for radar sensing and communications [1,2,3]. In particular, composed of a number of metasurface elements, RISs are capable of reflecting and refracting incident electromagnetic (EM) waves and imposing specified phases, which results in various EM functions, such as beamforming or interference nulling, by smartly configuring each element’s amplitude and phase shift [4,5,6]. With the abundant spatial degree of freedom (DoF) provided by the RIS, the information security performance can be enhanced from the physical-layer security (PLS) perspective [7,8], and the received signal-to-interference-plus-noise ratio (SINR) is also improved [9]. In comparison with traditional communication systems with radio frequency (RF) chains, RISs are expected to play a key role and to be one of the major technologies in future energy and spectrum-efficient wireless communications due to their straightforward structure. Also, the upcoming sixth-generation (6G) wireless networks regard RIS as a promising technology to facilitate massive connections [10].

In conventional passive RISs (PRISs), the metasurface elements are configured with low-power RF switches, allowing the noise to be ignored and resulting in relatively high power efficiency. It has been shown that implementing PRISs with large apertures and high array gains enhanced the performance of wireless communication systems [11]. Unfortunately, a PRIS brings only minimal performance improvement due to the “multiplicative fading” effect [12]. Even if large-scale PRISs with high array gain are employed, they are still ineffective to enlarge coverage ranges of wireless communication systems. To overcome the limitations of PRISs and extend their applications, the idea of active RIS (ARIS) has been introduced recently to scatter the incident signals with both amplifications and adjusted phase shifts [13,14,15,16]. Through amplifying incident waves, the strengths of the scattered signals are comparable to scenarios of direct link transmissions where the “multiplicative fading” effect is significant. Furthermore, an ARIS not only achieves an enhanced array gain and the required signal-to-noise ratio (SNR) at the receiver, but also becomes a PRIS when its amplifiers are turned off, making an ARIS more generalized than a PRIS.

Different from amplify-and-forward (AF) relays that also receive and radiate the amplified signals, ARISs have distinct architectures using a phase-reconfigurable reflection-type amplifier for metasurface elements [17,18]. Currently, ARISs can be implemented in several ways. For example, the incident waves can be reflected with orthogonal polarization by using an amplifying reflection array [19,20,21,22,23,24]. Furthermore, by using a two-layer patch antenna and a phase-reconfigurable reflection amplifier to formulate an active element, the ARIS is capable of beam-scanning and amplification [25]. In addition, both ARISs and PRISs operate in the full-duplex transmission model with full-band response, which is more attractive than half-duplex AF relays equipped with costly RF chains.

Recently, non-orthogonal multiple access (NOMA) technology has been incorporated as part of the upcoming 6G wireless networks [26]. In 6G wireless networks, NOMA plays a pivotal role in facilitating spectrally efficient massive wireless connections [27]. However, due to the open propagation of wireless signals, novel risks associated with information leakage and communication behavior exposure are introduced to the emerging NOMA systems. Ensuring the covertness of NOMA, especially preventing the exposure of NOMA communication behaviors, presents a significant and formidable challenge. In this context, covert communication emerges as a solution to address the security concerns in NOMA systems [10,28]. In [29], covert communications in a PRIS-aided NOMA system were investigated. It was shown that employing a PRIS significantly enhanced the covert communication performance of NOMA systems. Also, ARISs were utilized to maximize the covert communication rate in [30,31].

Since a large number of the metasurface elements are fabricated on a RIS, different functions could be allocated to different elements, and several element grouping schemes were proposed for RIS-aided wireless communication systems [32,33,34]. For example, the incident signal at the grouped PRIS elements could be adjusted to generate the jamming signal in the desired direction [32]. In [33], through element grouping, the incident signal on each PRIS element was amplified for information transmission or modulated into jamming, which effectively improved the PLS performance. On the other hand, using all ARIS elements to reflect the information signal led to certain security risks, whereas using all the ARIS elements to generate the jamming signal led to a decrease in reception quality for legitimate receivers. Therefore, the ARIS’s elements were divided into two groups, one working in the reflecting mode and the other in the jamming mode to improve the PLS performance [34]. It was shown that the element grouping scheme effectively improved the minimum secrecy rate among the NOMA users in an ARIS-aided NOMA system, achieving an enlarged channel gain difference between the legitimate and eavesdropping channels [34].

Different from PLS transmissions, proper randomness is needed to balance between the required covertness and legitimate transmission performance in NOMA systems. In particular, the superimposed signal of the NOMA users has already been regarded as a natural shield to conceal covert communication behaviors. When element grouping is applied in ARIS-aided NOMA systems, how to balance the covertness and legitimate transmission performance becomes critical in the presence of the NOMA superimposed signal. Although the element grouping has been investigated for the PLS, the applications of the element grouping in covert communications have never been investigated. Furthermore, although element grouping can improve the secrecy communication performance for the PLS, whether element grouping enhances the covert communication performance is still unknown. To the best of our knowledge, how to employ element grouping to realize covert communications in ARIS-aided NOMA systems has not been tackled.

In this work, the impacts of element grouping on covert communication performance are investigated for an ARIS-aided uplink NOMA system. In the element grouping scheme, each element of the ARIS works in at least one mode, i.e., information signal reflection or jamming signal reflection. By adjusting the reflection coefficient and working mode for each element, a trade-off between the covertness and legitimate transmission performance can be optimized. By jointly optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming at the base-station (BS), our design goal is to maximize the covert communication rate. To tackle the intractable optimization problem, we decouple the original problem into three sub-problems of optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming, respectively. We then derive closed-form expression for the sub-problem of the NOMA transmit power optimization. Mixed-integer non-linear programming for element grouping is effectively addressed by an arithmetic and geometric mean (AGM)-based penalized Dinkelbach transform approach. Then, an efficient alternating optimization (AO) algorithm is proposed to solve the formulated non-convex covert communication rate maximization problem [35,36]. Various simulation results are provided to verify whether element grouping enhances covert communication performance or not. Our work shows that the scheme without element grouping achieves a better covert communication performance than the scheme with element grouping, which is a valuable and novel finding that has never been reported before. Also, the findings in this paper are valuable and meaningful for a practical design of the ARIS-aided covert communications.

The remaining sections of the paper are organized as follows: Section 2 presents the system model, including the ARIS-aided transmission scheme with element grouping (denoted as the w-EG scheme in the following) and the ARIS-aided transmission scheme without element grouping (denoted as the w/o-EG scheme in the following). The problem formulation and the proposed solutions are presented in Section 3. Simulation results are illustrated in Section 4. Finally, Section 5 concludes the work.

Notations: Throughout the paper, scalars are represented by italic letters, and vectors and matrices are represented by boldfaced lowercase and uppercase letters, respectively. $\mathrm{Pr}\{·\}$ denotes probability, and $\parallel ·\parallel$ denotes the Euclidean norm of a complex vector. Superscript ${(·)}^T$, ${(·)}^{*}$, and ${(·)}^H$ denote vector/matrix transpose, conjugate, and Hermitian transpose, respectively. ${\mathbb{C}}^{M\times N}$ denotes the spaces of $M\times N$ complex-valued matrices. $\mathbf{A}⪰0$ indicates that $\mathbf{A}$ is a positive semi-definite matrix. The symbols $\mathrm{rank}\left(\mathbf{A}\right)$ and $\mathrm{tr}\left(\mathbf{A}\right)$ denote the rank and trace of matrix $\mathbf{A}$, respectively. ${\mathbf{0}}_N$ and ${\mathbf{1}}_N$ denote the N-dimensional all-zero and all-ones vectors, respectively. ${\mathbf{I}}_N$ denote the $N\times N$ identity matrix. $\mathrm{diag}\left(\mathbf{a}\right)$ denotes the $N\times N$ diagonal matrix with diagonal elements $a_1,\dots ,a_N$. The notation $\mathcal{C}\mathcal{N}\left(\nu ,{\sigma }^2\right)$ denotes a circular symmetric complex Gaussian (CSCG) random distribution with mean $\nu$ and variance ${\sigma }^2$.

2. System Model

The considered uplink NOMA system consists of a multi-antenna BS (Bob), a pair of NOMA users (Alice and Carol), and a warden (Willie), as shown in Figure 1. We assume that Alice is a covert user who has high-level of privacy and security requirements and Carol is a public user who only has transmission reliability requirements. For instance, Alice is a law enforcement node needing to hide its transmission pattern, while Carol transmits live streams possessing low or no confidential requirements. The ARIS is deployed between the NOMA users and Bob to not only aid the uplink transmissions from Alice and Carol, but also reflect jamming signals towards Willie to enhance the covert communication performance. In the considered system, the ARIS consists of N reflecting elements, Bob is equipped with M receive antennas, and the remaining nodes (Alice, Carol, and Willie) are each equipped with a single antenna. All the links associated with the ARIS are modeled to experience Rician fading, while all the direct links from the NOMA users (Alice and Carol) to Bob and Willie are modeled to experience Rayleigh fading. We assume that all the links follow quasi-statistic block fading. By exploiting the existing channel estimation methods for RIS-aided systems [37,38,39], Alice and Carol acquire the channel state information (CSI) associated with the ARIS and that of the direct links. To characterize the performance limit of the considered system, we further ignore the channel estimation error and assume that Bob has perfect CSI of Alice and Carol [29,30,31,37,38,39].

Let ${\mathbf{h}}_{ib}\in {\mathbb{C}}^{M\times 1}$ and $h_{iw}\in \mathbb{C}$ denote the direct links from node i to Bob and Willie, respectively, with $i\in \{a,c\}$ representing Alice and Carol, respectively. Similarly, the channels from node i to the ARIS are denoted by ${\mathbf{h}}_{ir}\in {\mathbb{C}}^{N\times 1}$, and the channels from the ARIS to Bob and Willie are denoted by ${\mathbf{H}}_{rb}\in {\mathbb{C}}^{M\times N}$ and ${\mathbf{h}}_{rw}^H\in {\mathbb{C}}^{1\times N}$, respectively. If Alice and Carol transmit the information signals simultaneously in a transmission block, the arrived signal at the ARIS can be written as:

$$\begin{array}{c}\hfill {\mathbf{y}}_r=\sqrt{P_a}{\mathbf{h}}_{ar}{x}_a+\sqrt{P_c}{\mathbf{h}}_{cr}{x}_c+{\mathbf{n}}_r,\end{array}$$

(1)

where $x_i\sim \mathcal{C}\mathcal{N}(0,1)$ is the signal transmitted by node i with $i\in \{a,c\}$, $P_i$ is the transmit power of node i satisfying $0\le P_i\le P_i^{max}$ with $P_i^{max}$ denoting the maximum transmit power, and ${\mathbf{n}}_r\sim \mathcal{C}\mathcal{N}({\mathbf{0}}_N,{\sigma }_r^2{\mathbf{I}}_N)$ is additive noise at the ARIS with zero mean and variance ${\sigma }_r^2$. After the amplitude and phase-shift adjustments, the departure signal from the ARIS can be expressed as:

$$\begin{array}{c}\hfill {\widehat{\mathbf{y}}}_r=\sqrt{P_a}\mathbf{\Theta }{\mathbf{h}}_{ar}{x}_a+\sqrt{P_c}\mathbf{\Theta }{\mathbf{h}}_{cr}{x}_c+\mathbf{\Theta }{\mathbf{n}}_r,\end{array}$$

(2)

where $\mathbf{\Theta }$ is the amplitude–phase-shift coefficient matrix of the ARIS.

The power of the departure signal at the ARIS is constrained by

$$P_a{\parallel \mathbf{\Theta }{\mathbf{h}}_{ar}\parallel }^2+P_c\parallel \mathbf{\Theta }{\mathbf{h}}_{cr}{\parallel }^2+ {\parallel \mathbf{\Theta }\parallel }_F^2{\sigma }_r^2\le P_r^{max},$$

(3)

where $P_r^{max}=P_{\mathrm{ris}}-N{P}_{\mathrm{circuit}}$ is the maximum power budget at the ARIS with $P_{\mathrm{ris}}$ and $P_{\mathrm{circuit}}$ denoting the total power of the ARIS and the power consumption per circuit element, respectively [34]. In the rest of this section, we present two ARIS-aided transmission schemes in detail, namely, the w-EG scheme and the w/o-EG scheme. Each scheme is discussed separately in the following two subsections.

2.1. w-EG Scheme

In this section, we investigate the impacts of the w-EG scheme on the covert communication performance of an ARIS-aided NOMA system. For the w-EG scheme, $\mathbf{\Theta }={\mathbf{\Theta }}_r+{\mathbf{\Theta }}_j$ is the amplitude–phase-shift coefficient matrix of the ARIS with the reflecting matrix ${\mathbf{\Theta }}_r={\mathbf{M}}_r{\overline{\mathbf{\Theta }}}_r$ and jamming matrix ${\mathbf{\Theta }}_j={\mathbf{M}}_j{\tilde{\mathbf{\Theta }}}_j$. Specifically, ${\mathbf{M}}_r=\mathrm{diag}\left({[{\overline{\alpha }}_1,{\overline{\alpha }}_2,\cdots ,{\overline{\alpha }}_N]}^T\right)$ and ${\mathbf{M}}_j=$$\mathrm{diag}\left({[{\tilde{\alpha }}_1,{\tilde{\alpha }}_2,\cdots ,{\tilde{\alpha }}_N]}^T\right)$ are the mode selection matrices with ${\overline{\alpha }}_n,{\tilde{\alpha }}_n\in \{0,1\}$, ${\overline{\alpha }}_n+{\tilde{\alpha }}_n=1$, and $n\in \mathcal{N}\triangleq \{1,2,\cdots ,N\}$. With the aid of the mode selection matrices, each element of the ARIS works in either the reflection mode or the jamming mode. The matrices ${\overline{\mathbf{\Theta }}}_r=\mathrm{diag}({[{\overline{\beta }}_1{e}^{\mathfrak{j}{\overline{\varphi }}_1},{\overline{\beta }}_2{e}^{\mathfrak{j}{\overline{\varphi }}_2},\cdots ,{\overline{\beta }}_N{e}^{\mathfrak{j}{\overline{\varphi }}_N}]}^T)$ and ${\tilde{\mathbf{\Theta }}}_j=$$\mathrm{diag}({[{\tilde{\beta }}_1{e}^{\mathfrak{j}{\tilde{\varphi }}_1},{\tilde{\beta }}_2{e}^{\mathfrak{j}{\tilde{\varphi }}_2},\cdots ,{\tilde{\beta }}_N{e}^{\mathfrak{j}{\tilde{\varphi }}_N}]}^T)$ contain the reflection coefficients for enhancing the NOMA transmission quality and deteriorating Willie’s detection performance, respectively, where ${\overline{\beta }}_n,{\tilde{\beta }}_n\in [0,{\eta }_n]$ and ${\overline{\varphi }}_n,{\tilde{\varphi }}_n\in [0,2\pi )$ denote the nth element’s reflecting amplitude and phase shift, respectively, with ${\eta }_n>1$ denoting the allowed maximum reflecting amplitude.

At Bob, the received signal can be expressed as:

$$\begin{array}{ccc}\hfill {\mathbf{y}}_b& =& \sqrt{P_a}{x}_a({\mathbf{g}}_{ab}+{\mathbf{z}}_{ab})+\sqrt{P_c}{x}_c({\mathbf{g}}_{cb}+{\mathbf{z}}_{cb})\hfill \\ & & +{\mathbf{H}}_{rb}\mathbf{\Theta }{\mathbf{n}}_r+{\mathbf{n}}_b,\hfill \end{array}$$

(4)

where ${\mathbf{g}}_{ib}={\mathbf{h}}_{ib}+{\mathbf{H}}_{rb}{\mathbf{\Theta }}_r{\mathbf{h}}_{ir}$ is the composite transmission channel from node i ($i\in \{a,c\}$) to Bob, ${\mathbf{z}}_{ib}={\mathbf{H}}_{rb}{\mathbf{\Theta }}_j{\mathbf{h}}_{ir}$ is the composite jamming channel from node i ($i\in \{a,c\}$) to Bob, and ${\mathbf{n}}_b\sim \mathcal{C}\mathcal{N}({\mathbf{0}}_M,{\sigma }_b^2{\mathbf{I}}_M)$ is additive noise at Bob with zero mean and variance ${\sigma }_b^2$. Then, a beamformer $\mathbf{w}\in {\mathbb{C}}^{M\times 1}$ is applied at Bob to obtain $y_b={\mathbf{w}}^H{\mathbf{y}}_b$ for further processing. In the considered ARIS-aided uplink NOMA system, the user ordering is conducted based on the channel gains [29,40]. Specifically, the two users, Alice and Carol, are paired based on their channel gains. At Bob, to guarantee the successful successive interference cancellation (SIC), the signal of the user with the stronger channel gain is decoded first. Then, the signal of the user with the weaker channel gain is decoded at the last stage of the SIC processing. To facilitate SIC at Bob, we assume that $\parallel {\mathbf{g}}_{cb}{\parallel }^2>\parallel {\mathbf{g}}_{ab}{\parallel }^2$ is guaranteed by not only $\parallel {\mathbf{h}}_{cb}{\parallel }^2\ge {\parallel {\mathbf{h}}_{ab}\parallel }^2$, but also the amplitude–phase-shift coefficient matrix $\mathbf{\Theta }$ as optimized in the next section. Consequently, $x_a$ is decoded at the last stage of SIC without suffering from inter-user interference. The achievable rates of decoding of $x_a$ and $x_c$ are given by $R_a={log}_2(1+{\gamma }_a)$ and $R_c={log}_2(1+{\gamma }_c)$, respectively, where ${\gamma }_a$ and ${\gamma }_c$ are the received SINRs given by ${\gamma }_a=\frac{P_a{|{\mathbf{w}}^H{\mathbf{g}}_{ab}|}^2}{P_a{|{\mathbf{w}}^H{\mathbf{z}}_{ab}|}^2+\Vert {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\Vert }^2{\sigma }_r^2+{\sigma }_b^2}$ and ${\gamma }_c=\frac{P_c{|{\mathbf{w}}^H{\mathbf{g}}_{cb}|}^2}{P_a{|{\mathbf{w}}^H({\mathbf{g}}_{ab}+{\mathbf{z}}_{ab})|}^2+P_c{|{\mathbf{w}}^H{\mathbf{z}}_{cb}|}^2+\Vert {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\Vert }^2{\sigma }_r^2+{\sigma }_b^2}$, respectively.

In order to judge whether Alice transmits an information signal to Bob or not, Willie performs a binary decision according to the Neyman–Pearson criterion [41], which results in two hypotheses as follows:

$$\begin{array}{c}\hfill {\mathcal{H}}_0:y_w=\sqrt{P_c}{x}_c(g_{cw}+z_{cw})+{\mathbf{h}}_{rw}^H\mathbf{\Theta }{\mathbf{n}}_r+n_w,\end{array}$$

(5)

$$\begin{array}{ccc}\hfill {\mathcal{H}}_1:y_w& =& \sqrt{P_a}(g_{aw}+z_{aw})+\sqrt{P_c}(g_{cw}+z_{cw})\hfill \\ & & +{\mathbf{h}}_{rw}^H\mathbf{\Theta }{\mathbf{n}}_r+n_w,\hfill \end{array}$$

(6)

where $g_{iw}=h_{iw}+{\mathbf{h}}_{rw}^H{\mathbf{\Theta }}_r{\mathbf{h}}_{ir}$ with $i\in \{a,c\}$, $z_{iw}={\mathbf{h}}_{rw}^H{\mathbf{\Theta }}_j{\mathbf{h}}_{ir}$ with $i\in \{a,c\}$, and $n_w\sim \mathcal{C}\mathcal{N}(0,{\sigma }_w^2)$ denotes additive noise at Willie with zero mean and variance ${\sigma }_w^2$. The likelihood functions of the received signals under ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$ can be written as $p_0\left(y_w\right)=e^{-|y_w{|}^2/{\sigma }_0^2}/\pi {\sigma }_0^2$ and $p_1\left(y_w\right)=e^{-|y_w{|}^2/{\sigma }_1^2}/\pi {\sigma }_1^2$, respectively, with ${\sigma }_0^2=P_c|g_{cw}+z_{cw}{|}^2+{\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }\parallel }^2{\sigma }_r^2+{\sigma }_w^2$ and ${\sigma }_1^2=P_a|g_{aw}+z_{aw}{|}^2+P_c|g_{cw}+z_{cw}{|}^2+{\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }\parallel }^2{\sigma }_r^2+{\sigma }_w^2$. Under the Neyman–Pearson criterion, Willie makes the likelihood ratio test ${\displaystyle \frac{p_1\left(y_w\right)}{p_0\left(y_w\right)}}\underset{{\mathcal{D}}_0}{\overset{{\mathcal{D}}_1}{\gtrless }}1$ to minimize the detection error probability (DEP), where ${\mathcal{D}}_0$ and ${\mathcal{D}}_1$ denote the binary decisions endorsing ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$, respectively. In practice, Willie utilizes its average received power $P_w\stackrel{\overline{M}\to \infty }{=}{\displaystyle \frac{1}{\overline{M}}}{\sum }_{\overline{m}=1}^{\overline{M}}{\left|y_w\left(\overline{m}\right)\right|}^2$ as the decision statistic with $\overline{m}$ denoting the transmission block index. Then, the optimal decision rule for Willie can be rewritten as $P_w\underset{{\mathcal{D}}_0}{\overset{{\mathcal{D}}_1}{\gtrless }}{\varphi }^{*}$, where ${\varphi }^{*}={\displaystyle \frac{{\sigma }_0^2{\sigma }_1^2}{{\sigma }_1^2-{\sigma }_0^2}}\ln {\displaystyle \frac{{\sigma }_1^2}{{\sigma }_0^2}}$ is the optimal detection threshold for Willie’s decision, which results in the following minimum DEP [41]:

$$\begin{array}{ccc}\hfill {\xi }^{*}& =& Pr\left({\mathcal{D}}_1|{\mathcal{H}}_0\right)+Pr\left({\mathcal{D}}_0|{\mathcal{H}}_1\right)\hfill \\ & =& Pr\left(P_w\ge {\varphi }^{*}|{\mathcal{H}}_0\right)+Pr\left(P_w\le {\varphi }^{*}|{\mathcal{H}}_1\right)\hfill \\ & =& 1+{\left({\displaystyle \frac{{\sigma }_1^2}{{\sigma }_0^2}}\right)}^{-{\displaystyle \frac{{\sigma }_1^2}{{\sigma }_1^2-{\sigma }_0^2}}}-{\left({\displaystyle \frac{{\sigma }_1^2}{{\sigma }_0^2}}\right)}^{-{\displaystyle \frac{{\sigma }_0^2}{{\sigma }_1^2-{\sigma }_0^2}}},\hfill \end{array}$$

(7)

where $Pr\left({\mathcal{D}}_1|{\mathcal{H}}_0\right)$ and $Pr\left({\mathcal{D}}_0|{\mathcal{H}}_1\right)$ denote the false alarm and missed detection probabilities, respectively. To facilitate a feasible covert communication design, we introduce a lower bound on ${\xi }^{*}$ as [41]:

$$\begin{array}{c}\hfill {\xi }^{*}\ge 1-\sqrt{{\displaystyle \frac{1}{2}}\mathcal{D}\left(p_0\left(y_w\right)\Vert p_1\left(y_w\right)\right)},\end{array}$$

(8)

where $\mathcal{D}\left(p_0\left(y_w\right)\Vert p_1\left(y_w\right)\right)$ is the Kullback–Leibler (KL) divergence, which can be evaluated as:

$$\begin{array}{ccc}\hfill \mathcal{D}\left(p_0\left(y_w\right)\Vert p_1\left(y_w\right)\right)& =& ln\left({\displaystyle \frac{{\sigma }_1^2}{{\sigma }_0^2}}\right)+\left({\displaystyle \frac{{\sigma }_0^2}{{\sigma }_1^2}}\right)-1.\hfill \end{array}$$

(9)

To guarantee the transmission covertness from Alice to Bob, the minimum DEP of Willie should satisfy ${\xi }^{*}\ge 1-\epsilon$ with $\epsilon >0$ being a small value to represent the required covertness level. By using the KL divergence and the lower bound on ${\xi }^{*}$ in (8), a tighter covertness constraint can be formulated as $\mathcal{D}\left(p_0\left(y_w\right)\Vert p_1\left(y_w\right)\right)\le 2{\epsilon }^2$. Due to the fact that the monotonically increasing function $f\left(\lambda \right)=ln\lambda +{\displaystyle \frac{1}{\lambda }}-1$ with respect to $\lambda \in \left[1,\infty \right)$ in addition to $f\left(1\right)=0$ and $f(\infty )=\infty$, the covertness constraint can be reformulated as:

$$\begin{array}{ccc}\hfill & & P_a|g_{aw}+z_{aw}{|}^2+(1-\kappa )P_c|g_{cw}+z_{cw}{|}^2\hfill \\ \hfill & & +(1-\kappa ) \Vert {\mathbf{h}}_{rw}^H\mathbf{\Theta }{\Vert }^2{\sigma }_r^2\le (\kappa -1){\sigma }_w^2,\hfill \end{array}$$

(10)

where $\kappa$ is the unique solution of $f\left(\lambda \right)=2{\epsilon }^2$ in the range $\left[1,\infty \right)$.

2.2. w/o-EG Scheme

In this section, we investigate the impacts of the w/o-EG scheme on the covert communication performance of the ARIS-aided NOMA system. It is noted that for the w/o-EG scheme, the amplitude–phase-shift coefficient matrix can be simplified as $\mathbf{\Theta }={\mathbf{\Theta }}_r$ with ${\mathbf{M}}_r={\mathbf{I}}_N$ regarding the system model of the w-EG scheme. In the w/o-EG scheme, the ARIS selects only one reflection mode for all the elements to reflect the information signal. Similarly to the w-EG scheme, the w/o-EG scheme and the corresponding system model can be described as follows.

At Bob, the received signal can be expressed as:

$$\begin{array}{c}\hfill {\widehat{\mathbf{y}}}_b=\sqrt{P_a}{x}_a{\mathbf{g}}_{ab}+\sqrt{P_c}{x}_c{\mathbf{g}}_{cb}+{\mathbf{H}}_{rb}\mathbf{\Theta }{\mathbf{n}}_r+{\mathbf{n}}_b.\end{array}$$

(11)

Then, a beamformer $\mathbf{w}$ is applied at Bob to obtain ${\widehat{y}}_b={\mathbf{w}}^H{\widehat{\mathbf{y}}}_b$ for further processing. Similarly, we decode $x_a$ at the last stage to guarantee a successful SIC at Bob. The achievable rates of decoding of $x_a$ and $x_c$ are given by ${\widehat{R}}_a={log}_2(1+{\widehat{\gamma }}_a)$ and ${\widehat{R}}_c={log}_2(1+{\widehat{\gamma }}_c)$, respectively, where ${\widehat{\gamma }}_a$ and ${\widehat{\gamma }}_c$ are the received SINRs given by ${\widehat{\gamma }}_a=\frac{P_a{|{\mathbf{w}}^H{\mathbf{g}}_{ab}|}^2}{\Vert {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\Vert }^2{\sigma }_r^2+{\sigma }_b^2}$ and ${\widehat{\gamma }}_c=\frac{P_c{|{\mathbf{w}}^H{\mathbf{g}}_{cb}|}^2}{P_a{|{\mathbf{w}}^H{\mathbf{g}}_{ab}|}^2+ \Vert {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\Vert }^2{\sigma }_r^2+{\sigma }_b^2}$, respectively.

In the w/o-EG scheme, the received signals at Willie under ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$, respectively, can be expressed as:

$$\begin{array}{c}\hfill {\mathcal{H}}_0:{\widehat{y}}_w=\sqrt{P_c}{x}_c{g}_{cw}+{\mathbf{h}}_{rw}^H\mathbf{\Theta }{\mathbf{n}}_r+n_w,\end{array}$$

(12)

$$\begin{array}{c}\hfill {\mathcal{H}}_1:{\widehat{y}}_w=\sqrt{P_a}{g}_{aw}+\sqrt{P_c}{g}_{cw}+{\mathbf{h}}_{rw}^H\mathbf{\Theta }{\mathbf{n}}_r+n_w.\end{array}$$

(13)

The likelihood functions of the received signals under ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$ can be written as $p_0\left(y_w\right)$ and $p_1\left(y_w\right)$, respectively, with ${\sigma }_0^2=P_c|g_{cw}{|}^2+{\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }\parallel }^2{\sigma }_r^2+{\sigma }_w^2$ and ${\sigma }_1^2=P_a|g_{aw}{|}^2+P_c|g_{cw}{|}^2+{\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }\parallel }^2{\sigma }_r^2+{\sigma }_w^2$. Similar to the derivation of (10), the covertness constraint under this setup can be formulated as:

$$\begin{array}{c}\hfill P_a|g_{aw}{|}^2+(1-\kappa )(P_c|g_{cw}{|}^2+\Vert {\mathbf{h}}_{rw}^H\mathbf{\Theta }{\Vert }^2{\sigma }_r^2)\le (\kappa -1){\sigma }_w^2.\end{array}$$

(14)

3. Problem Formulation and Proposed Solutions

3.1. Problem Formulation

The aim of the covert communication design is to maximize the covert communication rate $R_a$ and ${\widehat{R}}_a$ for the w-EG and w/o-EG schemes, respectively, by jointly optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming subject to the constraint on the covertness level, quality-of-service (QoS) requirements of Carol, power budget at the ARIS, and SIC processing constraint at Bob.

For the w-EG scheme, we formulate the covert communication rate $R_a$’s maximization problem as follows:

$$\begin{array}{cc}\hfill \left(\mathrm{P}1\right):& \underset{P_a,P_c,\mathbf{w},\mathbf{\Theta }}{max}{R}_a\hfill \end{array}$$

(15a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& P_i\le P_i^{max},\forall i\in \{a,c\},\hfill \end{array}$$

(15b)

$$\begin{array}{cc}\hfill & \parallel {\mathbf{g}}_{cb}{\parallel }^2>\parallel {\mathbf{g}}_{ab}{\parallel }^2,\hfill \end{array}$$

(15c)

$$\begin{array}{cc}\hfill & R_c\ge R_c^{min},\hfill \end{array}$$

(15d)

$$\begin{array}{cc}\hfill & {\overline{\alpha }}_n+{\tilde{\alpha }}_n=1,\forall n\in \mathcal{N},\hfill \end{array}$$

(15e)

$$\begin{array}{cc}\hfill & {\overline{\alpha }}_n,{\tilde{\alpha }}_n\in \{0,1\},\forall n\in \mathcal{N},\hfill \end{array}$$

(15f)

$$\begin{array}{cc}\hfill & {\overline{\beta }}_n,{\tilde{\beta }}_n\in [0,{\eta }_n],\forall n\in \mathcal{N},\hfill \end{array}$$

(15g)

$$\begin{array}{cc}\hfill & {\parallel \mathbf{w}\parallel }^2=1,\hfill \end{array}$$

(15h)

$$\begin{array}{cc}\hfill & \left(3\right)\mathrm{and}\left(10\right).\hfill \end{array}$$

(15i)

In problem (P1), constraint (15b) limits the maximum transmit power of node i, constraint (15c) guarantees a successful SIC at Bob, constraint (15d) ensures the minimum rate $R_c^{min}$ of Carol, constraint (15e) and (15f) make each element work in at least one mode, i.e., information signal reflection or jamming signal reflection, constraint (15g) sets the maximum signal amplification for each element, (15h) sets the unit power constraint on the receive beamformer, and constraint (15i) ensures the maximum reflection power of the ARIS and the required covertness level, respectively.

For the w/o-EG scheme, the covert communication rate ${\widehat{R}}_a$’s maximization problem is formulated as follows:

$$\begin{array}{cc}\hfill \left(\mathrm{P}2\right):& \underset{P_a,P_c,\mathbf{w},\mathbf{\Theta }}{max}{\widehat{R}}_a\hfill \end{array}$$

(16a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& P_i\le P_i^{max},\forall i\in \{a,c\},\hfill \end{array}$$

(16b)

$$\begin{array}{cc}\hfill & \parallel {\mathbf{g}}_{cb}{\parallel }^2>\parallel {\mathbf{g}}_{ab}{\parallel }^2,\hfill \end{array}$$

(16c)

$$\begin{array}{cc}\hfill & {\widehat{R}}_c\ge {\widehat{R}}_c^{min},\hfill \end{array}$$

(16d)

$$\begin{array}{cc}\hfill & {\overline{\beta }}_n\in [0,{\eta }_n],\forall n\in \mathcal{N},\hfill \end{array}$$

(16e)

$$\begin{array}{cc}\hfill & {\parallel \mathbf{w}\parallel }^2=1,\hfill \end{array}$$

(16f)

$$\begin{array}{cc}\hfill & \left(3\right)\mathrm{and}\left(14\right).\hfill \end{array}$$

(16g)

In problems (P1) and (P2), the optimization objectives (15a) and (16a) are non-concave, and the majority of the constraints are non-convex, so both problems (P1) and (P2) should be transformed into concave ones to be solved.

3.2. Proposed Solution to Problem (P1)

Due to the non-concave objective function and the highly coupled system parameters in the constraints, problem (P1) is intractable. Moreover, constraint (15f) formulates a mixed-integer non-linear programming problem. In the following, we first decouple the original problem into three sub-problems of optimizing the NOMA transmit power at Alice and Carol, ARIS beamforming, and receive beamforming, respectively. Then, the integer constraint and the rank-one constraint in the decoupled sub-problems are tackled, respectively, and a penalty-based AO algorithm is proposed to obtain the optimized solution for problem (P1).

With any given $\mathbf{\Theta }$ and $\mathbf{w}$, problem (P1) reduces to optimizing the NOMA transmit powers $P_a$ and $P_c$, which can be formulated as:

$$\begin{array}{ccc}\hfill \left(\mathrm{P}3\right):& \underset{P_a,P_c}{max}{\gamma }_a\hfill & \end{array}$$

(17a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\gamma }_c\ge {\gamma }_{\mathrm{th}},\hfill \end{array}$$

(17b)

$$\begin{array}{cc}\hfill & \left(3\right),\left(10\right)\mathrm{and}\left(15\mathrm{b}\right).\hfill \end{array}$$

(17c)

In problem (P3), we replace the objective function $R_a$ with ${\gamma }_a$, equivalently. Moreover, constraint (17b) represents the QoS requirements of Carol with ${\gamma }_{\mathrm{th}}=2^{R_c^{min}}-1$. Since the objective function ${\gamma }_a$ monotonically increases with $P_a$, the equality in constraint (15b) holds when the objective function achieves its maximum value, which results in $P_a\le P_a^{max}$. Furthermore, constraints (3), (10) and (17b) can be rewritten as $P_a\le {\Xi }_1$, $P_a\le {\Xi }_2$, and $P_a\le {\Xi }_3$, respectively.

$$\begin{array}{c}\hfill {\Xi }_1=\frac{P_r^{max}-P_c{\parallel \mathbf{\Theta }{\mathbf{h}}_{cr}\parallel }^2- {\parallel \mathbf{\Theta }\parallel }_F^2{\sigma }_r^2}{{\parallel \mathbf{\Theta }{\mathbf{h}}_{ar}\parallel }^2},\end{array}$$

(18)

$$\begin{array}{c}\hfill {\Xi }_2=\frac{(\kappa -1)({\sigma }_w^2+P_c|g_{cw}+z_{cw}{|}^2+ {\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }\parallel }^2{\sigma }_r^2)}{|g_{aw}+z_{aw}{|}^2},\end{array}$$

(19)

$$\begin{array}{c}\hfill {\Xi }_3=\frac{P_c|{\mathbf{w}}^H{\mathbf{g}}_{cb}{|}^2-{\gamma }_{\mathrm{th}}\left(P_c|{\mathbf{w}}^H{\mathbf{z}}_{cb}{|}^2+ \Vert {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\Vert }^2{\sigma }_r^2+{\sigma }_b^2\right)}{{\gamma }_{\mathrm{th}}|{\mathbf{w}}^H({\mathbf{g}}_{ab}+{\mathbf{z}}_{ab}){|}^2}.\end{array}$$

(20)

Thus, the optimal solution of the NOMA transmit power at Alice and Carol can be derived as:

$$\begin{array}{c}\hfill P_a^{\ast }=min\left\{P_a^{max},{\Xi }_1,{\Xi }_2,{\Xi }_3\right\}\mathrm{and}{P}_c^{\ast }=P_c^{max}.\end{array}$$

(21)

With any given $P_a$, $P_c$, and $\mathbf{w}$, the sub-problem of optimizing $\mathbf{\Theta }$ is formulated as:

$$\begin{array}{ccc}\hfill \left(\mathrm{P}4\right):& \underset{\mathbf{\Theta }}{max}{\gamma }_a\hfill & \end{array}$$

(22a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \left(15\mathrm{c}\right),\left(15\mathrm{e}\right),\left(15\mathrm{f}\right),\left(15\mathrm{g}\right),\left(15\mathrm{i}\right)\mathrm{and}\left(17\mathrm{b}\right).\hfill \end{array}$$

(22b)

It can be seen that the reflecting and jamming matrices are always in the form of a product, which motivates us to define ${\mathit{\theta }}_r=\left[\mathrm{diag}\left({\mathbf{\Theta }}_r\right);1\right]$ and ${\mathbf{A}}_r={\mathit{\theta }}_r{\mathit{\theta }}_r^H$. Meanwhile, we have ${\mathbf{A}}_r⪰0$, $\mathrm{rank}\left({\mathbf{A}}_r\right)=1$, and $\mathrm{diag}\left({\mathbf{A}}_r\right)={\mathbf{a}}_r$, where ${\mathbf{a}}_r=\mathrm{diag}\left({\mathbf{A}}_r\right)={[{\gamma }_1^r,\cdots ,{\gamma }_N^r,1]}^T$ with ${\gamma }_n^r={\left({\overline{\alpha }}_n{\overline{\beta }}_n\right)}^2$, $\forall n\in \mathcal{N}$. Similarly, we define ${\mathit{\theta }}_j=\left[\mathrm{diag}\left({\mathbf{\Theta }}_j\right);0\right]$ and ${\mathbf{A}}_j={\mathit{\theta }}_j{\mathit{\theta }}_j^H$. Meanwhile, we have ${\mathbf{A}}_j⪰0$, $\mathrm{rank}\left({\mathbf{A}}_j\right)=1$, and $\mathrm{diag}\left({\mathbf{A}}_j\right)={\mathbf{a}}_j$, where ${\mathbf{a}}_j=\mathrm{diag}\left({\mathbf{A}}_j\right)={[{\gamma }_1^j,\cdots ,{\gamma }_N^j,0]}^T$ with ${\gamma }_n^j={\left({\tilde{\alpha }}_n{\tilde{\beta }}_n\right)}^2$, $\forall n\in \mathcal{N}$. Furthermore, we introduce the variables ${\mathbf{G}}_{ib}=\left[\begin{array}{cc}{\overline{\mathbf{g}}}_{ib}^{\ast }{\overline{\mathbf{g}}}_{ib}^T& v_i{\overline{\mathbf{g}}}_{ib}^{\ast }\\ v_i{\overline{\mathbf{g}}}_{ib}^T& 0\end{array}\right]$, ${\mathbf{G}}_{iw}={\overline{\mathbf{g}}}_{iw}^{\ast }{\overline{\mathbf{g}}}_{iw}^T$, ${\mathbf{Z}}_{ib}=\left[\begin{array}{cc}{\overline{\mathbf{z}}}_{ib}^{\ast }{\overline{\mathbf{z}}}_{ib}^T& {\overline{\mathbf{z}}}_{ib}^{\ast }\\ {\overline{\mathbf{z}}}_{ib}^T& 0\end{array}\right]$, and ${\mathbf{Z}}_{iw}=\left[\begin{array}{cc}{\overline{\mathbf{z}}}_{iw}^{\ast }{\overline{\mathbf{z}}}_{iw}^T& {\overline{\mathbf{z}}}_{iw}^{\ast }\\ {\overline{\mathbf{z}}}_{iw}^T& 0\end{array}\right]$ with ${\overline{\mathbf{g}}}_{ib}=\mathrm{diag}\left({\mathbf{w}}^H{\mathbf{H}}_{rb}\right){\mathbf{h}}_{ir}$, $v_i={\mathbf{w}}^H{\mathbf{h}}_{ib}$, ${\overline{\mathbf{g}}}_{iw}=\left[\mathrm{diag}\left({\mathbf{h}}_{rw}^H\right){\mathbf{h}}_{ir};h_{iw}\right]$, ${\overline{\mathbf{z}}}_{ib}=\mathrm{diag}\left({\mathbf{w}}^H{\mathbf{H}}_{rb}\right){\mathbf{h}}_{ir}$, and ${\overline{\mathbf{z}}}_{iw}=\mathrm{diag}\left({\mathbf{h}}_{rw}^H\right){\mathbf{h}}_{ir}$ for $i\in \{a,c\}$. Then, we can express the composite channel gains in the form of $|{\mathbf{w}}^H{\mathbf{g}}_{ib}{|}^2=\mathrm{Tr}\left({\mathbf{G}}_{ib}{\mathbf{A}}_r\right)+|v_i{|}^2$, $|{\mathbf{w}}^H{\mathbf{z}}_{ib}{|}^2=\mathrm{Tr}\left({\mathbf{Z}}_{ib}{\mathbf{A}}_j\right)$, $|g_{iw}{|}^2=\mathrm{Tr}\left({\mathbf{G}}_{iw}{\mathbf{A}}_r\right)$, and $|z_{iw}{|}^2=\mathrm{Tr}\left({\mathbf{Z}}_{iw}{\mathbf{A}}_j\right)$ with $i\in \{a,c\}$. By introducing $\mathbf{S}=\mathrm{diag}\left({\left[{|{\left[\mathbf{s}\right]}_1|}^2,{|{\left[\mathbf{s}\right]}_2|}^2,\cdots ,{|{\left[\mathbf{s}\right]}_N|}^2,0\right]}^T\right)$ with $\mathbf{s}={\mathbf{w}}^H{\mathbf{H}}_{rb}$ and ${\mathbf{G}}_{kl}=\mathrm{diag}\left({\left[{|{\left[{\mathbf{h}}_{kl}\right]}_1|}^2,{|{\left[{\mathbf{h}}_{kl}\right]}_2|}^2,\cdots ,{|{\left[{\mathbf{h}}_{kl}\right]}_N|}^2,0\right]}^T\right)$ with $kl\in \left\{rw,ar,cr\right\}$, we have $\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }{\parallel }^2=\mathrm{Tr}\left({\mathbf{G}}_{rw}({\mathbf{A}}_r+{\mathbf{A}}_j\right))$, $\parallel \mathbf{\Theta }{\mathbf{h}}_{ar}{\parallel }^2=\mathrm{Tr}\left({\mathbf{G}}_{ar}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)$, $\parallel \mathbf{\Theta }{\mathbf{h}}_{cr}{\parallel }^2=\mathrm{Tr}\left({\mathbf{G}}_{cr}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)$, and $\parallel {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\parallel }^2=\mathrm{Tr}\left(\mathbf{S}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)$. Moreover, we have ${\parallel \mathbf{\Theta }\parallel }_F^2=\mathrm{Tr}(\mathbf{\Pi }({\mathbf{A}}_r+{\mathbf{A}}_j))$ with $\mathbf{\Pi }=\mathrm{diag}\left({\left[{\mathrm{I}}_{1\times N},0\right]}^T\right)$. Then, we can rewrite problem (P4) as problem (P4.1):

$$\begin{array}{ccc}\hfill \left(\mathrm{P}4.1\right):& \underset{{\mathbf{A}}_m}{max}{\displaystyle \frac{P_a\left(\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2\right)}{P_a\mathrm{Tr}\left({\mathbf{Z}}_{ab}{\mathbf{A}}_j\right)+{\sigma }_r^2\mathrm{Tr}\left(\mathbf{S}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)+{\sigma }_b^2}}\hfill & \end{array}$$

(23a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{Tr}\left({\mathbf{G}}_{cb}{\mathbf{A}}_r\right)+|v_c{|}^2>\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2,\hfill \\ \hfill & P_a\mathrm{Tr}\left({\mathbf{G}}_{ar}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)+P_c\mathrm{Tr}\left({\mathbf{G}}_{cr}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)\hfill \end{array}$$

(23b)

$$\begin{array}{cc}\hfill & +{\sigma }_r^2\mathrm{Tr}({\mathbf{A}}_r+{\mathbf{A}}_j)\le P_r^{max},\hfill \\ \hfill & P_a(\mathrm{Tr}\left({\mathbf{G}}_{aw}{\mathbf{A}}_r\right)+\mathrm{Tr}\left({\mathbf{Z}}_{aw}{\mathbf{A}}_j\right))+(1-\kappa )(P_c(\mathrm{Tr}\left({\mathbf{G}}_{cw}{\mathbf{A}}_r\right)\hfill \end{array}$$

(23c)

$$\begin{array}{cc}\hfill & +\mathrm{Tr}\left({\mathbf{Z}}_{cw}{\mathbf{A}}_j\right))+\mathrm{Tr}\left({\mathbf{G}}_{rw}({\mathbf{A}}_r+{\mathbf{A}}_j)\right){\sigma }_r^2)\le (\kappa -1){\sigma }_w^2,\hfill \\ \hfill & P_c(\mathrm{Tr}\left({\mathbf{G}}_{cb}{\mathbf{A}}_r\right)+|v_c{|}^2)\ge {\gamma }_{\mathrm{th}}(P_a(\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2\hfill \end{array}$$

(23d)

$$\begin{array}{cc}\hfill & +\mathrm{Tr}\left({\mathbf{Z}}_{ab}{\mathbf{A}}_j\right))+P_c\mathrm{Tr}\left({\mathbf{Z}}_{cb}{\mathbf{A}}_j\right)+\mathrm{Tr}\left(\mathbf{S}({\mathbf{A}}_r+{\mathbf{A}}_j)\right){\sigma }_r^2+{\sigma }_b^2),\hfill \end{array}$$

(23e)

$$\begin{array}{cc}\hfill & \mathrm{rank}\left({\mathbf{A}}_m\right)=1,{\mathbf{A}}_m⪰\mathbf{0},m\in \{r,j\},\hfill \end{array}$$

(23f)

$$\begin{array}{cc}\hfill & \left(15\mathrm{e}\right),\left(15\mathrm{f}\right)\mathrm{and}\left(15\mathrm{g}\right).\hfill \end{array}$$

(23g)

Nevertheless, problem (P4.1) is a concave–convex fractional programming problem in addition to the challenges of the rank-one constraint in (23f), the binary integer constraint in (15f), and the coupling between the amplification factors and mode selection coefficients. To tackle problem (P4.1), the Dinkelbach transform is applied to tackle the objective function and the following specific process is introduced to deal with the non-convex constraints.

With the definition of ${\gamma }_n^r$ and ${\gamma }_n^j$, constraint (15e) can be written as:

$$\begin{array}{c}\hfill {\gamma }_n^m\le {\eta }_n^2,\forall n\in \mathcal{N},m\in \{r,j\}.\end{array}$$

(24)

Since the value of ${\overline{\alpha }}_n$ (${\tilde{\alpha }}_n$) can only be 0 or 1, constraint (15f) is equivalent to

$$\begin{array}{c}\hfill {\gamma }_n^r+{\gamma }_n^j\le {\eta }_n^2,\forall n\in \mathcal{N}.\end{array}$$

(25)

To make the integrated variable ${\gamma }_n^m$ contain the mode selection function shown in (15f), it should satisfy ${\gamma }_n^r{\gamma }_n^j=0$, which is a non-convex form. As an alternation, we consider a more general case, i.e., ${\gamma }_n^r{\gamma }_n^j\ge 0$. To make the equation hold if and only if ${\gamma }_n^r$ and ${\gamma }_n^j$ equal to 0, we add the following constraint to ensure that each element works in at least one mode:

$$\begin{array}{c}\hfill {\gamma }_n^r+{\gamma }_n^j\ge 1,\forall n\in \mathcal{N}.\end{array}$$

(26)

At the same time, it is necessary to penalize the part of ${\gamma }_n^r{\gamma }_n^j>0$, and we obtain its upper bound by the AGM inequality, which is given by

$$\begin{array}{c}\hfill {\gamma }_n^r{\gamma }_n^j\le {\displaystyle \frac{1}{2}}{\left({\phi }_n{\gamma }_n^r\right)}^2+{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{{\gamma }_n^j}{{\phi }_n}}\right)}^2=\mathcal{A}\left({\gamma }_n^r,{\gamma }_n^j\right),\forall n\in \mathcal{N},\end{array}$$

(27)

where the equality holds if and only if ${\phi }_n=\sqrt{{\gamma }_n^j/\phantom{{\gamma }_n^j{\gamma }_n^r}{\gamma }_n^r}$. Then, a penalty term ${\sum }_{n=1}^N\mathcal{A}\left({\gamma }_n^r,{\gamma }_n^j\right)$ is introduced into the objective function to deal with the binary integer constraint.

Let $\parallel {\mathbf{A}}_m{\parallel }_{*}$ and $\parallel {\mathbf{A}}_m{\parallel }_2$ denote the nuclear norm and spectral norm of ${\mathbf{A}}_m$, respectively. To deal with the rank-one constraint, we add a penalty term ${\sum }_{m\in \{r,j\}}\mathcal{R}\left({\mathbf{A}}_m,{\mathbf{A}}_m^{\left(k\right)}\right)$ into the objective function, where ${\mathbf{A}}_m^{\left(k\right)}$ is the solution obtained in the kth iteration and $\mathcal{R}\left({\mathbf{A}}_m,{\mathbf{A}}_m^{\left(k\right)}\right)=\parallel {\mathbf{A}}_m{\parallel }_{*}+{\widehat{\mathbf{A}}}_m^{\left(k\right)}$ with ${\widehat{\mathbf{A}}}_m^{\left(k\right)}=-\parallel {\mathbf{A}}_m^{\left(k\right)}{\parallel }_2-\mathrm{Tr}\left({\mathbf{v}}_{max}^{\left(k\right)}{\left({\mathbf{v}}_{max}^{\left(k\right)}\right)}^H({\mathbf{A}}_m-{\mathbf{A}}_m^{\left(k\right)})\right)$ and ${\mathbf{v}}_{max}^{\left(k\right)}$ denoting the eigenvector corresponding to the largest eigenvalue of ${\mathbf{A}}_m^{\left(k\right)}$. Since $\mathrm{rank}\left({\mathbf{A}}_m\right)=1$ ensures $\parallel {\mathbf{A}}_m{\parallel }_{*}-\parallel {\mathbf{A}}_m{\parallel }_2=0$, the term $\parallel {\mathbf{A}}_m{\parallel }_{*}+{\widehat{\mathbf{A}}}_m^{\left(k\right)}$ forms an upper bound on $\parallel {\mathbf{A}}_m{\parallel }_{*}-\parallel {\mathbf{A}}_m{\parallel }_2$. Let $f_1\left({\mathbf{A}}_m\right)=P_a\left(\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2\right)$ and $f_2\left({\mathbf{A}}_m\right)=P_a\mathrm{Tr}\left({\mathbf{Z}}_{ab}{\mathbf{A}}_j\right)+{\sigma }_r^2\mathrm{Tr}\left(\mathbf{S}({\mathbf{A}}_r+{\mathbf{A}}_j)\right)+{\sigma }_b^2$; a new objective function after the Dinkelbach transform is formulated as:

$$\begin{array}{ccc}\hfill f\left({\mathbf{A}}_m\right)& =& f_1\left({\mathbf{A}}_m\right)-v_1{f}_2\left({\mathbf{A}}_m\right)-{\rho }_1\sum _{n=1}^N\mathcal{A}({\gamma }_n^r,{\gamma }_n^j)\hfill \\ & & -{\rho }_2\sum _{m\in \{r,j\}}\mathcal{R}\left({\mathbf{A}}_m,{\mathbf{A}}_m^{\left(k\right)}\right),\hfill \end{array}$$

(28)

where ${\rho }_1$ and ${\rho }_2$ are the penalty factors, each starting with a small value to ensure the feasibility of the initial point and gradually increasing to make the solution tightly satisfy the constraints. Then, the complete sub-problem after transformation is formulated as:

$$\begin{array}{ccc}\hfill \left(\mathrm{P}4.2\right):& \underset{{\mathbf{A}}_m,{\mathbf{a}}_m}{max}f\left({\mathbf{A}}_m\right)\hfill & \end{array}$$

(29a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathbf{A}}_m⪰\mathbf{0},m\in \{r,j\},\hfill \end{array}$$

(29b)

$$\begin{array}{ccc}\hfill & \mathrm{diag}\left({\mathbf{A}}_m\right)={\mathbf{a}}_m,m\in \{r,j\},\hfill & \end{array}$$

(29c)

$$\begin{array}{cc}\hfill & \left(23\mathrm{b}\right),\left(23\mathrm{c}\right),\left(23\mathrm{d}\right),\left(23\mathrm{e}\right),\left(24\right),\left(25\right)\mathrm{and}\left(26\right).\hfill \end{array}$$

(29d)

In problem (P4.2), the coefficient $v_1$ is updated by

$$\begin{array}{ccc}\hfill v_1^{(k+1)}& =& f_1{\left({\mathbf{A}}_m\right)}^{\left(k\right)}/f_2{\left({\mathbf{A}}_m\right)}^{\left(k\right)},\hfill \end{array}$$

(30)

where k is the iteration number with respect to the Dinkelbach transform coefficient $v_1$. By using a convex problem solver such as CVX, we can now obtain the optimal solution of problem (P4.2).

With any given $P_a$, $P_c$, and $\mathbf{\Theta }$, we need to optimize the receive beamforming only. By introducing $\mathbf{W}=\mathbf{w}{\mathbf{w}}^H$, ${\mathbf{\Omega }}_{ib}={\mathbf{g}}_{ib}{\mathbf{g}}_{ib}^H$, ${\mathbf{\Psi }}_{ib}={\mathbf{z}}_{ib}{{\mathbf{z}}_{ib}}^H$ with $i\in \{a,c\}$, and $\mathbf{F}=\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H$ with $\overline{\mathbf{F}}={\mathbf{H}}_{rb}\mathbf{\Theta }$, we can rewrite $|{\mathbf{w}}^H{\mathbf{g}}_{ib}{|}^2=\mathrm{Tr}\left({\mathbf{\Omega }}_{ib}\mathbf{W}\right)$, $|{\mathbf{w}}^H{\mathbf{z}}_{ib}{|}^2=\mathrm{Tr}\left({\mathbf{\Psi }}_{ib}\mathbf{W}\right)$, and $\parallel {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\parallel }^2=\mathrm{Tr}\left(\mathbf{F}\mathbf{W}\right)$. Now, the sub-problem of optimizing $\mathbf{w}$ can be formulated as:   

$$\begin{array}{ccc}\hfill \left(\mathrm{P}5\right):& \underset{\mathbf{w}}{max}{\displaystyle \frac{P_a\mathrm{Tr}\left({\mathbf{\Omega }}_{ab}\mathbf{W}\right)}{P_a\mathrm{Tr}\left({\mathbf{\Psi }}_{ab}\mathbf{W}\right)+{\sigma }_r^2\mathrm{Tr}\left(\mathbf{F}\mathbf{W}\right)+{\sigma }_b^2}}\hfill & \\ \hfill \mathrm{s}.\mathrm{t}.& P_c\mathrm{Tr}\left({\mathbf{\Omega }}_{cb}\mathbf{W}\right)\ge {\gamma }_{\mathrm{th}}(P_a\mathrm{Tr}\left(({\mathbf{\Omega }}_{ab}+{\mathbf{\Psi }}_{ab})\mathbf{W}\right)+\mathrm{Tr}\left(\mathbf{F}\mathbf{W}\right){\sigma }_r^2\hfill \end{array}$$

(31a)

$$\begin{array}{cc}\hfill & +P_c\mathrm{Tr}\left({\mathbf{\Psi }}_{cb}\mathbf{W}\right)+{\sigma }_b^2),\hfill \end{array}$$

(31b)

$$\begin{array}{cc}\hfill & \mathrm{rank}\left(\mathbf{W}\right)=1,\mathrm{Tr}\left(\mathbf{W}\right)=1,\mathbf{W}⪰\mathbf{0}.\hfill \end{array}$$

(31c)

To tackle the rank-one constraint in problem (P5), we add a penalty term ${\rho }_3\mathcal{R}\left(\mathbf{W},{\mathbf{W}}^{\left(k\right)}\right)={\rho }_3\left({\parallel \mathbf{W}\parallel }_{*}+{\widehat{\mathbf{W}}}^{\left(k\right)}\right)$ into the objective function, where ${\rho }_3$ is the penalty factor, k is the iteration number with respect to the penalty term, and ${\widehat{\mathbf{W}}}^{\left(k\right)}=-{\parallel {\mathbf{W}}^{\left(k\right)}\parallel }_2-\mathrm{Tr}\left({\tilde{\mathbf{v}}}_{max}^{\left(k\right)}{\left({\tilde{\mathbf{v}}}_{max}^{\left(k\right)}\right)}^H(\mathbf{W}-{\mathbf{W}}^{\left(k\right)})\right)$ with ${\mathbf{W}}^{\left(k\right)}$ denoting the solution obtained in the kth iteration and ${\tilde{\mathbf{v}}}_{max}^{\left(k\right)}$ denoting the eigenvector corresponding to the largest eigenvalue of ${\mathbf{W}}^{\left(k\right)}$. Let $f_3\left(\mathbf{W}\right)=\mathrm{Tr}\left({\mathbf{\Omega }}_{ab}\mathbf{W}\right)$ and $f_4\left(\mathbf{W}\right)=P_a\mathrm{Tr}\left({\mathbf{\Psi }}_{ab}\mathbf{W}\right)+{\sigma }_r^2\mathrm{Tr}\left(\mathbf{F}\mathbf{W}\right)+{\sigma }_b^2$. Then, with the introduced penalty term and applying the Dinkelbach transform, problem (P5) is reformulated as:

$$\begin{array}{cccc}\hfill \left(\mathrm{P}5.1\right):& \underset{\mathbf{W}}{max}{f}_3\left(\mathbf{W}\right)-v_2{f}_4\left(\mathbf{W}\right)-{\rho }_3\mathcal{R}\left(\mathbf{W},{\mathbf{W}}^{\left(k\right)}\right)\hfill & \hfill & \end{array}$$

(32a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{Tr}\left(\mathbf{W}\right)=1,\mathbf{W}⪰\mathbf{0},\mathrm{and}\left(31\mathrm{b}\right).\hfill \end{array}$$

(32b)

In problem (P5.1), the coefficient $v_2$ is updated by

$$\begin{array}{c}\hfill v_2^{(k+1)}=f_3{\left(\mathbf{W}\right)}^{\left(k\right)}/f_4{\left(\mathbf{W}\right)}^{\left(k\right)},\end{array}$$

(33)

where the Dinkelbach transform coefficient $v_2$ is updated by (33) in the kth iteration. Now, a convex problem solver can be applied to obtain the optimal solution of problem (P5.1).

Based on the optimized solutions for problems (P4.2) and (P5.1), the proposed AO algorithm to obtain the solution for problem (P1) is summarized in Algorithm 1. The complexity of solving problems (P4.2) and (P5.1) are $\mathcal{O}\left(I_a{I}_d{I}_p{\left(N+1\right)}^{3.5}\right)$ and $\mathcal{O}\left(I_a{I}_d{I}_p{\left(M\right)}^{3.5}\right)$, where $I_a$, $I_d$, and $I_p$ are the number of iterations of the AO procedure, iteration numbers with respect to the penalty term, and Dinkelbach coefficients, respectively.

Algorithm 1 AO algorithm for optimizing { $P_a$, $P_c$, ${\mathbf{A}}_m$, $\mathbf{W}$ }

1: Initialize $t\leftarrow 0$, $P_a^{\left(0\right)}$, $P_c^{\left(0\right)}$, ${\phi }^{\left(0\right)}$, ${\mathbf{A}}_m^{\left(0\right)}$, and ${\mathbf{W}}^{\left(0\right)}$  2: repeat  $t\leftarrow t+1$  3:   Update $P_a^{\left(t\right)}$ according to (21)  4:   Initialize ${\rho }_1>0$, ${\rho }_2>0$, $v_1^{\left(0\right)}>0$, $k\leftarrow 0$  5:   repeat  $k\leftarrow k+1$  6:     Update ${\mathbf{A}}_m^{\left(k\right)}$ by solving (P4.2)  7:     Update ${\phi }_n^{(k+1)}=\sqrt{{\left({\gamma }_n^j\right)}^{\left(k\right)}/{\left({\gamma }_n^r\right)}^{\left(k\right)}}$  8:     Update $v_1^{(k+1)}$ according to (30)  9:     Update ${\rho }_1\leftarrow c_1{\rho }_1$ and ${\rho }_2\leftarrow c_2{\rho }_2$ 10:   until ${\rho }_1{\displaystyle \sum _{n=1}^N}\mathcal{A}\left({\gamma }_n^r,{\gamma }_n^j\right)\le {\xi }_2$ and 11:     ${\rho }_2{\displaystyle \sum _{m\in \{r,j\}}}\mathcal{R}\left({\mathbf{A}}_m,{\mathbf{A}}_m^{\left(k\right)}\right)\le {\xi }_3$ 12:   ${{\mathbf{A}}_m}^{\left(t\right)}\leftarrow {{\mathbf{A}}_m}^{\left(k\right)}$ 13:   Initialize ${\rho }_3>0$, $v_2^{\left(0\right)}>0$, $k\leftarrow 0$ 14:   repeat  $k\leftarrow k+1$ 15:     Update ${\mathbf{W}}^{\left(k\right)}$ by solving (P5.1) 16:     Update $v_2^{(k+1)}$ according to (33) 17:     Update ${\rho }_3\leftarrow c_3{\rho }_3$ 18:   until  ${\rho }_3\mathcal{R}\left(\mathbf{W},{\mathbf{W}}^{\left(k\right)}\right)\le {\xi }_4$ 19:   ${\mathbf{W}}^{\left(t\right)}\leftarrow {\mathbf{W}}^{\left(k\right)}$ 20: until  ${\gamma }_a^{\left(t\right)}-{\gamma }_a^{(t-1)}\le {\xi }_1$

3.3. Proposed Solution to Problem (P2)

The main challenge for solving problem (P2) is the intractable non-concave objective function and non-convex constraints. To address these challenges, we decouple problem (P2) into three sub-problems of optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming, respectively. Similarly to Algorithm 1, we design an AO algorithm to obtain the optimal solution of problem (P2).

With any given $\mathbf{\Theta }$ and $\mathbf{w}$, problem (P2) reduces to optimizing the NOMA transmit power $P_a$ and $P_c$, which can be formulated as:

$$\begin{array}{ccc}\hfill \left(\mathrm{P}6\right):& \underset{P_a,P_c}{max}{\widehat{\gamma }}_a\hfill & \end{array}$$

(34a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\widehat{\gamma }}_c\ge {\widehat{\gamma }}_{\mathrm{th}},\hfill \end{array}$$

(34b)

$$\begin{array}{cc}\hfill & \left(3\right),\left(14\right)\mathrm{and}\left(16\mathrm{b}\right).\hfill \end{array}$$

(34c)

In problem (P6), we replace the objective function ${\widehat{R}}_a$ with ${\widehat{\gamma }}_a$, equivalently. Moreover, constraint (34b) represents the QoS requirements of Carol with ${\widehat{\gamma }}_{\mathrm{th}}=2^{{\widehat{R}}_c^{min}}-1$. Similar to the approach adopted in problem (P3), constraints (14) and (34b) can be rewritten as $P_a\le {\Xi }_4$ and $P_a\le {\Xi }_5$, respectively:

$$\begin{array}{c}\hfill {\Xi }_4=\frac{(\kappa -1)({\sigma }_w^2+P_c|g_{cw}{|}^2+{\parallel {\mathbf{h}}_{rw}^H\mathbf{\Theta }\parallel }^2{\sigma }_r^2)}{|g_{aw}{|}^2},\end{array}$$

(35)

$$\begin{array}{c}\hfill {\Xi }_5=\frac{P_c|{\mathbf{w}}^H{\mathbf{g}}_{cb}{|}^2-{\widehat{\gamma }}_{\mathrm{th}}\left(\Vert {\mathbf{w}}^H{\mathbf{H}}_{rb}\mathbf{\Theta }{\Vert }^2{\sigma }_r^2+{\sigma }_b^2\right)}{{\widehat{\gamma }}_{\mathrm{th}}|{\mathbf{w}}^H{\mathbf{g}}_{ab}{|}^2}.\end{array}$$

(36)

Thus, the optimal solution of the NOMA transmit power at Alice and Carol can be derived as:

$$\begin{array}{c}\hfill P_a^{*}=min\left\{P_a^{max},{\Xi }_1,{\Xi }_4,{\Xi }_5\right\}\mathrm{and}{P}_c^{*}=P_c^{max}.\end{array}$$

(37)

With any given $P_a$, $P_c$, and $\mathbf{w}$, the sub-problem of optimizing $\mathbf{\Theta }$ is formulated as:

$$\begin{array}{ccc}\hfill \left(\mathrm{P}7\right):& \underset{\mathbf{\Theta }}{max}{\widehat{\gamma }}_a\hfill & \end{array}$$

(38a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \left(16\mathrm{c}\right),\left(16\mathrm{e}\right),\left(16\mathrm{g}\right)\mathrm{and}\left(34\mathrm{b}\right).\hfill \end{array}$$

(38b)

Similar to the approach adopted in problem (P4.1), we can rewrite problem (P7) as problem (P7.1):

$$\begin{array}{ccc}\hfill \left(\mathrm{P}7.1\right):& \underset{{\mathbf{A}}_r}{max}{\displaystyle \frac{P_a\left(\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2\right)}{{\sigma }_r^2\mathrm{Tr}\left(\mathbf{S}{\mathbf{A}}_r\right)+{\sigma }_b^2}}\hfill & \end{array}$$

(39a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{Tr}\left({\mathbf{G}}_{cb}{\mathbf{A}}_r\right)+|v_c{|}^2>\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2,\hfill \end{array}$$

(39b)

$$\begin{array}{cc}\hfill & P_a\mathrm{Tr}\left({\mathbf{G}}_{ar}{\mathbf{A}}_r\right)+P_c\mathrm{Tr}\left({\mathbf{G}}_{cr}{\mathbf{A}}_r\right)+{\sigma }_r^2\mathrm{Tr}\left({\mathbf{A}}_r\right)\le P_r^{max},\hfill \\ \hfill & P_a\mathrm{Tr}\left({\mathbf{G}}_{aw}{\mathbf{A}}_r\right)+(1-\kappa )(P_c\mathrm{Tr}\left({\mathbf{G}}_{cw}{\mathbf{A}}_r\right)\hfill \end{array}$$

(39c)

$$\begin{array}{cc}\hfill & +\mathrm{Tr}\left({\mathbf{G}}_{rw}{\mathbf{A}}_r\right){\sigma }_r^2)\le (\kappa -1){\sigma }_w^2,\hfill \\ \hfill & P_c(\mathrm{Tr}\left({\mathbf{G}}_{cb}{\mathbf{A}}_r\right)+|v_c{|}^2)\ge {\widehat{\gamma }}_{\mathrm{th}}(P_a(\mathrm{Tr}\left({\mathbf{G}}_{ab}{\mathbf{A}}_r\right)+|v_a{|}^2)\hfill \end{array}$$

(39d)

$$\begin{array}{cc}\hfill & +\mathrm{Tr}\left(\mathbf{S}{\mathbf{A}}_r\right){\sigma }_r^2+{\sigma }_b^2),\hfill \end{array}$$

(39e)

$$\begin{array}{cc}\hfill & \mathrm{rank}\left({\mathbf{A}}_r\right)=1,{\mathbf{A}}_r⪰\mathbf{0},\hfill \end{array}$$

(39f)

$$\begin{array}{cc}\hfill & \left(16\mathrm{e}\right).\hfill \end{array}$$

(39g)

Nevertheless, problem (P7.1) is a concave–convex fractional programming problem in addition to the challenge of the rank-one constraint in (39f). Similar to the approach adopted in problem (P4.2), we transform problem (P7.1) into a concave one by applying the penalized Dinkelbach approach, which is formulated as:   

$$\begin{array}{ccc}\hfill \left(\mathrm{P}7.2\right):& \underset{{\mathbf{A}}_r,{\mathbf{a}}_r}{max}{f}_5\left({\mathbf{A}}_r\right)-v_3{f}_6\left({\mathbf{A}}_r\right)-{\rho }_4\mathcal{R}\left({\mathbf{A}}_r,{\mathbf{A}}_r^{\left(k\right)}\right)\hfill & \end{array}$$

(40a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathbf{A}}_r⪰\mathbf{0},\hfill \end{array}$$

(40b)

$$\begin{array}{ccc}\hfill & \mathrm{diag}\left({\mathbf{A}}_r\right)={\mathbf{a}}_r,\hfill & \end{array}$$

(40c)

$$\begin{array}{cc}\hfill & \left(39\mathrm{b}\right),\left(39\mathrm{c}\right),\left(39\mathrm{d}\right)\mathrm{and}\left(39\mathrm{e}\right).\hfill \end{array}$$

(40d)

In problem (P7.2), the coefficient $v_3$ is updated by

$$\begin{array}{ccc}\hfill v_3^{(k+1)}& =& f_5{\left({\mathbf{A}}_r\right)}^{\left(k\right)}/f_6{\left({\mathbf{A}}_r\right)}^{\left(k\right)},\hfill \end{array}$$

(41)

where the Dinkelbach transform coefficient $v_3$ is updated by (41) in the kth iteration.

With any given $P_a$, $P_c$, and $\mathbf{\Theta }$, we need to optimize the receive beamforming only. Now, the sub-problem of optimizing $\mathbf{w}$ can be formulated as:

$$\begin{array}{ccc}\hfill \left(\mathrm{P}8\right):& \underset{\mathbf{w}}{max}{\displaystyle \frac{P_a\mathrm{Tr}\left({\mathbf{\Omega }}_{ab}\mathbf{W}\right)}{{\sigma }_r^2\mathrm{Tr}\left(\mathbf{F}\mathbf{W}\right)+{\sigma }_b^2}}\hfill & \end{array}$$

(42a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& P_c\mathrm{Tr}\left({\mathbf{\Omega }}_{cb}\mathbf{W}\right)\ge {\gamma }_{\mathrm{th}}\left(P_a\mathrm{Tr}\left({\mathbf{\Omega }}_{ab}\mathbf{W}\right)+\mathrm{Tr}\left(\mathbf{F}\mathbf{W}\right){\sigma }_r^2+{\sigma }_b^2\right),\hfill \end{array}$$

(42b)

$$\begin{array}{cc}\hfill & \mathrm{rank}\left(\mathbf{W}\right)=1,\mathrm{Tr}\left(\mathbf{W}\right)=1,\mathbf{W}⪰\mathbf{0}.\hfill \end{array}$$

(42c)

To tackle the rank-one constraint in problem (P8), we apply the same penalized Dinkelbach approach as in problem (P5.1), which is formulated as:

$$\begin{array}{cccc}\hfill \left(\mathrm{P}8.1\right):& \underset{\mathbf{W}}{max}{f}_7\left(\mathbf{W}\right)-v_4{f}_8\left(\mathbf{W}\right)-{\rho }_5\mathcal{R}\left(\mathbf{W},{\mathbf{W}}^{\left(k\right)}\right)\hfill & \hfill & \end{array}$$

(43a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{Tr}\left(\mathbf{W}\right)=1,\mathbf{W}⪰\mathbf{0},\mathrm{and}\left(42\mathrm{b}\right).\hfill \end{array}$$

(43b)

In problem (P8.1), the coefficient $v_4$ is updated by

$$\begin{array}{c}\hfill v_4^{(k+1)}=f_7{\left(\mathbf{W}\right)}^{\left(k\right)}/f_8{\left(\mathbf{W}\right)}^{\left(k\right)},\end{array}$$

(44)

where the Dinkelbach transform coefficient $v_4$ is updated by (44) in the kth iteration.

Utilizing the optimized solutions for problems (P7.2) and (P8.1), Algorithm 2 summarizes the proposed AO algorithm to obtain the solution for problem (P2). In addition, the overall computational complexity of the algorithm for solving problems (P7.2) and (P8.1) can be characterized by $\mathcal{O}\left(I_a{I}_d{I}_p{\left(N+1\right)}^{3.5}\right)$ and $\mathcal{O}\left(I_a{I}_d{I}_p{\left(M\right)}^{3.5}\right)$.

Algorithm 2 AO algorithm for optimizing { $P_a$, $P_c$, ${\mathbf{A}}_r$, $\mathbf{W}$ }

1: Initialize $t\leftarrow 0$, $P_a^{\left(0\right)}$, $P_c^{\left(0\right)}$, ${\mathbf{A}}_r^{\left(0\right)}$, and ${\mathbf{W}}^{\left(0\right)}$  2: repeat  $t\leftarrow t+1$  3:   Update $P_a^{\left(t\right)}$ according to (37)  4:   Initialize ${\rho }_4>0$, $v_3^{\left(0\right)}>0$, $k\leftarrow 0$  5:   repeat  $k\leftarrow k+1$  6:     Update ${\mathbf{A}}_r^{\left(k\right)}$ by solving (P7.2)  7:     Update $v_3^{(k+1)}$ according to (41)  8:     Update ${\rho }_4\leftarrow c_4{\rho }_4$  9:    until  ${\rho }_4\mathcal{R}\left({\mathbf{A}}_r,{\mathbf{A}}_r^{\left(k\right)}\right)\le {\xi }_5$ 10:   ${{\mathbf{A}}_r}^{\left(t\right)}\leftarrow {{\mathbf{A}}_r}^{\left(k\right)}$ 11:   Initialize ${\rho }_5>0$, $v_4^{\left(0\right)}>0$, $k\leftarrow 0$ 12:   repeat  $k\leftarrow k+1$ 13:     Update ${\mathbf{W}}^{\left(k\right)}$ by solving (P8.1) 14:     Update $v_4^{(k+1)}$ according to (44) 15:     Update ${\rho }_5\leftarrow c_5{\rho }_5$ 16:   until  ${\rho }_5\mathcal{R}\left(\mathbf{W},{\mathbf{W}}^{\left(k\right)}\right)\le {\xi }_6$ 17:   ${\mathbf{W}}^{\left(t\right)}\leftarrow {\mathbf{W}}^{\left(k\right)}$ 18: until  ${\gamma }_a^{\left(t\right)}-{\gamma }_a^{(t-1)}\le {\xi }_1$

4. Simulation Results

In this section, we present simulation results to evaluate the covert communication performance of the proposed scheme. We assumed that the ARIS-aided uplink NOMA system operated in the mmWave band with a carrier frequency of 28 GHz and a bandwidth of 251.1886 MHz [42]. We set the simulation parameters as $M=4$, $N=16$ with the size of the ARIS panel being 2 cm × 2 cm [43], ${\sigma }_b^2={\sigma }_w^2={\sigma }_r^2=-90$ dBm, $\rho =0$, $R_c^{min}={\widehat{R}}_c^{min}=1$ bps/Hz, ${\eta }_n^2={\eta }^2$, $\forall n$, ${\xi }_1={10}^{-1}$, ${\xi }_2={\xi }_3={\xi }_4={10}^{-3}$, ${\rho }_1={\rho }_2={\rho }_3={10}^{-6}$, and $c_1=c_2=c_3=3.6$ [29,31]. The small-scale fading of all channels followed $\mathcal{CN}(0,1)$. The distance-related path loss of each channel was modeled as $\mathcal{L}={\mathcal{L}}_0{\left({\displaystyle \frac{d}{d_0}}\right)}^{-\chi }$, where ${\mathcal{L}}_0=-30$ dB, $d_0=1$ m, and d is the distance between two nodes. We considered the path-loss exponential factor $\chi =3.5$ for the direct links and $\chi =2.2$ for the channels associated with ARIS. The Rician factor for the links associated with the ARIS was set to 3 dB. The locations of Bob, ARIS, Alice, Carol, and Willie were set as $\left(0,15\right)$ m, $\left(50,45\right)$ m, $\left(80,30\right)$ m, $\left(70,15\right)$ m, and $\left(10,30\right)$ m, respectively. In the following discussions, we compare the impacts of element grouping on the covert communication performance of an ARIS-aided NOMA system against those without element grouping. Specifically, the impacts of the transmit power, the horizontal position of the ARIS, the covertness level, and the number of ARIS elements on the covert communication rate are investigated, respectively.

To gain insights on the implementation of the proposed schemes, we recorded the running time of the AO algorithms for the w/o-EG and w-EG schemes, as shown in

Table 1. Running time of the w/o-EG and w-EG schemes.

Running Time (s)	System Parameters	$\mathit{M}=4$, $\mathit{N}=16$	$\mathit{M}=8$, $\mathit{N}=16$	$\mathit{M}=8$, $\mathit{N}=32$
Scheme
w-EG		0.2821	0.2963	0.3212
w/o-EG		0.2806	0.2954	0.3176

. The simulation platform was a Lenovo Lecoo PC consisting of an Intel i5-12450H CPU and 16 G of memory. The operating system was Windows 11, and the software was Matlab R2023a. The results in Table 1 show that the AO algorithms of both the w/o-EG and w-EG schemes could be performed within 0.35 s, which was acceptable regarding the applied general PC platform. As expected, if a specific hardware/software platform was applied, e.g., FPGA and DSP, the real-time running speed of the proposed AO algorithms would be faster than that of a general PC.

The results in Figure 2 reveal the impacts of the transmit power budget on the covert communication rates for different schemes. From Figure 2, we can see that the covert communication rates achieved by all the schemes increased with the increasing $P_a^{max}$. The curves in Figure 2 confirmed that the proposed AO algorithm without employing element grouping achieved the highest covert communication rate in the whole transmit power regions. On the contrary, compared to the AO algorithm without employing element grouping, a decreased covert communication rate was achieved by the element grouping scheme. Furthermore, a higher ${\eta }^2$ rendered a higher covert communication rate for the proposed schemes, which indicated that the constraints (21) and (37) were inactive, respectively. The achieved covert communication rates of the randomized $\mathbf{\Theta }$ and $\{\mathbf{\Theta },\mathbf{W}\}$ tended to overlap with the increasing $P_a^{max}$, while the two considered schemes achieved a higher covert communication rate than the AO algorithm with element grouping in the low-transmit-power regime.

In Figure 3, the impacts of the ARIS’s location on the covert communication rate and Carol’s achievable rate were investigated, where we set $P_a^{max}=P_c^{max}=35$ dBm, $P_r^{max}=30$ dBm, $M=4$, $N=16$, and $ϵ=0.1$. Figure 3a shows the covert communication rate versus the ARIS’s horizontal position under different ARIS schemes, including the considered w-EG and w/o-EG schemes, where the vertical position of ARIS was fixed at 45 m. It shows that the covert communication rates achieved by the ARIS-aided NOMA system increased with the increasing horizontal position, until reaching about 45 m when they began to decrease, which indicated that when the ARIS was properly deployed close to Bob and the users, the highest covert communication rate could be achieved. Furthermore, Figure 3a reveals that the covert communication performance of the w/o-EG scheme outperformed that of the w-EG scheme in an ARIS-aided NOMA system. On the contrary, the covert communication rates achieved by the randomized $\mathbf{\Theta }$ and $\{\mathbf{\Theta },\mathbf{W}\}$ changed slightly with the increase in the horizontal position of the ARIS, whereas the achieved covert communication rates were far less than those achieved by the w-EG and w/o-EG schemes. Also, a higher ${\eta }^2$ rendered a higher covert rate for the proposed schemes. Furthermore, Figure 3b presents the results of Carol’s achievable rate versus the ARIS’s horizontal position for the considered ARIS schemes. The curves in Figure 3b show that the w/o-EG scheme obtained Carol’s highest achievable rate. Similarly to the covert communication rate, the location of the ARIS also affected Carol’s achievable rate. Carol’s highest achievable rate was obtained with a horizontal position of 45 m, which was the same position at which the maximum covert communication rate was obtained.

The covertness level $ϵ$ versus the covert communication rate and Carol’s achievable rate for different schemes were investigated in Figure 4, where we set $P_a^{max}=P_c^{max}=35$ dBm, $P_r^{max}=30$ dBm, $M=4$, and $N=16$. As clearly shown by the curves in Figure 4a, the covert communication rates achieved by all the considered schemes increased with the increasing $ϵ$. This result is consistent with the theoretical analysis, i.e., when $ϵ$ becomes larger, the covertness constraint becomes loose, and the allocatable transmit power for Alice and the reflection beamforming power towards the BS increase, which leads to a higher covert communication rate for the ARIS-aided NOMA system. The covert communication rate achieved in the ARIS-aided NOMA system increased dramatically at first from $ϵ=0.01$ to $ϵ=0.05$, and then increased smoothly with the continued increase in $ϵ$. Moreover, the AO algorithm without employing element grouping achieved a higher covert communication rate than the AO algorithm with element grouping, while the achieved covert communication rate for the randomized $\{\mathbf{\Theta },\mathbf{W}\}$ was lower than that of the randomized $\mathbf{\Theta }$. In Figure 4b, the curves of Carol’s achievable rate are plotted. The curves in Figure 4b show that Carol’s achievable rate increased with increasing $ϵ$. Also, when $ϵ$ increased from 0.01 to 0.05, the achievable rate had a significant increase. After $ϵ$ surpassed 0.05, Carol’s achievable rate increased slowly, having a trend similar to that of the covert communication rate in Figure 4a.

The curves in Figure 5 compare the covert communication rate and Carol’s achievable rate versus the number of ARIS elements N. In Figure 5, we set $P_a^{max}=P_c^{max}=35$ dBm, $P_r^{max}=30$ dBm, $M=4$, and $ϵ=0.1$. For the ARIS-aided NOMA system with the w/o-EG scheme, it is obvious that more reflecting elements of the ARIS provide a larger reflection beamforming gain and achieve a higher covert communication rate due to the exploitation of more DoFs to manipulate the propagation environments. Moreover, the covert communication rates achieved by the AO algorithm with element grouping increased with increasing N, until reaching about $N=12$, when they began to decrease. However, the covert communication rates achieved by the randomized $\mathbf{\Theta }$ and $\{\mathbf{\Theta },\mathbf{W}\}$ continued to decrease. Similarly to Figure 5a, the curves in Figure 5b show that the w/o-EG scheme achieved Carol’s highest achievable rate under the considered N range.

The trade-off between the covert communication rate performance of the w/o-EG and w-EG schemes was investigated in Figure 6a, where we set $P_a^{max}=P_c^{max}=35$ dBm, $P_r^{max}=30$ dBm, $M=4$, $N=16$, and $ϵ=0.1$. The results in Figure 6a show that the covert communication rate decreased with the increasing number of elements in the grouping. When the number of elements in the grouping was zero, which corresponded to the w/o-EG scheme, the maximum covert communication rate was achieved. If all the elements worked in the grouping, i.e., the w-EG scheme, the minimum covert communication rate was obtained. Furthermore, the trade-off between Carol’s achievable rate performance of the w/o-EG and w-EG schemes is plotted in Figure 6b. Similar to the results in Figure 6a, Carol’s maximum and minimum achievable rates were obtained by the w/o-EG and w-EG schemes, respectively. With the increasing number of elements in the grouping, the achievable rates obtained by all the considered schemes decreased monotonically.

5. Conclusions

In this paper, we proposed the w/o-EG and w-EG schemes to enhance the covert communication performance of an ARIS-aided uplink NOMA system. To reveal the impacts of element grouping on the covert communication performance of an ARIS-aided NOMA system, we formulated the covert communication rate maximization problem by jointly optimizing the NOMA transmit power, ARIS beamforming, and receive beamforming. To tackle the complicated non-concave objective function and highly coupled non-convex constraints, the original optimization problem was decoupled into the sub-problems of the NOMA transmit power allocation, ARIS beamforming, and receive beamforming, respectively. A closed-form expression for the optimal NOMA transmit power was derived. An AGM and penalized Dinkelbach transform based AO algorithm was proposed to tackle the mixed-integer non-linear programming for the element grouping-assisted covert communication rate maximization problem. The numerical results showed that while element grouping improved the covert communication performance, a decreased performance was observed compared to the scheme without employing element grouping. The simulation results also revealed that the number of elements in the grouping affected the trade-off in covert communication performance between the w/o-EG and w-EG schemes.