Simultaneously Transmitting and Reflecting Reconfigurable Intelligent Surfaces Empowered Cooperative Rate Splitting with User Relaying ^†

Abstract

In this work, we unveil the advantages of synergizing cooperative rate splitting (CRS) with user relaying and simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR RIS). Specifically, we propose a novel STAR RIS-assisted CRS transmission framework, featuring six unique transmission modes that leverage various combinations of the relaying protocols (including full duplex-FD and half duplex-HD) and the STAR RIS configuration protocols (including energy splitting-ES, mode switching-MS, and time splitting-TS). With the objective of maximizing the minimum user rate, we then propose a unified successive convex approximation (SCA)-based alternative optimization (AO) algorithm to jointly optimize the transmit active beamforming, common rate allocation, STAR RIS passive beamforming, as well as time allocation (for HD or TS protocols) subject to the transmit power constraint at the base station (BS) and the law of energy conservation at the STAR RIS. To alleviate the computational burden, we further propose a low-complexity algorithm that incorporates a closed-form passive beamforming design. Numerical results show that our proposed framework significantly enhances user fairness compared with conventional CRS schemes without STAR RIS or other STAR RIS-empowered multiple access schemes. Moreover, the proposed low-complexity algorithm dramatically reduces the computational complexity while achieving very close performance to the AO method.

1. Introduction

In sixth-generation (6G) and beyond mobile communications, an increasing number of devices connect to the wireless network, causing the following problems: (1) Multi-device connections introduce significant multi-user interference, severely degrading system performance. (2) Ensuring a favorable communication environment for each user becomes highly challenging due to the vast number of devices accessing the transmission network. To deal with these problems, a novel non-orthogonal interference management strategy in the physical (PHY) layer named rate-splitting multiple access (RSMA) comes into our sight. It emerges as a promising technique for enhancing spectral efficiency and user fairness [1]. RSMA follows the design principle where user messages are divided into common and private parts at the transmitter. The common parts are encoded into common streams and decoded by multiple users. Meanwhile, the private parts are independently encoded into private streams and decoded only by the corresponding users, following the decoding of the common streams and their subsequent removal through successive interference cancellation (SIC). This empowers RSMA to partially treat interference as noise and partially decode interference. It encompasses various multiple access (MA) schemes like space division MA (SDMA), which fully treats interference as noise, and non-orthogonal MA (NOMA), which fully decodes interference, treating them as special cases [2].

However, as the common streams of RSMA are required to be decoded by multiple users, the achievable rate will be limited by the user with the weakest channel strength. To overcome this limitation, a novel RSMA scheme named cooperative rate splitting (CRS) is proposed in [3,4]. This scheme empowers users with stronger channel conditions, acting as relaying users, to transmit the decoded common stream to users with weaker channel conditions. CRS thereby boosts the received signal strengths at the weaker users and improves the achievable common rate. It not only enhances spectral efficiency but also extends radio coverage, promotes user fairness, and reduces energy consumption [4,5]. There are two transmission phases in CRS, one is the direct transmission phase in which the base station (BS) transmits signals to all users, and the other is the cooperative transmission phase in which the relaying users transmit the decoded common streams to the destination users. According to whether these two phases are executed at the same time, there are two protocols in CRS: half duplex (HD) [4] and full duplex (FD) [6]. In the HD protocol, the direct and cooperative transmission phases are executed in different time slots, while in the FD protocol, these two transmission phases are executed at the same time. Both protocols demonstrate significant potential in realizing the aforementioned advantages of CRS.

In parallel, reconfigurable intelligent surface (RIS), gaining increasing attention recently, has been viewed as a promising technique in future wireless networks [7,8,9]. An RIS is a two-dimensional (2D) meta-surface containing numerous passive and low-cost reflecting elements that enable tuning the phase shift of the incident signal. By employing the RIS, the line-of-sight (LoS) channel between the BS and users can be reconfigured, effectively improving the spectral efficiency, and energy efficiency and extending the communication coverage. Nevertheless, conventional RIS can solely reflect signals, limiting its coverage to a ${180}^{\circ }$ area. This has motivated the emergence of a novel RIS that enables reflection and transmits the incident signal at the same time, which is called simultaneously transmitting and reflecting RIS (STAR RIS) [10] or intelligent omni-surface (IOS) [11]. According to different operating methods, STAR RIS has three protocols: energy splitting (ES), mode switching (MS), and time switching (TS). In the ES protocol, all elements can reflect and transmit the incident signal at the same time. In the MS protocol, each element either reflects or transmits the incident signal. In the TS protocol, the transmission time is split into two time slots. One is the reflection time slot, where all elements reflect the incident signal, and the other is the transmission time slot, where all elements transmit the incident signal. By adjusting the amplitude and phase shift of the reflected and transmitted signals, all three protocols of STAR RIS offer full spatial coverage and additional degrees of freedom (DoF) for the system.

1.1. Related Works

In view of the advantages of STAR RIS, many existing works have explored its integration with various MA schemes [12,13,14,15,16,17]. In [12], the authors investigated the coverage range of a two-user STAR RIS-aided orthogonal MA (OMA) or NOMA system. In [13], the beamforming optimization was studied in an uplink STAR RIS-aided NOMA system for outage probability minimization and secrecy capacity maximization. In [14], the sum rate maximization was investigated in a downlink STAR RIS-aided NOMA system, wherein the decoding order and transmit beamforming are jointly optimized. These studies have demonstrated that STAR RIS offers benefits such as extending communication coverage and enhancing the sum rate. Besides, the integration of STAR RIS and RSMA has recently been investigated in some existing works [15,16,17]. In [15], the primary focus was on maximizing the sum secrecy rate in a STAR RIS-aided simultaneous wireless information and power transfer (SWIPT) system using RSMA. In [16], the sum-rate maximization problem for a STAR RIS-aided uplink RSMA system is studied. In [17], efforts were directed toward minimizing the outage probability while considering the spatial correlation among the STAR RIS channels. The aforementioned STAR RIS-aided RSMA works [15,16,17] emphasize the advantages of STAR RIS in enabling full-space wireless networks and enhancing interference management, leading to improvements in both spectral and energy efficiency. However, so far, the integration of STAR RIS and CRS has not been investigated yet. And the performance of max-min fairness for STAR RIS-aided RSMA remains unexplored.

1.2. Contributions

Considering the aforementioned advantages of both CRS and STAR RIS, along with the identified research gaps in the current literature, there is a compelling incentive to integrate these two techniques. On one hand, the CRS transmission scheme provides robust and flexible interference management to enhance the performance of STAR RIS. On the other hand, STAR RIS reconfigures and improves wireless channels, thereby further enhancing the performance of CRS. Their integration can lead to a mutually beneficial solution. In this work, we delve into the STAR RIS-aided CRS system, and the main contributions of this paper are summarized as follows:

• We propose a novel downlink STAR RIS-aided CRS transmission framework, empowering a STAR RIS to assist both the direct and cooperative transmission phases of CRS. Within this framework, we investigate six different transmission modes including various combinations of CRS relaying protocols (HD and FD) and STAR RIS operating protocols (ES, MS, and TS).
• We formulate a new resource allocation problem with the objective of maximizing the minimum user rate. To solve this problem, the STAR RIS passive beamforming, the BS active beamforming, common rate allocation, and time slot allocation (in HD or TS protocols) are jointly optimized under the transmit power constraint at the BS and the energy conservation constraints at the STAR RIS. Due to the non-convexity of the formulated problem, we propose an alternative optimization (AO) algorithm to solve the problem. This approach involves decomposing the original problem into two subproblems: the STAR RIS passive beamforming optimization and transmit active beamforming optimization. Each subproblem is then solved using a successive convex approximation (SCA)-based method. Through iterative solving of the two subproblems, we attain a near-optimal solution until convergence.
• We further propose a low-complexity algorithm to solve the formulated problem. For the passive beamforming design, we derive a closed-form solution for the STAR RIS passive beamforming based on the gradient descent approach. To ensure the derived solution meets the STAR RIS constraints, we further use the symmetric unitary projection based on singular value decomposition (SVD) to project the solution into the feasible set of constraints. To transmit active beamforming, we use the zero-forcing (ZF) approach to fix the beamforming direction. Subsequently, we simply optimize the power allocation using SCA. Apart from perfect channel state information at the transmitter (CSIT), we also extend the two proposed algorithms to address the ergodic rates (ERs) optimization problems under imperfect CSIT.
• We evaluate the performance of the proposed STAR RIS-aided CRS framework and show the effectiveness of our proposed algorithms by numerical results. Our analysis reveals that the STAR RIS-aided CRS scheme outperforms other STAR RIS-aided MA schemes. We also offer insights into the preferred regions for the six proposed transmission protocols. Moreover, the results demonstrate that our proposed low-complexity algorithm achieves comparable performance while significantly reducing CPU time compared to the AO algorithm.

1.3. Organization

The subsequent sections of the paper are organized as follows. Section 2 delineates the system model and formulates the max-min fairness problem. Section 3 details the AO optimization framework for the problem and the proposed low-complexity algorithm. Section 4 presents numerical results. Section 5 concludes the paper.

2. System Model and Problem Formulation

As illustrated in Figure 1, we consider a STAR RIS-assisted multi-user multiple-input single-output (MISO) downlink transmission network, with CRS supported during the transmission. There is a BS equipped with L transmit antennas, a STAR RIS containing N elements indexed by the set $\mathcal{N}=\{1,\cdots ,N\}$, and K users indexed by the set $\mathcal{K}=\{1,2,\cdots ,K\}$. Each user is assumed to be low-mobility and equipped with a single transmit antenna and a receive antenna. Given the ability of the STAR RIS to transmit and reflect the incident signal simultaneously, we divide the transmission space into two distinct subspaces, namely, the reflection space and the transmission space. Users in the reflection space receive signals reflected by the STAR RIS and are indexed by the set ${\mathcal{K}}_r$. Conversely, users in the transmission space receive signals transmitted by the STAR RIS and are indexed by the set ${\mathcal{K}}_t$. The two user sets satisfy ${\mathcal{K}}_r\cup {\mathcal{K}}_t=\mathcal{K}$ and ${\mathcal{K}}_r\cap {\mathcal{K}}_t=\varnothing$. Both subspaces allow user relaying, which further divides the users in each subspace into two user groups: the relaying user group ${\mathcal{K}}_{i1}$ and the destination user group ${\mathcal{K}}_{i2},i\in \{r,t\}$. The user groups satisfy ${\mathcal{K}}_{r1}\cap {\mathcal{K}}_{r2}=\varnothing ,{\mathcal{K}}_{t1}\cap {\mathcal{K}}_{t2}=\varnothing$, ${\mathcal{K}}_{r1}\cup {\mathcal{K}}_{r2}={\mathcal{K}}_r$, and ${\mathcal{K}}_{t1}\cup {\mathcal{K}}_{t2}={\mathcal{K}}_t$. For simplicity, we denote the relaying user set and the destination user set in the full space as ${\mathcal{K}}_1={\mathcal{K}}_{t1}\cup {\mathcal{K}}_{r1}$ and ${\mathcal{K}}_2={\mathcal{K}}_{t2}\cup {\mathcal{K}}_{r2}$, respectively. Without loss of generality, we assume users in ${\mathcal{K}}_1$ have better channel conditions than users in ${\mathcal{K}}_2$.

In this work, we consider a block-fading channel model with T transmission blocks indexed by the set $\mathcal{T}=\{1,\cdots ,T\}$. The channels between the BS and STAR RIS, the BS and user-k, STAR RIS and user-k are, respectively, denoted by $\mathbf{E}\left(t\right)\in {\mathbb{C}}^{N\times L},{\mathbf{g}}_k\left(t\right)\in {\mathbb{C}}^{L\times 1}$, ${\mathbf{h}}_k\left(t\right)\in {\mathbb{C}}^{N\times 1}$, where t denotes the t-th block, $t\in \mathcal{T}$. Besides, the channel between user-i to user-j is denoted by $h_{i,j}$. As the pioneer study on STAR RIS-empowered CRS, we employ a simplified CSI model by assuming perfect knowledge of CSI at the BS for all communication links, i.e., perfect CSI at the transmitter (CSIT). The imperfect CSIT setting can be easily extended based on the existing works on RSMA with imperfect CSIT. It is important to highlight that our primary emphasis is on exploring the diverse performance achieved by various transmission modes of STAR RIS-empowered CRS.

2.1. Transmission Modes of STAR RIS-Empowered CRS

CRS involves two transmission phases: the direct transmission phase and the cooperative transmission phase. During the direct transmission phase, the BS sends signals to all users. In the cooperative transmission phase, users in ${\mathcal{K}}_1$ act as relaying users to transmit the decoded common stream to users in ${\mathcal{K}}_2$, utilizing either the HD or FD protocols. The FD protocol allows the simultaneous transmission and reception for relaying users. However, this approach introduces self-interference at the relaying users. In the HD protocol, each time block is divided into two consecutive parts, respectively, for the two transmission phases. Let $\lambda$$(0<\lambda \le 1)$ denote the fraction of time allocated to the direct transmission phase, and $(1-\lambda )$ is the fraction allocated to the cooperative transmission phase.

In this paper, both transmission phases of CRS are assisted by the STAR RIS. According to the principle of single-connected STAR RIS in [10], each element of STAR RIS can operate in the reflection mode and/or the transmission mode according to different operating protocols [10].

The reflection and transmission matrices are all diagonal matrices, denoted as ${\mathbf{\Theta }}_i=\mathrm{diag}\left(\sqrt{{\beta }_{1,i}}{e}^{j{\theta }_{1,i}},\cdots ,\sqrt{{\beta }_{N,i}}{e}^{j{\theta }_{N,i}}\right)$, $i\in \{r,t\}$. In the ES protocol, each element of STAR RIS simultaneously reflects and transmits the incident signal. Hence, ${\beta }_{n,r}$ and ${\beta }_{n,t}$ should satisfy ${\beta }_{n,r}+{\beta }_{n,t}=1$ and ${\beta }_{n,r},{\beta }_{n,t}\in [0,1]$. The MS protocol is a special case of ES where each element operates in either reflection or transmission mode, resulting in binary values for the amplitude coefficients, i.e., ${\beta }_{n,r},{\beta }_{n,t}\in \{0,1\}$. In the TS protocol, each element of STAR RIS switches between the transmission and reflection modes in each transmission block. Define $0\le {\lambda }_r\le 1$ as the percentage of time allocated to the reflection mode in each transmission block, and $(1-{\lambda }_r)$ as the percentage of time allocated to the transmission mode. The TS protocol focuses on designing ${\lambda }_r$ and the phase shifts, where ${\beta }_{n,r}=1,{\beta }_{n,t}=0$ during the ${\lambda }_r$ percentage of time and ${\beta }_{n,r}=0,{\beta }_{n,t}=1$ for the remaining $(1-{\lambda }_r)$.

In the proposed STAR RIS-assisted CRS transmission framework, we pair the aforementioned relaying protocols (HD and FD) with the three STAR RIS operating protocols (ES, MS, and TS), yielding six distinct transmission modes for STAR RIS-assisted CRS. These modes are summarized in Figure 2 and explained as follows:

• FD-ES (FE) mode: This refers to the use of the FD protocol for CRS and the ES protocol for the STAR RIS. As illustrated in Figure 2a, FE enables each element of the STAR RIS to operate in both transmission and reflection modes simultaneously in the non-orthogonal direct and cooperative transmission phases.
• HD-ES (HE) mode: This refers to the use of the HD protocol for CRS and the ES protocol for STAR RIS. As illustrated in Figure 2b, HE enables each element of the STAR RIS to operate in both transmission and reflection modes simultaneously in the orthogonal direct and cooperative transmission phases.
• FD-MS (FM) mode: This refers to the use of the FD protocol for CRS and the MS protocol for STAR RIS. As illustrated in Figure 2c, FM enables each element of the STAR RIS to operate in either transmission or reflection mode in the non-orthogonal direct and cooperative transmission phases.
• HD-MS (HM) mode: This refers to the use of the HD protocol for CRS and the MS protocol for the STAR RIS. As illustrated in Figure 2d, HM enables each element of the STAR RIS to operate in either transmission or reflection mode in the orthogonal direct and cooperative transmission phases.
• FD-TS (FT) mode: This refers to the use of the FD protocol for CRS and the TS protocol for the STAR RIS. As illustrated in Figure 2e, each transmission block is split into two time slots based on ${\lambda }_r$ ($0\le {\lambda }_r\le 1$). FT facilitates all elements of the STAR RIS to function in the reflection mode during the initial portion ${\lambda }_r$ of the time, while they operate in the transmission mode during the subsequent portion $1-{\lambda }_r$ of the time, across the non-orthogonal direct and cooperative transmission phases.
• HD-TS (HT) mode: This refers to the use of the HD protocol for CRS and the TS protocol for the STAR RIS. As illustrated in Figure 2f, each transmission block is split into two time slots based on ${\lambda }_r$: one for the reflection mode and the other for the transmission mode of STAR RIS. Then, each time slot is further split into two orthogonal parts for the direct and cooperative transmission phases of CRS, resulting in a total of four orthogonal transmission slots. Here, we introduce ${\lambda }_1$ ($0<{\lambda }_1\le {\lambda }_r$) and ${\lambda }_2$ ($0<{\lambda }_2\le 1-{\lambda }_r$) to represent the time allocation between the two transmission phases in the reflection and transmission modes, respectively.

Hereinafter, we use superscript j to represent the selected transmission mode, where $j\in \{\mathit{FE},\mathit{FM},\mathit{FT},\mathit{HE},\mathit{HM},\mathit{HT}\}$.

2.2. Transmit Signal Model

For all six transmission modes described in the previous subsection, the primary differences among them lie in the received signals and the approaches employed to process the received signals. The transmit signals in both the direct and cooperative transmission phases remain consistent, as specified in this subsection.

2.2.1. Direct Transmission Phase

We denote $W_k$ as the message intended for user-k. At the BS, according to the principle of one-layer RSMA [2,18], $W_k$ is split into a common sub-message $W_{c,k}$ and a private sub-message $W_{p,k}$. All common sub-messages $W_{c,1},\cdots ,W_{c,K}$ are collected and then encoded into a common stream $s_0$, which is required to be decoded by all users. The individual private sub-message $W_{p,k}$ is independently encoded into a private stream $s_k$, which is required to be decoded by user-k only. Denote $\mathbf{s}={[s_0,s_1,\dots ,s_K]}^T$ and $\mathbf{P}=[{\mathbf{p}}_0,{\mathbf{p}}_1,\dots ,{\mathbf{p}}_K]\in {\mathbb{C}}^{L\times (K+1)}$, respectively, as the stream vector and the transmit beamforming matrix, where ${\mathbf{p}}_k\in {\mathbb{C}}^{L\times 1}$. The transmit signal at the BS in the direct transmission phase is given by

$${\mathbf{x}}^{\left[1\right]}=\mathbf{P}\mathbf{s}=\sum _{k=0}^K{\mathbf{p}}_k{s}_k,$$

(1)

where the superscript $\left[1\right]$ denotes the direct transmission phase. Here, assuming $\mathbb{E}\left\{\mathbf{s}{\mathbf{s}}^H\right\}=\mathbf{I}$, the transmit power of the BS is limited by $\mathrm{tr}\left(\mathbf{P}{\mathbf{P}}^H\right)\le P_t$, where $P_t$ is the maximum transmit power at the BS.

2.2.2. Cooperative Transmission Phase

Users in ${\mathcal{K}}_1$ employ the decode-and-forward (DF) protocol to decode, re-encode, and subsequently transmit the decoded $s_0$ to users in ${\mathcal{K}}_2$. The transmit signal at the relaying user-k in the cooperative transmission phase is

$$x_k^{\left[2\right]}=\sqrt{P_k}{s}_0,$$

(2)

where the superscript $\left[2\right]$ denotes the cooperative transmission phase and $P_k$ is the relaying transmit power at user-k.

2.3. Received Signal Model for FD-Based Transmission Modes

In the FD-based transmission modes (FE, FM, and FT), both the direct and cooperative transmission phases share the same radio resources. Self-interference is inevitably among the relaying users in ${\mathcal{K}}_1$ as they simultaneously receive signals from the BS and transmit the common stream $s_0$ to users in ${\mathcal{K}}_2$, [19,20]. We denote the self-interference channel as $I_k\left(t\right)$, which follows the same distribution $I_k\left(t\right)\sim \mathcal{CN}(0,{\Omega }_I^2)$ for all $k\in {\mathcal{K}}_1$. The signal received by users in ${\mathcal{K}}_2$ from ${\mathcal{K}}_1$, denoted as ${\widehat{s}}_0\left(t\right)$, is a delayed version of $s_0$, where ${\widehat{s}}_0\left(t\right)=s_0(t-t_d)$. Here, $t_d$ represents the processing time for users in ${\mathcal{K}}_1$ to decode and forward $s_0$. To ensure that the relaying users transmit and receive ${\widehat{s}}_0\left(t\right)$ within the same block, we assume that $t_d$ is much smaller than one block period [20,21]. In this work, we focus on optimizing $\mathbf{P}$, ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$ in each transmission block. Hence, we omit t in the following. Next, we will specify the received signals and achievable rates for FE, FM, and FT, respectively.

2.3.1. FE/FM

When $j=\mathit{FE}$ or $j=\mathit{FM}$, the signal received at each relaying user-k in ${\mathcal{K}}_1$ is given by

$$\begin{array}{c}\hfill y_k^j=\underset{{\tilde{\mathbf{g}}}_{k,i}^H}{\left(\underbrace{{\mathbf{g}}_k^H+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i\mathbf{E}}\right)}{\mathbf{x}}^{\left[1\right]}+I_k\sqrt{P_k}{\widehat{s}}_0+n_k,i\in \{r,t\},\end{array}$$

(3)

where $n_k$ is the additive white Gaussian noise (AWGN) which follows $n_k\sim \mathcal{CN}(0,{\sigma }^2)$ and $I_k{\sqrt{P}}_k{\widehat{s}}_0$ is the self-interference suffered at user-k. Each user-k in ${\mathcal{K}}_1$ sequentially decodes the common stream $s_0$ and the private stream $s_k$ with the assistance of SIC. Let ${\tilde{\mathbf{g}}}_{k,i}$ denote the effective channel between the BS and user-k, the SINRs of decoding $s_0$ and $s_k$ at user-k in ${\mathcal{K}}_1$ are, respectively, given as

$$\begin{array}{cc}\hfill & {\gamma }_{c,k}^{\mathit{FD}}={\gamma }_{c,k}^{\mathit{FD},\left[1\right]}={\displaystyle \frac{{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_0\right|}^2}{{\sum }_{m\in \mathcal{K}}{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_m\right|}^2+{\left|I_k\right|}^2{P}_k+{\sigma }^2}},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & {\gamma }_k^{\mathit{FD}}={\displaystyle \frac{{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_k\right|}^2}{{\sum }_{m\in \mathcal{K},m\ne k}{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_m\right|}^2+{\left|I_k\right|}^2{P}_k+{\sigma }^2}}.\hfill \end{array}$$

(5)

Each user-k in ${\mathcal{K}}_2$ receives signals from both the BS and the relaying users in ${\mathcal{K}}_1$, which are given as

$$\begin{array}{c}\hfill y_k^j={\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{x}}^{\left[1\right]}+\sum _{m\in {\mathcal{K}}_1}\underset{{\tilde{h}}_{m,k,i}}{\left(\underbrace{h_{m,k}+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i^H{\mathbf{h}}_m}\right)}{x}_m^{\left[2\right]}+n_k,i\in \{r,t\}.\end{array}$$

(6)

Each user in ${\mathcal{K}}_2$ then employs the maximal ratio combining (MRC) approach to ensure the received signals are properly co-phased and merged [22]. Denote the effective channel between user-m and user-n as ${\tilde{h}}_{m,n,i}$, the SINRs of decoding the $s_0$ and $s_k$ at user-k in ${\mathcal{K}}_2$ are given as

$$\begin{array}{cc}\hfill & {\gamma }_{c,k}^{\mathit{FD}}=\underset{{\gamma }_{c,k}^{\mathit{FD},\left[1\right]}}{\underbrace{{\displaystyle \frac{{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_0\right|}^2}{{\sum }_{m\in \mathcal{K}}{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_m\right|}^2+{\sigma }^2}}}}+\underset{{\gamma }_{c,k}^{\mathit{FD},\left[2\right]}}{\underbrace{{\displaystyle \frac{{\sum }_{m\in {\mathcal{K}}_1}{\left|{\tilde{h}}_{m,k,i}\right|}^2{P}_m}{{\sigma }^2}}}},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & {\gamma }_k^{\mathit{FD}}={\displaystyle \frac{{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_k\right|}^2}{{\sum }_{m\in \mathcal{K},m\ne k}{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_m\right|}^2+{\sigma }^2}}.\hfill \end{array}$$

(8)

With the SINRs of $s_0$ and $s_k$ in (4), (5), (7) and (8), the achievable common and private rates are obtained as $R_{c,k}^j={log}_2(1+{\gamma }_{c,k}^{\mathit{FD}})$ and $R_k^j={log}_2(1+{\gamma }_k^{\mathit{FD}})$, $j\in \{\mathit{FE},\mathit{FM}\}$. To guarantee all users decode the common stream, the following condition should satisfy [2]:

$$R_c^j=min\left\{R_{c,k}^j\right|k\in \mathcal{K}\},$$

(9)

$R_c^j$ is shared by all users, we denote $C_k$ as the part of $R_c^j$ that is allocated to user-k, it satisfies ${\sum }_{k\in \mathcal{K}}{C}_k\le R_c^j$. Hence, the total achievable rate of user-k is

$$R_{tot,k}^j=R_k^j+C_k,\forall k\in \mathcal{K}.$$

(10)

Note that Equations (9) and (10) remain applicable for other transmission modes by setting j to $\mathit{FT}/\mathit{HE}/\mathit{HM}/\mathit{HT}$. In the following discussion of these transmission modes, the equations will not be repeated.

2.3.2. FT

In FT mode, there are two time slots: all elements of STAR RIS operate in the reflection mode during the first time slot (${\lambda }_r$) while all elements of STAR RIS operate in the transmission mode during the second time slot ($1-{\lambda }_r$). We denote the SINRs for decoding $s_0$ and $s_k$ in the reflection and transmission time slots, respectively, as ${\gamma }_{c,k}^{\mathit{FD},i}$ and ${\gamma }_k^{\mathit{FD},i}$, where $i\in \{r,t\}$. In the reflection time slot, users in ${\mathcal{K}}_r$ receive signals from the BS and the STAR RIS while users in ${\mathcal{K}}_t$ receive signals from the BS only. The corresponding SINRs are given as

$$\begin{array}{cc}\hfill {\gamma }_{c,k}^{\mathit{FD},r}& ={\gamma }_{c,k}^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_r},{\gamma }_k^{\mathit{FD},r}={\gamma }_k^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_r},k\in {\mathcal{K}}_r,\hfill \\ \hfill {\gamma }_{c,k}^{\mathit{FD},r}& ={\gamma }_{c,k}^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},{\gamma }_k^{\mathit{FD},r}={\gamma }_k^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},k\in {\mathcal{K}}_t,\hfill \end{array}$$

(11)

where ${\gamma }_{c,k}^{\mathit{FD}}$ and ${\gamma }_k^{\mathit{FD}}$ are defined in (4) and (5), respectively. In the transmission time slot, users in ${\mathcal{K}}_r$ receive signals from the BS only while users in ${\mathcal{K}}_t$ receive signals from both BS and STAR RIS. The corresponding SINRs are given as

$$\begin{array}{cc}\hfill {\gamma }_{c,k}^{\mathit{FD},t}& ={\gamma }_{c,k}^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},{\gamma }_k^{\mathit{FD},t}={\gamma }_k^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},k\in {\mathcal{K}}_r,\hfill \\ \hfill {\gamma }_{c,k}^{\mathit{FD},t}& ={\gamma }_{c,k}^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_t},{\gamma }_k^{\mathit{FD},t}={\gamma }_k^{\mathit{FD}}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_t},k\in {\mathcal{K}}_t.\hfill \end{array}$$

(12)

The achievable common and private rates for user-k are given as

$$\begin{array}{cc}\hfill R_{c,k}^{\mathit{FT}}& ={\lambda }_r{log}_2(1+{\gamma }_{c,k}^{\mathit{FD},r})+(1-{\lambda }_r){log}_2(1+{\gamma }_{c,k}^{\mathit{FD},t}),\hfill \\ \hfill R_k^{\mathit{FT}}& ={\lambda }_r{log}_2(1+{\gamma }_k^{\mathit{FD},r})+(1-{\lambda }_r){log}_2(1+{\gamma }_k^{\mathit{FD},t}).\hfill \end{array}$$

(13)

By substituting $R_{c,k}^{\mathit{FT}}$ and $R_k^{\mathit{FT}}$ into (9) and (10), we obtain the corresponding achievable rate of each user in the FT mode.

2.4. Received Signal Model for HD-Based Transmission Modes

In HD-based transmission modes, the direct and cooperative transmission phases are orthogonal in each transmission block.

In contrast to FD-based modes, there is no self-interference at relaying users, but the transmission rates for the direct and transmission phases are scaled by the pre-log factors $\lambda (0<\lambda \le 1)$ and $(1-\lambda )$, respectively. In the following, we specify the received signals and achievable rates for HE, HM, and HT.

2.4.1. HE/HM

In the direct transmission phase, the signal received at user-k is

$$\begin{array}{c}\hfill y_k^{\mathit{HD},\left[1\right]}={\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{x}}^{\left[1\right]}+n_k,\forall k\in \mathcal{K},j\in \{\mathit{HE},\mathit{HM}\}.\end{array}$$

(14)

The SINRs of decoding $s_0$ and $s_k$ in the direct transmission phase are given as

$$\begin{array}{cc}\hfill & {\gamma }_{c,k}^{\mathit{HD},\left[1\right]}={\displaystyle \frac{{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_0\right|}^2}{{\sum }_{m\in \mathcal{K}}{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_m\right|}^2+{\sigma }^2}},\hfill \end{array}$$

(15)

$$\begin{array}{c}\hfill {\gamma }_k^{\mathit{HD}}={\displaystyle \frac{{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_0\right|}^2}{{\sum }_{m\in \mathcal{K},m\ne k}{\left|{\tilde{\mathbf{g}}}_{k,i}^H{\mathbf{p}}_m\right|}^2+{\sigma }^2}}.\end{array}$$

(16)

In the cooperative transmission phase, users in ${\mathcal{K}}_1$ forward the decoded $s_0$ to users in ${\mathcal{K}}_2$. The signal received by user-k in ${\mathcal{K}}_2$ in the cooperative transmission phase is

$$\begin{array}{c}\hfill y_k^{\mathit{HD},\left[2\right]}=\sum _{m\in {\mathcal{K}}_1}{\tilde{h}}_{m,k,i}\sqrt{P_m}{s}_0+n_k,\forall k\in {\mathcal{K}}_2.\end{array}$$

(17)

The SINR of decoding $s_0$ at user-k in ${\mathcal{K}}_2$ in the cooperative transmission phase is given as

$$\begin{array}{c}\hfill {\gamma }_{c,k}^{\mathit{HD},\left[2\right]}={\displaystyle \frac{{\sum }_{m\in {\mathcal{K}}_1}{\left|{\tilde{h}}_{m,k,i}\right|}^2{P}_m}{{\sigma }^2}}.\end{array}$$

(18)

With the SINR in (15), (16) and (18), we obtain the achievable common and private rates in HE and HM modes ($j\in \mathit{HE}/\mathit{HM}$) as

$$\begin{array}{cc}\hfill R_{c,k}^j& =\lambda {log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right]}\right),\forall k\in {\mathcal{K}}_1\hfill \\ \hfill R_{c,k}^j& =\lambda {log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right]}\right)+(1-\lambda ){log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right]}\right),\forall k\in {\mathcal{K}}_2\hfill \end{array}$$

(19)

and

$$R_k^j=\lambda {log}_2\left(1+{\gamma }_k^{\mathit{HD}}\right).$$

(20)

By substituting (19) and (20) into (9) and (10), we obtain the corresponding achievable rate of each user in the HE and HM modes.

2.4.2. HT

In the HT mode, there are four time slots: two for direct transmission with STAR RIS reflection or transmission, and two for cooperative transmission with STAR RIS reflection or transmission. We use ${\gamma }_{c,k}^{\mathit{HD},\left[s\right],i}$ and ${\gamma }_k^{\mathit{HD},i}$, $i\in \{r,t\}$, $s\in \{1,2\}$ to denote the SINRs of decoding $s_0$ and $s_k$ in the direct or cooperative transmission phase. The superscript i indicates whether it is in the reflection or transmission time slot. Specifically, in the two reflection time slots, respectively, for cooperative and transmission phases, we have

$$\begin{array}{cc}\hfill {\gamma }_{c,k}^{\mathit{HD},\left[s\right],r}& ={\gamma }_{c,k}^{\mathit{HD},\left[s\right]}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_r},{\gamma }_k^{\mathit{HD},r}={\gamma }_k^{\mathit{HD}}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_r},k\in {\mathcal{K}}_r,\hfill \\ \hfill {\gamma }_{c,k}^{\mathit{HD},\left[s\right],r}& ={\gamma }_{c,k}^{\mathit{HD},\left[s\right]}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},{\gamma }_k^{\mathit{HD},r}={\gamma }_k^{\mathit{HD}}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},k\in {\mathcal{K}}_t,\hfill \end{array}$$

(21)

where ${\gamma }_{c,k}^{\mathit{HD},\left[s\right]},s\in \{1,2\}$ and ${\gamma }_k^{\mathit{HD}}$ are defined in (15), (18) and (16), respectively. In the two transmission time slots, respectively, for cooperative and transmission phases, we have

$$\begin{array}{cc}\hfill {\gamma }_{c,k}^{\mathit{HD},\left[s\right],t}& ={\gamma }_{c,k}^{\mathit{HD},\left[s\right]}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},{\gamma }_k^{\mathit{HD},t}={\gamma }_k^{\mathit{HD}}{\mid }_{{\mathbf{\Theta }}_i=\mathbf{0}},k\in {\mathcal{K}}_r,\hfill \\ \hfill {\gamma }_{c,k}^{\mathit{HD},\left[s\right],t}& ={\gamma }_{c,k}^{\mathit{HD},\left[s\right]}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_t},{\gamma }_k^{\mathit{HD},t}={\gamma }_k^{\mathit{HD}}{\mid }_{{\mathbf{\Theta }}_i={\mathbf{\Theta }}_t},k\in {\mathcal{K}}_t.\hfill \end{array}$$

(22)

Hence, the achievable common and private rates for user-k are given as

$$\begin{array}{cc}\hfill R_{c,k}^{\mathit{HT}}& ={\lambda }_1{log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right],r}\right)+{\lambda }_2{log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right],t}\right),\forall k\in {\mathcal{K}}_1,\hfill \\ \hfill R_{c,k}^{\mathit{HT}}& ={\lambda }_1{log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right],r}\right)+({\lambda }_r-{\lambda }_1){log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right],r}\right)\hfill \\ \hfill & + {\lambda }_2{log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right],t}\right)+(1-{\lambda }_r-{\lambda }_2){log}_2\left(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right],t}\right),\forall k\in {\mathcal{K}}_2.\hfill \end{array}$$

(23)

$$R_k^{\mathit{HT}}={\lambda }_1{log}_2\left(1+{\gamma }_k^{\mathit{HD},r}\right)+{\lambda }_2{log}_2\left(1+{\gamma }_k^{\mathit{HD},t}\right).$$

(24)

2.5. Problem Formulation

In this work, we aim to jointly optimize the active beamforming $\mathbf{P}$ at the BS, the common rate allocation $\mathbf{c}=[C_1,\cdots ,C_K]$, as well as the passive beamforming ${\mathbf{\Theta }}_r$, ${\mathbf{\Theta }}_t$ at the STAR RIS. For the HD-based transmission modes, the corresponding time allocation coefficients are also jointly optimized. Our objective is to maximize the minimum user rate (max-min rate) among all users subject to the transmit power constraint. The formulated problems for the six transmission modes of the proposed STAR RIS-aided CRS model are illustrated as follows:

2.5.1. FE

$j=\mathit{FE}$, the optimization problem is formulated as

$$\begin{array}{}\mathrm{(25a)}& \hfill {\mathcal{P}}_{\mathit{FE}}:& \underset{\mathbf{P},\mathbf{c},{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t}{max}\underset{k\in \mathcal{K}}{min}{R}_{tot,k}^{\mathit{FE}}\hfill \mathrm{(25b)}& \hfill \mathrm{s}.\mathrm{t}.& \sum _{k\in \mathcal{K}}{C}_k\le R_c^j,C_k\ge 0,\forall k\in \mathcal{K}\hfill \mathrm{(25c)}& \hfill & {\theta }_{n,i}\in [0,2\pi ),\forall n\in \mathcal{N},i\in \{r,t\},\hfill \mathrm{(25d)}& \hfill & \mathrm{tr}\left(\mathbf{P}{\mathbf{P}}^H\right)\le P_t\hfill \mathrm{(25e)}& \hfill & \hfill {\beta }_{n,r},{\beta }_{n,t}\in [0,1],{\beta }_{n,r}+{\beta }_{n,t}=1,\forall n\in \mathcal{N},\end{array}$$

where constraint (25b) guarantees the common stream and is successfully decoded by all users. Constraints (25c) and (25e) are the constraints for the phase shift and amplitude ranges for each element of the STAR RIS, respectively. (25d) limits the transmit power at the BS.

2.5.2. FM

$j=\mathit{FM}$, the formulated problem is given as

$$\begin{array}{}\mathrm{(26a)}& \hfill {\mathcal{P}}_{\mathit{FM}}:& \underset{\mathbf{P},\mathbf{c},{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t}{max}\underset{k\in \mathcal{K}}{min}{R}_{tot,k}^{\mathit{FM}}\hfill & \hfill \mathrm{s}.\mathrm{t}.& \left(\mathrm{25b}\right)\text{–}\left(\mathrm{25d}\right)\hfill \mathrm{(26b)}& \hfill & {\beta }_{n,r},{\beta }_{n,t}\in \{0,1\},{\beta }_{n,r}+{\beta }_{n,t}=1,\forall n\in \mathcal{N}.\hfill \end{array}$$

2.5.3. FT

$j=\mathit{FT}$, the time allocation variable ${\lambda }_r$ is jointly optimized with the beamforming matrices and the common rate allocation. The optimization problem is

$$\begin{array}{}\mathrm{(27a)}& \hfill {\mathcal{P}}_{\mathit{FT}}:& \underset{\mathbf{P},\mathbf{c},{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t,{\lambda }_r}{max}\underset{k\in \mathcal{K}}{min}{R}_{tot,k}^{\mathit{FT}}\hfill & \hfill \mathrm{s}.\mathrm{t}.& \left(\mathrm{25b}\right)\text{–}\left(\mathrm{25d}\right),\hfill \mathrm{(27b)}& \hfill & {\beta }_{n,r},{\beta }_{n,t}\in [0,1],\forall n\in \mathcal{N},\hfill \mathrm{(27c)}& \hfill & 0\le {\lambda }_r\le 1.\hfill \end{array}$$

2.5.4. HE

$j=\mathit{HE}$, the time allocation variable $\lambda$ is jointly optimized with the beamforming matrices and the common rate allocation. The formulated problem is

$$\begin{array}{}\mathrm{(28a)}& \hfill {\mathcal{P}}_{\mathit{HE}}:& \underset{\mathbf{P},\mathbf{c},{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t,\lambda }{max}\underset{k\in \mathcal{K}}{min}{R}_{tot,k}^{\mathit{HE}}\hfill & \hfill \mathrm{s}.\mathrm{t}.& \left(\mathrm{25b}\right)\text{–}\left(25e\right)\hfill \mathrm{(28b)}& \hfill & 0<\lambda \le 1.\hfill \end{array}$$

2.5.5. HM

$j=\mathit{HM}$, the corresponding optimization problem is formulated as

$$\begin{array}{cc}\hfill {\mathcal{P}}_{\mathit{HM}}:& \underset{\mathbf{P},\mathbf{c},{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t,\lambda }{max}\underset{k\in \mathcal{K}}{min}{R}_{tot,k}^{\mathit{HM}}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(\mathrm{25b}\right)\text{–}\left(\mathrm{25d}\right),\left(\mathrm{26b}\right),\left(\mathrm{28b}\right).\hfill \end{array}$$

(29)

2.5.6. HT

$j=\mathit{HT}$, ${\lambda }_r,{\lambda }_1,{\lambda }_2$ are jointly optimized. The corresponding optimization problem is formulated as

$$\begin{array}{}\mathrm{(30a)}& \hfill {\mathcal{P}}_{\mathit{HT}}:& \underset{\mathbf{P},\mathbf{c},{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t,{\lambda }_r,{\lambda }_1,{\lambda }_2}{max}\underset{k\in \mathcal{K}}{min}{R}_{tot,k}^{\mathit{HT}}\hfill & \hfill \mathrm{s}.\mathrm{t}.& \left(\mathrm{25b}\right)\text{–}\left(\mathrm{25d}\right),\left(\mathrm{27b}\right),\left(\mathrm{27c}\right)\hfill \mathrm{(30b)}& \hfill & 0<{\lambda }_1\le {\lambda }_r,0<{\lambda }_2\le 1-{\lambda }_r.\hfill \end{array}$$

From problems (25)–(30) for the six transmission modes, we observe that $\mathbf{P}$, ${\mathbf{\Theta }}_r$, ${\mathbf{\Theta }}_t$ are highly coupled in the non-convex and fractional expressions for the achievable common and private rates. Besides, problem (26) and (29) involve integer programming due to (25b). All problems are highly non-convex and difficult to solve. In the next section, we will propose two algorithms to solve the problems.

3. Proposed Optimization Frameworks

To solve the aforementioned six non-convex optimization problems, in this section, we first propose an AO framework that alternatively optimizes the STAR RIS passive beamforming matrices ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$, and the remaining variables. To ease the computational complexity, we further propose a low-complexity algorithm to address the optimization problems. In the end, we extend the proposed optimization framework to the imperfect CSIT scenario.

3.1. Proposed AO Algorithm

We divide the original non-convex problems into two subproblems, one for the STAR RIS passive beamforming design, and the other for the joint BS active beamforming, common rate, and time allocation design. The approaches for solving the two subproblems are detailed in the following:

3.1.1. STAR RIS Passive Beamforming Optimization

With the transmit beamforming $\mathbf{P}$, common rate allocation $\mathbf{c}$ and time allocation variables ${\lambda }_t$, ${\lambda }_1$ and ${\lambda }_2$ fixed, we focus on optimizing ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$. First, we introduce an auxiliary variable $t^j$ to represent the objective function for the transmission mode j, leading to the following constraint:

$$R_{p,k}^j+C_k\ge t^j,\forall k\in \mathcal{K},j\in \{\mathit{FE},\mathit{FM},\mathit{FT},\mathit{HE},\mathit{HM},\mathit{HT}\}.$$

(31)

Although the achievable rate expressions may vary across different transmission modes, our main focus is the SINR expressions as the optimization variables ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$ are embedded solely within the SINR. Note that the non-convexity of the SINR expressions remains consistent across all six transmission modes. The key distinction in optimizing the STAR RIS across these modes lies in the constraints imposed by the ES, MS, and TS protocols.

In the following, we begin by illustrating the SCA-based method employed to address the non-convexity of the SINR expressions, using the FE mode as an example. Following this, we elaborate on the penalty methods proposed to manage the STAR RIS constraints for the ES, MS, and TS protocols, respectively.

• Step 1.1: The SCA method for non-convex SINRs:

Let ${\psi }_{n,i}=\sqrt{{\beta }_{n,i}}{e}^{j{\theta }_{n,i}},n\in \mathcal{N},i\in \{r,t\}$ denote each element of ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$. With ${\mathit{\psi }}_i={[{\psi }_{1,i},\dots ,{\psi }_{N,i}]}^T$, we have ${\mathbf{g}}_k^H{\mathbf{p}}_m+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_m={\overline{g}}_{km}+{\mathbf{s}}_{km}^H{\mathit{\psi }}_i$, ${\mathbf{h}}_n^H{\mathbf{\Theta }}_i{\mathbf{h}}_m={\overline{\mathbf{h}}}_{m,n}^H{\mathit{\psi }}_i$, where ${\overline{g}}_{km}={\mathbf{g}}_k^H{\mathbf{p}}_m$, ${\mathbf{s}}_{km}={\left(\mathrm{diag}\left({\mathbf{h}}_k^H\right)\mathbf{E}{\mathbf{p}}_m\right)}^{*}$ and ${\overline{\mathbf{h}}}_{m,n}={\left(\mathrm{diag}\left({\mathbf{h}}_n^H\right){\mathbf{h}}_m\right)}^{*}\in {\mathbb{C}}^{N\times 1}$. In this way, the optimization variables are represented by ${\mathit{\psi }}_r$ and ${\mathit{\psi }}_t$.

Taking the FE problem (25) as an example, we first introduce slack variables $\mathit{\delta }={[{\delta }_1,\dots ,{\delta }_K]}^T$ and ${\mathit{\delta }}_c={[{\delta }_{c,1},\dots ,{\delta }_{c,K}]}^T$ to, respectively, denote the SINRs of $s_k$ and $s_0$ in the direct transmission phase. Furthermore, we introduce $\mathit{\xi }=[{\xi }_1,\dots ,{\xi }_{K_2}]$ to denote the SINRs of the destination users in the cooperative transmission phase, where $K_2$ is the number of users in ${\mathcal{K}}_2$. The optimization problem (25) for updating the STAR RIS passive beamforming matrices is then equivalently transformed into

$$\begin{array}{}\mathrm{(32a)}& \hfill {\mathcal{P}}_{\mathit{FE}-\mathit{1}:}& \underset{\begin{array}{c}{t}^{\mathit{FE}},{\mathbf{\psi }}_r,{\mathbf{\psi }}_t,\mathbf{\delta },{\mathbf{\delta }}_c,\mathbf{\xi }\end{array}}{max}{t}^{\mathit{FE}}\hfill \mathrm{(32b)}& \hfill \mathrm{s}.\mathrm{t}.& {log}_2(1+{\delta }_k)+C_k\ge t^{\mathit{FE}},\forall k\in \mathcal{K},\hfill \mathrm{(32c)}& \hfill & {log}_2(1+{\delta }_{c,k})\ge {\sum }_{m\in \mathcal{K}}{C}_m,k\in {\mathcal{K}}_1,\hfill \mathrm{(32d)}& \hfill & {log}_2(1+{\delta }_{c,k}+{\xi }_k)\ge {\sum }_{m\in \mathcal{K}}{C}_m,k\in {\mathcal{K}}_2,\hfill \mathrm{(32e)}& \hfill & {\delta }_{c,k}\le {\gamma }_{c,k}^{\mathit{FD},\left[1\right]},\forall k\in \mathcal{K},\hfill \mathrm{(32f)}& \hfill & {\delta }_k\le {\gamma }_k^{\mathit{FD}},\forall k\in \mathcal{K},\hfill \mathrm{(32g)}& \hfill & {\xi }_k\le {\gamma }_{c,k}^{\mathit{FD},\left[2\right]},\forall k\in {\mathcal{K}}_2,\hfill & \hfill & \left(25e\right).\hfill \end{array}$$

For constraints (32e) and (32f), we further introduce slack variables $\mathit{\eta }={[{\eta }_1,\dots ,{\eta }_K]}^T$ and ${\mathit{\eta }}_c={[{\eta }_{c,1},\dots ,{\eta }_{c,K}]}^T$ to, respectively, denote the denominators of the SINRs for $s_0$ and $s_k$. (32e) and (32f) are equivalently transformed to

$$\begin{array}{}\mathrm{(33a)}& {\eta }_{c,k}\ge \mathrm{de}\left({\gamma }_{c,k}^{\mathit{FD},\left[1\right]}\right),\forall k\in \mathcal{K}\hfill \mathrm{(33b)}& {\eta }_k\ge \mathrm{de}\left({\gamma }_k^{\mathit{FD}}\right),\forall k\in \mathcal{K}\hfill \mathrm{(33c)}& {\delta }_{c,k}{\eta }_{c,k}\le {\left|{\overline{g}}_{k0}+{\mathbf{s}}_{k0}^H{\mathit{\psi }}_i\right|}^2,\hfill \mathrm{(33d)}& {\delta }_k{\eta }_k\le {\left|{\overline{g}}_{kk}+{\mathbf{s}}_{kk}^H{\mathit{\psi }}_i\right|}^2,\hfill \end{array}$$

where $\mathrm{de}(·)$ is an operator defined to extract the denominator of a fraction, i.e., $\mathrm{de}\left({\displaystyle \frac{a}{b}}\right)=b$. For constraint (33c), the left-hand side (LHS) equals to ${\delta }_{c,k}{\eta }_{c,k}={\displaystyle \frac{1}{4}}{({\delta }_{c,k}+{\eta }_{c,k})}^2-{\displaystyle \frac{1}{4}}{({\delta }_{c,k}-{\eta }_{c,k})}^2$. By further approximating ${\delta }_{c,k}{\eta }_{c,k}$ at the point $({\delta }_{c,k}^{\left[z\right]},{\eta }_{c,k}^{\left[z\right]})$ in iteration $\left[z\right]$ using the first-order Taylor approximation of ${({\delta }_{c,k}-{\eta }_{c,k})}^2$, we have

$$\begin{array}{cc}\hfill {\delta }_{c,k}{\eta }_{c,k}\le & {\displaystyle \frac{1}{4}}{({\delta }_{c,k}+{\eta }_{c,k})}^2-{\displaystyle \frac{1}{2}}({\delta }_{c,k}^{\left[z\right]}-{\eta }_{c,k}^{\left[z\right]})({\delta }_{c,k}-{\eta }_{c,k})+{\displaystyle \frac{1}{4}}{({\delta }_{c,k}^{\left[z\right]}-{\eta }_{c,k}^{\left[z\right]})}^2\triangleq \nu ({\delta }_{c,k}^{\left[z\right]},{\eta }_{c,k}^{\left[z\right]}).\hfill \end{array}$$

(34)

We also approximate the right-hand side (RHS) of (33c) at ${\mathit{\psi }}_i^{\left[z\right]}$ by the first-order Taylor approximation of ${\left|{\overline{g}}_{k0}+{\mathbf{s}}_{k0}^H{\mathit{\psi }}_i\right|}^2$ as

$$\begin{array}{cc}\hfill {\left|{\overline{g}}_{k0}+{\mathbf{s}}_{k0}^H{\mathit{\psi }}_i\right|}^2\ge & 2\Re \left\{{({\mathbf{s}}_{k0}^H{\mathit{\psi }}_i^{\left[z\right]}+{\overline{g}}_{k0})}^H{\mathbf{s}}_{k0}^H{\mathit{\psi }}_i\right\}-|{\mathbf{s}}_{k0}^H{\mathit{\psi }}_i{|}^2+{|{\overline{g}}_{k0}|}^2\triangleq \varpi ({\mathit{\psi }}_i^{\left[z\right]},{\mathit{\psi }}_i,{\overline{g}}_{k0},{\mathbf{s}}_{k0}).\hfill \end{array}$$

(35)

Based on this approach, constraints (33c) and (33d) are transformed to

$$\begin{array}{}\mathrm{(36a)}& \nu ({\delta }_{c,k}^{\left[z\right]},{\eta }_{c,k}^{\left[z\right]})\le \varpi ({\mathit{\psi }}_i^{\left[z\right]},{\mathit{\psi }}_i,{\overline{g}}_{k0},{\mathbf{s}}_{k0}),\forall k\in {\mathcal{K}}_1,\hfill \mathrm{(36b)}& \nu ({\delta }_k^{\left[z\right]},{\eta }_k^{\left[z\right]})\le \varpi ({\mathit{\psi }}_i^{\left[z\right]},{\mathit{\psi }}_i,{\overline{g}}_{kk},{\mathbf{s}}_{kk}),\forall k\in \mathcal{K}.\hfill \end{array}$$

And constraint (32g) is approximated as

$${\xi }_k{\sigma }^2\le P_m\varpi ({\mathit{\psi }}_i^{\left[z\right]},{\mathit{\psi }}_i,h_{m,k},{\mathbf{h}}_{m,k}),\forall k\in \mathcal{K}.$$

(37)

Therefore, utilizing the SCA method, the non-convex SINR expressions (32e)–(32g) are replaced by (36) and (37), which are convex. This approach can be applied directly to the other five transmission modes, so we simplify by omitting their discussion here.

• Step 1.2: The penalty method for non-convex STAR RIS constraints:

After Step 1.1, the optimization problems for passive beamforming in all six modes become convex except for the constraints related to the phase and amplitude of the STAR RIS. In the following, we specify the penalty methods proposed to handle these constraints for the ES, MS, and TS protocols.

ES: For constraint (25e), we adopt the penalty method proposed in [23]. By introducing a large positive constant C, the objective function is transformed into $t^j+C{\sum }_{n=1}^N(|{\psi }_{n,r}{|}^2+|{\psi }_{n,t}{|}^2-1),j\in \{\mathit{FE},\mathit{HE}\}$ with an additional constraint

$$|{\psi }_{n,r}{|}^2+{|{\psi }_{n,t}|}^2\le 1.$$

(38)

Using the first-order Taylor approximation of $(|{\psi }_{n,r}{|}^2+|{\psi }_{n,t}{|}^2)$ at iteration $\left[z\right]$, we obtain the approximated objective function as

$$t^j+C\sum _{n=1}^N\Re \{2{\left({\psi }_{n,r}^{\left[z\right]}\right)}^{*}{\psi }_{n,r}-|{\psi }_{n,r}^{\left[z\right]}{|}^2+2{\left({\psi }_{n,t}^{\left[z\right]}\right)}^{*}{\psi }_{n,t}-|{\psi }_{n,t}^{\left[z\right]}{|}^2\}.$$

(39)

MS: For constraint (26b), it is obvious that the amplitude coefficient of each STAR RIS element in the FM and HM modes is an integer chosen from 0 to 1. To transform constraints (26b), it is worth noting that no matter whether each element is 0 or 1, it always satisfies (38) and

$$-|{\psi }_{n,i}{|}^2+|{\psi }_{n,i}|=0,i\in \{r,t\}.$$

(40)

(40) forces the amplitudes of ${\psi }_{n,r}$ and ${\psi }_{n,t}$ to 0 or 1. Due to constraint (38), $0 \le |{\psi }_{n,i}|\le 1$, we obtain that

$$-|{\psi }_{n,i}{|}^2+|{\psi }_{n,i}|\ge 0,i\in \{r,t\}.$$

(41)

Hence, constraint (26b) can be replaced by adding a penalty term to the objective function as $t^j-C{\sum }_{n=1}^N(-|{\psi }_{n,r}{|}^2+|{\psi }_{n,r}|-|{\psi }_{n,t}{|}^2+|{\psi }_{n,t}|),j\in \{\mathit{FM},\mathit{HM}\}$. Applying the first-order Taylor approximation to the penalty term $-|{\psi }_{n,r}{|}^2-{|{\psi }_{n,t}|}^2$ at iteration $\left[z\right]$, the objective function is transformed to

$$t^j-C\sum _{n=1}^N\left(A^{\left[z\right]}-2\Re \left\{{\left({\psi }_{n,r}^{\left[z\right]}\right)}^{*}{\psi }_{n,r}\right\}-2\Re \left\{{\left({\psi }_{n,t}^{\left[z\right]}\right)}^{*}{\psi }_{n,t}\right\}\right),$$

(42)

where $A^{\left[z\right]}={|{\psi }_{n,r}^{\left[z\right]}|}^2+{|{\psi }_{n,t}^{\left[z\right]}|}^2+|{\psi }_{n,r}|+|{\psi }_{n,t}|$.

TS: For TS modes, we also adopt the penalty method proposed in [23] to address constraints (27b), which transforms the objective function into $t^j+C{\sum }_{n=1}^N(|{\psi }_{n,r}{|}^2-1+|{\psi }_{n,t}{|}^2-1),j\in \{\mathit{FT},\mathit{HT}\}$ with an additional constraint

$$|{\psi }_{n,i}{|}^2\le 1,n\in \mathcal{N},i\in \{r,t\}.$$

(43)

The objective function is then approximated in the same way as (39).

After Steps 1.1 and 1.2, the STAR RIS optimization subproblems for all six transmission modes become convex quadratically constrained quadratic program (QCQP) and can be solved using the CVX optimization toolbox. The detailed process to solve the STAR RIS passive beamforming optimization problem is summarized in Algorithm 1.

Algorithm 1: STAR RIS passive beamforming optimization algorithm for six transmission modes

3.1.2. Joint Optimization of the Active Beamforming, Common Rate and Time Allocation

Given ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$, the effective channels of all users are fixed. To jointly optimize the remaining variables, we first introduce an auxiliary variable $x^j$ to denote the objective function for the transmission mode j, leading to the following constraints:

$$R_{p,k}^j+C_k\ge x^j,\forall k\in \mathcal{K},j\in \{\mathit{FE},\mathit{FM},\mathit{FT},\mathit{HE},\mathit{HM},\mathit{HT}\}.$$

(44)

In this subproblem, non-convexity arises from both the fractional SINR expressions and the coupling of time allocation variables within the rate expressions. Both of these challenges are tackled using SCA in the following steps.

• Step 2.1: The SCA method for time allocation variables:

In this step, we first specified the SCA methods employed to address the coupled time allocation variables within the rate expressions. Note that in the FE and FM modes, there are no time allocation variables. The non-convexity of the corresponding optimization problems arises solely from the SINR expressions, which are directly addressed in Step 2.2.

FT: To optimize ${\lambda }_r$ in problem (27), we first introduce the slack variables ${\mathit{\alpha }}^i={[{\alpha }_1^i,\dots ,{\alpha }_K^i]}^T$, and ${\mathit{\alpha }}_c^i={[{\alpha }_{c,1}^i,\dots ,{\alpha }_{c,K}^i]}^T$, $i\in \{r,t\}$ to denote the private and common rates. This allows us to transform (44) and (25b) to

$$\begin{array}{}\mathrm{(45a)}& {\lambda }_r{\alpha }_k^r+(1-{\lambda }_r){\alpha }_k^t+C_k\ge x^{\mathit{FT}},\hfill \mathrm{(45b)}& {\lambda }_r{\alpha }_{c,k}^r+(1-{\lambda }_r){\alpha }_{c,k}^t\ge {\sum }_{m\in \mathcal{K}}{C}_m,\hfill \mathrm{(45c)}& {\alpha }_k^i\le {log}_2(1+{\gamma }_k^{\mathit{FD},i}),\hfill \mathrm{(45d)}& {\alpha }_{c,k}^i\le {log}_2(1+{\gamma }_{c,k}^{\mathit{FD},i}).\hfill \end{array}$$

For the bilinear function ${\lambda }_r{\alpha }_k^r$, it can be approximated by the first-order Taylor approximation at iteration $\left[z\right]$ based on $\nu (·)$ in (34). Hence, constraints (45a) and (45b) are approximated as

$$\begin{array}{}\mathrm{(46a)}& -\nu ({\alpha }_k^{r,\left[z\right]},-{\lambda }_r)+{\alpha }_k^t-\nu ({\alpha }_k^{t,\left[z\right]},-{\lambda }_r)+C_k\ge x^{\mathit{FT}},\hfill \mathrm{(46b)}& -\nu ({\alpha }_{c,k}^{r,\left[z\right]},-{\lambda }_r)+{\alpha }_{c,k}^t-\nu ({\alpha }_{c,k}^{t,\left[z\right]},-{\lambda }_r)\ge {\sum }_{m\in \mathcal{K}}{C}_m.\hfill \end{array}$$

HE/HM: To optimize $\lambda$ in problems (28) and (29), akin to the FT mode, we introduce the slack variables $\mathit{\alpha }$, and ${\mathit{\alpha }}_c^i$ to represent the private and common rates, thereby facilitating the transformation of (44) and (25b) to

$$\begin{array}{}\mathrm{(47a)}& \lambda {\alpha }_k+C_k\ge x^j,\hfill \mathrm{(47b)}& \lambda {\alpha }_{c,k}\ge {\sum }_{m\in \mathcal{K}}{C}_m,k\in {\mathcal{K}}_1\hfill \mathrm{(47c)}& \lambda {\alpha }_{c,k}+(1-\lambda )log(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right]})\ge {\sum }_{m\in \mathcal{K}}{C}_m,k\in {\mathcal{K}}_2\hfill \mathrm{(47d)}& {\alpha }_k\le {log}_2(1+{\gamma }_k^{\mathit{HD}}),\hfill \mathrm{(47e)}& {\alpha }_{c,k}\le {log}_2(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right]}).\hfill \end{array}$$

Using the same first-order Taylor approximation method in (46), constraints (47a)–(47c) are approximated to

$$\begin{array}{cc}\hfill & -\nu ({\alpha }_k^{\left[z\right]},-\lambda )+C_k\ge x^j,\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & -\nu ({\alpha }_{c,k}^{\left[z\right]},-\lambda )\ge {\sum }_{m\in \mathcal{K}}{C}_m,\forall k\in {\mathcal{K}}_1,\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill & -\nu ({\alpha }_{c,k}^{\left[z\right]},-\lambda )+(1-\lambda ){log}_2(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right]})\ge {\sum }_{m\in \mathcal{K}}{C}_m,\forall k\in {\mathcal{K}}_2.\hfill \end{array}$$

(50)

HT: To jointly optimize ${\lambda }_r$, ${\lambda }_1$, and ${\lambda }_2$ in problems (28), we also introduce the slack variables ${\mathit{\alpha }}^i$, and ${\mathit{\alpha }}_c^i$, $i\in \{r,t\}$ to transform the problem. By applying the approximation method described in (46), we approximate (44) and (25b) to

$$\begin{array}{}\mathrm{(51a)}& -\nu ({\alpha }_k^{r,\left[z\right]},-{\lambda }_1)-\nu ({\alpha }_k^{t,\left[z\right]},-{\lambda }_2)+C_k\ge x^{\mathit{HT}},\hfill \mathrm{(51b)}& -\nu ({\alpha }_{c,k}^{r,\left[z\right]},-{\lambda }_1)-\nu ({\alpha }_{c,k}^{t,\left[z\right]},-{\lambda }_2)\ge {\sum }_{m\in \mathcal{K}}{C}_m,\forall k\in {\mathcal{K}}_1,\hfill & -\nu ({\alpha }_{c,k}^{r,\left[z\right]},-{\lambda }_1)+({\lambda }_r-{\lambda }_1){log}_2(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right],r})-\hfill \mathrm{(51c)}& \nu ({\alpha }_{c,k}^{r,\left[z\right]},-{\lambda }_2)+(1-{\lambda }_r-{\lambda }_2){log}_2(1+{\gamma }_{c,k}^{\mathit{HD},\left[2\right],t})\ge {\sum }_{m\in \mathcal{K}}{C}_m,\forall k\in {\mathcal{K}}_2,\hfill \mathrm{(51d)}& {\alpha }_{c,k}^i\ge log(1+{\gamma }_{c,k}^{\mathit{HD},\left[1\right],i}),\hfill \mathrm{(51e)}& {\alpha }_k^i\ge log(1+{\gamma }_k^{\mathit{HD},i}).\hfill \end{array}$$

Following Step 2.1, the time allocation variables are decoupled from the non-convex SINR expressions for all transmission modes. Next, we proceed to address the classical non-convex SINR expressions.

• Step 2.2: The SCA method for non-convex SINRs:

Similar to Step 1.1, the SCA method for SINRs is applicable to all transmission modes. Therefore, we illustrate this approach using the FE problem (25) as an example, without detailing other modes.

We first introduce slack variables $\mathit{\iota }={[{\iota }_1,\cdots ,{\iota }_K]}^T$, ${\mathit{\iota }}_c={[{\iota }_{c,1},\cdots ,{\iota }_{c,K}]}^T$ to denote the SINRs of $s_k$ and $s_0$. Constraints (44) and (25b) are equivalently transformed to

$$\begin{array}{}\mathrm{(52a)}& {log}_2(1+{\iota }_k)+C_k\ge x^{\mathit{FE}},\forall k\in \mathcal{K},\hfill \mathrm{(52b)}& {log}_2(1+{\iota }_{c,k})\ge {\sum }_{m\in \mathcal{K}}{C}_m,k\in \mathcal{K},\hfill \mathrm{(52c)}& {\iota }_{c,k}\le {\gamma }_{c,k}^{\mathit{FD},\left[1\right]},\forall k\in \mathcal{K},\hfill \mathrm{(52d)}& {\iota }_k\le {\gamma }_k^{\mathit{FD}},\forall k\in \mathcal{K}.\hfill \end{array}$$

Constraints (52c) and (52d) remain non-convex, we then introduce variables $\mathit{\zeta }={[{\zeta }_1,\cdots ,{\zeta }_K]}^T$, ${\mathit{\zeta }}_c={[{\zeta }_{c,1},\cdots ,{\zeta }_{c,K}]}^T$ to, respectively, denote the denominators of SINRs for $s_k$ and $s_0$. (52c) and (52d) are equivalently transformed to

$$\begin{array}{}\mathrm{(53a)}& {\zeta }_{c,k}\ge \mathrm{de}\left({\gamma }_{c,k}^{\mathit{FD},\left[1\right]}\right),\forall k\in \mathcal{K},\hfill \mathrm{(53b)}& {\zeta }_k\ge \mathrm{de}\left({\gamma }_k^{\mathit{FD}}\right),\forall k\in \mathcal{K},\hfill \mathrm{(53c)}& {\iota }_{c,k}{\zeta }_{c,k}\le {\left|{\tilde{\mathbf{g}}}_{i,k}^H{\mathbf{p}}_0\right|}^2,\hfill \mathrm{(53d)}& {\iota }_k{\zeta }_k\le {\left|{\tilde{\mathbf{g}}}_{i,k}^H{\mathbf{p}}_k\right|}^2.\hfill \end{array}$$

Following the approaches specified in Step 1.1 to address the non-convexity in (33), we approximate constraints (53c) and (53d) at iteration $\left[z\right]$ to

$$\begin{array}{}\mathrm{(54a)}& \nu ({\iota }_{c,k}^{\left[z\right]},{\zeta }_{c,k}^{\left[z\right]})\le \varpi ({\mathbf{p}}_0^{\left[z\right]},{\mathbf{p}}_0,0,{\tilde{\mathbf{g}}}_{i,k}),\forall k\in \mathcal{K},\hfill \mathrm{(54b)}& \nu ({\iota }_k^{\left[z\right]},{\zeta }_k^{\left[z\right]})\le \varpi ({\mathbf{p}}_k^{\left[z\right]},{\mathbf{p}}_k,0,{\tilde{\mathbf{g}}}_{i,k}),\forall k\in \mathcal{K}.\hfill \end{array}$$

Based on the approximation methods specified in Steps 2.1 and 2.2, the joint active beamforming, common rate, and time allocation optimization problems for all six transmission modes become QCQP and can be solved using CVX. The detailed process of the SCA methods to solve this subproblem is illustrated in Algorithm 2.

Algorithm 2: Joint transmit active beamforming and resource allocation algorithm

3.1.3. Alternative Optimization

Based on Algorithms 1 and 2, the proposed AO algorithm that solves the two subproblems iteratively is shown in Algorithm 3. Starting from a feasible point $({\mathbf{P}}^{\left[0\right]},{\mathbf{\Theta }}_r^{\left[0\right]},{\mathbf{\Theta }}_t^{\left[0\right]},{\mathbf{c}}^{\left[0\right]},{\lambda }_r^{\left[0\right]},{\lambda }^{\left[0\right]},{\lambda }_1^{\left[0\right]},{\lambda }_2^{\left[0\right]})$, at each iteration $\left[z\right]$, we first update ${\mathbf{\Theta }}_r^{\left[z\right]}$ and ${\mathbf{\Theta }}_t^{\left[z\right]}$ by Algorithm 1. Given ${\mathbf{\Theta }}_r^{\left[z\right]}$ and ${\mathbf{\Theta }}_t^{\left[z\right]}$, we then update ${\mathbf{P}}^{\left[z\right]}$, the common rate and time allocation based on Algorithm 2. By iteratively solving these two subproblems, the objective function is updated until convergence.

Algorithm 3: Proposed AO algorithm

3.2. Proposed Low-Complexity Algorithm

In this subsection, we propose a low-complexity algorithm for the STAR RIS passive beamforming and transmit beamforming optimization subproblems.

3.2.1. STAR RIS Passive Beamforming Optimization

We begin by specifying the proposed low-complexity algorithm tailored for the ES protocol. Subsequently, we extend this algorithm to the MS and TS protocols.

ES: To develop a low-complexity algorithm for designing ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$ in the ES protocol, we consider the following problem aimed at maximizing the sum of channel gains:

$$\begin{array}{cc}\hfill \underset{{\mathbf{\Theta }}_r,{\mathbf{\Theta }}_t}{max}& \sum _{k=1}^K\parallel {\mathbf{g}}_k^H+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i{\mathbf{E}\parallel }^2,i\in \{r,t\}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(\mathrm{25c}\right),\left(25e\right).\hfill \end{array}$$

(55)

Constraints (25c) and (25e) are non-convex, which make the problem (55) difficult to solve. To tackle such a non-convex problem, we first relax constraints (25c) and (25e) to $\parallel {\mathbf{\Theta }}_r{\parallel }_F^2+{\parallel {\mathbf{\Theta }}_t\parallel }_F^2\le N$, where ${\parallel ·\parallel }_F$ denotes the Frobenius norm. This approximation relaxes the feasible solutions of ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$ to a convex set $\mathcal{S}=\left\{({\mathbf{\Theta }}_r,{\mathbf{\Theta }}_r)\right|\parallel {\mathbf{\Theta }}_r{\parallel }_F^2+\parallel {\mathbf{\Theta }}_t{\parallel }_F^2\le N\}$. Therefore, problem (55) is relaxed to

$$\begin{array}{}\mathrm{(56a)}& \hfill \underset{\mathbf{\Theta }}{max}& f(\mathbf{\Theta })\triangleq \parallel {\mathbf{G}}^H+{\mathbf{H}}^H\mathbf{\Theta }{\mathbf{E}}_x{\parallel }_F^2\hfill \mathrm{(56b)}& \hfill \mathrm{s}.\mathrm{t}.& \mathbf{\Theta }\in \mathcal{S},\hfill \end{array}$$

where $\mathbf{\Theta }=\left[\begin{array}{cc}{\mathbf{\Theta }}_r& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Theta }}_t\end{array}\right]$, $\mathbf{G}=[{\mathbf{G}}_r;{\mathbf{G}}_t],{\mathbf{G}}_r=\left\{{\mathbf{g}}_k\right|k\in {\mathcal{K}}_r\},{\mathbf{G}}_t=\left\{{\mathbf{g}}_k\right|k\in {\mathcal{K}}_t\}$, $\mathbf{H}=[{\mathbf{H}}_r;{\mathbf{H}}_t],{\mathbf{H}}_r=\left\{{\mathbf{h}}_k\right|k\in {\mathcal{K}}_r\},{\mathbf{H}}_t=\left\{{\mathbf{h}}_k\right|k\in {\mathcal{K}}_t\}$ and ${\mathbf{E}}_x=[\mathbf{E};\mathbf{E}]$.

Inspired by the method proposed in [24], we employ the gradient decent method at the point ${\mathbf{\Theta }}_0=\mathbf{0}$ to derive a closed-form solution of (56). The corresponding gradient of $f(\mathbf{\Theta })$ at ${\mathbf{\Theta }}_0=\mathbf{0}$ is

$${▽}_{\mathbf{\Theta }}f\left({\mathbf{\Theta }}_0\right)=\mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H,$$

(57)

By using (57) as the decent direction, i.e., ${\mathbf{D}}_0={▽}_{\mathbf{\Theta }}f\left({\mathbf{\Theta }}_0\right)$, we then take the Armijo rule to determine the step size $\alpha$ as

$$f({\mathbf{\Theta }}_0+\alpha {\mathbf{D}}_0)\ge f\left({\mathbf{\Theta }}_0\right)+\varphi ·\alpha ·\mathrm{tr}\left({▽}_{\mathbf{\Theta }}f{\left({\mathbf{\Theta }}_0\right)}^H{\mathbf{D}}_0\right),$$

(58)

where $\varphi \in (0,0.5)$ is a constant shrinkage factor. Substituting (57) into (58), we obtain

$${\alpha }^2\parallel {\mathbf{H}}^H\mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H{\mathbf{E}}_x{\parallel }_F^2+\alpha ·(2-\varphi ){\parallel \mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H\parallel }_F^2\ge 0.$$

(59)

To guarantee the solution is on the boundary of the convex set $\mathcal{S}$, we set $\alpha ={\displaystyle \frac{\sqrt{N}}{\parallel \mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H{\parallel }_F}}$. Then, we obtain the following solution of problem (56) as

$$\mathbf{\Theta }={\displaystyle \frac{\sqrt{N}}{\parallel \mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H{\parallel }_F}}\mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H.$$

(60)

However, the solution (60) for the relaxed problem (56) may not satisfy the original constraints (25c), (25e). Therefore, we propose to project $\mathbf{\Theta }$ into the feasible region. Following [24], we first define the operator $\mathrm{sym}(·)$ that projects any square matrix $\mathbf{X}$ to a symmetric matrix, i.e.,

$$\mathrm{sym}\left(\mathbf{X}\right)={\displaystyle \frac{1}{2}}\left({\mathbf{X}}^T+\mathbf{X}\right),$$

(61)

and the operator $\mathrm{uni}(·)$ which projects an arbitrary matrix $\mathbf{X}$ to a unitary matrix as

$$\mathrm{uni}\left(\mathbf{X}\right)={\mathbf{UV}}^H,$$

(62)

where $\mathbf{U}$ and $\mathbf{V}$ are unitary matrices obtained by the singular value decomposition (SVD) of $\mathbf{X}$, i.e., $\mathbf{X}={\mathbf{USV}}^H$. Consider $\mathbf{X}$ with dimension N and rank R. Then, the unitary matrices $\mathbf{U}$ and $\mathbf{V}$ can be partitioned as $\mathbf{U}=[{\mathbf{U}}_R,{\mathbf{U}}_{N-R}]$ and $\mathbf{V}=[{\mathbf{V}}_R,{\mathbf{V}}_{N-R}]$, respectively. Based on the operators in (61) and (62), we could then define the symmetric unitary projection as [24]

$$\mathrm{symuni}\left(\mathbf{X}\right)=\mathrm{uni}\left(\mathrm{sym}\left(\mathbf{X}\right)\right)=\widehat{\mathbf{U}}{\mathbf{V}}^H,$$

(63)

where $\widehat{\mathbf{U}}=[{\mathbf{U}}_R,{\mathbf{V}}_{N-R}^{*}]$.

Utilizing the symmetric unitary projection, we then derive a feasible STAR RIS passive beamforming solution $\mathbf{\Theta }$. It is noteworthy that both ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$ are diagonal matrices, implying that $\mathbf{\Theta }$ is also diagonal. The diagonal entries of $\mathbf{\Theta }$ are determined by

$$\mathrm{diag}\{\mathbf{\Theta }\}={\displaystyle \frac{\sqrt{2}}{2}}\mathrm{symuni}\left\{\mathrm{diag}\left(\mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H\right)\right\}.$$

(64)

Since the unitary matrix $\mathrm{symuni}\left\{\mathrm{diag}\left(\mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H\right)\right\}\in {\mathbb{C}}^{2N\times 2N}$ whose dimension is $2N$, we have $\parallel \mathrm{symuni}\left\{\mathrm{diag}\left(\mathbf{H}{\mathbf{G}}^H{\mathbf{E}}_x^H\right)\right\}{\parallel }_F^2=2N$. To ensure constraint (25e), we scale each entry with ${\displaystyle \frac{\sqrt{2}}{2}}$.

MS: For the MS mode, we first utilize (64) to derive ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$. As each element can only operate in either reflection or transmission mode, we then compare the reflection and transmission amplitude coefficients of each STAR RIS element. The element with the higher coefficient is designated as 1, while the one with the lower coefficient is designated as 0. Based on (64), the reflection and transmission coefficients of each STAR RIS element are designed as

$$\begin{array}{cc}\hfill {\beta }_{r,n}& =\left\{\begin{array}{c}1, \mathrm{if} |{\mathbf{\Theta }}_{n,n}| - |{\mathbf{\Theta }}_{N+n,N+n}| \ge 0\\ 0, \mathrm{if} |{\mathbf{\Theta }}_{n,n}| - |{\mathbf{\Theta }}_{N+n,N+n}| <0\end{array}\right., \forall n\in \mathcal{N}\hfill \\ \hfill {\beta }_{t,n}& =1-{\beta }_{r,n}.\hfill \end{array}$$

(65)

TS: For the TS mode, in the reflection time slots, ${\mathbf{\Theta }}_t=\mathbf{0}$, the amplitudes of all elements in ${\mathbf{\Theta }}_r$ are 1. The solution of the reflection matrix is given as

$${\mathbf{\Theta }}_r=\mathrm{symuni}\left\{\mathrm{diag}\left({\mathbf{H}}_r{\mathbf{G}}_r^H{\mathbf{E}}^H\right)\right\}.$$

(66)

In the transmission time slots, the transmission matrix is given as

$${\mathbf{\Theta }}_t=\mathrm{symuni}\left\{\mathrm{diag}\left({\mathbf{H}}_t{\mathbf{G}}_t^H{\mathbf{E}}^H\right)\right\}.$$

(67)

3.2.2. Joint Optimization of the Active Beamforming, Common Rate and Time Allocation

With fixed ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$, we aim to simplify the design of the remaining variables. To achieve this, we reduce the optimization dimension by fixing the direction of the precoders. Subsequently, we focus on jointly optimizing the power, common rate, and time allocation variables.

Let ${\overline{\mathbf{g}}}_k={\mathbf{g}}_k/\left|\right|{\mathbf{g}}_k\left|\right|$, $\overline{\mathbf{G}}=[{\overline{\mathbf{g}}}_1,\dots ,{\overline{\mathbf{g}}}_K]$, ${\overline{\mathbf{p}}}_k={\mathbf{p}}_k/\left|\right|{\mathbf{p}}_k\left|\right|$, and $\overline{\mathbf{P}}=[{\overline{\mathbf{p}}}_1,\dots ,{\overline{\mathbf{p}}}_K]$. We transform the active beamformings in the form of ${\mathbf{p}}_c=\sqrt{{\rho }_c{P}_t}{\overline{\mathbf{p}}}_c,{\mathbf{p}}_k=\sqrt{{\rho }_k{P}_t}{\overline{\mathbf{p}}}_k$, where ${\rho }_c$ and ${\rho }_k$ are the power allocation coefficients. For the beamforming direction ${\overline{\mathbf{p}}}_k$ of the private streams, we adopt zero-forcing (ZF) beamforming, i.e., $\overline{\mathbf{P}}=\overline{\mathbf{G}}{({\overline{\mathbf{G}}}^H\overline{\mathbf{G}})}^{-1}$ [25]. For the beamforming direction ${\overline{\mathbf{p}}}_c$ of the common stream, we utilize maximum ratio transmission (MRT) and SVD methods as ${\overline{\mathbf{p}}}_c={\mathbf{u}}_c$, where ${\mathbf{u}}_c={\mathbf{U}}_c(:,1),\mathbf{G}={\mathbf{U}}_c{\mathbf{S}}_c{\mathbf{V}}_c$ [2]. Based on the aforementioned beamforming direction design, the optimization problems for all six transmission modes are simplified. In the following, we illustrate this simplification by considering the FD mode as a representative example:

$$\begin{array}{}\mathrm{(68a)}& \hfill & \underset{\mathit{\rho },\mathbf{c}}{max}\underset{k\in \mathcal{K}}{min}{C}_k+R_{p,k}^{\mathrm{FE}}\hfill \mathrm{(68b)}& \hfill \mathrm{s}.\mathrm{t}.& {\sum }_{m\in \mathcal{K}}{C}_m\le R_{c,k}^{\mathrm{FE}},\forall k\in \mathcal{K},\hfill \mathrm{(68c)}& \hfill & {\rho }_c+{\sum }_{k\in \mathcal{K}}{\rho }_k\le 1,\hfill \end{array}$$

where $\mathit{\rho }=[{\rho }_c,{\rho }_1,\dots ,{\rho }_K]$ are the power allocation coefficients of the BS, we have $R_{p,k}^{\mathrm{FE}}={log}_2\left(1+{\rho }_k{P}_t{\left|{\tilde{\mathbf{g}}}_k^H{\overline{\mathbf{p}}}_k\right|}^2/{\sigma }^2\right)$, $R_{c,k}^{\mathrm{FE}}={log}_2\left(1+{\rho }_c{P}_t|{\tilde{\mathbf{g}}}_k^H{\overline{\mathbf{p}}}_c{|}^2/({\rho }_k{P}_t|{\tilde{\mathbf{g}}}_k^H{\overline{\mathbf{p}}}_k{|}^2+{\sigma }^2)\right)$. The optimization problem can be directly solved using the SCA-based approach specified in Algorithm 2. The optimization approach for other transmission modes aligns with that of the FE mode. For FT/HE/HM/HT modes, the additional time allocation variables are jointly optimized using the SCA-based approach specified in Step 2.1.

The detailed process of the proposed low-complexity algorithm to solve problems (25)–(30) is outlined in Algorithm 4. Contrasted with Algorithm 3, the primary difference lies in the closed-form STAR RIS passive beamforming design and the closed-form active beamforming direction design. Additionally, there is no alternating optimization between the two subproblems. Each subproblem is solved once before the output.

Algorithm 4: Proposed low-complexity algorithm

1  Initialize: $j,{\mathbf{P}}^{\left[0\right]},{\mathbf{c}}^{\left[0\right]},{\lambda }_r^{\left[0\right]},{\lambda }^{\left[0\right]},{\lambda }_1^{\left[0\right]},{\lambda }_2^{\left[0\right]}$ 2  Update ${\mathbf{\Theta }}_t$ and ${\mathbf{\Theta }}_r$ through (64) if $j=\mathit{FE}/\mathit{HE}$, (65) if $j=\mathit{FM}/\mathit{HM}$, (66) and (67) if $j=\mathit{FT}/\mathit{HT}$. 3  Given ${\mathbf{\Theta }}_t$ and ${\mathbf{\Theta }}_r$, update $\mathbf{P}$, $\mathbf{c}$, ${\lambda }_r$, $\lambda$, ${\lambda }_1$, ${\lambda }_2$ by solving the joint power, common rate, and time allocation problem via Algorithm 2.

3.3. Convergence and Computational Complexity Analysis

3.3.1. Convergence Analysis

In Algorithm 1, the SCA method guarantees the objective $t^j$ monotonically increases, meaning that $t^{j,[z-1]}\le t^{j,\left[z\right]}$. This arises from the fact that the solution of ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$ at iteration $[z-1]$ is a feasible point of the STAR RIS problem at iteration $\left[z\right]$. Due to STAR RIS power constraint (25e), there is an upper bound for $t^{j,\left[z\right]}$. Hence, the convergence of Algorithm 1 is guaranteed. Similarly, in Algorithm 2, the SCA method guarantees that $x^{j,[z-1]}\le x^{j,\left[z\right]}$ since the solution at iteration $[z-1]$ is an also feasible point of the joint optimization problem at iteration $\left[z\right]$. Due to the power constraint (25d) at the BS, there is an upper bound for the objective value. Hence, the convergence of the Algorithm 2 is guaranteed. The convergence of Algorithms 1 and 2 implies that the objective of the AO algorithm increases monotonically, which guarantees the convergence of Algorithm 3.

3.3.2. Computational Complexity of Algorithm 3

The computational complexity of Algorithm 1 is $\mathcal{O}\left(N^{3.5}{log}_2(1/{ϵ}_1)\right)$, where ${ϵ}_1$ is the convergence tolerance. The complexity of Algorithm 2 is $\mathcal{O}\left({\left(K{N}_t\right)}^{3.5}{log}_2(1/{ϵ}_2)\right)$, where ${ϵ}_2$ is the corresponding convergence tolerance. Therefore, the computational complexity of Algorithm 3 is $\mathcal{O}\left(T\left(N^{3.5}{log}_2(1/{ϵ}_1)+{\left(K{N}_t\right)}^{3.5}{log}_2(1/{ϵ}_2)\right)\right)$, where T denotes the number of iterations of the AO algorithm.

3.3.3. Computational Complexity of Algorithm 4

For the proposed low-complexity algorithm, the computational complexity of the STAR RIS passive beamforming design is mainly occupied by the unitary projection with the SVD method, which is $\mathcal{O}\left(N^3\right)$. For the active beamforming optimization subproblem, the computational complexity is $\mathcal{O}\left(K^{3.5}{log}_2(1/{ϵ}_2)\right)$. Therefore, the overall computational complexity of Algorithm 4 is $\mathcal{O}\left(N^3+K^{3.5}{log}_2(1/{ϵ}_2)\right)$, which is significantly lower than that of Algorithm 3.

3.4. Imperfect CSIT

Due to quantization errors in CSI feedback, feedback delays, and many other issues, it is difficult for the BS to obtain the perfect CSIT. Hence, in this subsection, we focus on an imperfect CSIT scenario where the direct channel between the BS and users, the RIS and users is imperfect. Since the locations of the BS and STAR RIS are typically fixed, perfect CSIT between the BS and the STAR RIS is considered. In order to simplify the notations, we introduce a block diagonal matrix $\mathbf{Q}$ including all perfect channel coefficients, which is given as

$$\mathbf{Q}=\mathit{diag}\left(\left[{\mathbf{g}}_1,\dots ,{\mathbf{g}}_K\right],\left[{\mathbf{h}}_1,\dots ,{\mathbf{h}}_K\right],\mathbf{E}\right).$$

(69)

We denote the estimated CSI between the BS and user-k as ${\widehat{\mathbf{g}}}_k$, and the estimated CSI between the STAR RIS and user-k as ${\widehat{\mathbf{h}}}_k$. Besides, we denote ${\tilde{\mathbf{g}}}_k$ and ${\tilde{\mathbf{h}}}_k$ as the corresponding estimation errors, both following independent and identically distributed (i.i.d) zero-mean complex Gaussian distributions. The corresponding error covariance matrices are denoted as ${\mathbb{E}}_{{\mathbf{g}}_k|{\widehat{\mathbf{g}}}_k}\{{\mathbf{g}}_k{\mathbf{g}}_k^H|{\widehat{\mathbf{g}}}_k\}={\mathbf{R}}_{g,k}$ and ${\mathbb{E}}_{{\mathbf{h}}_k|{\widehat{\mathbf{h}}}_k}\{{\mathbf{h}}_k{\mathbf{h}}_k^H|{\widehat{\mathbf{h}}}_k\}={\mathbf{R}}_{h,k}$, respectively. The relationship between the real channel matrix $\mathbf{Q}$ and the estimated channel matrix $\widehat{\mathbf{Q}}$ is modeled as

$$\mathbf{Q}=\widehat{\mathbf{Q}}+\tilde{\mathbf{Q}},$$

(70)

where $\widehat{\mathbf{Q}}=\mathit{diag}\left(\left[{\widehat{\mathbf{g}}}_1,\dots ,{\widehat{\mathbf{g}}}_K\right],\left[{\widehat{\mathbf{h}}}_1,\dots ,{\widehat{\mathbf{h}}}_K\right],\mathbf{E}\right)$ and $\tilde{\mathbf{Q}}=\mathit{diag}\left(\left[{\tilde{\mathbf{g}}}_1,\dots ,{\tilde{\mathbf{g}}}_K\right],\left[{\tilde{\mathbf{h}}}_1,\dots ,{\tilde{\mathbf{h}}}_K\right],\mathbf{0}\right)$.

With partial CSIT, the instantaneous rate cannot guarantee decodability for the users. A more robust approach is to optimize the ER which characterizes the long-term rate performance across all channel states. The ERs for the common and private streams at user-k are defined as

$$\begin{array}{}\mathrm{(71a)}& {\overline{R}}_{c,k}^j={\mathbb{E}}_{\{\widehat{\mathbf{Q}},\mathbf{Q}\}}\left\{R_{c,k}^j\right\},\hfill \mathrm{(71b)}& {\overline{R}}_{p,k}^j={\mathbb{E}}_{\{\widehat{\mathbf{Q}},\mathbf{Q}\}}\left\{R_{p,k}^j\right\}.\hfill \end{array}$$

Optimizing ERs directly is very challenging. Instead, we can leverage average rates (ARs) to obtain ERs [26]. With a given channel estimation $\widehat{\mathbf{Q}}$, ARs for the common and private streams at user-k are defined as

$$\begin{array}{}\mathrm{(72a)}& {\widehat{R}}_{c,k}^j={\mathbb{E}}_{\{\mathbf{Q}|\tilde{\mathbf{Q}}\}}\left\{R_{c,k}^j|\widehat{\mathbf{Q}}\right\},\hfill \mathrm{(72b)}& {\widehat{R}}_{p,k}^j={\mathbb{E}}_{\{\mathbf{Q}|\tilde{\mathbf{Q}}\}}\left\{R_{p,k}^j|\widehat{\mathbf{Q}}\right\}.\hfill \end{array}$$

The relation between ERs and ARs is as follows [26]

$$\begin{array}{}\mathrm{(73a)}& {\overline{R}}_{c,k}^j={\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{{\widehat{R}}_{c,k}^j\right\},\hfill \mathrm{(73b)}& {\overline{R}}_{p,k}^j={\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{{\widehat{R}}_{p,k}^j\right\}.\hfill \end{array}$$

To ensure that the common message $s_0$ is successfully decoded by all users, the ER of the common stream $s_0$ should satisfy

$$\begin{array}{c}\hfill {\overline{R}}_c^j=\underset{k\in \mathcal{K}}{min}\left\{{\overline{R}}_{c,k}^j\right\}.\end{array}$$

(74)

By replacing the instantaneous rates in (25)–(30) with the average rates in (72), we obtain the stochastic AR optimization problem for a given channel estimation $\widehat{\mathbf{Q}}$. A conventional approach to address such stochastic problems is the sampled average approximation (SAA) approach [2], which transforms the problems into deterministic ones. However, such an approach requires a large number of channel samples and significantly increases the computational complexity. To avoid these issues, we follow [27] and obtain the lower bounds for the ARs. For example, the AR ${\widehat{R}}_{c,k}^j$ in (72) is bounded by

$$\begin{array}{cc}\hfill {\widehat{R}}_{c,k}^j& ={\mathbb{E}}_{\mathbf{Q}|\widehat{\mathbf{Q}}}\left\{{log}_2\left(1+{\displaystyle \frac{|{\mathbf{g}}_k^H{\mathbf{p}}_c+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_c{|}^2}{{\sum }_{j=1}^K{|{\mathbf{g}}_k^H{\mathbf{p}}_j+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_j|}^2+{\sigma }_k^2}}\right)\right\}\hfill \\ \hfill & \ge {log}_2\left(1+{\displaystyle \frac{|{\widehat{\mathbf{g}}}_k^H{\mathbf{p}}_c+{\widehat{\mathbf{h}}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_c{|}^2}{{\sum }_{j=1}^K\left(|{\widehat{\mathbf{g}}}_k^H{\mathbf{p}}_j{|}^2+{\mathbf{p}}_j^H{\mathbf{R}}_{g,k}{\mathbf{p}}_j+{\mathbf{q}}_{i,j}^H{\mathbf{R}}_{h,k}{\mathbf{q}}_{i,j}\right)+{\mathbf{p}}_c^H{\mathbf{R}}_{h,k}{\mathbf{p}}_c+{\mathbf{q}}_{i,c}^H{\mathbf{R}}_{h,k}{\mathbf{q}}_{i,c}+{\sigma }_k^2}}\right),\hfill \\ \hfill & \triangleq {\widehat{R}}_{c,k}^{j,lb},\hfill \end{array}$$

(75)

where ${\mathbf{q}}_{i,j}={\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_j$. The inequality is obtained by treating ${\mathbf{e}}_{g,k}^H{\mathbf{p}}_c{s}_c+{\mathbf{e}}_{h,k}^H{\mathbf{q}}_{i,c}{s}_c+{\sum }_{j=1}^K({\mathbf{e}}_k^H{\mathbf{p}}_j+{\mathbf{e}}_{h,k}^H{\mathbf{q}}_{i,j})s_j$ as independent Gaussian noise and applying Jensen’s inequality [28]. Similarly, the AR ${\widehat{R}}_{p,k}^j$ is bounded by

$$\begin{array}{cc}\hfill {\widehat{R}}_{p,k}^j& ={\mathbb{E}}_{\mathbf{Q}|\widehat{\mathbf{Q}}}\left\{{log}_2\left(1+{\displaystyle \frac{|{\mathbf{g}}_k^H{\mathbf{p}}_k+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_k{|}^2}{{\sum }_{j\ne k}^K{|{\mathbf{g}}_k^H{\mathbf{p}}_j+{\mathbf{h}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_j|}^2+{\sigma }_k^2}}\right)\right\}\hfill \\ \hfill & \ge {log}_2\left(1+{\displaystyle \frac{|{\widehat{\mathbf{g}}}_k^H{\mathbf{p}}_k+{\widehat{\mathbf{h}}}_k^H{\mathbf{\Theta }}_i\mathbf{E}{\mathbf{p}}_k{|}^2}{{\sum }_{j\ne k}^K{\left|{\widehat{\mathbf{g}}}_k^H{\mathbf{p}}_j\right|}^2+{\sum }_{j=1}^K{\mathbf{p}}_j^H{\mathbf{R}}_{g,k}{\mathbf{p}}_j+{\sum }_{j=1}^K{\mathbf{q}}_{i,j}^H{\mathbf{R}}_{h,k}{\mathbf{q}}_{i,j}+{\sigma }_k^2}}\right)\hfill \\ \hfill & \triangleq {\widehat{R}}_{p,k}^{j,lb}.\hfill \end{array}$$

(76)

With the lower bound (75), the ER of the common stream is, therefore, bounded by

$$\begin{array}{c}\hfill {\overline{R}}_c=\underset{k\in \mathcal{K}}{min}\left\{{\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{{\widehat{R}}_{c,k}^j\right\}\right\}\ge \underset{k\in \mathcal{K}}{min}\left\{{\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{{\widehat{R}}_{c,k}^{j,lb}\right\}\right\}\ge {\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{\underset{k\in \mathcal{K}}{min}\left\{{\widehat{R}}_{c,k}^{j,lb}\right\}\right\}.\end{array}$$

(77)

Similarly, the ER of the private stream is bounded by

$$\begin{array}{c}\hfill {\overline{R}}_p={\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{{\widehat{R}}_{p,k}^j\right\}\ge {\mathbb{E}}_{\widehat{\mathbf{Q}}}\left\{{\widehat{R}}_{p,k}^{j,lb}\right\}.\end{array}$$

(78)

By replacing the instantaneous rates $R_c^j$ and $R_{p,k}^j$ in (25)–(30) with the lower bounds $\underset{k\in \mathcal{K}}{min}{\widehat{R}}_{c,k}^{lb}$ and ${\widehat{R}}_{p,k}^{lb}$ in (75) and (76), we construct six long-term deterministic imperfect CSIT optimization problems. As the non-convexity in these problems remains the same as the perfect CSIT ones, our proposed two algorithms can be employed to address the imperfect CSIT optimization problems directly. Therefore, we omit the algorithm framework here.

4. Numerical Results

In this section, we evaluate the performance of our proposed STAR RIS-assisted CRS system framework along with the two proposed algorithms.

4.1. Simulation Setting

The setting of the simulation follows [29]. Consider a three-dimensional (3D) space, the BS is located at $(0,0,0)$ m, the STAR RIS is located at $(0,50,0)$ m. Users are randomly distributed within a circle, with the center being the location of the STAR RIS and a radius of $r=5$ m. We follow the method in [4] to separate the users into group ${\mathcal{K}}_1$ and ${\mathcal{K}}_2$. We only choose one user with the largest channel gain as the relaying user, other users are destination users. The channels between BS and users, as well as between user-m and user-n are assumed to be Rayleigh fading, and are denoted as ${\mathbf{g}}_k=L_{\mathrm{BU},k}{\overline{\mathbf{g}}}_k,h_{m,n}=L_{\mathrm{UU}}{\overline{h}}_{m,n}$, where $L_{\mathrm{BU},k}$ and $L_{\mathrm{UU}}$ denote the path loss between the BS and users as well as the path loss between users. ${\overline{\mathbf{g}}}_k$ and ${\overline{h}}_{m,n}$ follow the complex Gaussian distribution with a certain variance, i.e., ${\overline{\mathbf{g}}}_k\sim \mathcal{CN}(0,{\sigma }_k^2),{\overline{h}}_{m,n}\sim \mathcal{CN}(0,{\sigma }_{m,n}^2)$. The channels between the BS and STAR RIS, as well as between STAR RIS and users are assumed to follow the Rician fading model with the Rician factor being 3 dB. The path loss for all the aforementioned channels is defined as $L_x=L_0{(d_x/d_0)}^{-{\alpha }_x},x\in \{\mathrm{BU},\mathrm{UU},\mathrm{BR},\mathrm{RU}\}$, where $d_0=1$ m is the reference distance, $L_0=-30$ dB is the path loss at reference distance, $d_x$ is the distance of different channels, and ${\alpha }_x$ denotes different channel path exponents. The channel path loss exponents between the BS and STAR RIS, as well as between STAR RIS and users are 2.2 [14]. The channel path loss exponent between the BS and users is 3.76 [23]. Besides, the relaying power is $P_k=0.5P_t$, the convergence tolerance is $ϵ={10}^{-3}$, and the noise power is $\sigma =-90$ dBm. All results are obtained by averaging over 100 random channel realizations. The variances of user channels are set to ${\sigma }_k^2=1$ for the relaying user, ${\sigma }_k^2=0.3$ for the destination users, and ${\sigma }_{m,n}^2=1$.

The following schemes are compared in this section:

• CRS-FE/FM/FT/HE/HM/HT: This refers to the STAR RIS-assisted CRS for six different transmission modes, as we proposed in Section 2.
• CRS-HD/FD: This refers to the existing CRS transmission schemes based on the HD or FD protocols without the assistance of STAR RIS [4,6].
• RSMA-ES: This refers to the existing STAR RIS-assisted RSMA transmission scheme without user relaying [16]. Here we only consider the ES protocol since it is more general than the MS and TS protocols.
• SDMA-ES: This refers to the existing STAR RIS-assisted SDMA transmission scheme. Again, we only consider the ES protocol.
• NOMA-ES: This refers to the existing STAR RIS assisted NOMA transmission scheme [29] with the ES protocol.
• RSMA: This refers to the conventional one-layer RSMA transmission scheme, which does not involve user relaying or assistance from STAR RIS [2].

4.2. Simulation Results

4.2.1. Comparison Among STAR RIS Operating Protocols

From Figure 3, we observe that the FE/HE mode consistently achieves a higher max-min rate than the FM/HM and FT/HT modes. This is because the ES protocol is more general than the MS and TS protocol, and it ensures that each STAR RIS element has a larger tunable range of the amplitude coefficient. As the number of STAR RIS elements increases, the max-min rate performance gain between FM/HM and FT/HT modes decreases. Surprisingly, when N is larger than 110, the FT/HT mode achieves a higher max-min rate than the FM/HM mode. This indicates that the TS protocol outperforms the MS protocol when the STAR RIS element is large. With an increasing number of STAR RIS elements, the performance improvement gained from having all elements operating within the same transmission/reflection space is outweighed by the performance degradation caused by partial time service.

Due to the superior max-min rate performance of ES compared to MS and TS, in the following simulation results, we only illustrate the proposed STAR RIS-assisted CRS in FE and HE modes for clarity.

4.2.2. Comparison Between FD and HD Relaying Protocols

Figure 4 illustrates the max-min rate performance versus transmit SNR. We compare our proposed CRS-FE/HE with the conventional RSMA scheme and the conventional CRS schemes with HD and FD protocols. Algorithm 3 is employed to optimize the corresponding problems of the proposed schemes. We set $N=50,K=4,N_t=4$. The max-min rate increases with SNR for all transmission schemes. It is noteworthy that when SNR is below 25 dB, the CRS-FE scheme achieves a higher max-min rate than the CRS-HE scheme. Conversely, when the SNR exceeds 25 dB, the CRS-HE scheme outperforms the CRS-FE scheme. This divergence stems from the introduction of self-interference in FD relaying. As the SNR increases, the level of self-interference also rises. This indicates that the HD protocol is preferred in the high SNR regime while the FD protocol is preferred in the low and moderate SNR regimes. For consistently ensuring the optimal max-min rate performance gain, we can employ FD relaying at low and moderate SNR and HD relaying at high SNR.

From Figure 4, it is obvious that with the assistance of STAR RIS and CRS, CRS-FE/HE achieves explicit max-min rate gain over conventional schemes. Specifically, CRS-FE demonstrates an average relative performance gain of 117.1% and 52.4% over conventional RSMA and CRS-FD, respectively. This demonstrates the significant performance enhancement achieved by integrating STAR RIS and CRS.

4.2.3. Comparison Among Different MA Schemes

In Figure 5, we compare various STAR RIS-assisted transmission schemes with the ES protocol when $N=50$, $K=4$, $N_t=4$. The results show that our proposed STAR RIS-aided CRS scheme outperforms other MA schemes. It achieves an average max-min rate gain of 42.6% over the RSMA-ES scheme, 90.9% over the SDMA-ES scheme, and 77.2% over the NOMA-ES scheme. Besides, the RSMA-ES scheme outperforms NOMA-ES and SDMA-ES. By splitting the user message into common and private parts, RSMA facilitates more flexible interference management in all SNR regimes.

4.2.4. Comparison Between the Two Proposed Algorithms

In Figure 6, we compare the two proposed optimization algorithms for the proposed STAR RIS-aided CRS in the ES mode. We set $\mathrm{SNR}=20$ dB, $K=4$, $N_t=4$. In comparison to the proposed AO algorithm, the low-complexity algorithm reduces the average CPU time by 99.2% while incurring only a 9.1% performance loss at $N=64$. This confirms the effectiveness of the proposed low-complexity algorithm. This efficiency is primarily due to the low-complexity algorithm eliminating the need for CVX operations to update ${\mathbf{\Theta }}_r$ and ${\mathbf{\Theta }}_t$, along with the alternative optimization between two subproblems. Instead, CVX is exclusively used for joint power, common rate, and time allocation, leading to a significant reduction in the optimization dimension. The results suggest that if the system can accommodate high time complexity, Algorithm 3 should be employed to achieve near-optimal max-min rate performance. Conversely, if the system is time-sensitive, the low-complexity Algorithm is recommended.

4.2.5. Imperfect CSIT

Figure 7 illustrates the ergodic max-min rate versus the number of STAR RIS elements, where $\mathrm{SNR}=20$ dB, $K=4$, $N_t=4$. The estimation errors ${\mathbf{e}}_{g,k}$ and ${\mathbf{e}}_{h,k}$ follow $\mathcal{CN}(\mathbf{0},\kappa {\sigma }_k^2\mathbf{I})$, where ${\sigma }_k^2$ represents the channel variance and $\kappa$ represents the CSIT error variance ratio. In this work, we set $\kappa =0.2$. As the number of STAR RIS elements increases, the ergodic max-min rate increases accordingly. Our proposed low-complexity algorithm achieves performance close to that of the AO algorithm. This observation is consistent with our findings under perfect CSIT.

5. Conclusions

In this paper, we propose a novel STAR RIS-assisted CRS transmission with six different transmission modes, namely, HE, HM, HT, FE, FM, and FT. With the objective of maximizing the minimum user rate, we then propose a unified SCA-based AO algorithm to optimize the BS active beamforming and the STAR RIS passive beamforming iteratively under the transmit power constraint at the BS and the law of energy conservation at the STAR RIS. Meanwhile, we propose a novel low-complexity resource allocation algorithm that designs the BS active beamforming and the STAR RIS passive beamforming in closed form. Numerical results show that our proposed STAR RIS aid-CRS system achieves superior max-min rate performance gain over the existing CRS schemes and the STAR RIS-aided MA schemes. Furthermore, we show that the FD relaying has better max-min rate performance at low and moderate SNR while the HD relaying is preferred at high SNR. The ES protocol offers superior max-min rate performance compared to the MS and TS protocols, albeit with increased hardware complexity. With a large number of STAR RIS elements, the TS protocol demonstrates a significantly higher max-min rate gain compared to the MS protocol. Our proposed AO algorithm is better suited for systems that can accommodate higher time complexity, while the low-complexity algorithm is preferable for systems that are more time-sensitive.