STAR-RIS-UAV-Aided Coordinated Multipoint Cellular System for Multi-User Networks

Abstract

Different from conventional reconfigurable intelligent surfaces (RIS), simultaneous transmitting and reflecting RIS (STAR-RIS) can reflect and transmit signals to the receiver. In this paper, to serve more ground users and increase deployment flexibility, we investigate an unmanned aerial vehicle (UAV) equipped with STAR-RIS (STAR-RIS-UAV)-aided wireless communications for multi-user networks. Energy splitting (ES) and mode switching (MS) protocols are considered to control the reflection and transmission coefficients of STAR-RIS elements. To maximize the sum rate of the STAR-RIS-UAV-aided coordinated multipoint (CoMP) cellular system for multi-user networks, the corresponding beamforming vectors as well as transmitted and reflected coefficient matrices are optimized. Specifically, instead of adopting the alternating optimization, we design an iteration method to optimize all variables for both the ES and MS protocols at the same time. Simulation results reveal that the STAR-RIS-UAV-aided CoMP system has a much higher sum rate than systems with conventional RIS or without RIS. Furthermore, the proposed structure is more flexible than fixed STAR-RIS and could greatly promote the sum rate.

1. Introduction

Reconfigurable intelligent surfaces (RIS) have attracted attention from both the academic and industry field since it emerged. RIS can solve many existing problems (e.g., high system energy consumption and high hardware costs) in wireless communications by reflecting and controlling the phase of the received signals [1,2,3]. RIS has been seen as a key technology for sixth-generation communication networks (6G). Additionally, it has been adopted in many wireless applications, such as energy-harvesting systems [4,5], physical layer security [6,7], non-orthogonal multiple access (NOMA) techniques [8,9], massive multiple-input multiple-output (MIMO) methods [10], ultra-reliability low-latency communications (URLLC) [11], and so on. As opposed to relay stations, a RIS is a passive device. This means that it consumes little energy and is suitable for use in networks of unmanned aerial vehicles (UAVs) [12,13,14].

A RIS is a two-dimensional (2D) surface that consists of a controller and many low-power passive elements. The controller can adjust phases of the received signals. Then, the element will reflect the signal to the receiver. Thus, the RIS is similar to a relay. However, the RIS has significantly reduced costs and energy consumption. Since there are many advantages of RIS, it has been investigated in many research studies [15,16,17,18,19]. In [15], the authors designed a method of beamforming for a RIS-assisted wireless network and proved that the performance of all users could be greatly improved with the help of the RIS. Artificial noise (AN) and RIS-assisted secure wireless communications were investigated in [16]. In addition, block coordinate descent and majorization–minimization-based methods were proposed to enhance the security of these wireless communications. Physical layer security and energy harvesting were considered at the same time in [17]. Additionally, a novel transmit beamforming method was also proposed to meet the above two requirements. In [18], a RIS was also used to increase the secrecy rate in a directional modulation system. In order to further decrease circuit costs, researchers analyzed the performance of a discrete phase shifter in a RIS-aided system in [19].

However, a RIS is unhelpful when the user is behind the surface. Owing to the rapid development of materials, the simultaneously transmitting and reflecting RIS (STAR-RIS) method was proposed in recent years [20]. Since an incident signal can be reflected or refracted by the STAR-RIS by changing the transmitted and reflected coefficients (TARCs) of the elements, the STAR-RIS method could serve users whether they are in front of or behind the STAR-RIS. There are three operation protocols that can control the transmitted and reflected signals: energy splitting (ES), mode switching (MS), and time switching (TS). Thus, many works have been donated to STAR-RIS in recent years [8,20,21,22,23]. In [20], the performances of three protocols were analyzed. The authors of this paper pproved that the STAR-RIS method could obtain a better sum rate than a conventional RIS. More importantly, when it comes to broadcast and unicast communication, TS and ES, respectively, were found to have the best performance. In [8], to meet the requirements of a more stringent quality-of-service (QoS), STAR-RIS-assisted downlink NOMA networks were investigated. Afterwards, a method of joint optimization for the position and beamforming of STAR-RIS was proposed in [22]. Furthermore, this paper revealed that the optimized position of STAR-RIS could significantly enhance the sum rate.

Recently, to suppress interference between adjacent cellular signals, coordinated multipoint (CoMP) transmission and reception was proposed [24]. Some RIS-aided CoMP works have been conducted [25,26]. However, as far as we know, STAR-RIS-aided CoMP has not been investigated. Furthermore, typically, the configuration of users is not fixed, and it is hard to set all users in front of the RIS. Therefore, we integrate STAR-RIS into a CoMP system to further improve system performance. In addition, when STAR-RIS is deployed on the ground, the line-of-sight (LOS) part of the channel is usually obscured due to obstacles. Thus, in this paper, STAR-RIS is attached to a UAV, and the UAV hovers over a preset position. This ensures that the LOS exists and that the deployment position of STAR-RIS is more flexible. The following is a summary of our work’s significant contributions:

• A UAV equipped with STAR-RIS (STAR-RIS-UAV)-aided CoMP systems for multiple users is proposed in this paper. ES and MS protocols are considered in the system. To achieve high performance, the STAR-RIS-UAV hovers in the middle of two base stations (BS) in the sky. Then, we optimize the beamforming vector and TARCs matrices to maximize the sum rate. In addition, the transmission power for BS and the QoS of users are considered as constraints.
• For the ES protocol, in contrast with conventional alternate optimization, which optimizes one variable every time, the proposed method can optimize all variables in all iterations. Successive convex approximation (SCA) and a penalty function are adopted to create a convex version of this non-convex issue. Then, the optimized results are obtained by updating the penalty factor in subsequent iterations.
• The MS protocol is processed in a manner similar to the ES protocols. The key here is to deal with a binary constraint. We replace this binary constraint with a new penalty function. When the penalty factor increase to infinity, the modified problem is equal to the origin.
• We evaluate the performance by comparing the system sum rate of proposed methods with three schemes: no RIS, conventional RIS, and uniform energy splitting. Through simulation and analysis, we reveal that the proposed system and methods have the highest system sum rate. In addition, the performance of the ES protocol is better than the MS protocol. However, the MS protocol is more easily implemented.

The rest of this paper is arranged as follows. In Section 2, the system model of the STAR-RIS-UAV-aided CoMP cellular system for multi-user networks is presented. Then, in Section 3, two penalty-function-based iterative methods are proposed. The complexity is analyzed in Section 4. In Section 5, simulations are shown to investigate the performance and convergence of our proposed methods. Finally, we draw conclusions in Section 6.

2. System Model

As shown in Figure 1, a CoMP cellular system with two adjacent cells is considered, and a STAR-RIS-UAV is employed to enhance the communications. This system uses CoMP transmission to mitigate inter-cell interference while using the STAR-RIS-UAV to generate a stronger channel gain at the cell edge user (CEU). Thus, this system could improve the performance of the CEU without affecting the performance of the cell center user (CCU). The STAR-RIS-UAV is hovered at the middle of the two cells to divide the whole system into two spaces; the CEU is located in the reflective half, while the CCU is situated in the refractive half. The BS owns M antennas, and the STAR-RIS has N elements. CEU and CCU are both single-antenna users. We, respectively, denote the channels from the BS to the STAR-RIS, CEU, and CCU as ${\mathbf{G}}_k(k=1,2)\in {\mathbb{C}}^{N\times M}$, ${\mathbf{h}}_{k,E}^H\in {\mathbb{C}}^{1\times M}$, and ${\mathbf{h}}_{k,C}^H\in {\mathbb{C}}^{1\times M}$. The channels from the STAR-RIS to the CEU and CCU are given by ${\mathbf{h}}_{R,E}^H\in {\mathbb{C}}^{1\times N}$ and ${\mathbf{h}}_{R,k}^H\in {\mathbb{C}}^{1\times N}$. In addition, perfect CSIs are considered in this model.

For theES protocol, two modes are used by every component of the STAR-RIS. Typically, the energy of the received signal is divided with an energy splitting ratio, ${\beta }_n^t:{\beta }_n^r$, into the transmitted signal energy and the reflected signal energy. Thus, TARC matrices are, respectively, given by

$${\Theta }_t^{\mathrm{ES}}=\mathrm{diag}\left(\sqrt{{\beta }_1^t}{e}^{j{\theta }_1^t},\sqrt{{\beta }_2^t}{e}^{j{\theta }_2^t},\cdots ,\sqrt{{\beta }_N^t}{e}^{j{\theta }_N^t}\right)$$

(1)

and

$${\Theta }_r^{\mathrm{ES}}=\mathrm{diag}\left(\sqrt{{\beta }_1^r}{e}^{j{\theta }_1^r},\sqrt{{\beta }_2^t}{e}^{j{\theta }_2^r},\cdots ,\sqrt{{\beta }_N^t}{e}^{j{\theta }_N^r}\right),$$

(2)

where ${\beta }_n^t,{\beta }_n^r\in [0,1]$, ${\beta }_n^t+{\beta }_n^r=1$, and ${\theta }_n^t,{\theta }_n^r\in [0,2\pi )$$\forall n\in N$. A significant degree of design freedom for communication systems can be accomplished because the TARCs of each device can be optimized.

For the MS protocol, all elements of the STAR-RIS are separated into two categories. One category, which is in the transmission mode, owns $N_t$ elements. The other category has $N_r$ elements and works in the reflection mode, where $N_t+N_r=N$. In this case, the TARC matrices are given by

$${\Theta }_t^{\mathrm{MS}}=\mathrm{diag}\left(\sqrt{{\beta }_1^t}{e}^{j{\theta }_1^t},\sqrt{{\beta }_2^t}{e}^{j{\theta }_2^t},\cdots ,\sqrt{{\beta }_N^t}{e}^{j{\theta }_N^t}\right)$$

(3)

and

$${\Theta }_r^{\mathrm{MS}}=\mathrm{diag}\left(\sqrt{{\beta }_1^r}{e}^{j{\theta }_1^r},\sqrt{{\beta }_2^r}{e}^{j{\theta }_2^r},\cdots ,\sqrt{{\beta }_N^r}{e}^{j{\theta }_N^r}\right),$$

(4)

respectively, where ${\beta }_n^t,{\beta }_n^r\in \{0,1\}$, ${\beta }_n^t+{\beta }_n^r=1$, and ${\theta }_n^t,{\theta }_n^r\in [0,2\pi )$, $\forall n\in N$.

The MS protocol could be viewed as a unique instance of the ES protocol, where ${\beta }_n^t=0$ or ${\beta }_n^t=1$. Obviously, MS has lower performance than ES, but it is more popular in practical applications due to its ease of implementation.

Let ${\mathbf{w}}_{k,E}$ and ${\mathbf{w}}_{k,C}$ denote the beamforming vector for the CEU and CCU and let $s_E$ and $s_{k,C}$ denote the information symbol transmitted to the CEU and CCU, respectively. It is assumed that $\mathbb{E}\left\{\right|s_E{|}^2\}=1$ and $\mathbb{E}\left\{\right|s_{k,C}{|}^2\}=1\forall k$. Then, the received signals at the CEU and CCU are, respectively, given by

$$y_E=\sum _{k=1}^2({\mathbf{h}}_{k,E}^H+{\mathbf{h}}_{R,E}^H{\Theta }_t^{\mathrm{ES}/\mathrm{MS}}{\mathbf{G}}_k)({\mathbf{w}}_{k,E}{s}_E+{\mathbf{w}}_{k,C}{s}_{k,C})+n_E$$

(5)

and

$$y_{k,C}=({\mathbf{h}}_{k,C}^H+{\mathbf{h}}_{R,k}^H{\Theta }_r^{\mathrm{ES}/\mathrm{MS}}{\mathbf{G}}_k)({\mathbf{w}}_{k,E}{s}_E+{\mathbf{w}}_{k,C}{s}_{k,C})+n_{k,C},$$

(6)

where $n_E\sim \mathcal{CN}(0,{\sigma }_E^2)$ and $n_{k,C}\sim \mathcal{CN}(0,{\sigma }_{k,C}^2)$ are the additive white Gaussian noises at the CEU and CCU, respectively. Let us define

$$\begin{array}{cc}\hfill & {\mathbf{v}}_l={[\sqrt{{\beta }_1^l}{e}^{j{\theta }_1^l},\sqrt{{\beta }_2^l}{e}^{j{\theta }_2^l},\cdots ,\sqrt{{\beta }_N^l}{e}^{j{\theta }_N^l}]}^H\in {\mathbb{C}}^{N\times 1},l\in t,r,{\mathbf{u}}_l=[{\mathbf{v}}_l;1]\hfill \\ \hfill & {\mathbf{F}}_{R,k}=\mathrm{diag}\left\{{\mathbf{h}}_{R,k}^H\right\},{\mathbf{G}}_k\in {\mathbb{C}}^{N\times M},{\mathbf{F}}_k=[{\mathbf{F}}_{R,k};{\mathbf{h}}_{k,C}^H],\hfill \\ \hfill & {\mathbf{H}}_{R,k}=\mathrm{diag}\left\{{\mathbf{h}}_{R,E}^H\right\},{\mathbf{G}}_k\in {\mathbb{C}}^{N\times M},{\mathbf{H}}_R=[{\mathbf{H}}_{R,1},{\mathbf{H}}_{R,2}],\hfill \\ \hfill & {\mathbf{h}}_E={[{\mathbf{h}}_{1,E}^H,{\mathbf{h}}_{2,E}^H]}^H,\mathbf{H}=[{\mathbf{H}}_R;{\mathbf{h}}_E^H],\hfill \\ & {\mathbf{w}}_C={[{\mathbf{w}}_{1,C}^H,{\mathbf{w}}_{2,C}^H]}^H,{\mathbf{w}}_E={[{\mathbf{w}}_{1,E}^H,{\mathbf{w}}_{2,E}^H]}^H\hfill \end{array}$$

(7)

for notational simplicity. Therefore, the achievable rate at the CEU and CCU for ES and MS are given by

$$R_E={log}_2(1+\frac{|{\mathbf{u}}_t^H\mathbf{H}{\mathbf{w}}_E{|}^2}{|{\mathbf{u}}_t^H\mathbf{H}{\mathbf{w}}_C{|}^2+{\sigma }_E^2})$$

(8)

and

$$R_{k,C}={log}_2(1+\frac{|{\mathbf{u}}_r^H{\mathbf{F}}_k{\mathbf{w}}_{k,C}{|}^2}{|{\mathbf{u}}_r^H{\mathbf{F}}_k{\mathbf{w}}_{k,E}{|}^2+{\sigma }_{k,C}^2}),$$

(9)

respectively. In this paper, in order to increase the sum rate of the STAR-RIS-assisted CoMP network [20], which is given by

$$\sum _{k=1}^2{R}_{k,C}+R_E,$$

(10)

our goal is to jointly optimize the beamforming vectors and TARCs, subject to the transmit power constraints for the BSs, and the QoS constraint of the users.

3. Proposed Iterative Algorithm Employing a Penalty Function

In this section, two penalty-based iterative algorithms are proposed for the ES and MS protocols. Instead of employing the alternating optimization, we intend to optimize all optimization variables in each iteration. Then, the problem for MS could be solved by similar operations.

3.1. Joint Design of Beamforming Vector and STAR-RIS Coefficient Matrix for ES Protocol

Specifically, for the ES protocol, the problem can be written as

$$\begin{array}{cc}\hfill & \underset{{\mathbf{w}}_{k,C},{\mathbf{w}}_{k,E},{\mathbf{u}}_l}{max}\sum _{k=1}^2{R}_{k,C}+R_E\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \left|\right|{\mathbf{w}}_{k,C}{\left|\right|}^2+\left|\right|{\mathbf{w}}_{k,E}{\left|\right|}^2\le P,\forall k\hfill \end{array}$$

(11b)

$$\begin{array}{cc}\hfill & R_{k,C}\ge {\overline{R}}_{k,C},\forall k\hfill \end{array}$$

(11c)

$$\begin{array}{cc}\hfill & R_E\ge {\overline{R}}_E\hfill \end{array}$$

(11d)

$$\begin{array}{cc}\hfill & {\left[{\mathbf{u}}_l\right]}_n=\sqrt{{\beta }_n^l}{e}^{j{\theta }_n^l},{\theta }_n^l\in [0,2\pi ),\forall l,n\hfill \end{array}$$

(11e)

$$\begin{array}{cc}\hfill & {\left[{\mathbf{u}}_l\right]}_{N+1}=1\hfill \end{array}$$

(11f)

$$\begin{array}{cc}\hfill & {\beta }_n^l\in [0,1],{\sum }_l{\beta }_n^l=1,\forall l,n\hfill \end{array}$$

(11g)

where P denotes the maximum transmit powers for each BS, and ${\overline{R}}_E$ and ${\overline{R}}_{k,C}$ denote the minimum rate of the CEU and CCU, respectively. For the MS protocol, let us reformulate this equation by replacing ${\beta }_n^l\in [0,1]$ with ${\beta }_n^l\in [0,1]$. Note that it is hard to achieve the globally optimal solution directly for the non-convex problem. Specifically, the constraints from Equations (11c) and (11d) are non-convex for the coupled optimization variables. Furthermore, for the conventional reflecting-only RIS, only one coefficient needs to be designed, while STAR-RIS require the joint optimization of the TARCs, which increases the difficulty of solving the optimization problem.

Firstly, let us deal with the issue of coupled optimization variables. Let us define ${\mathbf{W}}_E={\mathbf{w}}_E{\mathbf{w}}_E^H$, ${\mathbf{W}}_C={\mathbf{w}}_C{\mathbf{w}}_C^H$, ${\mathbf{W}}_{k,E}={\mathbf{w}}_{k,E}{\mathbf{w}}_{k,E}^H$, ${\mathbf{W}}_{k,C}={\mathbf{w}}_{k,C}{\mathbf{w}}_{k,C}^H$. Then, the SINR expressions in Equations (8) and (9) can be equivalently rewritten as

$${\Gamma }_E=\frac{\mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_E{\mathbf{H}}^H\right)}{\mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H\right)+{\sigma }_E^2}$$

(12)

and

$${\Gamma }_{k,C}=\frac{\mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H\right)}{\mathrm{Tr}\left(\mathbf{U}{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H\right)+{\sigma }_{k,C}^2}$$

(13)

By employing Equations (12) and (13), the optimization problem for ES is given by

$$\begin{array}{cc}\hfill & \underset{\genfrac{}{}{0pt}{}{{\mathbf{W}}_{k,C},{\mathbf{W}}_{k,E},}{{\mathbf{W}}_C,{\mathbf{W}}_E,{\mathbf{U}}_l}}{min}-\sum _{k=1}^2{log}_2(1+{\Gamma }_{k,C})-{log}_2(1+{\Gamma }_E)\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{Tr}\left({\mathbf{W}}_{k,C}\right)+\mathrm{Tr}\left({\mathbf{W}}_{k,E}\right)\le P,\forall k\hfill \\ \hfill & {\gamma }_{k,C}\mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H\right)+{\gamma }_{k,C}{\sigma }_{k,C}^2\hfill \end{array}$$

(14b)

$$\begin{array}{cc}\hfill & -\mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H\right)\le 0,\forall k\hfill \\ \hfill & {\gamma }_E\mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H\right)+{\gamma }_E{\sigma }_E^2\hfill \end{array}$$

(14c)

$$\begin{array}{cc}\hfill & -\mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_E{\mathbf{H}}^H\right)\le 0,\hfill \end{array}$$

(14d)

$$\begin{array}{cc}\hfill & 0\le {\left[{\mathbf{U}}_l\right]}_{n,n}\le 1,{\left[{\mathbf{U}}_t\right]}_{n,n}+{\left[{\mathbf{U}}_r\right]}_{n,n}=1,\forall l,n\hfill \end{array}$$

(14e)

$$\begin{array}{cc}\hfill & {\left[{\mathbf{U}}_t\right]}_{N+1,N+1}={\left[{\mathbf{U}}_r\right]}_{N+1,N+1}=1\hfill \end{array}$$

(14f)

$$\begin{array}{cc}\hfill & {\mathbf{W}}_C,{\mathbf{W}}_E⪰0,{\mathbf{W}}_{k,C},{\mathbf{W}}_{k,E}⪰0,\forall k,\hfill \end{array}$$

(14g)

$$\begin{array}{cc}\hfill & {\mathbf{U}}_l⪰0,\forall l\hfill \\ \hfill & \mathrm{rank}\left({\mathbf{W}}_C\right)=1,\mathrm{rank}\left({\mathbf{W}}_C\right)=1,\hfill \end{array}$$

(14h)

$$\begin{array}{cc}\hfill & \mathrm{rank}\left({\mathbf{W}}_{k,C}\right)=1,\mathrm{rank}\left({\mathbf{W}}_{k,E}\right)=1,\forall k\hfill \end{array}$$

(14i)

$$\begin{array}{cc}\hfill & \mathrm{rank}\left({\mathbf{U}}_l\right)=1,\forall l\hfill \end{array}$$

(14j)

where ${\gamma }_E=2^{{\overline{R}}_E-1}$ and ${\gamma }_{k,C}=2^{{\overline{R}}_{k,C}-1}$. Note that, since there are rank-one constraints and the coupling in SINR expressions, Equation (14) is still non-convex. To circumvent this obstacle, we can transform Equation (14a) into the following equivalent form:

$$\sum _{k=1}^2{log}_2{\delta }_k-\sum _{k=1}^2{log}_2{\phi }_k+{log}_2\psi -{log}_2\varphi ,$$

(15)

where ${\delta }_k$, ${\phi }_k$, $\psi$, $\varphi$ are slack optimization variables. Afterwards, new constraints are added according to ${\delta }_k$, ${\phi }_k$, $\psi$, and $\varphi$, which are given by

$${\delta }_k\ge \mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H\right)+{\sigma }_{k,C}^2,$$

(16)

$${\phi }_k\le \mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H\right)+\mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H\right)+{\sigma }_{k,C}^2,$$

(17)

$$\psi \ge \mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H\right)+{\sigma }_E^2,$$

(18)

and

$$\varphi \le \mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H\right)+\mathrm{Tr}\left({\mathbf{U}}_t\mathbf{H}{\mathbf{W}}_E{\mathbf{H}}^H\right)+{\sigma }_E^2,$$

(19)

respectively. It can be seen that Equations (16)–(19) have a similar structure composed of a product of matrices. Then, we recast these equations in the form of the difference of the convex (DC) functions. For example, referring to Lemma 1 in [21], the first two terms in Equation (14c) can be converted into

$$\begin{array}{c}\hfill {\gamma }_{k,C}\mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H\right)=\frac{{\gamma }_{k,C}}{2}\left|\right|{\mathbf{U}}_r+{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2-\frac{{\gamma }_{k,C}}{2}\left|\right|{\mathbf{U}}_r{\left|\right|}_F^2-\frac{{\gamma }_{k,C}}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2\end{array}$$

(20)

$$\begin{array}{c}\hfill -\mathrm{Tr}\left({\mathbf{U}}_r{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H\right)=\frac{1}{2}\left|\right|{\mathbf{U}}_r-{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2-\frac{1}{2}\left|\right|{\mathbf{U}}_r{\left|\right|}_F^2-\frac{1}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2\end{array}$$

(21)

where the terms $-\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2$, $-\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2$ and $-\left|\right|{\mathbf{U}}_r{\left|\right|}_F^2$ are concave with respect to ${\mathbf{W}}_{k,E}$, ${\mathbf{W}}_{k,C}$ and ${\mathbf{U}}_r$, respectively, while the term $\left|\right|{\mathbf{U}}_r+{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2$ is convex with respect to ${\mathbf{W}}_{k,E}$ and ${\mathbf{U}}_r$, and the term $\left|\right|{\mathbf{U}}_r-{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2$ is convex with respect to ${\mathbf{W}}_{k,C}$ and ${\mathbf{U}}_r$. Then, we apply the SCA approach in each iteration to construct a global lower bound on the terms by their first-order Taylor approximations, respectively, which are expressed as

$$\left|\right|{\mathbf{U}}_r{\left|\right|}_F^2\ge -\left|\right|{\mathbf{U}}_r^{\left(t\right)}{\left|\right|}_F^2+2\mathrm{Tr}\left({\left({\mathbf{U}}_r^{\left(t\right)}\right)}^H{\mathbf{U}}_r\right),$$

(22)

$$\begin{array}{c}\hfill \left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2\ge -\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2+2\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,E}\right),\end{array}$$

(23)

and

$$\begin{array}{c}\hfill \left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2\ge -\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,C}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2+2\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,C}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,C}\right),\end{array}$$

(24)

respectively, where ${\mathbf{U}}_r^{\left(t\right)}$, ${\mathbf{W}}_{k,E}^{\left(t\right)}$, and ${\mathbf{W}}_{k,C}^{\left(t\right)}$ are feasible points in the t-th iteration of the SCA. Hence, the non-convex constraint is replaced by

$$\begin{array}{cc}\hfill & \frac{{\gamma }_{k,C}}{2}\left|\right|{\mathbf{U}}_r+{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2-({\gamma }_{k,C}+1)\mathrm{Tr}\left(\left({\mathbf{U}}_r^{\left(t\right)}\right){\mathbf{U}}_r\right)\hfill \\ \hfill & +\frac{{\gamma }_{k,C}+1}{2}\left|\right|{\mathbf{U}}_r^{\left(t\right)}{\left|\right|}_F^2-{\gamma }_{k,C}\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,E}\right)\hfill \\ \hfill & +\frac{{\gamma }_{k,C}}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2+\frac{1}{2}\left|\right|{\mathbf{U}}_r-{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2\hfill \\ \hfill & -\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,C}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,C}\right)+\frac{1}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,C}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2\hfill \\ \hfill & +{\gamma }_{k,C}{\sigma }_{k,C}^2\le 0\hfill \end{array}$$

(25)

Then, the constraint Equation (14d) can be dealt with by a similar method as Equation (14c). Therefore, the non-convex constraint is replaced by

$$\begin{array}{cc}\hfill & \frac{{\gamma }_E}{2}\left|\right|{\mathbf{U}}_t+\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H{\left|\right|}_F^2-({\gamma }_E+1)\mathrm{Tr}\left(\left({\mathbf{U}}_t^{\left(t\right)}\right){\mathbf{U}}_t\right)+\frac{{\gamma }_E+1}{2}\left|\right|{\mathbf{U}}_t^{\left(t\right)}{\left|\right|}_F^2\hfill \\ \hfill & -{\gamma }_E\mathrm{Tr}\left({\left({\mathbf{H}}^H\mathbf{H}{\mathbf{W}}_C^{\left(t\right)}{\mathbf{H}}^H\mathbf{H}\right)}^H{\mathbf{W}}_C\right)+\frac{{\gamma }_E}{2}\left|\right|\mathbf{H}{\mathbf{W}}_C^{\left(t\right)}{\mathbf{H}}^H{\left|\right|}_F^2+\frac{1}{2}\left|\right|{\mathbf{U}}_t-\mathbf{H}{\mathbf{W}}_E{\mathbf{H}}^H{\left|\right|}_F^2\hfill \\ \hfill & -\mathrm{Tr}\left({\left({\mathbf{H}}^H\mathbf{H}{\mathbf{W}}_E^{\left(t\right)}{\mathbf{H}}^H\mathbf{H}\right)}^H{\mathbf{W}}_E\right)+\frac{1}{2}\left|\right|\mathbf{H}{\mathbf{W}}_E^{\left(t\right)}{\mathbf{H}}^H{\left|\right|}_F^2+{\gamma }_E{\sigma }_E^2\le 0\hfill \end{array}$$

(26)

Similarly, the constraints Equations (16)–(19) are replaced by

$$\begin{array}{cc}\hfill & \frac{1}{2}\left|\right|{\mathbf{U}}_r+{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2-\mathrm{Tr}\left(\left({\mathbf{U}}_r^{\left(t\right)}\right){\mathbf{U}}_r\right)+\frac{1}{2}\left|\right|{\mathbf{U}}_r^{\left(t\right)}{\left|\right|}_F^2-\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,E}\right)\hfill \\ \hfill & +\frac{1}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2+{\sigma }_{k,C}^2-{\delta }_k\le 0,\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & \frac{1}{2}\left|\right|{\mathbf{U}}_r-{\mathbf{F}}_k{\mathbf{W}}_{k,C}{\mathbf{F}}_k^H{\left|\right|}_F^2-2\mathrm{Tr}\left(\left({\mathbf{U}}_r^{\left(t\right)}\right){\mathbf{U}}_r\right)+\left|\right|{\mathbf{U}}_r^{\left(t\right)}{\left|\right|}_F^2-\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,C}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,C}\right)\hfill \\ \hfill & +\frac{1}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,C}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2+\frac{1}{2}\left|\right|{\mathbf{U}}_r-{\mathbf{F}}_k{\mathbf{W}}_{k,E}{\mathbf{F}}_k^H{\left|\right|}_F^2-\mathrm{Tr}\left({\left({\mathbf{F}}_k^H{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\mathbf{F}}_k\right)}^H{\mathbf{W}}_{k,E}\right)\hfill \\ \hfill & +\frac{1}{2}\left|\right|{\mathbf{F}}_k{\mathbf{W}}_{k,E}^{\left(t\right)}{\mathbf{F}}_k^H{\left|\right|}_F^2+{\sigma }_{k,C}^2+{\phi }_k\le 0\hfill \end{array},$$

(28)

$$\begin{array}{cc}\hfill & \frac{1}{2}\left|\right|{\mathbf{U}}_t+\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H{\left|\right|}_F^2-\mathrm{Tr}\left(\left({\mathbf{U}}_t^{\left(t\right)}\right){\mathbf{U}}_t\right)+\frac{1}{2}\left|\right|{\mathbf{U}}_t^{\left(t\right)}{\left|\right|}_F^2-\mathrm{Tr}\left({\left({\mathbf{H}}^H\mathbf{H}{\mathbf{W}}_C^{\left(t\right)}{\mathbf{H}}^H\mathbf{H}\right)}^H{\mathbf{W}}_C\right)\hfill \\ \hfill & +\frac{1}{2}\left|\right|\mathbf{H}{\mathbf{W}}_C^{\left(t\right)}{\mathbf{H}}^H{\left|\right|}_F^2+{\sigma }_E^2-\psi \le 0\hfill \end{array},$$

(29)

and

$$\begin{array}{cc}\hfill & \frac{1}{2}\left|\right|{\mathbf{U}}_t-\mathbf{H}{\mathbf{W}}_C{\mathbf{H}}^H{\left|\right|}_F^2-2\mathrm{Tr}\left(\left({\mathbf{U}}_t^{\left(t\right)}\right){\mathbf{U}}_t\right)+\left|\right|{\mathbf{U}}_t^{\left(t\right)}{\left|\right|}_F^2-\mathrm{Tr}\left({\left({\mathbf{H}}^H\mathbf{H}{\mathbf{W}}_C^{\left(t\right)}{\mathbf{H}}^H\mathbf{H}\right)}^H{\mathbf{W}}_C\right)\hfill \\ \hfill & +\frac{1}{2}\left|\right|\mathbf{H}{\mathbf{W}}_C^{\left(t\right)}{\mathbf{H}}^H{\left|\right|}_F^2+\frac{1}{2}\left|\right|{\mathbf{U}}_t-\mathbf{H}{\mathbf{W}}_E{\mathbf{H}}^H{\left|\right|}_F^2-\mathrm{Tr}\left({\left({\mathbf{H}}^H\mathbf{H}{\mathbf{W}}_E^{\left(t\right)}{\mathbf{H}}^H\mathbf{H}\right)}^H{\mathbf{W}}_E\right)\hfill \\ \hfill & +\frac{1}{2}\left|\right|\mathbf{H}{\mathbf{W}}_E^{\left(t\right)}{\mathbf{H}}^H{\left|\right|}_F^2+{\sigma }_E^2+\varphi \le 0\hfill \end{array},$$

(30)

respectively. Next, we address the rank-one constraint, Equation (14j). The constraint Equation (14j) is able to be converted into

$$\left|\right|{\mathbf{U}}_l{\left|\right|}_{*}-\left|\right|{\mathbf{U}}_l{\left|\right|}_2\le 0,\forall l,$$

(31)

where $\left|\right|{\mathbf{U}}_l{\left|\right|}_{*}={\sum }_i{\sigma }_i\left({\mathbf{U}}_l\right)$ denotes the nuclear norm, and $\left|\right|{\mathbf{U}}_l{\left|\right|}_2=\underset{i}{max}{\sigma }_i\left({\mathbf{U}}_l\right)$ is the spectral norm. ${\sigma }_i\left({\mathbf{U}}_l\right)$ is the i-th largest singular value of ${\mathbf{U}}_l$. Since the equality holds if and only if the rank-one constraint is satisfied, $\left|\right|{\mathbf{U}}_l{\left|\right|}_{*}-\left|\right|{\mathbf{U}}_l{\left|\right|}_2\ge 0$ is always satisfied for any ${\mathbf{U}}_l\in {\mathbb{H}}^M,{\mathbf{U}}_l⪰0$. Hence, for rank-one matrices ${\mathbf{U}}_l$, Equation (31) is met. Significantly, the constraint Equation (31) is still non-convex. Then, we adopt the penalty-based method to deal with this issue. By augmenting the constraint Equation (31) into the (14a), the objective function can be written as

$$\begin{array}{c}\hfill \underset{\genfrac{}{}{0pt}{}{{\mathbf{W}}_{k,C},{\mathbf{W}}_{k,E},{\mathbf{W}}_C,}{{\mathbf{W}}_E,{\mathbf{U}}_l,{\delta }_k,{\phi }_k,\psi ,\varphi }}{min}{f}_{\mathrm{CU}}-g_{\mathrm{CU}}+f_{\mathrm{CE}}-g_{\mathrm{CE}}+\chi {\sum }_l\left(\right||{\mathbf{U}}_l{\left|\right|}_{*}-\left|\right|{\mathbf{U}}_l{\left|\right|}_2)\end{array}$$

(32)

where $f_{\mathrm{CU}}={\sum }_{k=1}^2{log}_2{\delta }_k$, $g_{\mathrm{CU}}={\sum }_{k=1}^2{log}_2{\phi }_k$, $f_{\mathrm{CE}}={log}_2\psi$ and $g_{\mathrm{CE}}={log}_2\varphi$. Constraints of (32) are (25)–(30) and (14e)–(14i).

Thus far, the constraint Equation (31) has become a penalty term in Equation (32), and $\chi >0$ denotes a penalty factor for any ${\mathbf{U}}_l$ whose rank is not one. When $\chi \to +\infty$, the problems Equations (14) and (32) are identical. The initial value of $\chi$ should be small because a large $\chi$ could result in a negligible effect of the sum rate in Equation (32). Then, gthe value of $\chi$ is gradually increased to meet the rank-one constraint. However, since Equation (32) is non-convex and there are non-convex constraints, Equation (32) is still non-convex.

Based on the proposed transformations, the objective function is also in the form of DC. To handle the non-convex objective function, for any feasible points ${\delta }_k^{\left(t\right)}$ and ${\psi }^{\left(t\right)}$ in the t-th iteration of SCA, the global lower bounds of $f_{\mathrm{CU}}$ and $f_{\mathrm{CE}}$ can be obtained by using first-order Taylor expansion, as follows:

$$f_{\mathrm{CU}}\ge \sum _{k=1}^2({log}_2{\delta }_k^{\left(t\right)}+\frac{{\delta }_k-{\delta }_k^{\left(t\right)}}{ln2{\delta }_k^{\left(t\right)}})\triangleq {\overline{f}}_{\mathrm{CU}}({\delta }_k,{\delta }_k^{\left(t\right)}),$$

(33)

$$f_{\mathrm{CE}}\ge {log}_2{\psi }^{\left(t\right)}+\frac{\psi -{\psi }^{\left(t\right)}}{ln2{\psi }^{\left(t\right)}}\triangleq {\overline{f}}_{\mathrm{CE}}(\psi ,{\psi }^{\left(t\right)}),$$

(34)

Similarly, for any feasible points ${\mathbf{U}}_l^{\left(t\right)}$ in the t-th iteration, we have

$$\left|\right|{\mathbf{U}}_l{\left|\right|}_{*}-\left|\right|{\mathbf{U}}_l{\left|\right|}_2\le \left|\right|{\mathbf{U}}_l{\left|\right|}_{*}-{\overline{\mathbf{U}}}_l^{\left(t\right)}$$

(35)

where

$${\overline{\mathbf{U}}}_l^{\left(t\right)}\triangleq \left|\right|{\overline{\mathbf{U}}}_l^{\left(t\right)}{\left|\right|}_2+\mathrm{Tr}\left(\mathbf{\lambda }\left({\mathbf{U}}_l^{\left(t\right)}\right){\left(\mathbf{\lambda }\left({\mathbf{U}}_l^{\left(t\right)}\right)\right)}^H({\mathbf{U}}_l-{\mathbf{U}}_l^{\left(t\right)}\right)$$

(36)

$\mathbf{\lambda }\left({\mathbf{U}}_l^{\left(t\right)}\right)$ is the eigenvector corresponding to the maximum eigenvalue of ${\mathbf{U}}_l^{\left(t\right)}$. Then, Equation (32) is replaced by

$$\begin{array}{c}\hfill \underset{\genfrac{}{}{0pt}{}{{\mathbf{W}}_{k,C},{\mathbf{W}}_{k,E},{\mathbf{W}}_C,}{{\mathbf{W}}_E,{\mathbf{U}}_l,{\delta }_k,{\phi }_k,\psi ,\varphi }}{min}{\overline{f}}_{\mathrm{CU}}-g_{\mathrm{CU}}+{\overline{f}}_{\mathrm{CE}}-g_{\mathrm{CE}}+\chi {\sum }_l\left(\right||{\mathbf{U}}_l{\left|\right|}_{*}-{\overline{\mathbf{U}}}_l^{\left(t\right)})\end{array}$$

(37)

Constraints of (37) are (25)–(30) and (14e)–(14i).

Here, the only non-convexity of Equation (37) lies in the rank-one constraint equations, and this constraint can be omitted using semi-definite relaxation (SDR) to solve the relaxation problem. The issue can thus be converted into a standard convex SDP problem, which can be solved directly using CVX [27]. Then, an iterative algorithm based on the penalty function is proposed for solving the problem, which consists of two loops. $\chi$ is gradually increased during each iteration in the outer loop according to $\chi =\omega \chi$, where $\omega >1$. The outer loop will be terminated when the following criterion is met:

$$max\left\{\left|\right|{\mathbf{U}}_l{\left|\right|}_{*}-\left|\right|{\mathbf{U}}_l{\left|\right|}_2,\forall l\in \{t,r\}\right\}\le {\epsilon }_1$$

(38)

where ${\epsilon }_1$ is the threshold set by Equation (31). Therefore, as $\chi$ increases, the constraint Equation (31) will eventually satisfy the accuracy ${\epsilon }_1$. In the inner loop, the variables will be continuously optimized by solving the relaxation problem Equation (37) with a fixed penalty factor. In the inner loop, Equation (37a) is non-increasing, and it has a lower bound. Therefore, the proposed penalty-based iterative method will converge to a stable point when $\chi$ approaches infinity. The details are shown in Algorithm 1.

Algorithm 1: Proposed penalty-based based iterative method for ES protocol.

3.2. Joint Design of Beamforming Vector and STAR-RIS Coefficient Matrix for MS Protocol

Now that we have solved the optimization problem in the ES mode, this subsection proceeds to study the maximum sum rate problem in the MS protocol, where the magnitude coefficients of STAR-RIS can only be chosen to be between 0 and 1. Therefore, the optimization problem on the STAR-RIS coefficients becomes a 0–1 integer programming problem, which can be given by

$$\begin{array}{cc}\hfill & \underset{{\mathbf{w}}_{k,C},{\mathbf{w}}_{k,E},{\mathbf{u}}_l}{max}\sum _{k=1}^2{R}_{k,C}+R_E\hfill \end{array}$$

(39a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \left|\right|{\mathbf{w}}_{k,C}{\left|\right|}^2+\left|\right|{\mathbf{w}}_{k,E}{\left|\right|}^2\le P,\forall k\hfill \end{array}$$

(39b)

$$\begin{array}{cc}\hfill & R_{k,C}\ge {\overline{R}}_{k,C},\forall k\hfill \end{array}$$

(39c)

$$\begin{array}{cc}\hfill & R_E\ge {\overline{R}}_E\hfill \end{array}$$

(39d)

$$\begin{array}{cc}\hfill & {\left[{\mathbf{u}}_l\right]}_n=\sqrt{{\beta }_n^l}{e}^{j{\theta }_n^l},{\theta }_n^l\in [0,2\pi ),\forall l,n\hfill \end{array}$$

(39e)

$$\begin{array}{cc}\hfill & {\left[{\mathbf{u}}_l\right]}_{N+1}=1\hfill \end{array}$$

(39f)

$$\begin{array}{cc}\hfill & {\beta }_n^l\in \{0,1\},\forall l,n\hfill \end{array}$$

(39g)

Compared with the ES protocol, this optimization problem involves an additional non-convex binary constraint, ${\beta }_n^l\in \{0,1\}$, in the MS protocol. Since other non-convex items can be handled in a similar way as in the previous subsection, it is only necessary to focus on how to solve this new binary constraint. This binary constraint can be transferred into an equation constraint as follows:

$${\beta }_n^l-{\left({\beta }_n^l\right)}^2=0,\forall l,n$$

(40)

because ${\beta }_n^l\in \{0,1\}$, ${\beta }_n^l-{\left({\beta }_n^l\right)}^2\ge 0$ is always satisfied and the equality holds if and only if ${\beta }_n^l=0$ or ${\beta }_n^l=1$. Thus, Equation (40) is satisfied when ${\beta }_n^l=0$ or ${\beta }_n^l=1$.

In the following, we will investigate how to extend the penalty-based iterative algorithm proposed in the ES protocol to the optimization problem in the MS protocol. Based on the method for the ES protocol, Equation (40) can be added to the objective function as an additional penalty term. Thus, Equation (39a) can be transformed into

$$\begin{array}{c}\hfill \underset{\genfrac{}{}{0pt}{}{{\mathbf{W}}_{k,C},{\mathbf{W}}_{k,E},{\mathbf{W}}_C,}{{\mathbf{W}}_E,{\mathbf{U}}_l,{\delta }_k,{\phi }_k,\psi ,\varphi }}{min}{\overline{f}}_{\mathrm{CU}}-g_{\mathrm{CU}}+{\overline{f}}_{\mathrm{CE}}-g_{\mathrm{CE}}+\chi {\sum }_l\left(\right||{\mathbf{U}}_l{\left|\right|}_{*}-{\overline{\mathbf{U}}}_l^{\left(t\right)})+\eta \sum _{n=1}^N{\sum }_l\left({\beta }_n^l-{\left({\beta }_n^l\right)}^2\right)\end{array}$$

(41)

where $\eta >0$ denotes a new penalty factor. When $\chi ,\eta \to +\infty$, the solution of Equation (41) is able to satisfy Equations (31) and (40). Constraints of (41) are (25)–(30) and (14e)–(14i). Then, the SCA can still be adopted to solve this non-convex optimization problem. For a given $\left\{{\beta }_l^{\left(t\right)}\right\}$, the upper bound can be achieved using the first-order Taylor approximation,

$$\begin{array}{cc}\hfill {\beta }_n^l-{\left({\beta }_n^l\right)}^2& \ge {\beta }_n^l-{\left({\beta }_n^{l\left(t\right)}\right)}^2-2{\beta }_n^{l\left(t\right)}\left({\beta }_n^l-{\beta }_n^{l\left(t\right)}\right)\hfill \\ \hfill & =\left(1-2{\beta }_n^{l\left(t\right)}\right){\beta }_n^l+{\left({\beta }_n^{l\left(t\right)}\right)}^2\hfill \\ \hfill & \triangleq \mathsf{\Omega }\left({\beta }_n^l,{\beta }_n^{l\left(t\right)}\right),\forall l,n\hfill \end{array}$$

(42)

Substituting Equation (42) into Equation (41), we have

$$\begin{array}{cc}\hfill \underset{\genfrac{}{}{0pt}{}{{\mathbf{W}}_{k,C},{\mathbf{W}}_{k,E},{\mathbf{W}}_C,}{{\mathbf{W}}_E,{\mathbf{U}}_l,{\delta }_k,{\phi }_k,\psi ,\varphi }}{min}& {\overline{f}}_{\mathrm{CU}}-g_{\mathrm{CU}}+{\overline{f}}_{\mathrm{CE}}-g_{\mathrm{CE}}+\chi {\sum }_l\left(\right||{\mathbf{U}}_l{\left|\right|}_{*}-{\overline{\mathbf{U}}}_l^{\left(t\right)})\hfill \\ \hfill & +\eta \sum _{n=1}^N{\sum }_l\mathsf{\Omega }\left({\beta }_n^l,{\beta }_n^{l\left(t\right)}\right)\hfill \end{array}$$

(43)

Constraints of (41) are (25)–(30) and (14e)–(14i).

Then, SDR is used again to remove the non-convex rank-one constraint, and Equation (43) is also convex and is able to be solved directly with CVX [27]. Similar to the ES protocol, the two-level cyclic penalty-based iterative algorithm is able to be employed for the optimization problem in the MS protocol. Since there are two penalty factors in Equation (43), the algorithm terminates when the following criterion is satisfied:

$$max\left\{\begin{array}{c}\left|\right|{\mathbf{U}}_l{\left|\right|}_{*}-\left|\right|{\mathbf{U}}_l{\left|\right|}_2\\ {\beta }_n^l-{\left({\beta }_n^l\right)}^2\end{array}\right\}\ge {\epsilon }_2,\forall l\in \{t,r\}$$

(44)

where ${\epsilon }_2$ is a maximum violation of constraint Equation (31) and (42). The specific steps of the algorithm are summarized as Algorithm 2:

Algorithm 2: Proposed penalty-based iterative method for ES protocol.

4. Computational Complexity

Next, we investigate the computational complexity of the proposed methods. The main complexity lies in solving the SDR after relaxation, and since the relaxation problem is an SDP problem, the computational complexity for each iteration is given by

$${\mathcal{C}}_s=\mathcal{O}\left(K{M}^{6.5}+2N^{6.5}\right)$$

(45)

Therefore, the total computational complexity of the proposed method can be expressed by

$${\mathcal{C}}_p=\mathcal{O}\left(I_{out}{I}_{in}\left(K{M}^{6.5}+2N^{6.5}\right)\right)$$

(46)

where $I_{out}$ and $I_{in}$ are the number of iterations for the inner and outer loops. It can be seen that the computational complexity of this method is a polynomial over N. Thus, even if the number of elements of STAR-RIS is too large, the algorithm still has a relatively low computational complexity. In addition, we can optimize the problem by employing alternating optimization, as many works have [7,15,17]. In each iteration, there are two SDR problems for optimizing ${\mathbf{W}}_{k,C}$ and ${\mathbf{W}}_{k,E}$, respectively. Thus, the computational complexity for each iteration will increase to $\mathcal{O}\left(2K{M}^{6.5}+2N^{6.5}\right)$. The total computational complexity is given by $\mathcal{O}\left(I_{out}{I}_{in}\left(2K{M}^{6.5}+2N^{6.5}\right)\right)$, which is higher than the proposed penalty-based methods. Thus, it is easier to implement the proposed methods in practical applications.

5. Simulation Results and Analysis

In this section, the performance of the STAR-RIS-UAV-aided CoMP system is simulated and verified. Consider a 3D coordinate system, where 2 BSs were located at (0 m, 0 m, 10 m) and (140 m, 0 m, 10 m), respectively, with the radius of each cell set to 80 m. STAR-RIS-UAV was located in the overlapping area of the two cells, set at (70 m, 20 m, 20 m). CCU1 and CCU2 were located in the reflection half of the STAR-RIS and were randomly distributed in the two discs centered on the adjacent BS with inner and outer diameters of 20 m and 40 m, respectively. The CEU was randomly distributed in the overlapping area of the two cells and was located in the transmission half of the area 20 m away from the STAR-RIS. The path loss was calculated as $PL=P{L}_0{(d/d_0)}^{-\alpha }$, where $P{L}_0={10}^{-3}$ was the channel gain when $d_0=1$ m, d denotes the path distance, and $\alpha =2.2$ is the path loss index. In the simulation, all channel models used the Rice fading channel, and the channel is given by

$$\mathit{G}=\sqrt{\frac{\rho }{\rho +1}}{\mathit{G}}^{LOS}+\sqrt{\frac{1}{\rho +1}}{\mathit{G}}^{NLOS}$$

(47)

where $\rho =5$ dB is the Rice coefficient. ${\mathit{G}}^{LOS}$ denotes the LOS component of the channel, obtained by multiplying the transmit and receive direction vectors. As modeled by Rayleigh fading, ${\mathit{G}}^{NLOS}$ is the NLOS component. Given that the maximum transmitting power of the two BSs was the same and the minimum required rate ${\overline{R}}_{k,C}$ and ${\overline{R}}_E$ were equal, ${\overline{R}}_{k,C}={\overline{R}}_E$. The noise variance was set as ${\sigma }_{k,C}^2={\sigma }_E^2=-90$ dBm. In addition, ${\epsilon }_1={\epsilon }_2={10}^{-7}$ and the initial values of the penalty factors $\chi$ and $\eta$ were set as ${10}^{-7}$.

In order to better investigate the advantages of STAR-RIS in the corresponding ES and MS protocols, three comparison schemes were proposed, as follows:

• Conventional RIS: Instead of using STAR-RIS, full coverage was achieved by two reflective-only RISs. The two RISs were adjacent to each other and deployed at the same locations. For a fair comparison, it was assumed that each conventional reflection/transmission RIS had $N/2$ elements; this number was set to be even for simplicity.
• Uniform energy splitting (UES): It was assumed that the TARCs of all components of STAR-RIS in ES mode were equal, ${\beta }_n^t={\overline{\beta }}_n^t$, ${\beta }_n^r={\overline{\beta }}_n^r$, $\forall n$, where $0\le {\overline{\beta }}_n^t,{\overline{\beta }}_n^r\le 1$, ${\overline{\beta }}_n^t+{\overline{\beta }}_n^r=1$. UES can be viewed as a specific example of STAR-RIS with a group/face amplitude design in ES protocol.
• No STAR-RIS: Without the aid of STAR-RIS, the system becomes a traditional collaborative multi-point transmission communication system.

The constraint violations over the number of iterations in the outer loop for ES and MS protocols are shown in Figure 2. From Figure 2, it can be seen that the constraint violations decrease rapidly for both ES and MS as the number of iterations increases and eventually satisfy the predetermined accuracy (i.e., ${\epsilon }_1={\epsilon }_2={10}^{-7}$) after 8 iterations. In other words, the TARC matrices $\left\{{\mathbf{U}}_l\right\}$, as well as the binary TARCs $\left\{{\beta }_n^l\right\}$, were optimized by the penalty-based iterative algorithm after 8 iterations.

Figure 3 shows a graph of the variations of the system sum rate versus the maximum transmit power. It can be seen that the system sum rate of all schemes increases monotonically with increases in the maximum transmit power. In addition, the ES mode obtains a higher system sum rate than the MS and UES schemes because the ES mode has a freer choice of amplitude and phase shift, while the MS and UES schemes have certain restrictions on the TARCs. the amplitude coefficients in MS can only be chosen from 0 and 1, while UES requires equal amplitude coefficients for all components. Therefore, the ES mode allows better adjustment of each element’s phase shift and amplitude coefficients, making full use of the degrees of freedom obtained by each element boost. From the mathematical and optimization point of view, both the MS mode and the UES scheme are a special case of the ES mode. In addition, for conventional RIS schemes, since only a fixed number of RIS elements perform a single reflection or transmission function, while STAR-RIS can flexibly control the number of elements involved in transmission and reflection, STAR-RIS can provide a greater degree of freedom than conventional RIS to enhance the signal strength. In particular, when compared with the ES and MS protocols, conventional RIS schemes suffer some performance loss. In addition, it is obvious that the performance of the STAR-free RIS is the worst among these schemes. Through the effective combination of the STAR-RIS and collaborative multipoint methods, passive beamforming can be reasonably designed with STAR-RIS to enhance the combined channel gain at CEU and eliminate the adverse effects of inter-cell interference, thus further improving the system’s performance.

Figure 4 depicts the variation of the system sum rate over the number of STAR-RIS elements. From this figure, it can be observed that only the system sum rate of the no-STAR-RIS scheme remains constant, while the system sum rate of all the remaining schemes increases with the increase of the number of STAR-RIS elements. This is due to the fact that more signal energy can reach the STAR-RIS when the number of STAR-RIS elements increases, so that the STAR-RIS could use more elements to control the energy and phase of the incident signal and optimize the TARC of each element rationally, thus increasing the system sum rate. Installing more elements also provides more freedom in resource allocation, which facilitates higher beamforming gain and thus higher sum rate. In addition, since the elements are passive, low power consumption and low cost, STAR-RIS with hundreds or even thousands of elements is expected to be used. it can also be seen from the figure that the performance of ES mode and MS is still better than other comparison schemes, once again proving the superiority of STAR-RIS.

Figure 5 plots the system sum rate over the variation in the minimum required rate. As observed in Figure 5, the system sum rate decreases monotonically as the minimum required rate of the user increases. In particular, when the user’s channel conditions are poor, the increase in the minimum required rate causes the system to be forced to allocate more resources to it, reducing the flexibility of the communication system’s resource allocation. In addition, compared with the no-STAR-RIS scheme, the system rate and the decreasing seen in it with increases in the minimum required rate are relatively slow in the scheme with STAR-RIS as well as the conventional RIS assistance. This is because the deployment of RIS can mitigate the effect of interference in the system, and without the assistance of STAR-RIS, the reflected signal cannot accurately reach the subscriber side to promote effective beamforming. The system is more sensitive to minimum rate requirements, and the additional degrees of freedom brought by collaborative multipoint techniques cannot be fully used to solve the inter-cell interference problem. Similar to the results in Figure 4 and Figure 5, the ES and MS modes with STAR-RIS still achieve the best performance, highlighting the importance of STAR-RIS as an aid to collaborative multipoint transmission systems.

Figure 6 shows the variation of the system sum rate versus the number of BS antennas. From the figure, it is clear that when the number of BS antennas rises, the system sum rate grows as well. This is because more BS antennas can obtain higher beamforming gain, which improves the performance of the system. At the same time, the ES and MS modes still outperform the other comparison schemes because STAR-RIS deploys a variable number of transmitting and reflecting elements, making use of the degrees of freedom in the system to enhance the signal strength.

6. Conclusions

In this work, to maximize the system sum rate under the maximum transmit power constraint of the BS as well as the minimum rate constraint of the users, we jointly designed a BS transmit beamforming method and TARCs matrices of the STAR-RIS method for STAR-RIS-UAV-aided CoMP systems. Furthermore, to effectively solve the non-convex optimization problem in both protocols, an iterative algorithm based on a penalty function was proposed for the ES protocol and further extended to the MS protocol. The simulation results proved that STAR-RIS-UAV is able to significantly increase the system sum rate compared with the conventional RIS scheme as well as the no-STAR-RIS scheme. More importantly, our proposed method can also be used when the system serves other UAVs. This conclusion provides useful guidance for STAR-RIS-UAV-aided CoMP systems. More importantly, our methods can also be used in UAV communications when the BS is replaced by a UAV. However, when our structure and method are employed in multi-UAV networks, UAVs will keep moving. Thus, the trajectories of drones may affect the sum rate. To deal with this problem, it is necessary to optimize the trajectories of the drones and the coefficients of STAR-RIS at the same time. This will become our future work. In general, our proposed system and methods are promising for use in practical applications.