Bilayer Intelligent Omni-Surface-Assisted Full-Duplex Systems: Simultaneous Self-Interference Cancellation and Sum Rate Maximization

Abstract

Recently, reconfigurable intelligent surfaces (RISs) have attracted increasing attentions in the design of full-duplex (FD) systems due to their novel capability of propagation environment reconfiguration. However, in conventional RIS-assisted FD systems, the beamforming for self-interference cancellation (SIC) and sum rate maximization (SRM) are highly coupled during RIS optimization, which significantly degrades the system performance. To tackle this issue, we exploit a novel bilayer intelligent omni-surface (BIOS) structure in FD systems. Compared with the conventional RIS designs, the BIOS provides independent beams on both sides, thus enabling more flexible achievement of SRM and SIC. For the BIOS-assisted FD system, we first formulate an optimization problem to achieve SRM and efficient SIC simultaneously. Then, we exploit the relationship between the SRM and mean square error (MSE), and propose a weighted MSE minimization with SIC algorithm to solve the problem. Specifically, we jointly design the beamforming at the base station and the BIOS with manifold optimization while guaranteeing an SIC constraint. Furthermore, we theoretically derive a lower band for the BIOS size required for efficient SIC in FD systems. Simulation results indicate that the BIOS outperforms the conventional RIS designs in FD systems, and verify the accuracy of the derived lower bound for the BIOS size.

1. Introduction

With the explosive growth of wireless communication devices, the scarcity of available spectrum resources has become increasingly acute. Consequently, effectively utilizing existing spectrum resources has emerged as a pivotal factor in advancing wireless communication technology [1,2,3]. Recently, in-band full-duplex (FD) systems have drawn notable interest in the design of next-generation wireless communication systems due to their capability of transmitting and receiving data on the same frequency band concurrently. Compared to traditional half-duplex (HD) systems, such as time division duplex or frequency division duplex, FD systems can significantly enhance spectrum utilization and reduce the time delay of wireless communication systems [4,5]. Additionally, FD systems can significantly improve confidentiality while maintaining the same data rate [4]. Therefore, FD technology is critical for communication in ultra-dense network environments in the future.

Despite the advantages mentioned above, a prominent problem in FD systems is the inevitable self-interference (SI) emanating from the transmitter to the receiver, as its high intensity significantly interferes with uplink (UL) signals. To tackle this issue, three typical categories of SI cancellation (SIC) methods are discussed in the literature: transmission-domain processing, analog-domain processing, and digital-domain signal processing [4]. The transmission-domain SIC scheme involves suppressing SI through hardware design, such as physical isolation, shielding materials, and antennas with different polarization directions, thereby minimizing the power of SI signals before they reach the receiver [6,7,8]. The analog-domain SIC scheme focuses on suppressing SI in the radio frequency (RF) chain by adding isolators or designing circuits to generate signals that counteract the SI signals [9,10,11,12,13]. In [9], the authors added an additional RF chain at the transmitter, connected to the receiver. They estimated the SI channel and used the results to digitally preprocess the transmitted signal, generating a counteracting signal with the same amplitude but opposite phase to the SI, thus canceling it at the receiver. Similarly, in [10], an active RF chain was utilized to generate a signal compensating for the inevitable leakage from passive analog SIC. The digital-domain SIC scheme employs digital signal processing techniques on baseband signals to cancel residual SI, exploiting the spatial degrees of freedom provided by multi-antenna systems. Typically, digital SIC methods involve beamforming techniques such as zero-forcing (ZF), minimum mean square error (MMSE), and least squares [14,15,16,17]. In [14], the authors used ZF precoding at the transmitter and maximum ratio combining at the receiver to effectively cancel SI. In [15], the authors employed the MMSE algorithm to jointly design transmitting and receiving beamforming, aiming to minimize the mean square error (MSE) and reduce the SI. However, despite numerous attempts, these SIC methods come with high hardware expenses, substantial complexity, and constrained performance. Thus, the reconfigurable intelligent surface (RIS) is considered a promising technology and has attracted increasing attention in the design of FD systems. Specifically, RIS is a two-dimensional (2D) metasurface consisting of programmable, low-cost, and passive elements [18,19]. Due to its flexibility and low-cost nature, RIS has been extensively applied in wireless communication systems [20,21,22,23,24].

For the FD systems, RIS has been proven a promising technology to tackle the SIC and thus is widely investigated in the literature [25,26,27,28,29]. In [26], intelligent reflecting surface (IRS), as the most famous type of RIS, was exploited in FD systems to achieve efficient SIC. Specifically, the IRS smartly reconfigured the cascade signal emitted from the base station (BS) transmitter (${\mathrm{BS}}_{\mathrm{Tx}}$) to the IRS and then reflected to the BS receiver (${\mathrm{BS}}_{\mathrm{Rx}}$) to create destructive interference with the SI signal. Moreover, taking into account the quantization noise of analog-to-digital converters (ADCs), the authors of [27] considered a more practical scenario and proposed to jointly design the downlink (DL) precoding matrix and the IRS coefficients. However, due to hardware limitations, the IRS can only reflect the signal back to the same side [30]. Therefore, it can only service user equipment (UE) on the same side of the BS. To tackle this constraint, the authors of [28] first introduced intelligent omni-surface (IOS) to FD systems, which is named simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) in some studies [31]. Compared with the IRS, the IOS can simultaneously reflect and refract the incident signal to both sides [30,31]. Thus, it can extend the wireless coverage to the full space while suppressing the SI signal in FD systems.

Although previous studies have focused on the performance of RIS-assisted FD systems, there remains a notable gap in the literature regarding the simultaneous SIC and sum rate maximization (SRM) of DL and UL. Ref. [28] only addressed SI constraints targeting DL rate optimization, while [25] assumed perfect SIC when maximizing the sum rate (SR). However, minimizing the residual SI is crucial for effective air interface SIC as excessive SI will lead to the saturation of ADCs, preventing it from functioning properly [7]. Thus, it is necessary to investigate beamforming optimization for the SRM under practical SI constraints in RIS-assisted FD systems. Moreover, it is notable that although current IOS models offer both reflection and refraction capabilities, they cannot independently adjust reflecting and refracting phase shifts [30,31,32]. This limitation results in coupled beamforming on both sides, which leads to a serious performance loss in the FD system as the IOS is expected to simultaneously suppress the SI signal and enhance the system data rate.

To deal with these issues, in this paper, we introduce a novel bilayer intelligent omni-surface (BIOS) structure to simultaneously suppress the SI and enhance the system SR in FD systems. This BIOS structure deploys two neighboring IOS layers, and thus can ensure independent beamforming design on both sides [33]. Unlike [33], which utilizes both reflected and refracted beams for communication, our design employs the reflected signals for SIC and the refracted signal for communication, thus achieving both objectives simultaneously. Our approach is grounded in a practical scenario that considers the power constraint of the SI signal. The main contributions are summarized as follows:

• We introduce a novel BIOS in a multi-user FD system. With the unique bilayer structure, the BIOS can provide flexible beamforming on both sides and thus can simultaneously improve the SR and cancel the SI more effectively than the conventional RIS designs.
• For the BIOS-assisted multi-user FD system, we jointly optimize the beamforming at the BS and BIOS to maximize the SR of UL and DL. We ensure effective over-the-air SIC by constraining the residual SI power below a threshold, marking a distinct approach from that of other studies. To solve this problem, we establish the proof of equivalence between the SRM and a weighted MSE minimization (WMMSE) and then propose a novel SIC-WMMSE algorithm.
• To further guide the practical deployment of BIOS-assisted FD systems, we theoretically derive a lower bound of the BIOS size required for efficient SIC, which is proven to be quite close to the accurate size required in practical systems.
• We provide simulation results to show that the BIOS-assisted FD system significantly outperforms that of the conventional IOS due to its capability of independent beamforming on both sides.

Notations: Matrices and vectors are denoted by boldface upper case and boldface lower case, respectively. ${(·)}^T$, ${(·)}^H$, ${(·)}^{-1}$, and ${(·)}^{*}$, respectively, denote the transpose, Hermitian transpose, inverse, and conjugate. ${[·]}_{ij}$ and ${[·]}_i$ denote the $(i,j)$-th element of a matrix and the i-th element of a vector. ${\mathbb{C}}^{M\times N}$ denotes the space of $M\times N$ complex-valued matrices. The trace, $l_1$ norm, and Frobenius norm are denoted by $\mathrm{Tr}(·)$, ${|·|}_1$, and ${\parallel ·\parallel }_F$, respectively. $\mathcal{R}\{·\}$ and $\mathbb{E}(·)$, respectively, denote the real-part and expectation operation. ∘ denotes Hadamard products. Furthermore, ${\mathbf{I}}_N$ denotes the $N\times N$ identity matrix. $\mathrm{diag}\left(\mathbf{a}\right)$ is a diagonal matrix with the elements of $\mathbf{a}$ on its main diagonal and $\mathrm{diag}\left(\mathbf{A}\right)$ represents the extraction of the diagonal of $\mathbf{A}$. $\mathcal{CN}(\mathbf{0},\mathbf{K})$ denotes the circularly symmetric complex Gaussian distribution with zero mean and covariance matrix $\mathbf{K}$.

2. System Model and Problem Formulation

2.1. BIOS Structure

As mentioned in Section 1, hardware limitations prevent IOS from independently controlling the phases of refracted and reflected signals, resulting in symmetric beams on both sides. However, in the proposed FD system, we aim to utilize the reflected signals to cancel SI and for the refracted signals to communicate. Thus, the symmetric beams will degrade the system performance. Specifically, the cascaded reflected signal, which is emitted from the ${\mathrm{BS}}_{\mathrm{Tx}}$ into the IOS and then reflected to the ${\mathrm{BS}}_{\mathrm{Rx}}$, serves exclusively for SIC while the refracted signals aim to maximize SR. Moreover, refraction beamforming must align DL signals with the DL UEs and enhance UL reception, complicating the optimization problem due to increased coupling and non-convexity. Consequently, performance trade-offs are necessary to find an optimal balance.

To mitigate the constraints, we propose a BIOS-assisted FD system as shown in Figure 1, which can achieve simultaneous SIC and SRM by flexibly controlling the reflection and refraction beams on both sides of the BIOS [33,34]. The BIOS consists of two neighboring IOSs (referred to as ${\mathrm{IOS}}_1$ and ${\mathrm{IOS}}_2$). Situated proximately to the BS, ${\mathrm{IOS}}_1$ functions like a traditional IOS, managing both reflection and refraction, while ${\mathrm{IOS}}_2$ is set to the full-transmitting mode. With this arrangement, the reflected cascaded signals can be smartly designed by controlling the ${\mathrm{IOS}}_1$ to suppress SI signals. Furthermore, the SR can be maximized by independently controlling the reflection and refraction beams on both sides of the BIOS. Thus, the BIOS can achieve the dual-function target of SIC and SRM in FD systems better than the conventional RIS designs. Specifically, compared to traditional IRS, BIOS allows the BS to achieve greater service coverage by reflecting and refracting signals into the full space. Moreover, the capability of independent beamforming on both sides enables separate and effective management for SIC and SRM. Compared to single-layer IOS, the addition of ${\mathrm{IOS}}_2$ enables independent beams on both sides, allowing for more flexible adjustment. Additionally, compared to multilayer refractive RIS investigated in the previous studies [35,36], the BIOS is distinguished by its specific dual function of simultaneous and decoupled reflection and refraction, which enhances its performance in FD systems.

Without loss of generality, we assume that both ${\mathrm{IOS}}_1$ and ${\mathrm{IOS}}_2$ are uniform square planar arrays consisting of $M=M_y\times M_z$ elements, where $M_y=M_z$. Thus, the effective reflecting and refracting coefficient matrices of BIOS can be respectively presented as follows:

$${\widehat{\mathbf{\Phi }}}_{\mathrm{r}}=\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{\Phi }}_{\mathbf{1}},{\widehat{\mathbf{\Phi }}}_{\mathrm{t}}=\sqrt{{ϵ}_{\mathrm{t}}{\xi }_{\mathrm{t}}}{\mathbf{\Phi }}_{\mathbf{2}}\mathbf{G}{\mathbf{\Phi }}_{\mathbf{1}},$$

(1)

where ${\mathbf{\Phi }}_{\mathbf{1}}\in {\mathbb{C}}^{M\times M}$ and ${\mathbf{\Phi }}_{\mathbf{2}}\in {\mathbb{C}}^{M\times M}$ denote the diagonal coefficient matrices of ${\mathrm{IOS}}_1$ and ${\mathrm{IOS}}_2$. ${\xi }_{\mathrm{r}}$ and ${\xi }_{\mathrm{t}}$ are the respective fractions of reflection and refraction signal power relative to the total power of the incoming signal for ${\mathrm{IOS}}_1$, satisfying the power constraint ${\xi }_{\mathrm{r}}+{\xi }_{\mathrm{t}}=1$. (Following the assumption in [32], we presume negligible penetration loss for IOS and BIOS. Nevertheless, the penetration loss will be an interesting and important issue that deserves further investigation.) Similarly, ${ϵ}_{\mathrm{r}}$ and ${ϵ}_{\mathrm{t}}$ denote those for ${\mathrm{IOS}}_2$ with ${ϵ}_{\mathrm{r}}+{ϵ}_{\mathrm{t}}=1$. According to the setup, ${ϵ}_{\mathrm{r}}=0$ and ${ϵ}_{\mathrm{t}}=1$. $\mathbf{G}\in {\mathbb{C}}^{M\times M}$ denotes the channel between ${\mathrm{IOS}}_1$ and ${\mathrm{IOS}}_2$.

2.2. System Model

Consider a narrowband multi-input single-output (MISO) FD system comprising a BS, a BIOS, $K_{\mathrm{D}}$ DL UEs, and $K_{\mathrm{U}}$ UL UEs in Figure 1. We assume that all the UEs are single-antenna, ${\mathrm{BS}}_{\mathrm{Tx}}$ employs $N_{\mathrm{T}}$ transmit antennas, and ${\mathrm{BS}}_{\mathrm{Rx}}$ has $N_{\mathrm{R}}$ receive antennas. The BIOS is deployed significantly close to the BS to ensure the reflected signals are strong enough to efficiently suppress the SI signals, so we assume that there is no line-of-sight (LoS) path between the BS and UEs [28].

For this FD system, the DL signal from the BS is denoted as ${\mathbf{s}}_{\mathrm{D}}=\mathbf{F}{\mathbf{x}}_{\mathrm{D}}$ with $\mathbb{E}\left\{\right|{\mathbf{s}}_{\mathrm{D}}{|}^2\}=P_{\mathrm{s}}$. Similarly, $\mathbf{F}=[{\mathbf{f}}_1,\cdots ,{\mathbf{f}}_{K_{\mathrm{D}}}]\in {\mathbb{C}}^{N_{\mathrm{T}}\times K_{\mathrm{D}}}$ denotes the precoder at the BS, ${\mathbf{x}}_{\mathrm{D}}=\left[x_{\mathrm{D},1},\cdots ,\right.{\left.x_{\mathrm{D},K_{\mathrm{D}}}\right]}^T\in {\mathbb{C}}^{K_{\mathrm{D}}\times 1}$ denotes the original symbol vector, and $x_{\mathrm{U},l}$ is the symbol transmitted by the l-th UL UE with the power constraint $\mathbb{E}\left\{\right|x_{\mathrm{U},l}{|}^2\}=P_{\mathrm{U}}$. The channels between the ${\mathrm{BS}}_{\mathrm{Tx}}$-${\mathrm{IOS}}_1$, ${\mathrm{BS}}_{\mathrm{Tx}}$-${\mathrm{BS}}_{\mathrm{Rx}}$, ${\mathrm{BS}}_{\mathrm{Rx}}$-${\mathrm{IOS}}_1$, ${\mathrm{IOS}}_2$-${\mathrm{UE}}_{\mathrm{D},k}$, and ${\mathrm{IOS}}_2$-${\mathrm{UE}}_{\mathrm{U},l}$ are denoted, respectively, as ${\mathbf{H}}_{\mathrm{TB}}\in {\mathbb{C}}^{M\times N_{\mathrm{T}}}$, ${\mathbf{H}}_{\mathrm{TR}}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{T}}}$, ${\mathbf{H}}_{\mathrm{BR}}\in {\mathbb{C}}^{M\times N_{\mathrm{R}}}$, ${\mathbf{h}}_{\mathrm{D},k}\in {\mathbb{C}}^{M\times 1}$, and ${\mathbf{h}}_{\mathrm{U},l}\in {\mathbb{C}}^{M\times 1}$. Therefore, the DL received signal at the k-th DL UE and the UL received signal at the ${\mathrm{BS}}_{\mathrm{Rx}}$ can be respectively represented as follows:   

$$y_{\mathrm{D},k}=\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{h}}_{\mathrm{D},k}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{f}}_k{x}_{\mathrm{D},k}+\underset{n_{\mathrm{D},k}^{\mathrm{MU}}}{\underbrace{\sum _{i=1,i\ne k}^{K_{\mathrm{D}}}\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{h}}_{\mathrm{D},k}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{f}}_i{x}_{\mathrm{D},i}}}+n_{\mathrm{D},k},$$

(2)

$$\begin{array}{cc}\hfill {\mathbf{y}}_{\mathrm{U}}& =\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{G}}^H{\mathbf{\Phi }}_2{\mathbf{h}}_{\mathrm{U},l}{x}_{\mathrm{U},l}+\underset{{\mathbf{n}}_{\mathrm{U},l}^{\mathrm{MU}}}{\underbrace{\sum _{i=1,i\ne l}^{K_{\mathrm{U}}}\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{G}}^H{\mathbf{\Phi }}_2{\mathbf{h}}_{\mathrm{U},i}{x}_{\mathrm{U},i}}}\hfill \\ \hfill & +\underset{{\mathbf{n}}^{\mathrm{SI}}}{\underbrace{\left({\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\right)\sum _{i=1}^{K_{\mathrm{D}}}{\mathbf{f}}_i{x}_{\mathrm{D},i}}}+{\mathbf{n}}_{\mathrm{U}},\hfill \end{array}$$

(3)

where $n_{\mathrm{D},k}^{\mathrm{MU}}$ represents the interference from other DL UEs for the k-th DL UE and ${\mathbf{n}}_{\mathrm{U},l}^{\mathrm{MU}}$ represents the interference from other UL UEs for the l-th UL UE. ${\mathbf{n}}^{\mathrm{SI}}$ is the SI at the BS. $n_{\mathrm{D},k}$ denotes the Gaussian noise at the k-th DL UE with mean zero and variance ${\sigma }_{\mathrm{D}}^2$, and ${\mathbf{n}}_{\mathrm{U}}$ denotes the noise at the BS satisfying the distribution of ${\mathbf{n}}_{\mathrm{U}}\sim \mathcal{CN}\left(\mathbf{0},{\sigma }_{\mathrm{U}}^2\mathbf{I}\right)$.

2.3. Channel Model

Before problem formulation, we first present detailed channel modeling. As the BIOS is deployed at the BS to achieve efficient SIC, ${\mathbf{H}}_{\mathrm{TB}}$, ${\mathbf{H}}_{\mathrm{TR}}$, and  ${\mathbf{H}}_{\mathrm{BR}}$ are considered near-field channels without non-line-of-sight (NLoS) paths. Thus, considering the properties of spherical waves, we model the LoS path gain from the n-th receive antenna of ${\mathrm{BS}}_{\mathrm{Tx}}$ to the m-th element of ${\mathrm{IOS}}_1$ as follows [37]:

$${\left[{\mathbf{H}}_{\mathrm{TB}}\right]}_{m,n}=\sqrt{\frac{a^2{G}_{\mathrm{TB}}{F}_{\mathrm{TB}}\left({\theta }_{m,n}^r,{\varphi }_{m,n}^r\right)}{4\pi d_{m,n}^2}}exp\left(\frac{-\mathrm{j}2\pi d_{m,n}}{\lambda }\right),$$

(4)

where a is the size of scattering elements, $\lambda$ is the carrier wavelength, and $d_{m,n}$ is the distance between the n-th antenna of ${\mathrm{BS}}_{\mathrm{Tx}}$ and the m-th element of ${\mathrm{IOS}}_1$. ${\theta }_{m,n}^r$ and ${\varphi }_{m,n}^r$, respectively, denote the elevation and azimuth angle of the received signal at the m-th element of ${\mathrm{IOS}}_1$. Moreover, $G_{\mathrm{TB}}=2$ and $F_{\mathrm{TB}}\left(\theta ,\varphi \right)=\left|cos\theta \right|$, respectively, denote parameters related to the equivalent antenna gain and the normalized power radiation pattern between ${\mathrm{BS}}_{\mathrm{Tx}}$ and ${\mathrm{IOS}}_1$.

Similarly, the channel gain of ${\mathbf{H}}_{\mathrm{BR}}$, ${\mathbf{H}}_{\mathrm{TR}}$, and $\mathbf{G}$ can be modeled as follows:

$${\left[{\mathbf{H}}_{\mathrm{BR}}\right]}_{m,n}=\sqrt{\frac{a^2{G}_{\mathrm{BR}}{F}_{\mathrm{BR}}\left({\theta }_{m,n}^{\mathrm{r}},{\varphi }_{m,n}^{\mathrm{r}}\right)}{4\pi d_{m,n}^2}}exp\left(\frac{-\mathrm{j}2\pi d_{m,n}}{\lambda }\right),$$

(5)

$${\left[{\mathbf{H}}_{\mathrm{TR}}\right]}_{m,n}=\sqrt{\frac{a^2}{4\pi d_{m,n}^2}}exp\left(\frac{-\mathrm{j}2\pi d_{m,n}}{\lambda }\right),$$

(6)

$${\left[\mathbf{G}\right]}_{m,n}=\sqrt{\frac{a^2{G}_{\mathrm{II}}{F}_{\mathrm{II}}\left({\theta }_{m,n}^{\mathrm{r}},{\varphi }_{m,n}^{\mathrm{r}}\right)}{4\pi d_{m,n}^2}}exp\left(\frac{-\mathrm{j}2\pi d_{m,n}}{\lambda }\right),$$

(7)

where $G_{\mathrm{BR}}=2$, $G_{\mathrm{II}}=1$, $F_{\mathrm{BR}}(\theta ,\varphi )=|cos\theta |$, and $F_{\mathrm{II}}(\theta ,\varphi )={|cos\theta |}^2$.

Considering the NLoS paths, ${\mathbf{h}}_{\mathrm{D},k}$ and ${\mathbf{h}}_{\mathrm{U},l}$ can be modeled as Rician channels. Specifically, the channel gain between the k-th DL UE and m-th element of ${\mathrm{IOS}}_2$ can be modeled as follows:

$${\left[{\mathbf{h}}_{\mathrm{D},k}\right]}_m=\sqrt{\frac{\kappa }{1+\kappa }}{h}_m^{\mathrm{LoS}}+\sqrt{\frac{1}{1+\kappa }}{h}_m^{\mathrm{NLoS}},$$

(8)

$$h_m^{\mathrm{LoS}}=\sqrt{{\alpha }_0{\left(\frac{d_m}{d_0}\right)}^{-\beta }{G}_{\mathrm{UE}}{F}_{\mathrm{UE}}\left({\theta }_m^{\mathrm{t}},{\varphi }_m^{\mathrm{t}}\right)}exp\left(\frac{-\mathrm{j}2\pi d_m}{\lambda }\right),$$

(9)

$$h_m^{\mathrm{NLoS}}=\sqrt{{\alpha }_0{\left(\frac{d_m}{d_0}\right)}^{-\beta }{G}_{\mathrm{UE}}{F}_{\mathrm{UE}}\left({\theta }_m^{\mathrm{t}},{\varphi }_m^{\mathrm{t}}\right)}{h}_m^{\mathrm{s}},$$

(10)

where $\kappa$ is the Rician factor. $h_m^{\mathrm{LoS}}$ and $h_m^{\mathrm{NLoS}}$, respectively, denote the LoS and NLoS path gain from the m-th element of ${\mathrm{IOS}}_2$ to the k-th DL UE. $d_m$ denotes the distance between the m-th element of ${\mathrm{IOS}}_2$ and the k-th DL UE. ${\alpha }_0$ denotes the pass loss of the reference distance $d_0$ with the path loss exponent $\beta$. $h_{m_1,n}^{\mathrm{s}}\sim \mathcal{CN}(0,1)$ denotes the cumulative effect of a large number of scattered paths. Moreover, $G_{\mathrm{UE}}=2$, $F_{\mathrm{UE}}(\theta ,\varphi )=|cos\theta |$. Similarly, ${\mathbf{h}}_{\mathrm{U},l}$ can be modeled in the same way.

2.4. Problem Formulation

According to (2) and (3), the data rate of the k-th DL UE can be expressed as

$$R_{\mathrm{D},k}={log}_2\left(1+\frac{{\left|\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{h}}_{\mathrm{D},k}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{f}}_k\right|}^2}{{\sum }_{\begin{array}{c}i=1,i\ne k\end{array}}^{K_{\mathrm{D}}}{\left|\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{h}}_{\mathrm{D},k}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{f}}_i\right|}^2+{\sigma }_{\mathrm{D}}^2}\right).$$

(11)

Similarly, the UL data rate of the l-th UL UE is given by

$$R_{\mathrm{U},l}={log}_2\left|\mathbf{I}+\frac{P_{\mathrm{U}}\widehat{\mathbf{H}}{\mathbf{h}}_{\mathrm{U},l}{\mathbf{h}}_{\mathrm{U},l}^H{\widehat{\mathbf{H}}}^H}{{\displaystyle \sum _{\begin{array}{c}j=1,j\ne l\end{array}}^{K_{\mathrm{U}}}}{P}_{\mathrm{U}}\widehat{\mathbf{H}}{\mathbf{h}}_{\mathrm{U},j}{\mathbf{h}}_{\mathrm{U},j}^H{\widehat{\mathbf{H}}}^H+{\displaystyle \sum _{i=1}^{K_{\mathrm{D}}}}{\mathbf{H}}_{\mathrm{SI}}{\mathbf{f}}_i{\mathbf{f}}_i^H{\mathbf{H}}_{\mathrm{SI}}^H+{\sigma }_{\mathrm{U}}^2\mathbf{I}}\right|,$$

(12)

where $\widehat{\mathbf{H}}=\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{G}}^H{\mathbf{\Phi }}_2$, ${\mathbf{H}}_{\mathrm{SI}}={\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}$.

In this paper, we jointly optimize the beamforming design at the BS and BIOS to maximize the SR of UL and DL while effectively managing SIC. It is worth noting that excessive residual SI will cause the saturation of ADC, preventing it from functioning properly. However, most of the existing works are based on perfect SIC. To tackle this issue, we take into account the hardware limitations and then impose a constraint to maintain the residual SI power below a threshold, which is different from other papers. Specifically, the problem can be formulated as follows:

$$\begin{array}{cc}\hfill \underset{{\mathbf{\Phi }}_1,{\mathbf{\Phi }}_2,\mathbf{F}}{max}& \sum _{l=1}^{K_{\mathrm{U}}}{R}_{\mathrm{U},l}+\sum _{k=1}^{K_{\mathrm{D}}}{R}_{\mathrm{D},k}\hfill \end{array}$$

(13a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {∥{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}∥}_F^2/N_{\mathrm{R}}\le \gamma ,\hfill \end{array}$$

(13b)

$$\begin{array}{cc}\hfill & {\left|{\mathbf{\Phi }}_1\right|}_{ii}={\left|{\mathbf{\Phi }}_2\right|}_{jj}=1,\forall i,j,\hfill \end{array}$$

(13c)

$$\begin{array}{cc}\hfill & Tr\left(\mathbf{F}{\mathbf{F}}^H\right)\le P_{\mathrm{s}}.\hfill \end{array}$$

(13d)

Constraint (13b) maintains the residual SI power below a threshold to achieve efficient SIC. Constraints (13c) and (13d) result from the constant modulus constraint of the BIOS scattering elements and the power limitation at the BS, respectively.

3. Beamforming Design for BIOS-FD Systems

Problem (13) is difficult to solve directly due to the severely coupled variables and non-convex constraints. Thus, we propose an SIC-WMMSE method to transform the primal problem into an equivalent weighted MSE minimization problem, and then employ alternating optimization (AO) to solve it.

3.1. SIC-WMMSE Approach

Due to the multiple coupled variables and non-convex constraints, problem (13) is difficult to solve directly. However, by leveraging the relationship between MSE and data rate, we convert the primal problem (13) into an equivalent WMMSE problem as follows:

$$\begin{array}{cc}{\displaystyle \underset{\begin{array}{c}{\mathbf{\Phi }}_1,{\mathbf{\Phi }}_2,{\mathbf{F}}_{,}\\ a_{\mathrm{D},k},u_{\mathrm{D},k},a_{\mathrm{U},l},{\mathbf{u}}_{\mathrm{U},l}\end{array}}{min}}& f=\sum _{k=1}^{K_{\mathrm{D}}}\left(a_{\mathrm{D},k}{e}_{\mathrm{D},k}-{log}_2\left|a_{\mathrm{D},k}\right|\right)+\sum _{l=1}^{K_{\mathrm{U}}}\left(a_{\mathrm{U},l}{e}_{\mathrm{U},l}-{log}_2\left|a_{\mathrm{U},l}\right|\right)\hfill \\ \mathrm{s}.\mathrm{t}.& {∥{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}∥}_F^2/N_{\mathrm{R}}\le \gamma ,\hfill \\ & {\left|{\mathbf{\Phi }}_1\right|}_{ii}={\left|{\mathbf{\Phi }}_2\right|}_{jj}=1,\forall i,j,\hfill \\ & Tr\left(\mathbf{F}{\mathbf{F}}^H\right)\le P_{\mathrm{s}},\hfill \end{array}$$

(14)

where $a_{\mathrm{D},k}$ and $a_{\mathrm{U},l}$ are the weighting coefficients for the k-th DL UE and the l-th UL UE, respectively. $u_{\mathrm{D},k}^{*}$ is the receive coefficient at the k-th DL UE and ${\mathbf{u}}_{\mathrm{U},l}^H$ is the receive combiner for the l-th UL UE at the ${\mathrm{BS}}_{\mathrm{Rx}}$, respectively. $e_{\mathrm{D},k}$ and $e_{\mathrm{U},l}$, respectively, denote the MSEs of the k-th DL UE and the l-th UL UE as follows:

$$\begin{array}{cc}\hfill e_{\mathrm{D},k}& =\mathbb{E}\left\{{∥u_{\mathrm{D},k}^{*}{y}_{\mathrm{D},k}-x_{\mathrm{D},k}∥}^2\right\}\hfill \\ \hfill & =\left(u_{\mathrm{D},k}^{*}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H{\mathbf{f}}_k-1\right){\left(u_{\mathrm{D},k}^{*}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H{\mathbf{f}}_k-1\right)}^H+\sum _{\begin{array}{c}i=1\\ i\ne k\end{array}}^{K_{\mathrm{D}}}{u}_{\mathrm{D},k}^{*}{u}_{\mathrm{D},k}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H{\mathbf{f}}_i{\mathbf{f}}_i^H{\widehat{\mathbf{h}}}_{\mathrm{D},k}+{\sigma }_{\mathrm{D},k}^2{u}_{\mathrm{D},k}^{*}{u}_{\mathrm{D},k}\hfill \\ \hfill & =u_{\mathrm{D},k}^{*}{u}_{\mathrm{D},k}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H\mathbf{F}{\mathbf{F}}^H{\widehat{\mathbf{h}}}_{\mathrm{D},k}-2Re\left\{u_{\mathrm{D},k}^{*}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H{\mathbf{f}}_k\right\}+{\sigma }_{\mathrm{D},k}^2{u}_{\mathrm{D},k}^{*}{u}_{\mathrm{D},k}+1,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill e_{\mathrm{U},l}& =\mathbb{E}\left\{{\left|{\mathbf{u}}_{\mathrm{U},l}^H{\mathbf{y}}_{\mathrm{U}}-x_{\mathrm{U},l}\right|}^2\right\}\hfill \\ \hfill & =P_{\mathrm{U}}\left({\mathbf{u}}_{\mathrm{U},l}^H{\widehat{\mathbf{h}}}_{\mathrm{U},l}-1\right){\left({\mathbf{u}}_{\mathrm{U},l}^H{\widehat{\mathbf{h}}}_{\mathrm{U},l}-1\right)}^H+\sum _{\begin{array}{c}i=1\\ i\ne l\end{array}}^{K_{\mathrm{U}}}{P}_{\mathrm{U}}{\mathbf{u}}_{\mathrm{U},l}^H{\widehat{\mathbf{h}}}_{\mathrm{U},i}{\widehat{\mathbf{h}}}_{\mathrm{U},i}^H{\mathbf{u}}_{\mathrm{U},l}+{\mathbf{u}}_{\mathrm{U},l}^H{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H{\mathbf{u}}_{\mathrm{U},l}\hfill \\ \hfill & +{\sigma }_{\mathrm{U}}^2{\mathbf{u}}_{\mathrm{U},l}^H\mathbf{I}{\mathbf{u}}_{\mathrm{U},l}\hfill \\ \hfill & ={\mathbf{u}}_{\mathrm{U},l}^H\left(P_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}^H+{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H+{\sigma }_{\mathrm{U}}^2\mathbf{I}\right){\mathbf{u}}_{\mathrm{U},l}-2\sqrt{P_{\mathrm{U}}}Re\left\{{\mathbf{u}}_{\mathrm{U},l}^H{\widehat{\mathbf{l}}}_{\mathrm{U},l}\right\}+1,\hfill \end{array}$$

(16)

where ${\widehat{\mathbf{h}}}_{\mathrm{D},k}^H=\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{h}}_{\mathrm{D},k}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}$ represents the equivalent channel of the k-th DL UE for simplification and ${\widehat{\mathbf{H}}}_{\mathrm{D}}^H={\left[{\widehat{\mathbf{h}}}_{\mathrm{D},1},\cdots ,{\widehat{\mathbf{h}}}_{\mathrm{D},K_{\mathrm{D}}}\right]}^H$. Similarly, ${\widehat{\mathbf{h}}}_{\mathrm{U},l}=\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{G}}^H{\mathbf{\Phi }}_2{\mathbf{h}}_{\mathrm{U},l}$ represents the equivalent channel of the l-th UL UE for simplification and ${\widehat{\mathbf{H}}}_{\mathrm{U}}=\left[{\widehat{\mathbf{h}}}_{\mathrm{U},1},\cdots ,{\widehat{\mathbf{h}}}_{\mathrm{U},K_{\mathrm{U}}}\right]$. The proof of the equivalence between problems (13) and (14) can be found in Appendix A. For a more concise representation, problem (14) can be rewritten as follows by transforming the scalars into matrices:

$$\begin{array}{cc}{\displaystyle \underset{{\mathbf{\Phi }}_1,{\mathbf{\Phi }}_2,\mathbf{F},{\mathbf{U}}_{\mathrm{D}},{\mathbf{A}}_{\mathrm{D}},{\mathbf{U}}_{\mathrm{U}},{\mathbf{A}}_{\mathrm{U}}}{min}}& f=Tr\left({\mathbf{A}}_{\mathrm{D}}{\mathbf{E}}_{\mathrm{D}}\right)-logdet\left({\mathbf{A}}_{\mathrm{D}}\right)+Tr\left({\mathbf{A}}_{\mathrm{U}}{\mathbf{E}}_{\mathrm{U}}\right)-logdet\left({\mathbf{A}}_{\mathrm{U}}\right)\hfill \\ \mathrm{s}.\mathrm{t}.& {∥{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}∥}_F^2/N_{\mathrm{R}}\le \gamma ,\hfill \\ & {\left|{\mathbf{\Phi }}_1\right|}_{ii}={\left|{\mathbf{\Phi }}_2\right|}_{jj}=1,\forall i,j,\hfill \\ & Tr\left(\mathbf{F}{\mathbf{F}}^H\right)\le P_{\mathrm{s}},\hfill \end{array}$$

(17)

where ${\mathbf{E}}_{\mathrm{D}}=diag\left(e_{\mathrm{D},1},\cdots ,e_{\mathrm{D},K_{\mathrm{D}}}\right)$ and ${\mathbf{E}}_{\mathrm{U}}=diag\left(e_{\mathrm{U},1},\cdots ,e_{\mathrm{U},K_{\mathrm{U}}}\right)$, respectively, denote diagonal matrices containing the MSEs of each DL and UL UE. ${\mathbf{A}}_{\mathrm{D}}=diag\left(a_{\mathrm{D},1},\cdots ,a_{\mathrm{D},K_{\mathrm{D}}}\right)$ and ${\mathbf{A}}_{\mathrm{U}}=diag\left(a_{\mathrm{U},1},\cdots ,a_{\mathrm{U},K_{\mathrm{U}}}\right)$, respectively, denote diagonal matrices containing the weighting coefficients of each DL and UL UE. ${\mathbf{U}}_{\mathrm{D}}=diag\left(u_{\mathrm{D},1},\cdots ,u_{\mathrm{D},K_{\mathrm{D}}}\right)$ denotes the diagonal matrix containing the receive coefficients of each DL UE, and ${\mathbf{U}}_{\mathrm{U}}=\left[{\mathbf{u}}_{\mathrm{U},1},\dots ,{\mathbf{u}}_{\mathrm{U},K_{\mathrm{U}}}\right]$ denotes the matrix containing the receive combiners of each UL UE.

Although we have transformed the SRM problem into an equivalent WMMSE problem (17), it is still difficult to solve it directly due to the highly non-convex constraints. Thus, we propose an efficient way to deal with the residual SI power constraint (13b). Specifically, move the SI suppression term into the objective function with a penalty factor $\rho$, resulting in a new optimization problem as follows:

$$\begin{array}{cc}{\displaystyle \underset{{\mathbf{\Phi }}_1,{\mathbf{\Phi }}_2,\mathbf{F},{\mathbf{U}}_{\mathrm{D}},{\mathbf{A}}_{\mathrm{D}},{\mathbf{U}}_{\mathrm{U}},{\mathbf{A}}_{\mathrm{U}}}{min}}& f=Tr\left({\mathbf{A}}_{\mathrm{D}}{\mathbf{E}}_{\mathrm{D}}\right)-logdet\left({\mathbf{A}}_{\mathrm{D}}\right)+Tr\left({\mathbf{A}}_{\mathrm{U}}{\mathbf{E}}_{\mathrm{U}}\right)-logdet\left({\mathbf{A}}_{\mathrm{U}}\right)+\rho \zeta \hfill \\ \mathrm{s}.\mathrm{t}.& {\left|{\mathbf{\Phi }}_1\right|}_{ii}={\left|{\mathbf{\Phi }}_2\right|}_{jj}=1,\forall i,j,\hfill \\ & Tr\left(\mathbf{F}{\mathbf{F}}^H\right)\le P_{\mathrm{s}},\hfill \end{array}$$

(18)

where $\zeta =\frac{1}{N_{\mathrm{R}}}{∥{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}∥}_F^2$ is the power of the SI signal.

In order to facilitate the optimization of each variable, we substitute all variables into objective function (18) to derive a more detailed expression of f, which can be expressed as follows:

$$f=f_{\mathrm{D}}+f_{\mathrm{U}}+f_{\mathrm{SI}},$$

(19)

where

$$\begin{array}{cc}\hfill f_{\mathrm{D}}& =Tr\left({\mathbf{A}}_{\mathrm{D}}{\mathbf{E}}_{\mathrm{D}}\right)-logdet\left({\mathbf{A}}_{\mathrm{D}}\right)\hfill \\ \hfill & =Tr\left({\mathbf{F}}^H{\widehat{\mathbf{H}}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{U}}_{\mathrm{D}}{\widehat{\mathbf{H}}}_{\mathrm{D}}^H\mathbf{F}\right)-2Re\left(Tr\left({\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\widehat{\mathbf{H}}}_{\mathrm{D}}^H\mathbf{F}\right)\right)\hfill \\ \hfill & +Tr\left({\mathbf{A}}_{\mathrm{D}}{\sigma }_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{U}}_{\mathrm{D}}\right)+Tr\left({\mathbf{A}}_{\mathrm{D}}\right)-{log}_2\left|{\mathbf{A}}_{\mathrm{D}}\right|\hfill \\ \hfill & =Tr\left({\xi }_{\mathrm{t}}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{TB}}^H{\mathbf{\Phi }}_1^H{\mathbf{G}}^H{\mathbf{\Phi }}_2^H{\mathbf{H}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{U}}_{\mathrm{D}}{\mathbf{H}}_{\mathrm{D}}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\mathbf{F}\right)+Tr\left({\mathbf{A}}_{\mathrm{D}}{\sigma }_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{U}}_{\mathrm{D}}\right)\hfill \\ \hfill & -2Re\left(Tr\left(\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{H}}_{\mathrm{D}}^H{\mathbf{\Phi }}_2\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\mathbf{F}\right)\right)+Tr\left({\mathbf{A}}_{\mathrm{D}}\right)-{log}_2\left|{\mathbf{A}}_{\mathrm{D}}\right|,\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill f_{\mathrm{U}}& =Tr\left({\mathbf{A}}_{\mathrm{U}}{\mathbf{E}}_{\mathrm{U}}\right)-logdet\left({\mathbf{A}}_{\mathrm{U}}\right)\hfill \\ \hfill & =Tr\left({\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H\left(P_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}^H+{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H+{\sigma }_{\mathrm{U}}^2\mathbf{I}\right)\right)\hfill \\ \hfill & -2Re\left(\sqrt{P_{\mathrm{U}}}Tr\left({\widehat{\mathbf{H}}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H\right)\right)+Tr\left({\mathbf{A}}_{\mathrm{U}}\right)-{log}_2\left|{\mathbf{A}}_{\mathrm{U}}\right|\hfill \\ \hfill & =Tr\left({\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H\left({\xi }_{\mathrm{t}}{P}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{G}}^H{\mathbf{\Phi }}_2{\mathbf{H}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{U}}^H{\mathbf{\Phi }}_2^H\mathbf{G}{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{BR}}\right.\right.\hfill \\ \hfill & \left.\left.+\left({\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\right)\mathbf{F}{\mathbf{F}}^H{\left({\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{RR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\right)}^H+{\sigma }_{\mathrm{U}}^2\mathbf{I}\right)\right)\hfill \\ \hfill & -2Re\left(Tr\left(\sqrt{P_{\mathrm{U}}{\xi }_{\mathrm{t}}}{\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{G}}^H{\mathbf{\Phi }}_2{\mathbf{H}}_{\mathrm{U}}\right)\right)+Tr\left({\mathbf{A}}_{\mathrm{U}}\right)-{log}_2\left|{\mathbf{A}}_{\mathrm{U}}\right|,\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill f_{\mathrm{SI}}& =\frac{\rho }{N_{\mathrm{R}}}{∥{\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}∥}_F^2\hfill \\ \hfill & =\frac{\rho }{N_{\mathrm{R}}}Tr\left(\left({\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\right){\left({\mathbf{H}}_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\right)}^H\right)\hfill \\ \hfill & =\frac{\rho }{N_{\mathrm{R}}}Tr\left({\mathbf{H}}_{\mathrm{TR}}{\mathbf{H}}_{\mathrm{TR}}^H+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{TR}}{\mathbf{H}}_{\mathrm{TB}}^H{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{BR}}\right.\hfill \\ \hfill & \left.+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{H}}_{\mathrm{TR}}^H+{\xi }_{\mathrm{r}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{H}}_{\mathrm{TB}}^H{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{BR}}\right).\hfill \end{array}$$

(22)

3.2. Optimize ${\mathbf{A}}_{\mathrm{D}}$, ${\mathbf{A}}_{\mathrm{U}}$, ${\mathbf{U}}_{\mathrm{D}}$, ${\mathbf{U}}_{\mathrm{U}}$ with Fixed $\mathbf{F}$, ${\mathbf{\Phi }}_1$, ${\mathbf{\Phi }}_2$

According to the WMMSE design approach [28] and the derivation in Appendix A, ${\mathbf{A}}_{\mathrm{D}}$, ${\mathbf{A}}_{\mathrm{U}}$, ${\mathbf{U}}_{\mathrm{D}}$, and ${\mathbf{U}}_{\mathrm{U}}$ all have closed-form solutions when other variables are fixed, which can be expressed as follows:

$${a_{\mathrm{D},k}}^{★}=e_{\mathrm{D},k}^{-1}={\left(u_{\mathrm{D},k}^{*}{u}_{\mathrm{D},k}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H\mathbf{F}{\mathbf{F}}^H{\widehat{\mathbf{h}}}_{\mathrm{D},k}-2Re\left\{u_{\mathrm{D},k}^{*}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H{\mathbf{f}}_k\right\}+{\sigma }_{\mathrm{D},k}^2{u}_{\mathrm{D},k}^{*}{u}_{\mathrm{D},k}+1\right)}^{-1},$$

(23)

$${u_{\mathrm{D},k}}^{★}={\left({\widehat{\mathbf{h}}}_{\mathrm{D},k}^H\mathbf{F}{\mathbf{F}}^H{\widehat{\mathbf{h}}}_{\mathrm{D},k}+{\sigma }_{\mathrm{D},k}^2\right)}^{-1}{\widehat{\mathbf{h}}}_{\mathrm{D},k}^H{\mathbf{f}}_k,$$

(24)

$${a_{\mathrm{U},l}}^{★}=e_{\mathrm{U},l}^{-1}={\left({\mathbf{u}}_{\mathrm{U},l}^H\left(P_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}^H+{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H+{\sigma }_{\mathrm{U}}^2\mathbf{I}\right){\mathbf{u}}_{\mathrm{U},l}-2\sqrt{P_{\mathrm{U}}}Re\left\{{\mathbf{u}}_{\mathrm{U},l}^H{\widehat{\mathbf{h}}}_{\mathrm{U},l}\right\}+1\right)}^{-1},$$

(25)

$${{\mathbf{u}}_{\mathrm{U},l}}^{★}=\sqrt{P_{\mathrm{U}}}{\left(P_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}{\widehat{\mathbf{H}}}_{\mathrm{U}}^H+{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H+{\sigma }_{\mathrm{U}}^2\mathbf{I}\right)}^{-1}{\widehat{\mathbf{h}}}_{\mathrm{U},l}.$$

(26)

3.3. Optimize $\mathbf{F}$ with Fixed ${\mathbf{A}}_{\mathrm{D}}$, ${\mathbf{A}}_{\mathrm{U}}$, ${\mathbf{U}}_{\mathrm{D}}$, ${\mathbf{U}}_{\mathrm{U}}$, ${\mathbf{\Phi }}_1$, ${\mathbf{\Phi }}_2$

When ${\mathbf{A}}_{\mathrm{D}}$, ${\mathbf{A}}_{\mathrm{U}}$, ${\mathbf{U}}_{\mathrm{D}}$, ${\mathbf{U}}_{\mathrm{U}}$, ${\mathbf{\Phi }}_1$, and ${\mathbf{\Phi }}_2$ are fixed, problem (18) is reduced into a sub-problem for the optimization of $\mathbf{F}$ with the power constraint, which can be rewritten as follows by ignoring constant terms:

$$\begin{array}{cc}\hfill \underset{\mathbf{F}}{min}& f\left(\mathbf{F}\right)=Tr\left({\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H{\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H\right)+Tr\left({\mathbf{F}}^H{\widehat{\mathbf{H}}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{U}}_{\mathrm{D}}{\widehat{\mathbf{H}}}_{\mathrm{D}}^H\mathbf{F}\right)\hfill \\ \hfill & -2Re\left(Tr\left({\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\widehat{\mathbf{H}}}_{\mathrm{D}}^H\mathbf{F}\right)\right)+\frac{\rho }{N_{\mathrm{R}}}Tr\left({\mathbf{H}}_{\mathrm{SI}}\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_{\mathrm{SI}}^H\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& Tr\left({\mathbf{F}}^H\mathbf{F}\right)\le P_{\mathrm{s}}.\hfill \end{array}$$

(27)

It can be observed that sub-problem (27) is convex. Thus, with other parameters fixed, the optimal $\mathbf{F}$ can be obtained by utilizing the Karush–Kuhn–Tucker (KKT) conditions as

$${\mathbf{F}}^{★}=\tilde{\mathbf{F}}{\widehat{\mathbf{H}}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}},$$

(28)

where $\tilde{\mathbf{F}}={({\mathbf{H}}_{\mathrm{SI}}^H{\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H{\mathbf{H}}_{\mathrm{SI}}+{\widehat{\mathbf{H}}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{U}}_{\mathrm{D}}{\widehat{\mathbf{H}}}_{\mathrm{D}}^H+\frac{\rho }{N_{\mathrm{R}}}{\mathbf{H}}_{\mathrm{SI}}^H{\mathbf{H}}_{\mathrm{SI}}+\delta \mathbf{I})}^{-1}$. $\delta$ is the KKT coefficient whose optimal solution can be obtained through a one-dimensional search based on the power constraint. The follow two subsections focus on the optimization of BIOS, i.e., ${\mathbf{\Phi }}_1$, ${\mathbf{\Phi }}_2$.

3.4. Optimize ${\mathbf{\Phi }}_1$ with Fixed $\mathbf{F}$, ${\mathbf{\Phi }}_2$, ${\mathbf{A}}_{\mathrm{D}}$, ${\mathbf{A}}_{\mathrm{U}}$, ${\mathbf{U}}_{\mathrm{D}}$, ${\mathbf{U}}_{\mathrm{U}}$

In this subsection, we will optimize ${\mathbf{\Phi }}_1$ by fixing other variables. By further defining ${\mathbf{H}}_{\mathrm{D}}=\left[{\mathbf{h}}_{\mathrm{D},1},\cdots ,{\mathbf{h}}_{\mathrm{D},K_{\mathrm{D}}}\right]$, ${\mathbf{H}}_{\mathrm{U}}=\left[{\mathbf{h}}_{\mathrm{U},1},\cdots ,{\widehat{\mathbf{h}}}_{\mathrm{U},K_{\mathrm{U}}}\right]$, $\overline{\mathbf{H}}={\mathbf{H}}_{\mathrm{U}}^H{\mathbf{\Phi }}_2^H{\mathbf{G}}^H$, $\overline{\mathbf{F}}={\mathbf{H}}_{\mathrm{TB}}\mathbf{F}$, ${\mathbf{H}}_{\mathrm{fle}}={\mathbf{H}}_{\mathrm{BR}}{\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H{\mathbf{H}}_{\mathrm{BR}}^H$, and ${\mathbf{H}}_{\mathrm{fra}}={\mathbf{G}}^H{\mathbf{\Phi }}_2^H{\mathbf{H}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}^H{\mathbf{H}}_{\mathrm{D}}^H{\mathbf{\Phi }}_2\mathbf{G}$, the objective function in (18) related to ${\mathbf{\Phi }}_1$ can be simplified as follows:

$$\begin{array}{cc}\hfill \underset{{\mathbf{\Phi }}_1}{min}{f}_1\left({\mathbf{\Phi }}_1\right)=& 2Re\left(Tr\left({\mathbf{\Phi }}_1^H\left(\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}{\overline{\mathbf{U}}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{TR}}\mathbf{F}{\overline{\mathbf{F}}}^H-\sqrt{{\xi }_{\mathrm{t}}{P}_{\mathrm{U}}}{\mathbf{H}}_{\mathrm{BR}}{\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}\overline{\mathbf{H}}\right.\right.\right.\hfill \\ \hfill & \left.\left.\left.-\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{G}}^H{\mathbf{\Phi }}_2^H{\overline{\mathbf{U}}}_{\mathrm{D}}{\overline{\mathbf{F}}}^H+\frac{\rho }{N_{\mathrm{R}}}\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}{\mathbf{H}}_{\mathrm{TR}}{\mathbf{H}}_{\mathrm{TB}}^H\right)\right)\right)\hfill \\ \hfill & +Tr\left({\xi }_{\mathrm{t}}{P}_{\mathrm{U}}{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{fle}}{\mathbf{\Phi }}_1{\overline{\mathbf{H}}}^H\overline{\mathbf{H}}\right)+Tr\left({\xi }_{\mathrm{r}}{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{fle}}{\mathbf{\Phi }}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H\right)\hfill \\ \hfill & +Tr\left({\xi }_{\mathrm{t}}{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{fra}}{\mathbf{\Phi }}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H\right)+Tr\left(\frac{\rho {\xi }_{\mathrm{r}}}{N_{\mathrm{R}}}{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{BR}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{H}}_{\mathrm{TB}}^H\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\left|{\mathbf{\Phi }}_1\right|}_{ii}=1,\forall i.\hfill \end{array}$$

(29)

Due to the constant modulus constraint of BIOS, sub-problem (29) remains highly non-convex. Thus, we propose to apply the manifold optimization (MO) algorithm to address this challenge. Similar to the gradient descent method, the MO algorithm aims to find the most rapid descent direction on the manifold which also minimally deviates from it, namely, the Riemann conjugate gradient (RCG). It can be derived that the RCG is the projection of the Euclidean gradient in the tangent space, which is the set of directions that least deviate from the manifold surface. Consequently, the key of MO lies in the derivation of the Euclidean gradient. To obtain the gradient of $f_1\left({\mathbf{\Phi }}_1\right)$, we first extract the main diagonal elements of ${\mathbf{\Phi }}_1$ containing useful information to form vector ${\mathit{\varphi }}_1$, i.e., ${\mathit{\varphi }}_1=diag\left({\mathbf{\Phi }}_1\right)$, and transform the objective function $f_1\left({\mathbf{\Phi }}_1\right)$ into $f_1\left({\mathit{\varphi }}_1\right)$ as follows:   

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f_1\left({\mathit{\varphi }}_1\right)=& 2Re\left(Tr\left(\mathbf{A}{\mathit{\varphi }}_1^H\right)\right)+Tr\left({\xi }_{\mathrm{t}}{P}_{\mathrm{U}}{\mathit{\varphi }}_1^H{\mathbf{H}}_{\mathrm{fle}}{\mathit{\varphi }}_1{\overline{\mathbf{H}}}^H\overline{\mathbf{H}}\right)+Tr\left({\xi }_{\mathrm{r}}{\mathit{\varphi }}_1^H{\mathbf{H}}_{\mathrm{fle}}{\mathit{\varphi }}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H\right)\hfill \\ \hfill & +Tr\left({\xi }_{\mathrm{t}}{\mathit{\varphi }}_1^H{\mathbf{H}}_{\mathrm{fra}}{\mathit{\varphi }}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H\right)+Tr\left(\frac{\rho {\xi }_{\mathrm{r}}}{N_{\mathrm{R}}}{\mathit{\varphi }}_1^H{\mathbf{H}}_{\mathrm{BR}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathit{\varphi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{H}}_{\mathrm{TB}}^H\right)\hfill \\ \hfill \stackrel{\left(a\right)}{=}& 2Re\left({\mathit{\varphi }}_1^H\mathbf{a}\right)+{\mathit{\varphi }}_1^H\mathbf{B}{\mathit{\varphi }}_1,\hfill \end{array}\end{array}$$

(30)

where $\mathbf{A}$ and $\mathbf{B}$ are equivalent matrices as

$$\begin{array}{cc}\hfill \mathbf{A}=& \sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}{\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}{\mathbf{U}}_{\mathrm{U}}^H{\mathbf{H}}_{\mathrm{TR}}\mathbf{F}{\overline{\mathbf{F}}}^H-\sqrt{{\xi }_{\mathrm{t}}{P}_{\mathrm{U}}}{\mathbf{H}}_{\mathrm{BR}}{\mathbf{U}}_{\mathrm{U}}{\mathbf{A}}_{\mathrm{U}}\overline{\mathbf{H}}\hfill \\ \hfill & -\sqrt{{\xi }_{\mathrm{t}}}{\mathbf{G}}^H{\mathbf{\Phi }}_2^H{\mathbf{H}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}{\overline{\mathbf{F}}}^H+\frac{\rho }{N_{\mathrm{R}}}\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{H}}_{\mathrm{BR}}{\mathbf{H}}_{\mathrm{TR}}{\mathbf{H}}_{\mathrm{TB}}^H,\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbf{B}=& {\xi }_{\mathrm{t}}{P}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{fle}}\odot {\left({\overline{\mathbf{H}}}^H\overline{\mathbf{H}}\right)}^T+{\xi }_{\mathrm{r}}{\mathbf{H}}_{\mathrm{fle}}\odot {\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H\right)}^T\hfill \\ \hfill & +{\xi }_{\mathrm{t}}{\mathbf{H}}_{\mathrm{fra}}\odot {\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H\right)}^T+\frac{\rho {\xi }_{\mathrm{r}}}{N}\left({\mathbf{H}}_{\mathrm{BR}}{\mathbf{H}}_{\mathrm{BR}}^H\right)\odot {\left({\mathbf{H}}_{\mathrm{TB}}{\mathbf{H}}_{\mathrm{TB}}^H\right)}^T,\hfill \end{array}$$

and equality $\left(a\right)$ holds due to the following properties: $\mathrm{Tr}\left({\mathbf{C}}^H\mathbf{P}\mathbf{C}\mathbf{Q}\right)={\mathbf{c}}^H(\mathbf{P}\odot {\mathbf{Q}}^T)\mathbf{c}$, $\mathrm{Tr}\left(\mathbf{P}\mathbf{C}\right)={\mathbf{p}}^T\mathbf{c}$ for any matrices $\mathbf{P}$, $\mathbf{Q}$, and diagonal matrix $\mathbf{C}$, with $\mathbf{p}=\mathrm{diag}\left(\mathbf{P}\right)$ and $\mathbf{c}=\mathrm{diag}\left(\mathbf{C}\right)$. Thus, the Euclidean gradient of $f_1\left({\mathit{\varphi }}_1\right)$ can be obtained as

$${\nabla }_{{\mathit{\varphi }}_1}{f}_1\left({\mathit{\varphi }}_1\right)=\mathbf{a}+\mathbf{B}{\mathit{\varphi }}_1.$$

(31)

Thus, the RCG can be derived to facilitate gradient descent with an appropriate step size. Subsequently, the optimized ${\mathit{\varphi }}_1$ is projected back onto the manifold surface to ensure the constant modulus constraint. Consequently, the optimal ${\mathit{\varphi }}_1$ can be obtained by iteratively descending along the RCG until a predefined stopping criterion is satisfied.

It is important to notice that the term $Tr\left(\frac{\rho {\xi }_{\mathrm{r}}}{N_{\mathrm{R}}}{\mathbf{\Phi }}_1^H{\mathbf{H}}_{\mathrm{BR}}{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}{\mathbf{H}}_{\mathrm{TB}}^H\right)$ results from the SI power constraint, while the other terms are associated with the SR. Consequently, the optimization of ${\mathbf{\Phi }}_1$ takes into account SRM and SIC simultaneously.

3.5. Optimize ${\mathbf{\Phi }}_2$ with Fixed $\mathbf{F}$, ${\mathbf{\Phi }}_1$, ${\mathbf{A}}_{\mathrm{D}}$, ${\mathbf{A}}_{\mathrm{U}}$, ${\mathbf{U}}_{\mathrm{D}}$, ${\mathbf{U}}_{\mathrm{U}}$

Similarly, we optimize ${\mathbf{\Phi }}_2$ with other variables fixed in this subsection. By defining equivalent matrices ${\overline{\mathbf{H}}}_{\mathrm{BR}}=\sqrt{P_{\mathrm{U}}}{\mathbf{U}}_{\mathrm{U}}^H{\mathbf{H}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1\mathbf{G}$, ${\overline{\mathbf{H}}}_{\mathrm{TB}}=\mathbf{G}{\mathbf{\Phi }}_1{\mathbf{H}}_{\mathrm{TB}}\mathbf{F}$, ${\overline{\mathbf{H}}}_{\mathrm{D}}={\mathbf{H}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}$, the sub-problem related to ${\mathbf{\Phi }}_2$ can be expressed as

$$\begin{array}{cc}\hfill \underset{{\mathbf{\Phi }}_2}{min}{f}_2\left({\mathbf{\Phi }}_2\right)=& -2\sqrt{{\xi }_{\mathrm{t}}}Re\left(Tr\left(\left({\overline{\mathbf{H}}}_{\mathrm{BR}}^H{\mathbf{A}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{U}}^H+{\mathbf{H}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\overline{\mathbf{H}}}_{\mathrm{TB}}^H\right){\mathbf{\Phi }}_2^H\right)\right)\hfill \\ \hfill & +{\xi }_{\mathrm{t}}Tr\left({\mathbf{\Phi }}_2^H{\overline{\mathbf{H}}}_{\mathrm{BR}}^H{\mathbf{A}}_{\mathrm{U}}{\overline{\mathbf{H}}}_{\mathrm{BR}}{\mathbf{\Phi }}_2{\mathbf{H}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{U}}^H\right)\hfill \\ \hfill & +{\xi }_{\mathrm{t}}Tr\left({\mathbf{\Phi }}_2^H{\overline{\mathbf{H}}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\overline{\mathbf{H}}}_{\mathrm{D}}^H{\mathbf{\Phi }}_2{\overline{\mathbf{H}}}_{\mathrm{TB}}{\overline{\mathbf{H}}}_{\mathrm{TB}}^H\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\left|{\mathbf{\Phi }}_2\right|}_{ii}=1,\forall i.\hfill \end{array}$$

(32)

Similarly, we can also employ MO to solve (33). Extract the main diagonal elements of ${\mathbf{\Phi }}_2$ to form vector ${\mathit{\varphi }}_2$; the objective $f_2\left({\mathbf{\Phi }}_2\right)$ can be transformed into $f_2\left({\mathit{\varphi }}_2\right)$ as follows:

$$f_2\left({\mathbf{\Phi }}_2\right)=-2\sqrt{{\xi }_{\mathrm{t}}}Re\left({\mathit{\varphi }}_2^H\mathbf{c}\right)+{\xi }_{\mathrm{t}}{\mathit{\varphi }}_2^H\mathbf{D}{\mathit{\varphi }}_2,$$

(33)

where ${\mathit{\varphi }}_2=diag\left({\mathbf{\Phi }}_2\right)$, and $\mathbf{C}$ and $\mathbf{D}$ are equivalent matrices as

$$\mathbf{C}={\overline{\mathbf{H}}}_{\mathrm{BR}}^H{\mathbf{A}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{U}}^H+{\mathbf{H}}_{\mathrm{D}}{\mathbf{U}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\overline{\mathbf{H}}}_{\mathrm{TB}}^H,$$

(34)

$$\mathbf{D}={\overline{\mathbf{H}}}_{\mathrm{BR}}^H{\mathbf{A}}_{\mathrm{U}}{\overline{\mathbf{H}}}_{\mathrm{BR}}\odot {\left({\mathbf{H}}_{\mathrm{U}}{\mathbf{H}}_{\mathrm{U}}^H\right)}^T+{\overline{\mathbf{H}}}_{\mathrm{D}}{\mathbf{A}}_{\mathrm{D}}{\overline{\mathbf{H}}}_{\mathrm{D}}^H\odot {\left({\overline{\mathbf{H}}}_{\mathrm{TB}}{\overline{\mathbf{H}}}_{\mathrm{TB}}^H\right)}^T.$$

(35)

Thus, the Euclidean gradient of $f_2\left({\mathit{\varphi }}_2\right)$ can be obtained as

$${\nabla }_{{\mathit{\varphi }}_2}{f}_2\left({\mathit{\varphi }}_2\right)=-\sqrt{{\xi }_{\mathrm{t}}}\mathbf{c}+{\xi }_{\mathrm{t}}\mathbf{D}{\mathit{\varphi }}_2.$$

(36)

It is worth noting that, despite the similar form, the optimization of ${\mathbf{\Phi }}_1$ and ${\mathbf{\Phi }}_2$ has different meanings for this system. As indicated in the UL rate expression (12), the SI is tied to ${\mathbf{\Phi }}_1$ while not directly related to the optimization of ${\mathbf{\Phi }}_2$. Therefore, we can ignore the penalty item related to SIC in (19) during the ${\mathbf{\Phi }}_2$ optimization. Furthermore, as the BIOS can independently control the reflection and refraction beams on both sides, the additional freedom enables ${\mathbf{\Phi }}_2$ to focus on the beamforming for SRM without affecting the SIC quality.

The proposed SIC-WMMSE algorithm for solving the sub-problem is summarized in Algorithm 1. It is worth noting that the application of the KKT conditions and the MO algorithm enables a reduction in the objective function for each variable within each iteration. When the decrement in the weighted MSE between two iterations is below a predefined threshold, the iteration is terminated. Since the objective function decreases after each step of updating, Algorithm 1 can converge to a locally optimal solution. To analyze its computational complexity, we count the number of complex multiplications of the key steps in one iteration turn. Assuming that $N_{\mathrm{T}}=N_{\mathrm{T}}=N$, $K_{\mathrm{D}}=K_{\mathrm{U}}=K$, and $M>N>K$, the complexity can be summarized as follows: updating ${\mathbf{F}}^{\left(i\right)}$: $O\left(K{M}^2+N^3\right)$; updating ${{\mathbf{\Phi }}_1}^{\left(i\right)}$: $O\left(M^3\right)$; updating ${{\mathbf{\Phi }}_2}^{\left(i\right)}$: $O\left(N{M}^2\right)$; updating ${{\mathbf{A}}_{\mathrm{D}}}^{\left(i\right)}$: $O\left(K{M}^2\right)$; updating ${{\mathbf{A}}_{\mathrm{U}}}^{\left(i\right)}$: $O\left(K{M}^2+M{N}^2\right)$; updating ${{\mathbf{U}}_{\mathrm{D}}}^{\left(i\right)}$: $O\left(N{K}^2\right)$; updating ${{\mathbf{U}}_{\mathrm{U}}}^{\left(i\right)}$: $O\left(N^3\right)$.

Algorithm 1 Proposed SIC-WMMSE Algorithm

1: Randomly initialize ${\mathbf{\Phi }}_1^{\left(0\right)}$, ${\mathbf{\Phi }}_2^{\left(0\right)}$ and ${\mathbf{F}}^{\left(0\right)}$ and set $i=0$. 2: Calculate ${\mathbf{A}}_{\mathrm{D}}^{\left(0\right)}$, ${\mathbf{A}}_{\mathrm{U}}^{\left(0\right)}$, ${\mathbf{U}}_{\mathrm{D}}^{\left(0\right)}$, and ${\mathbf{U}}_{\mathrm{U}}^{\left(0\right)}$ according to (23)–(26). 3: repeat 4:    Update ${\mathbf{F}}^{\left(i\right)}$ according to (28). 5:    Update ${\mathbf{\Phi }}_1^{\left(i\right)}$, ${\mathbf{\Phi }}_2^{\left(i\right)}$ according to (31), (36). 6:    Update ${\mathbf{U}}_{\mathrm{D}}^{\left(i\right)}$ and ${\mathbf{U}}_{\mathrm{U}}^{\left(i\right)}$ according to (24), (26). 7:    Update ${\mathbf{A}}_{\mathrm{D}}^{\left(i\right)}$, ${\mathbf{A}}_{\mathrm{U}}^{\left(i\right)}$ according to (23), (25). 8:    $i\leftarrow i+1$. 9: until the stopping condition is satisfied.

4. Analysis of BIOS Size

To further guide the practical deployment of BIOS-assisted FD systems, a lower bound of BIOS size required for efficient SIC is derived here for the single-input single-output (SISO) scenario. Specifically, the channels between the ${\mathrm{BS}}_{\mathrm{Tx}}$-${\mathrm{BS}}_{\mathrm{Rx}}$, ${\mathrm{BS}}_{\mathrm{Tx}}$-${\mathrm{IOS}}_1$, and ${\mathrm{BS}}_{\mathrm{Rx}}$-${\mathrm{IOS}}_1$ are denoted as $h_{\mathrm{TR}}$, ${\mathbf{h}}_{\mathrm{TB}}$, and ${\mathbf{h}}_{\mathrm{BR}}$, respectively.

Theorem 1.

To achieve efficient SIC, the number of elements in one layer of the BIOS must exceed a threshold M that satisfies the following inequality:

$$\sum _{m=1}^M{\left(\frac{1}{d_m^{\mathrm{BR}}{d}_m^{\mathrm{TB}}}\right)}^{\frac{3}{2}}\ge \frac{2\sqrt{\pi }}{\sqrt{{\xi }_{\mathrm{r}}}\lambda d_{\mathrm{BB}}{d}_{\mathrm{TR}}},$$

(37)

where $d_m^{\mathrm{BR}},d_m^{\mathrm{TB}}$, respectively, denote the distances from the m-th element of the ${IOS}_1$ to the ${BS}_{\mathrm{Rx}}$ and ${BS}_{\mathrm{Tx}}$, and $d_{\mathrm{BB}}$ and $d_{\mathrm{TR}}$ denote the distance between BS-BIOS and ${BS}_{\mathrm{Rx}}$-${BS}_{\mathrm{Tx}}$.

Proof of Theorem 1.

To ensure effective SIC in the SISO scenario, the residual SI should satisfy the following power constraint:

$$\parallel (h_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{h}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{h}}_{\mathrm{TB}})s_{\mathrm{D}}{\parallel }_F^2\le \gamma .$$

(38)

Notably, for effective SIC, the threshold $\gamma$ should be minimized, and the power constraint (38) should be universally valid for any $s_{\mathrm{D}}$. This requires that the residual interference approaches zero, which can be expressed as

$$\left(h_{\mathrm{TR}}+\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{h}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{h}}_{\mathrm{TB}}\right)\Rightarrow 0.$$

(39)

To clarify, the necessary condition for efficient SIC is that the modulus of the cascaded channel $\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{h}}_{\mathrm{BR}}^H{\mathbf{\Phi }}_1{\mathbf{h}}_{\mathrm{TB}}$ should exceed that of the SI channel $h_{\mathrm{TR}}$, which can be formulated as follows:

$${\left|\sqrt{{\xi }_{\mathrm{r}}}{\mathbf{h}}_{\mathrm{BR}}^H\mathbf{\Phi }{\mathbf{h}}_{\mathrm{TB}}\right|}^2\ge {\left|h_{\mathrm{TR}}\right|}^2.$$

(40)

We define $g\left(M\right)$ as the maximum gain of the cascade channel generated by BIOS with M scattering elements in one layer, which is given by

$$\begin{array}{c}\hfill g\left(M\right)=\underset{\mathbf{\Phi }}{max}{\left|{\xi }_{\mathrm{r}}{\mathbf{h}}_{\mathrm{BR}}^H\mathbf{\Phi }{\mathbf{h}}_{\mathrm{TB}}\right|}^2=\underset{\mathit{\varphi }}{max}{\xi }_{\mathrm{r}}{\left|\sum _{m=1}^M{\left[{\mathbf{h}}_{\mathrm{BR}}^H\right]}_m{\left[\mathit{\varphi }\right]}_m{\left[{\mathbf{h}}_{\mathrm{TB}}\right]}_m\right|}^2\end{array}$$

(41)

It can be observed that maximizing the cascade channel gain occurs when the phase shift element ${\left[\mathit{\varphi }\right]}_m$ is aligned in phase with ${\left[{\mathbf{h}}_{\mathrm{BR}}^H\right]}_m{\left[{\mathbf{h}}_{\mathrm{TB}}\right]}_m$. Thus, by setting ${\left[{\mathit{\varphi }}_1\right]}_m=\frac{{\left[{\mathbf{h}}_{\mathrm{BR}}\right]}_m{\left[{\mathbf{h}}_{\mathrm{TB}}^{*}\right]}_m}{\left|{\left[{\mathbf{h}}_{\mathrm{BR}}^{*}\right]}_m{\left[{\mathbf{h}}_{\mathrm{TB}}\right]}_m\right|}$, $g\left(M\right)$ has an evident closed-form solution as

$$\begin{array}{c}\hfill g\left(M\right)={\xi }_{\mathrm{r}}{∥{\mathbf{h}}_{\mathrm{BR}}\circ {\mathbf{h}}_{\mathrm{TB}}∥}_1^2={\xi }_{\mathrm{r}}{\left(\sum _{m=1}^M\left|{\left[{\mathbf{h}}_{\mathrm{BR}}^{*}\right]}_m{\left[{\mathbf{h}}_{\mathrm{TB}}\right]}_m\right|\right)}^2.\end{array}$$

(42)

By substituting the channel settings, we have

$$g\left(M\right)=\frac{{\lambda }^2{d}_{\mathrm{BB}}}{8\pi }\sum _{m=1}^M{\left(\frac{1}{d_m^{\mathrm{BR}}{d}_m^{\mathrm{TB}}}\right)}^{\frac{3}{2}}.$$

(43)

$$\left|h_{\mathrm{TR}}\right|=\frac{\lambda }{4\sqrt{\pi }{d}_{\mathrm{TR}}}.$$

(44)

According to the derivations mentioned above, the lower bound of the element number can be derived through the inequality constraint

$$\frac{{\lambda }^2{d}_{\mathrm{BB}}}{8\pi }\sum _{m=1}^M{\left(\frac{1}{d_m^{\mathrm{BR}}{d}_m^{\mathrm{TB}}}\right)}^{\frac{3}{2}}\ge \frac{\lambda }{4\sqrt{\pi }{d}_{\mathrm{TR}}}.$$

(45)

The proof is thus completed. □

Theorem 1 provides a lower bound of BIOS size required for SIC, which is of practical importance in the design of FD systems. As will be shown in Section 5, the derived lower bound is rather close to the required BIOS size in the simulation.

5. Simulation Results

This section presents numerical results to illustrate the performance evaluation of the proposed BIOS-assisted FD system. For comparison, an IOS-assisted FD system is employed as a benchmark, which is characterized by identical phase matrices for reflecting and refracting with ${\xi }_{\mathrm{r}}={\xi }_{\mathrm{t}}=1/2$. Throughout simulations, the centers of the BS and BIOS are set at the origin of a three-dimensional coordinate system. The distance between two neighboring IOSs of the BIOS is set as $d_{\mathrm{II}}=0.05$ m. The DL and UL UEs are randomly distributed within two respective circles of 3m radius centered at $(20,-10,0)$ and $(10,10,0)$. Considering the NLoS paths, ${\mathbf{h}}_{\mathrm{D},k}$ and ${\mathbf{h}}_{\mathrm{U},l}$ are modeled as Rician channels with a Rician factor $\kappa =4$. Moreover, ${\alpha }_0={10}^{-4}$ is the pass loss of the reference distance $d_0=1$ m with the path loss exponent $\beta =2.2$. Furthermore, $\lambda =0.05$ m and ${\sigma }_{\mathrm{U}}^2={\sigma }_{\mathrm{D}}^2=-80$ dBm.

Figure 2 shows the SR performance as a function of the BIOS/IOS size for different designs in a multi-user MISO system with $N_{\mathrm{T}}=N_{\mathrm{R}}=3$ and $K_{\mathrm{D}}=K_{\mathrm{U}}=2$. Specifically, the solid lines illustrate the SR of FD systems, and the dashed lines indicate the rate of HD systems, with blue for BIOS assistance and red for IOS assistance, respectively. Meanwhile, as the BIOS may be too small to enable effective SIC, we switch the system from the FD mode to the HD one with only the DL transmission when the SIC constraint cannot be satisfied, which is indicated by the shaded area. It is worth noting that as the channels are randomly generated, the shaded area means that in most of the simulation realizations, the system has to switch to the HD mode as the size of the BIOS/IOS is too small to achieve the SIC threshold. It can be observed that for all systems, the data rate monotonically increases with $M_y$ for both BIOS and IOS due to the enhanced beamforming gains. For FD systems, as $M_y$ increases, the SR of BIOS-assisted systems significantly outperforms that of HD systems. However, the IOS-assisted FD system demonstrates only a slight enhancement over HD systems. This is because the IOS beams for SIC and SRM are tightly coupled, limiting the system’s efficiency. Specifically, if the IOS is designed to cancel the SI signal on the reflecting side, it cannot be efficiently optimized for SRM on the refracting side. Conversely, the BIOS-assisted FD system can provide flexible beamforming on both sides, thereby enhancing communication performance and achieving SIC simultaneously.

To further investigate the beamforming performance of the BIOS and IOS in FD systems, Figure 3 shows the detailed UL and DL rates of the BIOS/IOS-assisted system with different M. As can be observed, the system switches from the FD mode to the HD mode with $M_y<9$. In this range, the data rate, i.e., the DL rate, consistently increases with M. This is because the expansion of the array scale provides higher degrees of freedom, thus enhancing the beamforming gain. However, when $M_y$ changes from 8 to 9, the DL rate unexpectedly decreases while the UL rate substantially increases. This is because the power of the cascaded signal generated by the BIOS/IOS is strong enough to facilitate effective SIC, thus leading to a trade-off between the SRM and SIC for beamforming. It is worth noting that, compared with the IOS-assisted system, the DL rate in the BIOS-assisted system is less affected due to the independent beamforming on both sides. Consequently, the SR performance of the BIOS-assisted FD system significantly surpasses that of the IOS-assisted system.

Furthermore, Figure 4 shows the performance comparison between the proposed BIOS and an IOS with the same scattering elements. As the element spacing is kept unchanged, the IOS with $2M$ elements has a larger aperture size, it can be observed that the IOS with $2M$ elements slightly outperforms the IOS with M elements due to the additional beamforming gain from the increase in the element number and the aperture size. Similarly, it outperforms the BIOS when $M_y$ is small for the same reasons. However, with the increase in $M_y$, the performance of the IOS with $2M$ elements remains significantly inferior to that of the BIOS due to the coupling between the reflected and refracted beams.

Finally, to visualize the SIC capability of the BIOS and verify the theoretical lower bound discussed in Section 4, Figure 5 shows the residual SI power as a function of $M_y$ in SISO FD systems with different $d_{\mathrm{BB}}$. Specifically, the solid line represents the residual SI power, and the dashed line of the same color represents the theoretical lower bound of M for efficient SIC in the system with the same $d_{\mathrm{BB}}$. It can be observed that the residual SI power with the proposed scheme monotonically decreases as M increases due to the beamforming gain. Meanwhile, despite the approximation, the theoretical lower bound calculated according to (37) is consistent with the simulation result.

6. Conclusions

We proposed a novel BIOS-assisted FD system for simultaneous SIC and SRM, and formulated the beamforming optimization problem aiming at maximizing the SR with the SIC requirement constraint. The BS precoder and the BIOS phase shifts were jointly optimized by the proposed SIC-WMMSE approach. Furthermore, a theoretical lower bound of BIOS size required for efficient SIC was derived and proved to be very tight through simulation. The simulation results also showed that the BIOS-assisted FD system significantly outperforms the conventional IOS-assisted system, as the BIOS can flexibly control the beamforming for SIC and SRM.