Optimization of Robust and Secure Transmit Beamforming for Dual-Functional MIMO Radar and Communication Systems

Abstract

This paper investigates a multi-antenna, multi-input multi-output (MIMO) dual-functional radar and communication (DFRC) system platform. The system simultaneously detects radar targets and communicates with downlink cellular users. However, the modulated information within the transmitted waveforms may be susceptible to eavesdropping. To ensure the security of information transmission, we introduce non-orthogonal multiple access (NOMA) technology to enhance the security performance of the MIMO-DFRC platform. Initially, we consider a scenario where the channel state information (CSI) of the radar target (eavesdropper) is perfectly known. Using fractional programming (FP) and semidefinite relaxation (SDR) techniques, we maximize the system’s total secrecy rate under the requirements for radar detection performance, communication rate, and system energy, thereby ensuring the security of the system. In the case where the CSI of the radar target (eavesdropper) is unavailable, we propose a robust secure beamforming optimization model. The channel model is represented as a bounded uncertainty set, and by jointly applying first-order Taylor expansion and the S-procedure, we transform the original problem into a tractable one characterized by linear matrix inequalities (LMIs). Numerical results validate the effectiveness and robustness of the proposed approach.

1. Introduction

With the continuous evolution of mobile communication technology, the world has entered a critical phase of 6G research. 6G is expected to usher humanity into a new era of deep integration between the physical and virtual worlds. Dual-Functional Radar–Communication (DFRC), recognized as a pivotal technology for achieving the 6G vision, leverages the design of wireless signal systems to facilitate the sharing of signal processing modules and environmental information. This integration delivers significant advantages, including improved spectrum efficiency, reduced processing costs, and enhanced system performance. In recent years, the development of radar and communication systems toward DFRC has become a focal point of research for scholars both domestically and internationally. Specifically, in [1,2], the author proposed a cooperative scheme for MIMO communication and radar, which achieves the sharing of sensing and communication systems by jointly designing the radar transmission coding matrix and the communication transmission covariance matrix. The research [3,4] addressed the design of MIMO sensing transmission beam patterns coexisting with wireless communication systems, enhancing the required sensing and communication performance across various scenarios through solving optimization problems. DFRC can also be applied to many other scenarios, including Internet of Things (IoT) networks [5,6], smart homes or vehicular networks [7,8], and autonomous driving technology networks [9].

In addition, the proper integration of DFRC with other wireless technologies or devices can lead to many emerging applications, including orthogonal frequency division multiplexing (OFDM), intelligent reflecting surfaces (IRSs), and non-orthogonal multiple access (NOMA) technologies. For example, in [10], the author proposes an OFDM-DFRC system, which effectively utilizes the spectrum while maintaining robust communication to prevent multipath fading and improving radar estimation accuracy. Similarly, in [11], the author explores a RIS-aided DFRC system. By designing the transmit beamformer and IRS reflection, they achieve up to a 32 dB improvement in the radar SNR compared to passive RIS-aided DFRC systems.

As a key technology in 5G networks, NOMA can serve multiple users simultaneously, improving the utilization of shared spectrum. By combining NOMA with DFRC systems, spectrum sharing can be achieved. The research in [12] proposed a NOMA-DFRC framework, utilizing NOMA to provide services to multiple communication users while performing target sensing, enhancing the system’s communication performance and target sensing ability. In [13], the author investigated a NOMA-aided DFRC system, where NOMA serves multiple users while detecting a set of radar targets. A joint optimization framework is proposed to maximize sensing efficiency while simultaneously ensuring the communication performance of NOMA users and the sensing accuracy of designated targets. This approach effectively balances the trade-offs between communication and sensing requirements, achieving optimal system performance. The research in [14] introduced an iterative channel estimation method for DFRC that maintains both sensing and communication performance while providing spectrum efficiency. This method achieves uplink transmission through estimating the channel of the received signals. In [15], the author proposed a NOMA-assisted communication, sensing, and computing application. The proposed application maximizes computation offloading capabilities and suppresses inter-functional interference, achieving high-quality sensing and computing performance. By adopting NOMA technology, the degrees of freedom in DFRC systems are greatly increased, significantly enhancing both sensing and communication performance while also offering possibilities for other applications.

However, the transmitted signals in DFRC systems are designed to carry communication signals and data through specific waveforms. When eavesdropping devices are present during system operation, the communication security of users in DFRC systems cannot be guaranteed. Therefore, physical layer security is a critical consideration in DFRC system design. In [16], the authors leveraged the inherent interference of strong radar signals to ensure the physical layer security of coexisting multi-user single-input single-output communication systems and collaborative MIMO radar. In [17], a drone system integrated with DFRC technology was proposed, addressing a joint optimization problem involving user scheduling, transmission power, and source drone trajectories to maximize secure data transmission rates. The research in [18,19] both utilized intelligent reflecting surfaces (IRSs) to enhance the physical layer security of DFRC systems. The former formulated a joint optimization problem to maximize user capacity and secure rates through active and passive beamforming, while the latter investigated the joint design of communication and radar beamformer phase shifts on the IRS and base station to maximize the gain of the perceptual beam pattern towards targets, ensuring minimal information leakage to eavesdroppers. This demonstrated the significant potential of IRSs in enhancing the security of DFRC systems. The research in [20] combined the Capon method and approximate maximum likelihood techniques to identify potential eavesdropper directions, achieving a mutually beneficial outcome for both sensing and security functions. In [21,22,23], an artificial noise (AN)-assisted strategy was proposed to ensure the security of MIMO communication systems. The strategy entails transmitting AN in conjunction with information-bearing signals to effectively disrupt eavesdropping devices, ensuring secure communication.

Additionally, the wireless security challenges associated with NOMA systems merit significant attention. Due to the higher transmission power of the weak-channel user, the signals are more susceptible to interception. A secure transmission scheme for a MISO-NOMA framework is discussed in [24], using artificial jamming generated at the BS to ensure security in the presence of an eavesdropping device. By maximizing the transmission power of the AN and using SIC to eliminate the jamming signal before it interferes with legitimate transmissions, eavesdropping can be effectively disrupted without affecting lawful communication. In [25], the author explored the wireless security of NOMA-SISO systems. Each user has predefined QoS requirements, and the feasible transmission power region that satisfies all users’ QoS needs was identified. Additionally, the research in [26] proposed a secure transmission scheme in dual-user NOMA channels using deep neural networks (DNNs). By applying linear precoding to each user’s signal using deep neural networks, the computation time is significantly reduced, and the secrecy capacity increases to approximately 98 precent. At the same time, spectrum efficiency is enhanced, with dynamic complexity several times lower than that of existing iterative methods.

Nevertheless, the research on the secure transmission of DFRC systems under the NOMA framework is still limited. Currently, only the research in [27] has addressed the secure transmission problem in a NOMA-assisted DFRC system by introducing artificial interference. Based on the above analysis, we propose a secure transmission framework for NOMA-assisted DFRC systems in scenarios involving eavesdroppers. Sensing and communication are achieved by superimposing signals in the power domain, with SIC employed to remove inter-user interference. We fully leverage the high degrees of freedom that NOMA technology brings to DFRC systems and investigate the beamforming problem to maximize the system’s security performance, constrained by minimum communication rate requirements and specific radar performance limitations. Additionally, since the eavesdroppers are non-cooperative, we also consider the robust beamforming problem in scenarios where the exact CSI of the eavesdropper is unavailable. At present, there has been no prior research dedicated to addressing the robust secure transmission challenges in NOMA-assisted DFRC systems. This gap highlights the novelty and significance of the proposed work. In summary, our contributions are as follows:

• A secure communication deployment scheme for NOMA-assisted DFRC systems is proposed, considering two scenarios where the eavesdropper’s CSI is precisely known and where the CSI of the eavesdropper is unavailable. The security design issues related to beamforming patterns, user signal-to-noise ratios, and power constraints are addressed for each scenario.
• In the scenario where the eavesdropper’s CSI is known, a secure communication optimization model based on transmit beamforming is proposed. Subject to minimum communication rate constraints and radar beam threshold requirements, the model iteratively addresses the beamforming optimization challenge, aiming to enhance the aggregate secrecy rate for multiple users by integrating fractional programming (FP) with semidefinite relaxation (SDR) methods. The high degree of freedom NOMA technology provides is integrated into the secure communication of DFRC systems.
• In the scenario where the CSI of the eavesdropping device is unavailable, a robust transmit beamforming optimization model for secure communication is proposed, representing the uncertain channel model as a bounded uncertainty set. Under the constraints of ensuring the minimum communication rate for system users and meeting radar beam threshold requirements, the model constructs a bounded beamforming optimization to maximize the sum secrecy rate for users. By jointly applying Taylor series expansion and the S-procedure, the original problem is transformed into a tractable problem characterized by linear matrix inequalities (LMIs), enabling efficient and tractable solutions using standard optimization techniques.

Notations: All scalars are in italicized letters, and vectors and matrices are in bold letters. ${\mathbb{C}}^{x\times y}$ denotes the $x\times y$ dimensional complex matrix. $∥·∥$ denotes its Euclidean norm and $diag\left(·\right)$ denotes a diagonal matrix. For any general matrix $\mathit{Z}$, ${\mathit{Z}}^{†}$ denote its conjugate transpose.

2. System Model

As shown in Figure 1, we consider a dual-function MIMO-DFRC system equipped with a uniform linear array (ULA) of M antennas, where the sensing signals carrying communication information are used for both sensing and communication purposes simultaneously. The system detects information about the target of interest while serving K legitimate users via NOMA (since the users are generally cooperative, we can estimate their CSI through pilot signals). At the same time, there may be potential eavesdroppers near the users, and the transmitted communication symbols may be intercepted by adversarial malicious devices. We address two possible scenarios: one where the eavesdropper’s CSI is precisely known and another where the CSI of the eavesdropper is unavailable.

2.1. Communication Model of NOMA-DFRC System

By employing the transmit beamforming design, the transmit signal of the MIMO-DFRC system can be given by

$$y_k={\mathit{h}}_k^{†}{\sum }_{i=1}^K{\mathit{w}}_i{\mathit{s}}_i+n_k={\sum }_{i=1}^K{\mathit{h}}_k^{†}{\mathit{w}}_i{\mathit{s}}_i+n_k$$

(1)

where ${\mathit{s}}_k\in {\mathbb{C}}^{M\times 1}$ denotes the information intended for transmission to user k and ${\mathit{h}}_k={\zeta }_k\widetilde{{\mathit{h}}_k}$ denotes the channel of DFRC to user k. The Rayleigh distributed channel can be utilized between the DFRC system and the users. Typically, in NOMA transmission, we assume ${\zeta }_1\le {\zeta }_2\le \cdots \le {\zeta }_K$, where user 1 has the weakest channel gain, and user K has the strongest channel gain. Based on this, when the DFRC system transmits communication symbols to users using NOMA, the k-th user first applies SIC to detect and remove interference from all users with weaker channel gains than user k ($j<k$); at the same time, the signal of stronger users $j>k$ is treated as noise. Therefore, the achievable rate at user k is

$$R_k^k={log}_2\left(1+{\displaystyle \frac{{\left|{\mathit{h}}_k^{†}{\mathit{w}}_k\right|}^2}{{\sum }_{j=k+1}^K{\left|{\mathit{h}}_k^{†}{\mathit{w}}_j\right|}^2+{\sigma }_n^2}}\right)$$

(2)

In addition, the information for user k, apart from the part it demodulates itself, can also be decoded by all users with stronger channel gains through SIC. Therefore, for users $j>k$, the following achievable rate is produced:

$$R_k^j={log}_2\left(1+{\displaystyle \frac{{\left|{\mathit{h}}_j^{†}{\mathit{w}}_k\right|}^2}{{\sum }_{i=k+1}^K{\left|{\mathit{h}}_j^{†}{\mathit{w}}_i\right|}^2+{\sigma }_n^2}}\right)$$

(3)

Therefore, the achievable rate of $s_k$ should take the lowest value of communication rates, which can be expressed as

$$R_k=min\left\{R_k^k,R_k^{k+1},\cdots ,R_k^K\right\}$$

(4)

At user K, SIC can be used to remove interference signals from other users, and the transmission rate can be simplified to

$$R_K={log}_2\left(1+{\left|{\mathit{h}}_K^{†}{\mathit{w}}_K\right|}^2/{\sigma }_n^2\right)$$

(5)

2.2. Secure Transmission Problem of MIMO-DFRC

Regarding the security of user information transmission, users who are located farther away and have higher transmission power are more likely to face threats from eavesdropping devices. We assume the presence of a potential eavesdropper near such high-power users, aiming to intercept communication between the DFRC system and legitimate users. Since the eavesdropper lacks the necessary information to perform SIC, the SNR observed by the eavesdropper for the k-th user’s received signal can be represented as follows:

$$R_e^k={log}_2\left(1+{\displaystyle \frac{{\left|{\mathit{g}}_e^{†}{\mathit{w}}_k\right|}^2}{{\sum }_{j=1,j\ne k}^K{\left|{\mathit{g}}_e^{†}{\mathit{w}}_j\right|}^2+{\sigma }_k^2}}\right)$$

(6)

where $g_e\in {\mathbb{C}}^{M\times 1}$ represents the CSI from the DFRC system to the eavesdropper. We propose the secrecy rate to evaluate the security performance of the system [28], which can be expressed as

$$R_s^k={\left[R_c^k-R_e^k\right]}^{+},k\in \mathbb{K}$$

(7)

2.3. Design of the Beamforming for Radar Sensing

In order to better detect and estimate targets, it is often necessary to design beam patterns with excellent performance. Therefore, in this paper, we utilize beam pattern similarity metrics to assess sensing performance, using the minimum MSE between the designed beam pattern and the ideal beam pattern as the evaluation criterion, which can be expressed as follows [29]:

$$\begin{array}{cc}\hfill \underset{\xi ,{\mathit{S}}_d}{min}& {\mathrm{\Gamma }}_{mse}\triangleq {\displaystyle \frac{1}{L}}{{\sum }_{l=1}^L\left|\xi P_d\left({\theta }_0\right)-{\mathit{\alpha }}^{†}\left({\theta }_0\right){\mathit{S}}_d\mathit{\alpha }\left({\theta }_0\right)\right|}^2\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill s.t.& \mathrm{Tr}\left({\mathit{S}}_d\right)=P_0\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill & {\mathit{S}}_d\ge 0,{\mathit{S}}_d={\mathit{S}}_d^{†}\hfill \end{array}$$

(8c)

$$\begin{array}{cc}\hfill & \xi \ge 0\hfill \end{array}$$

(8d)

In this context, ${\theta }_0$ represents the azimuth of the target to be estimated, the normalized $P_d\left({\theta }_0\right)$ represents the ideal beam pattern, $\mathit{\alpha }\left({\theta }_0\right)$ denotes the signal steering vector, and $\xi$ is a proportionality factor. By minimizing the mean square error ${\mathrm{\Gamma }}_{mse}$, we can derive the covariance matrix ${\mathit{S}}_d$ of the ideal beam pattern. We will aim to optimize ${\mathit{S}}_d$ to closely approximate the ideal beam pattern, thereby meeting the performance requirements for detection by the sensing device.

3. Secure Beamforming Design with Known Eavesdropper CSI

In this section, the eavesdropper’s CSI can be accurately estimated, and the eavesdropper lacks sufficient information to perform SIC on the signals of legitimate users. Assuming that all users require secure transmission, we aim to optimize the overall secrecy rate by designing secure beamforming vectors for the DFRC system, while ensuring the system’s sensing and communication capabilities. To begin, we define ${\mathit{W}}_k\triangleq {\mathit{w}}_k{\mathit{w}}_k^{†}$. The DFRC secure beamforming problem can be given by

$$\begin{array}{cc}\hfill \underset{{\mathit{W}}_k}{max}& {\sum }_{k=1}^K{R}_s^k\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill s.t.& {\displaystyle \frac{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{W}}_k\right)-{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}}\le {\gamma }_{bp}\hfill \end{array}$$

(9b)

$$\begin{array}{cc}\hfill & min\left\{R_k^k,R_k^{k+1},\cdots ,R_k^K\right\}\ge {\gamma }_k,k=1,2,...K-1\hfill \end{array}$$

(9c)

$$\begin{array}{cc}\hfill & R_K\ge {\gamma }_K\hfill \end{array}$$

(9d)

$$\begin{array}{cc}\hfill & {\sum }_{k=1}^{K}tr\left({\mathit{W}}_k\right)\le P_0\hfill \end{array}$$

(9e)

$$\begin{array}{cc}\hfill & {\mathit{W}}_k={\mathit{W}}_k^{†},{\mathit{W}}_k\underline{\succ }0,k\in \mathbb{K}\hfill \end{array}$$

(9f)

$$\begin{array}{cc}\hfill & rank\left({\mathit{W}}_k\right)=1\hfill \end{array}$$

(9g)

Constraint (9b) ensures that the designed DFRC system’s transmit beam pattern closely approximates the ideal radar beam pattern, meeting the target detection requirements. Constraints (9c) and (9d) represent the minimum rate conditions for NOMA multicast transmission, ensuring reliable communication. Constraint (9e) reflects the total power constraint for the DFRC system. Both the objective function and constraints (9c) and (9d) are non-convex, making the problem challenging to solve directly. By substituting (4)–(7) into (9a), we obtain

$$\begin{array}{cc}\hfill & {\sum }_{k=1}^K{R}_s^k=\sum _{k=1}^{K-1}{log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}{{\sum }_{j=k+1}^K{h}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2}}\right)\hfill \\ \hfill & -\sum _{k=1}^K{log}_2\left(1+{\displaystyle \frac{{\mathit{g}}_e^{†}{\mathit{W}}_k{\mathit{g}}_e}{{\sum }_{j=1,j\ne k}^K{\mathit{g}}_e^{†}{\mathit{W}}_j{\mathit{g}}_e+{\sigma }_n^2}}\right)+{log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_K^{†}{\mathit{W}}_K{\mathit{h}}_K}{{\sigma }_n^2}}\right)\hfill \end{array}$$

(10)

The objective function contains a difference of logarithmic terms. To handle this, we first apply a basic transformation to simplify it. Equation (11) can then be reformulated as

$$\begin{array}{cc}\hfill & {\sum }_{k=1}^K{R}_s^k=\sum _{k=1}^{K-1}{log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}{{\sum }_{j=k+1}^K{\mathit{h}}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2}}\right)\hfill \\ \hfill & +\sum _{k=1}^K{log}_2\left({\displaystyle \frac{{\sum }_{j=1,j\ne k}^K{\mathit{g}}_e^{†}{\mathit{W}}_j{\mathit{g}}_e+{\sigma }_n^2}{{\sum }_{i=1}^K{\mathit{g}}_e^{†}{\mathit{W}}_i{\mathit{g}}_e+{\sigma }_n^2}}\right)+{log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_K^{†}{\mathit{W}}_K{\mathit{h}}_K}{{\sigma }_n^2}}\right)\hfill \end{array}$$

(11)

Due to the presence of logarithmic functions with fractional terms in Equation (12), the problem remains non-convex. Fortunately, fractional programming (FP) methods can eliminate the fractional terms and make it easier to solve [30]. First, we apply the Lagrangian dual transformation technique, utilizing the Closed-Form FP approach to address the first term in Equation (12). We first consider the auxiliary variables ${\lambda }_{u,k}$ and ${\varsigma }_{u,k}$; the first term in Equation (12) can be reformulated as follows:

$$\begin{array}{cc}\hfill & \sum _{k=1}^{K-1}{log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}{{\sum }_{j=k+1}^K{\mathit{h}}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2}}\right)={\sum }_{k=1}^{K-1}{log}_2\left(1+{\lambda }_{u,k}\right)-{\sum }_{k=1}^{K-1}{\lambda }_{u,k}\hfill \\ \hfill & +2{\sum }_{k=1}^{K-1}{\varsigma }_{u,k}\sqrt{\left(1+{\lambda }_{u,k}\right){\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}-{\sum }_{k=1}^{K-1}{\left|{\varsigma }_{u,k}\right|}^2\left({\sum }_{j=k+1}^K{\mathit{h}}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2\right)\hfill \\ \hfill & \triangleq C\left({\lambda }_{u,k},{\varsigma }_{u,k}\right)+\tilde{G}\left({\lambda }_{u,k},{\varsigma }_{u,k},{\mathit{W}}_k\right)\hfill \end{array}$$

(12)

When ${\mathit{w}}_k$ is fixed, the optimal values for ${\lambda }_{u,k}^{*}$ and ${\varsigma }_{u,k}^{*}$ can be obtained by taking the partial derivatives and setting them equal to zero, i.e.,

$${\lambda }_{u,k}^{*}={\displaystyle \frac{{\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}{{\sum }_{j=k+1}^K{\mathit{h}}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2}}$$

(13)

$${\varsigma }_{u,k}^{*}={\displaystyle \frac{\sqrt{\left(1+{\lambda }_{u,k}\right){\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}}{{\sum }_{j=k+1}^K{\mathit{h}}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2}}$$

(14)

It can be observed that the fractional term in the first part of Equation (12) has been successfully transformed into a convex form. Next, we apply the Direct FP approach to handle the second term in Equation (12). Through the quadratic transformation, we can rewrite the second term as follows:

$$\begin{array}{cc}\hfill & \sum _{k=1}^K{log}_2\left({\displaystyle \frac{{\sum }_{j=1,j\ne k}^K{\mathit{g}}_e^{†}{\mathit{W}}_j{\mathit{g}}_e+{\sigma }_n^2}{{\sum }_{i=1}^K{\mathit{g}}_e^{†}{\mathit{W}}_i{\mathit{g}}_e+{\sigma }_n^2}}\right)={log}_2\left(2q_k\sqrt{\sum _{j=1,j\ne k}^K{\mathit{g}}_e^{†}{\mathit{W}}_j{\mathit{g}}_e+{\sigma }_n^2}-q_k^2\left(\sum _{i=1}^K{\mathit{g}}_e^{†}{\mathit{W}}_i{\mathit{g}}_e+{\sigma }_n^2\right)\right)\hfill \\ \hfill & \triangleq {log}_2\left(2y_k\sqrt{A\left({\mathit{W}}_k\right)}-q_k^{2}B\left({\mathit{W}}_k\right)\right)\hfill \end{array}$$

(15)

When ${\mathit{w}}_k$ is fixed, the value of $q_k^{*}$ can be obtained from the following formula:

$$\begin{array}{c}\hfill q_k^{*}={\displaystyle \frac{\sqrt{{\sum }_{j=1,j\ne k}^K{\mathit{g}}_e^{†}{\mathit{W}}_j{\mathit{g}}_e+{\sigma }_n^2}}{{\sum }_{i=1}^K{\mathit{g}}_e^{†}{\mathit{W}}_i{\mathit{g}}_e+{\sigma }_n^2}}\end{array}$$

(16)

At this point, problems (9a)–(9c) can be reformulated into the following form:

$$\begin{array}{cc}\hfill \underset{{\mathit{W}}_k}{max}& C\left({\lambda }_{u,k},{\varsigma }_{u,k}\right)+\tilde{G}\left({\lambda }_{u,k},{\varsigma }_{u,k},{\mathit{W}}_k\right)+R_K+{log}_2\left(2y_k\sqrt{A\left({\mathit{W}}_k\right)}-q_k^{2}B\left({\mathit{W}}_k\right)\right)\hfill \end{array}$$

(17a)

$$\begin{array}{cc}\hfill s.t.& {\displaystyle \frac{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{W}}_k\right)-{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}}\le {\gamma }_{bp}\hfill \end{array}$$

(17b)

$$\begin{array}{cc}\hfill & min\left\{R_k^k,R_k^{k+1},\cdots ,R_k^K\right\}\ge {\gamma }_k,k=1,2,...K-1\hfill \\ \hfill & s.t.\left(9d\right),\left(9e\right),\left(9f\right),\left(9g\right)\hfill \end{array}$$

(17c)

Where the variables ${\lambda }_{u,k},{\varsigma }_{u,k},q_k$ are updated in each iteration; when these variables are known, the objective function in Equation (18) becomes a convex problem concerning ${\mathit{W}}_k$, as it combines both linear and logarithmic functions, and can be directly solved. Next, we address the non-convex constraint (17c). By defining ${\mathit{H}}_k\triangleq {\mathit{h}}_k{\mathit{h}}_k^{†}$, constraint (17c) is reformulated as follows:

$$\begin{array}{c}ll{R}_k={log}_2\left({\displaystyle \frac{\sum _{i=k}^K{\left|{\mathit{h}}_j^{†}{\mathit{w}}_i\right|}^2+{\sigma }_n^2}{\sum _{i=k+1}^K{\left|{\mathit{h}}_j^{†}{\mathit{w}}_i\right|}^2+{\sigma }_n^2}}\right)={log}_2\left(\sum _{i=k}^{K}tr\left({\mathit{H}}_j{\mathit{W}}_i\right)+{\sigma }_n^2\right)\\ -{log}_2\left(\sum _{i=k+1}^{K}tr\left({\mathit{H}}_j{\mathit{W}}_i\right)+{\sigma }_n^2\right)\ge {\gamma }_k\end{array}$$

(18)

The non-convexity of Equation (19) stems from the second term, which contains a negative sign. Regarding this problem, we use a first-order Taylor expansion to construct a locally approximated surrogate function for Equation (19), transforming it into a convex problem. To begin, we review several key results from [31], as demonstrated in Lemma 1.

Lemma 1.

If a convex function $f\left(x\right)$ is differentiable, we approximate it by the tangent function $g\left(x\left|x_t\right.\right)$ at a given point $x_t$. The first-order expansion is

$$\begin{array}{c}\hfill f\left(x\right)\ge g\left(x\left|x_t\right.\right)=f\left(x_t\right)+\nabla f^{†}\left(x_t\right)\left(x-x_t\right)\end{array}$$

(19)

Based on Lemma 1, we reformulate constraint (17c) into a convex form. For each iteration point ${\mathit{W}}^{\left(t\right)}$, the problem (19) approximates the lower bound of the constraint function at the current local point ${\mathit{W}}^{\left(t\right)}$. Using the first-order Taylor expansion, the second term of Equation (19) is reformulated as follows:

$$\begin{array}{cc}\hfill & f\left({\mathit{W}}_k\left|{\mathit{W}}_k^{\left(t\right)}\right.\right)=-{log}_2\left(\sum _{i=k+1}^{K}tr\left({\mathit{H}}_j{\mathit{W}}_i^{\left(t\right)}\right)+{\sigma }_n^2\right)-{\displaystyle \frac{{\sum }_{i=k+1}^{K}tr\left({\mathit{H}}_j\left({\mathit{W}}_i-{\mathit{W}}_i^{\left(t\right)}\right)\right)}{\left({\sum }_{i=k+1}^{K}tr\left({\mathit{H}}_j{\mathit{W}}_i^{\left(t\right)}\right)+{\sigma }_n^2\right)ln2}}\hfill \end{array}$$

(20)

By substituting (21) into (19), constraint (17c) is given by

$$\begin{array}{c}\hfill \widetilde{R_k}={log}_2\left(\sum _{i=k}^{K}tr\left({\mathit{H}}_j{\mathit{W}}_i\right)+{\sigma }_n^2\right)+f\left({\mathit{W}}_k\left|{\mathit{W}}_k^{\left(t\right)}\right.\right)\ge {\delta }_k\end{array}$$

(21)

By substituting (22a)–(22c) into (18), the resulting new optimization problem is as follows:

$$\begin{array}{cc}\hfill \underset{{\mathit{W}}_k}{max}& C\left({\lambda }_{u,k},{\varsigma }_{u,k}\right)+\tilde{G}\left({\lambda }_{u,k},{\varsigma }_{u,k},{\mathit{W}}_k\right)+R_K+{log}_2\left(2q_k\sqrt{A\left({\mathit{W}}_k\right)}-q_k^{2}B\left({\mathit{W}}_k\right)\right)\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill s.t.& {\displaystyle \frac{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{W}}_k\right)-{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}}\le {\gamma }_{bp}\hfill \end{array}$$

(22b)

$$\begin{array}{cc}\hfill & \widetilde{R_k}\ge {\delta }_k,\forall k\in \mathbb{K},k\ne K\hfill \\ \hfill & s.t.\left(9d\right),\left(9e\right),\left(9f\right),\left(9g\right)\hfill \end{array}$$

(22c)

It is evident that Equation (23) has been reformulated as a convex optimization problem. Subsequently, we can reconstruct the rank-one solution by eigenvalue decomposition or Gaussian randomization techniques. The overall algorithmic procedure is summarized as follows.

4. Robust Secure Beamforming Design for DFRC with Imperfect CSI of the Eavesdropper

In Section 3, we assumed that the CSI between the DFRC system and the eavesdropper was fully available. However, in practical scenarios, the eavesdropper’s location is unknown due to its passive nature. Given the presence of estimation errors, obtaining perfect CSI for adversarial eavesdroppers becomes challenging [32,33]. Therefore, we adopt a norm-bounded channel uncertainty model, assuming that the base station has only obtained the uncertainty region of the eavesdropper’s channel [34], which is described as follows:

$$\begin{array}{c}\hfill {\mathsf{\Xi }}_e=\left\{{\mathit{g}}_e\left|{\mathit{g}}_e=\widetilde{{\mathit{g}}_e}+\Delta {\mathit{g}}_e\right.,∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2\right\}\end{array}$$

(23)

where ${\mathsf{\Xi }}_e$ represents the channel uncertainty sets of the eavesdropper, $\widetilde{{\mathit{g}}_e}$ is the channel estimation CSI of the eavesdropper, $\Delta {\mathit{g}}_e$ represents estimation error, and ${\epsilon }_1$ represents the maximum threshold of channel estimation error.

Considering the uncertainty of channel, the worst-case secrecy rate (WCSR) is introduced as the design objective to replace the objective function in (9a)–(9g); it can be described as follows:

$$\begin{array}{cc}\hfill & \begin{array}{c}{R}_{\mathrm{wc}}\triangleq {\displaystyle \sum _{k=1}^K}{\displaystyle \underset{∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2}{min}}{\left[R_c^k-{\tilde{R}}_e^k\right]}^{+}={\displaystyle \sum _{k=1}^K}{\left[R_c^k-{\displaystyle \underset{∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2}{max}}{\tilde{R}}_e^k\right]}^{+}\end{array}={\displaystyle {\displaystyle \sum _{k=1}^K}}\left(R_c^k-\underset{∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2}{max}{\tilde{R}}_e^k\right)\hfill \end{array}$$

(24)

where ${\tilde{R}}_e^k$ represents the eavesdropper’s rate under the uncertain channel model, i.e., ${\tilde{R}}_e^k={log}_2\left(1+{\displaystyle \frac{{\left|{\left(\widetilde{{\mathit{g}}_e}+\Delta {\mathit{g}}_e\right)}^{†}{\mathit{w}}_k\right|}^2}{\sum _{j=1,j\ne k}^K{\left|{\left(\widetilde{{\mathit{g}}_e}+\Delta {\mathit{g}}_e\right)}^{†}{\mathit{w}}_j\right|}^2+{\sigma }_k^2}}\right),∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2$. Our goal is to maximize the WCSR for the legitimate users and the eavesdropper with imperfect CSI, while simultaneously satisfying the sensing beamforming constraint, communication rate constraint, and transmission power constraint. Specifically, the optimization problem can be formulated as follows:

$$\begin{array}{cc}\hfill \underset{{\mathit{W}}_k}{max}& \sum _{k=1}^K\left(R_c^k-\underset{∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2}{max}{\tilde{R}}_e^k\right)\hfill \end{array}$$

(25a)

$$\begin{array}{cc}\hfill s.t.& {\displaystyle \frac{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{W}}_k\right)-{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}{{\mathrm{\Gamma }}_{mse}\left(\xi ,{\mathit{S}}_d\right)}}\le {\gamma }_{bp}\hfill \end{array}$$

(25b)

$$\begin{array}{cc}\hfill & min\left\{R_k^k,R_k^{k+1},\cdots ,R_k^K\right\}\ge {\gamma }_k,k=1,2,...K-1\hfill \\ \hfill & s.t.\left(9d\right),\left(9e\right),\left(9f\right),\left(9g\right)\hfill \end{array}$$

(25c)

It is evident that the uncertainty in the eavesdropper’s CSI results in a non-convex problem. To address this, we reformulate the objective function by employing auxiliary variables ${\left\{{\varpi }_1^k\right\}}_{k=1}^K$ and ${\left\{{\varpi }_2^k\right\}}_{k=1}^K$. The aforementioned non-convex problem can be redefined as

$$\begin{array}{cc}\hfill \underset{{\mathit{W}}_k,{\varpi }_1^k,{\varpi }_2^k}{max}& \sum _{k=1}^K\left({\varpi }_1^k-{\varpi }_2^k\right)\hfill \end{array}$$

(26a)

$$\begin{array}{cc}\hfill s.t.& {log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_k^{†}{\mathit{W}}_k{\mathit{h}}_k}{{\sum }_{j=k+1}^K{\mathit{h}}_k^{†}{\mathit{W}}_j{\mathit{h}}_k+{\sigma }_n^2}}\right)\ge {\varpi }_1^k,k\in \mathbb{K},k\ne K\hfill \end{array}$$

(26b)

$$\begin{array}{cc}\hfill & {log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_K^{†}{\mathit{W}}_K{\mathit{h}}_K}{{\sigma }_n^2}}\right)\ge {\varpi }_1^K,\hfill \end{array}$$

(26c)

$$\begin{array}{cc}\hfill & {log}_2\left(1+{\displaystyle \frac{tr\left({\mathit{G}}_e{\mathit{W}}_k\right)}{{\sum }_{j=1,j\ne k}^{K}tr\left({\mathit{G}}_e{\mathit{w}}_j\right)+{\sigma }_k^2}}\right)\le {\varpi }_2^k,∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2,k\in \mathbb{K}\hfill \\ \hfill & \left(25b\right),\left(25c\right),\left(9d\right),\left(9e\right),\left(9f\right),\left(9g\right)\hfill \end{array}$$

(26d)

where ${\mathit{G}}_e={\left(\widetilde{{\mathit{g}}_e}+\Delta {\mathit{g}}_e\right)}^{†}\left(\widetilde{{\mathit{g}}_e}+\Delta {\mathit{g}}_e\right)$. The non-convexity of (27a)–(27d) is primarily concentrated in constraints (27b) and (27c). To address this, we first apply the Taylor expansion method from Lemma 1. Similar to (19), (27a)–(27d) can be reformulated as follows:

$$\begin{array}{cc}\hfill \underset{{\mathit{W}}_k,{\varpi }_1^k,{\varpi }_2^k}{max}& \sum _{k=1}^K\left({\varpi }_1^k-{\varpi }_2^k\right)\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill s.t.& {log}_2\left({\sum }_{i=k}^{K}tr\left({\mathit{h}}_j{\mathit{W}}_i\right)+{\sigma }_n^2\right)+f\left({\mathit{W}}_k\left|{\mathit{W}}_k^{\left(t\right)}\right.\right)\ge {\varpi }_1^k,\forall k\in \mathbb{K},k\ne K\hfill \end{array}$$

(27b)

$$\begin{array}{cc}\hfill & {log}_2\left(1+{\displaystyle \frac{{\mathit{h}}_K^{†}{\mathit{W}}_K{\mathit{h}}_K}{{\sigma }_n^2}}\right)\ge {\varpi }_1^K,\hfill \\ \hfill & {log}_2\left({\sum }_{k=1}^{K}tr\left({\mathit{G}}_e{\mathit{W}}_k\right)+{\sigma }_k^2\right)\hfill \end{array}$$

(27c)

$$\begin{array}{cc}\hfill & -{log}_2\left({\sum }_{j=1,j\ne k}^{K}tr\left({\mathit{G}}_e{\mathit{W}}_j\right)+{\sigma }_k^2\right)\le {\varpi }_2^k,∥\Delta {\mathit{g}}_e∥\le {\epsilon }_1^2,k\in \mathbb{K}\hfill \\ \hfill & \left(25b\right),\left(25c\right),\left(9d\right),\left(9e\right),\left(9f\right),\left(9g\right)\hfill \end{array}$$

(27d)

After processing, the original non-convex terms have been transformed into convex constraints. However, due to the channel uncertainty, the resulting infinite cone constraints remain unsolvable directly. By introducing auxiliary variables ${\left\{{\varpi }_3^k\right\}}_{k=1}^K$ and ${\left\{{\varpi }_4^k\right\}}_{k=1}^K$, we rewrite constraint (27d) in the following form:

$$\begin{array}{c}\hfill {\sum }_{k=1}^{K}tr\left({\mathit{G}}_e{\mathit{W}}_k\right)+{\sigma }_k^2\ge {\varpi }_3^k\end{array}$$

(28a)

$$\begin{array}{c}\hfill {\sum }_{j=1,j\ne k}^{K}tr\left({\mathit{G}}_e{\mathit{W}}_j\right)+{\sigma }_k^2\le {\varpi }_4^k\end{array}$$

(28b)

$$\begin{array}{c}\hfill {log}_2\left({\varpi }_3^k\right)-{log}_2\left({\varpi }_4^k\right)-{\varpi }_2^k\le 0\end{array}$$

(28c)

The uncertainty constraints in (29a) and (29b) can be equivalently expressed as

$$\begin{array}{cc}\hfill & \Delta {{\mathit{g}}_e}^{†}{\sum }_{i=1}^k{\mathit{W}}_i\Delta {\mathit{g}}_e+2Re\left({\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{i=1}^k{\mathit{W}}_i\Delta {\mathit{g}}_e\right)+{\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{i=1}^k{\mathit{W}}_i\widetilde{{\mathit{g}}_e}+{\sigma }_k^2-{\varpi }_3^k\ge 0\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill & \Delta {{\mathit{g}}_e}^{†}{\sum }_{j=1,j\ne k}^k{\mathit{W}}_j\Delta {\mathit{g}}_e+2Re\left({\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{j=1,j\ne k}^k{\mathit{W}}_j\Delta {\mathit{g}}_e\right)+{\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{i=1}^k{\mathit{W}}_i\widetilde{{\mathit{g}}_e}+{\sigma }_k^2-{\varpi }_4^k\le 0\hfill \end{array}$$

(29b)

Next, we present Lemma 2, which transforms the constraints in (29a) and (29b) into linear matrix inequalities. We begin by introducing Lemma 2.

Lemma 2.

(S-procedure) Define the function

$$\begin{array}{c}\hfill {\psi }_m\left(\mathit{x}\right)\triangleq {\mathit{x}}^H{\mathit{F}}_m\mathit{x}+2Re\left\{{\mathsf{\Theta }}_m^{H}x\right\}+z_m,m=1,2\end{array}$$

(30)

where ${\mathit{F}}_m\in {\mathbb{C}}^{M\times M}$ is a Hermitian matrix, ${\mathsf{\Theta }}_m\in {\mathbb{C}}^{M\times 1}$, $\mathit{x}\in {\mathbb{C}}^{M\times 1}$, and $z_m\in \mathbb{R}$. The expression ${\psi }_1\left(\mathit{x}\right)\le 0\Rightarrow {\psi }_2\left(\mathit{x}\right)\le 0$ holds if and only if there exists $\mu \ge 0$ such that

$$\begin{array}{c}\hfill \mu \left[\begin{array}{c}{\mathit{F}}_1{\mathsf{\Theta }}_1\\ {\mathsf{\Theta }}_1^H{\mathrm{z}}_1\end{array}\right]-\left[\begin{array}{c}{\mathit{F}}_2{\mathsf{\Theta }}_2\\ {\mathsf{\Theta }}_2^H{z}_2\end{array}\right]\ge 0\end{array}$$

(31)

From (24), we derive that $\Delta {\mathit{g}}_e^{†}\Delta {\mathit{g}}_e\le {\epsilon }_1^2$. Based on Lemma 2, by introducing a relaxation variable ${\left\{{\mu }_1^k\right\}}_{k=1}^K$ and ${\left\{{\mu }_2^k\right\}}_{k=1}^K$, we can reformulate (29a) and (29b) as follows:

$$\begin{array}{c}\hfill \left[\begin{array}{cc}{\mu }_1^k{\mathit{I}}_M+{\sum }_{i=1}^k{\mathit{W}}_i& {\sum }_{i=1}^k{\mathit{W}}_i\widetilde{{\mathit{g}}_e}\\ {\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{i=1}^k{\mathit{W}}_i& C_1\end{array}\right]\ge 0\end{array}$$

(32)

$$\begin{array}{c}\hfill \left[\begin{array}{cc}{\mu }_2^k{\mathit{I}}_M-{\sum }_{j=1,j\ne k}^k{\mathit{W}}_j& {\sum }_{j=1,j\ne k}^k{\mathit{W}}_j\widetilde{{\mathit{g}}_e}\\ {\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{j=1,j\ne k}^k{\mathit{W}}_j& C_2\end{array}\right]\ge 0\end{array}$$

(33)

where

$$C_1={\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{i=1}^k{\mathit{W}}_i\widetilde{{\mathit{g}}_e}+{\sigma }_k^2-{\varpi }_3^k-{\mu }_1^k{\epsilon }_1^2$$

$$C_2={\varpi }_4^k-{\widetilde{{\mathit{g}}_e}}^{†}{\sum }_{j=1,j\ne k}^k{\mathit{W}}_j\widetilde{{\mathit{g}}_e}-{\sigma }_k^2-{\mu }_2^k{\epsilon }_1^2$$

${\mathit{I}}_M$ denotes the identity matrix.

Finally, by employing the first-order Taylor expansion from Lemma 1, (28c) can be reformulated as follows:

$$\begin{array}{cc}\hfill & {\varpi }_2^k-{log}_2\left({\left({\varpi }_3^k\right)}^{\left(t\right)}\right)-\left\{{\varpi }_3^k-{\left({\varpi }_3^k\right)}^{\left(t\right)}\right\}/{\left({\varpi }_3^k\right)}^{\left(t\right)}ln2+{log}_2\left({\varpi }_4^k\right)\ge 0\hfill \end{array}$$

(34)

By substituting the above transformations into (26a)–(26d), the resulting optimization problem becomes more tractable and can be denoted as

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}{\mathit{W}}_k,{\varpi }_1^k,{\varpi }_2^k\\ {\varpi }_3^k,{\varpi }_4^k\end{array}}{max}\sum _{k=1}^K\left({\varpi }_1^k-{\varpi }_2^k\right)\hfill \\ \hfill & s.t.\left(27b\right),\left(27c\right),\left(32\right),\left(33\right),\left(34\right),\left(25b\right),\left(25c\right),\left(9d\right),\left(9e\right),\left(9f\right),\left(9g\right)\hfill \end{array}$$

(35)

For any given ${\mathit{W}}_k$, the problem (36) is a convex problem. We can reconstruct the rank-one solution by eigenvalue decomposition or Gaussian randomization techniques [35], the algorithmic procedure for solving (26a)–(26d) is as follows.

5. Numerical Results

The numerical results are provided to assess the secure transmission performance of the MIMO-DFRC system under two distinct scenarios: one in which the eavesdropper’s CSI is available and one where it is not. The DFRC system is assumed to have a uniform linear array (ULA) comprising $M=6$ or $M=8$ antennas. The system serves $K=4$ communication users while detecting radar targets at angles $\theta =30°$ and $\theta =-30°$. The noise power at the user receivers is set to ${\sigma }_n=-110$ dBm. Path loss and fading channel models are taken from [36]. The correlation between the users’ channels is set to $t=0.5$. The Rayleigh coefficient of the BS and user channels is 2, and the distance between the BS and the user is in the range of 50 m to 200 m. Numerical results are derived from 2000 Monte Carlo simulations to ensure statistical reliability.

5.1. Performance Analysis of Secure NOMA-ISAC

We begin by considering a secure transmission scenario within the DFRC system, assuming the CSI of the eavesdropper can be perfectly estimated. Algorithm 1 is employed to solve the proposed optimization problem. Figure 2 presents the secrecy rate analysis curves for varying power budgets $P_0$, with $M=6$ and $M=8$ transmit antennas. The results show that the total secrecy rate of the system increases monotonically with the number of iterations. After five iterations, the values converge to a stable point.

Algorithm 1 Secure beamforming design of MIMO-DFRC system with known eavesdropper CSI.

Input: ${\mathit{h}}_k$, ${\gamma }_k$, $\forall k$, ${\mathit{g}}_e$, ${\sigma }_n^2$, $P_0$, ${\gamma }_{bp}$, $P_d\left({\theta }_0\right)$, $\mathit{\alpha }\left({\theta }_0\right)$ Output: ${\mathit{w}}_k$, $R_s^k$, $\forall k$, $\xi$ 1: From the provided directional pattern template $P_d\left({\theta }_0\right)$, the solution to problems (8a)–(8d) yields the covariance matrix ${\mathit{R}}_d$ and scale factor $\xi$ with an ideal directional pattern. 2: Initialize ${{\mathit{w}}_k}^{\left(t-1\right)}$ and algorithm iterations $t=1$. 3: From the given ${{\mathit{w}}_k}^{\left(t-1\right)}$, ${\mathit{h}}_k$, ${\mathit{g}}_e$, ${\sigma }_n^2$, calculate ${{\lambda }_{u,k}}^{\left(t-1\right)}$ and ${q_k}^{\left(t-1\right)}$ through (14) and (17a)–(17b). 4: From the given values of ${{\mathit{w}}_k}^{\left(t-1\right)}$, ${\mathit{h}}_k$, ${\sigma }_n^2$, along with the calculated ${{\lambda }_{u,k}}^{\left(t-1\right)}$, compute the current values of ${{\varsigma }_{u,k}}^{\left(t-1\right)}$ using (15). 5: Substitute the values of ${{\lambda }_{u,k}}^{\left(t-1\right)}$, ${{\varsigma }_{u,k}}^{\left(t-1\right)}$, ${q_k}^{\left(t-1\right)}$ into Equations (18) and (22a)–(22c), yielding (23). By solving (23) and applying eigenvalue decomposition or Gaussian randomization methods, the variable ${{\mathit{w}}_k}^{\left(t\right)}$ is obtained. 6: Calculate ${R_s^k}^{\left(t\right)}$ using the obtained values of ${{\mathit{w}}_k}^{\left(t\right)}$ through Equation (11). If ${R_s^k}^{\left(t\right)}$ converges, the iteration is completed. Otherwise, set $t=t+1$ and proceed to step 3.

Figure 3 depicts the influence of the number of transmit antennas M and the power budget $P_0$ on the system’s secrecy performance under varying minimum communication user rate thresholds $R_{min}$. Obviously, the system’s secrecy rate improves as both the power budget and the number of transmit antennas are increased. This improvement arises because higher power budgets provide the system with more energy resources, while a greater number of transmit antennas enhances array gain and increases the degrees of freedom available for optimization. These factors collectively enable the system to improve mutual information with legitimate users while reducing information leakage to eavesdroppers, thereby ensuring enhanced system secrecy performance.

Additionally, a smaller $R_{min}$ results in a higher achievable total secrecy rate, an expected outcome. A smaller $R_{min}$ imposes more relaxed constraints on system design, allowing for greater flexibility and enhanced secrecy performance. In practical scenarios, it is crucial to strike an appropriate balance between system secrecy and communication rates.

Similarly, Figure 4 shows the effect of the number of transmit antennas and the power budget on the secrecy performance of the DFRC system under different beamforming thresholds ${\gamma }_{bp}$. The system’s secrecy rate improves with an increase in both the power budget and the number of transmit antennas. Additionally, a larger ${\gamma }_{bp}$ imposes more relaxed constraints on the DFRC system design, leading to higher secrecy rates. It is noteworthy that as ${\gamma }_{bp}$ increases, the difference in total secrecy rates across systems with varying numbers of transmit antennas becomes more pronounced. This is because looser constraints provide greater system flexibility and higher degrees of freedom.

In Figure 5, we present the transmit beam patterns achieved by the proposed scheme under random channel conditions. The parameters are set as $N=6$, $R_{min}=1$ bit/s/Hz, ${\gamma }_{bp}=-20$ dB, ${\gamma }_{bp}=-15$ dB, and ${\gamma }_{bp}=-10$ dB. The beam pattern for “Radar-only” is obtained by solving problems (8a)–(8d). Obviously, the beam pattern designed using the NOMA scheme closely approximates the ideal radar beam pattern, satisfying the sensing requirements for the target to be detected.

Moreover, as ${\gamma }_{bp}$ decreases, the beam pattern generated by the NOMA scheme becomes increasingly similar to that of the ideal radar system. However, this improvement comes at the cost of reduced secrecy rates.

To demonstrate the superiority of the proposed scheme, we compare the secrecy rate of the proposed method (“NOMA-DFRC”) with that of the conventional DFRC system (“Conventional DFRC”) and the artificial noise-aided DFRC system [37] (“AN-aid DFRC”). The secrecy rate maximization mathematical model for the “Conventional DFRC” can be denoted as

$$\begin{array}{cc}\hfill lll\underset{W_k}{max}& \sum _{k=1}^K{log}_2\left(1+{\displaystyle \frac{h_k^{†}{W}_k{h}_k}{{\sum }_{j=1,j\ne k}^K{h}_k^{†}{W}_j{h}_k+{\sigma }_k^2}}\right)-{log}_2\left(1+{\displaystyle \frac{g_e^{†}{W}_k{g}_e}{{\sum }_{j=1,j\ne k}^K{g}_e^{†}{W}_j{g}_e+{\sigma }_e^2}}\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\displaystyle \frac{h_k^{†}{W}_k{h}_k}{{\sum }_{j=1,j\ne k}^K{h}_k^{†}{W}_j{h}_k+{\sigma }_k^2}}\ge {\gamma }_b,k\in \mathbb{K}\hfill \\ \hfill & \left(9b\right)\left(9e\right)\left(9f\right)\left(9g\right)\hfill \end{array}$$

(36)

As shown in Figure 6, The proposed scheme has a significantly higher secrecy rate than other traditional schemes. Additionally, for the AN-aided DFRC system under the same power budget, the system provides higher degrees of freedom, leading to a more substantial improvement in secrecy performance.

5.2. Performance Analysis of Robust Secure NOMA-DFRC System

Now, let us consider the scenario where the eavesdropper’s CSI is unknown. First, we present the convergence analysis curves of Algorithm 2 for $M=6$ and $M=8$ transmit antennas, and various channel estimation errors ${\epsilon }_1=0.1$, ${\epsilon }_1=0.3$, and ${\epsilon }_1=0.5$. As shown in Figure 7, it can be observed that, for different channel estimation errors and numbers of transmit antennas, the system’s WCSR increases monotonically with the number of iterations. After three iterations, the values converge to a fixed point, demonstrating the convergence and effectiveness of the algorithm.

Algorithm 2 Robust secure beamforming design for DFRC with imperfect CSI of the eavesdropper.

Input: ${\mathit{h}}_k$, ${\gamma }_k$, $\forall k$, $g_e$, ${\sigma }_n^2$, $P_0$, ${\gamma }_{bp}$, $P_d\left({\theta }_0\right)$, $\mathit{\alpha }\left({\theta }_0\right)$, ${\epsilon }_1$ Output: ${\mathit{w}}_k$, $R_s^k$, $\forall k$, $\xi$ 1: From the provided directional pattern template $P_d\left({\theta }_0\right)$, the solution to problems (8a)–(8d) yields the transmit signal covariance matrix ${\mathit{R}}_d$ and scale factor $\xi$ with an ideal directional pattern. 2: Initialize ${{\mathit{w}}_k}^{\left(t-1\right)}$ and algorithm iterations $t=1$. 3: From the given ${{\mathit{w}}_k}^{\left(t-1\right)}$, ${\mathit{h}}_k$, ${\mathit{g}}_e$, ${\sigma }_n^2$, calculate ${{\mathit{W}}_k}^{\left(t\right)}$, ${\left({\varpi }_1^k\right)}^{\left(t\right)}$, ${\left({\varpi }_2^k\right)}^{\left(t\right)}$, ${\left({\varpi }_3^k\right)}^{\left(t\right)}$ and ${\left({\varpi }_4^k\right)}^{\left(t\right)}$ through problem (36). 4: Based on the obtained values of ${{\mathit{W}}_k}^{\left(t\right)}$, ${\left({\varpi }_1^k\right)}^{\left(t\right)}$ and ${\left({\varpi }_2^k\right)}^{\left(t\right)}$, compute ${R_{wc}}^{\left(t\right)}$ by using problem (36). If the value of ${R_{wc}}^{\left(t\right)}$ converges, the iteration is complete. Otherwise, set $t=t+1$ and proceed to step 3. 5: Based on the rank of ${{\mathit{W}}_k}^{\left(t\right)}$, determine whether to use eigenvalue decomposition or randomization to obtain the final solution.

Figure 8 shows the performance of the robust method under varying error coefficients ${\epsilon }_1$. As anticipated, the greater CSI uncertainty results in a lower WCSR for the system. This is because a certain level of secrecy performance must be sacrificed to mitigate the effects of channel uncertainty. Additionally, an increase in transmit antennas enhances the system’s available degrees of freedom, leading to a higher WCSR.

Figure 9 compares the system security performance of the robust secure transmission Algorithm 2 (“Robust Method”) with that of the conventional non-robust method (“Non-robust”). The conventional non-robust method directly uses outdated CSI, i.e., the estimated eavesdropper channel with errors, as the actual channel [38]. “Perfect CSI” represents the system’s security performance obtained using Algorithm 1 in the absence of channel errors. Through comparison, we can draw the following conclusions:

(1) When channel errors are present, the proposed robust Algorithm 2 achieves a higher WCSR compared to the conventional method. (2) The smaller the channel error, the closer the WCSR of the system obtained by Algorithm 2 to that of the perfect CSI system, leading to better security performance and robustness. (3) As the transmit power increases, the corresponding system WCSR also increases.

Therefore, to achieve near-optimal secrecy performance, the eavesdropper’s CSI uncertainty should be strictly controlled within a narrow range.

6. Discussion

This entire article addresses the problem of secure transmit beamforming design in NOMA-assisted DFRC systems. The transmitted superimposed signals are utilized for communication and radar sensing, while SIC is employed to remove inter-user interference. Two potential scenarios are considered: one where the eavesdropper’s CSI is precisely known, and another where the eavesdropper’s CSI is unavailable. For the scenario with precise eavesdropper CSI, the optimization problem is converted into a tractable convex problem that can be efficiently addressed using fractional programming. The total secrecy rate of the system is then solved iteratively through optimization. For the scenario where the eavesdropper’s CSI is unavailable, a robust transmit beamforming optimization model is developed. By leveraging the S-procedure and Taylor series expansion, the original problem is reformulated into a convex optimization problem characterized by LMI, enabling efficient and tractable solutions using standard optimization techniques. Numerical results demonstrate that the proposed algorithms significantly enhance the system’s security performance in both scenarios. In practical applications, the relevant metrics should be adjusted in real time based on specific requirements to meet diverse demands.