Deep Reinforcemnet Learning for Robust Beamforming in Integrated Sensing, Communication and Power Transmission Systems

Abstract

A communication network integrating multiple modes can effectively support the sustainable development of next-generation wireless communications. Integrated sensing, communication, and power transfer (ISCPT) represents an emerging technological paradigm that not only facilitates information transmission but also enables environmental sensing and wireless power transfer. To achieve optimal beamforming in transmission, it is crucial to satisfy multiple constraints, including quality of service (QoS), radar sensing accuracy, and power transfer efficiency, while ensuring fundamental system performance. The presence of multiple parametric constraints makes the problem a non-convex optimization challenge, underscoring the need for a solution that balances low computational complexity with high precision. Additionally, the accuracy of channel state information (CSI) is pivotal in determining the achievable rate, as imperfect or incomplete CSI can significantly degrade system performance and beamforming efficiency. Deep reinforcement learning (DRL), a machine learning technique where an agent learns by interacting with its environment, offers a promising approach that can dynamically optimize system performance through adaptive decision-making strategies. In this paper, we propose a DRL-based ISCPT framework, which effectively manages complex environmental states and continuously adjusts variables related to sensing, communication, and energy harvesting to enhance overall system efficiency and reliability. The achievable rate upper bound can be inferred through robust, learnable beamforming in the ISCPT system. Our results demonstrate that DRL-based algorithms significantly improve resource allocation, power management, and information transmission, particularly in dynamic and uncertain environments with imperfect CSI.

1. Introduction

As wireless networks evolve to meet the increasing demands of ultra-reliable low-latency communication (uRLLC), massive machine-type communication (mMTC), and enhanced mobile broadband (eMBB), the efficient utilization of available spectrum and the improvement of signal quality have become critical priorities [1,2,3]. In this context, integrated sensing and communications (ISAC) has emerged as a promising solution, enabling the simultaneous transmission of data and environmental sensing via shared wireless signals [4,5]. Concurrently, the development of energy-harvesting technologies has generated interest in simultaneous wireless information and power transfer (SWIPT), which extends the operational lifespan of wireless devices while maintaining high-quality data transmission [6]. SWIPT has shown significant promise in applications such as the Internet of Things (IoT) and low-power device scenarios [7,8,9]. Building on these advancements, the concept of integrated sensing, communications, and power transfer (ISCPT) has been introduced. ISCPT combines the principles of ISAC and SWIPT to create a versatile system that not only transmits information and harvests energy, but also gathers situational awareness within a unified framework [10]. Using shared signal designs and optimized resource allocation, ISCPT systems have the potential to transform traditional wireless networks by simultaneously addressing the challenges of data transmission, environmental sensing, and energy harvesting [11,12].

The principles underpinning ISCPT highlight the complementary nature of its constituent technologies [13]. ISCPT systems have demonstrated substantial capacity in improving spectrum efficiency, energy sustainability, and system scalability. A core area of investigation in ISCPT research is signal design and resource optimization, where researchers aim to develop multi-functional waveforms that address diverse system requirements [14,15]. Furthermore, advances in hardware architecture, including energy-harvesting circuits and sensing-optimized antenna designs, have enabled the practical deployment of ISCPT systems [16]. Among these enabling technologies, beamforming is a fundamental technique in ISCPT systems, as it governs the spatial allocation of wireless signals to optimize overall system performance [17,18,19,20,21,22]. Traditional beamforming approaches often rely on static optimization frameworks that assume precise and accurate knowledge of the environment and system parameters. For instance, in [23], a joint optimization problem involving power allocation and beamforming in wideband millimeter-wave (mmWave) channels is proposed, leveraging limited feedback on channel information. However, in practical deployments, obtaining perfect or complete channel state information (CSI) is often challenging due to various factors such as hardware limitations, feedback delays, and environmental dynamics. Imperfect CSI introduces uncertainties that can significantly degrade the performance of ISCPT systems. It affects critical aspects such as beamforming accuracy, power transfer efficiency, and sensing reliability, leading to suboptimal system operation. For example, inaccurate CSI may result in incorrect spatial signal allocation, increasing interference and reducing the achievable communication rate. Moreover, imperfect CSI complicates the optimization problem by introducing non-convexity and stochastic variations, making traditional static approaches less effective [24]. Addressing these challenges requires adaptive and robust optimization methods capable of dynamically handling uncertainties in CSI while maintaining high system performance.

Recent advancements in artificial intelligence (AI) and optimization techniques have provided innovative solutions for dynamically allocating resources [25] and mitigating interference [26]. To ensure robust system performance across heterogeneous network environments, deep learning (DL) has emerged as a particularly promising approach for physical-layer optimization. DL offers data-driven methods to optimize beamforming without relying on exact system models or perfect environmental knowledge. Moreover, deep reinforcement learning (DRL) introduces a novel paradigm for beamforming by modeling the optimization problem as a Markov decision process (MDP) [27,28,29,30,31]. In this framework, an agent dynamically selects beamforming actions based on environmental states. DRL leverages neural networks to approximate policies or value functions, enabling efficient exploration and exploitation in non-convex and stochastic environments, such as those encountered in reconfigurable intelligent surfaces (RISs) [32], cell-free networks [33], and movable antennas [34]. By enabling multiple functionalities with wireless communication and signal processing, DRL significantly enhances spectrum utilization, addressing the challenges posed by limited bandwidth in densely populated networks [35]. One of the most compelling advantages of DRL is its ability to maintain strong robustness in the presence of environmental uncertainties, dynamic conditions, and imperfect CSI. Unlike traditional optimization methods, which rely on fixed assumptions about system parameters and often suffer from performance degradation under mismatched conditions, DRL dynamically adapts to varying environments by continuously learning from real-time interactions [36]. This adaptability makes DRL especially effective in scenarios with stochastic dynamics, such as rapid channel fluctuations, unexpected interference, and mobility-induced changes [37]. For example, in ISCPT systems, where multiple objectives, including communication, sensing, and power transfer, compete for resources, DRL agents can adjust their policies in response to real-time feedback, ensuring that the system maintains optimal performance even under adverse conditions. So far, different agents can also design to deal with the relationship between each group of variables according to different state representations. In ISAC system, DRL can collect specialized waveforms that are optimized for both high communication rates and accurate sensing, ensuring minimal trade-offs between the two functions [38]. By allowing devices to harvest wireless energy, SWIPT reduces dependence on traditional power sources, enabling extended operation in battery-constrained scenarios. It enables devices such as IoT sensors and wearable electronics to harvest energy from ambient signals while maintaining seamless communication [39].

This capability makes DRL particularly well suited for ISCPT systems, where beamforming must balance competing objectives—sensing accuracy, communication quality, and harvested energy—while adapting to dynamic network conditions. DRL combines the adaptability of reinforcement learning with the representational power of deep neural networks, enabling the learning of optimal policies in high-dimensional, uncertain environments. DRL excels in environments with stochastic dynamics, making it ideal for ISCPT systems, where channel conditions, interference, and user requirements are constantly evolving. By modeling optimization as an MDP, DRL allows intelligent agents to learn policies that adapt to these changing conditions [32]. ISCPT systems require a delicate balance between sensing, communication, and power transfer objectives, which can be achieved through DRL by utilizing reward functions that navigate the trade-offs among these competing goals. For instance, a DRL agent can dynamically prioritize sensing accuracy in critical scenarios or optimize energy transfer when device power levels are low. DRL’s ability to explore the solution space enables the identification of better-performing policies. Once properly trained, the framework can generalize across diverse network configurations and operational conditions, making it highly scalable for large-scale ISCPT deployments involving multiple users and devices. DRL learns from interactions with the environment, allowing it to handle uncertainties and unpredictable interference patterns. Beamforming is central to ISAC, as it directs signal energy for both sensing and communication [29]. DRL-based methods dynamically optimize beam patterns to guarantee sensing accuracy and communication quality under dynamic conditions. DRL has also been used to optimize resource allocation in SWIPT systems, ensuring efficient energy harvesting while maintaining communication reliability. This is particularly useful in IoT networks, where devices have stringent energy constraints. ISCPT systems involve interactions across multiple layers, including the physical, media access control (MAC), and application layers. DRL enables cross-layer optimization by learning policies that account for dependencies and trade-offs across these layers. DRL-powered ISCPT systems can support applications such as real-time video streaming in autonomous vehicles, where high communication rates and low latency must coexist with environmental sensing and energy harvesting.

Achieving robust beamforming in ISCPT systems is critical for supporting real-world IoT applications. By integrating DRL with ISCPT technologies, this study aims to address the critical challenges in multi-objective optimization and real-time adaptability, providing a foundation for advanced solutions in future wireless networks. This paper investigates the application of DRL to robust beamforming in ISCPT systems, addressing the trade-offs among multi-user communication quality, sensing accuracy, and harvested energy efficiency. Specifically, we propose a DRL-based framework that jointly optimizes beamforming vectors while accounting for system uncertainties and dynamic network conditions. The key contributions of this work are as follows:

• DRL Framework for ISCPT Beamforming: We design and implement an innovative DRL-based framework tailored for beamforming in ISCPT systems. By simulating a dynamic environment, the DRL agent learns optimal beamforming policies through iterative trial-and-error interactions. This approach captures complex system behaviors and provides a scalable solution for optimizing multi-objective performance. Unlike traditional optimization methods, the DRL framework enables continuous adaptation to changes in CSI and environmental dynamics, ensuring robust and efficient beamforming under varying conditions.
• Achievable Upper Bound in ISCPT System: To quantify and maximize system performance, we formulate a joint optimization problem aimed at achieving the theoretical upper bound of ISCPT systems. This involves simultaneous optimization of transmit beamforming vectors while adhering to stringent system constraints, such as power budgets, quality-of-service requirements, and energy-harvesting thresholds. Furthermore, the framework addresses critical challenges posed by uncertainties in CSI and dynamically evolving parameters, ensuring a balance among communication, sensing, and energy objectives.
• Robustness to Environmental Dynamics: The proposed framework demonstrates exceptional adaptability and resilience in realistic scenarios. By leveraging DRL’s capability to learn from dynamic interactions, the system achieves robust performance even under unpredictable environmental conditions. Extensive simulations validate the effectiveness of our approach, showcasing significant improvements in multi-objective performance metrics compared to conventional optimization techniques. This adaptability positions the framework as a practical solution for next-generation wireless networks requiring real-time and robust optimization.

This work bridges the gap between the emerging field of ISCPT and advanced machine learning techniques, offering a scalable and robust solution for beamforming in multi-functional wireless networks. The proposed framework significantly enhances the practicality and efficiency of ISCPT systems, paving the way for their deployment in next-generation communication networks, including 6G and beyond.

This paper is organized as follows: Section 2 introduces the novel integrated IoT devices and sensing target communication model. Section 3 presents the problem formulation and derives the upper bound of the achievable rate under imperfect CSI. Section 4 proposes the DRL framework for joint transmit beamforming design in ISCPT systems. Section 5 provides the corresponding configurations and demonstrates the performance of the proposed algorithm through numerical results. Finally, Section 6 summarizes the findings of this paper and outlines future research directions.

Notations: ${\left(·\right)}^T$ and ${\left(·\right)}^H$ denote transpose and conjugate transpose, respectively. ${\parallel ·\parallel }_2$ and $\mathrm{tr}(·)$ represents Euclidean norm and trace, respectively. ⊙ expresses the Hadamard Product, and $\mathbb{C}$ is the sets of complex numbers. Scalar x is represented by lowercase letters, vector $\mathbf{x}$ by bold lowercase letters and matrice $\mathbf{X}$ by bold uppercase letters.

2. System Model

This section presents the system model for the ISCPT system. As depicted in Figure 1, the system consists of a base station (BS) equipped with $N_t$ antennas, $K(K<N_t)$ single-antenna IoT devices, and a target that needs to be perceived. The BS not only facilitates communication with the IoT devices but also transfers energy to sustain their operations. Simultaneously, the BS engages in target sensing by transmitting dedicated signals designed for environmental perception. The integration of communication, power transfer, and sensing is crucial to the system’s functionality and efficiency. In the following subsections, we provide a detailed description of the channel propagation model, data transmission model, energy-harvesting model, and target sensing model that form the foundation of this integrated system.

2.1. Wireless Channel Propagation Model

This section clarifies the wireless channel model used for communication between the BS and the IoT devices. The channel between the BS and the k-th IoT device, denoted as ${\mathbf{h}}_k\in {\mathbb{C}}^{N_t\times 1}$, consists of two components: line-of-sight (LOS) ${\mathbf{h}}_{k,\mathrm{LOS}}$ and non-line-of-sight (NLOS) ${\mathbf{h}}_{k,\mathrm{NLOS}}$. The overall channel is the sum of these two components, where the LOS provides a direct path and the NLOS accounts for multipath effects such as reflections, scattering, and diffraction.

The BS is equipped with a uniform linear array (ULA), and the response of the antenna array to a received signal from the k-th IoT device depends on the angle of departure (AoD) ${\theta }_k$. The array response can be represented by the array steering vector $\mathbf{a}\left({\theta }_k\right)$, which is a function of the angle of arrival ${\theta }_k$. The steering vector for an antenna array in the far field is given by

$$\mathbf{a}\left({\theta }_k\right)={\displaystyle \frac{1}{\sqrt{N_t}}}{\left[1,e^{j{\displaystyle \frac{2\pi d}{\lambda }}sin\left({\theta }_k\right)},\dots ,e^{j{\displaystyle \frac{2\pi d}{\lambda }}(N_t-1)sin\left({\theta }_k\right)}\right]}^T,$$

(1)

where d is the distance between adjacent antenna elements, and $\lambda$ is the carrier wavelength.

The LOS channel component ${\mathbf{h}}_{k,\mathrm{LOS}}$ is deterministic and represents the direct line-of-sight path between the BS and the IoT device. This component is typically modeled as

$${\mathbf{h}}_{k,\mathrm{LOS}}=\sqrt{{\beta }_{k,\mathrm{LOS}}}\odot \mathbf{a}\left({\theta }_k\right),$$

(2)

where ${\beta }_{k,\mathrm{LOS}}$ is the path gain of the LOS component, which accounts for free-space path loss and large-scale fading effects. The NLOS ${\mathbf{h}}_{k,\mathrm{NLOS}}$ component accounts for multipath propagation caused by reflections, scattering, and diffractions. The overall channel between the BS and the k-th IoT device combines both LOS and NLOS components as follows:

$${\mathbf{h}}_k={\mathbf{h}}_{k,\mathrm{LOS}}+{\mathbf{h}}_{k,\mathrm{NLOS}}=\sqrt{{\displaystyle \frac{\kappa }{\left(1+\kappa \right)}}}\mathbf{a}\left({\theta }_k\right)+{\displaystyle \frac{1}{\sqrt{1+\kappa }}}{\mathbf{g}}_k,$$

(3)

where ${\beta }_{k,\mathrm{NLOS}}$ is the path gain of the NLOS component, which depends on the scattering environment and large-scale fading. $\kappa$ denotes the Rician factors, and ${\mathbf{g}}_k$ is the small-scale fading vector, representing independent and identically distributed (i.i.d.) complex Gaussian random variables for each antenna.

This composite channel model, combining both the deterministic LOS path and the random NLOS multipath effects, provides a realistic representation of the wireless channel in typical communication environments.

2.2. Data Transmission Model

For data transmission, the BS utilizes beamforming to direct the signal toward the IoT devices. The signal transmitted to the k-th IoT device is given by ${\mathbf{s}}_k\left(t\right)={\mathbf{w}}_k{x}_k\left(t\right)$, where ${\mathbf{w}}_k\in {\mathbb{C}}^{N_t\times 1}$ is the beamforming vector for the k-th device, and $x_k\left(t\right)$ is the transmitted signal, with the condition $\mathbb{E}\left\{x_i\left(t\right)x_j^H\left(t\right)\right\}=1$ for $i\ne j$, ensuring the signals are orthogonal. The received signal at the k-th IoT device is influenced by both the desired signal and interference from other devices. The channel is modeled as an additive white Gaussian noise (AWGN) channel, and the received signal at the k-th IoT device is

$$y_k\left(t\right)=\sqrt{{\rho }_k}\left[{\mathbf{h}}_k^H{\mathbf{s}}_k\left(t\right)+\sum _{j\ne k}{\mathbf{h}}_k^H{\mathbf{s}}_j\left(t\right)+n_k\left(t\right)\right]+{\overline{n}}_k\left(t\right),$$

(4)

where ${\rho }_k\left(0\le {\rho }_k\le 1\right)$ is the power split factor between communication and power transfer for the k-th device, $n_k\left(t\right)\sim \mathcal{CN}\left(0,{\sigma }_k^2\right)$ and ${\overline{n}}_k\left(t\right)\sim \mathcal{CN}\left(0,{\overline{\sigma }}_k^2\right)$ denote the additional complex Gaussian noises. The beamforming vector ${\mathbf{w}}_k$ is typically chosen to maximize the received signal power at the k-th device, improving the signal-to-interference-plus-noise ratio (SINR). The SINR for the k-th device, which is a key metric for evaluating the quality of the received signal, is given by the ratio of the desired signal power to the total interference (from other devices) and noise power. The SINR is formulated as

$${SINR}_k={\displaystyle \frac{{\rho }_k{\left|{\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2}{{\rho }_k{\sum }_{j\ne k}{\left|{\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2+{\rho }_k{\sigma }_k^2+{\overline{\sigma }}_k^2}}.$$

(5)

2.3. Energy-Harvesting Model

In the proposed system, the BS not only communicates with the IoT devices but also transfers power to sustain their operations using SWIPT. The power transfer is performed concurrently with data communication by combining the communication signal with the power transfer signal. The received signal for power harvesting at the k-th IoT device is given by

$$y_k^{EH}\left(t\right)=\sqrt{1-{\rho }_k}\left[{\mathbf{h}}_k^H{\mathbf{s}}_k\left(t\right)+\sum _{j\ne k}^K{\mathbf{h}}_k^H{\mathbf{s}}_j\left(t\right)+n_k\left(t\right)\right].$$

(6)

The energy-harvesting process depends on the signal strength and the efficiency of the energy conversion. To reduce interference and simplify the signal processing, the transmitted signal for the k-th device satisfies $\mathbb{E}\left[{\left|x_k\right|}^2\right]=1$. The energy collected by the k-th IoT device can be followed as

$$E_k={\eta }_k(1-{\rho }_k)\left[{\left|{\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2+\sum _{j\ne k}^K{\left|{\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2+{\sigma }_k^2\right],$$

(7)

where ${\eta }_k\in (0,1]$ is the energy conversion efficiency for the k-th IoT device. This model assumes a linear energy conversion process and no interference between the information and power transfer signals. The model presented here captures the dual-functionality of the BS in simultaneously serving communication and power transfer needs for the IoT devices, which is critical for the operation of energy-constrained devices in the system.

2.4. Sensing Signal Model

Along with communication and energy harvesting, the BS is responsible for sensing a target in the environment. The BS directs beams towards the target and analyzes the signals that are reflected or scattered by the target to estimate its position and properties. For simplicity, the target is assumed to only reflect the transmitted signals, and the channel $h_s\in {\mathbb{C}}^{N_t\times 1}$ is considered with a LOS path similar as (2). The received signal at the BS is then used for detection and localization [5]. The received signal from the target at the BS can be expressed as

$${\mathbf{y}}_s\left[n\right]=\sum _{i=1}^K\alpha {\mathbf{h}}_s{\mathbf{h}}_s^H{\mathbf{w}}_i{x}_i\left[n\right]+{\mathbf{n}}_s,$$

(8)

where $\alpha$ is the reflection coefficient of the target, ${\mathbf{h}}_s$ is the channel vector between the BS and the target, and ${\mathbf{n}}_s$ represents the noise associated with the target sensing. The BS employs beamforming to enhance the signal reflected by the target. The channel between the BS and the target can be represented by the steering vector ${\overline{\mathbf{a}}}_T\left({\phi }_{DoA}\right)\in {\mathbb{C}}^{N_{Tx}\times 1}$, which depends on the direction of arrival (doa) ${\phi }_{DoA}$ of the reflected signal. The steering vector is defined as

$${\overline{\mathbf{a}}}_T\left({\phi }_{DoA}\right)={\left[1,e^{-j\pi sin{\phi }_{DoA}},\dots ,e^{-j\left(N_t-1\right)\pi sin{\phi }_{DoA}}\right]}^T.$$

(9)

The target detection is enhanced by time-of-flight (ToF) measurements, which help improve the accuracy of the target’s localization [40]. The total received signal over N sampling moments is

$${\mathbf{Y}}_s=\left[\begin{array}{c}{\mathbf{y}}_s\left[1\right]\\ ⋮\\ {\mathbf{y}}_s\left[N\right]\end{array}\right]=\left[\begin{array}{c}\sum _{i=1}^K\alpha {\mathbf{h}}_s{\mathbf{h}}_s^H{\mathbf{w}}_i{x}_i\left[1\right]+{\mathbf{n}}_s\left[1\right]\\ ⋮\\ \sum _{i=1}^K\alpha {\mathbf{h}}_s{\mathbf{h}}_s^H{\mathbf{w}}_i{x}_i\left[N\right]+{\mathbf{n}}_s\left[N\right]\end{array}\right],$$

(10)

where ${\mathbf{Y}}_s$ is received signal corresponding to the target. To simplify the expression, it is intuitional that can define ${\mathbf{H}}_s={\left({\mathbf{h}}_s^H\right)}^H{\mathbf{h}}_s^H\in {\mathbb{C}}^{N_t\times N_t}$ and $\mathbf{S}=[{\mathbf{w}}_i{\mathrm{x}}_i\left[1\right],\dots ,{\mathbf{w}}_i{\mathbf{x}}_i\left[N\right]]\in {\mathbb{C}}^{N_t\times N}$ contains N independent unit-power data steams. The simplified model for the total received signal at the BS can thus be expressed as

$${\mathbf{Y}}_s=\alpha {\mathbf{H}}_s\mathbf{S}+{\mathbf{N}}_s,$$

(11)

where ${\mathbf{N}}_s\in {\mathbb{C}}^{N_t\times N}$ is the noise matrix, with each column representing the noise vector at a given time instant. This model is crucial for estimating the reflection coefficient $\alpha$ and the channel matrix ${\mathbf{H}}_s$, which are essential for target detection and characterization. The reflection coefficient $\alpha$ and the channel ${\mathbf{H}}_s$ are used to estimate the target’s location, motion, and other properties.

3. Problem Formulation

In this section, we formulate the optimization problem for the integrated system, which includes communication, wireless power transfer, and target sensing. The goal is to find an optimal beamforming strategy that maximizes the system’s overall performance while satisfying constraints on power usage and ensuring the effective operation of all system components. In practice, perfect CSI is probably impossible to obtain. Therefore, it is essential to consider the case of imperfect channel estimation. For simplify notation, we define the actual CSI as $\widehat{\mathbf{h}}=\mathbf{h}+\Delta \mathbf{h}$ represents the channel estimation error, satisfying $\Delta \mathbf{h}\sim \mathcal{N}\left(0,{\sigma }^2\right)$ that ${\sigma }^2$ is the variance of $\Delta \mathbf{h}$. The corresponding received signal at the k-th device is then described by

$${\widehat{y}}_k\left(t\right)=\sqrt{{\rho }_k}\left[{\mathbf{h}}_k^H{\mathbf{s}}_k\left(t\right)+\sum _j\Delta {\mathbf{h}}_k^H{\mathbf{s}}_j\left(t\right)+\sum _{j\ne k}{\mathbf{h}}_k^H{\mathbf{s}}_j\left(t\right)+n_k\left(t\right)\right]+{\overline{n}}_k\left(t\right),$$

(12)

where $\Delta {\mathbf{h}}_k$ is the reconstructed channel error of the k-th device. The achievable data rate for the k-th device, which is based on the Shannon capacity formula [41], can be expressed as a function of the SINR as follows:

$${\widehat{R}}_k={log}_2\left(1+{\displaystyle \frac{{\rho }_k{\left|{\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2}{{\rho }_k{\sum }_j{\left|\Delta {\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2+{\rho }_k{\sum }_{j\ne k}{\left|{\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2+{\rho }_k{\sigma }_k^2+{\overline{{\sigma }_k}}^2}}\right),$$

(13)

where ${\widehat{R}}_k$ is the data rate for the k-th device. The closed-form upper bound for this rate is derived in Appendix A.

Power transmission in ISCPT systems relies on the ability to transmit energy efficiently to devices, based on the estimated CSI. The power efficiency of energy transfer is related to the power allocated to each device, the channel gain, and the beamforming vector used. As with communication, the beamforming mismatch due to the imperfect CSI will lead to a decrease in the efficiency of power transfer. Thus, the total energy received with imperfect CSI becomes

$$\begin{array}{cc}\hfill {\widehat{E}}_k={\eta }_k\left(1-{\rho }_k\right)& \left[\underset{\mathrm{Signal}\mathrm{Power}\mathrm{Term}}{\underbrace{\left({\left|{\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2+2\Re {\left({\mathbf{h}}_k^H{\mathbf{w}}_k\right)}^H\Delta {\mathbf{h}}_k+{\left|\Delta {\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2\right)}}\right.\hfill \\ \hfill & \left.+\underset{\mathrm{Interference}\mathrm{Power}\mathrm{Term}}{\underbrace{{\sum }_{j\ne k}^K\left({\left|{\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2+2\Re {\left({\mathbf{h}}_k^H{\mathbf{w}}_j\right)}^H\Delta {\mathbf{h}}_k+{\left|\Delta {\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2\right)}}+{\sigma }_k^2\right].\hfill \end{array}$$

(14)

For target sensing, we employ the Cramér–Rao bound (CRB) as a fundamental lower bound on the variance of any unbiased estimator. The CRB quantifies the best achievable accuracy for estimating a parameter, such as the target’s position, given a set of measurements [42]. As mentioned above, the parameters of interest are the reflection coefficient $\alpha$ and the channel vector ${\mathbf{H}}_s$, which depend on the DOA ${\phi }_{DoA}(\theta ={\phi }_{DoA})$ of the signal from the target. We extend the problem by representing the reflection coefficient $\alpha$ as a tuple of its real and imaginary parts, i.e., $\{\Re \left(\alpha \right),\Im \left(\alpha \right)\}=[{\alpha }_r,{\alpha }_i]$. This allows us to derive the Fisher information matrix (FIM) for the parameters $\tau ={\left[{\alpha }_r,{\alpha }_i,\theta \right]}^T$, which can be written as

$$\mathbf{F}=\left[\begin{array}{ccc}{F}_{{\alpha }_r{\alpha }_r}& F_{{\alpha }_r{\alpha }_i}& F_{{\alpha }_r\theta }\\ F_{{\alpha }_i{\alpha }_r}& F_{{\alpha }_i{\alpha }_i}& F_{{\alpha }_i\theta }\\ F_{\theta {\alpha }_r}& F_{\theta {\alpha }_i}& F_{\theta \theta }.\end{array}\right]$$

(15)

According to Formula (11), the likelihood function for the received signal vector ${\mathbf{y}}_s$, given the unknown parameters $\alpha$ and $\theta$, is given by the probability density function (PDF) of the received signal, assuming that the noise is complex Gaussian:

$$\mathcal{L}\left(\alpha \right)=-{\displaystyle \frac{1}{{\sigma }^2}}Tr\left[{\left({\mathbf{Y}}_s-\alpha {\mathbf{H}}_s\mathbf{S}\right)}^H\left({\mathbf{Y}}_s-\alpha {\mathbf{H}}_s\mathbf{S}\right)\right]+\mathrm{constant}.$$

(16)

The log-likelihood function is a quadratic form in $\alpha$, so we can differentiate it with respect to $\tau$ to find the Fisher information.

The derivative of the log-likelihood function with respect to $\tau$ can be expressed as

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_r}}={\displaystyle \frac{2}{{\sigma }^2}}\Re \left[{\mathbf{S}}^H{\mathbf{H}}_s^H\left({\mathbf{Y}}_s-\alpha {\mathbf{H}}_s\mathbf{S}\right)\right],\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\partial \mathcal{L}}{\partial {\alpha }_i}}={\displaystyle \frac{2}{{\sigma }^2}}\Im \left[{\mathbf{S}}^H{\mathbf{H}}_s^H\left({\mathbf{Y}}_s-\alpha {\mathbf{H}}_s\mathbf{S}\right)\right],\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\partial \mathcal{L}}{\partial \theta }}={\displaystyle \frac{2}{{\sigma }^2}}\Re \left[{\mathbf{S}}^H{\mathbf{H}}_s^H\left({\displaystyle \frac{\partial \left(\alpha {\mathbf{H}}_s\mathbf{S}\right)}{\partial \theta }}\right)\right].\hfill \end{array}$$

(19)

The final expression can use the chain rule of derivative with respect to $\theta$ as follows:

$${\displaystyle \frac{\partial \left(\alpha {\mathbf{H}}_s\mathbf{S}\right)}{\partial \theta }}=\left({\displaystyle \frac{\partial \alpha }{\partial \theta }}\right){\mathbf{H}}_s\mathbf{S}=\left|\alpha \right|\left[-sin\left(\theta \right){\mathbf{H}}_s\mathbf{S}+cos\left(\theta \right){\mathbf{H}}_s\mathbf{S}\right].$$

(20)

The Fisher information matrix for the parameters ${\alpha }_r$, ${\alpha }_i$, and $\theta$ is the expectation of the second derivatives of the log-likelihood function. Since the noise $n_s$ is Gaussian, the Fisher information for each parameter is given by

$$\mathbf{F}\left(\alpha \right)=-\mathbb{E}\left[{\displaystyle \frac{{\partial }^2\mathcal{L}}{\partial {\alpha }^2}}\right]=CRB{\left(\alpha \right)}^{-1}.$$

(21)

The CRB for the parameters ${\alpha }_r$, ${\alpha }_i$, and $\theta$ is the inverse of the Fisher information matrix. For the imperfect CSI case, the calculation of the concrete Fisher information matrix is provided in Appendix B. The CRB for $\theta$ can be expressed as

$${\mathrm{CRB}}_{\theta }={\left[F_{\theta \theta }-{\displaystyle \frac{F_{\theta {\alpha }_r}{F}_{{\alpha }_r\theta }}{F_{{\alpha }_r{\alpha }_r}}}-{\displaystyle \frac{F_{\theta {\alpha }_i}{F}_{{\alpha }_i\theta }}{F_{{\alpha }_i{\alpha }_i}}}\right]}^{-1}.$$

(22)

Now, we can formulate the complete optimization problem as follows:

$$\begin{array}{cc}\hfill \underset{{\mathbf{w}}_k,\Delta {\mathbf{h}}_k}{maximize}& {\displaystyle \frac{1}{K}}\sum _{k=1}^K{\widehat{R}}_k,\hfill \end{array}$$

(23a)

$$\begin{array}{cc}\hfill \mathrm{subject}\mathrm{to}& {∥{\mathbf{w}}_k∥}^2\le P_{max}\forall k,,\hfill \end{array}$$

(23b)

$$\begin{array}{cc}\hfill & {SINR}_k\ge {\gamma }_k^{min}\forall k,\hfill \end{array}$$

(23c)

$$\begin{array}{cc}\hfill & E_k\ge E_k^{min}\forall k,\hfill \end{array}$$

(23d)

$$\begin{array}{cc}\hfill & {\mathrm{CRB}}_{\theta }\le ϵ.\hfill \end{array}$$

(23e)

This optimization problem is a non-convex optimization problem because of the logarithmic terms in the objective function and the quadratic power and energy constraints. Implementing robust beamforming in systems with imperfect CSI is particularly challenging. It is vital to combine multiple objectives that ensure reliable environmental sensing performance under varying interference and environmental conditions, while also maintaining high data rates, low latency for diverse applications, and efficient wireless power transfer with minimal energy overheads. These challenges are further compounded by dynamic real-world conditions, such as user mobility and fluctuating interference levels. Traditional static optimization methods often fall short in addressing these complexities, leading to suboptimal system performance. To tackle these issues, we propose a structured approach that formulates the problem as an adaptive reinforcement learning task and leverages deep learning models to handle the system’s complexity [43].

4. Proposed DRL Framework for ISCPT System

In this section, we present a novel DRL framework tailored for optimizing beamforming strategies in the ISCPT system. The main goal is to maximize the information transmission rate while respecting multiple system constraints, such as the maximum transmit power, QoS, and energy-harvesting requirements. DRL is particularly well suited for this task due to its ability to learn complex, adaptive policies in environments where both the state and action spaces are large and continuous. DRL has shown promising results in addressing optimization problems with dynamic and uncertain environments, making it an ideal candidate for wireless communication systems with imperfect CSI. Traditional optimization methods, although effective in simpler, static systems, struggle with the real-time adaptability required in complex and fluctuating environments. The DRL framework, however, allows the learning agent to continually update its policy based on the evolving system state, making it highly adaptable to changing channel conditions [27].

The objective of our optimization, as defined in the objective function (23), is to find an optimal set of beamforming vectors ${\mathbf{w}}_k$ that maximize the total information transmission rate of the system, while ensuring the satisfaction of various operational constraints. These constraints are particularly challenging to incorporate into traditional optimization methods but can be effectively handled within the DRL framework. Given the dynamic nature of the ISCPT system, where the beamforming decisions must adapt continuously over time, we propose the use of the deep deterministic policy gradient (DDPG) algorithm to optimize the beamforming vectors in a way that maximizes the overall system performance.

As shown in Figure 2, the DRL framework operates in a continuous interaction loop between the agent and the environment. The BS acts as the learning agent, continuously adjusting its beamforming strategy in response to the feedback from the wireless channel. The state information, which includes the channel conditions, power levels, and interference, is collected at each time step. Using this information, the agent learns to select the optimal beamforming vectors that maximize the data rate while respecting system constraints such as SINR, energy-harvesting requirements, and other performance metrics. The DDPG algorithm, which is designed to operate in high-dimensional continuous action spaces, is employed to update the agent’s policy through a combination of actor–critic networks [44].

4.1. Action and State Space Definition

To facilitate the learning process, it is crucial to properly define the action and state spaces in the context of the DRL framework. The action space ${\mathbf{a}}_t$ refers to the beamforming vectors ${\mathbf{w}}_k$ that the agent chooses at each time step. Since the beamforming vectors are complex-valued and high-dimensional, they form a continuous action space. At each time step t, the action vector is defined as

$${\mathbf{a}}_t=\left[{\mathbf{w}}_{t,1},{\mathbf{w}}_{t,2},\cdots ,{\mathbf{w}}_{t,K},\right].$$

(24)

The state space ${\mathbf{s}}_t$, on the other hand, captures all the necessary information about the environment that the agent uses to make its decisions. In the ISCPT system, the state includes information about the channel conditions, such as the channel gains ${\widehat{\mathbf{h}}}_{t,k}$ for each device, as well as the previous beamforming actions ${\mathbf{w}}_{t-1,k}$ taken by the agent. This information allows the agent to learn the relationship between past actions and their impact on the system performance. The state vector can be expressed as

$${\mathbf{s}}_t=\left[\underset{\mathrm{the}\mathrm{current}\mathrm{state}{\mathbf{s}}^{\prime }}{\underbrace{{\widehat{\mathbf{h}}}_{t,1},{\widehat{\mathbf{h}}}_{t,2},\dots ,{\widehat{\mathbf{h}}}_{t,K},{\widehat{\mathbf{h}}}_{t,s}}},\underset{\mathrm{the}\mathrm{last}\mathrm{action}{\mathbf{a}}_{t-1}}{\underbrace{{\mathbf{w}}_{t-1,1},{\mathbf{w}}_{t-1,2},\dots ,{\mathbf{w}}_{t-1,K}}}\right].$$

(25)

The state space is a Markov process, meaning that the future state only depends on the current state and the action taken. This property allows the DRL agent to make decisions based on the current conditions and adapt to changes in the wireless channel over time.

4.2. Reward Function Design

The design of the reward function is critical for guiding the learning process and ensuring that the agent maximizes the desired system objective. In our case, the objective is to maximize the average achievable rate while adhering to various operational constraints. The reward function is closely tied to the system’s sum rate, as well as penalties for constraint violations. The reward at each time step is designed to be a weighted combination of the achievable rate and penalties. The total reward is the sum of the achievable rates for all devices:

$$r_t=\sum _{k=1}^K{R}_k=\sum _{k=1}^K{log}_2\left(1+{\mathrm{SINR}}_k\right).$$

(26)

In addition to maximizing the sum rate in (23b), the reward function includes penalty terms for violating constraints. For example, the SINR for each device should exceed a minimum threshold ${\gamma }_k$, the energy harvested by the devices should be above a minimum value $E_{min}$, and the CRB for DoA estimation should remain within a predefined limit $ϵ$. The reward function can thus be formulated as

$$\begin{array}{cc}\hfill \sum _{k=1}^K{R}_k=\underset{{\mathbf{w}}_k,\Delta {\mathbf{h}}_k}{maximize}& \sum _{k=1}^K{\widehat{R}}_k-{\lambda }_1\sum _{k=1}^K\left({SINR}_k-{\gamma }_k\right)\hfill \\ \hfill & -{\lambda }_2\sum _{k=1}^K\left({\mathbf{E}}_k-E_{min}\right)-{\lambda }_3\left({\mathrm{CRB}}_{\theta }-ϵ\right),\hfill \end{array}$$

(27)

where ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are positive penalty parameters that balance the trade-off between maximizing the rate and satisfying the constraints. These penalty terms help ensure that the optimization respects the necessary system requirements, such as QoS, energy constraints, and estimation accuracy.

4.3. Actor-Critic Networks

The DRL framework employs two neural networks: the actor and the critic. The actor network $\pi \left(\mathbf{s};{\theta }_a\right)$ outputs a deterministic action (beamforming vector) given the current state ${\mathbf{s}}_t$, while the critic network $Q\left(\mathbf{s},\mathbf{a};{\theta }_c\right)$ evaluates the quality of the action chosen by the actor. The critic estimates the expected cumulative reward (Q-value) for taking action ${\mathbf{a}}_t$ in state ${\mathbf{s}}_t$, based on the Bellman equation [45]. The goal of training the actor is to maximize the Q-value, which corresponds to maximizing the average rate (reward). The critic helps to evaluate how good a particular beamforming action is [46].

The objective of the critic network is to minimize the Bellman error, which is the difference between the current Q-value and the target Q-value. The Bellman error is defined as

$$\mathbf{L}\left({\theta }^Q\right)=\mathbb{E}\left[{\left(Q\left({\mathbf{s}}_t,{\mathbf{a}}_t\mid {\theta }^Q\right)-\left(r_t+\gamma Q^{\prime }\left({\mathbf{s}}_{t+1},{\mathbf{a}}_{t+1}\mid {\theta }^{Q^{\prime }}\right)\right)\right)}^2\right],$$

(28)

where $Q^{\prime }$ is the target critic network, and $\gamma$ is the discount factor that represents the importance of future rewards.

The actor network is trained to maximize the Q-value output by the critic, i.e., to select actions (beamforming vectors) that maximize the expected cumulative reward. The update rule for the actor is given by

$${\nabla }_{{\theta }^{\pi }}J\left({\theta }^{\pi }\right)=\mathbb{E}\left[{\nabla }_{{\mathbf{a}}_t}Q\left({\mathbf{s}}_t,{\mathbf{a}}_t\mid {\theta }^Q\right){\nabla }_{{\theta }^{\pi }}\pi \left({\mathbf{s}}_t\mid {\theta }^{\pi }\right)\right],$$

(29)

where $J\left({\theta }^{\pi }\right)$ is the objective function for the actor.

To stabilize training, both the actor and critic networks are updated using soft target updates. The target networks $\pi \left(\mathbf{s};{\theta }_{a^{\prime }}\right)$ and $Q\left(\mathbf{s},\mathbf{a};{\theta }_{c^{\prime }}\right)$ are updated periodically:

$$\begin{array}{cc}\hfill & {\theta }^{{\pi }^{\prime }}\leftarrow \delta {\theta }^{\pi }+(1-\delta ){\theta }^{{\pi }^{\prime }},\hfill \\ \hfill & {\theta }^{Q^{\prime }}\leftarrow \delta {\theta }^Q+(1-\delta ){\theta }^{Q^{\prime }}.\hfill \end{array}$$

(30)

where $\delta$ is a small constant that controls the rate of target network updates. The agent stores its experiences $\left({\mathbf{s}}_t,{\mathbf{a}}_t,r_t,{\mathbf{s}}_{t+1}\right)$ in a replay buffer $\mathcal{D}$. In the procedure of training, the agent samples mini-batches $\mathcal{M}$ from the buffer to break the correlation between consecutive experiences. At each time step, the actor network $\pi \left(\mathbf{s};{\theta }_a\right)$ is used to select an action $a_t$. To encourage exploration, Ornstein–Uhlenbeck noise is added to the action, making the action selection stochastic [47].

4.4. Constraint Handling with Neural Networks

As mentioned earlier, neural networks can be employed to satisfy the constraint conditions in (23b). This is a clever approach that utilizes the transformation of Euler’s formula to meet the constraint of maintaining constant beam values, while also addressing the requirement for complex variables [48]. Only half the number of neurons is required to represent the corresponding beamforming vector. To satisfy the maximum power constraint $P_{m}ax$, a softmax activation function is added, which allows for free adjustment within the given limitations. The corresponding formula can be expressed as follows:

$${\mathbf{w}}_k=\left({\displaystyle \frac{e^{{\mu }_{\mathbf{2}}^{\top }{\mathbf{a}}_k}}{{\sum }_{i=1}^K{\mu }_{\mathbf{2}}^{\top }{\mathbf{a}}_i}}\right)\times e^{\mathrm{j}·2\pi {\mu }_{\mathbf{1}}^{\top }{\mathbf{a}}_k},$$

(31)

where ${\mu }_1$ and ${\mu }_2$ represent the results of the neuronal output before Euler’s formula and softmax’s activation function, respectively. This method enforces constant beam values while maintaining flexibility for adjusting the beamforming vectors within the required power and phase bounds.

4.5. DDPG Algorithm for ISCPT System Optimization

The DDPG algorithm is a model-free, off-policy reinforcement learning approach designed for continuous action spaces, making it ideal for our problem. The algorithm employs two neural networks: the actor network $\pi \left(\mathbf{s};{\theta }_a\right)$, which produces deterministic actions (beamforming vectors), and the critic network $Q\left(\mathbf{s},\mathbf{a};{\theta }_c\right)$, which estimates the expected return for a given state-action pair. The pseudo-code for the DDPG algorithm in the context of ISCPT system optimization is as follows (Algorithm 1):

Algorithm 1 DDPG algorithm for ISCPT system optimization.

1: Initialize actor network $\pi (s;{\theta }_a)$ and critic network $Q(s,a;{\theta }_c)$ with random parameters. 2: Initialize target networks $\pi (s;{\theta }_{a^{\prime }})$ and $Q(s,a;{\theta }_{c^{\prime }})$ as copies of the actor and critic networks, respectively. 3: Initialize replay buffer $\mathcal{D}$. 4: Set the exploration noise parameters Ornstein–Uhlenbeck noise). 5: for each episode do 6:     Initialize the state $s_0$ (e.g., environment configuration at time step 0). 7:     for each time step t in the episode do 8:   Select action $a_t=\pi \left(s_t\right)+{\mathcal{N}}_t$ (exploration with noise). 9:   Execute action $a_t$ in the environment to obtain next state $s_{t+1}$ and reward $r_t$. 10:   Store the transition $(s_t,a_t,r_t,s_{t+1})$ in the replay buffer $\mathcal{D}$. 11:   Sample a mini-batch $\mathcal{M}$ of transitions $(s_t,a_t,r_t,s_{t+1})$ from $\mathcal{D}$. 12:   Update critic network by minimizing the loss in (28). 13:   Update actor network by maximizing the policy gradient in (29). 14:   Update target networks with soft updates in (30). 15:     end for 16: end for

This process allows the agent to gradually improve its beamforming strategy through exploration and exploitation. The learning process is repeated across multiple episodes, where each episode represents a new set of channel conditions and environmental configurations. Through iterative training, the agent learns an optimal policy that maximizes the total transmission rate while respecting system constraints.

It is worth noting that the proposed DRL framework demonstrates remarkable robustness, particularly in mitigating the effects of CSI errors, such as $\Delta {\mathbf{h}}_k$. Through the combination of online learning, adaptive policy updates, experience replay, multi-task learning, and robust reward function design, the agent is able to continuously adjust its beamforming strategy to accommodate imperfect CSI. This ensures that the system maintains optimal performance even in the presence of channel estimation inaccuracies. Furthermore, the use of neural network regularization and data augmentation techniques enhances the robustness of the model, enabling it to generalize effectively to various real-world scenarios. As a result, the DRL-based approach offers a scalable and reliable solution to the complex optimization problem of robust beamforming in ISCPT systems, overcoming the challenges posed by CSI imperfections.

5. Simulation Results

In this section, we present the simulation results for the DRL-based ISCPT system. The results include a detailed analysis of the system’s performance under varying conditions, with particular emphasis on the effects of learning rates, transmit power, SINR, energy harvesting, and CRB on the system’s efficiency and robustness. The parameters of the channel model used in the simulations are summarized in

Table 1. Channel model parameters for ISCPT system.

Symbol	Parameter	Value
F	Carrier frequency	4 GHz
$N_t$	Transmit antenna	64
$a_G$	Path loss exponents	3
$\kappa$	Rician factor	2.2
$\alpha$	Target reflection coefficient	0.01
$\eta$	Energy conversion efficiency	0.7
${\sigma }^2$	Noise Power	−70 dBm

. We assume a flat-fading channel with AWGN and consider typical parameters for a wireless communication system operating in the C-band at 4 GHz. The IoT devices are randomly deployed within a 50 m radius of the BS, covering a sensing target, and the poss loss is configured as $-20$ dB with the reference distance 1 m.

In the DRL model for ISCPT systems, several key neural network parameters need to be optimized. The model is trained and tested on a system equipped with a CPU (AMD Ryzen 5 5600X) and a GPU (NVIDIA RTX 3070). These parameters control the training process of the neural network, which aims to optimize beamforming strategies while accounting for power and spectrum constraints. Critical hyperparameters, such as the learning rate, discount factor, batch size, and network architecture, play a crucial role in determining the network’s convergence and performance in multi-objective optimization. These parameters are summarized in

Table 2. DDPG algorithm for ISCPT system parameters.

Symbol	Parameter	Value
$N_{epi}$	The number of episodes	1000
$T_{max}$	The number of maxstep in each episode	100
${\alpha }_a$	Learning rate for actor network	0.0001
${\alpha }_c$	Learning rate for critic network	0.0001
${\delta }_a$	Target actor network update rate	0.0001
${\delta }_c$	Target critic network update rate	0.0001
$\lambda$	Decaying rate	0.00001
$\gamma$	Discount factor	0.95
$\mathcal{M}$	The size of mini-batch	32
$\mathcal{D}$	Replay buffer size	1,000,000
U	The number of steps’ synchronization	1

.

A dataset consisting of 1000 samples is used to train the DDPG agent, with 100 samples reserved for validation. The training set is utilized to optimize the beamforming strategy while simultaneously addressing constraints related to power allocation and spectrum management. The validation set is employed to assess the generalization performance of the trained model, ensuring its robustness on statistical data. Both the actor and critic networks are implemented as three-layer deep neural networks (DNNs) to model the continuous control policy and value function, respectively, within the DDPG framework. The actor network is designed to output the beamforming vector ${\mathbf{w}}_k$ for each device, given a specific state. It consists of three fully connected layers, each comprising 512 neurons. The ReLU activation function is applied to all layers to introduce non-linearity, enabling the network to approximate complex mappings from state to action effectively. Similarly, the critic network evaluates the quality of the actions selected by the actor by estimating the Q-value. The critic shares the same architecture as the actor network, comprising three fully connected layers with 512 neurons per layer and ReLU activations. This structure enhances the learning capacity and flexibility of the model, facilitating effective training in complex environments.

Figure 3 illustrates the learning performance of the proposed DRL-based framework under different learning rates, specifically $0.01$, $0.001$, $0.0001$, and $0.00001$. At the start of training, the model receives negative rewards, reflecting its initial difficulty in adapting to the environment’s dynamics. During the first 100 episodes, the system encounters a penalty term in (27), causing the rewards to oscillate within the negative range. This negative oscillation underscores the challenges the agent faces in exploring the solution space effectively during the early training phase. As the model gradually adapts to the environment and learns the underlying channel conditions, the rewards steadily improve, transitioning into a positive incentive phase. The results demonstrate that the learning rate plays a critical role in influencing the model’s training performance. Among the tested values, the learning rate of $0.0001$ achieves the best performance, striking an optimal balance between convergence speed and training stability. In contrast, excessively low learning rates impede the model’s ability to generalize, resulting in underfitting and suboptimal performance.

As shown in Figure 4, the system with perfect CSI demonstrates faster and more stable convergence, with the rewards steadily increasing and eventually stabilizing at a higher value. In comparison, the system with imperfect CSI experiences a slight degradation in performance. Specifically, the imperfect CSI condition results in slower learning, as evidenced by the delayed rise in rewards. The oscillations are more pronounced, and the overall learning process takes longer to reach the positive reward phase compared to the perfect CSI scenario. This suggests that the uncertainty introduced by imperfect CSI compromises the model’s ability to robustly adapt to the environment, thereby slowing down the learning process and affecting performance stability.

Next, we evaluate the average rate of the ISCPT system as a function of the maximum transmit power, which ranges from 0 dBm to 30 dBm. The results, presented in Figure 5, demonstrate how the average rate varies with transmit power. Here, DRL refers to the method proposed in this paper, while semidefinite relaxation (SDR) represents a optimal solution as explored in previous studies [21]. ZF indicates beamforming via the zero-forcing algorithm, and random denotes random beamforming.

The experimental results demonstrate that as the transmit power increases, the average rate of all algorithms improves due to enhanced signal strength and resource utilization. Among the algorithms tested, the DRL-based approach consistently outperforms SDR, ZF, and Random under both perfect $({\sigma }^2=0)$ and imperfect $({\sigma }^2=0.01)$ CSI conditions, and it provides a good stability compared to other algorithms even with the channel error. DRL exhibits exceptional robustness, maintaining high performance in the presence of CSI imperfections, which significantly degrade the performance of other algorithms. While SDR performs well under perfect CSI, it suffers from a substantial performance decline with imperfect CSI due to its reliance on precise channel information. ZF offers moderate performance but is limited by its sensitivity to channel estimation errors. Random beamforming predictably performs the worst, highlighting the necessity of optimization strategies.

In practical systems, it is crucial to adapt to various external environmental changes, including delays, frequency offsets, and inherent errors in channel estimation, which make it difficult to guarantee the accuracy of CSI. Therefore, we need to adopt a method that is both adaptive and stable. As shown in Figure 6, the DRL method demonstrates significant robustness compared to other approaches under varying levels of uncertainty. To further emphasize the robustness of DRL, we analyze the percentage decrease in the average rate as CSI uncertainty increases. Specifically, as the uncertainty rises from 0 to $0.6$, the rate reduction for DRL is notably smaller than for SDR, ZF, and Random methods. The results clearly show that the DRL method exhibits superior robustness under increasing uncertainty. At $\sigma =0.6$, DRL retains $80\%$ of its original rate, while SDR retains only $64.3\%$, ZF $58.3\%$, and Random $48.8\%$. This significant difference highlights the advantage of DRL in adapting to channel uncertainty and maintaining stable system performance, making it a more reliable solution in scenarios with uncertain or imperfect CSI.

From a computational complexity perspective, DRL, despite its superior performance and robustness, has a higher computational overhead due to its iterative learning process and the training phase, which involves neural network updates and extensive interactions with the environment. The complexity is given by $\mathcal{O}\left(2N_{epi}{T}_{max}({\sum }_{l_{=}1}^L{n}_{l-1}·n_l+n_l·b_l)\right)$, where n and b represent the amounts of neurons and bias, respectively. However, once trained, the DRL model operates with relatively low complexity during inference, making it suitable for real-time applications. In contrast, SDR relies on solving semidefinite programming (SDP) problems with a complexity of $\mathcal{O}\left(t{\left(N_t(K+1)\right)}^{3.5}\right)$, which makes it computationally prohibitive for large-scale systems. These results underscore the trade-off between computational complexity and performance, with DRL emerging as the most robust and efficient approach for dynamic, real-time scenarios.

Further, we compare the average IoT rate for varying numbers of devices. Figure 7 illustrates the influence of different deviations of imperfect CSI on the system’s performance, reflected by the average rate for different numbers of IoT devices. As $\sigma$ increases, representing greater uncertainty in the CSI, the system’s performance gradually deteriorates. Specifically, higher $\sigma$ values result in increased instability in the optimization problem, making it more challenging to achieve stable performance. These results highlight the superior robustness and performance of the DRL-based beamforming algorithm under imperfect CSI conditions. By continuously adjusting its policy based on environmental feedback, the DRL model effectively manages uncertainty and adapts to dynamic changes in the wireless channel. In contrast, traditional beamforming algorithms, which rely on static or approximate models, are more prone to performance degradation in high-uncertainty environments.

The results further reinforce the superior robustness and adaptability of the DRL-based beamforming algorithm, particularly in environments where CSI is imperfect or subject to noise. Unlike traditional methods, which rely on static or approximate models and are more sensitive to inaccuracies in CSI, the DRL-based model can continuously adapt to changing conditions, effectively mitigating the effects of imperfect information. This capability positions DRL as a promising solution for next-generation wireless networks, where channel conditions are often dynamic and unpredictable.

Then, we investigate the impact of the minimum energy-harvesting requirement on the transmission rate. Figure 8 and Figure 9 compare the average rates achieved by different algorithms without and with CSI error. The results clearly show that the DRL-based beamforming algorithm significantly outperforms other methods, particularly in scenarios where energy-harvesting constraints are present. In ISCPT systems, energy fluctuations can directly impact the quality of communication, as the available energy is crucial for powering transmitters, receivers, and related electronics. A decrease in energy availability can result in lower transmission power, leading to signal degradation and performance loss. The DRL-based beamforming approach, by continuously adapting to the available energy, ensures that the system can maintain a high level of performance even in the presence of energy fluctuations.

Finally, we analyze the influence of transmit power and energy harvesting on the CRB for angle estimation, ${\mathrm{CRB}}_{\theta }$, as shown in Figure 10. Both transmit power and energy harvesting contribute significantly to reducing the CRB, thereby improving the accuracy of angle estimation. This, in turn, enhances the overall system performance by providing more precise angle information for beamforming. The results underscore the critical role of transmit power and energy harvesting in optimizing the performance of the system, particularly in scenarios requiring accurate angle estimation for effective beamforming.

6. Conclusions

In this section, we presented the simulation results for a DRL-based ISCPT system. The results demonstrated the effects of various parameters, such as transmit power, energy harvesting, SINR, and CRB, on the system’s performance. The DRL algorithm successfully adapted to these parameters, improving the system’s efficiency and robustness under dynamic conditions. Future work will focus on further optimizing the DRL algorithm and exploring advanced learning techniques to enhance system performance. Additionally, we plan to investigate non-linear energy-harvesting models to better capture real-world energy transfer dynamics. Furthermore, research will be extended to hybrid beamforming techniques, aiming to achieve more efficient and flexible beamforming designs that balance complexity and performance in integrated sensing and communication systems.