Smart Energy Borrowing and Relaying in Wireless-Powered Networks: A Deep Reinforcement Learning Approach

Abstract

Wireless energy harvesting (EH) communication has long been considered a sustainable networking solution. However, it has been limited in efficiency, which has been a major obstacle. Recently, strategies such as energy relaying and borrowing have been explored to overcome these difficulties and provide long-range wireless sensor connectivity. In this article, we examine the reliability of a wireless-powered communication network by maximizing the net bit rate. To accomplish our goal, we focus on enhancing the performance of hybrid access points and information sources by optimizing their transmit power. Additionally, we aim to maximize the use of harvested energy, by using energy-harvesting relays for both information transmission and energy relaying. However, this optimization problem is complex, as it involves non-convex variables and requires combinatorial relay selection indicator optimization for decode and forward (DF) relaying. To simplify this problem, we utilize the Markov decision process and deep reinforcement learning framework based on the deep deterministic policy gradient algorithm. This approach enables us to tackle this non-tractable problem, which conventional convex optimization techniques would have difficulty solving in complex problem environments. The proposed algorithm significantly improved the end-to-end net bit rate of the smart energy borrowing and relaying EH system by $13.22\%, 27.57\%,$ and $14.12\%$ compared to the benchmark algorithm based on borrowing energy with an adaptive reward for Quadrature Phase Shift Keying, 8-PSK, and 16-Quadrature amplitude modulation schemes, respectively.

1. Introduction

The deployment of ultra-low-power electronic sensors has increased significantly with the advanced wireless communication networks [1]. These sensors are used for various data collection and signal processing applications of the Internet of Things (IoT) domain [2]. However, the lifetime of the IoT networks is limited by the battery constraints of the individual sensor devices. To address this issue, dedicated radio frequency energy transfer (RF-ET) ensures uninterrupted long-duration network operation by providing controllable on-demand energy replenishment of sensor devices [3], which includes long-range beamforming capabilities for energy harvesting (EH), joint energy, and information transfer provisioning over the same signal [4]. This introduces a research paradigm on wireless-powered communication networks (WPCN), in which the uplink information transfer (IT) is governed by downlink ET from the hybrid access point (HAP).

The RF-EH system can operate independently in remote and harsh locations, but it has some limitations. These include low energy sensitivity, low rectification efficiency at lower input power, high attenuation due to path loss, and energy dispersion loss [5]. Additionally, the energy harvested from ambient sources cannot be accurately predicted dynamically because the channel conditions are constantly changing [6]. Therefore, it is necessary to have a backup power supply, such as a power grid (PG), to facilitate energy cooperation. This secondary power supply can efficiently handle energy transactions when EH devices require additional power for uninterrupted WPCN operation. This paper investigates the artificial intelligence (AI) enabled smart energy sharing and relaying in cooperative WPCN to enhance the end-to-end system performance.

1.1. Related Works

Several studies, including those referenced in citations [7,8,9,10,11,12,13], have explored implementing autonomous cooperative energy harvesting (EH) techniques with unknown channel gains. These techniques involve energy-constrained sensor devices transmitting information using harvested energy in wireless power transfer networks (WPCN). For instance, one study proposed an optimization model in [7] to maximize two-hop radio frequency energy transfer efficiency with optimal relay placement. In [8], the authors maximized the overall bit rate by optimizing time and power allocation for downlink energy transfer and uplink information transfer and relaying. Another study by Chen et al. in [9] approximated the average throughput for wireless-powered cooperative networks using the harvest-then-cooperate protocol. In addition to fixed relaying approaches in [8,9], an adaptive transmission protocol in [10] dynamically determines whether the information source (IS) should access the point (AP) directly or cooperatively with relays based on estimated channel state information (CSI). Beamforming optimization was performed in [11] to maximize received power for evaluating the performance of relay-assisted schemes under EH efficiency constraints. In [12], a cooperative relaying system was developed to improve the quality of experience (QoE) for cell-edge users. In [13], Wei et al. proposed wireless power transfer (WPT) to enhance spectral efficiency (SE) by jointly adjusting time slot duration, subcarriers, and the transmit power of the source and relay. However, the harvested energy from WPT at sensor device batteries cannot transmit data over long distances. Therefore, energy cooperation and sharing strategies are necessary to overcome dynamic green energy arrival conditions for perpetual WPCN operation.

In network optimization, ref. [14] proposed a method to minimize network delay through simplified energy management and conservation constraints for fixed data and energy routing topologies. Meanwhile, ref. [15] explored various energy-sharing mechanisms among multiple EH devices within the network. When data transmission is possible, but there is insufficient energy in the device battery, external energy supply from nearby secondary power sources must be considered. Ref. [16] addressed this issue by examining the external energy supply provided by PG to EH devices in WPCN. In contrast, ref. [17] proposed that EH devices borrow energy from PG for information transmission and return it with additional interest as a reward. Sun et al. developed a schedule [18,19,20] to maximize system throughput through energy borrowing and returning. However, these approaches rely on predefined statistical parameters and dynamics, whereas in reality, channel gains and harvested energy are subject to random variation. Therefore, a decision-making deep reinforcement learning (DRL) algorithm is needed to determine current network parameters based on previously gained knowledge of the environment.

Wireless network management has recently seen an increase in deep reinforcement learning (DRL) use as part of machine learning (ML) due to its decision-making capabilities through a trial-and-error approach. The sophisticated combination of neural networks (NNs) in DRL makes it ideal for handling complex situations with high-dimensional problems. The authors of [21] developed an NN model to extract key rates with high reliability, considerable tightness, and great efficiency. Qie et al. [22] used DRL based on the deep deterministic policy gradient (DDPG) algorithm to develop an optimal energy management strategy for an EH wireless network. Resource allocation policies were also developed using DRL in [23] to maximize achievable throughput, considering EH, causal information of the battery state, and channel gains. DRL based on borrowing energy with an adaptive reward (BEAR) algorithm was proposed in [24] to optimize energy borrowing from a secondary power source and efficient data transfer utilizing harvested energy. In [25], cooperative communications with adaptive relay selection in a wireless sensor network were investigated as a Markov decision process (MDP), and deep-Q-network (DQN) was proposed to evaluate network performance. DRL based on the actor-critic method was used in [26,27] to maximize the energy efficiency (EE) of a heterogeneous network for optimal user scheduling and resource allocation. However, the impact of energy scheduling and transmit power allocation of IS maximizing the transmission rate of an energy borrowing and relaying aided WPCN is still a research gap, needs to be explored.

1.2. Motivation and Key Contributions

In current EH cooperative relaying techniques for WPCN, such as those mentioned in references [10,11,12,13], having complete knowledge of the CSI at the receiver is necessary. However, such simplified channel models fail to account for the dynamic communication environment, which is crucial for optimizing resource allocation and analyzing system performance. Alternative approaches, like energy scheduling and management methods, have been adopted in references [15,16,17,18], which assume a practical probability distribution model for energy arrival. However, these methods do not consider optimal power allocation, energy borrowing, and returning schedules for harvested energy relaying for IT. Only the authors of reference [21,23,24] have considered a practical EH channel model, where a single EH relay wirelessly transfers energy to the IS. However, this model does not apply to multiple EH relay-assisted WPCN, where maximizing throughput and minimizing transmission delay are essential. Our article addresses these issues by exploring the RF-powered joint information and energy relaying (JIER) protocol for WPCN, which efficiently allocates resources to maximize system reliability. Specifically, we consider an EH-HAP that effectively manages energy transactions with the PG and transmits RF energy to the IS through multiple EH relays. It then receives information from the source via uplink DF-relay-assisted channels. This timely investigation focuses on maximizing the efficacy of EH in WPCN by optimally utilising the available energy resources by enabling intelligent energy relaying and borrowing. Our specific contribution is four-fold, which can be summarized as follows:

• Considering a novel smart energy borrowing and relaying-enabled EH communication scenario, we investigate the end-to-end net bit rate maximization problem in WPCN. Here, we jointly optimize the transmit power of HAP and IS, fractions of harvested energy transmitted by the relays, and the relay selection indicators for DF relaying within the operational time.
• We decompose the formulated problem into multi-period decision-making steps using MDP. Specifically, we propose a nontrivial transformation where the state represents the current onboard battery energy level and the instantaneous channel gains. In contrast, the corresponding action indicates the transmit power allocations. Since HAP selects the relay based on the maximum achievable signal-to-noise ratio (SNR) among all the relays for receiving the information, the instantaneous transmission rate attained by HAP is treated as an immediate reward.
• We observed that the initial joint optimization problem was analytically intractable to be solved using traditional convex optimization techniques due to the realistic parameter settings of complex communication environments. Therefore, we suggest a DRL framework using the DDPG algorithm to train the DNN model, enabling the system to discover the best policy in an unfamiliar communication environment. The proposed approach determines the current policy using the Q-value for all state-action pairs. Additionally, we have examined the convergence and complexity of the proposed algorithm to improve the learning process.
• Our analysis is validated by the extensive simulation results, which offer valuable insights into the impact of key system parameters on optimal decision-making. Additionally, we compared the performance of various modulation schemes, including QPSK, 8-phase shift keying (PSK), and 16-quadrature amplitude modulation (QAM), that leverage the same algorithm. Our resource allocation technique improves the net bit rate of the system compared to the BEAR-based benchmark algorithm.

The rest of the article is organized as follows: Section 2 presents the system model of efficient WPCN (Figure 1). Section 3 elaborates on the mathematical formulation of our objective. Section 4 introduces the DRL approach and the proposed DDPG algorithm for resource allocation corresponding to the optimal policy. Section 5 gives insights into extensive simulation results for performance evaluation. Finally, Section 6 outlines the conclusion, where the list of references is located at the end of this article.

We use bold letters to denote vector quantity; $\left|.\right|$ represents the magnitude of a complex quantity; $\mathrm{erfc}\left(.\right)$ stands for complementary error function; $\mathcal{CN}\left(\lambda ,\omega \right)$ indicates a circularly symmetric complex Gaussian random variable with mean $\lambda$ and variance $\omega$; $\mathbb{E}\left[.\right]$ is the expectation operator; and $\mathcal{O}\left(.\right)$ denotes the big-O notation.

2. System Model

Consider a JIER-assisted WPCN consisting of a PG and three types of transceiver modules such as a HAP, two RF-EH relays, and an IS, as depicted in Figure 1. Here, HAP is harvesting energy from ambient sources, such as RF power signals, and subsequently stores the harvested energy in its limited charge storage capable internal battery. We assume that IS can only harvest energy into its small-size battery storage from the RF energy transfer mode as it has no direct external energy supply. When IS accumulates sufficient energy, it can transmit information to relays and HAP. Furthermore, HAP can borrow the required energy from the PG while it faces the potential energy shortage for RF energy transmission towards relays and IS. To reduce PG’s additional burden, HAP returns the borrowed energy to PG along with interest based on the borrowing price, where no energy leakage is considered within the energy transmission deadline [28]. For ease of calculation, we subdivide the operational period into N equally spaced discrete time slots of duration $\delta$ each. Let, at the nth $\left(n\in \mathcal{N}=\left\{1,2,\dots ,N\right\}\right)$ time slot, the battery energy level of HAP is $B\left[n\right]$ and its harvested energy from the ambient sources is $E_H\left[n\right]$, where $E_H\left[n\right]$ follows the Gaussian distribution of mean ${\mu }_H$ and variance ${\sigma }_H^2$. The channel gain coefficient for the communication link between source and destination at the nth time slot also follows the complex Gaussian distribution of mean ${\mu }_h$ and variance ${\sigma }_h^2$ [29].

2.1. JIER Protocol

We consider that HAP can transfer energy to IS by the two RF-EH relays, i.e., ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$ via full-duplex two-hops downlink channel and then IS transmits data to the HAP via uplink half-duplex decode and forward (DF) relaying channel. The entire protocol is divided into four phases as follows:

• Phase 1: Energy is harvested at ${\mathcal{R}}_1$, ${\mathcal{R}}_2$ and the IS for $\left(N-3\right)$ time slots by the directly transmitted RF signals from the HAP.
• Phase 2: Energy is further harvested at the IS in $\left(N-2\right)$th time slot by energy relaying from ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$, where the relays transmit a fraction of the harvested energy from phase 1. Let ${\sigma }_1\left[n\right]$ and ${\sigma }_2\left[n\right]$ be the fractions of the harvested energy transmitted by ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$, respectively.
• Phase 3: At the $\left(N-1\right)$th time slot, the IS directly transmits information to ${\mathcal{R}}_1$, ${\mathcal{R}}_2$ and the HAP using the harvested energy stored in its onboard battery.
• Phase 4: Finally, at Nth time slot, the information is transmitted to the HAP from ${\mathcal{R}}_1$ or ${\mathcal{R}}_2$ via DF relaying by utilizing the remaining harvested energy stored in the relays’ battery. The HAP receives information from a single relay at a time and selects that relay depending on the maximum achievable SNR among all the relays.

2.2. Energy Scheduling

The instantaneous battery energy level of the HAP depends on its current harvesting energy, energy borrowing, and returning of energy to PG, which can be calculated as [28]

$$B\left[n\right]=min\left\{\left(B\left[n-1\right]+E_H\left[n\right]\right),B_{max}\right\}+E_B\left[n\right]-E_R\left[n\right]-\delta P_H\left[n\right].$$

(1)

where $B_{max}$ is the maximum battery capacity of HAP, $E_B\left[n\right]$ represents the instantaneous borrowed energy from PG, $E_R\left[n\right]$ denotes the returned energy to PG at nth time slot, and $P_H\left[n\right]$ is the instantaneous transmit power of HAP.

2.2.1. Energy Borrowing

If HAP’s current energy level is less than its energy consumption at a slot while transmitting with the power of $P_H\left[n\right]$, HAP borrows the required energy from the PG, which can be expressed as [28]

$$E_B\left[n\right]=\left\{\begin{array}{c}\delta P_H\left[n\right]-\left(B\left[n-1\right]+E_H\left[n\right]\right),\mathrm{if}\delta P_H\left[n\right]>\left(B\left[n-1\right]+E_H\left[n\right]\right)\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right..$$

(2)

2.2.2. Energy Returning

Since sometimes HAP borrows the required energy from PG according to (2), it has to be returned to PG along with the interest based on the borrowing price. Hence, HAP returns it by utilizing the harvested energy at future time slots, defined by [24]

$$E_R\left[n\right]=\left\{\begin{array}{c}\varsigma E_E\left[n\right],\mathrm{if}{E}_U\left[n-1\right]>E_E\left[n\right]\hfill \\ \varsigma E_U\left[n-1\right],\mathrm{Otherwise}\hfill \end{array}\right.,$$

(3)

where $E_E\left[n\right]=B\left[n-1\right]+E_H\left[n\right]+E_B\left[n\right]-\delta P_H\left[n\right]$ is the instantaneous excess energy, $\varsigma$ denotes the energy transfer efficiency from HAP to PG, and $E_U\left[n\right]$ indicates the unreturned energy, which is defined at nth time slot as [24]

$$E_U\left[n\right]=\left\{\begin{array}{c}{E}_U\left[n-1\right]+E_B\left[n\right],\mathrm{if}\delta P_H\left[n\right]>\left(B\left[n-1\right]+E_H\left[n\right]\right)\hfill \\ E_U\left[n-1\right]+E_I\left[n\right]-E_R\left[n\right],\mathrm{if}\delta P_H\left[n\right]\le \left(B\left[n-1\right]+E_H\left[n\right]\right)\hfill \\ \mathrm{and}{E}_R\left[n\right]\le \left(E_U\left[n-1\right]+E_I\left[n\right]\right)\hfill \\ 0,\mathrm{Otherwise}\hfill \end{array}\right..$$

(4)

where $E_I\left[n\right]=\varrho E_U\left[n-1\right]$ is the excess returnable energy in the form of interest because of delay in returning the borrowed energy, and $\varrho$ denotes the rate of interest. To restrict excessive energy borrowing, we set the upper bound of unreturned energy $E_{max}^U$ during the entire energy transmission process, expressed by $E_U\left[n\right]<E_{max}^U$.

2.3. RF Energy Harvesting

As we mentioned earlier in Section 2.1 that energy is harvested in ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$ and IS during the first two phases, we employ a linear harvesting model in this case. According to this model, instantaneous stored harvested energies at ${\mathcal{R}}_1$, ${\mathcal{R}}_2$ and IS are calculated as [29]

$$E_H^{{\mathcal{R}}_1}\left[n\right]=\eta \delta P_H\left[n\right]{\left|h_1\left[n\right]\right|}^2,$$

(5)

$$E_H^{{\mathcal{R}}_2}\left[n\right]=\eta \delta P_H\left[n\right]{\left|h_2\left[n\right]\right|}^2,$$

(6)

$$E_H^S\left[n\right]=\eta \delta P_H\left[n\right]{\left|h_3\left[n\right]\right|}^2+{\eta }^2{P}_H\left[n\right]\left({\sigma }_1\left[n\right]{\left|h_1\left[n\right]\right|}^2{\left|h_4\left[n\right]\right|}^2+{\sigma }_2\left[n\right]{\left|h_2\left[n\right]\right|}^2{\left|h_5\left[n\right]\right|}^2\right).$$

(7)

where $\eta$ is RF-EH efficiency, $h_1\left[n\right]$, $h_2\left[n\right]$, $h_3\left[n\right]$, $h_4\left[n\right]$, and $h_5\left[n\right]$ are the instantaneous channel gains between the links of HAP to ${\mathcal{R}}_1$, HAP to ${\mathcal{R}}_2$, HAP to IS, ${\mathcal{R}}_1$ to IS, and ${\mathcal{R}}_2$ to IS respectively.

3. Problem Definition

3.1. DF Relay-Assisted Information Transfer

The performance metric of the proposed WPCN is defined as the instantaneous bit rate, which refers to the number of bits transmitted per unit of time over the communication channel. Various factors, including the modulation scheme, channel bandwidth, coding scheme, and the presence of any error correction or data compression techniques, influence the bit rate that can be expressed at nth time slot as [29]

$$R\left[n\right]=\frac{\xi \rho }{\zeta }{\left(1-P_e\left[n\right]\right)}^{\xi \rho },$$

(8)

where $\xi$ is the number of bits containing a symbol, $\rho$ represents the number of symbols in a packet, $\zeta$ denotes the packet duration, and $P_e\left[n\right]$ is the instantaneous end-to-end bit error rate (BER), which can be expressed for DF relaying system as [28]

$$P_e\left[n\right]=\left\{\begin{array}{c}{\sum }_{d}w\left(m,d\right)\frac{1}{2}\mathrm{erfc}\left(\sqrt{\frac{z\left(m,d\right)\left(P_{{\mathcal{R}}_1}\left[n\right]{\left|h_1\left[n\right]\right|}^2+P_S\left[n\right]{\left|h_4\left[n\right]\right|}^2+P_S\left[n\right]{\left|h_3\left[n\right]\right|}^2\right)}{2N_0}}\right),\hfill \\ \mathrm{if}{\mathcal{R}}_1\mathrm{is}\mathrm{selected}\hfill \\ {\sum }_{d}w\left(m,d\right)\frac{1}{2}\mathrm{erfc}\left(\sqrt{\frac{z\left(m,d\right)\left(P_{{\mathcal{R}}_2}\left[n\right]{\left|h_2\left[n\right]\right|}^2+P_S\left[n\right]{\left|h_5\left[n\right]\right|}^2+P_S\left[n\right]{\left|h_3\left[n\right]\right|}^2\right)}{2N_0}}\right),\hfill \\ \mathrm{if}{\mathcal{R}}_2\mathrm{is}\mathrm{selected}\hfill \\ {\sum }_{d}w\left(m,d\right)\frac{1}{2}\mathrm{erfc}\left(\sqrt{\frac{z\left(m,d\right)\left(P_S\left[n\right]{\left|h_3\left[n\right]\right|}^2\right)}{2N_0}}\right),\mathrm{if}\mathrm{direct}\mathrm{transmission}\mathrm{without}\mathrm{any}\mathrm{relay}\hfill \end{array}\right.,$$

(9)

where $P_S\left[n\right]$ is the instantaneous transmit power of IS, $P_{{\mathcal{R}}_1}\left[n\right]=\left(1-{\sigma }_1\left[n\right]E_H^{{\mathcal{R}}_1}\left[n\right]\right)/\delta$ and $P_{{\mathcal{R}}_2}\left[n\right]=\left(1-{\sigma }_2\left[n\right]E_H^{{\mathcal{R}}_2}\left[n\right]\right)/\delta$ are the instantaneous transmit power of ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$ respectively to relay the information toward HAP, $N_0$ is noise power received at ${\mathcal{R}}_1$, ${\mathcal{R}}_2$, and HAP. $w\left(m,d\right)$ and $z\left(m,d\right)$ are two modulation-related parameters whose values are provided in

Table 1. Different values of modulation parameters for the three considered schemes.

Modulation	$\left(\mathit{w}\left(\mathit{m},\mathit{d}\right),\mathit{z}\left(\mathit{m},\mathit{d}\right)\right)$
QPSK	$\left(w\left(m,0\right),z\left(m,0\right)\right)=\left(1,1\right)$
8-PSK	$\left(w\left(m,0\right),z\left(m,0\right)\right)=\left(\frac{2}{3},2{sin}^2\left(\frac{\pi }{8}\right)\right)$
	$\left(w\left(m,1\right),z\left(m,1\right)\right)=\left(\frac{2}{3},2{sin}^2\left(\frac{3\pi }{8}\right)\right)$
16-QAM	$\left(w\left(m,0\right),z\left(m,0\right)\right)=\left(\frac{3}{4},\frac{1}{5}\right)$
	$\left(w\left(m,0\right),z\left(m,0\right)\right)=\left(\frac{1}{2},\frac{9}{5}\right)$

. Here, d stands for the particular constant that modulation index m determines in the nth time slot. Furthermore, as HAP receives information from a single relay at a time slot, we define relays ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$ selection indicators at nth time slot respectively as

$${\Omega }_1\left[n\right]=\left\{\begin{array}{c}1,\mathrm{if}{P}_{{\mathcal{R}}_1}\left[n\right]{\left|h_1\left[n\right]\right|}^2/N_0>\mathrm{Y}\mathrm{and}{P}_{{\mathcal{R}}_1}\left[n\right]{\left|h_1\left[n\right]\right|}^2>P_{{\mathcal{R}}_2}\left[n\right]{\left|h_2\left[n\right]\right|}^2\hfill \\ 0,\mathrm{Otherwise}\hfill \end{array}\right.,$$

(10)

$${\Omega }_2\left[n\right]=\left\{\begin{array}{c}1,\mathrm{if}{P}_{{\mathcal{R}}_2}\left[n\right]{\left|h_2\left[n\right]\right|}^2/N_0>\mathrm{Y}\mathrm{and}{P}_{{\mathcal{R}}_2}\left[n\right]{\left|h_2\left[n\right]\right|}^2>P_{{\mathcal{R}}_1}\left[n\right]{\left|h_1\left[n\right]\right|}^2\hfill \\ 0,\mathrm{Otherwise}\hfill \end{array}\right..$$

(11)

3.2. Optimization Formulation

To improve the reliability of the proposed WPCN, we maximize the end-to-end net bit rate from IS to HAP by finding the optimal transmit power of HAP and IS, fractions of harvested energy transmitted by relays, and relay selection indicator for DF relaying within the operational period. The associated optimization problem is formulated as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{OP}:& \underset{\left\{\underset{\forall n\in \mathcal{N}}{P_H\left[n\right],P_S\left[n\right],{\sigma }_1\left[n\right],{\sigma }_2\left[n\right],{\Omega }_1\left[n\right],{\Omega }_2\left[n\right]}\right\}}{max}\sum _{n=1}^{N}R\left[n\right],\hfill \\ \hfill \mathrm{Subject}\mathrm{to}\\ & \left(C1\right):P_H\left[n\right]\ge B\left[n\right]/\delta ,\forall n\in \mathcal{N},\hfill \\ & \left(C2\right):0\le P_S\left[n\right]\le \left(B^S\left[n\right]+E_H^S\left[n\right]\right),\forall n\in \mathcal{N},\hfill \\ & \left(C3\right):E_B\left[n\right]\ge 0,\forall n\in \mathcal{N},\hfill \\ & \left(C4\right):E_R\left[n\right]\ge 0,\forall n\in \mathcal{N},\hfill \\ & \left(C5\right):E_U\left[N\right]=0,E_U\left[n\right]\le E_{max},\forall n\in \mathcal{N},n\ne N,\hfill \\ & \left(C6\right):0\le {\sigma }_1\left[n\right]\le 1,0\le {\sigma }_2\left[n\right]\le 1,\forall n\in \mathcal{N},\hfill \\ & \left(C7\right):{\Omega }_1\left[n\right],{\Omega }_2\left[n\right]\in \left\{0,1\right\},\forall n\in \mathcal{N}.\hfill \end{array}\end{array}$$

Here, $C1$, $C2$, $C3$, and $C4$ set the instantaneous boundary conditions for the HAP and IS’s transmit power, borrowing, and the returning energy of the HAP, respectively; $C5$ implies that the unreturned energy of the HAP at the end of the operation has to be zero, but it should not exceed the certain threshold during the operation; $C6$ specifies the fractions of harvested energy transmitted by the relays ${\mathcal{R}}_1$ and ${\mathcal{R}}_2$; and $C7$ verifies the relay selection indicators.

The formulated problem is combinatorial because the fractions of the harvested energy transmitted by the relays at phase 2 are related to their transmit power at phase 4, which is also associated with HAP’s transmit power at phase 1. Furthermore, since the optimization problem is nontrivial, due to the nonlinear structure of the objective function and non-convex constraints, traditional convex optimization requires several approximation steps to obtain suboptimal solutions.In addition, the channel gain and energy arrival rate are unpredictable in a practical wireless communication environment. Hence, we propose a DRL model using the DDPG algorithm to maximize objective value, which also guarantees fast convergence.

4. DRL-Based Solution Methodology

The original optimization problem has multiple decision variables, making it combinatorial and originating several nonconvexity issues. Hence, we formulate an MDP-based DRL framework in which the system interacts with the unknown environment to learn the best decision-making policy for improving the objective value.

4.1. MDP Framework

A centralized controller executes the DRL framework while simultaneously observing PG, HAP, EH relays, and IS. As the current state of these network elements depends on the immediate past state value, the optimization problem can be simplified by an MDP-based sequential decision-making policy.

4.1.1. State Space

Since the transmit powers, fractions of harvested energy transmitted by relays, and relay selection depend on current channel gains, battery level, and harvested energy, the instantaneous state vector is defined as

$$\begin{gathered} \mathbf{s}\left[n\right]=[{\left|h_1\left[n\right]\right|}^2,{\left|h_2\left[n\right]\right|}^2,{\left|h_3\left[n\right]\right|}^2,{\left|h_4\left[n\right]\right|}^2,{\left|h_5\left[n\right]\right|}^2,B\left[n\right],B^{{\mathcal{R}}_1}\left[n\right],B^{{\mathcal{R}}_2}\left[n\right], \\ {B}^S\left[n\right],E_H\left[n\right],E_H^{{\mathcal{R}}_1}\left[n\right],E_H^{{\mathcal{R}}_2}\left[n\right],E_H^S\left[n\right]], \end{gathered}$$

(12)

where $B^{{\mathcal{R}}_1}\left[n\right]\in \left(0,B_{max}^{{\mathcal{R}}_1}\right)$, $B^{{\mathcal{R}}_2}\left[n\right]\in \left(0,B_{max}^{{\mathcal{R}}_2}\right)$, and $B^S\left[n\right]\in \left(0,B_{max}^S\right)$ are the instantaneous battery level of the relay ${\mathcal{R}}_1$, ${\mathcal{R}}_2$, and IS respectively, $B_{max}^{{\mathcal{R}}_1}$, $B_{max}^{{\mathcal{R}}_2}$, and $B_{max}^S$ are their respective maximum battery capacity.

4.1.2. Action Space

According to the decision-making policy, the values of the optimizing variables are determined. Therefore, these are characterized by the transmit power of HAP and IS, fractions of harvested energy transmitted by relays, and relay selection indicators at every instance. Hence, the instantaneous action vector is defined as:

$$\mathbf{a}\left[n\right]=\left[P_H\left[n\right],P_S\left[n\right],{\sigma }_1\left[n\right],{\sigma }_2\left[n\right],{\Omega }_1\left[n\right],{\Omega }_2\left[n\right]\right],$$

(13)

4.1.3. Reward Evaluation

Since the reward defines the quality of an action taken at a particular state, the immediate reward function to maximize the objective value over the long term is expressed as an instantaneous end-to-end net bit rate

$$r\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)=R\left[n\right],$$

(14)

4.1.4. State Transition

It is the probability that the current state $\mathbf{s}\left[n\right]$ transits to the next state $\mathbf{s}\left[n+1\right]$ after taking a current action $\mathbf{a}\left[n\right]$. In our model, channel gains and harvested energy are uncertain and must be learned during decision-making. As these decision variables mostly follow Gaussian distribution, we must estimate their distribution parameters, such as mean and variance, over the simulation episode to maximize the cumulative long-term reward. We define the instantaneous channel gain values and harvested energy according to their current distribution parameters as $h_1\left[n\right]\sim \mathcal{CN}\left({\mu }_{h_1}\left[n\right],{\sigma }_{h_1}^2\left[n\right]\right)$, $h_2\left[n\right]\sim \mathcal{CN}\left({\mu }_{h_2}\left[n\right],{\sigma }_{h_2}^2\left[n\right]\right)$, $h_3\left[n\right]\sim \mathcal{CN}\left({\mu }_{h_3}\left[n\right],{\sigma }_{h_3}^2\left[n\right]\right)$, $h_4\left[n\right]\sim \mathcal{CN}\left({\mu }_{h_4}\left[n\right],{\sigma }_{h_4}^2\left[n\right]\right)$, $h_5\left[n\right]\sim \mathcal{CN}\left({\mu }_{h_5}\left[n\right],{\sigma }_{h_5}^2\left[n\right]\right)$, and $E_H\left[n\right]\sim \mathcal{CN}\left({\mu }_H\left[n\right],{\sigma }_H^2\left[n\right]\right)$ respectively. Depending on their values, battery levels at the next time slot are measured as

$$B_S\left[n+1\right]=min\left(B_S\left[n\right]+E_H^S\left[n\right],B_{max}^S\right)-\delta P_S\left[n\right],$$

(15)

$$B_{{\mathcal{R}}_1}\left[n+1\right]=min\left(B_{{\mathcal{R}}_1}\left[n\right]+E_H^{{\mathcal{R}}_1}\left[n\right],B_{max}^{{\mathcal{R}}_1}\right)-{\sigma }_1\left[n\right]E_H^{{\mathcal{R}}_1}\left[n\right]-\delta P_{{\mathcal{R}}_1}\left[n\right],$$

(16)

$$B_{{\mathcal{R}}_2}\left[n+1\right]=min\left(B_{{\mathcal{R}}_2}\left[n\right]+E_H^{{\mathcal{R}}_2}\left[n\right],B_{max}^{{\mathcal{R}}_2}\right)-{\sigma }_2\left[n\right]E_H^{{\mathcal{R}}_2}\left[n\right]-\delta P_{{\mathcal{R}}_2}\left[n\right].$$

(17)

4.2. Decision-Making Policy

The system gathers information on the surrounding environment through interaction to obtain a sub-optimal decision-making policy. Initially, it does not know the properties of the communication environment. Therefore, it tentatively selects action at a given state, immediately receives reward, and obtains $Q\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)$ value for the current state-action pair. Then the current state $\mathbf{s}\left[n\right]$ is updated to the next state $\mathbf{s}\left[n+1\right]$. Here, the expected mapping value between the current state and the following action can be mathematically represented by [29]

$$Q\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)=\mathbb{E}\left[\sum _{n=\mathfrak{n}}^{\infty }{\gamma }^{n-\mathfrak{n}}r\left(\mathbf{s}\left[\mathfrak{n}\right],\mathbf{a}\left[\mathfrak{n}\right]\right)|\mathbf{s}\left[\mathfrak{n}\right],\mathbf{a}\left[\mathfrak{n}\right],\Pi \right],$$

(18)

where $\gamma \in \left(0,1\right)$ denotes the discount factor, and $\Pi$ represents a deterministic policy for the decision-making process.

Due to the difficulties of finding the gradient of the current policy $\Pi \left(P\left[n\right]|s\left[n\right],\theta \right)$, we model it as a Gaussian distribution with mean $\mu \left(s\left[n\right],{\theta }_{\mu }\right)$ and standard deviation $\sigma \left(s\left[n\right],{\theta }_{\sigma }\right)$, where the policy distribution can be expressed as [27]

$$\Pi \left(P\left[n\right]|s\left[n\right],\theta \right)=\frac{1}{\sqrt{2\pi {\left(\sigma \left(s\left[n\right],{\theta }_{\sigma }\right)\right)}^2}}exp\left[-\frac{{\left(\delta P_H\left[n\right]-\mu \left(s\left[n\right],{\theta }_{\mu }\right)\right)}^2}{2{\left(\sigma \left(s\left[n\right],{\theta }_{\sigma }\right)\right)}^2}\right],$$

(19)

Since the maximum and minimum values of policy parameters lie within a range, we apply the hyperbolic tangent function to restrict its output between $-1$ and 1, represented by [24]

$$\mu \left(s\left[n\right],{\theta }_{\mu }\right)=max\left\{0,E_A\left[n\right]\left(\frac{1+tanh\left({\theta }_{\mu }^T\varphi \left(s\left[n\right]\right)\right)}{2}\right)\right\}.$$

(20)

where $E_A\left[n\right]=B\left[n-1\right]+E_H\left[n\right]-E_U\left[n-1\right]$ is the effective energy level, and $\varphi \left(s\left[n\right]\right)=\left[{\varphi }_1\left(s\left[n\right]\right),{\varphi }_2\left(s\left[n\right]\right)\right]$ is the feature vector of the current state $s\left[n\right]$. It contains two binary functions ${\varphi }_1\left(s\left[n\right]\right)$ and ${\varphi }_2\left(s\left[n\right]\right)$ such that, ${\varphi }_1\left(s\left[n\right]\right),{\varphi }_2\left(s\left[n\right]\right)=1$ if the battery energy level exceeds its minimum value, and the battery energy level has achieved its maximum value respectively. Otherwise, they are set to zero for other cases. Since the standard deviation should be positive, it is modeled by a linear exponent as $\sigma \left(s\left[n\right],{\theta }_{\sigma }\right)=exp\left({\theta }_{\sigma }^T\varphi \left(s\left[n\right]\right)\right)$. This proposed Gaussian policy allocates transmit power at each time slot. In all but the N-th time slot within the last k time slots, the transmission power abides by a certain condition. This condition is put in place to guarantee that any energy that was borrowed is properly returned to the power grid and it can be expressed as follows:

$$P_H\left[n\right]=\left\{\begin{array}{c}max\left\{0,E_H\left[N-k\right]-E_U\left[N-k\right]\right\},\mathrm{if}{E}_U\left[N-k\right]>\frac{k}{\kappa }\frac{{\sum }_{i=0}^{N-k}{E}_H\left[i\right]}{N-k}\hfill \\ \mathrm{Sample}\mathrm{from}\left(19\right),\mathrm{Otherwise}\hfill \end{array}\right.,$$

(21)

This condition holds for a sufficient amount of energy can be returned to the PG efficiently, where k is proportional to $\left(E_{max}^U/\mathrm{mean}\left(E_H\left[n\right]\right)\right)$ and $P_H\left[N\right]=max\left\{0,E_H\left[N\right]-E_U\left[N\right]\right\}$. Here $\kappa \in (0,1]$ plays a critical role in determining the margin due to variations in harvested energy.

4.3. Proposed DDPG Algorithm

DDPG is an RL framework that can handle the continuous state and action spaces based on policy and Q-value evaluation. It utilizes policy evaluation NN $\mathbf{a}\left[n\right]=\mu \left(\mathbf{s}\left[n\right]|{\theta }^{\mu }\right)$ with parameter ${\theta }^{\mu }$ which takes the state vector $\mathbf{s}\left[n\right]$ as input and outputs corresponding action vector $\mathbf{a}\left[n\right]$. Policy evaluation NN $\mu \left(\mathbf{s}\left[n\right]|{\theta }^{\mu }\right)$ consists of an input layer of thirteen neurons, three successive hidden layers of $N_1$, $N_2$, and $N_3$ neurons, and an output layer of six neurons. As the normalized action vector can only be a positive value for a given positive value state vector, we apply the sigmoid activation function to better tune the policy NN model. After taking action $\mathbf{a}\left[n\right]$ at the current state $\mathbf{s}\left[n\right]$, the immediate reward $r\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)$ is generated and the current state is updated to the next state $\mathbf{s}\left[n+1\right]$. Then, the sample data tuple $\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right],r\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right),\mathbf{s}\left[n+1\right]\right)$ is stored in the experience memory. During the reinforcement learning process, a data tuple consisting of the current state vector, action vector, reward resulting from the action, and the next state vector is stored in the experience memory. The Q-value evaluation neural network is trained using a batch of randomly selected $N_B$ samples from the experience memory. The neural network, denoted as $Q\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]|{\theta }^Q\right)$ with parameter ${\theta }^Q$, takes the state and action vectors as input and provides the state-action value $Q\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)$ as output. This neural network has an input layer comprising nineteen neurons, three hidden layers consisting of $N_4$, $N_5$, and $N_6$ neurons, respectively, and an output layer with a single neuron. As the desired output Q value is always a positive number, we apply the sigmoid activation function to tune the Q-value evaluation NN. The policy and Q-value target NNs, respectively represented by $\mu \prime \left(\mathbf{s}\left[n\right]|{\theta }^{\mu \prime }\right)$ and $Q\prime \left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]|{\theta }^{Q\prime }\right)$ with parameters ${\theta }^{\mu \prime }$ and ${\theta }^{Q\prime }$ replicating the same structure as the policy and Q-value evaluation NNs respectively are applied to stabilize the training process. The parameter of the Q-value evaluation NN, ${\theta }^Q$, is updated by minimizing the temporal difference (TD) error loss, which is expressed as [21]:

$$L\left({\theta }^Q\right)=\frac{1}{N_B}\sum _n{\left(\overline{Y}\left[n\right]-Q\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]|{\theta }^Q\right)\right)}^2,$$

(22)

where $\overline{Y}\left[n\right]$, the output of the Q-value target NN is calculated using the output of the policy target network as [21]:

$$\overline{Y}\left[n\right]=r\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)+\gamma Q\prime \left(\mathbf{s}\left[n+1\right],\mu \prime \left(\mathbf{s}\left[n+1\right]|{\theta }^{\mu \prime }\right)|{\theta }^{Q\prime }\right),$$

(23)

where $\gamma$ is the discount factor. The parameters of policy evaluation NN can be updated through the deterministic policy gradient method, which is given as [21]:

$${\nabla }_{{\theta }^{\mu }}J\left({\theta }^{\mu }\right)=\frac{1}{N_B}\sum _n\left({\nabla }_{\mu }Q\left(\mathfrak{s},\mathfrak{a}|{\theta }^Q\right)|\mathfrak{s}=\mathbf{s}\left[n\right],\mathfrak{a}=\mu \left(\mathbf{s}\left[n\right]|{\theta }^{\mu }\right){\nabla }_{{\theta }^{\mu }}\mu \left(\mathfrak{s}|{\theta }^{\mu }\right)|\mathfrak{s}=\mathbf{s}\left[n\right]\right),$$

(24)

Finally, the parameters of the target NNs are updated slowly with respect to learning rate $\tau \ll 1$ as [21]:

$${\theta }^{\mu \prime }\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{\mu \prime },$$

(25)

$${\theta }^{Q\prime }\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q\prime }.$$

(26)

4.4. Implementation Details

Algorithm 1 implements the end-to-end net bit rate maximization in the proposed EH relay-assisted WPCN. In the beginning, the four NNs, namely, policy evaluation NN, policy target NN, Q-value evaluation NN, and Q-value target NN, are initialized with random weight vectors. Then, inside the main loop, policy evaluation NN takes the current state as input for each time slot and approximates the action value. In order to keep exploration, we add Gaussian noise of variance $ϵ$ to the current action. After choosing the action, the current state updates to a new state and generates an immediate reward by (14). Then, the transition data set, consisting of the current state, action, reward, and the next state, is stored in experience memory to train the DRL model. When the filled memory length is greater than the batch size, randomly sample a mini-batch of transition data from memory, calculate the loss values, and update the parameters of policy and Q-value evaluation NNs by (24) and (22), respectively. Then, the algorithm updates the parameters of the target NNs by (25) and (26) and also updates the current state as the next state. Finally, the running episode is terminated when the system elapsed maximum operational time steps, and the obtained policy corresponding to the last episode makes optimal decision variables.

Algorithm 1 DRL based on DDPG for end-to-end net bit rate maximization

Require: N, T, $N_B$, $N_0$, $\eta$, $\delta$, $\xi$, $\rho$, $\zeta$

Ensure: $P_H\left[n\right]$, $P_S\left[n\right]$, ${\sigma }_1\left[n\right]$, ${\sigma }_2\left[n\right]$, ${\Omega }_1\left[n\right]$, ${\Omega }_2\left[n\right]$, $\forall n\in \mathcal{N}$

1: Initialize policy evaluation and target NNs, Q-value evaluation, and target NNs as $\mu \left(\mathbf{s}\left[n\right]|{\theta }^{\mu }\right)$, $\mu \prime \left(\mathbf{s}\left[n\right]|{\theta }^{\mu \prime }\right)$, $Q\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]|{\theta }^Q\right)$, $Q\prime \left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]|{\theta }^{Q\prime }\right)$, and corresponding parameters are ${\theta }^{\mu \prime }={\theta }^{\mu }$ and ${\theta }^{Q\prime }={\theta }^Q$, respectively. 2: Initialize an empty experience buffer as $M_E=$ where initial memory length is set as $|M_E|\hspace{0.17em}=0$ 3: for $t=1,2,\dots ,T$ do 4:     Reset the state vector at the initial condition as $\mathbf{s}\left[1\right]$ 5:     for $n=1,2,\dots ,N$ do 6:     Get the current normalized action vector as $\mathbf{a}\left[n\right]=\mu \left(\mathbf{s}\left[n\right]|{\theta }^{\mu }\right)+\mathcal{CN}\left(0,ϵ\right)$ 7:     Update the next state vector $\mathbf{s}\left[n+1\right]$ by (1), (15), (16), (17) 8:     Generate immediate reward $r\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)$ by (14) 9:     Store state, action, and reward transition data in the experience memory buffer as a tuple of $\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right],\mathbf{s}\left[n+1\right],r\left(\mathbf{s}\left[n\right],\mathbf{a}\left[n\right]\right)\right)$ 10:      $|M_E|\hspace{0.17em}=\hspace{0.17em}|M_E|\hspace{0.17em}+\hspace{0.17em}1$ 11:      if $|M_E|\hspace{0.17em}\ge N_B$ then 12:       Randomly sample a batch of data from memory 13:       Update parameters of policy and Q-value evaluation NNs by (24) and (22) respectively 14:       Update the parameters of policy and Q-value target NNs by (25) and (26), respectively 15:     end if 16:     Update the current state vector as $\mathbf{s}\left[n\right]=\mathbf{s}\left[n+1\right]$ 17:     end for 18:     Update action noise as $ϵ=ϵ\left(1-\left(t/T\right)\right)$ 19: end for 20: Finally the optimal policy can be obtained as $\left(P_H\left[n\right],P_S\left[n\right],{\sigma }_1\left[n\right],{\sigma }_2\left[n\right],{\Omega }_1\left[n\right],{\Omega }_2\left[n\right]\right)$, $\forall n\in \mathcal{N}$

The centralized controller implements the proposed algorithm to train the above-mentioned NN configurations. The proposed algorithm’s computational complexity depends entirely on the defined NNs’ structure and the number of operations in the network model. Let $W_1$ and $W_2$ denote the number of fully connected layers in the policy and Q-value NNs, respectively. In each time slot, the total transition made by policy evaluation NN is calculated as ${\sum }_{u=0}^{W_1-1}{\Theta }_u^P{\Theta }_{u+1}^P$, where ${\Theta }_u^P$ is the neurons of the u-th layers of the policy NN. Similarly, the total transition faced by Q-value NN can be obtained as ${\sum }_{w=0}^{W_2-1}{\Theta }_w^Q{\Theta }_{w+1}^Q$, where ${\Theta }_w^Q$ is the neurons of the w-th layer of the Q-value NN. Therefore, the overall computational complexity for successive N timeslots in each of the T episodes will be $\mathcal{O}\left(NT\left({\sum }_{u=0}^{W_1-1}{\Theta }_u^P{\Theta }_{u+1}^P+{\sum }_{w=0}^{W_2-1}{\Theta }_w^Q{\Theta }_{w+1}^Q\right)\right)$. According to this expression, the computational complexity of the proposed algorithm increases with the operational period.

5. Simulation Results

In this section, we validate the effectiveness and convergence of the proposed algorithm using various simulation results. We used the Pytorch 1.10.1 module in Python 3.7.8 to build the DDPG environment and conduct the simulations on a high computing system. The Adam optimizer was applied to update the parameters of policy and Q value evaluation NNs. We compared the performance of the proposed methodology with the adaptive BEAR algorithm [24], where the underlying transmission power allocation was modeled with a parameterized Gaussian distribution, ensuring a maximum sum bit rate over a given time slot while learning the EH rate and channel conditions. Furthermore, the primary simulation parameters are taken from [24,29] which are given as $B_{max}=B_{max}^{{\mathcal{R}}_1}=B_{max}^{{\mathcal{R}}_2}=B_{max}^S=3$ Joul, $E_{max}=5$ Joul, $N=100$, $\varsigma =0.8$, $\varrho =0.1$, $\eta =0.6$, $\rho =1000$, $\zeta =0.01$ s, $\delta =1$ s, ${\rm Y}=15$ dB, ${\mu }_{h_1}={\mu }_{h_2}={\mu }_{h_3}={\mu }_{h_4}={\mu }_{h_5}={\mu }_H=0.5$, ${\sigma }_{h_1}^2={\sigma }_{h_2}^2={\sigma }_{h_3}^2={\sigma }_{h_4}^2={\sigma }_{h_5}^2={\sigma }_H^2=0.75$, $N_B=128$, $\left|M_E\right|=\mathrm{50,000}$, $\gamma =0.9$, $\tau =0.001$, $ϵ=0.1$, $T=3000$, $N_1=128$, $N_2=64$, $N_3=32$, $N_4=64$, $N_5=128$, and $N_6=32$.

5.1. Convergence Analysis

Figure 2 demonstrates the converging behavior of the training performance at $-25$ dB noise power for various learning rates under the QPSK modulation scheme. According to this figure, the end-to-end net bit rate increases with each episode and eventually converges. If we choose a low learning rate, the training process runs slowly, because the low learning rate updates the NNs’ weights on a small scale. However, if we set a high learning rate, the loss function of the NNs encounters undesirable divergence. Consequently, the NNs experience high oscillation during training, but the objective value converges quickly. This figure shows that fluctuations and convergence rates over the training episode increase when the learning rate varies from $1\times {10}^{-5}$ to $5\times {10}^{-3}$. Hence, considering the model’s stability, we set the learning rate to $5\times {10}^{-4}$ for the subsequent simulations.

Figure 3 presents the variation of the end-to-end net bit rate over the training episodes to analyze the performance of the proposed and benchmark algorithms corresponding to different modulation schemes, while the noise power was set to $-25$ dB. This figure shows that the converged objective value increased with the lower modulation schemes because this required less power and fewer signal points to transmit the same amount of data, which allowed a more efficient use of the available frequency spectrum. Therefore, QPSK achieved a higher end-to-end net bit rate than 8-PSK and 16-QAM. Furthermore, as the benchmark BEAR algorithm entailed more computational complexity, due to the off-policy samples from the replay buffer, the achievable performance metric corresponding to the BEAR algorithm was lower than the proposed DDPG algorithm. Hence, according to this figure, the DDPG algorithm outperformed the BEAR algorithm by $20.67\%$, $16.21\%$, and $26.38\%$ in the cases of the QPSK, 8-PSK, and 16-QAM modulation schemes, respectively.

5.2. Performance Evaluation

We plot the transmit power variation of the HAP in Figure 4, corresponding to the proposed and benchmark algorithms for different modulation schemes. It can be observed that the average transmit power of the HAP increased with the higher modulation schemes because they required higher peak-to-average power ratios and higher transmit power levels to generate complex signal constellations. On the other hand, the conventional BEAR algorithm updated the parameters of the policy evaluation NN using a distributional shift correction method to reduce the overestimation of the Q-value. This limited the ability of the algorithm to explore the search space and find optimal policies in complex environments. From Figure 4, it is clear that the proposed methodology reduced the average transmit power of the HAP by $19.12\%$, $17.69\%$, and $11.58\%$ as compared to the BEAR algorithm in the cases of the QPSK, 8-PSK, and 16-QAM modulation schemes, respectively, over the operational period.

Figure 5 illustrates the end-to-end net bit rate for the different modulation schemes under various noise power levels at the receiver. This figure shows that the 16-QAM modulation scheme performed better than 8-PSK and QPSK at a lower noise power, whereas QPSK outperformed 8-PSK and 16-QAM at a higher noise power. This is because the higher modulation schemes encode more bits per symbol, which allows higher data rates to be transmitted over a given channel bandwidth. However, since they typically use more complex signal constellations with smaller distances between signal points, they are highly susceptible to the distortion caused by noise or channel impairments. On the other hand, the lower modulation techniques are more straightforward to implement than the higher modulation techniques. This makes them more suitable for low-power at low-complexity systems, making them more bandwidth-efficient at a higher noise power. From Figure 3 and Figure 4, we can justify that the benchmark BEAR algorithm faced several challenges in finding the optimal policy. Hence, the proposed DDPG algorithm improves the performance metric by $13.22\%$, $27.57\%$, and $14.12\%$ as compared to the BEAR algorithm in the cases of the QPSK, 8-PSK, and 16-QAM modulation schemes, respectively.

The variation of the allotted IS’s transmit power with respect to the noise power under different modulation techniques is shown in Figure 6. It can be observed that the transmit power of the IS increased with the noise power because the IS utilized more transmit power to achieve a minimum SNR and BER value for a higher noise power, which also maintained an adequate quality of service (QoS). Since we mentioned earlier that higher modulation techniques are more susceptible to distortion caused by channel noise impairments, 16-QAM requires more transmit power at higher noise power levels compared to the 8-PSK and QPSK modulation techniques. Moreover, as the benchmark algorithm is inefficient, the proposed technique reduced the transmit power of IS by $15.17\%$, $9.94\%$, and $8.18\%$ compared to the BEAR algorithm in the cases of the QPSK, 8-PSK, and 16-QAM modulation schemes, respectively.

6. Conclusions

Facing the great challenges of reliable WPCN, this article introduced a JIER protocol that allocates resources efficiently, for effective energy management in EH wireless networks. Specifically, the formulated joint optimization of HAP and IS transmit power, the fraction of harvested energy transmitted by relays, and relay selection indicators maximized the end-to-end net bit rate of the system under energy borrowing and return scheduling constraints. The formulated problem was highly non-convex, nontrivial, and difficult to solve directly. Hence, we leveraged the DRL framework, which decomposed the original problem into multiple sub-problems based on MDP, and then proposed the DDPG algorithm to find the optimal policy. The simulation results validated the proposed scheme and enhanced the end-to-end net bit rate of the system by $13.22\%$, $27.57\%$, and $14.12\%$ compared with the BEAR algorithm for the QPSK, 8-PSK, and 16-QAM modulation schemes, respectively. In the future, we will extend this work to an energy borrowing and returning strategy in multi-user and multi-antenna relay-assisted EH systems using multi-agent DRL, where the objective will be the minimization of the age of information, to ensure a tolerable latency.