Intelligent Energy Efficiency Maximization for Wirelessly-Powered UAV-Assisted Secure Sensor Network

Abstract

The rapid proliferation of Internet of Things (IoT) devices and applications has led to an increasing demand for energy-efficient and secure communication in wireless sensor networks. In this article, we firstly propose an intelligent approach to maximize the energy efficiency of the UAV in a secure sensor network with wireless power transfer (WPT). All sensors harvest energy via downlink signal and use it to transmit uplink information to the UAV. To ensure secure data transmission, the UAV needs to optimize the transmission parameters to decode received information under malicious interference from an attacker. Code Division Multiple Access (CDMA) is adopted to improve uplink communication robustness. To maximize the UAV’s energy efficiency in data collection tasks, we formulate a constrained optimization problem that jointly optimizes charging power, charging duration, and data transmission duration. Applying Deep Deterministic Policy Gradient (DDPG) algorithm, we train an action policy to dynamically determine near-optimal transmission parameters in real time. Numerical results validate the superiority of proposed intelligent approach over exhaustive search and gradient ascent techniques. This work provides some important guidelines for the design of green secure wireless-powered sensor networks.

1. Introduction

The future Internet of Things (IoT) is anticipated to enable massive deployments of intelligent sensors that can securely and efficiently connect to the internet and interact with one another. However, powering these large-scale sensor networks sustainably remains a critical challenge, as relying on grid electricity for a vast number of devices is often impractical. Wireless power transfer (WPT) has emerged as a promising solution, allowing sensors to harvest energy wirelessly while eliminating the need for frequent battery replacements. Despite its advantages, WPT systems face significant wireless path loss and heightened security risks, making the development of secure and energy-efficient transmission schemes essential for real-world applications. Recent research has investigated the approach for optimizing secrecy performance in wireless communication networks, e.g., [1,2,3,4,5,6]. For instance, ref. [1] proposed a secure communication paradigm at physical layer for wireless-powered sensor networks, where corresponding challenges, countermeasures, and road ahead were discussed in detail. Ref. [2] proposed a certificateless linearly homomorphic signature scheme, for data transmission and authentication, to ensure the authenticity, integrity and non-repudiation of data. Ref. [3] analyzed secrecy outage probability (SOP) for dual-hop relaying systems with hybrid MIMO RF/FSO links and proposed transmit antenna selection (TAS) schemes to enhance secrecy under imperfect channel state information (CSI). Similarly, ref. [4] examined SOP for FSO systems with a distinct eavesdropper (ED) near the relay, deriving closed-form expressions for various scenarios. Additionally, ref. [5] proposed a learning-based secure transmission framework using deep neural networks to optimize transmission parameters for maximizing secrecy throughput, while [6] focused on robust secure transmission by optimizing the worst-case secrecy rate under energy-harvesting and power constraints. However, these studies primarily address secrecy issues rather than energy efficiency issues, leaving a significant gap in developing integrated frameworks that ensure energy-efficient and secure communication for intelligent sensor networks in IoT applications.

Refs. [7,8,9] further investigated the improvement approach for energy efficiency in terms of wireless communication network with eavesdroppers. Ref. [7] proposed a novel secure and energy-efficient message communication system, called MComIoV, where MComIoV was evaluated through security proof and analysis against various attacks to verify corresponding robustness. Ref. [8] proposed a beamformer design approach to optimize the energy efficiency, in terms of secrecy bits per Joule under secrecy quality-of-service (QoS) constraints, for a multi-user MIMO communication network with an eavesdropper. Ref. [9] optimized the resource allocation strategy, the UAV’s trajectory, and the jamming policy of the jammer UAV are jointly optimized for maximizing overall energy efficiency for secure unmanned aerial vehicle (UAV) communication system. However, these literature did not focus on secure communication network with wireless power transfer.

Refs. [10,11,12,13,14,15,16,17] further investigated energy efficient simultaneous information and power transfer (SWIPT) system. Ref. [10] realized high energy efficient transmission by optimizing power allocation and Energy Harvesting (EH) relay selection in clustered wireless sensor network. Ref. [11] investigated the energy efficiency maximization for coordinated multi-point (CoMP) SWIPT heterogeneous networks (HetNets) where joint beam-forming and power allocation are optimized under intra-cell and inter-cell interferences. Ref. [12] studied the optimal resource allocation and data forwarding in integrated data and energy transmission. Ref. [13] focused on the energy efficient SWIPT in MIMO system where optimal power allocation and precoding have been realized. Ref. [14] developed a tractable model in terms of joint downlink and uplink transmission of K-tier heterogeneous cellular networks by SWIPT, in which authors derive the outage probability and the average ergodic rate of a random mobile user under different cell associations. This will provide guideline for energy efficient SWIPT system. Ref. [15] proposed a heuristic algorithm to optimize the transfer time allocation based on energy consumption distribution of nodes under scene of SWIPT. Ref. [16] proposed a mode switching scheme for SWIPT on basis of random beamforming technique. This can generate artificial channel fading to increase energy harvesting efficiency of receiver. Ref. [17] has achieved the energy efficient secure transmission system by jointly optimizing energy beamformers, information beamformers and transmit time switching ratio, subject to constrained usersharvested energy as well as power budget of BS. Refs. [18,19,20,21] mainly investigated the wirelessly powered network with separate information transmission. Refs. [22,23,24] aimed at improving the energy efficiency of wireless powered network by utilizing renewable energy. Ref. [22] proposed a hybrid framework that combines the two technologies, namely solar energy harvesting and wireless charging, in which cluster head placement has been optimized to minimize corresponding energy consumption. In [23], a M/M/1 make-to-stock queuing model is proposed to investigate decentralized decisions on how much amount of renewable energy should be supplied to BS for minimizing the individual cost of renewable source. Ref. [24] investigated the tradeoff between energy consumption and QoS, where the wireless network is powered by both grid and harvested renewable energy. However, these literature did not consider the security issue and mainly use iterative algorithms to optimize transmission parameters. Since online iterative calculation needs to consume some time, these approaches can not be used for determining optimal transmission parameters under strict latency constraint.

Recently, deep reinforcement learning (DRL) become a powerful tool for handing complicated optimization issues in communication network [25]. Refs. [26,27,28] introduced the deep reinforcement learning algorithm to achieve energy efficient sensing, navigation and wireless transmission, respectively. Ref. [26] utilized distributed DDPG to enhance EE in UAV mobile sensing (MCS). Ref. [27] proposed a decentralized deep reinforcement learning (DRL) framework to arrive at energy efficient navigation for distributed UAVs. Ref. [28] proposed a reinforcement learning based energy management strategy to access good long-term average net bit rate.However, DRL approach, applicable for realizing green secure data collection in IoT, has not been investigated yet.

In a word, green secure transmission scheme for wirelessly powered sensor network was not well investigated yet in all the above literature. In this article, we design a DDPG approach to arrive at a near-optimal secure transmission scheme for maximizing the energy efficiency of data collection in wireless sensor network with malicious attackers and WPT. With well-trained policy model, optimal transmission parameters can be determined in real time. As such, our proposed approach can effectively reduce the online latency of wireless transmissions. To our best knowledge, this is the first work to investigate the adaptive scheme for green secure IoT transmissions under strict latency constraint and WPT.

2. System and Channel Model

We consider a wirelessly-powered sensor network with an UAV, an attacker and multiple sensors, as illustrated in Figure 1. More specifically, the UAV firstly transfers energy to all sensors through downlink and then, all sensors send back collected data to the UAV through uplink using harvested energy. Note also that the attacker would constantly transfer interference signal to the UAV for degrading decoding performance. To achieve secure data collection, the UAV needs to be capable of normally collecting sensor data under the interference from attacker. We assume that all channels experience slow flat fading, where $g_i$ denotes the channel power gain from the UAV to sensor i, $g_i$ consists of fading channel gain $h_i$ and path loss $d_i^{-\alpha }$, $i=1,2,3,\cdots ,M$. As such, channel power gain is a random variable depending upon distance $d_i$ and path loss exponent $\alpha$, $i=1,2,3,\cdots ,M$. Applying linear energy harvesting (EH) model, the harvested energy at sensor i, $i=1,2,3,\cdots ,M$, is given by

$$E_{H_i}=P_S{g}_i{\tau }_{wp}\eta ,$$

(1)

where $P_S$ is charging power level of the UAV, ${\tau }_{wp}$ is the charging duration and $\eta$ is DC energy transfer efficiency. All sensors’ positions are assumed to follow 2D poisson point process (PPP) with the density of $\lambda$. During uplink data collection, both TDMA and CDMA are applied to facilitate the data transmission.

2.1. TDMA

In terms of TDMA mode, all sensors send collected data to the UAV in turn through uplink transmission. ${\tau }_i$ is the transmission duration for sensor i, $i=1,2,3,\cdots ,M$. As such, effective data rate can be denoted by

$$\begin{array}{c}\hfill R_{TDMA}=\sum _{i=1}^M{\tau }_{i}B{log}_2(1+{\displaystyle \frac{P_S{g}_i^2\eta \tau }{{\tau }_i(P_{at}{g}_{at}+{\sigma }^2)}})/T.\end{array}$$

(2)

Here, B is the effective bandwidth of wirelessly-powered network, ${\sigma }^2$ is the average noise power, T is the duration of data collection and ${\sum }_{i=1}^M{\tau }_i$ is equal to $T-\tau$, $P_{at}$ is the transmit power level of the attacker and $g_{at}$ is the channel power gain from the attacker to the UAV.

2.2. CDMA

In terms of CDMA mode, all sensors employ orthogonal codes to simultaneously transmit collected data to the UAV. Then, while noting that CDMA can considerably reduce the inter-user interference [29], we ignore the decoding interference from all normal sensors. Accordingly, effective data rate can be denoted by

$$\begin{array}{c}\hfill R_{CDMA}=\sum _{i=1}^M{\displaystyle \frac{B{\tau }_{up}{log}_2(1+\frac{P_S{g}_i^2\eta {\tau }_{wp}}{{\tau }_{up}(P_{at}{g}_{at}+{\sigma }^2)})}{T}}.\end{array}$$

(3)

Here, B is the effective bandwidth, ${\sigma }^2$ is the average noise power and ${\tau }_{up}$ is the uplink transmission duration, equal to $T-{\tau }_{wp}$.

3. Energy Efficiency Maximization

In this section, we intend to maximize the energy efficiency for wirelessly-powered sensor network under CDMA mode and TDMA mode. With TDMA mode, action policy for determining near-optimal transmission parameters is obtained, and all closed-form expressions for optimal transmission parameters are derived accordingly.

3.1. TDMA

3.1.1. Optimization Without QoS Constraint

We maximize the energy efficiency of data collection under an attacker’s interference, which is calculated as the product of effective power and operating duration. General optimization problem is formulated as follows

$$\begin{array}{cc}\hfill & \underset{P_S,\tau ,{\tau }_1,{\tau }_2,\cdots ,{\tau }_M}{max}Eff={\displaystyle \frac{{\sum }_{i=1}^M\frac{{\tau }_i}{T}B{log}_2(1+\frac{P_S{g}_i^2\eta \tau }{{\tau }_i(P_{at}{g}_{at}+{\sigma }^2)})}{(P_S\frac{\tau }{T}+P_C)}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill \sum _{i=1}^M{\tau }_i& \hfill =T-\tau ,0<P_S\le P_{max},\end{array}\hfill \end{array}$$

where ${\sum }_{i=1}^M{\tau }_{i}B{log}_2(1+{\displaystyle \frac{P{g}_i^2\eta \tau }{{\tau }_i(P_{at}{g}_{at}+{\sigma }^2)}})/T$ is the effective throughput and $P_S\tau /T+P_C$ is the effective transmit power of the UAV. Note that iterative algorithms have high online calculative complexity and as such, they are not suitable for adaptive transmission subject to strict latency constraint. Accordingly, we intend to derive closed-form analytical expressions for optimal transmission parameters, where corresponding parameters can be determined in real time without involving any iterative calculation. First of all, we present a lemma to show the condition of energy efficiency maximization.  

Lemma 1.

Optimal ${\tau }_1$, ${\tau }_2$, ⋯, ${\tau }_M$ need to satisfy the condition $P_S{g}_1^2\eta \tau /{\tau }_1=P_S{g}_2^2\eta \tau /{\tau }_2=\cdots =P_S{g}_M^2\eta \tau /{\tau }_M=C$.

We provide corresponding proof as below

Proof. 

We assume that $P_S$ and $\tau$ are given and as such, optimization problem can be updated to

$$\begin{array}{cc}\hfill & \underset{{\tau }_1,{\tau }_2,\cdots ,{\tau }_M}{max}{E}_{Eff}={\displaystyle \frac{{\sum }_{i=1}^M{\tau }_{i}B{log}_2(1+P_S{g}_i^2\eta \tau /{\tau }_i(P_{at}{g}_{at}+{\sigma }^2))}{P_S\tau +P_{C}T}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill \sum _{i=1}^M{\tau }_i& \hfill =T-\tau .\end{array}\hfill \end{array}$$

Note that the second derivative, for this objective function, with respect to ${\tau }_i$ is always less than zero, $i=1,2,\cdots ,M$. Hence, the above function and constraint are both jointly concave with respect to ${\tau }_1$, ${\tau }_2$, ⋯ and ${\tau }_M$. According to Ref. [30], we apply KKT condition to arrive at the mathematical expression $L=EE+\lambda (T-\tau -{\sum }_{i=1}^M{\tau }_i)$. As such, we can further transform ${\nabla }_{{\tau }_1,\cdots ,{\tau }_M}L=0$ to

$$\begin{array}{c}\hfill {\displaystyle \frac{B(ln(1+\frac{P_S{g}_i^2\eta \tau }{{\tau }_i(P_{at}{g}_{at}+{\sigma }^2)})-\frac{P_S{g}_i^2\eta \tau /({\tau }_i(P_{at}{g}_{at}+{\sigma }^2))}{(1+P_S{g}_i^2\eta \tau /({\tau }_i(P_{at}{g}_{at}+{\sigma }^2)))})}{(P_S\tau +P_{C}T)ln2}}-\lambda =0,\end{array}$$

(4)

where $i=1,\cdots ,M$. We can see that Equation (4) can hold if and only if $P_S{g}_1^2\eta \tau /{\tau }_1=P_S{g}_2^2\eta \tau /{\tau }_2=\cdots =P_S{g}_M^2\eta \tau /{\tau }_M=C$, where C is a constant.

Accordingly, by substituting the condition of Lemma 1 into original optimization problem, we can rewrite corresponding problem to

$$\begin{array}{cc}\hfill & \underset{P_S,\tau ,{\tau }_1,\cdots ,{\tau }_M}{max}{E}_{Eff}={\displaystyle \frac{B({\sum }_{i=1}^M{\tau }_i){log}_2(1+\frac{C}{(P_{at}{g}_{at}+{\sigma }^2)})}{P_S\tau +P_{C}T}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill \sum _{i=1}^M{\tau }_i& \hfill =T-\tau ,0<P_S\le P_{max},\\ \hfill & \hfill P_S{g}_1^2\eta \tau /{\tau }_1& \hfill =\cdots =P_S{g}_M^2\eta \tau /{\tau }_M=C.\end{array}\hfill \end{array}$$

According to ${\sum }_{i=1}^M{\tau }_i$=$T-\tau$ and ${\displaystyle \frac{P_S{g}_1^2\eta \tau }{{\tau }_1}}=\cdots ={\displaystyle \frac{P_S{g}_M^2\eta \tau }{{\tau }_M}}=C$, we have $C={\sum }_{i=1}^M{\displaystyle \frac{P_S{g}_i^2\eta \tau }{T-\tau }}$ and ${\tau }_i=(T-\tau ){\displaystyle \frac{g_i^2}{{\sum }_{i=1}^M{g}_i^2}}$, $i=1,2,\cdots ,M$. Then, we further simplify the optimization problem to

$$\begin{array}{cc}\hfill & \underset{P_S,\tau }{max}{E}_{Eff}={\displaystyle \frac{B(T-\tau ){log}_2(1+\frac{P_S({\sum }_{i=1}^M{g}_i^2)\eta \tau }{(T-\tau )(P_{at}{g}_{at}+{\sigma }^2)})}{(P_S\tau +P_{C}T)}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill \tau & \hfill >0,0<P_S\le P_{max}.\end{array}\hfill \end{array}$$

Such objective function is still not jointly concave with respect to $P_S$ and $\tau$. As such, we define a variable x as $P_S(\tau /(T-\tau ))$. Then, this optimization problem can be rewritten to

$$\begin{array}{cc}\hfill & \underset{P_S,x}{max}{E}_{Eff}={\displaystyle \frac{B{log}_2(1+x({\sum }_{i=1}^M{g}_i^2)\eta /(P_{at}{g}_{at}+{\sigma }^2))}{(x+P_C(1+\frac{x}{P_S}))}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill x& \hfill >0,0<P_S\le P_{max}.\end{array}\hfill \end{array}$$

While noting that $\tau /(T-\tau )$ ranges from 0 to ∞, x and $P_S$ are independent with each other. This is because that x is possibly to be any value regardless of how much is $P_S$. Accordingly, the objective function is monotonically increasing with $P_S$, based on which optimal $P_S$ is equal to its peak value $P_{max}$. Subsequently, such optimization problem can be transformed to an univariate optimization problem, shown as follows

$$\begin{array}{cc}\hfill & \underset{x}{max}{E}_{Eff}={\displaystyle \frac{B{log}_2(1+x({\sum }_{i=1}^M{g}_i^2)\eta /(P_{at}{g}_{at}+{\sigma }^2))}{(x(1+\frac{P_C}{P_{max}})+P_C)}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill x& \hfill >0.\end{array}\hfill \end{array}$$

By setting $\partial Eff/\partial x=0$, we arrive at the optimal solution as below

$$\begin{array}{c}\hfill x^{*}={\displaystyle \frac{\frac{P_C}{1+\frac{P_C}{P_{max}}}-\frac{(P_{at}{g}_{at}+{\sigma }^2)}{({\sum }_{i=1}^M{g}_i^2)\eta }}{W_0[e^{-1}(\frac{P_C}{1+\frac{P_C}{P_{max}}}(\frac{{\sum }_{i=1}^M{g}_i^2\eta }{(P_{at}{g}_{at}+{\sigma }^2)})-1)]}}-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{({\sum }_{i=1}^M{g}_i^2)\eta }},\end{array}$$

(5)

where $W_0[.]$ denotes the positive branch of Lambert W function [31]. Accordingly, optimal charging duration $\tau$ can be calculated by

$$\begin{array}{c}\hfill {\tau }^{*}=T-{\displaystyle \frac{T}{1+x^{*}/P_{max}}}.\end{array}$$

(6)

Accordingly, on the basis of the derived condition ${\tau }_i=(T-\tau )g_i^2/{\sum }_{i=1}^M{h}_i^2$, optimal data transmission duration ${\tau }_i$, $i=1,2,\cdots ,M$, can be denoted by

$$\begin{array}{c}\hfill {\tau }_i^{*}=(T-{\tau }^{*}){\displaystyle \frac{g_i^2}{{\sum }_{i=1}^M{g}_i^2}}.\end{array}$$

(7)

3.1.2. Optimization with QoS Constraint

In real application scenarios, we usually need to ensure that effective throughput is greater than a threshold for satisfying the QoS requirement. Then, updated optimization problem can be shown as follows

$$\begin{array}{cc}\hfill & \underset{P_S,\tau }{max}{E}_{Eff}={\displaystyle \frac{B(T-\tau ){log}_2(1+\frac{P_S({\sum }_{i=1}^M{g}_i^2)\eta \tau }{(T-\tau )(P_{at}{g}_{at}+{\sigma }^2)})}{(P_S\tau +P_{C}T)}},\hfill \\ \hfill & \begin{array}{ccc}s.t.\hfill & \tau \hfill & >0,0<P_S\le P_{max},\hfill \\ \hfill & \hfill B& \hfill {\displaystyle \frac{(T-\tau )}{T}}{log}_2(1+{\displaystyle \frac{P_S({\sum }_{i=1}^M{g}_i^2)\eta \tau }{(T-\tau )(P_{at}{g}_{at}+{\sigma }^2)}})\ge R_{min},\end{array}\hfill \end{array}$$

where $R_{min}$ is the lower bound of effective throughput. Following a similar process in Section 3.1, optimal $P_S$ is equal to $P_{max}$. Accordingly, the above constraint for effective throughput can be transformed to ${\displaystyle \frac{B(T-\tau )}{T}}{log}_2(1+{\displaystyle \frac{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta \tau }{(T-\tau )(P_{at}{g}_{at}+{\sigma }^2)}})\ge R_{min}$. Through some mathematical manipulations, effective throughput constraint can be further updated to

$$\begin{array}{cc}\hfill & -{\displaystyle \frac{B}{R_{min}ln2}}(W_0[-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}exp({\displaystyle \frac{R_{min}ln2}{B}}\hfill \\ \hfill & (1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))]+{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}})\hfill \\ \hfill & \le {\displaystyle \frac{\tau }{T-\tau }}\le -{\displaystyle \frac{B}{R_{min}ln2}}(W_{-1}[-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}\hfill \\ \hfill & exp({\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))]+{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}).\hfill \end{array}$$

(8)

We present corresponding proof as follows

Proof. 

First of all, we use simple mathematical manipulations to transform effective throughput constraint to

$$\begin{array}{c}\hfill {\displaystyle \frac{-R_{min}ln2}{B}}({\displaystyle \frac{\tau }{T-\tau }}+{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }})exp({\displaystyle \frac{-R_{min}ln2}{B}}({\displaystyle \frac{\tau }{T-\tau }}+{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))\\ \hfill \le {\displaystyle \frac{-R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}exp(-{\displaystyle \frac{R_{min}ln2}{B}}({\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}-1)).\end{array}$$

(9)

using the property of lambert W function, Equation (8) can be derived.

Also note that ${\displaystyle \frac{\tau }{T-\tau }}$ is equal to or greater than zero and as such, $-{\displaystyle \frac{B}{R_{min}ln2}}(W_{-1}[-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}exp({\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))]+{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}})$ has to be greater than zero constantly for ensuring the existence of a feasible solution. Accordingly, we have $W_{-1}[-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}exp({\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))]<-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}$. When

$-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}$≤$-1$, this inequality is equivalent to ${\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }})$<$-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}$, since $W_{-1}[x]$ is a monotonically decreasing function. While noting that ${\displaystyle \frac{R_{min}ln2}{B}}$ is greater than zero, ${\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }})$<$-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}$ is not possibly to hold. When $-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}$>$-1$, $-{\displaystyle \frac{B}{R_{min}ln2}}(W_{-1}[-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}exp({\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))]+{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}})$ must be greater than zero because $W_{-1}[-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B{P}_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}exp({\displaystyle \frac{R_{min}ln2}{B}}(1-{\displaystyle \frac{(P_{at}{g}_{at}+{\sigma }^2)}{P_{max}({\sum }_{i=1}^M{g}_i^2)\eta }}))]$ is less than $-1$. Accordingly, $-{\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{P_{max}B\eta ({\sum }_{i=1}^M{g}_i^2)}}$ need to be greater than $-1$ to satisfy minimum effective throughput constraint, which means $P_{max}({\sum }_{i=1}^M{g}_i^2)$≥${\displaystyle \frac{R_{min}(P_{at}{g}_{at}+{\sigma }^2)ln2}{B\eta }}$. Meanwhile, ${\sum }_{i=1}^M{g}_i^2$ is a random variable, which may approach zero. So $P_{max}$ has to be large enough to make such condition satisfiable. Combining Equation (6) and derivation process in III. A, optimal charging duration $\tau$ should approach zero and as such, effective throughput actually approaches $B{log}_2(1+x({\sum }_{i=1}^M{g}_i^2)\eta /(P_{at}{g}_{at}+{\sigma }^2))$. Here, x was defined in III. A. Accordingly, minimum effective throughput constraint can be equivalent to condition that effective received signal to noise ratio (SNR) is greater than $2^{{\displaystyle \frac{R_{min}}{B}}}-1$, denoted by $\gamma (R_{min})$. According to the above analysis, With optimal parameters, the UAV’s received SNR is equal to $x({\sum }_{i=1}^M{g}_i^2)\eta (P_{at}{g}_{at}+{\sigma }^2)$. Accordingly, x needs to be not less than ${\displaystyle \frac{\gamma (R_{min})}{({\sum }_{i=1}^M{g}_i^2)\eta /(P_{at}{g}_{at}+{\sigma }^2)}}$. Then, we can rewrite the optimal x to

$$\begin{array}{c}\hfill \overline{x^{*}}=max\{x^{*},{\displaystyle \frac{\gamma (R_{min})}{({\sum }_{i=1}^M{g}_i^2)\eta /(P_{at}{g}_{at}+{\sigma }^2)}}\}.\end{array}$$

(10)

On the basis of Equation (7), the optimal ${\tau }_1$, ${\tau }_2$, ⋯, ${\tau }_M$ can be updated accordingly.

3.2. CDMA

In terms of CDMA mode, to maximize overall energy efficiency of the UAV, we formulate an optimization problem as follows

$$\begin{array}{cc}\hfill & \underset{P_S,{\tau }_{wp}}{max}Eff={\displaystyle \frac{{\sum }_{i=1}^{M}B(T-{\tau }_{wp}){log}_2(1+\frac{P_S\eta g_i^2{\tau }_{wp}}{(P_{at}{g}_{at}+{\sigma }^2)(T-{\tau }_{wp})})}{P_S{\tau }_{wp}+P_{C}T}},\hfill \\ \hfill & \begin{array}{ccc}\hfill s.t.& \hfill 0<P_S\le P_{max},0& \hfill \le {\tau }_{wp}\le T.\end{array}\hfill \end{array}$$

We can see that it is not possibly to derive closed-form optimal solution in terms of this case. As mentioned above, iterative algorithms have high online computational complexity and can not determine near-optimal transmission parameters in real time, e.g., [7,8,9,18,19,20,21]. While noting that machine learning can well train a DNN model for performing adaptive optimal data collection, it can offer lower latency than conventional iterative algorithms. As such, we intend to design a DDPG approach to arrive at a near-optimal solution.

DDPG Solution

Applying DDPG approach, we construct one critic network and one actor network, respectively, as illustrated in Figure 2, where ${\theta }^Q$ denotes the parameter set of critic network and ${\theta }^{\mu }$ denotes the parameter set of actor network. The input of critic network is the state vector and action vector, and the output is estimated Q value. The input of actor network is the state vector, and corresponding output is the action vector. During the training process, the critic network feeds back an estimated Q value for current action to the actor network to update corresponding parameter set using chain rule based gradient ascent. Adopted critic-actor configuration is presented in Figure 2, where ${\overrightarrow{s}}_t$ is the state vector at time instant t and ${\overrightarrow{a}}_t$ is the action vector at time instant t. In this case, we define the state vector ${\overrightarrow{s}}_t$ as ${[h_1,h_2,\cdots ,h_M]}^T$, which consists of channel gains from all sensors to the UAV at time instant t. Action vector ${\overrightarrow{a}}_t$ is defined as ${[P_S,{\tau }_{wp}]}^T$. Following DDPG approach, we use expected reward function to evaluate the action policy [27], shown as follows

$$\begin{array}{c}\hfill y_t=R_t(s_t,a_t)+\gamma Q(s_{t+1},\mu (s_{t+1}|{\theta }^{\mu })|{\theta }^{\mu })|{\theta }^Q).\end{array}$$

(11)

Here, $y_t$ is the expected reward for adopted action ${\overrightarrow{a}}_t$, $\gamma$ is the discounted factor, $R_t({\overrightarrow{s}}_t,{\overrightarrow{a}}_t)$ is the instantaneous reward with given state ${\overrightarrow{s}}_t$ and action ${\overrightarrow{a}}_t$, ${\overrightarrow{s}}_{t+1}$ is the state at time instant $t+1$, $\mu ({\overrightarrow{s}}_{t+1}|{\theta }^{\mu })$ is the output of actor network. Prior to offline training, state transition tuples $\{{\overrightarrow{s}}_i,{\overrightarrow{a}}_i,R_i,{\overrightarrow{s}}_{i+1}\}$, $i=1,2,\cdots ,N$, need to be randomly generated at first, and then put into a memory buffer of size K. Note that expected reward can be estimated using a critic network. During each training iteration, we need to obtain an action vector from current actor network and then, calculate the resulting reward as well as the state vector in the next time instant. Subsequently, one new experience tuple can be obtained for randomly replacing one existing experience tuple in the memory buffer. Note that within each training iteration, corresponding action vector is the output with random exploration, shown as below

$$\begin{array}{c}\hfill {\overrightarrow{a}}_t=\mu ({\overrightarrow{s}}_t|{\theta }^{{\mu }^{\prime }})+N_t.\end{array}$$

(12)

$N_t$ denotes the variable following normal distribution with the mean value of zero and the variance of v. At the end of each training iteration, v is updated to $\beta$v, where $\beta$ is a constant ranging from 0 to 1. After that, we can obtain the critic network parameter ${\theta }^{Q^{\prime }}$ through minimizing loss function, shown as follows

$$\begin{array}{c}\hfill {\theta }^{Q^{\prime }}=\underset{{\theta }^Q}{min}{\displaystyle \frac{1}{N^{\prime }}}\sum _{i=1}^N{(y_i-Q({\overrightarrow{s}}_i,{\overrightarrow{a}}_i|{\theta }^Q))}^2.\end{array}$$

(13)

The state transition tuples $\{{\overrightarrow{s}}_i,{\overrightarrow{a}}_i,R_i{\overrightarrow{s}}_{i+1}\}$, $i=1,2,\cdots ,N¡$, is directly extracted from the memory buffer. In terms of this case, $R_t({\overrightarrow{s}}_t,{\overrightarrow{a}}_t)$ is defined as instantaneous energy efficiency for the UAV, denoted by

$$\begin{array}{c}\hfill R_t(s_t,a_t)={\displaystyle \frac{{\sum }_{i=1}^{M}B(T-{\tau }_{wp}){log}_2(1+\frac{P_S{g}_i^2{\tau }_{wp}}{(P_{at}{g}_{at}+{\sigma }^2)(T-{\tau }_{wp})})}{(P_S{\tau }_{wp}+P_{C}T)H}}.\end{array}$$

(14)

Applying chain rule based gradient ascent, through extracting $M^{\prime }$ tuples in random from the memory buffer, we further calculate the renewed parameter set of actor network as follows

$$\begin{array}{c}\hfill {\theta }^{{\mu }^{\prime }}={\theta }^{\mu }+{\displaystyle \frac{1}{M^{\prime }}}{\Sigma }_{i=1}^{M^{\prime }}{▽}_{a}Q(s,a|{\theta }^{Q^{\prime }}){{|}_{s=s_i,a=u(s_i)}{▽}_{{\theta }^{\mu }}\mu (s|{\theta }^{\mu })|}_{s=s_i}.\end{array}$$

(15)

Then, we arrive at the updated parameter set of critic network as follows

$$\begin{array}{c}\hfill {\theta }^Q=k_1{\theta }^{Q^{\prime }}+(1-k_1){\theta }^Q,\end{array}$$

(16)

and the updated parameter set of actor network as follows

$$\begin{array}{c}\hfill {\theta }^{\mu }=k_1{\theta }^{{\mu }^{\prime }}+(1-k_1){\theta }^{\mu },\end{array}$$

(17)

where $k_1$ is the exploration constant during the process of learning. The pseudo code is shown in Algorithm 1.

For each training iteration, the above mentioned training process needs to be repeated. After sufficient amount of training iterations, one actor network can be well trained to determine near-optimal transmission parameters in real time.

Algorithm 1 pseudo-code for action policy training

Initialize the critic network $Q(\overrightarrow{s},\overrightarrow{a}|{\theta }^Q)$ and the actor network $\mu (\overrightarrow{s},\overrightarrow{a}|{\theta }^{\mu })$. Initialize the network parameter ${\theta }^Q$ and ${\theta }^{\mu }$. for episode $\in [1,2,\cdots ,M]$ do    Initialize a Rayleigh fading random process and generate initial state ${\overrightarrow{s}}_t$.    for t $\in [1,2,\cdots ,T]$ do      Select action ${\overrightarrow{a}}_t=\mu ({\overrightarrow{s}}_t|{\theta }^{{\mu }^{\prime }})+N_t$.      Randomly generate new state ${\overrightarrow{s}}_{t+1}$.      Use ${\overrightarrow{a}}_t$ and ${\overrightarrow{s}}_{t+1}$ to calculate the resulting reward $R_t$.      Save state transition tuple {${\overrightarrow{s}}_t$, ${\overrightarrow{a}}_t$, $R_t$, ${\overrightarrow{s}}_{t+1}$} into the memory buffer.      Extract a random minibatch of N state transition tuples {${\overrightarrow{s}}_i$, ${\overrightarrow{a}}_i$, $R_i$, ${\overrightarrow{s}}_{i+1}$} from the memory buffer, $i=1,2,\cdots ,N$.      Set expected value function $y_i$ to $r_i+\gamma Q({\overrightarrow{s}}_{i+1},{\mu }^{\prime }({\overrightarrow{s}}_{i+1}|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})$.      Update critic network parameter to ${\theta }^{Q^{\prime }}$ by minimizing the loss function as Equation (13).      Update actor network parameter by performing gradient ascent, shown as ${\theta }^{{\mu }^{\prime }}={\theta }^{\mu }+{▽}_{\overrightarrow{a}}Q(\overrightarrow{s},\overrightarrow{a}|{\theta }^{Q^{\prime }}){|}_{\overrightarrow{s}={\overrightarrow{s}}_i,\overrightarrow{a}=u({\overrightarrow{s}}_i)}{▽}_{{\theta }^{\mu }}\mu (\overrightarrow{s}|{\theta }^{\mu })$.      Update corresponding network parameters as ${\theta }^Q=k_1{\theta }^{Q^{\prime }}+(1-k_1){\theta }^Q$ and ${\theta }^{\mu }=k_1{\theta }^{{\mu }^{\prime }}+(1-k_1){\theta }^{\mu }$.    end for end for

4. Numerical Results

To test the validity of theoretical analysis and proposed optimization approach, we presented some numerical examples on important parameters, such as maximum energy efficiency, effective throughput. All the parameters, used in simulation, have been given in

Table 1. Parameters used in the simulation.

Parameter	Meaning	Value
${\sigma }^2$	Noise Power	1$\times {10}^{-8}$ Watt
B	Available Bandwidth	10 kHz
$\rho$	Path Loss Exponent	2
PC	Circuit Power of Source Node	0.01 Watt
$\eta$	DC Conversion Efficiency	0.8

. To simplify the simulation process, $P_{at}$ is set to a very small value. All channel gains are assumed to be the same and follow rice distribution, all sensors’ positions are assumed to follow normal distribution in a circle area, and the UAV is assumed to be in stationary hovering state. Since the noise at the sensor is usually very tiny, Noise power consumption is set to be $1\times {10}^{-8}$ Watt.

Figure 3 presents the minimum energy consumption with respect to channel power gain under TDMA mode. Note that under TDMA mode, minimum energy consumption can be calculated by substituting optimal charging power and optimal charging duration into the objective function in Section 3.1. As expected by intuition, we see that energy efficiency monotonically increases with channel power gain. This is because that when channel condition deteriorate, charging power level needs to be higher for keeping the amount of collected data unchanged.

Figure 4 presents the effective transmit power x with respect to channel power gain under TDMA mode. As expected, we see that the optimal power level generally decreases as channel power gain increases. We also see that descending velocity slightly decreases with the increasing channel power gain. This is because that when channel condition becomes better, less charging power is needed in general for collecting a fixed amount of data.

Figure 5 presents the effective throughput with respect to channel power gain under TDMA mode. We can see that the resulting throughput increases as channel power gain increases. This is because that received SNR increases as channel power gain increases. Note that since optimal charging power and charging duration varies with channel power gain, effective throughput increases with channel power gain in approximately linear manner.

Figure 6 presents the numerical results on average energy efficiency per sensor, under CDMA mode, from gradient ascent, DDPG based approach and exhaustive search. It shows that the result from exhaustive search result is very close to that from DDPG approach and as such, our proposed DDPG approach achieves near-optimal performance. Furthermore, we compare the performance of DDPG approach with that of gradient ascent and show that DDPG approach has much better performance. Note that as the step size is relatively big, the result from gradient ascent is not good.

5. Conclusions

In this paper, we investigated the issue of energy efficiency maximization in wirelessly-powered sensor network with an attacker. In the case of TDMA, we maximized the energy efficiency through deriving closed-form optimal charging power, charging duration and transmission durations. Additionally, in the case of CDMA, we proposed a DDPG based approach to arrive at an action policy for determining near-optimal transmission parameters in real time, and reduce the calculation complexity. These results will be very valuable for arriving at energy efficient wirelessly-powered sensor network.