Adaptive Dynamic Programming-Based Multi-Sensor Scheduling for Collaborative Target Tracking in Energy Harvesting Wireless Sensor Networks

Abstract

Collaborative target tracking is one of the most important applications of wireless sensor networks (WSNs), in which the network must rely on sensor scheduling to balance the tracking accuracy and energy consumption, due to the limited network resources for sensing, communication, and computation. With the recent development of energy acquisition technologies, the building of WSNs based on energy harvesting has become possible to overcome the limitation of battery energy in WSNs, where theoretically the lifetime of the network could be extended to infinite. However, energy-harvesting WSNs pose new technical challenges for collaborative target tracking on how to schedule sensors over the infinite horizon under the restriction on limited sensor energy harvesting capabilities. In this paper, we propose a novel adaptive dynamic programming (ADP)-based multi-sensor scheduling algorithm (ADP-MSS) for collaborative target tracking for energy-harvesting WSNs. ADP-MSS can schedule multiple sensors for each time step over an infinite horizon to achieve high tracking accuracy, based on the extended Kalman filter (EKF) for target state prediction and estimation. Theoretical analysis shows the optimality of ADP-MSS, and simulation results demonstrate its superior tracking accuracy compared with an ADP-based single-sensor scheduling scheme and a simulated-annealing based multi-sensor scheduling scheme.

1. Introduction

A wireless sensor network (WSN) is usually deployed to monitor the physical phenomena in the geographic area covered by a large number of sensor nodes. It has the advantages of low cost, rapid deployment, self-organization, and fault tolerance, with wide applications such as environmental monitoring [1,2], medical care [3], pension service [4], and intelligent transportation [5].

Target tracking is a typical research problem in WSNs for studying collaborative signal and information processing where the sensors are scheduled by considering the tracking performance and the constrained network resources (e.g., the wireless bandwidth and limited sensor energy).

Usually, sensor scheduling relates to tasking the appropriate sensors at the right time to achieve satisfactory performance by considering the limited sensing, computing, and communication resources of the sensors. Effective sensor scheduling algorithms are needed for collaborative target tracking in order to get accurate estimates and effective resource utilization. For example, sensors are scheduled by using dynamic clusters and duty cycling technology to economize the limited energy of the sensors in [6]. An adaptive sensor scheduling scheme was proposed in [7] to improve tracking accuracy on the premise of employing the same number of sensors based on extended Kalman filter (EKF). However, these existing sensor scheduling methods are based on the local optimization of performance for limited time steps, and may be sub-optimal from a global perspective (especially regarding the performance for all the time steps from the current one to the end or infinity).

Additionally, with the development of new technologies such as new materials, microelectronics, energy storage, and conversion, energy harvesting technologies that can obtain energy from a variety of energy sources in the environment (e.g., light [8], wind [9], thermoelectric energy [10,11], electromagnetic radiation [12]) have recently been developed. Energy harvesting can be used to charge batteries from time to time, and can help to avoid battery replacement. Energy harvesting technologies make it possible for the sensor nodes to obtain energy from the environment at a low cost. Accordingly, an energy harvesting-based WSN was introduced in which the sensor nodes could harvest energy from the environment to power the sensors [13].

Theoretically, the lifetime of an energy harvesting-enabled network is infinite. Subsequently, the key problem for sensor scheduling is no longer to maximize the network lifetime, but to optimize the network performance under the given energy harvesting capability. Hence, the development of energy harvesting technologies has provided a new challenge of infinite-horizon sensor scheduling with finite energy harvesting capability for high-performance target tracking. In this paper, we apply adaptive dynamic programming (ADP) to sensor scheduling for collaborative target tracking in an energy-harvesting WSN over an infinite horizon.

ADP was proposed by Werbos [14]. The typical structure of ADP consists of a critic module, a state transition module, and an action module. Each module can be realized by a neural network (NN) [15,16,17]. Characterized by strong abilities of self-learning and adaptation, ADP has demonstrated a strong capability to find the optimal control policy and solve the discrete system dynamic programming (DP) problem [18], including the adaptive critic design, reinforcement learning [19], and so on, obtaining the approximate optimal performance and the optimal control to satisfy the Bellman optimal principle through the function approximation structure. An ADP-based sensor scheduling scheme for target tracking in an energy-harvesting WSN was proposed in [20], which made the sensor energy consumption and tracking accuracy optimal over the system operational horizon for WSNs. However, only one sensor was scheduled for each time step, and therefore the tracking accuracy improvement was limited. In contrast, due to the energy harvesting capabilities, the motivation of this paper is to present a novel multi-sensor scheduling scheme for global performance optimization over an infinite horizon.

The rest of this paper is organized as follows: In Section 2, the related work is introduced. In Section 3, the energy harvesting model is established based on solar energy harvesting, and the EKF-based target motion model and measurement model are introduced. In Section 4, the details of the target tracking system and the predicted energy consumption model are given. In Section 5, the optimal multi-sensor scheduling problem is abstracted to a mathematical model and the ADP-based multi-sensor scheduling algorithm (ADP-MSS) is proposed. In Section 6, the performance of the proposed algorithm is examined by theoretical analysis and simulations. In Section 7, conclusions are drawn and some suggested future work is discussed.

2. Related Work

In the existing work for target tracking in WSNs, various sensor scheduling schemes have been proposed. By considering the number of scheduled sensors for each time step, they can be classified into one-sensor scheduling schemes and multi-sensor scheduling schemes. Meanwhile, by considering the scheduling mechanisms, they can be classified into non-adaptive sensor scheduling schemes and adaptive sensor scheduling schemes.

For example, a periodic sensor scheduling (PSS) scheme in which sensors sense the target alternatively within the predefined time slots was presented to avoid the inter-sensor interference (ISI) problem and utilize the sensors more effectively [21]. The drawback of PSS is the existence of an empty detection when a scheduled sensor cannot generate an effective measurement, which results in lower tracking accuracy and the wasting of sensor power. A distributed sensor scheduling scheme was proposed in [22] where the tasking sensor is elected spontaneously from the sensors with pending sensing tasks via random competition based on carrier sense multiple access. Each node does not need to know the location of the other nodes, which requires less occupied memory. However, the computation burden of the scheduling is shared to all the active nodes. Aiming to optimize the tradeoff between the tracking accuracy and the energy cost for collaborative target tracking in WSNs, a dynamic sensor selection scheme based on genetic algorithms was proposed in [23].

In the above approaches, only one sensor node is scheduled for performing the measurement at each time step. Generally, tracking performance can be further improved by multi-sensor scheduling. For example, a distributed multi-sensor target tracking algorithm was proposed in [24] by using a cluster-based Kalman filter (KF). At each time step, one sensor is selected as the head to fuse the measurements from the other sensors, estimate the target state using EKF, and send the results to the base station. A distributed-saturation-degree-based algorithm was proposed for target tracking with multiple ultrasonic sensors, where the ISI avoidance problem is converted to the problem of multiple access in a shared channel and the scheduling problem is transformed to a coloring problem [25]. In [26], probability-based prediction and sleep scheduling protocols were presented to improve energy efficiency with limited tracking performance loss. The above work does not analyze the energy consumption of the sensor node, and lacks the adaptation mechanism in response to energy changes in the network.

Some adaptive sensor scheduling solutions can be found in the literature. For example, an adaptive sensor scheduling scheme was introduced by scheduling the next tasking sensor for the next time step according to the predicted tracking accuracy derived from the trace of the covariance matrix of the state estimation [21]. In [27], an energy-efficient target tracking method was proposed, where the KF is used to predict the target location for the next time step, then the sensor node and the cluster are selected to minimize the energy consumption. A multi-step sensor scheduling scheme is adopted based on the adaptive sampling interval approach to achieve fast tracking speed and superior energy efficiency without degrading the tracking accuracy [28]. To improve the performance of energy efficiency and tracking accuracy, authors in [29] proposed a multi-step sensor scheduling approach using the branch-and-bound algorithm. It could achieve the optimal multi-step sensor scheduling solution, but easy led to the “curse of dimensionality” problem.

Nevertheless, the above adaptive sensor scheduling solutions only choose one sensor node for each time step. Similarly, a flexible mechanism to improve the tracking performance can be obtained by adaptive multi-sensor scheduling. For example, in [30], a distributed adaptive multi-sensor scheduling was presented to implement the target tracking with the cooperation of the sensor nodes. In the adaptive sampling interval approach for single target tracking, the sensors are scheduled in alternative tracking mode to implement energy-efficient tracking according to the predicted tracking accuracy based on EKF [31]. To minimize the estimation error over multiple time steps in a computationally tractable fashion, Huber [32] proposed an information-based pruning algorithm for multi-step sensor scheduling by using the information matrices of the sensors and the monotonicity of the Riccati equation. In [33], several suboptimal scheduling algorithms were proposed with the performance expressed by the weighted sum of the estimated error covariance matrix in KF and the energy consumption. The posterior Cramer–Rao lower bound was proposed as a sensor selection metric, which put a constraint on the total number of selected sensors to observe the target over a time window [7,34].

However, all these methods dispatch the sensors based on the optimization of local performance, instead of global performance. In energy-harvesting WSNs, novel design criteria are required to achieve an overall performance optimization over an infinite horizon.

There are some studies in the literature on energy harvesting-based WSNs. For example, the Markov decision process (MDP) was presented to maximize the long-term expected throughput to derive the optimal power level [35]. However, the computational complexity of the MDP-based approaches is generally high due to the large volume of the state and action space. To optimize the transmission performance, Lyapunov optimization theory was used in an energy harvesting wireless communication system [36]. DP was proposed for the optimization of the task scheduling to maximize the quality of service in a solar energy-harvesting Internet of Things [37]. However, DP easily leads to the “curse of dimensionality”.

Recently, data-driven approaches have been widely used in the control field to realize a variety of data-based linear and nonlinear systems, for prediction, evaluation, scheduling, monitoring, diagnosis, decision-making, and optimization [38,39,40,41,42,43,44,45,46]. ADP is a typical data-driven approach for control over an infinite horizon, which can avoid the “curse of dimensionality” problem and the reverse solving problem existing in DP [47]. Up to now, ADP has been applied to nonlinear zero-/nonzero-sum differential games [48,49], optimal tracking control problems [50], optimal control of intelligent grid [51,52], and optimal time slot scheduling of MAC protocol [53]. Recently ADP was also proposed as an optimal sensor scheduling scheme for target tracking in an energy-harvesting WSN [20], by scheduling one sensor for each time step over an infinite horizon considering the global tracking accuracy and energy consumption. However, ADP-based multi-sensor scheduling for collaborative target tracking in energy-harvesting WSNs remains as an open and challenging problem.

3. Basic Models

A sensor node usually consists of a sensing unit, a processing unit, a transceiver unit, and a power unit. The energy is finite in a power unit, while the energy of an energy-harvesting sensor node can be collected by the energy harvesting device through ambient energy from time to time and stored as electric energy.

In this paper, we assume that solar energy-based harvesting technology is adopted by the sensor nodes. We also assume that the WSN of this paper is composed of one sink node and M energy-harvesting sensor nodes. Each sensor node with enough energy can sense the target in its sensing region and transmit the perceived information to the sink node. The sink node fuses the received measurements, predicts the target states, and performs sensor scheduling.

3.1. Solar Energy Harvesting Model of the Sensor Nodes

In this paper, we assume that solar energy harvesting is used by the sensor.

If the sensor’s energy storage capacity is unlimited, the harvested energy of a sensor node can be modeled as [54]:

$$E_h(t_0,\Delta T)={\displaystyle {\int }_{t_0}^{t_0+\Delta T}{\eta }_e{f}_e(t)dt}={\eta }_e{\displaystyle {\int }_{t_0}^{t_0+\Delta T}{f}_e(t)dt},$$

(1)

where $t_0$ is the starting time for energy harvesting, $\Delta T$ is the time duration, $f_e(t)$ is the statistical distribution of the solar energy, and ${\eta }_e$ is the conversion efficiency of the solar panel.

However, the unlimited storage capacity is impractical. Suppose that the maximal energy storage capacity of sensor $i$ is $H_i^{\max }(0<H_i^{\max }<\infty )$. Then, the harvested energy of sensor $i$ with the residual energy $E_{left}$ is $\min \left(E_{left}+E_h(t_0,\Delta T),H_i^{\max }\right)$.

3.2. EKF-Based Prediction and Estimation Model for Target State

In this paper, we will apply EKF to the target tracking problem. The basic idea is to use minimum mean square error as the best estimation criterion and update the current estimated state with the previous prediction and the current measurements [55]. In this paper, we adopt a linear target motion model and a non-linear measurement model, both with Gaussian noise distributions.

The state of the target at the k-th time step at $t_k$ is

$$X(k)={\left[\begin{array}{cc}\begin{array}{cc}x(k)& v_x(k)\end{array}& \begin{array}{cc}y(k)& v_y(k)\end{array}\end{array}\right]}^T,$$

(2)

where $(x(k)$, $y(k))$ are the location coordinates of the target and $\left(v_x(k),v_y(k)\right)$ are the velocity of the target along the x-axis and the y-axis at $t_k$. The target motion is modeled by the following constant velocity motion model

$$X(k+1)=A(\Delta t_k)X(k)+w(\Delta t_k),$$

(3)

$$A(\Delta t_k)=\left[\begin{array}{cccc}1& \Delta t_k& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& \Delta t_k\\ 0& 0& 0& 1\end{array}\right], Q(\Delta t_k)=q\left[\begin{array}{cccc}\frac{\Delta t_k^3}{3}& \frac{\Delta t_k^2}{2}& 0& 0\\ \frac{\Delta t_k^2}{2}& \Delta t_k& 0& 0\\ 0& 0& \frac{\Delta t_k^3}{3}& \frac{\Delta t_k^2}{2}\\ 0& 0& \frac{\Delta t_k^2}{2}& \Delta t_k\end{array}\right].$$

(4)

If the target is detected by n sensors, then the sink will obtain n measurements $z_j(k)\left(j=1,\cdots ,n\right)$. Let $Z(k)={\left[z_1(k),z_2(k),\cdots ,z_n(k)\right]}^T$, then the measurement model is given by

$$Z(k)=\left[\begin{array}{c}{h}_1\left(X(k)\right)\\ h_2\left(X(k)\right)\\ ⋮\\ h_n\left(X(k)\right)\end{array}\right]+\left[\begin{array}{c}{v}_1(k)\\ v_2(k)\\ ⋮\\ v_n(k)\end{array}\right]=h\left(X(k)\right)+v(k).$$

(5)

Some notations used in EKF are listed as follows:

• $\widehat{X}(k+1|k)$: Step prediction for the (k + 1)-th time step using state estimation at the k-th time step.
• $\widehat{X}(k+1|k+1)$: State estimation at the (k + 1)-th time step.
• $\Delta t_k$: Sampling time interval between two successive time steps.
• $w(\Delta t_k)$: Process noise at the k-th time step.
• $Q(\Delta t_k)$: Covariance matrix of the process noise at the k-th time step.
• $q$: Given scalar that represents the intensity of the process noise.
• $v_i(k)$: Measurement noise of sensor i at the k-th time step.
• $v(k)$: Measurement noise at the k-th time step.
• $R_i(k)$, $R(k)$: Covariance matrix of the measurement noise at the k-th time step.
• $h_i\left(X(k)\right)$, $h\left(X(k)\right)$: Measurement function at the k-th time step.
• $H(k+1)$: Jacobian matrix of h at $t_{k+1}$ with respect to $\widehat{X}(k+1|k)$.
• $P(k+1|k)$: Error covariance matrix of the state prediction for the (k + 1)-th time step.
• $P(k+1|k+1)$: Error covariance matrix of the state estimation at the (k + 1)-th time step.
• $I$: Unit matrix.
• $K(k+1)$: Kalman gain at the (k + 1)-th time step.

Both $w(\Delta t_k)$ and $v_i$ are independent and assumed to have zero-mean, white, Gaussian probability distributions. $h_i$ is generally non-linear depending on $X(k)$, the measurement characteristic, and the parameters (e.g., the location) of sensor i.

In EKF, the prediction is operated as

$$\widehat{X}(k+1|k)=A(\Delta t_k)\widehat{X}(k|k),$$

(6)

$$P(k+1|k)=A(\Delta t_k)P(k|k)A^T(\Delta t_k)+Q(\Delta t_k).$$

(7)

The estimation is operated as

$$K(k+1)=P(k+1|k)H^T(k+1){\left[H(k+1)P(k+1|k)H^T(k+1)+R(k+1)\right]}^{-1},$$

(8)

$$\widehat{X}(k+1|k+1)=\widehat{X}(k+1|k)+K(k+1)\left(Z(k+1)-h\left(\widehat{X}(k+1|k)\right)\right),$$

(9)

$$P(k+1|k+1)=\left(I-K(k+1)H(k+1)\right)P(k+1|k).$$

(10)

3.3. Tracking Accuracy

At the k-th step, the sink node schedules the sensors to minimize the global performance, which is composed of the energy consumption and tracking accuracy, under the limited energy harvesting capabilities. However, it is impractical to calculate the error through the difference between the real state and the estimated state because the measurement is unobtainable prior to the sink scheduling the sensors. However, the error covariance based on EKF is available before measuring, and it describes the degree of the difference between the estimation and the expectation. Hence, (11) can be used to evaluate the tracking accuracy:

$$T(k)=\mathrm{trace}\left(P(k|k)\right).$$

(11)

4. The Optimal Sensor Scheduling Problem

4.1. System Assumptions

In our proposed algorithm, the assumptions made about the network model are as follows.

• The sensors and sink node are stationary.
• The sink node has strong computing ability and energy storage capacity with enough memory.
• The sink is aware of the locations of the sensor nodes.
• All sensor nodes are homogeneous (i.e., having the same sensing, processing, and communication capabilities).
• A sensor node and the sink node can communicate directly with each other via a single-hop link.

4.2. Target Tracking Mechanism

Figure 1 illustrates a target tracking scenario in an energy-harvesting WSN. When observed the target, the tasking sensors transmit the perceived measurements to the sink node. Then the sink node fuses the received measurements, predicts the target state for the next time step based on EKF, schedules the next tasking sensor nodes by ADP-MSS and notifies them by the low-power paging channel.

The general tracking system in the energy-harvesting WSN works as follows.

• Initialization. When the target enters the sensor field, the energy-harvesting sensor with enough energy that detects the target for the first time becomes the first tasking sensor. It sends the measurement to the sink node.
• State estimation and prediction. When the sink gets the new measurements, it estimates and predicts the state and error covariance by EKF.
• Sensor scheduling. Based on the above solar energy harvesting model, the sink performs the sensor scheduling by ADP-MSS to minimize the performance which consists of the predicted tracking accuracy and energy consumption.
• Mode swapping. The sink wakes up the tasking sensors for the current time step and switches the others to the sleeping mode via the low-power paging channel.
• Monitoring and transmitting. The tasking sensors monitor the target and transmit the measurements to the sink.

4.3. Energy Consumption Analysis

At the k-th time step, the detection model of sensor i is described as

$$D_i(k)=\{\begin{array}{ll}0& E_i(k)<E^h\\ 1& E_i(k)\ge E^h\end{array},$$

(12)

where $E^h$ is a threshold value for sensing the target, $E_i(k)$ represents the received signal level, and $E_i(k)=E_i^0\exp \left(-\beta d_{(x,i)}\right)$, in which $E_i^0$ and $\beta$ are constant and $d_{(x,i)}$ is the Euclidean distance between the target and sensor $i$. The set of tasking sensors scheduled to track the target ${\Omega }_T(k)$ is a subset of ${\Omega }_D(k)=\{i|D_i(k)=1\}$, which denotes the set of all candidate sensors that possibly detect the target. At $t_k$, the energy consumption of sensor $i$ is

$$E_{con}(i)=\{\begin{array}{ll}{E}_r+E_t(i)+E_p& u_i(k)=1\\ E_s& u_i(k)=0\end{array}.$$

(13)

If $u_i(k)=1$, the sensor i is scheduled, otherwise the sensor i is sleeping. $E_r=e_r{b}_r$ represents the energy consumed to receive $b_r$ bits of data. $E_t(i)=(e_t+e_d{d}_{(s,i)}^2)b_t$ represents the energy consumption due to transmitting $b_t$ bits of data to the sink node s. $E_p=e_p{b}_p$ represents the energy consumption due to sensing and data processing of $b_p$ bits, and $E_s$ represents the energy required for sleeping. $e_r$, $e_t$, $e_d$, and $e_p$ are decided by the specifications of the sensor.

In this paper, the design objective is to schedule the sensors for high tracking performance over an infinite horizon. Set the system state as the residual energy of the energy-harvesting sensors and the system control as the sensor scheduling solution. At $t_k$, before scheduled, the residual energy of sensor $i$ is

$$E_i^{bf}(k)=\min \left(E_i^{af}(k-1,u_i(k-1))+E_h(t_k,\Delta t_k),H_i^{\max }\right)=f(E_i^{af}(k-1,u_i(k-1))),$$

(14)

where $E_i^{af}(k-1,u_i(k-1))$ is the residual energy of sensor $i$ after being scheduled at the (k − 1)-th time step. If scheduled, sensor i must satisfy the restriction

$$E_i^{bf}(k)\ge E_r+E_t(i)+E_p.$$

(15)

Let ${\Omega }_u(k)$ be the subset in ${\Omega }_D(k)$. For time step k, the scheduled sensors must be a subset of ${\Omega }_u(k)$. After being scheduled at the k-th time step, the consumed energy and residual energy of sensor i are

$$E_i^c(k,u_i(k))=\left(E_r+E_t(i)+E_p\right)u_i(k)+E_s\left(1-u_i(k)\right),$$

(16)

$$E_i^{af}(k,u_i(k))=E_i^{bf}(k)-E_i^c(k,u_i(k)).$$

(17)

According to (14), (15), and (17), we can obtain

$$E_i^{af}(k,u_i(k))=f(E_i^{af}(k-1))+g_i{u}_i(k)-E_s,$$

(18)

where $g_i=E_s-\left(E_r+E_t(i)+E_p\right)$. Let $g=\left[g_1,g_2,\cdots ,g_M\right]$, the system state of the k-th time step is $E_{af}(k)=\left[E_1^{af}\left(k,u_1(k)\right),E_2^{af}\left(k,u_2(k)\right),\cdots ,E_M^{af}\left(k,u_M(k)\right)\right]$, and the control of the k-th time step is $u(k)=\left[u_1(k),u_2(k),\cdots ,u_M(k)\right]$. Then, the system model is

$$E_{af}\left(k\right)=f\left(E_{af}\left(k-1\right)\right)+g\times u(k)-E_s,$$

(19)

where $g\times u(k)$ means the Hadamard product (i.e., element-wise product) between $g$ and $u(k)$.

5. ADP-Based Optimal Multi-Sensor Scheduling Algorithm

5.1. The Proposed Algorithm

We analyzed the predicted tracking accuracy and energy consumption respectively for time step k. To acquire the trade-offs between the potentially infinite network lifetime and the tracking accuracy, we define the utility function at time step k as

$$U(k)={\beta }_{1}T\left(k\right)+{\displaystyle \sum _{i=1}^M{E}_i^c\left(k,u(k)\right)},$$

(20)

in which ${\beta }_1>0$ is a coefficient to adjust the weight of the tracking accuracy [7]. It is obvious that $U(k)$ is finite. Define the global performance index as the weight sum of the utility function from time step k to the infinite:

$$J\left(k\right)={\displaystyle \sum _{j=k}^{\infty }{\gamma }^{j-k}U(j)},$$

(21)

where $0<\gamma \le 1$ is a discount factor. Then, we can derive a Hamilton–Jacobi–Bellman (HJB) equation:

$$J\left(k\right)=U\left(k\right)+\gamma J\left(k+1\right).$$

(22)

Hence, the objective function of the optimization multi-sensor scheduling problem for target tracking in an energy-harvesting WSN is

$$\begin{array}{l}\hspace{1em}\underset{u(k)}{\min } J\left(k\right)\\ st \{\begin{array}{ll}{D}_i(k)=1& \forall i\in \left\{i|u_i(k)=1\right\}\\ E_{af}(k)\ge 0& \end{array}\end{array}$$

(23)

Let $J^{*}\left(k\right)=\underset{u(k)}{\min }J\left(k\right)$. Then, we can get the following HJB equation

$$J^{*}\left(k\right)=\underset{u(k)}{\min }\left\{U\left(k\right)+\gamma J^{*}\left(k+1\right)\right\}$$

(24)

and the optimal control sequence $u^{*}(k)$ by

$$u^{*}(k)=\arg \underset{u(k)}{\min }\left\{U\left(k\right)+\gamma J^{*}\left(k+1\right)\right\}$$

(25)

Generally, the optimal performance index function $J^{*}\left(k\right)$ is nonlinear, and it is difficult to obtain the optimal control by directly solving (24). To overcome the above problem, the ADP-MSS is proposed to get the approximate optimal solution in this paper.

A diagram of the proposed ADP-MSS is shown in Figure 2, which is comprised of three modules: model, critic network, and action. The model describes the relationship between the next system state $E_{af}(k+1)$ with the current system state $E_{af}(k)$ and the system control $u(k)$ (i.e., the model in (19)). The critic network evaluating the infinite horizon performance is realized by a neural network, in which the input is the system state and the output is the evaluated performance index $\Phi (k)$ which tends to satisfy the HJB equation defined as in (22). The action is executed to find the optimal control for the evaluated performance in the critic network.

It runs as follows.

At first, let ${\Phi }^{[0]}(k)=0$ for any $k$, then we can obtain the optimal performance index at the first iteration step

$${\Phi }^{[1]}(k)=\underset{u(k)}{\min }\left\{U(k)+\gamma {\Phi }^{[0]}(k+1)\right\}$$

(26)

and the optimal control strategy

$$u^{[0]}(k)=\arg \underset{u(k)}{\min }\left\{U(k)+\gamma {\Phi }^{[0]}(k+1)\right\}.$$

(27)

Next, when the iteration step i = 1, 2, ⋯, we can obtain

$${\Phi }^{[i+1]}(k)=\underset{u(k)}{\min }\left\{U(k)+\gamma {\Phi }^{[i]}(k+1)\right\},$$

(28)

$$u^{[i]}(k)=\arg \underset{u(k)}{\min }\left\{U(k)+\gamma {\Phi }^{[i]}(k+1)\right\}.$$

(29)

The critic network is designed to approximate ${\Phi }^{[i+1]}$. The input is $E_{af}(k)\in {\mathrm{R}}^{1\times M}$ where $\mathrm{R}$ is the set of real numbers and the output is

$${\Phi }^{[i+1]}(k)=w_c^T\left(k\right)\sigma \left(v_c^T\left(k\right)E_{af}\left(k\right)\right).$$

(30)

The optimal object can be expressed as

$${\tilde{\Phi }}^{[i+1]}(k)=U(k)+\gamma {\Phi }^{[i]}(k+1).$$

(31)

Hence, we can define the error of the network as

$$e_c^{[i+1]}(k)={\Phi }^{[i+1]}(k)-{\tilde{\Phi }}^{[i+1]}(k).$$

(32)

Therefore, the objective function needed to be minimized in the critic network is $E_c^{[i+1]}={\left(e_c^{[i+1]}(k)\right)}^2/2$. The steepest descent method is used for the weight update:

$$w_c^{\prime }(k)=w_c(k)-{\alpha }_c\partial E_c^{[i+1]}(k)/\partial w_c(k),$$

(33)

$$v_c^{\prime }(k)=v_c(k)-{\alpha }_c\partial E_c^{[i+1]}(k)/\partial v_c(k),$$

(34)

in which $0<{\alpha }_c<1$ is the learning rate and the updated weights are $w_c^{\prime }(k)$ and $v_c^{\prime }(k)$.

5.2. The ADP-MSS Implementation Process

The pseudocode for ADP-MSS at time step k is given in Algorithm 1. Here, the system state $E_{af}\left(k\right)$ is known, $\delta$ is a very small positive value defined by the user, and ${\Phi }^{[i]}(k)$ denotes the iterative global performance index from time step k to the infinite, at iteration step i. This iteration procedure can be terminated after a predefined number of iteration step (MI) is reached.

Algorithm 1 ADP-MSS

1: set the value of $R_i$, $q$, ${\beta }_1$, $\gamma$, ${\alpha }_c$, $X(1)$, $P(1)$

2: select the initial value of $w_c\left(k\right)$, $v_c\left(k\right)$ randomly from a given region

3: set i = 0; ${\Phi }^{[i]}(k)=0$, $\forall k>0$ and termination = false

4: while (termination = false) do

5:  $u^{[i]}\left(k\right)=\arg \underset{u(k)}{\min }\left\{U\left(k\right)+\gamma {\Phi }^{[i]}\left(k+1\right)\right\}$ (Action)

6:  $E_{af}^{[i]}\left(k+1\right)=f\left(E_{af}\left(k\right)\right)+g\times u^{[i]}\left(k\right)-E_s$ (Model)

7:  ${\Phi }^{[i]}\left(k+1\right)=w_c^T\left(k\right)\sigma \left(v_c^T\left(k\right)E_{af}^{[i]}\left(k+1\right)\right)$ (Critic Network)

8:  ${\tilde{\Phi }}^{[i+1]}\left(k\right)=U\left(k\right)+\gamma {\Phi }^{[i]}\left(k+1\right)$ (HJB equation)

9:  ${\Phi }^{[i+1]}\left(k\right)=w_c^T\left(k\right)\sigma \left(v_c^T\left(k\right)E_{af}\left(k\right)\right)$ (Critic Network)

10:  $e_c^{[i+1]}\left(k\right)={\Phi }^{[i+1]}\left(k\right)-{\tilde{\Phi }}^{[i+1]}\left(k\right)$

11:  $i=i+1$

12:  if $\parallel e_c^{[i+1]}\left(k\right)\parallel <\delta$ or $i>MI$ then

13:   termination = true

14:  else

15:   updated the weight $w_c\left(k\right)$, $v_c\left(k\right)$ by the steepest descent method

16:  end if

17: end while

18: return: $u^{[i]}\left(k\right)$

6. Performance Analysis

6.1. Theoretical Analysis

Now we will prove the convergence of ADP-MSS. That is, when $i\to \infty$, ${\Phi }^{[i]}(k)\to J^{*}(k)$.

Theorem 1.

Let the proposed ADP-MSS be implemented according to (26)–(29), then$\left\{{\Phi }^{[i]}(k),i=0,1,2,\cdots \right\}$is a bounded sequence.

Proof: 

Define a new sequence as follows:

$${\Psi }^{[i+1]}(k)=U^{[i]}(k)+\gamma {\Psi }^{[i]}(k),$$

(35)

in which $U^{[i]}(k)$ is the utility under the control $u^{[i]}(k)$, then $U^{[i]}(k)$ is bounded because the number of scheduled sensors is finite. Set ${\Psi }^{[0]}(k)=0$ for any $k$, and we can obtain

$$\begin{array}{ll}{\Psi }^{[i+1]}(k)& =U^{[i]}(k)+\gamma {\Psi }^{[i]}(k+1)\\ & =U^{[i]}(k)+\gamma \left[U^{[i-1]}(k+1)+\gamma {\Psi }^{[i-1]}(k+2)\right]\\ & =U^{[i]}(k)+\gamma U^{[i-1]}(k+1)+{\gamma }^2\left[U^{[i-2]}(k+2)+\gamma {\Psi }^{[i-2]}(k+3)\right]\\ & =\cdots \\ & ={\displaystyle \sum _{j=0}^i{\gamma }^j{U}^{[i-j]}(k+j)}+{\gamma }^{i+1}{\Psi }^{[0]}(k+i+1)\\ & ={\displaystyle \sum _{j=0}^i{\gamma }^j{U}^{[i-j]}(k+j)}\end{array}$$

(36)

Hence, ${\Psi }^{[i+1]}(k)$ is bounded. According to (28), we can conclude that ${\Phi }^{[i+1]}(k)\le {\Psi }^{[i+1]}(k)$, so $\left\{{\Phi }^{[i]}(k),i=0,1,2,\cdots \right\}$ is a bounded sequence. □

Theorem 2.

Let the proposed ADP-MSS be implemented according to (26)–(29), then$\left\{{\Phi }^{[i]}(k),i=0,1,2,\cdots \right\}$is a monotone non-decreasing sequence, that is,

$${\Phi }^{[i+1]}(k)\ge {\Phi }^{[i]}(k).$$

(37)

Proof: 

Mathematical induction is used in the proof.

At first, when $i=0$, $\forall k, {\Phi }^{[0]}(k)=0$. Then

$${\Phi }^{[1]}(k)\ge {\Phi }^{[0]}(k).$$

(38)

Assuming that ${\Phi }^{[l]}(k)\ge {\Phi }^{[l-1]}(k)$ for $i=l-1, l=1,2,\cdots$, when $i=l$,

$$\begin{array}{ll}{\Phi }^{[l+1]}(k)& =\underset{u(k)}{\min }\{U(k)+\gamma {\Phi }^{[l]}(k+1)\}\\ & \ge \underset{u(k)}{\min }\{U(k)+\gamma {\Phi }^{[l-1]}(k+1)\}\\ & ={\Phi }^{[l]}(k)\end{array}$$

(39)

Therefore, $\left\{{\Phi }^{[i]}(k),i=0,1,2,\cdots \right\}$ is a monotonic non-decreasing sequence. □

From Theorem 1 and Theorem 2, it can be inferred that $\left\{{\Phi }^{[i]}(k)\right\}$ is convergent. Denote

$${\Phi }^{\infty }(k)=\underset{i\to \infty }{\lim }{\Phi }^{[i]}(k).$$

(40)

Theorem 3.

For any$k$, ${\Phi }^{\infty }(k)$is the optimal performance index, that is,${\Phi }^{\infty }(k)$satisfies the HJB equation

$${\Phi }^{\infty }(k)=\underset{u(k)}{\min }\{U(k)+\gamma {\Phi }^{\infty }(k+1)\}.$$

(41)

Proof: 

From Theorem 2, we can get ${\Phi }^{\infty }(k)\ge {\Phi }^{[i+1]}(k)$. Let $i\to \infty$, then we can obtain

$${\Phi }^{\infty }(k)\ge \underset{u(k)}{\min }\{U(k)+\gamma {\Phi }^{\infty }(k+1)\}.$$

(42)

Based on the definition of ${\Phi }^{\infty }(k)$, $\forall \epsilon >0$, $\exists {\Phi }^{[p]}(k)$, such that

$${\Phi }^{[p]}(k)\le {\Phi }^{\infty }(k)\le {\Phi }^{[p]}(k)+\epsilon .$$

(43)

Then, we have

$${\Phi }^{\infty }(k)\le \underset{u(k)}{\min }\{U(k)+\gamma {\Phi }^{[p-1]}(k+1)\}+\epsilon .$$

(44)

$\epsilon$ can be ignored because it is any positive value. Let $p\to \infty$, then

$${\Phi }^{\infty }(k)\le \underset{u(k)}{\min }\{U(k)+\gamma {\Phi }^{[\infty ]}(k+1)\}.$$

(45)

From (42) and (45), we can get (41), which is just the definition of $J^{*}(k)$ after replacing ${\Phi }^{\infty }(·)$ by $J^{*}(·)$. Hence, we can conclude that ${\Phi }^{\infty }(k)=J^{*}(k)$, which means that the sequence of the iterative performance indexes in the proposed ADP-MSS will converge to the optimal solution. □

6.2. Simulation Results

In this paper we used Matlab 2014 as the simulation tool and considered a numerical example in which a WSN is deployed to monitor the moving target in a closed region with 10 m × 10 m square. The WSN contained 24 sensor nodes and one sink located at the center, as shown in Figure 3. For each sensor node, the sensing region was a circle centered on its own location with a radius of 3 m.

In the simulations of this paper, the ranging sensors were used to measure the distance between the sensor and the target. For sensor $i$ located at $\left(x_i,y_i\right)$, the measurement function $h_i$ is

$$h_i\left(X(k)\right)=\sqrt{{\left(x(k)-x_i\right)}^2+{\left(y(k)-y_i\right)}^2}.$$

(46)

The Jacobian matrix for the measurement function is

$$H(k+1)=\left[\begin{array}{cccc}\frac{x(k)-x_1}{\sqrt{{\left(x(k)-x_1\right)}^2+{\left(y(k)-y_1\right)}^2}}& 0& \frac{y(k)-y_1}{\sqrt{{\left(x(k)-x_1\right)}^2+{\left(y(k)-y_1\right)}^2}}& 0\\ ⋮& ⋮& ⋮& ⋮\\ \frac{x(k)-x_n}{\sqrt{{\left(x(k)-x_n\right)}^2+{\left(y(k)-y_n\right)}^2}}& 0& \frac{y(k)-y_n}{\sqrt{{\left(x(k)-x_n\right)}^2+{\left(y(k)-y_n\right)}^2}}& 0\end{array}\right].$$

(47)

We assumed the solar panel’s area was 5 cm × 5 cm. The harvested energy rate was 0.1 W/cm², and energy conversion efficiency was 15%. The max capacity of each battery was $5\times {10}^{-2}$ J with the initial energy being $2.5\times {10}^{-3}$ J, and they had infinite recharge cycles. Meanwhile, the variance of the measurement noise of the sensor nodes changed from 0.01 to 0.1. Except for $E_s$, the energy consumption parameters were borrowed from [31] as shown in

Table 1. Parameters in the energy consumption model.

Parameters	Value
$e_t$	$4.5\times {10}^{-5} \mathrm{J}/\mathrm{bit}$
$e_d$	$1\times {10}^{-11} \mathrm{J}/\mathrm{bit}·{\mathrm{m}}^2$
$e_r$	$1.35\times {10}^{-4} \mathrm{J}/\mathrm{bit}$
$e_p$	$5\times {10}^{-5} \mathrm{J}/\mathrm{bit}$
$E_s$	$1\times {10}^{-6} \mathrm{J}$

, and the other constant parameters are given in

Table 2. Simulation parameters.

Parameters	Value
process noise parameter q	1
coefficient ${\beta }_1$	0.10
discount factor $\gamma$	0.70
learning rate ${\alpha }_c$	0.20
sampling interval	0.05 s
Packet size in each transmission	10 bits
number of nodes in the NN hidden layer	30
initial location of the target	(8.81, 6.23)
computation precision $\delta$	1 × 10−3
max iteration step MI in ADP algorithm	1000

.

In the simulations, the true trajectory of the target was a circle with a radius of 4 m centered at the center of the WSN. The residual energy of the 24 sensors at time step k $E_{af}(k)$ was used as the system state, and could be obtained by the previous system state estimation and the control according to (19). The control was the sensor scheduling scheme $u(k)=\left[u_1(k),u_2(k),\cdots ,u_{24}(k)\right]$, where $u_i(k)\in \left\{0,1\right\}$ and $u_i(k)=1$ means that sensor i was scheduled as one of the tasking sensors at time step k, otherwise the sensor i was not scheduled and could remain in the sleeping mode. While the target was moving in the monitoring area, the tracking system iteratively performed target detection by the scheduled tasking sensors, transmitting the measurements from the tasking sensors to the sink, target state estimation and prediction by the sink, and sensor scheduling by the sink. If the sensors are not properly scheduled, it can result in the failure of the tracking or degradation of the overall tracking performance.

The structure of the adopted critic network was 24–30–1 with 24 inputs, 30 nodes in the hidden layer, and 1 output. Its initial weight values were set randomly from the range (0, 0.5). Figure 4 shows the changes of the performance indexes for the first time-step of ADP-MSS, initialized at 0. It can be found that the change of the performance indexes was monotone non-decreasing as analyzed in Theorem 2, and the curve converged after about 600 iterations.

The true trace and estimated trajectories of the target are shown in Figure 5 when the variance of the measurement noise was 0.05 and the target speed was 5 m/s. The corresponding tracking error is shown in Figure 6, which consists of the Euclidean distance from the true coordinate to the estimated coordinate of the target at time step k. To evaluate the tracking accuracy of ADP-MSS, the tracking errors of an ADP-based single-sensor scheduling algorithm (ADP-SSS) and simulated annealing algorithm-based multi-sensor scheduling (SAA-MSS) are also shown in the same figure, where it is obvious that the proposed approach ADP-MSS was more stable and accurate.

Figure 7 and Figure 8 show the tracking errors respectively while the target speed increased from 1 to 10 m/s and the variance of the measurement noise changed from 0.01 to 0.1. From the curves in these two figures, we can find that the results of the two multi-sensor scheduling schemes (ADP-MSS and SAA-MSS) were more stable and accurate than those of the single-sensor one (ADP-SSS). This is because multiple sensors can provide more information to improve the tracking accuracy using data fusion. In addition, it is easy to find that the results from ADP-MSS scheme were better than those from SAA-MSS. The main reason is that the SAA-MSS only takes the local optimization of the performance into account. In fact, the node’s state is associated with its previous state and may influence its states at the following steps. Hence, from the overall performance perspective, local optimal solutions are not the most reasonable decisions, and may have a negative impact on the global performance.

From Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8, the following conclusions can be drawn:

• The performance index of ADP-MSS was monotonically non-decreasing and converged.
• The multi-sensor scheduling schemes were more stable and reliable than the single one.
• The proposed ADP-MSS could achieve global performance optimality.

7. Conclusions

ADP is an efficient method to solve the dynamic programming problems of discrete systems. This paper introduces the ADP approach (ADP-MSS) to the optimal multi-sensor scheduling problem for target tracking in energy-harvesting WSNs. We present an adaptive scheme to schedule the tasking sensors by considering the global optimization of the performance composed of the energy consumption and tracking accuracy over an infinite time horizon. Theoretical analysis proved that the iterative control by ADP-MSS will converge to the optimal solution. Through simulation results, we found that the multi-sensor scheduling schemes were more stable and reliable than the single sensor scheduling scheme and the proposed ADP-MSS was superior to an SAA-based multi-sensor scheduling scheme from a global perspective. As future work, more advanced ADP based cross-layer sensor network design schemes can be studied by jointly designing the network protocols with the sensor scheduling.