1. Introduction
Wireless sensor networks (WSN) have gradually served as an essential infrastructure and have been widely used in modern defense, national health, intelligent transportation, environmental monitoring, and so on [
1,
2]. For a long time, however, the limited battery energy of sensor nodes has been a key factor in restricting the performance and popularization of WSN. Benefiting form the breakthrough of wireless charging technology, a novel and practical solution is to deploy charging systems for WSN to form wireless rechargeable sensor networks (WRSN) [
3], which consist of one or more mobile chargers (MC) with high-capacity batteries, a base station (B), a service station (S) that can recharge MC, and lots of sensor nodes. The base station and the service station are usually at the same location, collectively known as BS. MC can move autonomously and is equipped with a wireless charging device to replenish energy for nodes by wireless charging to reduce the number of dead nodes.
Since the death of nodes in WSN may lead to serious consequences such as link disconnection, how to optimize the scheduling of MC to reduce the number of node deaths has become critical and has attracted much attention from the research community. Many studies have been conducted to explore the charging schemes for WRSN, which can be divided into two main categories: offline schemes [
4,
5,
6,
7,
8,
9] and online schemes [
10,
11,
12,
13,
14,
15,
16,
17]. In the offline schemes, it is assumed that the energy consumption rates of sensor nodes are fixed or regular, and thus, MC can plan its charging path in advance and moves periodically along the planned path to replenish energy for the sensor nodes. Offline schemes usually study a fixed charging path that minimizes MC travel distance while minimizing the number of dead nodes. For example, work of [
5] is to calculate a charging path that minimizes node mortality while reducing MC travel distance. However, due to WRSNs usually being deployed in close contact with the environment, the energy consumption of sensor nodes is dynamic and diverse in practice, affected by the surrounding environment, which shows that the pre-defined charging scheduling in the offline schemes cannot well adapt to the dynamicity of the surroundings, leading to a lack of flexibility and to inefficiency.
Unlike the offline schemes, in online schemes, each sensor node monitors its residual energy and proactively sends a charging request to MC once it falls below a threshold. MC will make real-time scheduling to choose a charging candidate node among the receiving requests. Therefore, due to the different request times of charging nodes, the number of request nodes is different, and a fixed charging path cannot be calculated for online charging scheme. Therefore, the optimization objective of the online charging scheme is not the charging path, but the charging sequence. Since the online charging schemes are closely linked to the environment, online charging schemes can be well adapted to the changing energy demand of the WRSN, which can further reduce node dead ratio, but the existing online works still have performance problems for the following reasons. Firstly, most of them do not take the optimization of the charging path into account in whole, which could bring about unnecessary movement of MC and thus increase charging cost. For example, the classical online charging scheme NJNP [
18] only considers the spatial position relationship between nodes and MC, and ignores the influence of node remaining power.
Secondly, most existing online schemes are based on one-to-one charging, by which MC can only charge one sensor at a time, making it prone to resulting in node failure for too long waiting time for charging, and is thus only suitable for small-scale WRSN.
Figure 1 depicts the performance under the charging schemes HPSOGA [
19] and NJNP using one-to-one charging scheme. As shown in
Figure 1, HPSOGA and NJNP significantly reduce the node dead ratio when the number of nodes is small; however, its performance decreases severely as the network size increases. When the number of network nodes exceeds 100, the node dead ratio increases significantly and HPSOGA cannot effectively reduce the node dead ratio.
Thirdly, the existing online charging schemes lack explicitly analyzing and optimizing the charging amount that MC transfers to the sensor node(s) in each charging task. The charging amount is an important factor in determining the performance of the charging scheme; specifically, too much charging amount can lead to a long charging time, causing uncharged nodes to die before they are charged, while too little charging amount can make it difficult for nodes to run efficiently for a long time.
Aiming at the above problems, this paper proposes a novel solution. First of all, one-to-multiple charging [
20], i.e., MC can charge multiple sensor nodes simultaneously, is adopted to increase the rate of energy replenishment of MC. Then, a new scheduling scheme is proposed to optimize both the charging sequence and the charging amount to improve performance. Classical optimization methods [
21,
22] are usually difficult to converge when tackling the above optimization problem with the large dimensionality of dynamic environment state, while reinforcement learning (RL) [
23] uses a trial-and-error approach that allows agents to continuously learn as they interact with the environment in order to achieve a specific goal.
In classical RL process, the agent perceives state
s from the environment and then selects action
a based on state
s and its own parameters; after performing action
a, the environment state transitions from state
s to the next state
and provides a reward value
. The intelligent body learns between state transitions in the form of a sequence
and continuously optimizes its own parameters. Thus, RL has adaptability to dynamic environments and powerful decision-making capabilities, and is thus more suitable for optimizing MC scheduling. For example, the work of [
15] uses reinforcement learning Q-learning, which combines RL and MC to reduce node mortality while increasing MC autonomy. However, the work of [
15] is an offline charging scheme, which is difficult to apply in a dynamic WRSN environment.
Classical reinforcement learning, such as Q-learning, is only applicable to problems with limited state space and action space; it requires a data approximation function approach to deploy value functions and perform state updates, and requires manual design of high-quality learning features. In WRSN, with the large variety and number of sensor nodes and the large amount of continuous high-dimensional data such as residual energy, RL is difficult to apply to such problems. Deep learning [
24], with its rapid development, has more powerful processing ability for continuous high-dimensional data. Therefore, reinforcement learning can be combined with deep learning to form deep reinforcement learning (DRL) with both high-dimensional continuous data processing ability and powerful decision-making ability. Therefore, DRL utilizes deep learning for high-dimensional continuous data processing and reinforcement learning for real-time decision making, which can well solve the problem of optimizing MC for scheduling in WRSN.
Specifically, in this paper, we propose an online one-to-multiple (OTM) charging scheme based on Double Dueling Deep Q-Network (3DQN), called OTM3DQN. Based on the effective charging distance of MC, OTM3DQN cellularizes the whole network and then uses 3DQN to determine the optimal charging cell sequence from the global consideration to minimize the node dead ratio. Meanwhile, in order to improve the charging efficiency, this scheme calculates the charging amount of each cell being recharged according to the nodes’ energy demand in the cell, the network survival time, and MC’s remaining energy. Moreover, the scheme further utilizes Double DQN and Dueling DQN to reduce overestimation to speed up the training process. The main contributions of this paper are as follows:
In contrast to existing work, this paper considers both the optimization of charging sequences and the charging amount in one-to-multiple charging. To the best of our knowledge, we present for the first time an optimization scheme that dynamically adjusts the one-to-many charging sequence and the charging amount according to the real-time network state.
We use the 3DQN algorithm to decide the next charging cell from the candidate charging cells and to calculate the optimized amount of charging for the sensor nodes in the cell to improve the performance when avoiding node failure.
Finally, we conducted comprehensive tests to evaluate the performance of our charging scheme under various input parameters and compared our charging scheme with existing typical schemes to determine the superiority of our charging scheme.
The paper is organized as follows:
Section 2 briefly reviews the related work. In
Section 3, we provide the network model and the problem formulation.
Section 4 presents the detailed process of using 3DQN to optimize the charging sequences. Then,
Section 5 gives the proposed charging framework and the optimization of node charge amount. In
Section 6, the simulation experiment setup is described and the simulation results are discussed. Finally,
Section 7 concludes the paper and presents future work.
4. Optimized Charging Sequence
In this paper, we combine DQN, Dueling DQN, and Double DQN into the 3DQN model. In this section, we explain in detail the 3DQN network model, the construction of the learning model, and how to optimize the charging sequence by 3DQN.
4.1. Learning Model Construction
MC will not perform the next charging task until it has performed one; therefore, the online one-to-multiple charging scheme is one that contains sequence of decisions. The charging process is shown in
Figure 3. Suppose MC receives charging requests from A, B, C, and D, then MC scheduling is optimized by optimizing the charging decision sequence as B, A, D, and C. Since RL has a natural advantage for optimizing decision sequences, we use RL to optimize MC charging sequences.
In WRSN, traditional RL is difficult to be applied to such problems because of the presence of large and high-dimensional data. However, deep learning has a powerful high-dimensional data processing capability. Therefore, RL can be combined with deep learning to form deep reinforcement learning with both high-dimensional continuous data processing capability and powerful decision-making capability, which can well solve the optimization problem of scheduling MC in WRSN. As shown in
Figure 4, we have the following definition.
Definition (Step): The moment at which MC makes a charging decision based on the current state.
This paper uses state, action, and reward, and the state after the action is performed to define one-to-multiple charge scheme [
34]. The states, actions, and rewards of a one-to-many charging scheme are defined below.
State: In online one-to-multiple charging, we take the status information of MC and nodes as the status space of WRSN, which is specifically . is the specific position coordinate of MC. contains node status information and charging cell location information, which is formulated as . , where is the node location and remaining energy in the , specifically .
Action: MC makes a decision based on the current state, which is an action. In one-to-multiple charging, it is decided that the charging cell to be charged is one action, and the number of charging cells are the same as the number of action spaces.
Reward: The reward is obtained by MC after completing the action and is used to evaluate the action. The main objective of this paper is to optimize the scheduling of MC in order to decrease the node dead ratio in WRSN. Reducing charging latency is a secondary goal. We utilize the trip length of MC, the node dead ratio, and the request charging cell ratio in online one-to-multiple charging. The reward is defined as:
where
is the distance MC moves between adjacent steps.
is the node dead ratio. The number of charging cells requesting charging is expressed as
, and
m is the total number of charging cells.
At the initial state, MC is in the service station. When MC is working, MC will select a charging cell to charge or will go back to S due to its low energy. As shown in
Figure 4, during a CR, at step
k, the state of WRSN is
; thus, MC determines action
and executes it. MC obtains
and next state
after performing the action
[
17]. We assume that MC chooses
to charge at step
k; in the process of MC moving and charging the charging cell
, the remaining nodes consume energy in the waiting process. After charging the charging cell
, energy updates of nodes in the charging cell
are as follows:
The energy of the remaining nodes is updated as follows:
MC consumes energy due to movement and charging, and its energy updates are as follows:
where
is time spent by MC to move and
is MC’s charging time in
. Compared to the traveling time, the movement time of MC in the charging cell can be ignored. MC will return to the service station to replenish its energy due to lack of energy. After returning to the service station to replenish its energy, MC’s energy will be updated as follows:
According to Formulas (
12) and (
13), when the node is within the charging range, its energy will increase. On the contrary, nodes not within the charging range will reduce their energy due to the waiting process. At the same time, the maximum residual energy of a node is
E when supplementing energy, and the minimum residual energy of a node is 0 when consuming energy.
We applied some criteria for training and creating suitable solutions, namely:
When the remaining energy of MC is sufficient, MC can charge any of the charging cells.
MC does not consume energy when not moving and not charging.
MC does not accept charging requests during the charging process.
MC cannot be interrupted during the charging process.
If the residual energy of MC cannot support the charging task, it must return to the S to replenish the energy.
4.2. Deep Q-Networks
Q-learning is the classical method for charging sequential decision problems [
15]. Under the following policy
, the expected value of the total return obtained by choosing a series of actions
is called the
-value, and then the sequential decision problem can be solved by learning bootstrap estimates based on the
-value. An action
a in state
s follows the ground-truth value of policy
as
where
represents the discount factor. The discount factor’s function is to provide incentives received at various points in the future with various weights and values. The best value is
. Since the
-table can only calculate
-values for a limited state space, it is difficult to apply in the constantly varying WRSN environment; thus, deep neural networks are used as a nonlinear function to approximate
-values, i.e., deep neural networks are combined with classical
-learning to form a deep
-network (DQN).
The DQN improves training efficiency and reduces fluctuations by means of experience replay buffer
and dual networks [
35]. The neural network used to estimate the
-value function is the main network with parameters of
, and the neural network used to output iterative learning target values is the target network with parameters of
. We randomly take a certain number of
b groups of samples from the
for gradient descent calculation to train the main network and eliminate the correlation between the sample data to improve the learning efficiency. The target value of iterative learning is expressed as:
The main network parameter update formula is as follows:
where
is the learning rate; the main network updates the parameters to the target network every
n steps.
4.3. Double Dueling Deep Q-Networks
To pick and evaluate the greedy action, the maximized operators in Formula (
17) employ identical
values. Since action selection and evaluation use the same network, it is more likely that an overestimation of the true action value is generated because of bootstrapping. The double deep
-network (DDQN) [
36] has two DQN networks; the main DQN network outputs the actions and the target DQN network computes the target values for iterative learning to avoid bootstrapping and reduce overestimation. Therefore, the DDQN updates the objective as:
It can be seen from Formula (
19) that we use the main network to select the action and the target network to participate in the calculation of the updated target value. The main network parameters are likewise updated to the target network every
n steps.
In order to further improve its steady-state control, we combine DDQN and Dueling DQN [
37] to produce the Double Dueling Deep
-Network (3DQN) to attain better outcomes. We further stabilize the training process with 3DQN, which contains two different estimators:
is used to estimate the state value and
is the action advantage function. Finally, the output values of the two functions are combined as follows:
where
is a scalar and
is a
-dimensional vector, the
and
are output by two independent streams, and the last hidden layer of the original DQN is divided into these two independent streams. The suggested 3DQN’s design is shown in
Figure 5. The main network is in the upper part of
Figure 5, and the target network is in the lower part of
Figure 5. Two DQNs have Dueling structures and are jointly trained to represent two Q-valued functions, but they are structurally independent.
4.4. Training with 3DQN
As shown in
Figure 5, we combine the
and the
into
, which is the input of the convolution layer. After passing through the fully connected layer, MC position information is combined with the convolution layer’s output as the next layer’s input.
In 3DQN, the main network is represented by , and the target network is represented by . At each step, the generated by the -greedy policy is executed, and the value of decreases monotonically with the training process, with a minimum value of 0.01. Then, through the reward function, we obtain the reward value , and obtain the next state . We store as a tuple into .
In the process of training, a certain number of samples are randomly taken from the
for gradient descent calculation each time, which is less efficient in learning the differences of effects produced by different actions in the samples. Therefore, a weighted sample pool (Prioritized experience replay) [
38] is introduced to set different sampling weights for different samples. The worse the performance of an action in a sample, i.e., the greater the deviation relative to the estimate, the greater the probability that it will be selected by the action to improve learning efficiency. In this paper, we use a stepwise approach to calculate the sampling weights of the samples. The estimation error of sample
i is as follows:
The sampling weight of the sample set is the inverse of the estimation error of the sample, and then the sampling probability of sample
i can be obtained as:
where
indicates the magnitude of the action of the sampling weights. When
, the weighted sampling becomes ordinary random sampling. The OTM3DQN training is described in detail in Algorithm 1.
Algorithm 1: OTM3DQN |
![Sensors 23 03903 i001]() |
5. Charging Scheme Details and Procedures
This section describes the specific steps of the charging scheme in this paper, which includes how to calculate the charging request threshold and the node charging amount, which are aimed at reducing the sensor node dead ratio. For the former, we calculate the node charging threshold by computing the node’s minimum tolerated charging time and determining the conditions under which charging cells send charging requests based on the node charging threshold. In particular, the charging requests for the charging cells are sent by the aggregation nodes. For the node charging amount, we calculate it from three perspectives: individual node demand, average residual time of all nodes in the whole WRSN, and MC’s residual energy, which changes dynamically according to the network state. Then the center of gravity method is used to adjust the charging location based on the node charging amount. In addition, since MC energy is limited, MC will determine whether its energy is sufficient before performing the charging task.
Section 5.2 describes the charging request threshold.
Section 5.3 and
Section 5.4 calculate the node charging amount and MC charging location.
Section 5.5 determines whether MC performs the next charging task.
5.1. The Charging Scheme Overview
Our charging scheme is described as follows: we use MC effective charging distance as the side length of the regular hexagonal division network; the hexagon with internal nodes is named as the charging cell and expressed as . MC maintains the 3DQN algorithm and a service pool , and when a charging cell reaches the charging request condition, it will send a message to MC with the request to charge, then MC receives this message, updates the service pool, and then performs the following operations:
As the proposed scheme is non-preemptive, MC performs a new charging task only after the current charging task is completed. After performing the charging task, MC obtains the current WRSN state as an input to the 3DQN algorithm and derives the next optimal charging cell.
5.2. Charging Request Threshold
The charging request threshold (defined as
) determines when a cell sends a charging request. The node residual energy reaches threshold when the node residual energy falls below a certain level, so the threshold
for node
is
where
is the threshold factor. If whole charging cell is considered, a simple analysis from Formula (
23) shows that the threshold for sending a charging request from the charging cell
should be a simple sum of the thresholds of each node:
This paper takes into account that, when more than half of the number of sensor nodes within the charging cell residual energy drop below the threshold, or the value of the residual energy within the charging cell is below the
, then this charging cell sends a charging request. Therefore, the service pool of MC is expressed as:
5.3. Charging Amount of Nodes
In previous studies, both the full and partial charging schemes of sensor nodes have significant limitations which cannot meet the dynamic WRSN environment. The charging amount of node is discussed from three aspects below.
- (1)
Analysis from individual sensor node
Since lowering the node dead ratio as much as feasible is the aim of this paper, the energy replenished to each sensor node should be as much as possible to ensure that it does not die before the next charging service; thus, the value of
for
is taken as follows:
The charging amount
of the
where
is located is:
- (2)
Analysis from all charging cells
When charging the charging cell, the energy variation of the sensor nodes in the other charging cells should be considered by MC and it should not lose sight of them. After MC finishes this charging task, MC still needs to go to
. We use
to represent the distance between
and
centers; thus, the distance matrix between the centers of charging cells in WRSN is:
The matrix column summation is:
Then, Matrix row summation is
. Thus, the average distance between the center positions of any two charging cells is:
We replace the average length of the movement in MC charging task with
d. Considering only the latency of MC movement, the average waiting time of sensor nodes in the network waiting for MC charging is:
The overall residual energy of the
is expressed as
. Similarly, the received power of the
is
. The average energy surplus of charging cells in the WRSN is:
We use the total energy consumption rate of the internal nodes of the charging cell to represent the energy consumption rate of the charging cell. Therefore, the energy consumption of
is
, and the energy consumption rate of charging cells in the WRSN is:
Then the average network survival time is
. So that not too many sensor nodes die in the network, there is the following equation:
The equation can be rectified to obtain:
After obtaining the energy required for the charging cell, the energy required for each node within the charging cell can be calculated. In this paper, according to the principle that, the longer the survival time, the less energy is allocated, and the survival time of
is
, therefore, the energy that should be allocated to
in
is:
- (3)
Analysis from MC residual energy
To minimize the node dead ratio and simultaneously increase MC’s energy efficiency, the service time of MC to perform a charging task must be manageable. The average charging efficiency of the charging cells in the network is:
Assume that the number of charging cells in the service pool requesting charging is
x. Therefore, we have the following equation:
where
is the residual energy of MC, the
of
is obtained as:
According to the above analysis, the value of
should be the median of the values taken by the three formulas of (
27), (
35), and (
39). If the value of
is greater than the value found in Formula (
27), then
. Otherwise, the value of
is obtained according to Formula (
36), and the energy is allocated according to the principle of energy consumption rate of sensor nodes. Algorithm 2 presents the algorithm’s pseudo-code.
5.4. Mc Charging Location Determination
Through the analysis in
Section 5.3, MC obtains the charging amount of node. Then, MC determines the appropriate charging location to replenish nodes in the charging cell with the energy they need. The coordinates of
in the
in 2D are
, and the required energy
is the weight; the center of gravity of the object in the solution plane are:
Therefore, the adjusted charging position
of MC after reaching the
to be charged is:
Algorithm 2: Charging amount of nodes determination |
![Sensors 23 03903 i002]() |
5.5. Analyze Whether MC Returns to the S
After selecting the next charging cell to be recharged, MC will check whether it has enough energy before recharging. MC can check whether its energy is enough in two steps. First, MC can judge whether its energy is enough to go to the charging cell
center
and return to the S. The energy for MC to move from the current position to the center of
is
The energy returned to the S from the charging cell
center is
If , MC returns to S and this CR ends. Otherwise, MC heads to the charging cell center.
If MC has enough energy to return to S, it should also be determined whether MC has enough energy to perform the charging task. The energy of MC moving from the charging cell
center
to the charging position
is
. The charging time
of MC in the
is
The
energy consumed by MC replenishes energy for sensor nodes in
. From the charging position of the
, the energy returned by MC to the S is
. Therefore, if
, MC returns to S and this CR ends. Otherwise, the nodes in the charging cell are charged by MC. The specifics are explained in Algorithm 3.
Algorithm 3: Whether MC returns S |
![Sensors 23 03903 i003]() |
6. Experiment
In this section, we analyze our proposed scheme through experiments. We provide further experimental information, along with a comparison of several approaches.
6.1. Experimental Settings
We take into account three networks of varying sizes, with 50, 100, and 200 sensor nodes. A 3.7 V/450 mAh alkaline rechargeable battery powers the node. Therefore, maximum energy capacity of node is E = 3.7 V × 0.45 A × 3600 s = 6 kJ. In that initial state of the network environment, the node positions are uniformly and randomly drawn in the square area with a side length of 1000 m, and the BS is located at (500,500). The energy of the node is randomly distributed between 500 and 6000 J. The capacity of MC is = 25 × 3.7 V × 3 A × 3600 s = 1000 kJ since it has 25 Li-ion batteries, each of which has a voltage and power of 3.7 V and 3000 mAh, respectively. The speed of MC is 5 m/s, the transmit power of MC is 20 w, and the energy consumption of MC is 50 J/m. We use the Python simulator (Python v3.6.8) and Pytorch 1.7.0 to build the simulation environment
In the training process, we use a mini-batch size of 32 as the input, 1D convolution layer as the input layer of the neural network, and the dimension of the input layer is the same as that of the state s. We combine the MC position information processed by the fully connected layer with the state-of-nodes information processed by the convolutional layer as the input of the following fully connected layer.
The main network has four hidden layers: two convolutions, one fully connected layer, and two parallel fully connected layers outputting
A and
V, respectively. The number of output channels of
A is the same as the action space, and
V has only single output channel. Finally, these two values are calculated by Formula (
20) to obtain the Q. All networks use the ReLU activation. We used the Adam optimizer to train our model, and the 1 ×
learning rate was applied to the training of all scale networks. The target network structure is the same as the main network structure. The OTM3DQN is trained for 1200 CRs on 100 nodes, 150 nodes, and 200 nodes, respectively. The experimental parameters are shown in
Table 3.
6.2. Result of Comparison with Others
We carry out comparison experiments to show the performance of our proposed scheme, OTM3DQN. The charging schemes we compare include the Greedy scheme and the FCFS scheme. We apply two deep reinforcement learning algorithms RMP-RL [
16] and ACRL [
17] in this paper, they schedule MC to move to center of charging cell and apply the fully charging method. The FCFS scheme and the Greedy scheme are introduced as follows.
(1) FCFS scheme: In the FCFS scheme, when the charging cell meets the conditions for entering MC service pool, the charging cell sends a charging request. Then, MC inserts the charging cell identification that sent the charging request into after receiving the request. Charging cells are selected on a first-come, first-served basis. MC selects a charging cell from and goes to the center of the charging cell to fully charge all nodes in the charging cell. After charging is complete, MC removes the charging cell identification from and continues to receive charging requests.
(2) Greedy scheme: In the Greedy scheme, when the charging cell meets the conditions for entering the MC service pool, the charging cell sends a charging request. Then, MC inserts the charging cell identification that sent the charging request into after receiving the request. At each step, with the Greedy scheme min , MC selects a charging cell from and goes to the center of the charging cell to fully charge all nodes in the charging cell. After charging is complete, MC removes the charging cell identification from and continues to receive charging requests.
We first compare the performance of two deep reinforcement learning algorithms RMP-RL and ACRL in training; these two schemes were studied for 1200 CRs.
Figure 6 shows the change in reward values during training for OTM3DQN, RMP-RL, and ACRL at the scale of 100 nodes, MC speed of 5 m/s, and transmit power of 20 w. As shown in
Figure 6, the performance of OTM3DQN is the highest and most stable. OTM3DQN has a higher reward value because MC supplements a more reasonable energy value for the node when charging the node, thus reducing charging time which can replenish energy for other nodes in time and reduce node dead ratio. Meanwhile, OTM3DQN is more stable and faster than RMP-RL because OTM3DQN uses Double DQN to alleviate the overestimation caused by
bootstrap operation during training, and then uses Dueling DQN to improve the stability of training, which makes OTM3DQN obtain a faster convergence rate and better stability.
ACRL has the best stability during training, but the reward value is lower than RMP-RL because RMP-RL selects actions based on value. In contrast, ACRL selects actions based on probability during training. In the process of training, when performing an action yields a higher reward, ACRL will increase the probability of the action choice; this will give ACRL a greater probability of selecting some of the better-performing actions. Therefore, the ACRL training has good stability, but it can also lead to the appearance of fear forward and fear backward when facing unknown strategies. With unknown strategies, this makes it possible that there are better policies that are not being executed, which is the reason for the lower ACRL reward value.
6.3. Impact of MC Transmitting Power
We mainly talk about how MC transmitting power
affects charging effectiveness. The movement speed of MC is fixed at 5 m/s. The value of
uniformly increases from 10 w to 50 w. We count the number of dead nodes at the completion of each charging task of the MC and calculate the node dead ratio when MC returns to S. In
Figure 7, we show the comparison results with OTM3DQN, RMP-RL, ACRL, Greedy, and FCFS in networks with different numbers of sensor nodes.
As depicted in
Figure 7a,b, in 100, 150, and 200 nodes, the three different network sizes, we observe that node dead ratio and average charging latency are relatively high at the beginning at lower
, and node dead ratio and average charging latency show decreasing trend as
increases. The reason is that, when
is too small, resulting in longer charging time, the residual energy of subsequent nodes rapidly decreases, leading to more nodes with insufficient energy, a sharp increase in charging requests, a sharp increase in node dead ratio, and a rapid increase in the average charging latency. With the gradual increase of
, MC can replenish energy for nodes faster, with more energy remaining in subsequent nodes and shorter waiting time for charging, leading to a lower node dead ratio and average charging latency.
It can be seen from
Figure 7a,b that OTM3DQN has the lowest node dead ratio and the lowest average charging latency among the five charging schemes. This is because OTM3DQN uses the 3DQN algorithm to select the optimal charging cell from the global perspective, rather than just calculating the next charging cell from the data in the
, which takes into account the nodes that are likely to fall into energy deficit, thus reducing the node dead ratio. At the same time, OTM3DQN analyzes the three aspects to derive a reasonable node charging amount, thus making MC not spend too much time charging, allowing subsequent nodes to wait less time, and also then renewing nodes to have more energy to support until the next charge, which results in less node dead ratio and average charging latency.
Although the reward value function is used to guide the deep reinforcement learning training process, Greedy and FCFS can still derive data, such as the number of dead nodes, which can be substituted into the reward function to derive the reward value. To unify the reward values, we use
n for 100 and
m for 32. In
Figure 7c, comparing the three network scales, we observed that the average reward for all five charging schemes tends to increase as
. This is because, when
increases, nodes can replenish their energy faster. The remaining nodes will have a shorter waiting time, resulting in fewer dead nodes and requesting cells and a rise in the average reward value.
We find that there is no optimal value of transmitting power, but after reaching a specific value, the tendency of various indexes to improve with increasing transmitting power becomes slower. For a network size of 100 nodes using the OTM3DQN charging scheme, this value is about 35 w, which varies for different network sizes and charging schemes, but is about 25 w to 40 w, and this range should continue to increase as the network size continues to expand. We can also see that all three charging schemes using deep reinforcement learning outperform the traditional charging schemes, with the OTM3DQN charging scheme performing the best. It can also be seen in
Figure 7c that the traditional charging schemes have substantial limitations due to the overemphasis on one aspect, such as the Greedy, because the Greedy charging scheme averages the traveling distance weight and the residual energy weight of the nodes and focuses only on the charging cells in the
, making the Greedy charging scheme short-sighted.
6.4. Impact of MC Speed
We analyze the performance of the three metrics for five charging schemes with a range of MC speed as a sequence of arithmetic progression from 1 m/s to 10 m/s, and we fix MC transmitting power
is 20 w. From
Figure 8, we can observe that the node dead ratio and average charging latency decrease with the increase of MC movement speed. The average reward also increases with the increase of MC movement speed. The OTM3DQN continues to be the best performer among the five charging schemes, again because OTM3DQN uses the 3DQN algorithm and analyzes the node charging amount, making the selection of charging cells and charging amounts more reasonable, and thus reducing node dead ratio and average charging latency.
It can be seen from
Figure 8 that the performance of FCFS and Greedy charging schemes almost do not improve with the increase of MC speed after reaching a specific value, which is due to the short-sightedness caused by the limitations of FCFS and Greedy. Since FCFS only focuses on the residual energy, and Greedy, although it considers both the traveling distance and the residual energy, cannot dynamically adjust the weights between the traveling distance and the residual energy, MC only moves between a few charging cells under these two schemes, thus making FCFS and Greedy fall into local optimal solutions.
Meanwhile, in
Figure 8c, it can be observed that, when MC speed is low, the enhancement of the performance of the charging schemes by increasing the speed is pronounced. However, as MC speed increases, the enhancement of charging schemes’ performance by increasing the speed is too limited. It can be concluded that, when MC speed reaches a specific range, it is difficult to improve the charging efficiency by increasing MC speed.
6.5. Charging Cost and Energy Efficiency
Table 4 shows the average travel length and energy efficiency obtained by different schemes for 100, 150, and 200 nodes, MC speed of 5 m/s, and transmit power of 20 w. The average travel length is called average cost.
Energy efficiency is the ratio of the replenishment energy of the sensor node to the maximum available energy of MC in CR. We observe that the OTM3DQN charging scheme outperforms the remaining four charging schemes in terms of energy efficiency and average cost. In contrast, the Greedy charging scheme performs just below the OTM3DQN charging scheme in terms of energy efficiency over 100 and 150 nodes. This is because the Greedy charging scheme averages the traveling distance weight and the residual energy weight of the nodes and focuses only on the charging cells in the
, making the Greedy charging scheme short-sighted. In other words, the OTM3DQN charging scheme is more long-term by dynamically adjusting the traveling distance weight and node residual energy weight from a global perspective. The FCFS charging scheme performs the worst since the FCFS charging scheme only focuses on the residual energy of the nodes. As a result, MC charges only a few charging cells in WRSN, which consumes much energy during the movement, leading to higher average cost and lower energy efficiency. We also observe that the Greedy and FCFS charging schemes perform worse as the network size increases, while the OTM3DQN charging scheme performs better, which proves the superiority and scalability of our charging scheme.