GDR: A Game Algorithm Based on Deep Reinforcement Learning for Ad Hoc Network Routing Optimization

Abstract

Ad Hoc networks have been widely used in emergency communication tasks. For dynamic characteristics of Ad Hoc networks, problems of node energy limited and unbalanced energy consumption during deployment, we propose a strategy based on game theory and deep reinforcement learning (GDR) to improve the balance of network capabilities and enhance the autonomy of the network topology. The model uses game theory to generate an adaptive topology, adjusts its power according to the average life of the node, helps the node with the shortest life to decrease the power, and prolongs the survival time of the entire network. When the state of the node changes, reinforcement learning is used to automatically generate routing policies to improve the average end-to-end latency of the network. Experiments show that, under the condition of ensuring connectivity, GDR has smaller residual energy variance, longer network lifetime, and lower network delay. The delay of the GDR model is 10.5% higher than that of existing methods on average.

1. Introduction

The Ad Hoc network is a center-less, multi-hop wireless network with self-organizing properties. There is no fixed infrastructure for the entire network; each node is on the move and can stay connected to other nodes in any way dynamically [1]. Due to the particularity of its communication mechanism and the limitation of node resources, the energy is usually powered by batteries, which are difficult to supplement. Therefore, the energy consumption of nodes must be considered. In actual deployment, its nodes are randomly and densely deployed in different locations in the monitoring area. The network load is out of balance because each node consumes energy at different rates. So, how to improve energy utilization efficiency and ensure balanced energy consumption to make the network obtain the maximum life cycle is the question that needs to be considered. At the same time, the Ad Hoc network is dynamic, and the network nodes can move anywhere and can also be turned on and off at any time, which will make the network topology change at any time. The dynamic topology makes it more difficult to guarantee a long lifetime while maintaining connectivity.

At present, some achievements have been made in the research on the energy consumption management of Ad Hoc network. The vehicular edge computing (VEC) proposed in [2] reduces the energy consumption by using edge caching approach to reduce the cost, which treats unused nodes as static nodes with reduced computational overhead. It can be seen as adding some static nodes and changing the topology of the network to control the energy consumption. In addition, to balance the energy, Du Y et al. [3] proposed a topology control game algorithm (EBTG). In order to consider both nodal energy balance and energy efficiency, a new exponent is introduced to improve the utility function. Based on this, they use non-cooperative game theory to optimize the algorithm in [4], which includes two sub-algorithms. One of them only guarantees the single connectivity of the network and another one can guarantee the dual connectivity of the network. In contrast to other algorithms based on game theory, this algorithm can effectively extend the network lifetime and performs better in terms of robustness when guaranteeing the dual connectivity of the network. The above algorithms can effectively control the network topology, improve the network performance and balance the energy consumption of the nodes, but they cannot completely guarantee the full coverage of the network and the connectivity of all sensor nodes. Moreover, the average survival time of nodes and the transmission power of nodes should be given consideration.

Moreover, the control method only for topology has many limitations in practical deployment. Because of the dynamic properties of Ad Hoc networks, a single topology control cannot adapt to the changing network environment. In recent years, the speedy advancement of reinforcement learning has opened up the possibility of solving routine engineering problems. Compared with supervised learning and unsupervised learning, which can only classify network traffic, reinforcement learning can directly generate routes by training intelligent agents in unlabeled data sets through newly connected nodes [5,6]. Many related studies apply reinforcement learning to routing engineering or traffic flow engineering [7,8].

The application effect of reinforcement learning in routing engineering is not so significant at present, because the existing methods [9] are mainly based on neural network (NN for short) architectures (e.g., Convolution Neural Network [10] and Recurrent Neural Network [11]), which are not appropriate for modeling information about graph structures. This is because changes in the topological of dynamic networks imply that the inputs and outputs of CNNs or RNNs are not fixed. Also, for the input or output of dynamic Ad Hoc network information, CNN or RNN cannot generalize well. Even if it is represented, it is very inconvenient to store, because the change of topology means the modification of the whole representation. Graph neural networks (GNN) use graphs to represent network topology information, which can clearly characterize the Ad Hoc network structure and store it more efficiently. In [12], a new GNN-based multipath routing model is proposed to explore the complexity between links, paths and MPCP connection on various topologies. GNN models can predict the topology and network throughput for a given network, providing support for optimal multipath routing. In [13], packets routing strategy based on deep reinforcement learning (PRMD) were proposed to reduce data packet transmission time by learning the data information of the network to forward the packets. These reinforcement learning strategies do not focus on the consumption of network energy, which can lead to changes in the overall network topology and therefore cannot make the network stable for a long time.

To obtain a stable, longer life cycle and lower delay network, and to solve the problem of unbalanced energy consumption during dynamic networking, we propose an adaptive routing algorithm based on game theory and deep reinforcement learning. The method first describes the network topology and network transmission parameters in terms of graph neural networks, and then dynamically adjusts the topology using a game algorithm. In order to maintain stability when the network changes dynamically, reinforcement learning is used to generate automatic routing strategies, so that GNN can attain an approximate optimal performance without a priori information about the environment, and is capable of independent exploration and optimization decisions. The contributions of our work are summarized as follows:

• We apply game theory to the graph representation of routing engineering and pay attention to the performance of each node in the network in terms of energy consumption while ensuring network connectivity and efficiency;
• In view of the dynamic nature of the network, we train a network with reinforcement learning to achieve a near-optimal routing policy without prior information about the environment. It is worth pointing out that our input is a graph representation that is adapted by the game model, which is less computationally expensive and faster;
• We collect and experiment with network traffic data in the real world. The results show that the average end-to-end delay of GDR model is 10.5% higher than that of AutoGNN [14] on average. Meanwhile, the average lifespan of the network nodes is longer, the network energy consumption is more balanced, the energy efficiency of the entire network is improved, and the network structure is more robust.

The remainder of this paper is structured as follows. Section 2 summarizes the related work. Section 3 presents the details of the GRD model. Section 4 is the simulation and result analysis. The conclusion is presented in Section 5.

2. Related Works

Early approaches to enhance the survival time of Ad Hoc networks were mainly based on graph-theoretic topology control approaches. This approach constructs the minimum spanning tree of network topology with Euclidean distance and lacks adaptive capability. Currently more promising approaches mainly use Game Theory and Deep Reinforcement Learning.This paper uses GNN to describe the topology of the network, then uses Game Theory to adjust the topology according to the present state of the network, and finally uses DRL methods to determine a routing policy.

Graph Neural Networks. A graph neural network is an emerging network that operates on graph structural information. Since they were introduced in [15], they have made great strides in many fields. In wireless networks, many related works are presented. In [16], the problem of downlink power control in wireless networks was considered. They utilize a graph neural network and use unsupervised optimization methods to learn the decisions for optimal power allocation. In [17], the crucial interference features are extracted using graph neural networks and the DRL framework is used to explore the optimal allocation strategy. Reference [18] characterizes diverse link characteristics and interference relationships using graphs and proposes a heterogeneous interference graph neural network that enables each link to obtain its individual transmission scheme with neighboring links after a limited information exchange. Therefore, based on the ability of graphs to represent the network structure, it is inevitable to use graphs for network structure representation.

Game theory based approach to network topology adjustment. Wang [19] proposed an algorithm based on this topology control model. The nodes adjust their power according to the average lifetime of the nodes in order to help the nodes with the shortest lifetime to reduce the transmitting power, which can extend the lifetime of the whole network. But in this model, the dynamic adjustment of the network topology relies on a set period or threshold. In practical applications, this threshold is often difficult to define. Aiming at the problems of the existing topology control algorithm of wireless Ad Hoc network based on game theory, such as an unbalanced load of individual “bottleneck nodes”, many redundant links, and short life cycle, a multi-objective fusion network topology control algorithm is proposed in [20]. The weight factor of the game model in this algorithm is manually adjusted, which cannot adapt to the dynamics of the Ad Hoc network.

Deep reinforcement learning based routing Policy. Practices and theories that lead by deep reinforcement learning have been studied for a long time. Kao [21] demonstrates the promise of applying reinforcement learning (RL) to optimize Network-on-Chip (NoC) runtime performance. A multi-agent model-free RL scheme for resource allocation is presented in [22], which mitigates interference and eliminates the need of network model. The proposed schemes are implemented in a decentralized cooperative manner with CRs acting as multi-agent, forming a stochastic dynamic team to obtain an optimal energy-efficient resource allocation strategy. As a result, reinforcement learning techniques in multi-agent environments and distributed networks are becoming increasingly popular.

3. GDR Framework

In this section, we will describe the GDR framework in detail. In Section 3.1, we introduce how to represent Ad Hoc networks with a dynamic graph. Section 3.2 provides the topology control game algorithm and a detailed description of the optimization problem. Section 3.3 develops the routing algorithm based on the DRL framework with GNN.

3.1. Dynamic Graph Construction for Ad Hoc Network

In real practice, there are several examples where the number of neighbors per node is variable, as opposed to the fixed neighbor size of images and text. With such data, it is difficult to express them in any way other than in a graph. The network topology is just such data. This section will describe how Ad Hoc networks can be represented by graphs.

To build the graph of an Ad Hoc network with N transceiver pairs, we regard the ith pair of transceivers as the ith node of the graph. Each node has a feature vector, including environmental information and direct channel state information $h_{ii}^1$, such as the remaining energy $w_i$ of the ith node. The feature vectors of two directed edges between nodes $v_i$ and $v_j$ can include $h_{ij}$ and $h_{ij}$, respectively. Figure 1 shows a construction method for a graph for Ad Hoc networks with three transceivers.

In an Ad Hoc network, each transceiver is frequently distinct from the other, and the information carried by the graph is diverse.The above representation does not do well enough to perform the network, so we define the graph as $H=〈V,E,P〉$, where V represents the set of all transceiver nodes in the network, i.e., N nodes randomly deployed in a region; the link set E represents the set of links between two nodes in the node set V. Each node can communicate with its neighboring reachable nodes. $V_i$ stores the coordinates $d_i$ of point i. For convenience, we treat the graph representation as undirected, i.e., $E_{ij}=E_{ji}$. $E_i$ contains the propagation delay of the link, which is related to the quality of the link. P represents the set of states of a node, including network connectivity, transmitting power, node degree and link quality. In this paper, these specific feature values are integrated into the environment information, which will be introduced in Section 3.2 and Section 3.3.

Figure 2 depicts the graph representation of an Ad Hoc network with two state types, where $v_i$ is the feature vector of node i, and $e_{ij}$ is the feature vector of the edge between nodes $v_i$ and $v_j$. Different colors indicate different states, and the states are stored in $p_i$.

3.2. Game Algorithm for Topology Control

Having represented the network topology in a graph, we can then use the game model to adjust the topology generation to regulate energy consumption through the nodes while ensuring network connectivity. To start with, a simple example of topological control to regulate energy consumption is considered as shown in Figure 3. Nodes A, B, C, and D are deployed within the transmission range of each other, where each node is represented by a small circle for ease of description and is assigned an integer remaining lifetime value, and the number inside the circle indicates the remaining lifetime of the node.

Figure 3a shows the topology generation diagram of nodes at a certain moment, assuming that the current average lifetime is 4, and the remaining lifetimes of nodes A, B, C and D are 7, 3, 4 and 2, respectively. Node A can be accessible to Node B with the smallest transmit power and the least energy consumption, and node B can reach node C with the largest transmit power and the most energy consumption. If the operation continues according to Figure 3a, node B will surely die due to excessive energy consumption. In Figure 3b, the remaining lifetime of node A is 7, which is longer than the average lifetime. Consider increasing the transmitting power to help the surrounding nodes reduce energy consumption, so the communication radius of node A increases from the original node B to the average lifetime node C. In Figure 3c, the node B has a shorter than average lifetime, so consider reducing the transmitting power to reduce energy consumption to extend its lifetime under the condition of ensuring network connectivity. Therefore, the communication radius of node B is reduced from reaching the distant node C to the nearest node B. The remaining lifetime of node C is the same as the average lifetime, so its transmitting power remains unchanged. After the node transmitting power is adjusted, the network remains connected and point B will not be dead prematurely.

In this paper, the topological control game model uses the above idea, and the details of the game model will be presented later in this section.

3.2.1. Game Model

The definition strategy game is $\theta =〈N,C,U〉$, where $N=\left\{1,2,\cdots ,n\right\}$ denotes the game participants, which is the same as the number of nodes in the graph. The strategy space is denoted as $C=\left\{C_1,C_2,\cdots ,C_n\right\}$, where $C_i$ represents the set of strategies that participant i can choose. If there are k alternative strategies for i, then we have $C_i=\left\{c_i\left[1\right],c_i\left[2\right],\cdots ,c_i\left[k\right]\right\}$; $U=\left\{u_1,u_2,\cdots ,u_n\right\}$ is the payoff value obtained by the participants after the game, and $u_i\left(c_i,c_{-i}\right)$ denotes the payoff value obtained by participant i in strategy combination $\left(c_i,c_{-i}\right)$, with $c_i$ denoting the strategy chosen by participant i, and $c_{-i}$ denoting the strategy chosen by the remaining of participants.

3.2.2. Utility Function

It is not easy to quantify the benefits of nodes in the complex node deployment environment of Ad Hoc networks. In order to reflect the network realistically and exactly, we consider the utility function of nodes through the following aspects.

Network connectivity. By introducing network connectivity as a parameter when a node changes its own transmit power, it is possible to ensure that the topology keeps connected after various iterations of the game. Equation (1) represents the connectivity of the network:

$$f_i\left(c_i,c_{-i}\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{connected}\hfill \\ 1,\hfill & \mathrm{unconnected}.\hfill \end{array}\right.$$

(1)

Transmitting power. Depending on the denseness of the nodes within the communication range and sparsity, the neighboring nodes are controlled by adjusting the magnitude of node transmit power nodes and thus achieve the purpose of network load balancing. Definition $p_i^j$ is defined as the number of nodes i with node j the transmission power consumed to establish the communication rate. In Equation (2), $d\left(i,j\right)$ denotes the length of the communication link between node i and neighbor node j; $p_i^{\min }$ denotes minimum transmission power; $a\left(f\right)$ is the communication loss coefficient.

$$p_i^j=d\left(i,j\right)p_i^{\min }{\left({10}^{a{\left(f\right)}^{/10}}\right)}^{d\left(i,j\right)}.$$

(2)

Node degree. An appropriate node degree can effectively improve the energy utilization efficiency. Consequently, the node degree density probability function is added to the integrated utility function. Y represents the probability that the degree of node i is m. In Equation (3), m is the degree of node i; $\lambda$ is the mean value of Poisson random variable.

$$Y\left\{i=m\right\}=\frac{{\lambda }^m\exp \left(-\lambda \right)}{m!}.$$

(3)

Link quality. In the process of data transmission, link quality has an essential influence on data transmission, and choosing a path with high link quality to transmit data can effectively reduce energy consumption. The received signal strength indication value in both positive and negative directions is used to combine the link quality $b\left(i,j\right)$. In Equation (4), $R\left(i\right)$, $R\left(j\right)$ is the signal reception capacity of the node i, j.

$$b\left(i,j\right)=\frac{R\left(i\right)R\left(j\right)}{R{\left(i\right)}^2+R{\left(j\right)}^2}.$$

(4)

Combining the above analysis, the payoff of the utility function is presented as Equation (5), where $\alpha$,$\beta$,$\eta$ and $\mu$ is the weighting factor greater than zero. The weight factor constraint is $\alpha +\beta +\eta +\mu =1$. $p_i^{\max }$ is the maximum transmission power of node i; $E_o\left(i\right)$ is the initial energy of node i; $E_r\left(i\right)$ is the residual energy of node i.

$$u_i=f\left(c_i,c_{-i}\right)\left(\alpha p_i^{\max }\frac{E_o\left(i\right)}{E_r\left(i\right)}+\beta \overline{E_i\left(p_i\right)}+\eta \frac{{\lambda }^m\exp \left(-\lambda \right)}{p_i^{j}m!}+\mu \frac{b\left(i,j\right)}{p_i^{j}d\left(i,j\right)}\right)-\alpha p_i\frac{E_o\left(i\right)}{E_r\left(i\right)}.$$

(5)

From Equation (5), we can see that the utility function is topologically controlled under the premise of guaranteeing network connectivity. If $f_i\left(c_i,c_{-i}\right)=0$, the gain of the utility function is 0, which indicates that, at this time, node i can only have a negative gain after the game. When $u_i\left(c_i,c_{-i}\right)=-\alpha p_i\frac{E_o\left(i\right)}{E_r\left(i\right)}$, from Equations (1)–(4) and combined with Equation (5), we can see that node energy consumption, node degree and link length are negatively related to the utility function, the higher the energy consumption and the longer the link length, the lower the gain. The residual energy and link quality of neighboring nodes are positively correlated with the utility function; the higher the residual energy of neighboring nodes and the higher the link quality, the higher the gain. Therefore, this game model not only reduces node energy consumption, but also has higher link quality between nodes and lower node load.

3.2.3. Nash Equilibrium

If the node i achieves a higher gain value with a smaller transmit power, the node adjusts to the transmit power at this point and updates the neighbor list. At this point, the node does not need to adjust the power again and obtains a larger gain, thus achieving Nash equilibrium. That is, the selection strategy of node i is $c_i=\arg {\max }_{c_iϵC_i}{u}_i\left(c_i,c_{-i}\right)$. The algorithm of the potential game phase is described as Algorithm 1. In Algorithm 1, $p\left(i,j\right)$ denotes the need for power from i to j; $\widehat{P}$ denotes the power set of all points, which can be used to represent the topology; $S_i$ denotes the set of neighbor nodes.

Algorithm 1 Topology Control Game Algorithm.

1: Initialization: 2: $\widehat{P}=\left\{{\widehat{p}}_1,{\widehat{p}}_2,\cdots ,{\widehat{p}}_n\right\},nϵN$ 3: foriinN: 4:   Node i broadcasts its own information with $p_{\max }$ power. 5:   if ($p\left(i,j\right)⩽p_{\max }$) 6:    $j\to S_i$ 7:    number of $S_i\to k$ 8:    sort $p_i\left[1\right]<p_i\left[2\right]<\cdots <p_i\left[k\right]$ 9:    $S_i\&p_i\iff \left\{c_i\left[1\right],c_i\left[2\right],\cdots ,c_i\left[k\right]\right\}\to C_i$ 10: Adjustment: 11: do 12: foriinN 13:   for t in $C_i$ do $u_i^k\left(t,-t\right)$ 14:   $c_i^{\ast }=\arg {max}_{c_i^{\ast }ϵC_i}{u}_i\left(c_i^{\ast },c_{-i}^{\ast }\right)$ 15:   update $S_i\&p_i$ 16: end for 17: while ($\widehat{P}$ no longer changes)

After the Algorithm 1, we are able to obtain a new and more concise topology. The new topology not only reduces node energy consumption, but also has higher link quality between nodes and lower node load. Figure 4 shows the process of establishing the topology of the topology control game algorithm. From Figure 4a, we can see that in the initial state of the network, the communication links constructed between nodes can cause high energy consumption of nodes and lead to premature death of nodes.

From Figure 4b, we can see that, after applying the potential game model, the node can complete the forwarding task without establishing communication to all neighboring nodes in the communication range. It can be concluded that the game model in the algorithm can reduce the node load to improve the link quality. At the same time, this results in smaller inputs and better computational performance for the reinforcement learning agent.

3.3. GNN-Based DRL Agent

In this section, we provide details of the deep reinforcement learning routing optimization algorithm, which is a DRL approach with GNN. First, we briefly introduce reinforcement learning and graph neural networks (GNNs). Then, we illustrate how to define the key reinforcement learning elements in the routing optimization problem. After that, we describe the process of generating actions by agents based on GNN. It includes feature extraction and policy generation. Finally, the specifics of the training process are presented. The framework of the routing optimization is illustrated in Figure 5.

3.3.1. Deep Reinforcement Learning

Since the node state and the link state of the network in the GDR model are dynamic, the network topology, routing strategy, and transmission power need to be dynamically adjusted according to the current node energy. The parameters $\alpha$, $\beta$, $\eta$, and $\mu$ for topology control and energy control in the utility function of Equation (5) need to be determined and, since the network state is dynamic, manually setting these parameters for the network will not be adaptive. Therefore, a DRL model is designed here to learn the properties of the network and pave the way for optimization decisions.

DRL is suitable for achieving decision optimization in situations where the environment model is unknown. In a system of environmental state change that can be represented as a Markov process, the agent first obtains the environment state $s_t$ at each step t, and then the agent performs action $a_t$, which causes the environmental state to change to $s_{t+1}$. In this phase, the changed environment is evaluated by a specific standard, and the agent receives a reward $r_t$. When this process is completed, the performance of this decision can be represented by the sum of each reward. The decision process is shown in Figure 6.

During reinforcement learning model training, the parameters of DRL are first randomly initialized. Through interaction with the environment, the parameters of DRL are constantly adjusted, so as to generate near-optimal actions. Each complete state transition in reinforcement learning is called an episode. The interaction steps between the agent and environment in each episode can be represented as $s_t$,$a_t$,$r_t$, and the reward for this episode is $E\left({\sum }_{t=0}^T{r}_t\right)$, where T is the number of steps for each episode’s state transition. To make the DRL model converge better, the cumulative rewards are replaced with $E\left({\sum }_{t=0}^T{\gamma }^t{r}_t\right)$, where $\gamma$ is the discount factor. So, the policy of the agent learned is $\pi :\mathcal{O}\to \mathcal{A}$, which can be expressed as ${\pi }_{\theta }\left(a_{i,t}|o_{i,t}\right)$.

The policy function ${\pi }_{\theta }\left(a_{i,t}|o_{i,t}\right)$ gives the probability of taking any possible behavior given a particular state and parameters, and similar to other neural network solutions, we can solve the policy function as long as the objective function is determined. Specifically, the target function can be optimized by simply finding the gradient of the parameter $\theta$ and then updating it in the direction of the rising gradient. This gradient expressed as:

$${\nabla }_{\theta }{E}_{\pi }\left[{\sum }_{t=0}^T{r}_t\right]=E_{\pi }\left[{\sum }_{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }\left(a_{i,t}|o_{i,t}\right){\sum }_{t=0}^{T}r\left(o_{i,t},a_{i,t}\right)\right].$$

(6)

3.3.2. Graph Neural Networks

The GNN model can extract features of the graph structure well, and it can capture the dependencies in the graph through the information propagation between vertices. GNN can obtain the hidden state embedding ${\hslash }_c,cϵ\mathcal{C}$ based on graph perception. Specifically, the hidden state representation of each vertex iteratively is updated by aggregating the features of each vertex’s edges and neighboring vertices. The hidden state embedding of vertex c at time $t+1$ is updated as follows:

$${\hslash }_c^{t+1}=f\left({\chi }_c,{\chi }_c^{\mathrm{con}\left(c\right)},{\chi }_c^{\mathrm{nei}\left(c\right)},{\hslash }_{\mathrm{nei}\left(c\right)}^t\right),$$

(7)

where ${\chi }_c^{\mathrm{con}\left(c\right)}$ is the features of edge adjacent to vertex c, $f\left(·\right)$ is the local transaction function, ${\chi }_c^{\mathrm{nei}\left(c\right)}$ represents the features of the neighbor vertices of c, and ${\hslash }_{\mathrm{nei}\left(c\right)}^t$ represents the hidden state embedding of the neighbor vertices at time t.

In our work, the network based on GNN is built. The network structure is shown in Figure 7. To extract the features of the graph mode representation, we use four graph convolution layers. If too few stacking layers are set, the network does not learn easily. Since the vertices can only identify and aggregate a few neighbors, the vertices will lack feature information of their neighbors. According to general common sense, we tend to think that the deeper the network is trained, the better it will perform, but it actually does not perform that well. If there are too many hidden layers in the network, after multi-hop propagation, almost all vertices are determined to be adjacent to each other. However, in reality, our input is an optimized topology, and each vertex in the graph should not present a highly similar performance. For the routing optimization task, it is sufficient to see only the closer nodes and to consider the global effect. By stacking four GNN layers, features can be extracted efficiently. In addition, a fully connected layer is used to gradually adapt to route choices by adjusting the parameters, and using the softmax layer as the output. In the deep reinforcement learning framework, we use this output to generate the probability distribution of actions. The softmax function converts the input vector to a new vector of values between (0,1) and uses it to represent the probability of choosing a certain action.

3.3.3. DRL Framework

The DRL framework of the routing optimization is illustrated in Figure 5. We will describe each component of the reinforcement learning framework in detail.

• Agent: Based on the “GNN+DRL” framework, a central controller is considered as an agent, which means that our work uses a centralized reinforcement learning algorithm. The agent has the ability to learn and make decisions in the interaction with the environment. The central controller is responsible for performing routing optimization for the network;
• State: State in the reinforcement learning framework represents the information that the agent can obtain from the environment. This information is represented as a graph, including vertexes, edges and global information.

In our work, the state mainly consists of the utility function $U\left(t\right)$ of Section 3.2 and the distance distribution matrix $D\left(t\right)$. $D\left(t\right)$ is derived from the location information carried by each node itself interacting with other nodes. Consequently, the GNN-Based DRL agent system state is defined as:

$$\mathcal{O}\left(t\right)=\left\{D\left(t\right),U\left(t\right)\right\}.$$

(8)

• Action: An action is a process of valid routing. Due to the dynamic nature of the Ad Hoc network, new incoming or outgoing nodes can cause changes in the state of the environment.

At every single step, we define the actions as the allocation of the newly accessed nodes, which can be expressed as:

$$\mathcal{A}\left(t\right)=\left\{a_1\left(t\right),a_2\left(t\right),\cdots \right\},$$

(9)

where $a_i\left(t\right)$ denotes a choice of action. In our approach, the action values are a matrix of information about the newly accessed nodes, including $〈V,P〉$ described in Section 3.1. Each edge information E is calculated after the information in P and V. The utility function value of the route is calculated from the carried utility function content.

• Rewards: Instead of following predetermined labels, the learning agent optimizes the behavior of the algorithm by continuously obtaining rewards from the external environment. The principle of the “reward” mechanism is to tell the agent how comparatively good the current action is. In other words, the reward function directs the optimization direction of the algorithm. Accordingly, if we link the design of the reward function to the optimization goal, the performance of the system will be improved, driven by the rewards.

In general, actions that satisfy all nodes’ requirements without breaking constraints and that are more stable are considered favorable and are encouraged, and the agent will receive a positive reward. This means that the maximum probability of choosing the current action should be enforced. Conversely, actions that violate constraints or cause a severe imbalance in the network are considered to be failures and are discouraged, and negative rewards are fed back to the agent. This means that the agent has more possibilities to find alternative routing optimization decisions. Consistent with the maximization of cumulative discount rewards, the overall optimization goal is achieved by continuously promoting routing optimization policies.

During the learning process, the agent continuously updates the policy driven by the cumulative reward function, until the best policy for routing optimization is learned. We optimize the loss and back-propagate the gradients through the policy network. The loss function are as follows:

$${\mathcal{L}}_{\theta }={\sum }_{t=0}^T\left(\log {\pi }_{\theta }\left(a_{i,t}\left|o_{i,t}\right.\right)\left({\sum }_{t^{{}^{\prime }}=t}^{T}r\left(o_{i,t^{{}^{\prime }}},a_{i,t^{{}^{\prime }}}\right)-b_{i,t^{{}^{\prime }}}\right)\right).$$

(10)

Therefore, the total loss from each small batch is given by:

$${\mathcal{L}}_{\mathrm{total}}=\frac{1}{Z}{\sum }_{i=1}^Z{\mathcal{L}}_{\theta }.$$

(11)

Algorithm 2 is a pseudo-code that describes the operation of the DRL agent. We use the data from each small batch to perform the optimization. At the beginning, nodes are initialized randomly with an arbitrary position $\vartheta$ and utility information $\xi$. After that, some nodes are dropped randomly as Z. At the same time, we initialize the environment by randomly setting the parameter of the GNN model and the parameter of the utility function as $\theta$, including $\alpha$, $\beta$, $\eta$ and $\mu$, where $\alpha +\beta +\eta +\mu =1$. Then, we operate on each node in Z until all nodes have been traversed. We initialize the environment $\theta$ for each node. Then, we construct the graph $H=〈V,E,P〉$ of the current topology, which is introduced in Section 3.1 and Section 3.2.1. Observe the state $\mathcal{O}\left(t\right)$ from the graph for each case, which can be found in Equation (8) in detail. What is more, we select routing link $a_{i,t^{{}^{\prime }}}$ form $\mathcal{A}\left(t\right)$, which can be found in Equation (9) in detail and perform routing link selection. After that, we use the game model to adjust the topology generation to obtain a more optimal structure, as shown in Section 3.2. The performance is especially judged by the reward $r\left(o_{i,t^{{}^{\prime }}}\right)$ after execution.

Algorithm 2 DRL Agent Operation.

1: Initialization: 2: Nodes is initialized randomly with an arbitrary position $\vartheta$ and utility information $\xi$ 3: Some nodes is dropped randomly as Z. 4: The parameter of GNN model and utility function is initialized as $\theta$, where $\alpha +\beta +\eta +\mu =1$ in utility function 5: begin 6:   For i in Z, do 7:   Initialize the environment $\theta$ 8:   For the current moment t, do 9:     Construct the graph $H=〈V,E,P〉$ of current topology 10:    Observe the state $\mathcal{O}\left(t\right)$ from the graph 11:    Select routing link $a_{i,t^{{}^{\prime }}}$ form $\mathcal{A}\left(t\right)$ and perform routing link selection 12:     Use the game model to adjust the topology generation b 13:     Obtain the reward $r\left(o_{i,t^{{}^{\prime }}},a_{i,t^{{}^{\prime }}}\right)$ 14:   $t=t+1$ 15:   End for 16:  Calculate the loss of routing link selection ${\mathcal{L}}_{\theta }$. 17:  Calculate the total loss ${\mathcal{L}}_{total}$. 18:  Update the network parameter $\theta$ and utility function parameter, $\alpha$, $\beta$, $\eta$ and $\mu$. 19:  End for 20: end

4. Simulation and Evaluation

In this section, we present experiments to evaluate our proposed routing optimization game algorithm. It was conducted in Windows OS (CPU is Intel core i7-7700 3.6HZ; GPU is NVIDIA GeForce RTX 3080, which includes 8GB of graphics memory). Some hyperparameters are summarized in

Table 1. Simulation parameters.

Parameter	Value
Number of initial nodes	50
Initial energy of node	50 J
cell radius	100 m
system loss	1
Wave length	0.1224 m
Monitoring area	300 × 300 m2
Node residual energy	The Poisson distribution that obeys $\lambda$ of 25

.

4.1. Simulation

To start with, we compare our proposed algorithm (GDR) with the following methods: 1. the stochastic policy based on our proposed network structure (SP for short); 2. the CNN-based DRL approach (CNN+DRL for short); 3. the AutoGNN [14] uses GNN as the DRL framework but does not use game theory; 4. the game model with manually modified parameters (GMM for short).

Table 2. A comparison of different algorithms.

	SP	CNN+DRL	GMM	AutoGNN [14]	GDR
Neural networks	GNN	CNN	✘	GNN	GNN
DRL framework	✘	✔	✘	✔	✔
Convergence time	6000 s	⩾60,000 s	✘	48,000 s	25,000 s
Optimal solution	✘	✘	✘	✔	✔
Scalability	✘	✘	✔	✔	✔

illustrates the comparison of different algorithms.

Figure 8a depicts the convergence process of the algorithm we proposed in this paper. As the number of iterations increases, the reward increases and becomes stable. At the beginning of the training, the reward is relatively small and the algorithm is just experimenting and not learning anything useful. After 25,000 iterations, the reward gradually stabilizes. This indicates that the agent has learned and analyzed the historical trajectory, and has mastered the specific method to automatically update the strategy and converge to the optimum in subsequent attempts. Additionally, Figure 8b illustrates the total loss during training. It is evident that after 4800 iterations, the loss is minimized. The slight fluctuations in the training process do not affect the performance of our algorithm.

Figure 9 illustrates the smoothing curve showing the achievable data rates of the various algorithms. The achievable data rate of the GDR proposed in the paper is stable at [5500, 5800], which performs the best. Although the convergence time of the stochastic strategy is relatively short, it is not the optimal routing optimization scheme because the achievable data rate is stable at [2700, 2900]. In addition, the performance of the CNN+DRL scheme is worth exploring. As seen in the figure, it fails to converge because the CNN can only process Euclid data. The GDR convergence time is 25,000 s and the optimal routing optimization strategy is learned. Meanwhile, it is known from [14] that the reward of AutoGNN is stable after 48,000 iterations. It is obviously much slower than GDR. This is because the input is the result of the complete network topology, which is much more computationally intensive. In contrast, the input of the algorithm in this paper is the adjusted network, and the graph representation of the network is simpler and more computationally effective.

4.2. Evaluation

The effectiveness of topology control has been demonstrated in [3,4,19,20] and will not be repeated here. To show the superiority of dynamic topology control and deep reinforcement learning, we compare GDR (proposed in this paper) with the Energy Balance Topology control game algorithm ([3] EBTG) and the routing strategy based on deep reinforcement learning ([13] PRMD) for both residual energy and survival time.

Residual energy. The standard deviation variation of node residual energy can visualize the inter-node load. To verify the load balancing of the GDR algorithm, Figure 10 shows the comparison of the variation of the standard deviation of the residual energy of the nodes.

The standard deviation of the GDR algorithm is lower and increases more slowly than that of the PRMD and EBTG algorithm. The PRMD algorithm is concerned with energy balance, but the dynamics of the nodes cannot be properly considered, and it is easy to subjectively misjudge the transmit power of some nodes. It tends to cause the total energy consumption of the link to increase and the residual energy standard deviation of the nodes to rise faster. On the contrary, the GDR and algorithm can be dynamically adjusted according to the network conditions to improve the energy efficiency of the network and make the node lifetime in the network more balanced.

Survival time. The ultimate goal of topology control algorithms is to extend the survival time of the network, which can be taken as the survival time of the network by the first node death time. In this paper, we compare the relationship between the survival time and the number of nodes of PRMD, EBTG and GRD, as Figure 11.

As can be seen in Figure 11, the survival time of algorithm GDR is slightly higher than that of algorithm PRMD and EBTG as the number of nodes increases, and the life cycle of the GDR algorithm improves by about 2.6%, 12.9%, 13.6%, and 14.8%. This is due to the limitations of the manually given parameters, which cannot be adapted to the dynamic network environment. In contrast, the parameters in the GDR algorithm are updated dynamically as the network changes, and the effects of node load and node state on the network survival time are taken into account comprehensively, so the network survival time is longer.

To evaluate the whole model, we conduct experiments from network delay. To demonstrate our advantage in comparison, we train GDR instances and AutoGNN [14] instances in two network scenarios. To demonstrate the efficiency of our algorithm, the two network scenarios are random network structures with 15 and 40 nodes, respectively. It is worth mentioning that the demand lists of both GDR and AutoGNN solutions are identical and randomly generated. We conducted 200 random traffic generation experiments for each to obtain representative conclusions.

Network delay. Network delay is measured by the average end-to-end delay, which is reflected as a specific score in the algorithm of this paper. Figure 12 and Figure 13 show the results of 2 experiments based on different number of nodes, 15 and 20, respectively. In each box plot, the Y-axis shows the specific score, which is the delay calculated from the environmental information available at the time of each evaluation. From the figure, we can see that the end-to-end delay distribution of the network does not differ much when the network has only fifteen nodes. This is mainly due to the simple topology of the network when there are fewer nodes and the complexity of the graph representation of the network is not high. When the network has forty nodes, GDR significantly outperforms AutoGNN. e.g., in about 75% of the experiments, the performance improvement is more than 30%. This is because the graph representation after game model simplification not only improves computational efficiency, but also allows the model to learn something useful to help decision making. In particular, Section 4.1 shows that we can know that GDR training is roughly 1.92 times faster than AutoGNN. As a result, the GDR model is better.

There is variability in the characteristics of the data under different network structures. Without knowing the specific network, we do not have a good criterion upon which to judge whether a model is good or bad. Therefore, in this paper, we use comparison experiments to evaluate the generalization ability of our model by the score of the average reward.

Generalization ability. We evaluated the ability of our GDR model to generalize to real-world network topologies obtained from a small self-study room. From the data collected in that study room, we selected topologies with more than 10 and less than 40 nodes. In particular, we did not consider ring and star topologies. This is because, in these topologies, the number of effective candidate paths for distribution requirements is usually very limited (in many cases a node is connected to only 1–2 nodes). By filtering, we obtained 140 real network topologies with which to perform tests.

To evaluate the generalization ability of our model, we selected the best model during training for comparison experiments. On each topology, we performed 500 evaluation experiments and stored the rewards achieved by GDR and AutoGNN routing strategies and calculated the average value. Figure 14 shows the performance of different models on different topologies (X-axis). The X-axis represents different topologies, which are ordered according to the GDR model scores. The Y-axis indicates the performance of our model relative to the AutoGNN model. It is shown that in 80% of the cases, our model works better than AutoGNN. The delay of the network under the GDR model is 10.5% higher than that of existing methods, on average.

5. Conclusions

In this paper, we proposed a strategy based on game theory and reinforcement learning to improve the balance of network capabilities and enhance the autonomy of the network topology. The model uses game theory to generate an adaptive topology, adjust the node’s power according to the average life, help the node with the shortest life to reduce the power, and prolong the survival time of the entire network. When nodes move in and out of the network dynamically, reinforcement learning is used to automatically generate routing policies to improve the average end-to-end latency of the network. Through theoretical analysis and experimental results, it is proved that, under the condition of ensuring connectivity, GDR has lower load balancing, longer network lifetime, and lower network delay. It reduces the average end-to-end delay of the network and exhibits greater robustness to topology changes.