Power Allocation Based on Multi-Agent Deep Deterministic Policy Gradient for Underwater Acoustic Communication Networks

Abstract

This paper proposes a reinforcement learning-based power allocation for underwater acoustic communication networks (UACNs). The objective function is formulated as maximizing channel capacity under constraints of maximum power and minimum channel capacity. To solve this problem, a multi-agent deep deterministic policy gradient (MADDPG) approach is introduced, where each transmitter node is considered as an agent. Given the definition of a Markov decision process (MDP) model for this problem, the agents learn to collaboratively maximize the channel capacity by deep deterministic policy gradient (DDPG) learning. Specifically, the power allocation of each agent is obtained by a centralized training and distributed execution (CTDE) method. Simulation results show the sum rate achieved by the proposed algorithm approximates that of the fractional programming (FP) algorithm and improves by at least 5% compared with the DQN (deep Q-learning network) -based power allocation algorithm.

1. Introduction

Underwater acoustic communication networks (UACNs) have many applications for underwater environments, such as underwater environment monitoring, target tracking, and ocean data collection, which have attracted a lot of research [1]. The authors studied the channel state information (CSI) prediction in UACNs based on machine learning [2,3]. Q. Ren et al. investigated the energy-efficient data collection method for an underwater magnetic induction (MI)-assisted system [4]. In this paper, we focus on the power allocation study because it plays an important role in UACNs optimization. First, the total channel capacity can be improved through power allocation for transmitters, which reduces the negative impact of the limited bandwidth of underwater acoustic channels. Second, power allocation among nodes balances energy consumption and reduces total energy consumption, which is suitable for an energy-limited system. Finally, power allocation can reduce the interference between nodes and improve the service quality of the network. Therefore, considering the particular environment of UACNs, power allocation can overcome problems such as restricted bandwidth, limited energy, and interference, which have a substantial impact on underwater acoustic communication.

According to the characteristics of the underwater acoustic communication environment, many studies have proposed power allocation algorithms to optimize channel capacity [5,6,7,8,9]. K. Shen et al. analyzed the multiple-ratio concave–convex fractional programming (FP) problem and its application in solving power control problems [5]. Jin et al. proposed a joint optimization of slot scheduling and power allocation of sensor nodes to maximize the channel capacity for clustered networks [6]. Authors in [7] investigated a joint power allocation and transmission scheduling algorithm for UACNs, where the transmission start-up time and transmission power are co-optimized to maximize the total transmission capacity. Zhao et al. proposed power allocation based on genetic algorithms and adaptive greedy algorithms [8], which can maximize the channel capacity and system robustness. To adapt to the dynamic underwater acoustic channel, Qarabaqi et al. proposed an adaptive power allocation method that models the channel as an autoregressive process and allows the transmitter to adaptively adjust the power allocation based on channel state information to maximize the signal interference noise ratio (SINR) at the receiver [9]. However, these algorithms require full channel state information (CSI).

Due to the dynamic channel and long propagation delay underwater, it is not efficient to obtain full CSI and execute model-based optimization. Therefore, mode-free-based reinforcement learning (RL) has been introduced to optimize the power control problem, whose model is data driven. The Q-learning and deep Q-networks (DQN) algorithms have been applied to solve power allocation problems in UACNs [10,11,12]. However, the Q-learning-based algorithms result in large action spaces that severely impact computational complexity. In contrast, the deterministic policy gradient (DPG) approach applies to the continuous action space. In response, the authors in [13] proposed to combine DQN and DPG into a deep deterministic policy gradient (DDPG) algorithm based on the actor–critic (AC) framework, which can solve high-dimensional continuous action space problems. Based on this, S. Han et al. proposed a DDPG strategy to optimize the continuous power allocation [14]. However, it takes the nodes as individual agents and does not consider the collaborative learning of the agents.

The multi-agent deep deterministic policy gradient (MADDPG) [15], as one of the AC algorithms, has been applied to much research such as unmanned aerial vehicle (UAV) [16], vehicle networks [17], and other resource allocation because of its high efficiency and collaboration. It also has been applied to power allocation in wireless mobile networks [18]. Inspired by these works, we proposed a power allocation algorithm based on MADDPG for UACNs in this paper, because the multiple underwater nodes generate high-dimensional action and state space, and the collaboration of nodes has the advantage in learning. We take the transmitter nodes as agents and multiple agents can cooperate and share information for network training. Accordingly, we propose to maximize the channel capacity as the objective function, with the constraints of maximum power and minimum channel capacity. We model the power allocation problem as a Markov decision process (MDP) and apply the MADDPG approach to optimize power allocation. The actor and critic network of DDPG is trained using a central trainer, and its parameters are broadcast to multiple agents. Each agent updates its own actor network and inputs state to obtain actions for execution. This centralized training and distributed execution (CTDE) method iteratively trains the neural network until convergence to obtain a power allocation strategy. The main contributions of this study are as follows.

We propose a MADDPG-based power allocation scheme for UACNs. A MDP model is formulated and then the MADDPG is used to solve it. To the best of our knowledge, we first study using the MADDPG approach to solve the power allocation problem in UACNs. Although the MADDPG structure comes from [15], we define the action, state, observation space, and reward function according to the objective function, and thus the MADDPG can be applied to the underwater network power allocation problem.

The consideration of the history information of CSI in the MDP model makes the proposed algorithm applicable to the underwater network involving mobility. Through the CTDE process, the multiple agents are trained collaboratively and make power allocation decisions adaptively to adapt to the changing underwater environment. Our approach is therefore better suited to underwater channels that vary due to fading and node movement.

The MDP model proposed in this paper can provide more QoS requirements in design. In the study, we guarantee QoS by requiring a minimum channel capacity. However, other QoS metrics, such as throughput, delay, or success transmission rate, can also be combined into the objective function. As a result, the MDP model can be adjusted to meet these QoS requirements and the MADDPG structure is still valid in these cases.

Simulation results show the total channel capacity of the proposed MADDPG power allocation performs better than that of DQN-based [19] and DDPG-based [13] algorithms with independent agent training. Also, the proposed method has a much lower running time compared with the FP algorithm, particularly with large networks.

2. System Model and Problem Formulation

2.1. System Model

In this paper, we consider a UACN consisting of $M$ transmitter nodes and $N$ receiver nodes, where the transmitter nodes are deployed at the water bottom and each node is configured with an underwater acoustic transducer, as shown in Figure 1. We use $ℳ\triangleq \left\{1,2,\cdots M\right\}$ and $\mathcal{N}\triangleq \left\{1,2,\cdots N\right\}$ to denote the transmitter nodes and the receiver nodes, respectively. Thus, there are $M\times N$ links in the system. When a node transmits the signal to the target receiver, the signals from other transmitter nodes are considered as interference. We assume that the transmission of all nodes in the network starts and ends at the same time slot for a duration of $T_s$.

We assume the channel is slowly time varying and quasi-stationary with flat fading during a time slot. At the time slot $t$, the channel gain $g_{j,i}^{\left(t\right)}$ between receiver node ${\mathcal{N}}_j \left({\mathcal{N}}_j\in \mathcal{N}\right)$ and transmitter node ${ℳ}_i \left({ℳ}_i\in ℳ\right)$ is denoted by [20]:

$$g_{j,i}^{\left(t\right)}=\kappa \cdot 10\log d_{j,i}^{\left(t\right)}+d_{j,i}^{\left(t\right)}\cdot 10\log a\left(f\right),$$

(1)

where $i\in ℳ$, $j\in \mathcal{N}$. $f$ represents the signal transmission frequency, and $d_{j,i}^{\left(t\right)}$ represents the distance between ${\mathcal{N}}_j$ and ${ℳ}_i$ in the time slot $t$. The expansion factor $\kappa$ is typically 1.5. The $a\left(f\right)$ represents the absorption coefficient. According to the $Francois\propto Garrison$ model [20], the coefficient $a\left(f\right)$ is expressed by

$$a\left(f\right)=\frac{A_1{P}_1{f}_1{f}^2}{f^2+f_1^2}+\frac{A_2{P}_2{f}_2{f}^2}{f^2+f_2^2}+A_3{P}_3{f}^2,$$

(2)

where $A_1, A_2,A_3$ denote the impacts from boric acid, magnesium sulfate salt, and pure water components, respectively. They are functions of seawater temperature, the potential of hydrogen (pH), sound speed, and salinity. The symbols $P_1,P_2,P_3$ denote the water depth pressure of boric acid, magnesium sulfate salt, and pure water components. The $f_1,f_2$ denote the relaxation frequency of the boric acid and magnesium sulfate salts, which also depend on seawater temperature and salinity [21].

Considering the real network scenario of acoustic communication, the signal is interfered with underwater noise. The power spectral density of the noise $N\left(f\right)$ is denoted by

$$N\left(f\right)=N_t\left(f\right)+N_s\left(f\right)+N_w\left(f\right)+N_{th}\left(f\right),$$

(3)

where $N_t\left(f\right)$, $N_s\left(f\right)$, $N_w\left(f\right),N_{th}\left(f\right)$ denote turbulence noise, shipping noise, wave noise, and thermal noise, respectively. These noises are mainly affected by signal frequency, shipping activity coefficient, and wind speed, as discussed in [21].

2.2. Problem Formulation

At time slot $t$, the SINR of the communication link $\left(j,i\right)$ formed from the transmitter node ${ℳ}_i$ to receiver node ${\mathcal{N}}_j$ is expressed as

$$SIN{R}_j^{\left(t\right)}=\frac{{\left|g_{j,i}^{\left(t\right)}\right|}^2{p}_i^{\left(t\right)}}{{\sum }_{k\in ℳ, k\ne i}{\left|g_{j,k}^{\left(t\right)}\right|}^2{p}_k^{\left(t\right)}+{\sigma }_n^2} i,k\in ℳ,j\in \mathcal{N},$$

(4)

where $g_{j,i}^{\left(t\right)}$ denotes the channel gain of the link $\left(j,i\right)$. The symbols $p_i^{\left(t\right)}$ and $p_k^{\left(t\right)}$ denote the transmit power of ${ℳ}_i$ and ${ℳ}_k$ at time slot $t$, respectively. The ${\sigma }_n^2$ is noise power. Accordingly, the channel capacity of ${\mathcal{N}}_j$ in time slot $t$ is denoted by

$$C_j^{\left(t\right)}={\log }_2\left(1+SIN{R}_j^{\left(t\right)}\right).$$

(5)

Our objective is to maximize total channel capacity by optimizing the power allocation, which is subject to maximum transmitting power and quality of service (QoS) requirements. This optimization problem is then formulated as

$$\begin{array}{c}{\mathcal{P}}_1:\underset{p_i^{\left(t\right)}}{\mathrm{maximize}}{\displaystyle {\displaystyle \sum }_{j=1}^N}{C}_j^{\left(t\right)}\\ s.t. 0\le p_i^{\left(t\right)}\le P_{max},\forall i\in M\\ C_j^{\left(t\right)}\ge q_{th},\forall j\in N,\end{array}$$

(6)

where $P_{max}$ is the maximum transmit power and $q_{th}$ is a threshold. The $q_{th}$ ensures the minimum channel capacity of a single link, which is regarded as the QoS requirement. To solve ${\mathcal{P}}_1$, we formulate this problem as a Markov decision process (MDP) model and then apply the MADDPG to solve the problem, which can obtain the power allocation policy.

3. Reinforcement Learning

3.1. Introduction to Actor–Critic

In RL, the agent interacts with the environment and learns the optimal policy to maximize the expected total reward over a time horizon. At time slot $t$, the agent takes action $a^{\left(t\right)}\in \mathcal{A}$ in state $s^{\left(t\right)}\in \mathcal{S}$, where $\mathcal{A}$ and $\mathcal{S}$ represent action space and state space, respectively. After that, the environmental feedbacks reward $r^{\left(t\right)}$ to the agent, and then the agent moves to the next state $s^{\prime \left(t\right)}$. It then forms a sample of experience $(a^{\left(t\right)},s^{\left(t\right)},r^{\left(t\right)},s^{\prime \left(t\right)}$) and stores it into replay memory $\mathcal{D}$. The agent trains the neural network to maximize the discounted future reward when it obtains enough experience samples and then obtains optimal decision strategy. The discounted future reward $R^{\left(t\right)}$ is defined as [22]:

$$R^{\left(t\right)}={\displaystyle \sum }_{\eta =0}^{\infty }{\gamma }^{\eta }{r}^{\left(t+\eta +1\right)},$$

(7)

where $\gamma$ is a discount factor.

The policy updates include the value function-based method and the policy gradient-based method. The actor–critic framework combines these two methods. As shown in Figure 2, the AC network consists of an actor neural network and a critic neural network, with network parameters $\theta$ and $\mu$, respectively. Considering continuous action and state space, we exploit DDPG to solve our objective function; therefore, the actor network updates $\theta$ using the deterministic policy gradient ${\pi }_{\theta }$, while the critic updates $\mu$ using the gradient of the loss function.

The actor and critic are defined as follows:

Actor: The actor network updates policy ${\pi }_{\theta }$, which maps state space $\mathcal{S}$ into action space $\mathcal{A}$, which is denoted by

$${\pi }_{\theta }\left(\mathcal{S} \right):\mathcal{S}\mapsto \mathcal{A}.$$

(8)

According to policy ${\pi }_{\theta }\left(\mathcal{S}\right)$, the actor selects the action by the following rules:

$$a^{\left(t\right)}={\pi }_{\theta }\left(\mathcal{S}\right)+{\mathcal{U}}^{\left(t\right)} \left(a^{\left(t\right)}\in \mathcal{A}\right),$$

(9)

where ${\mathcal{U}}^{\left(t\right)}$ is a random process.

Critic: The critic network estimates the action value $Q_{\mu }\left(s^{\left(t+1\right)}\right)$. It evaluates the new state by the temporal difference (TD) error, which is

$${\delta }^{\left(t\right)}=r^{\left(t+1\right)}+\gamma Q_{\mu }\left(s^{\left(t+1\right)}\right)-Q_{\mu }\left(s^{\left(t\right)}\right).$$

(10)

The action selection weight will be enhanced if the TD error is positive. Otherwise, it is decreased with a negative TD error. The critic network and actor network parameters are updated as follows:

(1) Updates: μ AC uses replay buffer $\mathcal{D}$ to store empirical samples $(a^{\left(t\right)},s^{\left(t\right)},r^{\left(t\right)},s^{\prime \left(t\right)})$. The critic network randomly selects $G$ mini-batch samples ${\{(a^{\left(g\right)},s^{\left(g\right)},r^{\left(g\right)},s^{\prime \left(g\right)})\}}_{g=1}^G$ for network training, and updates the parameters by minimizing the mean-squared loss function between the target Q-value and the estimated Q-value. The loss function is formulated by [13]:

$$L\left(\mu \right)=\frac{1}{G}{\displaystyle \sum }_{g=1}^G{\left[\left(y^{\left(g\right)}-Q_{\mu }\left(s^{\left(g\right)},a^{\left(g\right)}\right)\right)\right]}^2,$$

(11)

$$\mu \leftarrow \mu +{\alpha }_{\mu }\nabla L\left(\mu \right),$$

(12)

where $y^{\left(g\right)}=r^{\left(g\right)}+\gamma Q_{{\mu }^{\prime }}\left(s^{\prime \left(g\right)},{\pi }_{\theta }^{\prime }\left(s^{\prime \left(g\right)}\right)\right)$ denotes the Q-value calculated by the target network and ${\alpha }_{\mu }\in \left(0,1\right)$ is the step size of the iterative update. The target network with parameter ${\mu }^{\prime }$ is used to maintain the stability of the Q-value, where ${\mu }^{\prime }$ is updated periodically by $\mu$ as

$${\mu }^{\prime }\leftarrow \tau \mu +\left(1-\tau \right){\mu }^{\prime },\tau \ll 1,$$

(13)

where $\tau$ is used to slowly update the target network.

(2) Update $\theta$: The actor network is performed by a deterministic strategy, whose parameters are also trained from randomly selected samples. The goal of the actor is to find strategies that maximize the average long-term reward. The network parameters $\theta$ are updated by [13]:

$${\nabla }_{\theta }J\left(\theta \right)\approx {\mathbb{E}}_{s^{\left(g\right)}~\mathcal{D}}\left[{\nabla }_{{\pi }_{\theta }}{Q}_{\mu }\left(s^{\left(g\right)},a^{\left(g\right)}\right){|}_{a^{\left(g\right)}={\pi }_{\theta }\left(s^{\left(g\right)}\right)}\ast {\nabla }_{\theta }{\pi }_{\theta }\left(s^{\left(g\right)}\right)\right],$$

(14)

$$\begin{array}{c}\theta \leftarrow \theta +{\alpha }_{\theta }\nabla J\left(\theta \right),\end{array}$$

(15)

where ${\alpha }_{\theta }\in \left(0,1\right)$ is the step size of an iterative update to ensure that the critic is updated faster than the actor. The operation $\nabla$ represents gradient descent for functions. Similar to the critic network, the t update for the actor target network parameter ${\theta }^{\prime }$ is

$${\theta }^{\prime }\leftarrow {\tau }^{\prime }\theta +\left(1-{\tau }^{\prime }\right){\theta }^{\prime },{\tau }^{\prime }\ll 1,$$

(16)

where ${\tau }^{\prime }$ is used to update the target network.

3.2. MADDPG

The UACNs environment contains multiple nodes. It is more efficient to use multi-agent reinforcement learning like MADDPG than DDPG with independent training by a single agent. However, training multiple agents leads to instability and invalid experience replay. To address these challenges, MADDPG utilizes a centralized training and decentralized execution (CTDE) framework, where a central trainer handles the learning process using the DDPG and broadcasts the training parameters to each agent. The central trainer includes the actor network, target actor network, critic network, and target critic network. The single agent only contains an independent actor network, whose parameters come from the central trainer. The single agent inputs the state to its actor network and obtains the action. This separation of training and execution allows more stable and efficient multi-agent learning. Each agent benefits from the shared learning while acting independently during execution.

There are $M$ agents in the UACNs, and for the agent ${ℳ}_i$, the parameters of the actor network and its local policy are denoted by ${\theta }_i$ and ${\pi }_{{\theta }_i}$ respectively. Therefore, the network parameters related to $M$ agents are described by $\theta =\left({\theta }_1,\dots ,{\theta }_M\right)$ and $\pi =\left({\pi }_{{\theta }_1},\dots ,{\pi }_{{\theta }_M}\right)$. The learning processes of multiple agents can be represented by a MDP model, which is defined by the state $\mathcal{S}$, action ${\mathcal{A}}_1,\dots ,{\mathcal{A}}_M$, observation ${\mathcal{O}}_1,\dots ,{\mathcal{O}}_M,$ and state transfer function $\Gamma$. Agent ${ℳ}_i$ uses the deterministic policy ${\pi }_{{\theta }_i}\left(\mathcal{O}\right):{\mathcal{O}}_i\mapsto {\mathcal{A}}_i$ for action selection and moves to the next state according to the state transition function $\Gamma :\mathcal{S}\times {\mathcal{A}}_i\times \cdots \times {\mathcal{A}}_M\mapsto \mathcal{S}$. It then receives the reward $r_i:\mathcal{S}\times {\mathcal{A}}_i\mapsto {ℛ}_i$ and also obtains the observation $o_i:\mathcal{S}\mapsto$ ${\mathcal{O}}_i$.

The central trainer updates the parameters of the critic network by minimizing the loss function [15]:

$$L\left({\theta }_i^{}\right)={\mathbb{E}}_{s,a,r,s^{\prime }\sim \mathcal{D}}\left[{\left(y^{\left(g\right)}-Q_i^{{\pi }^{}}\left(s^{\left(g\right)},a_1^{\left(g\right)},\cdots ,a_M^{\left(g\right)}\right)\right)}^2\right],$$

(17)

where $y^{\left(g\right)}=r_i^{\left(g\right)}+\gamma Q_i^{{\pi }^{\prime }}\left(s^{\prime \left(g\right)},a_1^{\prime \left(g\right)},\cdots ,a_M^{\prime \left(g\right)}\right){\mid }_{a_i^{\left(g\right)}={\pi }_{{\theta }_i}\left(o_i\right)}$ is the Q-value of the target network, and ${\pi }^{\prime }=\left({\pi }_{{\theta }_1}^{\prime },\cdots ,{\pi }_{{\theta }_M}^{\prime }\right)$ is the set of target policies, which is updated by Equation (16).

The actor network of agent ${ℳ}_i$ performs parameter updates by the gradient descent algorithm with the deterministic policy ${\pi }_{{\theta }_i}$. The loss function is [15]:

$${\nabla }_{{\theta }_i}J\left({\theta }_i\right)\approx {\mathbb{E}}_{s^{\left(g\right)},a^{\left(g\right)}~\mathcal{D}}\left[{\nabla }_{a_i}{Q}_{{\pi }_{{\theta }_i}}\left(s^{\left(g\right)},a_1^{\left(g\right)},\dots ,a_M^{\left(g\right)}\right){|}_{a_i^{\left(g\right)}={\pi }_{{\theta }_i}\left(o_i\right)}\ast {\nabla }_{{\theta }_i}{\pi }_{{\theta }_i}\left(o_i\right)\right],$$

(18)

where the replay buffer $\mathcal{D}$ stores samples from $M$ agents at each time slot, which are $\left(s^{\left(t\right)},a_1^{\left(t\right)},\dots ,a_M^{\left(t\right)},r_1^{\left(t\right)},\dots ,r_M^{\left(t\right)},s^{\prime \left(t\right)}\right)$. After the training in the central trainer, the parameters of the ith actor network are broadcasted to each agent. Each agent then uses the received parameters to independently update the actor network.

4. Power Allocation Based on MADDPG

4.1. MDP Model

In this paper, we regard each transmitter node as an agent. Therefore, there are multiple agents in the system. The multi-agents must consider both their observations and other agents’ actions, and their actions also affect surrounding agents’ policies. To obtain a collaborative advantage, the CTDE framework is used in this paper to train the network by the centralized trainer, as is shown in Figure 3. Each agent interacts with the environment and other agents with the information exchange demonstrated by ①/②/③/④. The central trainer includes the actor and critic training networks of all agents and their respective target networks. Training samples come from experiences $\left(s^{\left(t\right)},a_1^{\left(t\right)},\cdots ,a_M^{\left(t\right)},r_1^{\left(t\right)},\cdots ,r_M^{\left(t\right)},s^{{(}^{\prime }t)}\right)$ sent by each agent and stored in replay memory $\mathcal{D}$. During the centralized training, the central trainer selects the samples randomly from $\mathcal{D}$ and updates AC network parameters via DDPG. After central training, the trainer broadcasts new actor parameters to each agent. For the distributed execution, the individual agent executes its action output by its local actor and then receives rewards and moves to the new state. A new sample experience is then obtained by each agent, which is sent back to the central trainer.

Based on the objective function, we define the action, state, and reward functions to formulate a MDP model.

(1) Action Space $\mathcal{A}$: We assume all nodes have the same maximum power constraint. During the time slot, the action $a_i^{\left(t\right)}$ of agent ${ℳ}_i$ is defined as the transmission power allocated by an agent. This results in the action space being defined as

$$\begin{array}{c}A=\left\{a_i^{\left(t\right)}=p_i|0\le p_i\le P_{max}\right\},\\ p_i=P_{min}+\frac{x}{X}\left(P_{max}-P_{min}\right), x=0,\cdots ,X, \end{array}$$

(19)

where $X$ is the number of discretized power levels and $P_{min}=0$ is assumed.

(2) State Space $\mathcal{S}$: The state of the ${ℳ}_i$ at time slot $t$ consists of two parts, i.e., $s_i^t=\left\{o_i^{\left(t\right)},a_i^{\left(t\right)}\right\}$, in which $o_i^{\left(t\right)}=\left\{{\varphi }_i^{\left(t\right)},{\rho }_i^{\left(t\right)}\right\}$ is the current observation. The symbol ${\varphi }_i^{\left(t\right)}=\parallel g_{\left(j,i\right)}^{\left(t-1\right)},g_{\left(j,i\right)}^{\left(t\right)}\parallel$ represents the channel state information, containing the channel gain from the transmitter ${ℳ}_i$ to receiver ${\mathcal{N}}_j$ at time slot $t-1$ and $t$. The ${\rho }_i^{\left(t\right)}$ denotes the system state information. It includes the interference and noise received by ${\mathcal{N}}_j$ at the previous two time slots, and channel capacity $C_{j,i}^{\left(t-1\right)}$ and $C_{l,k}^{\left(t-1\right)}$ for the adjacent link, which is denoted by

$$\begin{array}{c}\begin{array}{c}{\rho }_i^{\left(t\right)}=\left[{\displaystyle \sum }_{k\in M,k\ne i}{g}_{j,k}^{\left(t-2\right)}{p}_k^{\left(t-2\right)}+N\left(f\right),{\displaystyle \sum }_{k\in M,k\ne i}{g}_{j,k}^{\left(t-1\right)}{p}_k^{\left(t-1\right)}+N\left(f\right),C_{j,i}^{\left(t-1\right)},C_{l,k}^{\left(t-1\right)}\right]\end{array}\end{array},$$

(20)

where $N\left(f\right)$ denotes noise power. The historical information of CSI considered in state space makes our proposed algorithm applicable to varying channel environments.

(3) Reward: In the objective function, we require each user to maintain a minimum channel capacity. Therefore, we define the reward function such that channel capacity yields a positive reward, while interference results in a negative reward. We also consider the minimum channel capacity constraint. At time slot $t$, the reward function $r_i^{\left(t\right)}$ is defined as

$$r_i^{\left(t\right)}=C_j^{\left(t\right)}-{\zeta }_1\underset{l\in N,l\ne j}{{\displaystyle \sum }{I}_{l,i}^{\left(t\right)}}-{\zeta }_2\left|C_j^{\left(t\right)}-q_{th}\right|,$$

(21)

where $C_j^{\left(t\right)}$ denotes the channel capacity. The elements $\underset{l\in N,l\ne j}{{{\displaystyle \sum }}^{ }{I}_{l,i}^{\left(t\right)}}$ represents the interference caused by signals from ${ℳ}_i$ to receivers ${\mathcal{N}}_l \left(l\ne j\right)$, calculated as $I_{l,i}^{\left(t\right)}=C_{l\setminus i}^{\left(t\right)}-C_l^{\left(t\right)}$, with the $C_{l\setminus i}^{\left(t\right)}$ being the channel capacity of ${\mathcal{N}}_l$ without interference from ${ℳ}_i$. The coefficient ${\zeta }_1$ adjusts the penalty ratio for interference. The $\left|C_j^{\left(t\right)}-q_{th}\right|$ represents the deviation from the minimum capacity threshold $q_{th}$, weighted by ${\zeta }_2$.

4.2. Power Allocation Algorithm

Based on the MDP model defined in Section 4.1, we present a power allocation algorithm using MADDPG. At time slot $t$, agent ${ℳ}_i$ inputs the state $s_i^{\left(t\right)}$ into its local actor network, and outputs action $a_i^{\left(t\right)}=p_i^{\left(t\right)}$ according to the policy ${\pi }_{{\theta }_i}$. Agent ${ℳ}_i$ then transmits signals to the receiver ${\mathcal{N}}_j$ with power $p_i^{\left(t\right)}$. If the signals are received successfully, the receiver ${\mathcal{N}}_j$ feedbacks the channel gain. Meanwhile, agent ${\mathcal{M}}_j$ sends communication requests to the neighboring agent ${ℳ}_k$. The ${ℳ}_k$ responds to ${ℳ}_i$ with information including of $g_{l,k}^{\left(t\right)}, g_{j,k}^{\left(t\right)}$, $p_k^{\left(t\right)},$ and $N\left(f\right)$. With the receiving information and the stored history information in $t-1 \mathrm{and} t-2$, the ${ℳ}_i$ calculates the current observation $o_i^{\left(t\right)}$ and obtains reward $r_i^{\left(t\right)}$ to form the state $s_i^{\left(t\right)}=\left\{o_i^{\left(t\right)},a_i^{\left(t\right)}\right\}$. Note that this process is carried out in parallel for all agents. Although agents can interact with all neighbors for information exchange, we assume that each agent interacts only with its two nearest neighbors in the simulation due to long propagation delays in underwater channels.

After the communication process completes at the end of time slot $t$, each agent sends experience data $\left(s^{\left(t\right)}, a_1^{\left(t\right)}, \cdots ,a_M^{\left(t\right)}, r_1^{\left(t\right)},\cdots ,r_M^{\left(t\right)}, s^{\prime \left(t\right)}\right)$ to the central trainer and the experience data is stored in $\mathcal{D}$. Once sufficient sample data is collected, the central trainer randomly selects a batch of $G$ samples to update the actor–critic network parameters through gradient descent. The updating process has been described in Equations (17) and (18). The trainer broadcasts the updated actor parameters after completing centralized training. The agents then update their actor networks for action selection in the next time slot. If the environment changes slowly, the actor network parameters by centralized training can broadcast to the agent at intervals of several time slots. For an individual agent, the centralized training reduces computational requirements and saves energy.

We conclude the proposed method in Algorithm 1.

Algorithm 1: MADDPG power allocation

Initialization: Randomly initialize $\mathbf{\theta } \mathrm{and} \mathbf{\mu }; \mathrm{Initialize} \mathcal{D}$, $\mathit{G}$,$\mathrm{target} \mathrm{network} \mathrm{parameter} \mathrm{update} \mathrm{period} {\mathit{T}}_{\mathit{u}}$

1: for each episode do    Initialize the environment and state space $\mathcal{S}$.

2:  for   $\mathit{t}=1 \mathit{to} {\mathit{T}}_{\mathit{m}}$ do

3:     for  $\mathit{i}=1 \mathit{to} \mathit{M}$ do

4:        Input state ${\mathit{s}}_i^{\left(\mathit{t}\right)}$$\mathrm{to} {\mathit{Actor}}_{\mathit{i}}$, and agent ${\mathbf{ℳ}}_i$ outputs action: ${\mathit{a}}_{\mathit{i}}^{\left(\mathit{t}\right)}={\mathit{p}}_{\mathit{i}}^{\left(\mathit{t}\right)}$

5:         ${\mathbf{ℳ}}_{\mathit{i}}$$\mathrm{interacts} \mathrm{with} {\mathbf{ℳ}}_{\mathit{j}}$$\mathrm{to} \mathrm{obtain} {\mathit{g}}_{\mathit{j},\mathit{i}}^{\left(\mathit{t}\right)}$

6:         ${\mathbf{ℳ}}_{\mathit{i}}$$\mathrm{interacts} \mathrm{with} {\mathbf{ℳ}}_{\mathit{k}}$$\mathrm{to} \mathrm{obtain} {\mathit{g}}_{\mathit{l},\mathit{k}}^{\left(\mathit{t}\right)}, {\mathit{g}}_{\mathit{j},\mathit{k}}^{\left(\mathit{t}\right)}, {\mathit{p}}_{\mathit{k}}^{\left(\mathit{t}\right)}$$\mathrm{and} \mathit{N}\left(\mathit{f}\right)$

7:         $\mathrm{Calculates} \mathrm{observations}: {\mathit{o}}_{\mathit{i}}^{\left(\mathit{t}\right)}$

8:         $\mathrm{Receives} \mathrm{reward}: {\mathit{r}}_{\mathit{i}}^{\left(\mathit{t}\right)}$

9:         $\mathrm{Forms} \mathrm{state}: {\mathit{s}}_{\mathit{i}}^{\left(\mathit{t}\right)}=\left\{{\mathit{o}}_{\mathit{i}}^{\left(\mathit{t}\right)}, {\mathit{a}}_{\mathit{i}}^{\left(\mathit{t}\right)},\right\}$

10:        Updates $\mathrm{next} \mathrm{state}: {\mathit{s}}_{\mathit{i}}^{\prime \left(\mathit{t}\right)}={\mathit{s}}_{\mathit{i}}^{\left(\mathit{t}\right)}$

11:        Forms sample data and transmits to $\mathcal{D}:({\mathit{s}}_{\mathit{i}}^{\left(t\right)},{\mathit{a}}_{\mathit{i}}^{\left(\mathit{t}\right)},{\mathit{r}}_{\mathit{i}}^{\left(\mathit{t}\right)},{\mathit{s}}_{\mathit{i}}^{\prime \left(\mathit{t}\right)})$

12:     end for

13:      $\mathrm{Selects} \mathit{G}$$\mathrm{batches} \mathrm{samples} \mathrm{from} \mathcal{D}: \left({\mathit{s}}^{\left(\mathit{g}\right)},{\mathit{a}}^{\left(\mathit{g}\right)},{\mathit{r}}^{\left(\mathit{t}\right)},{\mathit{s}}^{\prime \left(\mathit{t}\right)}\right)$

14:     Calculate  ${\mathit{y}}^{\left(\mathit{g}\right)}={\mathit{r}}_i^{\left(\mathit{g}\right)}+\mathit{\gamma }{\mathit{Q}}_{\mathit{i}}^{{\mathit{\pi }}^{\prime }}\left({\mathit{s}}^{\prime \left(\mathit{g}\right)},{\mathit{a}}_1^{\prime \left(\mathit{g}\right)},\cdots ,{\mathit{a}}_{\mathit{M}}^{\prime \left(\mathit{g}\right)}\right){\mid }_{{\mathit{a}}_{\mathit{i}}^{\left(\mathit{g}\right)}={\mathit{\pi }}_{{\mathit{\theta }}_{\mathit{i}}}\left({\mathit{o}}_{\mathit{i}}\right)}$

15:     Update Critic network by Equation (17)

16:     Update Actor network by Equation (18)

17:     Broadcast  ${\mathit{Actor}}_{\mathit{i}}$ network parameter to agent

18:   end for

19:   if $\mathit{t}=={\mathit{T}}_{\mathit{u}}$ then

20:    Update the target network parameters for critic and actor by Equations (13) and (16)

21:   end if

22: end for

5. Simulation Results

In this section, we evaluate the performance of the proposed MADDPG power allocation through simulations. We assume that $M=10 \mathrm{and} N=10$, and all source nodes are deployed underwater, as shown in Figure 4. The source nodes are located $20 \mathrm{m}$ underwater with $100 \mathrm{m}$ between adjacent nodes. The receivers are on the water surface, with communication distances of $20~500 \mathrm{m}$ from the source nodes. The simulation parameters of the underwater acoustic environment are shown in

Table 1. Simulation parameters.

Parameters	Value
Frequency ($f$)	$20,000 \mathrm{Hz}$
Maximum Doppler frequency ($f_d$)	$12 \mathrm{Hz}$
Shipping activity coefficient ($\mu$)	$0.5$
Maximum transmission power ($P_{max}$)	$1 \mathrm{W}$
Time slot length ($T_s$)	$2.0 \mathrm{s}$
Wind speed ($w$)	$0.1 \mathrm{m}/\mathrm{s}$
Salinity ($H$)	31.8%
pH	$8.17$
Temperature ($T$)	$9.6 ℃$
Speed of sound ($c$)	$1480.9 \mathrm{m}/\mathrm{s}$

, where the wind speed, salinity, pH, temperature, and sound speed data are measured from the Yellow Sea of China in 2015 [23]. We also assume the underwater acoustic channel is slow time varying and quasi-static flat fading, which means $g_{j,i}^{\left(t\right)}$ is constant within a time slot.

According to [24], the underwater sensor nodes are anchored and restricted by a cable, which can float in water. The nodes move at a speed of 0.83–1.67 m/s within the limit of the cable length [24]. We adopt a moving speed of 0.9 m/s in the simulation. Therefore, the maximum Doppler frequency is 12 Hz. To avoid the space–time uncertainty caused by the long propagation delay of underwater acoustic transmission, we assume that the time slot length is long enough to complete information exchange and power allocation in the same time slot. Based on transmission distance and sound velocity, the time slot is assumed to be 2 s.

We compared the proposed MADDPG algorithm with fractional programming (FP) power allocation algorithm [5], DQN training-based power allocation algorithm [19], DDPG algorithm without collaboration [13], random power allocation, and maximum transmitting power (full power).

Figure 5 shows that the proposed MADDPG algorithm obtains a better sum rate compared with other power allocation strategies. The sum rate here refers to the sum channel capacity of all links. The sum rate of the proposed MADDPG power allocation remains above $1.7 \mathrm{bps}/\mathrm{Hz}$, in which the $\mathrm{bps}$ means bits per second. The FP algorithm is a model-driven method and has full CSI, whereas the deep learning methods such as DQN, DDPG, and MADDPG are only data driven without full CSI, resulting in lower performance than the FP algorithm. In the DQN and DDPG algorithms, each single agent is trained independently without interacting with the surrounding environment. However, the agents in MADDPG interact with each other and can use global data for centralized training, which obtains a better performance than DQN and DDPG. Random power and full power do not optimize power allocation, thus resulting in the worst performance.

Figure 6 compares the spectral efficiency (SE) performance of different algorithms in single-episode training. It can be seen that the SE of MADDPG is close to FP and outperforms other algorithms. Moreover, the MADDPG power allocation obtains convergence within 5000 training time steps, which has the same convergence rate as DQN.

In the objective function ${\mathcal{P}}_1$, the threshold $q_{th}$ is required to ensure the minimum channel capacity of each link. Figure 7 compares the sum rate of MADDPG power allocation with ($q_{th}=1 \mathrm{bps}/\mathrm{Hz}$) and without ($q_{th}=0 \mathrm{bps}/\mathrm{Hz}$) minimum channel capacity constraint. The algorithm considering $q_{th}=1 \mathrm{bps}/\mathrm{Hz}$ maintains a channel capacity of approximately 1.75 bps/Hz which performs 75% higher than that of without considering minimum channel capacity. Therefore, considering the minimum channel capacity constraint and penalty for interference in the reward function of our algorithm, each link can ensure a minimum channel capacity, which improves the system sum rate.

Figure 8 compares the computational complexity of different algorithms as the number of network nodes increases. We use the program running time to complete one-time power allocation as a metric. From Figure 8, we can see that the number of nodes affects the complexity of the algorithm. The algorithms of MADDPG, DQN, and DDPG have approximately the same complexity, while FP is much higher. The per-iteration complexity of FP is $O(M^2$), while the others are $O\left(M\right)$ or less. Note that random and maximum power allocation algorithms are excluded from Figure 8 since they do not need additional calculations.

6. Conclusions

This paper proposes a power allocation algorithm based on reinforcement learning, which optimizes the channel capacity of UACNs. The action, state, observation, and reward functions of a MDP model are designed to solve the objective function. To take advantage of collaborative training, the MADDPG structure is applied to this problem, which is implemented by centralized training and distributed execution. An actor–critic network of all agents is trained by a centralized trainer, while the independent actor network of each agent is used to execute actions. The minimum channel capacity constraint ensures the QoS requirement of each link. Simulation results demonstrate that the proposed algorithm outperforms the DQN- and DDPG-based power allocation algorithms of both sum rate and spectral efficiency. Furthermore, as the number of network nodes increases, the proposed method has a much lower running time compared with the FP algorithm.