Adaptive Data Collection and Offloading in Multi-UAV-Assisted Maritime IoT Systems: A Deep Reinforcement Learning Approach

Abstract

This paper studies the integration of data collection and offloading for maritime Internet of Things (IoT) systems with multiple unmanned aerial vehicles (UAVs). In the considered multi-UAV maritime IoT system, the UAVs act as the aerial base stations to complete the missions of data collection from buoys and data offloading to the offshore base station (OBS). In this case, the UAVs need to adaptively select the mission mode between data collection and data offloading according to the network resources and mission requirements. In this paper, we aimed to minimize the completion time of data collection and offloading missions for all UAVs by jointly optimizing the UAV trajectories, mission mode selection, transmit power of buoys, and association relationships between the UAVs and buoy/OBS. In order to solve the mixed-integer non-convex minimization problem, we first designed a multi-agent deep reinforcement learning algorithm based on a hybrid discrete and continuous action space to preliminarily obtain the UAV trajectories, mission mode selection, and the transmit power of buoys. Then, we propose an algorithm based on the stable marriage problem to determine the buoy–UAV and UAV–OBS association relationships. Finally, the simulation results show that the proposed algorithms can effectively shorten the total mission completion time of data collection and offloading for the multi-UAV-assisted maritime IoT system.

1. Introduction

The development of the maritime Internet of Things (IoT) facilitates various maritime activities of humans more frequently and greatly promotes the development of the maritime economy. The surging demand for wireless communication puts great pressure on maritime wireless communication systems for the next generation of wireless communication technologies (such as beyond Fifth-Generation (5G) and Six-Generation (6G)) [1,2]. However, maritime wireless communication technology is relatively behind compared with that for land [3]. For example, the maritime wireless communication system lacks the construction conditions of communication facilities (such as base stations), and the maritime spectrum resources are scarce [4]. Besides, complex sea surface changes (such as wave motion and evaporation duct) make the maritime wireless communication environment unstable [5]. Hence, it is of great significance to research the maritime IoT to realize flexible, efficient, and reliable transmission.

In the maritime IoT system, buoys embedded with a variety of sensors are usually deployed on the sea surface to monitor the maritime environment. However, these buoys are generally out of the service coverage of the offshore base stations (OBSs). Although monitoring data can be transmitted through the multi-hop network formed by buoys and ships, the connectivity of the network will be seriously damaged when nodes in the network fail [6]. Satellite communication could mitigate the issue of network failure, but it has fatal drawbacks in terms of high delay, low-link reliability, and high cost [7]. The unmanned-aerial-vehicle (UAV)-assisted maritime communication system is regarded as a promising solution to deal with the high delay and low reliability. The UAV as the aerial mobile base station has the following advantages in assisting maritime wireless communication. On the one hand, the high maneuverability of the UAV makes it easier to establish line-of-sight (LoS) communication link between UAVs and buoys and also allows them to flexibly approach buoys to enhance the communication links [8]. Hence, the UAV can better adapt to the complex maritime wireless communication environment and is very suitable for emergency communication scenarios. On the other hand, the UAV usually has the advantages of a small size, low cost, and easy control [9]. The application of UAVs can reduce the risk of manual operation in some dangerous maritime environments. The above advantages also make UAVs have high concealment and be widely used in maritime military activities.

However, the size of the batteries embedded in UAVs and buoys is limited. It will affect the efficiency of UAV’s mission execution, if UAVs frequently return to the charging area. Moreover, frequent battery replacement will increase the maintenance cost of buoys, and in fact, it difficult to achieve due to the bad condition of the sea environment. Hence, the performance of UAV-assisted maritime wireless communication is severely limited by the energy consumption of UAVs and buoys [10]. The energy consumption disadvantage could be compensated by multi-UAV cooperative data transmission. The multi-UAV communication system has a wider communication range and a shorter transmission distance, which can reduce the communication delay and improve the energy efficiency [11]. In the multi-UAV communication system, UAVs need to collect the data cached in buoys and offload the data to the OBS. In this process, the UAV will be far away from the OBS, which may make it take a long time to offload the data. Hence, it is important to design the trajectories of multiple UAVs to ensure the data transmission requirement under the condition of limited energy and shorten the mission completion time of data collection and data offloading.

Recently, there has been extensive research work focused on the multi-UAV-assisted wireless communication scenarios. Diao et al. [12] solved the multi-UAV deployment problem by a circular deployment scheme and jointly optimized the offloading and scheduling strategies to minimize the weighted sum of transmission and hover energy consumption. Kuo et al. [13] studied multi-UAV-assisted data collection scenarios with circular trajectories and considered the deployment of UAVs and device association to minimize the total energy consumption of IoT devices. Gao et al. [14] jointly optimized UAV trajectories, ground equipment scheduling, and power allocation by using the technologies of block coordinate descent and successive convex approximation to maximize the overall throughput of ground devices. However, the above work only considered the uplink or downlink transmission scenarios. Hua et al. [15] studied the simultaneous uplink and downlink transmission systems of multiple UAVs, in which one UAV is responsible for downlink transmission and the other UAV is responsible for uplink transmission. Furthermore, Liu et al. [16] studied that each UAV can dynamically change the transmission mode in each time slot, but the premise of this work is to know the UAVs’ trajectories. It is significant that UAVs can change the mission mode according to the transmission requirements. For example, a UAV adopts the mode of offloading after completing the collection mission. When the UAV fails during the collection mission, all the data collected by the UAV cannot be returned to the OBS, resulting in data loss. The high coupling of UAV trajectories, association relationships, and mission mode is a challenge for us, which can usually be described as a mixed-integer non-convex problem. The above works effectively solved this kind of problem through traditional convex optimization techniques and iterative methods, but they have high computational complexity and the curse of dimensionalitywith the increase of the action and state dimension of the system due to the high dynamics of UAVs.

With the development of machine learning technology, deep reinforcement learning (DRL) algorithms have received extensive attention with the advantage of dealing with highly dynamic environments [17,18,19]. In reinforcement learning, the agent maximizes the cumulative reward by constantly exploring the environment. DRL combines the advantages of the deep neural network (DNN) and can handle more complex actions and states. At present, much research has applied DRL algorithms to multi-UAV trajectory optimization. References [20,21,22] studied the multi-UAV trajectory optimization scheme based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm and regarded the UAVs as the agents. Wu et al. [20] aimed to minimize the average age-of-information (AoI) and proposed an MADDPG algorithm to jointly optimize the UAV sensing location and transmission location. However, the MADDPG algorithm can only handle a continuous action space. Hence, Wang et al. [21] combined a low-complexity mission decision-making method with the MADDPG algorithm. Gao et al. [22] used a potential game approach to determine the service allocation between users and multiple UAVs, which effectively solved the mixed-integer non-linear problem. For using DRL algorithms to process the hybrid discrete and continuous action space, it undoubtedly reduces the flight accuracy of UAVs if a continuous action is discretized, and this is inconsistent with the actual situation of UAV flight actions; relaxing the discrete action space into a continuous action space increases the complexity of the action space, such as in Hausknecht et al. [23]. Xiong et al. [24] proposed the parameterized deep Q-network learning (PDQN) algorithm based on the parameterized action space. Yin et al. [25] proposed a multi-agent parametrized deep Q-network (MAPDQN) algorithm to maximize both the overall throughput and the fairness throughput. However, the action decision-making method of the PDQN algorithm may reduce the maximum Q-value [26]. Then, Fan et al. [26] proposed a hybrid proximal policy optimization (HPPO) algorithm, which can efficiently solve the hybrid action space based on the proximal policy optimization (PPO) algorithm. We summarize the difference between our work and the existing related references in

Table 1. Comparison between our work and the related references.

Reference	UAV Trajectory	UAV Hovering Location	Uplink	Downlink	Types of Action Variables	Deep Reinforcement Learning
[12]		✓	✓	✓
[13]	✓		✓
[14]	✓		✓
[15]	✓		✓	✓
[16]			✓	✓
[20]	✓			✓	continuous	MADDPG
[21]	✓			✓	continuous	MADDPG
[22]	✓		✓	✓	continuous	MADDPG
[25]	✓			✓	discrete and continuous	MAPDQN
our work	✓		✓	✓	discrete and continuous	MAHPPO

.

Based on the above discussion, this paper studies the multi-UAV-assisted maritime IoT systems, in which the missions of the UAVs consist of collecting data from buoys and offloading the data to the OBS. The UAVs could adaptively change the mission mode according to the network resources and mission requirements. Our goal was to minimize the total mission completion time by jointly optimizing the UAV trajectories, mission mode selection, the transmit power of the buoys, and the buoy–UAV and UAV–OBS association relationships with the constraints of energy consumption and a no-fly zone. The main contributions of our paper are summarized as follows:

• We propose an adaptive data transmission mission mode framework for the multi-UAV-assisted maritime IoT system. Specifically, UAVs are allowed to change the mission mode in each time slot according to the channel quality and data volume, which improves the flexibility of offshore data collection and offloading.
• A multi-UAV trajectory optimization algorithm based on multi-agent HPPO (MAHPPO) is proposed to overcome the problem of a hybrid discrete and continuous action space. The total mission completion time is effectively shortened through the flexible scheduling of the UAV trajectories, mission mode, and transmit power of buoys.
• In order to reduce the exploration difficulty and action space dimension of the MAHPPO algorithm, we propose an algorithm based on the stable marriage problem (SMP) to determine the buoy–UAV and UAV–OBS association relationships. The simulation results show that the proposed algorithm can effectively reduce the total mission completion time with low computational complexity.

The remainder of the paper is organized as follows. The multi-UAV data collection and offloading model and the problem formulation are given in Section 2. Then, the problem analysis and the proposed algorithm are given in Section 3. The simulation results are given in Section 4. Finally, we give a conclusion in Section 5.

2. System Model and Problem Formulation

As shown in Figure 1, we considered a UAV-assisted maritime IoT system where U UAVs with the same fixed flight height H were used as the aerial base stations to execute the data collection–offloading mission in the target area. The mission of the UAVs is to collect the hydrometeorological data sensed by M buoys with a random distribution in the target area and offload all the collected data to the OBS. The sets of UAVs and buoys are denoted as $u\in \mathcal{U}=\left\{1,2,\dots ,U\right\}$ and $m\in \mathcal{M}=\left\{1,2,\dots ,M\right\}$, respectively. The total mission completion time of the UAVs is divided into multiple equal time slots and is denoted as $T_{\mathrm{total}}=K\delta$, where K is the number of time slots, $k\in \mathcal{K}=\left\{\infty ,\in ,\dots ,\mathcal{K}\right\}$ is the set of time slots, and $\delta$ is the duration of each time slot. Note that, at each time slot, the UAVs need to choose the appropriate operation mode between data collection and data offloading according to the current network states. In each time slot, the UAV u cannot collect and offload data at the same time.

The horizontal coordinate of UAV u at time slot k is denoted as $q_{u,k}=\left\{x_{u,k},y_{u,k}\right\},\forall u\in \mathcal{U},k\in \mathcal{K}$. The flight direction of UAV u is limited by an angle of ${\psi }_{u,k}\in \left.\left[0,2\pi \right.\right)$ and a velocity of $V_{u,k}\in \left[0,V_{max}\right]$, where $V_{max}$ is the maximum flight velocity of the UAV. The coordinates of UAV u can be expressed as $x_{u,k}=x_{u,0}+{\sum }_{k^{*}=1}^k{V}_{u,k^{*}}\delta cos\left({\psi }_{u,k^{*}}\right)$ and $y_{u,k}=y_{u,0}+{\sum }_{k^{*}=1}^k{V}_{u,k^{*}}\delta sin\left({\psi }_{u,k^{*}}\right)$, respectively. Considering the limitation of the flight range, $x_{u,k}$ and $y_{u,k}$ are respectively constrained by

$$0\le x_{u,k}\le x_{max},\forall u\in \mathcal{U},k\in \mathcal{K}$$

(1)

$$0\le y_{u,k}\le y_{max},\forall u\in \mathcal{U},k\in \mathcal{K}$$

(2)

where $x_{max}$ and $y_{max}$ are the length and width of the target area, respectively. Moreover, the distance between UAV u and UAV $u^{*}$ at == time slot k can be expressed as $d_{u,u^{*},k}=\sqrt{{\left(x_{u,k}-x_{u^{*},k}\right)}^2+{\left(y_{u,k}-y_{u^{*},k}\right)}^2}$. In order to avoid a collision, it is necessary to ensure the minimum safety distance $d^{min}$ between UAVs, which is expressed as

$$d_{u,u^{*},k}\ge d^{min}.\forall u,u^{*}\in \mathcal{U},u\ne u^{*},k\in \mathcal{K}$$

(3)

Denote the set $i\in \mathcal{I}=\left\{0,1,\dots ,M\right\}$ as the set of OBS and buoys, where $i=0$ means the OBS. The horizontal coordinate of buoy m and the OBS is $(x_i,y_i)$. Hence, the distance of the buoy–UAV and UAV–OBS at time slot k can be given by $d_{u,i,k}=\sqrt{{\left(x_{u,k}-x_i\right)}^2+{\left(y_{u,k}-y_i\right)}^2+H^2}$.

2.1. Channel Model

Due to the special maritime propagation, we adopted the model of the air-to-ground channel and the two-ray path loss model [27]. More specifically, the probability of the LoS link is expressed as $P_{u,i,k}^{\mathrm{LoS}}=\frac{1}{1+aexp(-b({\psi }_{u,i,k}-a))}$, where a and b are two constant values depending on the environment. ${\psi }_{u,i,k}$ denotes the elevation angle of the buoy–UAV and UAV–OBS, which is given by ${\psi }_{u,i,k}=\frac{180^{{}^{\circ }}}{\pi }\times arcsin\left(\frac{H}{\sqrt{{\left(x_{u,k}-x_i\right)}^2+{\left(y_{u,k}-y_i\right)}^2}}\right)$.

The average path loss of the buoy–UAV and UAV–OBS links is figured out according to the probability of establishing the LoS, which can be expressed as

$$L_{u,i,k}=P_{u,i,k}^{\mathrm{LoS}}{L}_{u,i,k}^{\mathrm{LoS}}+\left(1-P_{u,i,k}^{\mathrm{LoS}}\right)L_{u,i,k}^{\mathrm{NLoS}},\forall u\in \mathcal{U},i\in \mathcal{I},k\in \mathcal{K}$$

(4)

where $L_{u,i,k}^{\mathrm{LoS}}$ and $L_{u,i,k}^{\mathrm{NLoS}}$ are the average path loss of the LoS and NLoS, respectively, which can be given by

$$L_{u,i,k}^{\mathrm{LoS}}={\left(\frac{4\pi d_{u,i,k}}{\lambda }\right)}^{{\upsilon }^{\mathrm{LoS}}}{\mu }_{u,i,k}{\xi }_{\mathrm{LoS}},$$

(5)

$$L_{u,i,k}^{\mathrm{NLoS}}={\left(\frac{4\pi d_{u,i,k}}{\lambda }\right)}^{{\upsilon }^{\mathrm{NLoS}}}{\mu }_{u,i,k}{\xi }_{\mathrm{NLoS}},$$

(6)

where ${\mu }_{i,k}=1$, $\lambda$ is the wavelength, ${\xi }_{\mathrm{LoS}}$ and ${\xi }_{\mathrm{NLoS}}$ are the excessive path loss for the LoS and NLoS paths, respectively, and ${\upsilon }^{\mathrm{LoS}}$ and ${\upsilon }^{\mathrm{NLoS}}$ are the LoS and NLoS link factors, respectively.

According to the above discussions, the channel gain of the buoy–UAV and UAV–OBS links can be expressed as

$$h_{u,i,k}=1/L_{u,i,k}.\forall u\in \mathcal{U},i\in \mathcal{I},k\in \mathcal{K}$$

(7)

The signal-to-noise-ratio (SNR) of the buoy–UAV and UAV–OBS at time slot k is given by

$$g_{u,i,k}=\frac{P_{u,i,k}{h}_{u,i,k}}{{\sigma }^2},\forall u\in \mathcal{U},i\in \mathcal{I},k\in \mathcal{K}$$

(8)

where ${\sigma }^2$ is the Gaussian noise power and $P_{u,m,k},\forall m\in \mathcal{M}$ and $P_{u,0,k}$ are the transmit power of buoy m and UAV u at time slot k, respectively. Moreover, $0\le P_{u,m,k}\le P_{max}$, and $P_{u,0,k}$ is a constant.

2.2. Transmission Model

Let ${\alpha }_{u,i,k},\forall u\in \mathcal{U},i\in \mathcal{I},k\in \mathcal{K}$ denote the association indicator of the buoy–UAV and UAV–OBS. ${\alpha }_{u,m,k}=1,\forall m\in \mathcal{M}$ if the buoy m is associated with UAV u at time slot k. Otherwise, ${\alpha }_{u,m,k}=0,\forall m\in \mathcal{M}$. Similarly, ${\alpha }_{u,0,k}=1$ if UAV u is associated with the OBS at time slot k; otherwise, ${\alpha }_{u,0,k}=0$. At time slot k, each UAV can only collect the data from one buoy, and each buoy can only be associated with one UAV at most.

$${\alpha }_{u,i,k}\in \left\{0,1\right\},\forall u\in \mathcal{U},i\in \mathcal{I},k\in \mathcal{K}$$

(9)

$${\sum }_{i=0}^M{\alpha }_{u,i,k}\le 1,\forall u\in \mathcal{U},k\in \mathcal{K}$$

(10)

$${\sum }_{u=1}^U{\alpha }_{u,m,k}\le 1.\forall m\in \mathcal{M},k\in \mathcal{K}$$

(11)

The system total spectrum bandwidth is B and allocated to each buoy–UAV link and UAV–OBS link equally. Each buoy–UAV link and UAV–OBS link uses mutually independent and non-overlapping spectrum resources. Then, the spectrum bandwidth of each buoy–UAV link and UAV–OBS link at time slot k can be given by

$$B_k=\frac{B}{{U_k}^{*}},$$

(12)

where ${U_k}^{*}\le U$ is the number of UAVs successfully associated with the buoys and OBS at time slot k.

The transmission rate of the buoy–UAV and UAV–OBS at time slot k is expressed as

$$R_{u,i,k}=B_k{\alpha }_{u,i,k}{log}_2\left(1+g_{u,i,k}\right).\forall u\in \mathcal{U},i\in \mathcal{I},k\in \mathcal{K}$$

(13)

In order to ensure that the data collection requirements of the buoys are satisfied, the following condition is considered:

$$\sum _{u=1}^U\sum _{k=1}^K{R}_{u,m,k}\ge C_m,\forall m\in \mathcal{M}$$

(14)

where $C_m$ denotes the data volume that needs to be collected from buoy m. It is necessary to guarantee that all data collected by the UAVs are offload to the OBS. Moreover, the UAV that collects enough data is allowed to offload the data at time slot k, even though the data in the buoys are not completely collected. The total data volume collected by UAV u before time slot k is expressed as $C_{u,k-1}^{\mathrm{co}}={\sum }_{k^{*}=1}^{k-1}{\sum }_{m=1}^M{R}_{u,m,k^{*}}\delta$. The total data volume offload from UAV u to the OBS before time slot k is expressed as $C_{u,k-1}^{\mathrm{of}}={\sum }_{k^{*}=1}^{k-1}{R}_{u,0,k^{*}}\delta$, where $k^{\prime }$ denotes the time slot in which the data in all buoys are collected completely, and $k^{\prime }<K$. Hence, each UAV should satisfy the following constraints:

$$C_{u,k-1}^{\mathrm{co}}-C_{u,k-1}^{\mathrm{of}}\ge R_{u,0,k}\delta ,\forall u\in \mathcal{U},k\in \left\{1,2,\dots ,k^{\prime }\right\}$$

(15)

$${\sum }_{u=1}^U{\sum }_{k=1}^K{R}_{u,0,k}\delta \ge {\sum }_{m=1}^M{C}_m.$$

(16)

In particular, even if UAV u does not have enough data, it can only offload the data to the OBS rather than at time slot $k^{\prime }$.

2.3. Energy Consumption Model

The energy consumption of the UAV mainly includes propulsion energy and communication energy [28]. The propulsion energy is to keep the UAV hovering and support the flight movement. The communication energy consumption is mainly caused by data transmission, signal processing, the communication circuit, etc. The order of magnitude of the propulsion energy is much larger than that of the communication energy. Hence, in this paper, we mainly considered the propulsion energy consumption and set the transmit power of the UAVs $P_{u,0,k}$ to be a constant. Then, the communication energy caused by the data transmission from the UAVs to the OBS is $E_u^{\mathrm{cm}}={\sum }_{k=1}^K{\alpha }_{u,0,k}{P}_{u,0,k}\delta$.

The propulsion power of UAV u mainly depends on the flight velocity, given by

$$P_u\left(V_{u,k}\right)=P_{\mathrm{o}}\left(1+\frac{3V_{u,k}^2}{U_{\mathrm{tip}}^2}\right)+P_i{\left(\sqrt{1+\frac{V_{u,k}^4}{4v_{\mathrm{o}}^4}}-\frac{V_{u,k}^2}{2v_{\mathrm{o}}^2}\right)}^{1/2}+\frac{1}{2}{d}_{\mathrm{o}}\rho lf{V}_{u,k}^3,$$

(17)

where $U_{\mathrm{tip}}$ denotes the tip speed of the rotor blade, $v_{\mathrm{o}}$ is the mean rotor-induced velocity, $d_{\mathrm{o}}$ is the fuselage drag ratio, $P_{\mathrm{o}}$ and $P_i$ are the blade profile power and induced power, respectively, and $\rho$, l, and f denote the air density, rotor solidity, and blade angular velocity, respectively. The propulsion energy consumption of UAV u can be expressed as

$$E_u^{\mathrm{pr}}={\sum }_{k=1}^{K}P\left(V_{u,k}\right)\delta .$$

(18)

Then, the total energy consumption of UAV u is given by $E_u^{\mathrm{total}}=E_u^{\mathrm{cm}}+E_u^{\mathrm{pr}}$, where $E_u^{\mathrm{total}}\le {\overline{E}}_u$ and ${\overline{E}}_u$ is the energy threshold of UAV u.

The energy consumption of the buoys is mainly generated by data transmission, expressed as $E_m={\sum }_{u=1}^U{\sum }_{k=1}^K{\alpha }_{u,m,k}{P}_{u,m,k}\delta$, where $E_m\le {\overline{E}}_m$, and ${\overline{E}}_m$ is the energy threshold of buoy m.

2.4. Problem Formulation

Let $\mathit{A}=\{{\alpha }_{u,i,k},\forall u,i,k\}$ denote the associated variables, $\mathit{P}=\{P_{u,m,k},\forall u,m,k\}$ denote the transmit power of the buoys, $\mathit{V}=\{V_{u,k},\forall u,k\}$ denote the flight velocity of the UAVs, and $\mathit{\psi }=\{{\psi }_{u,k},\forall u,k\}$ denote the flight angle of the UAVs. Our goal is to minimize the completion time of the data collection–offloading missions for all UAVs by the joint optimization of $\mathit{A}$, $\mathit{P}$, $\mathit{V}$, and $\mathit{\psi }$. Then, the total mission completion time minimization problem can be formulated as

$$\begin{array}{cc}\hfill \left(\mathrm{P}1\right):\underset{\mathit{A},\mathit{P},\mathit{V},\mathit{\psi }}{min}& T_{\mathrm{total}}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathrm{C}1:0\le V_{u,k}\le V_{max},\forall u\in \mathcal{U},k\in \mathcal{K}\hfill \\ & \mathrm{C}2:0\le {\psi }_{u,k}<2\pi ,\forall u\in \mathcal{U},k\in \mathcal{K}\hfill \\ & \mathrm{C}3:g_{u,m,k}\ge {\overline{g}}_{\mathrm{co}},\forall u\in \mathcal{U},m\in \mathcal{M},k\in \mathcal{K}\hfill \\ & \mathrm{C}4:g_{u,0,k}\ge {\overline{g}}_{\mathrm{of}},\forall u\in \mathcal{U},k\in \mathcal{K}\hfill \\ & \mathrm{C}5:0\le P_{u,m,k}\le P_{max},\hfill \\ & \forall u\in \mathcal{U},m\in \mathcal{M},k\in \mathcal{K}\hfill \\ & \mathrm{C}6:E_u^{\mathrm{total}}\le {\overline{E}}_u,\forall u\in \mathcal{U}\hfill \\ & \mathrm{C}7:E_m\le {\overline{E}}_m,\forall m\in \mathcal{M}\hfill \\ & \mathrm{C}8-\mathrm{C}16:\left(1\right)-\left(3\right),\left(9\right)-\left(11\right),\left(14\right)-\left(16\right).\hfill \end{array}$$

In problem P1, C1 and C2 are the UAV maximum velocity and maximum angle constraints, respectively. C3 and C4 are the SNR constraints, where ${\overline{g}}_{\mathrm{co}}$ and ${\overline{g}}_{\mathrm{of}}$ are the threshold of the SNR of the buoy–UAV and UAV–OBS, respectively. C5 restricts the maximum transmit power of the buoys. C6 and C7 mean that the energy consumption of the UAVs and buoys should be smaller than their maximum energy. C8-C10 ensure that the UAVs keep apart from each other to avoid collision and do not fly beyond the target area. C11–C13 are the association constraints. C14–C16 are the data transmission constraints of the process of UAV data collection and offloading.

3. Joint Optimization Algorithm Design

In this section, due to the binary constraint (C11) and the non-convex constraints (C6, C7, C14–C16), it is difficult to solve the formulated mixed-integer non-convex problem P1 effectively and directly. Hence, we first propose a DRL-based algorithm to preliminarily determine the UAV trajectories, mission mode, and transmit power of the buoys. Then, we designed an SMP-based association algorithm (SAA) and an UAV–OBS association algorithm (UAA) to solve the buoy–UAV and UAV–OBS association subproblem, respectively.

3.1. MAHPPO Algorithm

As we all known, RL is based on the continuous interaction between the agent and the environment to select the best action for the observed state so as to maximize the cumulative reward [18]. Each UAV in the multi-UAV system is controlled by a specialized agent, and all agents are deployed in the OBS. Each agent receives the status information obtained by the UAVs from the environment and then selects an action to obtain an immediate reward. Specifically, the agents send all the action information to the UAVs, and then, the UAVs forward the action information to the buoys. The action space $\mathcal{A}$, state space $\mathcal{S}$, and reward function $\mathcal{R}$ are designed as follows.

3.1.1. Action Space

The action space is denoted as $\mathcal{A}=\left\{A_{u,k},\forall u\in \mathcal{U}\right\}$. $A_{u,k}=\left\{{\chi }_{u,k},{\psi }_{u,k},V_{u,k},P_{u,k}\right\}$ represents the action space of each agent, where ${\chi }_{u,k}=\{0,1\}$ is denoted as the mission mode selection of UAV u. If ${\chi }_{u,k}=0$, the agent chooses to make UAV u collect data at time slot k, meaning that ${\sum }_{m=1}^M{\alpha }_{u,m,k}=1$. Similarly, if ${\chi }_{u,k}=1$, UAV u offloads the data at time slot k, meaning that ${\alpha }_{u,0,k}=1$. $P_{u,k}$ represents the transmit power of the buoy corresponding to UAV u at time slot k. Note that the value of $P_{u,k}$ could be obtained by the agent action selection, but we do not know which buoy is associated to UAV u at time slot k.

3.1.2. State Space

The state space is denoted as the global state ${\mathcal{S}}_k=\left\{{\kappa }_k,c_k^{\mathrm{buoy}},E_k^{\mathrm{buoy}},x_k,y_k,c_k^{\mathrm{uav}},E_k^{\mathrm{uav}},{\chi }_k,{\chi }_k^{\mathrm{new}},B_k\right\}$, where the meanings of each variable are as follows:

• ${\kappa }_k=\left\{{\kappa }_{u,k},\forall u\in \mathcal{U}\right\}$ is defined as the associated state of the buoy–UAV and UAV–OBS, where ${\kappa }_{u,k}=i,\forall i\in \mathcal{I}$ represents the number of buoys or OBS associated with UAV u. If UAV u is not associated with any buoy or OBS, it yields that ${\kappa }_{u,k}=-1$.
• $c_k^{\mathrm{buoy}}=\left\{c_{m,k}^{\mathrm{buoy}},\forall m\in \mathcal{M}\right\}$ and $c_k^{\mathrm{uav}}=\left\{c_{u,k}^{\mathrm{uav}},\forall u\in \mathcal{U}\right\}$ represent the set of remaining data volume for the buoys and UAVs, respectively.
• $E_k^{\mathrm{buoy}}=\left\{E_{m,k}^{\mathrm{buoy}},\forall m\in \mathcal{M}\right\}$ and $E_k^{\mathrm{uav}}=\left\{E_{u,k}^{\mathrm{uav}},\forall u\in \mathcal{U}\right\}$ represent the set of remaining energy for the buoys and UAVs, respectively.
• $x_k=\left\{x_{u,k},\forall u\in \mathcal{U}\right\}$ and $y_k=\left\{y_{u,k},\forall u\in \mathcal{U}\right\}$ represent the set of horizontal coordinates of the UAVs.
• In MAHPPO, the agent action selection is not limited by the constraints, such as the SNR threshold, energy consumption, and so on. Therefore, it is necessary to make a series of constraint judgments on the action choices of the agents. If the selected action of the agent violates the constraint, the action will be changed to meet the constraint conditions. Therefore, we define ${\chi }_k^{\mathrm{new}}=\left\{{\chi }_{u,k}^{\mathrm{new}},\forall u\in \mathcal{U}\right\}$ as the changed mission mode selection of the UAVs and ${\chi }_{u,k}^{\mathrm{new}}=\left\{{\chi }_{u,k},2\right\}$, where ${\chi }_{u,k}^{\mathrm{new}}=2$ means that UAV u neither collects nor offloads data.
• $B_k=\left\{B_{u,k},\forall u\in \mathcal{U}\right\}$, where $B_{u,k}$ indicates the bandwidth occupied by UAV u at time slot k.

3.1.3. Reward Function

The following cases are possible during the multi-UAV mission performance process:

(1)

If the mission is not completed, the UAV with sufficient energy will continue to collect or offload data.

(2)

Although the mission is not completed, the UAV does not have sufficient energy. Hence, the UAV is forced to stop performing the mission at this time.

(3)

The mission is successfully completed. Note that the UAVs must be offloading data in the last time slot.

Therefore, we designed the reward function as

$$\begin{array}{c}\hfill r_k=\left\{\begin{array}{cc}{\sum }_{u=1}^U\left({\alpha }_{u,0,k}{R}_{u,0,k}\delta +{\rho }_{u,k}\right)+K^{*},\hfill & \mathrm{Case}3\\ {\sum }_{u=1}^U\left({\sum }_{i=0}^I{\alpha }_{u,i,k}{R}_{u,i,k}\delta +{\rho }_{u,k}\right)+E_{\mathrm{pe}},\hfill & \mathrm{Case}1\&2\end{array}\right.\end{array}$$

(19)

where $E_{\mathrm{pe}}$ is defined as the energy penalty. ${\rho }_{u,k}$ is the position penalty when UAV u violates constraints C8-C10. Furthermore, the UAV mission completion time is upper-bounded by $K^{\max }$. If the UAVs complete the mission ahead of time, let $K^{*}=K^{max}-K$ denote the time reward.

According to the reward function (18), P1 can be rewritten as

$$\begin{array}{cc}\hfill \left(\mathrm{P}2\right):\underset{\mathit{A},\mathit{P},\mathit{V},\mathit{\psi }}{max}& {\sum }_{k=1}^K{\sum }_{i=0}^M{R}_{u,i,k}\hfill \\ & \mathrm{s}.\mathrm{t}.\mathrm{C}1-\mathrm{C}16.\hfill \end{array}$$

(20)

Since the action space contains two kinds of actions: a discrete variable ${\chi }_{u,k}$ and three continuous variables ${\psi }_{u,k},V_{u,k}$, and $P_{u,k}$, the MAHPPO algorithm [26,29] that can solve the hybrid discrete and continuous action space is employed. As shown in Figure 2, agents send state information ${\mathcal{S}}_k$ to actor networks, and the number of actor networks is U. These actor networks provide action policies ${\pi }_{{\theta }_u}\left(A_{\mathrm{d},{\mathrm{k}}_{\mathrm{u}}}\left|S_k\right.\right)$ and ${\pi }_{{\theta }_u}\left(A_{\mathrm{c},{\mathrm{k}}_{\mathrm{u}}}\left|S_k\right.\right)$ for the corresponding agents, where subscripts d and c represent the discrete and continuous parts of the actions, respectively. ${\theta }_u$ is the parameter of the actor network for agent u. For the convenience of expression, let ${\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)$ denote the discrete or continuous strategies. In an actor network, the output branches of ${\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)$ share multiple neural network layers. Since the MAHPPO algorithm adopts a random strategy, the discrete part of the actor network outputs the probability value of each discrete action. Moreover, the discrete action is sampled randomly based on the Softmax distribution. The continuous part of the actor network follows the parameterized Gaussian distribution and outputs the mean and variance of the continuous actions. The continuous part adopts the rule function in the last layer of the neural network.

RL is a process in which an agent constantly interacts with the environment to find the optimal policy to maximize the cumulative discount reward $R_k={\sum }_{\tau =0}^{\infty }{{\gamma }^{\tau }r}_{k+1+\tau }$, where $\gamma$ is a discount factor. The critic network outputs the state value function $V_{\varphi }\left(S_t\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[R_k\left|S=S_k\right.\right]$, where the expectation of $R_k$ is obtained with the policy ${\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)$. The critic network is updated by minimizing the loss function, shown as

$$L\left(\varphi \right)=\mathbb{E}\left[{\left(R_k-V_{\varphi }\left(S_k\right)\right)}^2\right],$$

(21)

where $\varphi$ is the critic network parameter.

In order to prevent excessive differences in the policy updates of the actor networks, a clipped surrogate loss function was adopted [30], which is expressed as

$$L^{\mathrm{CLIP}}\left({\theta }_u\right)=\mathbb{E}\left[min\left(r_{{\theta }_u}{\widehat{A}}_k\right),clip\left(r_{{\theta }_u},1-\Delta ,1+\Delta \right){\widehat{A}}_k\right],$$

(22)

where $r_{{\theta }_u}=\frac{{\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)}{{\pi }_{{\theta }_u^{\mathrm{old}}}\left(A_{k_u}|S_k\right)}$ is the ratio of the current policy to the old policy and ${\pi }_{{\theta }_{old}}\left(A_k|S_k\right)$ represents the old policy. $\Delta$ is a hyperparameter. ${\widehat{A}}_k$ is the advantage function estimated by the critic network, which is used to measure the advantage of action $A_k$ in state $S_k$ compared with the average action. The detailed definition of ${\widehat{A}}_k$ is given by

$${\widehat{A}}_k=R_k-V_{\varphi }\left(S_k\right).$$

(23)

In order to avoid falling into a suboptimal policy and encourage agents to explore, an entropy coefficient $\mathcal{H}\left({\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)\right)$ is introduced into the loss function of the actor networks. Then, the loss function of the actor network is expressed as

$$L\left({\theta }_u\right)=\sum _{u=1}^U\left[L^{\mathrm{CLIP}}\left({\theta }_u\right)+\iota \mathbb{E}\left[\mathcal{H}\left({\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)\right)\right]\right],$$

(24)

where $\iota$ is a hyperparameter.

The detailed flow of the MAHPPO algorithm is shown in Algorithm 1. In order to enhance the robustness of the algorithm, we normalized $\mathcal{S}$ and $\mathcal{A}$ to the range $[-1,1]$. Moreover, the reward scaling technique was used for improving algorithm performance [31].

3.2. SMP-Based Buoy–UAV Association Algorithm

Although the designed MAHPPO algorithm can determine the UAV mission modes, the association relationship between the buoys and UAVs is still unknown. In this subsection, we focus on addressing the association relationship problem. Given the transmit power of the buoys and the flight velocity and flight angle of the UAVs, P1 can be further written as

$$\begin{array}{cc}\hfill \left(\mathrm{P}3\right):\underset{\mathit{A}}{max}& {\sum }_{k=1}^K{\sum }_{i=0}^M{R}_{u,i,k}\hfill \\ & \mathrm{s}.\mathrm{t}.\mathrm{C}3,\mathrm{C}4,\mathrm{C}6,\mathrm{C}7,\mathrm{C}11-\mathrm{C}16.\hfill \end{array}$$

(25)

Since P3 is a mixed-integer non-convex problem, the SMP method is used to obtain the optimal solution in an iterative manner.

The SMP was first proposed in [32], which contains two groups of elements, called the male set and female set. Everyone in the set has his/her own preference list for the opposite sex. According to the preference list, the current match was considered stable if there was not one male or female who rated the female or male in the other couples as better than the current partner in any two couples. The Gale–Shapley (GS) algorithm effectively solves the above problems. The main idea of the GS algorithm is to take the male’s point of view (either male or female) and let a male select a female according to his/her preference list. The female temporarily accepts the choice until a better man chooses her and then updates her choice. According to the above process, the two sets continuously cycle and update the selection until the match is stable.

Algorithm 1 Algorithm.

1: Initialize the actor network parameters ${\theta }_u,\forall u\in \mathcal{U}$ and the critic network parameter $\varphi$.   2: Initialize the experience replay buffer $\mathcal{D}$ with size D, mini-batch size $D^{\prime }$, sample reuse time N, and state $S_0$.   3: for episode = 1 to $E{p}_{\max }$ do   4:    Initialize the number of total mission completion time slots $k=1$, the energy penalty $E_{\mathrm{pe}}$, and the position penalty ${\rho }_k$.   5:    while done is not True do   6:      Every agent sample action $A_{u,k}$ with ${\pi }_{{\theta }_u}\left(A_{k_u}|S_k\right)$.   7:      if the UAV u flies beyond the target area then   8:         ${\rho }_k=1/U$, cancel the actions of $V_{u,k}$ and ${\psi }_{u,k}$, and update $r_k$, $S_{k+1}$ based on the current state.   9:      end if 10:      if $E_u^{\mathrm{total}}>{\overline{E}}_u,\forall u\in \mathcal{U}$ then 11:         Let done be True, and $E_{\mathrm{pe}}=-50$ 12:      end if 13:      Get the reward $r_k$ and next state $S_{k+1}$. 14:      Store experience data $\left(S_k,A_k,r_k,S_{k+1}\right)$ in $\mathcal{D}$. 15:      $k=k+1$. 16:      if UAVs complete the mission or $k=K^{\max }$ then 17:         Let done be True. 18:      end if 19:      if $\mathcal{D}$ is full then 20:         Compute advantages ${\widehat{A}}_k$ and the state value $V_{\varphi }\left(S_t\right)$. 21:         for epoch = 1 to $N\times \left(D/D^{\prime }\right)$ do 22:           Sample $D^{\prime }$ from $\mathcal{D}$. 23:           Compute $L\left({\theta }_u\right)$ of each actor network by (21). 24:           Update each actor network by a gradient method ${\theta }_u\leftarrow {\theta }_u-{\beta }_a{\nabla }_{{\theta }_u}L\left({\theta }_u\right)$. 25:           Compute $L\left(\varphi \right)$ by (18). 26:           Update critic network by $\varphi \leftarrow \varphi -{\beta }_a{\nabla }_{\varphi }L\left(\varphi \right)$. 27:         end for 28:         Clear the experience replay buffer $\mathcal{D}$. 29:      end if 30:    end while 31: end for

In this paper, the buoy–UAV association relationship problem can be regarded as a preference ordering problem, which is a standard SMP. The UAVs and buoys can be regarded as males and females, respectively. In contrast to the traditional SMP problem, U and M may not be equal. At time slot k, the goal of minimizing the males’ cost can be regarded as maximizing the total amount of data collected by the UAVs. The amount of data collected is positively correlated with the SNR, so the preference value could be denoted by the SNR between the UAVs and buoys. According to the $P_{u,k}$ obtained by the MAHPPO algorithm, the transmit power matrix of the buoys at time slot k can be expressed as

$$\mathit{P}={\left[\begin{array}{ccccc}{P}_{1,k}& \cdots & P_{1,k}& \cdots & P_{1,k}\\ ⋮& \ddots & & ⋰& ⋮\\ P_{u,k}& & P_{u,k}& & P_{u,k}\\ ⋮& ⋰& & \ddots & ⋮\\ P_{U,k}& \cdots & P_{U,k}& \cdots & P_{U,k}\end{array}\right]}_{U\times M}$$

(26)

Then, the preference matrix can be expressed as

$$\mathit{G}={\left[\begin{array}{ccccc}{g}_{1,1,k}& \cdots & g_{1,m,k}& \cdots & g_{1,M,k}\\ ⋮& \ddots & & ⋰& ⋮\\ g_{u,1,k}& & g_{u,m,k}& & g_{u,M,k}\\ ⋮& ⋰& & \ddots & ⋮\\ g_{U,1,k}& \cdots & g_{U,m,k}& \cdots & g_{U,M,k}\end{array}\right]}_{U\times M}$$

(27)

It is unfair to match the association relationship directly according to the above preference matrix. For example, the distance between UAV u and buoy m is further than that between UAV $u^{\prime }$ and buoy m, but the agent u chooses higher transmit power to make $g_{u,m,k}>g_{u^{\prime },m,k}$, which causes buoy m to tend to be associated with UAV u and consume more energy of buoy m. Hence, the transmit power of all buoys was set to be the same.

Since the distances between the buoys and UAV u are different, there is no element value the same in any row. For buoy m, the distances between the UAVs and buoy m are also different due to the limit of $d^{min}$, so there is no element value the same in any column, except in the case of $P_{u,k}^{\prime }=0$. Hence, there must be $u\ne u*$ and $m\ne m*$ if there is $g_{u,m,k}=g_{u*,m*,k}$ in $\mathit{G}$. The maximum SNR is defined as $g^{max}$. Combined with the idea of the GS algorithm, we introduce Lemma 1.

Lemma 1. 

The buoy–UAV with $g^{max}$ in ${\mathit{G}}_{U,M,k}$ is the best choice for each.

Proof of Lemma 1. 

Suppose $U=2$ and $M=3$ and the preference of the first and second UAVs is $g_{1,1,k}>g_{1,2,k}>g_{1,3,k}$ and $g_{2,1,k}>g_{2,3,k}>g_{2,2,k}$, respectively. The preferences of the three buoys are $g_{2,1,k}>g_{1,1,k}$, $g_{1,2,k}>g_{2,2,k}$, and $g_{2,3,k}>g_{1,3,k}$. Hence, we have $g^{max}=g_{2,1,k}$. First, UAV 1 and UAV 2 both choose Buoy 1 according to their own preferences. However, Buoy 1 is selected to be associated with UAV 2 because of $g_{2,1,k}>g_{1,1,k}$. The second row and first column of ${\mathit{G}}_{U,M,k}$ are deleted to obtain ${\mathit{G}}^{\prime }$. UAV 1 chooses the maximum value in ${\mathit{G}}^{\prime }$, which means it chooses to be associated with Buoy 2, and so on.    □

The above result could be also obtained by changing the goal to the cost of the buoys. Hence, the SAA algorithm achieves stable matching while obtaining the global optimal solution and is shown in Algorithm 2.

3.3. Joint Optimization Algorithm

After obtaining the association relationship between the buoys and UAVs, we aimed to determine the UAV–OBS association relationship. The considered association relationship subproblem of the UAV–OBS with the given $\mathit{V}$, $\mathit{\psi }$ and $\mathit{P}$ is formulated as

$$\begin{array}{cc}\hfill \left(\mathrm{P}4\right):\underset{\mathit{A}}{max}& {\sum }_{k=1}^K{\sum }_{u=1}^U{R}_{u,0,k}\hfill \\ & \mathrm{s}.\mathrm{t}.\mathrm{C}4,\mathrm{C}6,\mathrm{C}11,\mathrm{C}12,\mathrm{C}15,\mathrm{C}16.\hfill \end{array}$$

(28)

Although agents choose to offload the data, they cannot determine whether the actions satisfy the relevant constraints of subproblem P4. Hence, a heuristic algorithm of the UAV–OBS association is proposed to guarantee that the UAV’s actions satisfy these constraints. The detailed procedure is shown in Algorithm 3. In particular, lines 3–10 determine whether the current action selections ${\chi }_k$ of the UAVs satisfy the relevant constraints and update the action selections ${\chi }_k^{\mathrm{new}}$. In line 5, the bandwidth occupied by each UAV is not available due to the uncertainty of the UAV–OBS association. Hence, we take the maximum value B satisfying constraints C15 and C16 when solving the problem P4.

Algorithm 2 SMP-based buoy-UAV association algorithm (SAA).

1: ${\mathcal{U}}^{\mathrm{co}}$, ${\chi }_k^{\mathrm{new}}$, $V_k$, $c_k^{\mathrm{buoy}}$, ${\kappa }_k$, $\mathcal{P}=\left\{P_{u,k},\forall u\in {\mathcal{U}}^{\mathrm{co}}\right\}$, and $h_{u,m,k},\forall u\in \mathcal{U},m\in \mathcal{M}$.   2: Set $i=0$.   3: for$u=1$to$U^{\mathrm{co}}$do   4:    $P^{\max }=max\left(\mathcal{P}\right)$.   5:    Let the transmit power of all buoys be $P^{\max }$, and delete $P^{\max }$ from $\mathcal{P}$.   6:    Obtain the preference matrix $\mathit{G}$ according to $h_{u,m,k}$ and $P^{\max }$.   7:    Obtain the energy consumption $E_u^{\mathrm{pr}},\forall u\in {\mathcal{U}}^{\mathrm{co}}$ according to $V_k$.   8:    Obtain the energy consumption of all buoys $E_m,\forall m\in \mathcal{M}$ according to $P^{\max }$.   9:    for $m=1$ to M do 10:      if $c_{m,k}^{\mathrm{buoy}}=0$, or $g_{u,m,k}$, $E_u^{\mathrm{pr}}$, or $E_m$ does not satisfy C3 and C6 then 11:         Set $g_{u,m,k}=0$ in $\mathit{G}$, so as to obtain the new preference matrix ${\mathit{G}}^{\prime }$. 12:      end if 13:    end for 14:    for $n=1$ to $U^{\mathrm{co}}$ do 15:      Obtain the maximum SNR $g^{max}$ in ${\mathit{G}}^{\prime }$. 16:      if $g^{max}\ne 0$ then 17:         Obtain the UAV number u and buoy number m corresponding to $g^{max}$, and ${\kappa }_{u,k}=m$. 18:         Delete the UAV number u from ${\mathcal{U}}^{\mathrm{co}}$, and set ${\chi }_{u,k}^{\mathrm{new}}=0$. 19:         Set the row u and column m of ${\mathbf{G}}^{\prime }$ equal to 0, and then, update ${\mathit{G}}^{\prime }$. 20:      else 21:         break. 22:      end if 23:    end for 24:    if ${\mathcal{U}}^{\mathrm{co}}=\varnothing$ then 25:      break. 26:    end if 27: end for 28: Let ${\kappa }_k^{\prime }$ indicate the set of UAVs that are first transformed. 29: if UAV u is not associated with any buoy then 30:    Set ${\chi }_{u,k}^{\mathrm{new}}=1$, and add the number of UAVs u to ${\kappa }_k^{\prime }$. 31: end if 32: if${\chi }_{u,k}=1$ and ${\chi }_{u,k}^{\mathrm{new}}=1$ then 33:    Set ${\chi }_{u,k}^{\mathrm{new}}=2$, and ${\kappa }_k=-1$. Then, add the number of UAVs u to ${\kappa }_k^{\prime }$. 34: end if 35: return${\chi }_{u,k}^{\mathrm{new}}$, ${\kappa }_k$, ${\kappa }_k^{\prime }$.

In summary, the problem P1 could be solved effectively and efficiently with the proposed SAA, UAA, and MAHPPO algorithms. The detailed process is shown in Algorithm 4. Specifically, we first obtain ${\chi }_k$ according to the actions of the agents at time slot k in Algorithm 1. According to ${\chi }_k$, we divide the number of UAVs into two initial groups: collection ${\mathcal{U}}^{\mathrm{co}}$ and offloading ${\mathcal{U}}^{\mathrm{of}}$. In line 8, we perform Algorithm 3 to judge ${\mathcal{U}}^{\mathrm{of}}$. In lines 11–24 of Algorithm 3, the UAV whose action does not satisfy the constraints is transformed to ${\chi }_{u,k}^{\mathrm{new}}=0$, and its number is formed into a new set ${\kappa }_k^{\prime }$. In lines 9–10, we update ${\mathcal{U}}^{\mathrm{co}}$ by ${\mathcal{U}}^{\mathrm{co}}\cap {\kappa }_k^{\prime }$. Then, we perform Algorithm 2 to judge ${\mathcal{U}}^{\mathrm{co}}$. Some UAVs in ${\kappa }_k^{\prime }$ may not satisfy the constraints of both data collection and data offloading after two judgments. Hence, in lines 32–34 of Algorithm 2, we set ${{\chi }^{*}}_{u,k}=2$ if ${\chi }_{u,k}=0$ and ${\chi }_k^{\mathrm{new}}=0$. In lines 12–13, for UAVs that have only experienced the judgment of Algorithm 2 once, but did not satisfy the constraints, we perform Algorithm 3 to make the second judgment. In lines 22–24 of Algorithm 3, we set ${{\chi }^{*}}_{u,k}=2$ if ${\chi }_{u,k}=1$ and ${\chi }_k^{\mathrm{new}}=1$. Hence, there may be some UAVs that neither collect, nor offload data in a certain time slot. After the above three judgments, the association relationship of all UAVs can be determined. The above procedure is looped until done is True.

Algorithm 3 UAV–OBS association algorithm (UAA).

1: Input ${\overline{g}}_{\mathrm{of}}$, $V_{u,k},\forall u\in \mathcal{U}$, ${\mathcal{U}}^{\mathrm{of}}$, ${\chi }_k^{\mathrm{new}}$, ${\kappa }_k$, $P_{u,0,k},\forall u\in \mathcal{U}$, and current channel gain $h_{u,0,k},\forall u\in \mathcal{U}$.   2: repeat   3:    Obtain the SNR of UAV–OBS $g_{u,0,k}$ according to $P_{u,0,k}$.   4:    Obtain the maximum transmission rate of UAV ${R_{max}}_{u,0,k}$ according to B.   5:    Obtain the energy consumption of UAVs $E_u^{\mathrm{total}}$ according to $V_u,k$,.   6:    if $g_{u,0,k}$, $E_u^{\mathrm{total}}$, and ${R_{max}}_{u,0,k}$ satisfy C4, C6, and C15, respectively then   7:      Set ${\chi }_k^{\mathrm{new}}=1$, and ${\kappa }_{u,k}=0$.   8:    else   9:      Set ${\chi }_k^{\mathrm{new}}=0$ 10:    end if 11:    if ${R_{max}}_{u,0,k}$ does not satisfy C15, and the data of all buoys are fully collected then 12:      if $g_{u,0,k}$ and $E_u^{\mathrm{total}}$ satisfy C4 and C6 then 13:         Set ${\chi }_k^{\mathrm{new}}=1$ and ${\kappa }_{u,k}=0$. 14:      else 15:         Set ${\chi }_k^{\mathrm{new}}=0$. 16:      end if 17:    end if 18:    $u=u+1$ 19: until$u=U^{\mathrm{of}}$ 20: Let ${\kappa }_k^{\prime }$ indicate the set of UAVs that are first transformed. 21: if${\chi }_{u,k}=0$ and ${\chi }_k^{\mathrm{new}}=0$ then 22:    Set ${{\chi }^{*}}_{u,k}=2$ and ${\kappa }_k=-1$. 23:    Add the number of UAVs u to ${\kappa }_k^{\prime }$. 24: end if 25: return${\chi }_{u,k}^{\mathrm{new}}$, ${\kappa }_k$, ${\kappa }_k^{\prime }$.

Algorithm 4 Joint optimization algorithm based on SAA, UAA, and MAHPPO (SU-MAHPPO).

Input:  The initial positions of UAVs $q_{u,1},\forall u\in \mathcal{U}$, the positions of buoys $x_m,y_m,\forall m\in \mathcal{M}$, and the position of OSB $x_0,y_0$. Output:  $\mathit{A},\mathit{P},\mathit{V},\mathit{\psi }$.   1: /* In Algorithm 1 */   2: for episode = 1 to $E{p}_{\max }$ do   3:    while done is not True do   4:      Perform $Lines$ 6–9 of Algorithm 1.   5:      Obtain ${\chi }_k$ according to $A_{u,k}$.   6:      Obtain initial ${\mathcal{U}}^{\mathrm{co}}$ and ${\mathcal{U}}^{\mathrm{of}}$ according to ${\chi }_k$.   7:      Let ${\chi }_k^{\mathrm{new}}={\chi }_k$ and ${\kappa }_{u,k}=-1,\forall u\in \mathcal{U}$.   8:      Update ${\chi }_k^{\mathrm{new}}$, ${\kappa }_k$, and ${\kappa }_k^{\prime }$ by performing Algorithm 3.   9:      if ${\kappa }_k^{\prime }$ is not null or ${\chi }_{u,k}^{\mathrm{new}}=0$ exists in ${\chi }_k^{\mathrm{new}}$ then 10:         ${\mathcal{U}}^{\mathrm{co}}\leftarrow {\mathcal{U}}^{\mathrm{co}}\cap {\kappa }_k^{\prime }$. 11:         Update ${\chi }_{u,k}^{\mathrm{new}}$, ${\kappa }_k$, and ${\kappa }_k^{\prime }$ by performing Algorithm 2. 12:         if ${\kappa }_k^{\prime }$ is not null then 13:           ${\mathcal{U}}^{\mathrm{of}}\leftarrow {\kappa }_k^{\prime }$. 14:           Update ${\chi }_{u,k}^{\mathrm{new}}$, ${\kappa }_k$, and ${\kappa }_k^{\prime }$ by performing Algorithm 3. 15:         end if 16:      end if 17:      Perform $Lines$ 10–26 of Algorithm 1. 18:    end while 19: end for

3.4. Complexity Analysis

The complexity of a neural network usually depends on the state dimension of the input layer, the number of neural network layers, and the number of neurons in the hidden layers and output layer. Since the MAHPPO algorithm contains U actor networks and a critic network, the complexity of the MAHPPO algorithm can be expressed as $\mathcal{O}\left(E_{p_{max}}{K}^{max}\left(U{\sum }_{l=1}^{L^a-1}{e}_l^a{e}_{l+1}^a+{\sum }_{l=1}^{L^c-1}{e}_l^c{e}_{l+1}^c\right)\right)$, where $L^a$ and $L^c$ represent the number of neural network layers of the actor and critic networks, respectively, and $e_l^c$ and $e_l^a$ represent the number of neurons in the l-th layer of the actor and critic networks, respectively. Since the maximum value of $U_{\mathrm{co}}$ and $U_{\mathrm{of}}$ is U, the highest algorithm complexity of Algorithm 2 and Algorithm 3 is $\mathcal{O}\left(U\left(U+M\right)\right)$ and $\mathcal{O}\left(U\right)$. Obviously, the complexity of Algorithm 2 is higher than that of Algorithm 3. In order to reduce the complexity of Algorithm 4, Algorithm 3 is executed with priority so that Algorithm 2 only needs to be executed once. In summary, the computational complexity of Algorithm 4 is $\mathcal{O}\left(E_{p_{max}}{K}^{max}\left(U{\sum }_{l=1}^{L^a-1}{e}_l^a{e}_{l+1}^a+{\sum }_{l=1}^{L^c-1}{e}_l^c{e}_{l+1}^c+U\left(U+M\right)+U\right)\right)$.

4. Simulation Results and Discussion

In the simulations, we set a 5000 m × 5000 m target area and a 1500 m × 1500 m no-fly zone. $M=10$ buoys with a data volume of 10 Mbits and a maximum transmit power $P_{\max }=24$ dBm were randomly distributed in the target area. The horizontal coordinates of the OBS were $(0,0)$ m. The initial horizontal coordinates of the UAVs with the flight height $H=100$ m were $(0,0)$ m, $(0,2500)$ m, and $(2500,0)$ m, respectively. The maximum flight velocity and angle of the UAVs were $V_{\max }=50$$\mathrm{m}/\mathrm{s}$ and ${\psi }_{u,k}\in [0,2\pi ]$, respectively. The transmit power of each UAV was $P_{u,0,k}=0.1$ W. The minimum safety distance between the UAVs was $d_{\min }=50$ m. The time slot duration was $\delta =1$ s. For the MAHPPO algorithm, we set the length of the episode $E{p}_{\max }=18,000$, $K^{\max }=250$. The actor network and the critic network both had three hidden layers with $\left(256,128,64\right)$ neurons. The learning rates of the actor network and the critic network were $bet{a}_a=0.0001$ and $bet{a}_b=0.0003$, respectively. The discount factor was $\gamma =0.99$. The experience replay buffer size D, mini-batch size $D^{\prime }$, and sample reuse time N were set to 1024, 256, and 8, respectively. The hyperparameters were $\Delta =0.2$ and $\iota =0.01$. Moreover, our simulation was based on Python 3.8 with the Pytorch package. Other simulation parameters are shown in

Table 2. Simulation parameters.

Parameter	Value
environment parameter, a, b	9.61, 0.16
excessive path loss, ${\xi }_{\mathrm{LoS}}$, ${\xi }_{\mathrm{NLoS}}$	1, 20
channel parameter, ${\mu }_{u,i,k}$	1
wavelength, $\lambda$	0.15 m
noise power, ${\sigma }^2$	−104 dBm
SNR threshold, ${\overline{g}}_{\mathrm{co}}$, ${\overline{g}}_{\mathrm{of}}$	8 dB, 2 dB
energy threshold, ${\overline{E}}_{\mathrm{u}}$, ${\overline{E}}_{\mathrm{m}}$	150 kJ, 1.25 J
tip speed of the rotor blade, $U_{\mathrm{tip}}$	120
fuselage drag ratio, $d_0$	0.6
air density, $\rho$	1.225
rotor solidity, l	0.05
blade angular velocity, f	0.503
blade profile power and induced power, $P_o$, $P_i$	79.86 W, 0.99 W
LoS and NLoS link factor, ${\upsilon }^{\mathrm{LoS}}$, ${\upsilon }^{\mathrm{NLoS}}$	2, 2

.

Furthermore, we compared the proposed SU-MAHPPO algorithm with three baseline algorithms, which are listed as follows:

• SU-MAPDQN algorithm: This algorithm refers to using the multi-agent PDQN (MAPDQN) algorithm to replace the MAHPPO algorithm in the algorithm proposed in our paper.
• SU-MAPPO algorithm: This algorithm refers to using the multi-agent PPO (MAPPO) algorithm to replace the MAHPPO algorithm in the algorithm proposed in this paper. Furthermore, the continuous action space of the UAVs is discretized in this algorithm. The flight angle of UAV u at time slot k is expressed in four directions: up, down, left, and right, i.e., ${\psi }_{u,k}\in \left\{0,\pi /2,\pi ,3\pi /2\right\}$. The flight velocity of the UAV at time slot k is expressed as $V_{u,k}\in \left\{0,10,20,30,40,50\right\}$ m/s.
• MAHPPO algorithm: In this algorithm, the association relationships of the buoy–UAV and UAV–OBS are optimized. Hence, the action space of each agent of this algorithm is denoted as $A_{u,k}=\left\{\mathcal{I},{\psi }_{u,k},V_{u,k},P_{u,k}\right\}$. The state space is denoted as ${\mathcal{S}}_k=\left\{{\kappa }_k,{\kappa }_k^{\mathrm{a}},c_k^{\mathrm{buoy}},E_k^{\mathrm{buoy}},x_k,y_k,c_k^{\mathrm{uav}},E_k^{\mathrm{uav}},B_k\right\}$, where ${\kappa }_k^{\mathrm{a}}$ represents the discrete actions of all agents. The MAHPPO algorithm is shown in Algorithm 5.

Algorithm 5 MAHPPPO algorithm.

1: /* In Algorithm 1 */   2: Perform $Lines$ 1–6 of Algorithm 1.   3: Obtain ${\kappa }_k^{\mathrm{a}}$ according to the discrete actions of $A_k$.   4: Let ${\kappa }_k={\kappa }_k^{\mathrm{a}}$   5: for$u=1$toUdo   6:    for $u*=1$ to U do   7:      if ${\kappa }_{u,k}^{\mathrm{a}}={\kappa }_{u*,k}^{\mathrm{a}}$ and ${\kappa }_{u,k}^{\mathrm{a}}\ne 0$ then   8:         if $g_{u,k}>g_{u*,k}$ then   9:           Set ${\kappa }_{u,k}=-1$. 10:         end if 11:      end if 12:      if does not satisfy C3, C4, C6, C7, and C15 then 13:         Set ${\kappa }_{u,k}=-1$. 14:      end if 15:    end for 16: end for 17: Perform $Lines$ 7–31 of Algorithm 1.

As shown in Figure 3, we first compared the accumulative reward of SU-MAHPPO with the other three algorithms. The curves were smoothed for the convenience of observation. We can see that the proposed SU-MAHPPO algorithm was significantly better than the other algorithms. The SU-MAHPPO scheme could be stable and convergent after 10,000 episodes. Although the SU-MAPPO algorithm can quickly converge, the accumulated reward was lower than the SU-MAHPPO algorithm, and the curve suddenly dropped after 15,000 episodes of training. Moreover, the SU-MAPDQN algorithm cannot be convergent after 18,000 episodes. This is because the continuous action values corresponding to each value of discrete actions are output in MAPDQN, which obviously increases the computational complexity. Furthermore, the performance of the MAHPPO algorithm was far lower than that of our proposed algorithm within 18,000 episodes. This is because the complex action space and state space make it more difficult for the agents to explore. This confirms the advantages of the combination of the SAA algorithm and UAA algorithm to narrow the exploration domain of the agents.

Figure 4 shows the effectiveness of the technologies mentioned in Section 3.1. It can be seen that the proposed SU-MAHPPO algorithm without reward scaling had difficulty converging and was more unstable within 18,000 episodes. The reason is that the reward scaling is beneficial to the fitting of the value function. The curve of the SU-MAHPPO algorithm without state and action normalization shows an upward trend, but it also cannot converge within 18,000 episodes. This is because the order of magnitude difference between the parameters in the state ${\mathcal{S}}_k$ and action space ${\mathcal{A}}_k$ is large, which is not conducive to the training of neural networks. Reward scaling is beneficial to the fitting of the value function. Hence, the above two techniques can effectively enhance the convergence and stability of the proposed algorithm.

Figure 5 shows the UAV trajectories’ comparison between the SU-MAHPPO algorithm and SU-MAPPO algorithm. The mission completion time of SU-MAHPPO and SU-MAPPO was 53 s and 67 s, respectively. SU-MAHPPO is better, since the designed reward function is positively related to the data transmission rate in each time slot. Moreover, the continuous action space of the SU-MAHPPO algorithm allows the agents to more accurately control the positions of the UAVs in each time slot. Besides, the continuous action space makes it more accurately adjust the transmit power of the buoys to maximize the accumulative reward. The discretized action space makes it difficult for the UAVs to reach the optimal trajectory for collecting and offloading data. Furthermore, it can be seen that the UAV trajectories of the proposed algorithm were not close to the buoy or OBS. This is because the UAVs can change the mission mode in different time slots. Take the trajectory of the UAV close to the buoy for example. In this case, the time of data collection will be reduced, but the UAV will be far away from the OBS, which makes it difficult to meet the SNR threshold of data offloading. Hence, the UAV needs more time to offload the data, so that the total mission completion time is longer.

Figure 6 shows the trajectories of the three groups of UAVs taking off from different initial positions. The initial positions of the first group of UAVs are the same as the default positions, namely Position 1. The initial positions of the second group of UAVs are (1000, 1000), (1500, 2000), and (3000, 2000), namely Position 2. The initial positions of the third group of UAVs are (1500, 1500), (1500, 3000), and (3000, 1500), namely Position 3. It can be seen that the UAVs can avoid the no-fly zone even if they take off at the edge of the no-fly zone. The performance advantage benefits what we set as the position penalty in the reward function, which can guide the agent to control the UAVs flying in the target area.

Figure 7 shows the comparison of the multi-UAV trajectories based on the SU-MAPPO algorithm with different SNR thresholds of data collection. It can be seen that the UAV trajectories were closer to the buoys with the growth of the SNR threshold. This is because the designed reward function is mainly related to the transmission rate and the time reward $K^{*}$. With the SNR thresholds increasing, the UAV needs to be close enough to the buoy to achieve the minimum transmission rate requirement. The agent chooses to let the UAV be closer to the buoy to enhance the data collection transmission rate, which contributes to shortening the data collection time and obtaining a larger value of $K^{*}$.

Figure 8 shows the total mission completion time of the SU-MAHPPO algorithm and the SU-MAPPO algorithm with different SNR thresholds of data collection. Figure 9 and Figure 10 show the total mission completion time of the SU-MAHPPO algorithm and the SU-MAPPO algorithm with different channel conditions. It can be seen that the performance of our proposed algorithm is significantly better than the SU-MAPPO algorithm. The reason is that the discretized action space makes the SU-MAPPO algorithm unable to explore and select the optimal action in the considered environment, which further verifies the advantage of the proposed algorithm in the continuous action space.

5. Conclusions

This paper studied the multi-UAV-assisted data collection and offloading problem in the maritime IoT. The proposed MAHPPO-based joint optimization algorithm could significantly reduce the total mission completion time. First, a MAHPPO-based algorithm was proposed to obtain the UAV trajectories, transmit power of the buoy, and mission mode of the UAVs. Second, an SMP-based algorithm was designed to solve the association relationship subproblem. Finally, a joint optimization algorithm was proposed in which the SAA algorithm and the UAA algorithm were executed alternately to finally determine the association relationship of the buoy–UAV and UAV–OBS. The simulation results showed that the proposed algorithm can effectively reduce the total mission completion time of the UAVs, and its performance was significantly better than the comparison schemes.

In the future work, we will investigate cooperative multi-UAV communication to further improve the data collection and offloading performance. Moreover, the location uncertainties of the buoys will be considered to enhance the channel estimation accuracy to improve the buoy–UAV association efficiency.