Sea-Based UAV Network Resource Allocation Method Based on an Attention Mechanism

Abstract

As humans continue to exploit the ocean, the number of UAV nodes at sea and the demand for their services are increasing. Given the dynamic nature of marine environments, traditional resource allocation methods lead to inefficient service transmission and ping-pong effects. This study enhances the alignment between network resources and node services by introducing an attention mechanism and double deep Q-learning (DDQN) algorithm that optimizes the service-access strategy, curbs action outputs, and improves service-node compatibility, thereby constituting a novel method for UAV network resource allocation in marine environments. A selective suppression module minimizes the variability in action outputs, effectively mitigating the ping-pong effect, and an attention-aware module is designed to strengthen node-service compatibility, thereby significantly enhancing service transmission efficiency. Simulation results indicate that the proposed method boosts the number of completed services compared with the DDQN, soft actor–critic (SAC), and deep deterministic policy gradient (DDPG) algorithms and increases the total value of completed services.

1. Introduction

With the continuous development of communication technology, human activity areas have gradually expanded from land to the ocean. Hence, maritime, wide-area, all-weather communication support is urgently required to provide uninterrupted network services for space-based, air-based, ground-based, and naval information services [1]. This is also an important topic in current network communication technology research. For example, B5G/6G integrates air, land, and sea network architectures into its core structure [2]. Owing to the extensive coverage and intricate structure of maritime wireless networks, coupled with diverse demands for network resources, service attributes, and quality of service (QoS), unmanned aerial vehicle networks [3] have emerged as a vital component of maritime wireless networks, leveraging their high scalability, high mobility, and broad coverage. Simultaneously, limited communication resources and environmental uncertainty significantly increase the difficulty of human production and activities. Hence, methods for orchestrating and allocating resources with different network dynamics are urgently required [4] to maximize the use of limited communication resources [5].

The current research on wireless network resource allocation focuses on the following three aspects: power allocation [6,7,8,9], spectrum allocation [10,11,12], and access selection [13,14]. Radio access technology (RAT) is a key technology that can achieve seamless connections for users under wide-area coverage, and it is the first step in resource orchestration and allocation. Current access selection algorithms are mainly divided into two categories, one of which relies on traditional operations and research methods for technical support, and the other approach relies on machine learning methods to determine optimal access strategies. The first category [15,16,17,18,19,20] relies on traditional operations research methods as technical support to provide accurate access result calculations. For example, Zhu et al. [16] proposed an innovative access selection mechanism based on remns to improve the quality of experience issues for 5G Internet of Things users under any motion state. Using a comprehensive preference assessment framework based on the analytic hierarchy process (AHP) and entropy weight method, they measured the preference degree of each Internet of Things (IoT) service for network attributes to reduce network congestion and total latency, improve user service quality, and enhance transmission efficiency under the constraint of limited network capacity.

For the other category [21,22,23,24,25,26], traditional operational research methods are typically used to solve low-dimensional discrete problems. However, scenarios involving air, ground, and sea are complex, and they contain many constraints as well as a large solution space, rendering them difficult for traditional methods to handle. When facing complex high-dimensional problems, machine learning methods are required to provide the means to approximate functions. For instance, Bo [13] proposed a joint relay selection and access control algorithm based on distributed Q-learning (DQL) to address the issues of low transmission rates and network congestion in multiuser multirelay uplink satellite–terrestrial networks. By learning the optimal relay transmission slots, the proposed algorithm was significantly better than traditional algorithms in terms of throughput, access delay, and rate. Please refer to

Table 1. Existing typical wireless access technologies.

Traditional Operations Research Methods		Machine Learning Methods
Technical Approach	Application Scenarios	Technical Approach	Application Scenarios
Evolutionary game and AHP [15]	5G, LTE	Distributed Q-learning [21]	Satellite ground relay network
AHP and entropy weighting method [16]	5G Internet of Things (IoT)	Deep Q-learning [22]	Large-scale machine communication scenarios in the IoT
Clustering [17]	Ultra-dense wireless communication system	Transfer learning [23]	Internet of Vehicles
MADM and AHP [18]	5G HetNet	Deep Q-learning [24]	Disaster response network
Discrete-time MDP and Lagrangian method [19]	5G NR-WiFi HetNet	Distributed Q-learning [25]	FRAN
Paris Metro pricing scheme [20]	4G/5G HetNet	Bayesian deep Q-learning [26]	Industrial IoT

for a detailed comparison.

Although the algorithms mentioned in previous studies have addressed the problem of wireless network resource allocation on land from different perspectives, maritime scenarios are characterized by fast-moving nodes [27], a vast area of wireless networks, and a variety of business demands with significant differences. Hence, migrating existing land and air wireless network resource allocation schemes directly to a maritime scenario is difficult. Therefore, this study focuses on the adaptability issue of resource allocation in maritime UAV networks and proposes a maritime wireless network resource allocation method based on an attention mechanism. This method is divided into three modules as follows: a deep reinforcement learning algorithm based on the double DQN (DDQN), an action suppression module based on action-value constraints, and an attention–perception module based on the attention mechanism. These address the issue of the agent’s actions easily, thereby leading to the mitigation of the ping-pong effect; an enhancement in the matching of the network, nodes, and services; and a reduction in the number of switches. The contributions of this study are as follows:

(1) A multidimensional resource management model and objective optimization function for maritime wireless networks were constructed. To improve the business transmission efficiency of nodes within the network, a non-convex objective optimization function for node access to the network and the transmission rate was provided by integrating the network throughput model into the traditional Shannon communication model. Moreover, the joint optimization of the nodes’ business access method and network access selection strategy was performed.

(2) An action suppression module that controls the output of the actions was designed. An action-value constraint function was introduced to regulate the output of the actions, address the issue of the ping-pong effect when an agent outputs actions, and effectively reduce the number of link switches.

(3) An attention mechanism from the field of computer vision was proposed to introduce an attention-aware method for resource allocation. The observation information of the nodes was divided into self-state information and environmental state information, with targeted processing of different feature information, and a hybrid attention weight was formed to enhance the state perception ability of the nodes. This led to the optimization of node business access methods; promoted matching among networks, nodes, and services; and provided a new approach to resource allocation adaptability methods.

2. System Model

Maritime UAV networks deeply integrate the multidimensional resources of air, space, and sea to support the real-time and diverse needs of maritime users. Their architectural compositions include the ocean, airspace, and a small part of the shore base. Network nodes can be divided into low-speed nodes (e.g., ships, light unmanned aerial vehicles, and small unmanned aerial vehicles) and high-speed nodes (e.g., medium-sized unmanned aerial vehicles, large unmanned aerial vehicles, and airplanes), based on their mobility rates, as shown in Figure 1. It is worth noting that the aforementioned network nodes can act as user nodes as well as temporary base stations. Despite the significant differences among entities when abstracted as network nodes, their main effective differences are their mobility rates. Therefore, in the proposed model, they were constructed as low-speed/high-speed user nodes and low-speed/high-speed temporary base stations.

For ease of analysis and generality, we assumed that high-speed mobile nodes perform continuous random motion within the sea area, each of which is capable of executing up to N_max parallel services, with their service list information being $B=\left\{b_{1,1}^1,\dots ,b_{p,q}^i\right\}, p\le N_{\max },i\in I$, and j maritime mobile base stations move continuously and randomly throughout the sea area at a slower speed, with the network parameter set of the base stations being $P=\left\{P_1^1,P_2^1,P_3^1,P_4^1,\dots ,P_1^j,P_2^j,P_3^j,P_4^j\right\}, j\in J$. In the scenario considered in this study, the heterogeneous wireless communication network has a common cloud service resource pool that is responsible for collecting the services generated by each maritime base station and distributing them to various nodes. Each base station can provide users with access to multiple networks with different communication ranges, transmission powers, bandwidths, and other parameters with upper limits. User nodes continuously interact with surrounding base stations or other mobile nodes for business purposes during their maritime movements, enabling mobile nodes to execute services without interruption.

Given the intimate connection between network throughput and transmission rate, establishing a relationship between the two enables the optimization of multidimensional resource allocation at nodes. Thus, based on the traditional Shannon channel model, we introduce the TCP throughput model for wireless networks to formulate the relationship between transmission rate and network parameters, thereby developing a multidimensional resource allocation model for wireless network scenarios. The specific construction is detailed as follows:

The TCP throughput model [28] for wireless networks chosen by a node is

$$C_s={\displaystyle \frac{M_s}{\left(\overline{D_s}+J_s\right)\sqrt{L_s}}}$$

(1)

where M_S represents the maximum packet segment size, $\overline{D_s}$ represents average packet delay, J_S represents the packet jitter, and L_s represents packet loss rate. In addition, according to Shannon’s formula, the relationship between the transmission rate of node i and the bandwidth of network s is given by

$${\Gamma }_{i,s}={\displaystyle \frac{C_{\max }}{I_s}}\log \left(1+SNR\right)$$

(2)

where the maximum throughput of the node is the full occupation of the base station bandwidth $C_s\le C_{\max }$ and the number of node connections I_S. The signal-to-noise ratio of a node can be expressed as follows:

$$SNR={\displaystyle \frac{{\mathrm{P}}_r\cdot g_{i,s}\left(t\right)}{{\sigma }_s}}$$

(3)

where $g_{i,s}\left(t\right)$ is the channel gain and ${\sigma }_s$ is the channel noise. The received power of the node P_r is represented as

$${\mathrm{P}}_r={\displaystyle \frac{P_t{G}_{s,t}{G}_{s,r}{\lambda }_s^2}{{\left(4\pi d_s\right)}^2}}$$

(4)

where $\lambda$ is the signal frequency of the wireless network. By substituting Equation (1) into Equation (2), we obtain the wireless network transmission rate formula, as follows:

$${\Gamma }_{i,s}={\displaystyle \frac{M_s}{I_s\left(\overline{D_s}+J_s\right)\sqrt{L_s}}}\log \left(1+{\displaystyle \frac{P_t{G}_{s,t}{G}_{s,r}{\lambda }_s^2{g}_{i,s}\left(t\right)}{{\left(4\pi d_s\right)}^2{\sigma }_s}}\right)$$

(5)

Therefore, we aim to maximize the transmission rates of the wireless networks. Among them, the maximum packet segment size M_S and other parameters ${\displaystyle \frac{P_t{G}_{s,t}{G}_{s,r}}{{\left(4\pi d_s\right)}^2}}$ are fixed attributes of network s in this study. Hence, $\max {\Gamma }_{i,s}$ can be simplified to $\begin{array}{cc}\max {\Gamma }_{i,s}=& \max \left\{{\displaystyle \frac{M_s}{I_s\left(\overline{D_s}+J_s\right)\sqrt{L_s}}}\log \left(1+{\displaystyle \frac{P_t{G}_{s,t}{G}_{s,r}{\lambda }_s^2{g}_{i,s}\left(t\right)}{{\left(4\pi d_s\right)}^2{\sigma }_s}}\right)\right\}\\ ={\displaystyle \frac{M_s}{\min \left[I_s\left(\overline{D_s}+J_s\right)\sqrt{L_s}\right]}}\log \left(1+{\displaystyle \frac{P_t{G}_{s,t}{G}_{s,r}}{{\left(4\pi d_s\right)}^2}}\cdot \max \left[{\displaystyle \frac{{\lambda }_s^2{g}_{i,s}\left(t\right)}{{\sigma }_s}}\right]\right)\end{array}$.

Simultaneously, we assumed that the total value of the business completed by the node is also as large as possible. Here, the value of the j-th business completed by node i is $V_j^i$, the number of completed businesses is $N_j$, the total value of completed businesses is $V_{\max }^i={\displaystyle \sum _{j=1}^{N_j}{V}_j^i}$, the duration of the j_t-th business completed by node i is $t_{j_t}^i$, the average value of the j_t-th business completed by node i is ${\overline{V}}_{j_t}^i$, and the number of j_t related to the time t is a function of the time t. This can be simplified to maximize the transmission business value $\max {\displaystyle \sum _{j=1}^{N_j}{V}_j^i}$:

$$\max {\displaystyle \sum _{j=1}^{N_j}{V}_j^i}=\max {\displaystyle \sum _{t=1}^{t_{\max }}{\displaystyle \sum _{j=1}^{N_{j,t}}{\overline{V}}_{j_t}^i{t}_{j_t}^i}}\to {\displaystyle \sum _{j=1}^{N_{j,t}}\min t_{j_t}^i}$$

(6)

The data volume $K_m^i$ of the m-th business of node i is represented as $K_m^i={\displaystyle \sum _{t=1}^{t_j^i}{\Gamma }_{i,s,t}}$, and ${\Gamma }_{i,s,t,j}$ is the transmission rate when node i is connected to network s at time t. By substituting this into Equation (6), we obtain

$$\min t_{j_t}^i=\min \left\{t|{\displaystyle \sum _{t=1}^{t_j^i}{\Gamma }_{i,s,t}}\ge K_m^i\right\}$$

(7)

In summary, the joint optimization constraint function for maximizing the transmission rate and total business value is obtained as follows:

$$\left\{\begin{array}{c}\begin{array}{c}\begin{array}{c}{\displaystyle \frac{M_s}{\min \left[I_s\left(\overline{D_s}+J_s\right)\sqrt{L_s}\right]}}\log \left(1+{\displaystyle \frac{P_t{G}_{s,t}{G}_{s,r}}{{\left(4\pi d_s\right)}^2}}\cdot \max \left[{\displaystyle \frac{{\lambda }_s^2{g}_{i,s}\left(t\right)}{{\sigma }_s}}\right]\right)\\ \min \left\{t|{\displaystyle \sum _{t=1}^{t_j^i}{\Gamma }_{i,s,t}}\ge K_m^i\right\}\end{array}\\ \begin{array}{cc}s.t.& C1:C_i\le C_{i\max }\end{array}\\ \begin{array}{c}C2:{\displaystyle \sum _i^{u_{i,s,k}=1}{C}_{i,s}}\le C_{s\max }\end{array}\\ \begin{array}{c}C3:N_s\le N_{\max }\end{array}\end{array}\\ \begin{array}{c}C4:{\displaystyle \sum _i^{u_{i,s,k}=1}{\Gamma }_{i,s}\le {\Gamma }_s}\end{array}\end{array}\right.$$

(8)

where the set of nodes I and the set of available networks S represent the collection of all individuals participating in business interactions in the current sea area. $C_{i\max }$ represents the maximum throughput obtained by node i from network k for base station j. Constraint C1 indicates that the bandwidth used by node i must not exceed the maximum bandwidth of the node. Constraint C2 indicates that the bandwidth allocated by the network must not exceed the maximum bandwidth of the network. Constraint C3 indicates that the number of businesses obtained by node i must not exceed the maximum number of businesses that the business list can carry. Constraint C4 indicates that the total rate allocated by network s to the connected nodes is less than or equal to the maximum rate of the base station.

For maritime communication, the common air–sea path loss model is based on the classic logarithmic path loss model formula, and the propagation environment of the ocean and sea wave conditions are corrected using correction factors. The channel noise reference is [29], and the channel gain model [30] is as follows:

$$g_{i,s}\left(t\right)=L(h_t,h_r,h_e,d_{m,k,t})+10n{\log }_{10}({\displaystyle \frac{d_t}{d_{m,k,t}}})+{\chi }_{\sigma }^{m,k,t}+\varsigma F_t$$

(9)

where d_m,k,t represents the reference distance from node m to base station k. The calculation formula for the reference distance is as follows:

$$d_{m,k,t}=\sqrt{{\left(x_{m,t}-x_{k,t}\right)}^2+{\left(y_{m,t}-y_{k,t}\right)}^2+{\left(z_{m,t}-z_{k,t}\right)}^2}$$

(10)

Owing to the considerable distance between the nodes and base stations, $L(h_t,h_r,h_e,d_{m,k})$ employs a three-ray path loss model. The formula [21] is as follows:

$$L\left(h_t,h_r,h_e,d_t\right)=-10{\log }_{10}\left({\left({\displaystyle \frac{\lambda }{4\pi d_t}}\right)}^2{\left(2\left(1+\Delta \right)\right)}^2\right)$$

(11)

where h_t and h_r represent the effective heights of the transmitter (Tx) and receiver (Rx) antennas, respectively, whereas h_e denotes the effective height of the evaporation duct. The formula for $\Delta$ is as follows:

$$\Delta =2\sin \left({\displaystyle \frac{2\pi h_t{h}_r}{\lambda d_t}}\right)\sin \left({\displaystyle \frac{2\pi \left(h_e-h_t\right)\left(h_e-h_r\right)}{\lambda d_t}}\right)$$

(12)

where n is the path loss exponent, which is typically set to 1.1, owing to the surface waveguide effect of the sea surface [30]. In addition, ${\chi }_{\sigma }^{m,k}$ represents shadowing, and the higher the sea level, the greater the shadowing. To better adapt to the rapid movement of nodes, an adjustment parameter F is introduced. This term is set to −1 when far from the shore base and 1 otherwise.

Issues with the network access strategy and efficiency of business completion can be transformed into an optimization problem of access network parameters; that is, the better the network parameters of the strategy network, the more businesses will be completed. If the business access strategy is further optimized, the efficiency of business completion can be increased. The interactive factors among access network parameters are affected by the natural environment and random human disturbances, rendering it difficult to describe the dynamic change relationship directly with a mathematical expression and further form an access strategy. Deep reinforcement learning methods integrate the advantages of neural networks with learning. They not only fit the policy function through data but also explore and adapt to a dynamic environment with a lower training cost and a trial-and-error learning approach. Therefore, this study uses a deep reinforcement learning method to determine the optimal values of the network parameters in the constraint formula.

3. Resource Allocation Method Based on an Attention Mechanism

Considering the characteristics of maritime wireless networks, the following considerations were made to improve the efficiency of resource allocation methods:

(1) When the difference in network parameters between better and optimal links is insignificant, it is entirely reasonable and acceptable to lose some transmission benefits in exchange for the stability of the communication link connection.

(2) The resource allocation algorithm based on deep reinforcement learning, which requires a large amount of environmental information, must accurately explore the internal logic between the environment and nodes to provide action outputs that efficiently and accurately match the current state of the intelligent agent.

(3) The requirements of different services and the corresponding sensitive network parameters are all different, which means that the network parameters corresponding to the service requirements should be prioritized without necessarily seeking the optimal network parameters.

Considering the aforementioned considerations, this section proposes a resource allocation method based on the attention mechanism, known as the attention double DQN (A-DDQN), from the following two perspectives: action output and environmental feature extraction. This method is composed of three modular components, namely, the DDQN algorithm module, the attention–perception module, and the selection suppression module, all of which are designed to enhance the efficiency of network transmission.

3.1. Basic Principle

A flowchart of the entire method is shown in Figure 2. The method is divided into three modules. The first module is the deep reinforcement learning module, which introduces the DDQN algorithm [31] and has high learning efficiency and smoother convergence. The second module is the attention–perception module. The environmental output state values are first transmitted to the attention–perception module to obtain the selection weights for the current resource pool business types. It executes the business selection action of $a_t^j$ and merges the intelligent agent’s business list information with the environmental state values into the Q-network and then outputs the action. The third module is the selection suppression module. The actions and their action value outputs from the Q-network are first checked using the selection suppression module. If the conditions are met, they are output directly; otherwise, the $a_t^i$ output from the previous moment replaces the current network output, outputting the network access selection action. The structure and function of each module are described in sequence below.

3.1.1. Action Suppression Module

Given that frequent switching of intelligent agents’ access choices can lead to transmission instability and information security issues, it is necessary to ensure the stability of the agent’s action output over a period of time. Therefore, this module introduces a method for setting the optimal action-value difference threshold at adjacent moments. For the action value outputted by the Q-network at time t, denoted by Q_t, and the action value outputted at the previous moment, denoted by Q_t−1, a sensitivity value Q_s is set as follows:

$$\left|Q_t-Q_{t-1}\right|\ge Q_s$$

(13)

As the output of the algorithm gradually stabilizes, the sensitivity value Q_s decayed over time to adapt to the output of the algorithm. Therefore, the decay rate is set to the sensitivity value Q_s, and the time decay coefficient is denoted as μ. By combining these two points, an action-value constraint function is obtained. This function enables the selection suppression unit to switch the link for the intelligent agent. If this condition is not satisfied, then the current connection action is maintained, as follows:

$$\left\{\begin{array}{c}\left|Q_t-Q_{t-1}\right|\ge Q_s\\ {\epsilon }_{s,t}={\epsilon }_{s,t}\cdot {\left(\mu \right)}^t\\ Q_s=Q_s\cdot {\epsilon }_{s,t}\\ Q_s\in \left[Q_{s\min },Q_s\cdot \epsilon \right]\end{array}\right.$$

(14)

3.1.2. Attention Perception Module

In resource allocation machine learning algorithms, where neural networks have the advantage of autonomously filtering useful information from an indifferent input of environmental data, the challenge lies in enabling neural networks to locate and process this useful information swiftly to enhance learning efficiency. This represents a breakthrough in improving the performance of these algorithms. Given that heterogeneous network business requirements often feature local optimizations of network parameters sufficient for global optimization, the proposed method aims to select information more pertinent to current business needs from extensive environmental data. This enables the neural network to focus selectively on relevant environmental information, thereby enhancing the performance and generalizability of resource allocation algorithms. Consequently, attention mechanisms from computer vision were integrated to develop an attention-aware module tailored for wireless network resource allocation. The approach is detailed as follows:

(1) The attention–perception module first splits the “perception field” of the agent to distinguish between the self-state and environmental state, to perform targeted processing on different states in the future. The self-state includes the agent’s current connection network parameters and other state parameters, whereas the environmental state includes the network parameters.

(2) For the self-state information, a linear perceptron is used to map each state parameter according to its dimensions to obtain a sub-weight, calculate the correlations between the attributes of a note and environmental parameters, and normalize these correlations into attention weights. A flowchart is shown in Figure 3. The sub-weight of an agent at time t is represented as

$$\begin{array}{c}{s}_t^{self}\end{array}\begin{array}{c}=\mathrm{softmax}\left[\left(\begin{array}{c}{s}_1^s\\ s_2^s\\ \cdots \\ s_p^s\end{array}\right)·\left(\begin{array}{cccc}{\Gamma }_1& {\Gamma }_2& \cdots & {\Gamma }_p\end{array}\right)\right]\\ =\mathrm{softmax}\left(\begin{array}{cccc}{\Gamma }_1\left(s_1^s\right)& {\Gamma }_2\left(s_2^s\right)& \cdots & {\Gamma }_p\left(s_p^s\right)\end{array}\right)\end{array}$$

(15)

Then, all sub-weights are weighted to obtain the self-attention weight for the selection of business types under self-state awareness, as follows:

$${\Gamma }_p^t\left(s_p^s\right)=\left(\begin{array}{cccc}{w}_1^1& w_1^2& \cdots & w_1^p\\ w_2^1& w_2^2& \cdots & w_2^p\\ \cdots & \cdots & \cdots & \cdots \\ w_l^1& w_l^2& \cdots & w_l^p\end{array}\right)\to \left(\begin{array}{c}{w}_1^1\\ w_2^1\\ \cdots \\ w_l^1\end{array}\right)\otimes \left(\begin{array}{c}{w}_1^2\\ w_2^2\\ \cdots \\ w_l^2\end{array}\right)\otimes \cdots \otimes \left(\begin{array}{c}{w}_1^p\\ w_2^p\\ \cdots \\ w_l^p\end{array}\right)\to \left(\begin{array}{c}{w}_1^{s,t}\\ w_2^{s,t}\\ \cdots \\ w_l^{s,t}\end{array}\right)$$

(16)

(3) For the environmental state information, a simple linear perceptron cannot be used for processing because it contains a variety of spatial and temporal information. For instance, maritime scene location information that affects network transmission can be considered spatial information. Here, a multilayer perceptron is employed to calculate the similarity of parameters, extract implicit information from the state through multiple fully connected layers, and output the business type selection spatiotemporal attention weights under the environmental state. This is represented as follows:

$$s_t^{env}\otimes F\left(\cdot \right)=\left(\begin{array}{c}F\left(s_1^e,k\right)\\ F\left(s_2^e,k\right)\\ \cdots \\ F\left(s_{n-p}^e,k\right)\end{array}\right)=\left(\begin{array}{c}{v}_a^T\tanh \left(W_{s_1^e}+U_k\right)\\ v_a^T\tanh \left(W_{s_2^e}+U_k\right)\\ \cdots \\ v_a^T\tanh \left(W_{s_{n-p}^e}+U_k\right)\end{array}\right)\to \left(\begin{array}{c}{w}_1^{e,t}\\ w_2^{e,t}\\ \cdots \\ w_l^{e,t}\end{array}\right)$$

(17)

(4) After obtaining the self-attention and spatiotemporal attention weights, the different impact levels of the two types of attention on the selection of business types are considered. Using self-attention as the baseline and adjusting for errors in spatiotemporal attention, the weights are multiplied and combined proportionally. Finally, this results in rescaled weights of the business-type features. These are called hybrid attention weights and are represented as follows:

$$w^{s,t}\otimes w^{e,t}=\left(\begin{array}{c}{w}_1^{s,t}\\ w_2^{s,t}\\ \cdots \\ w_l^{s,t}\end{array}\right)\otimes \left(\begin{array}{c}{w}_1^{e,t}\\ w_2^{e,t}\\ \cdots \\ w_l^{e,t}\end{array}\right)=\left(\begin{array}{c}{w}_1^{t,j}\\ w_2^{t,j}\\ \cdots \\ w_l^{t,j}\end{array}\right)$$

(18)

The structure of the attention–perception method is shown in Figure 4.

3.1.3. Deep Reinforcement Learning Module

Given that intelligent agents operate in highly dynamic environments with limited communication and computing resources, they necessitate deep reinforcement learning algorithms with high learning efficiency and low computational costs. Consequently, the proposed method incorporates the DDQN algorithm to develop a deep reinforcement learning module. Compared with other intricate deep reinforcement learning algorithms such as deep deterministic policy gradient (DDPG) and soft actor–critic (SAC), this approach maintains a simpler implementation and lower computational costs, resulting in higher learning efficiency and smoother convergence.

The process of agent action selection involves maximizing the reward function. In this context, the agent’s primary task is to maximize the value of a completed business; hence, the main reward is the business completion reward. Given the delayed nature of business completion rewards, to prevent sparse rewards, an auxiliary function is introduced to guide the agent’s behavior. A reward function based on dynamic potential energy is established, which disintegrates the primary task into subtasks to accelerate the algorithm’s convergence speed and enhance performance. The reward function is expressed as follows:

$$\left\{\begin{array}{c}r\left(s,a,a\prime \right)={\displaystyle \sum _{i\le N_{\max }}{r}_i^b}+\overline{r}\left(p,\varphi ,a,a\prime \right)\\ \overline{r}\left(p,\varphi ,a,a\prime \right)=r\left(p,\varphi ,a,a\prime \right)+\gamma \varphi \left(a\prime \right)-\varphi \left(a\right)\end{array}\right..$$

(19)

where ${\displaystyle \sum _{i\le N_{\max }}{r}_i^b}$ is the mainline reward, which is rewarded based on the business completed by the intelligent agent at the current time; $\overline{r}\left(p,\varphi ,a,a\prime \right)$ is an auxiliary reward function that rewards or punishes based on the network parameters of the agent’s previous access to the network.

3.2. Algorithmic Process

The pseudocode of the A-DDQN algorithm is presented in the following Algorithm 1:

Algorithm 1. Attention Double DQN Algorithm

Initialize the experience replay buffer D with capacity N; initialize the maximum size of the experience replay pool to Nr;

Initialize the action-value function Q with random weights θ; initialize the training batch size to Nb;

Initialize the target action-value function $\widehat{Q}$ with weights θ− = θ; initialize the target network update frequency to N−.

Initialize the greediness probability ε and the action-value sensitivity Qs.

For episode = 1, M do

Initialize action sequence ε = {x1} and initial state sequence ϕ1 = ϕ(s1).

For t = 1, T do

If the probability of greediness ε is less than the threshold value

Choose a random action $a_t^i$ with probability ε;

Else

Select an action $a_t^i$ using the Q-network model with a probability of less than Qs.

The greediness probability ε decays, the action-value sensitivity Qs decays, and the state εs decays with a time decay factor μ.

Given the state $S_t$, choose the action $a_t^j$, and update the Q-network with the state input $\left(s^{t+1},r^{t+1}\right)\to \left(s^{t+1},r^{t+1},b_m^{a_t^j}\right)$

$\mathrm{Execute} \mathrm{the} \mathrm{action}{a}_t^i$$\mathrm{and} a_t^j$ to obtain the reward value rt+1 and state st+1.

$\mathrm{Store} \mathrm{the} \mathrm{quadruple} (s_t, a_t^i$, rt+1, st+1) into the experience replay pool D.

$\mathrm{Sample} \mathrm{a} \mathrm{quadruple} (s_t, a_t^i$, rt, st+1) from the experience replay pool D with a minimum batch size.

Calculate the target value of Nb for each training batch:

$\mathrm{Define} a^{\max }\left(s\prime ;\theta \right)=\arg {\max }_{a\prime }Q\left(s\prime ,a\prime ;\theta \right)$.

$\mathrm{Calculate} y_i=\left\{\begin{array}{cc}{r}_j& \mathrm{If} \mathrm{the} \mathrm{training} \mathrm{round} \mathrm{stops} \mathrm{at} \mathrm{j}+1\\ r_j+\gamma \widehat{Q}(s\prime ,a^{\max }\left(s\prime ,\theta \right);{\theta }^{-})& other\end{array}\right.$.

$\mathrm{Calculate} \mathrm{the} \mathrm{gradient} \mathrm{descent} {‖y_j-Q(s,a;\theta )‖}^2$.

$\mathrm{Update} \mathrm{the} \mathrm{target} \mathrm{network} \mathrm{parameters}. {\theta }^{-}\leftarrow \theta$$\mathrm{every} N^{-}$ steps.

End For

End For

3.3. Algorithmic Complexity

When the input size of the algorithm is n, the algorithm complexity of the action suppression module is O (1), the algorithm complexity of the attention–perception method is O (1), and the algorithm complexity of the intelligent agent neural network part is O (TNN), where

$$T_{NN}={\displaystyle \sum _{i=1}^L{n}_i\cdot n_{i+1}}.$$

(20)

Therefore, the overall time complexity of the algorithm is O (TNN), which is not significantly different from the complexity of other deep learning algorithms. Furthermore, the neural network used by the intelligent agent consists of four fully connected layers, with a node count of 300/200/60/24 in each layer, resulting in a computational complexity of approximately 73,440 for the entire algorithm.

3.4. Algorithmic Convergence

The convergence of the DDQN algorithm is based on the convergence of Q-learning, and the overestimation of the Q value is reduced through the dual Q-learning method, further enhancing the stability and convergence of the algorithm. The proposed method does not change the basic properties of the Q-learning algorithm; therefore, its convergence is consistent with the Q-learning algorithm. The convergence proof can be found in reference [32].

4. Simulation Results and Analysis

To evaluate the performance of the proposed method in a maritime UAV network scenario, the proposed algorithm was simulated along with the other control algorithms. This section first introduces the simulation scenario and its parameter settings and then analyzes and discusses the simulation results.

4.1. Simulation Parameter Setup

4.1.1. Intelligent Agent Parameter Settings

In the simulation scenario, in addition to intelligent agents, other greedy nodes with tasks that are consistent with those of the agents exist. The order in which all nodes take business from the resource pool is sorted in real time, according to a uniform distribution. The schematic of the simulation scene is shown in Figure 5. The algorithm performance test scenario was random, and the randomness was reflected in the continuous random high-speed movement of nodes, continuous random low-speed movement of base stations, random generation of business attribute values in the resource pool, and random interference of network parameters. Traditional methods and greedy algorithms that do not require training are directly applied to the test scenarios. Owing to the strong randomness of the scenario, the following performance indicators were used as average values per 20 rounds. For the basic parameters of the scenario, refer to [33], for maritime channel parameters, refer to [30], and for link parameters, refer to [34]. Please refer to

Table 2. Simulation parameter settings.

Simulation Parameter Name	Simulation Parameter
Number of intelligent agents	1
Number of high-speed nodes	13
Number of base station nodes	6
High-speed node speed (m/s)	[15, 150]
Base station node speed (m/s)	[1, 15]
Node turning angle	[−0.25π, 0.25π]
Run time (s)	600
Model training times	2000

for the detailed parameter settings.

Each base station was set up to provide four types of networks to user nodes with the basic attributes of the links, as listed in

Table 3. Comparison of maritime base station communication link parameters.

Category	Coverage (km)	Transmission Power	Bandwidth	Outage Probability
Link 1	80	200	8	0.3
Link 2	50	120	15	0.25
Link 3	25	30	10	0.2
Link 4	300	105	20	0.35

.

4.1.2. Network–User Interaction Setup

The various links of each base station in the scenario are regarded as heterogeneous wireless communication networks. User QoS is mainly influenced by the following four network parameter indicators: packet delay, packet jitter, bandwidth, and packet loss rate. In this study, we assume that base stations can meet users’ communication network switching and access requests with negligible switching time and cost. Under the joint action of the four basic network parameters and random environmental factors (weather, Doppler shift, etc.), the business transmission rate and bandwidth of the users vary to varying degrees. This closed-loop mutual influence constitutes the interaction between the network and the user, as shown in the interaction diagram in Figure 6.

4.1.3. Transmission Business Setup

Base stations generate businesses with different demands according to a Poisson distribution and place them in the resource pool for nodes to access. The characteristics of these business types are presented in

Table 4. Comparison of service business characteristics.

Business Type	Business	Average Bandwidth/MHz	Mean Duration/s	Proportion/%
Ordinary business (conversation type)	Real-time location, maritime management, planned route information, etc.	3	8	33
Maritime safety business (interactive type)	Maritime distress and rescue information, emergency support information, etc.	12	16	34
Remote sensing business (stream type)	Hydrological information, sensor raw signals, maps, etc.	50	24	33

. Nodes randomly access businesses from the resource pool of the base station and place them in their business lists for execution, with the maximum number of parallel executions set at 10. We assume that the maximum number is exceeded or the node’s bandwidth is insufficient. In this case, they will not access more businesses until there is a vacancy in the business list and the bandwidth is free. Considering business timeliness, a business execution failure mechanism is introduced. That is, if the running time of the current business exceeds 1.2 times the duration, it is considered a business execution failure. The base station does not accept failed businesses, which are directly removed from the business list and penalized.

Businesses are distinguished by priority levels, with Level 1 being the highest priority and Level 3 being the lowest. High-priority businesses offer greater rewards for completion and penalties for non-completion. The proportion of businesses from high to low priority is [0.35, 0.5, 0.15].

Table 5. Comparison of service business rewards/penalties.

Business Completion Reward/Incompletion Penalty	Level 1	Level 2	Level 3
General business (conversation type)	9/−6	5/−3	3/−2
Maritime safety business (interactive type)	10/−7	6/−4	4/−3
Remote sensing business (stream type)	12/−8	8/−5	6/−4

compares the reward and penalty situations for different types of businesses.

The value of a business is defined based on its priority level and type. The completion value of a business is determined by multiplying its instantaneous value by its duration, as listed in

Table 6. Comparison of business instantaneous value.

Business Instantaneous Value	Level 1	Level 2	Level 3
General business (conversation type)	4	3	2
Maritime safety business (interactive type)	5	4	3
Remote sensing business (stream type)	6	5	4

.

4.1.4. Control Algorithms and Hyperparameters

To verify the performance of the proposed method, it was compared with those of the DDPG [35], SAC [36], and DDQN [31] deep reinforcement learning algorithms; the traditional D-AHP method [37]; and a greedy algorithm. In addition, ablation comparison experiments were conducted to verify the performance of the two modules. The SAC_ls, DDPG_ls, and DDQN_ls algorithms have a selection suppression unit added to their original versions to control the number of switches within 10% of the total duration. The D-AHP algorithm is an improved version of the traditional AHP method and is adapted for dynamic maritime wireless network environments. The greedy algorithm refers to an access selection strategy based on the closest distance to the business selection in random order. The hyperparameters of the proposed algorithm are listed in

Table 7. A-DDQN algorithm parameters.

Parameter	Numerical Value
Learning rate	1 × 10−4
Number of network layers	4
Number of neurons	300/200/60/24
Sampling time	1
Discount factor	0.95
Batch sample size	600
Experience buffer length	1.2 × 105
Smoothing factor	1 × 10−5
Target network update frequency	5

.

4.2. Analysis of Simulation Results

This section compares and analyzes the performances of the proposed method and control algorithms in terms of the following five aspects: the total number of switches, the quantity and value of business completion, average network parameters, the quantity of business priority completion, and the quantity of business type completion.

4.2.1. Total Number of Switches

Figure 7 shows a comparison of the total number of switches per algorithm round. The statistical standard for the number of switches is defined as follows. If the access selection strategy is different at times t and t+1, then the number of switches is incremented by one, accumulating the total number of switches in a complete round. As shown in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7, the inclusion of suppression units yields positive outcomes. The algorithm introduced exhibits a 69.23% reduction in switching compared with the original DDQN algorithm. Although this is slightly higher than other deep learning algorithms subject to switch frequency constraints, the switch count constitutes only 10% of the total action time, which is deemed reasonable and acceptable. When compared with the standard algorithm, the algorithm with selection suppression units maintains a switch count of less than 10% of the total action time. This robustly demonstrates the universal applicability of suppression units in effectively managing frequent algorithm switches. To fit a real situation, the subsequent comparison algorithms of the proposed algorithm all target deep learning algorithms with the switch count constrained to within 10% of the total action time, namely, the DDPG_ls, SAC_ls, and A-DDQN_ls algorithms.

4.2.2. Business Completion Quantity and Value

Figure 8a presents the total number of businesses completed per round using each algorithm. The proposed algorithm increased the total completion amount by approximately 6.4% and 11.73% compared with the DDQN_ls and DDPG_ls algorithms, respectively. Additionally, the results of the ablation study indicated that the attention–perception module of the A-DDQN algorithm increased the number of business completions by 22.7%. This demonstrates that the attention-aware module of the proposed method can effectively enhance the number of completed tasks even under constrained action-switching conditions. This is because the attention-aware module better aligns with node status and business requirements, ensuring a smooth node business queue. Combining Figure 7 and Figure 8a shows that after constraining the number of switches, the SAC algorithm’s business completion quantity decreased by 11.37% and that of the DDQN algorithm decreased by 13.53%. Although the selection suppression unit can effectively suppress the number of switches, it sacrifices some optimal actions at the cost of reducing the business transmission efficiency.

Figure 8b presents the total number of unfinished businesses per round for each algorithm. Notably, the proposed algorithm reduced the total amount of overdue business by 24.56% compared with the DDQN_ls algorithm and by 4.4% compared with the SAC_ls algorithm. This occurs because all three algorithms experience a business transmission efficiency below the normal transmission standards during certain periods. However, the proposed method matches node status with the traffic, and when transmission efficiency is low, it matches with low-duration traffic, thereby mitigating some losses.

Figure 8c illustrates the total value of the completed business. As the time for the agent to execute a business is limited, the total value of the business completed per round has an upper limit. The total value of the business completed by the proposed algorithm increased by 13.72% and 6.57% compared with those of the other two algorithms. This demonstrates that the attention–perception module plays a role in adapting to network parameters and selecting business types. The module fully utilizes its connected network to maximize transmission efficiency. In other words, when selecting non-highest-priority businesses, it adopts a strategy of choosing short-duration and low-value businesses under low network parameters and long-duration and high-value businesses under high network parameters.

4.2.3. Average Network Parameters

Figure 9 presents a comparison chart of the average network parameters per round for the intelligent agents. The average network parameter refers to the mean value of each network parameter per round and reflects the quality of the algorithm’s access strategy within that round. In this scenario, the packet delay, packet loss, and transmission rate are considered to have better transmission efficiency ranges of (0, 70], (10, 45], and (1, 2], respectively. Figure 10 shows that the three network parameters of the proposed algorithm are within the better transmission efficiency range for 80% of the rounds. Compared with the other comparison methods, the proposed method has an increase of approximately 8%–15%, which means that the intelligent agent can complete more business in resource-constrained environments and better adapt to strong dynamic environments.

4.2.4. Completion Quantity of Business Priorities

Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 present a comparative analysis of the completion rates across various business priorities. The proposed algorithm outperforms the DDQNls and SACls algorithms, achieving an improvement of 6.1% and 11.73%, respectively. This demonstrates that our approach prioritizes the completion of high-value business tasks. As shown in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8, in contrast to the other algorithms, our proposed method facilitates a positive cycle; it completes more tasks, updates the list more frequently, and achieves a greater total value of completed tasks while eliminating low-value tasks.

4.2.5. Quantity Completed by Business Type

Figure 11a presents a comparison chart of the completion quantity for Type I businesses. The proposed algorithm increased the completion quantity by 8.7% and 14.83% compared with the DDQN_ls and SAC_ls algorithms, respectively. As shown in Figure 9, the strategy for the intelligent agent to select a business is to choose as many conversational and interactive businesses as possible; it will select a small number of high-bandwidth stream businesses for execution only when the network parameters are sufficiently good, which is in line with the basic strategy of maximizing the value of business completion. In contrast to other algorithms that can only find a single optimal path, the SAC algorithm learns various solutions to problems through maximum entropy, which is more robust against dynamic environments. However, it does not distinguish the adaptability between different businesses and nodes. Algorithms based on traditional methods have low transmission efficiency and slow business list updates. Therefore, when they access businesses from the resource pool, most of the businesses in the pool are those left unchosen by other nodes in the past. Thus, more interactive and stream businesses are selected. The proposed algorithm perceives and adapts the compatibility between nodes and businesses through the attention–perception module. That is, it perceives its own and environmental states in real time and adjusts the action strategy to determine the optimal solution for each state.

In summary, although the proposed algorithm did not have the highest number of optimal performance metrics, its overall performance was the best. The proposed algorithm is applicable to maritime wireless network application scenarios in which business completion is the core performance indicator. While suppressing the ping-pong effect of access selection, it enhances the total quantity and value of business completion. A comparison of the performance parameters of the algorithms is presented in

Table 8. Comparison of algorithm performance.

	Total Business Value	Quantity Completed	Quantity Unfinished	Level 1	Level 2	Type 1	Type 2	Switching Frequency
Greedy	6780	185	136	57	87	143	34	449
D-AHP [27]	7370	224	123	70	108	202	15	448
DDQN	18533	377	45	349	23	356	14	278
SAC	20133	404	40	378	22	387	13	557
DDPG_ls [25]	13210	290	83	279	25	275	12	37
DDQN_ls [22]	15593	326	61	325	18	325	15	71
SAC_ls [26]	17367	358	57	326	23	337	13	66
A-DDQN	19750	400	43	373	30	387	18	79

Bold font represents the method proposed in this article.

.

4.2.6. Limitations of the Method

The proposed method primarily targets the highly dynamic, resource-constrained environment of maritime wireless communications, addressing issues such as frequent agent switching and low service completion rates. Consequently, this method relies on or is sensitive to certain environmental assumptions, such as the following:

(1) With the aim of simulation verification, to ensure the intelligent agent can acquire a greater number of services within a restricted timeframe, the average service duration and the proportion of services with a longer duration are set at relatively lower values as environment-sensitive parameters.

(2) With an emphasis on nodes completing as many operations as possible, the incentives for completing low-duration operations and the penalties for noncompletion are set relatively high.

5. Conclusions

This study addresses wireless resource allocation within the framework of offshore wireless networks by jointly optimizing service-access methods and network-entry strategies to provide new insights into wireless network resource management. To address the issues of unreasonable node switching frequencies and low service completion rates, this study proposes a resource allocation method based on an attention mechanism. The proposed method employs an action-value constraint function to mitigate the ping-pong effect in agent network access decisions and incorporates an attention mechanism to generate mixed attention weights that optimize service-access strategies and enhance the alignment among nodes, networks, and services. Simulation results demonstrate that the proposed method significantly outperforms the benchmark method, further validating the effectiveness of attention mechanisms in multidimensional resource allocation for wireless networks. These findings not only extend the application of existing attention mechanisms but also open new avenues for wireless network resource management. In the future, we will focus on the following: 1. Expanding the channel model to include land and air scenarios. 2. Collecting and validating real-world wireless communication network data at sea. 3. Extending the method to multiagent systems to further enhance the algorithm’s generalization ability.