A Throughput Performance Analysis Method for Multimode Underwater Acoustic Communication Network Based on Markov Decision Process

Abstract

The multimode underwater acoustic communication network is a novel form of underwater acoustic communication that adjusts its communication mode to enhance overall performance. Current performance analysis methods are primarily applied to single-mode networks and assume uniform communication capability across all nodes, making them unsuitable for multimode networks. This paper investigates the multimode communication of the physical layer, considering factors such as the marine environment, the node transmitting sound source level, and the transmitting distance. A decoding conflict model is proposed to support multimode concurrent transmission scenarios. The communication mode is designed to be compatible with the channel and node characteristics. Additionally, using a Markov decision process, this paper establishes a performance evaluation and analysis model for multimode underwater acoustic networks to determine throughput performance limits in real underwater environments. Simulations across various scenarios validate that the throughput performance limits obtained by this method are more accurate under multimode networks, with an improvement in accuracy of over 67.5% compared to existing methods.

1. Introduction

Underwater acoustic communication networks [1,2,3] can facilitate the exchange of information among multiple types of observation equipment in diverse underwater locations. This technology has been extensively applied in marine resource exploitation, marine environment monitoring, and related fields [4,5,6]. As a result, this topic has emerged as a significant area of research.

After extensive development, underwater acoustic communication technology has evolved to include a wide range of communication modes [7,8,9,10,11,12], including single-carrier coherent phase shift keying (PSK), noncoherent multifrequency shift keying (MFSK), and random spread spectrum (PRSS). Each mode offers distinct advantages in terms of communication capability, reliability, and invisibility. However, the current underwater acoustic communication network typically relies on a single mode, which may not effectively adapt to complex and dynamic marine environments or the increasing diversity of applications. By dynamically selecting the optimal communication mode [13,14,15], an underwater network can maximize efficiency in various environments and application requirements. In a previous paper [14], a multimode underwater acoustic network was introduced, although it only demonstrated its capability for multimode transmission without analyzing its impact on network performance. In another paper [15], Liu proposed a multimode network in which the communication regime is selected based on the channel delay and SNR. However, the choice of the communication regime is mainly determined by the marine environment and the point-to-point receiving SNR, focusing solely on the node’s perspective rather than considering a multimode optimization scheme that maximizes the network’s overall performance. Throughput serves as a crucial metric for evaluating the transmission performance of an underwater acoustic network. Therefore, this paper aims to analyze the throughput performance of a multimode underwater acoustic communication network and determine the optimal multimode solution.

The extended propagation delay of the underwater acoustic channel gives rise to an ambiguous channel state (busy/idle) and uncertain signal arrival time. This complexity creates challenges when designing traditional protocols for underwater acoustic communication networks and significantly reduces network throughput performance. However, recent research [15,16,17,18] has presented an alternative viewpoint. Chitre [16] proposed a novel method for analyzing throughput in networks with large propagation delays, which deviates from the traditional approach of suppressing such delays. Instead, it utilizes propagation delay as an advantage to achieve higher network throughput. Building upon this approach, Said [17] proposed a method for analyzing the upper limit of throughput performance in linear underwater acoustic networks. In [18], the authors focused on the physical layer and examined the throughput performance in a time-slotted UASN with guard time. Kan [19] analyzed the throughput performance of network nodes with different packet sizes. The study of network performance heavily relies on the state of the physical layer transmission. However, these studies have oversimplified the physical layer of underwater acoustic communication by assuming that each node possesses identical transmission capability without accounting for the variations in channel characteristics and communication rates across different links. Furthermore, refs. [15,16,17,18] utilized a traditional interference conflict model that assumes that conflicting signals cannot be successfully decoded when they overlap. In reality, multiple communication signals can overlap and be decoded successfully under certain conditions. This inconsistency with real-world scenarios leads to inaccurate limitations on network throughput performance.

Scholars have studied the influence of underwater acoustic channels on network transmission. Zhong [20] conducted research on the difference between the horizontal and vertical propagation of underwater acoustic channels and proposed a method for analyzing the throughput performance based on this differentiation. However, this research failed to acknowledge that the discrepancy between horizontal and vertical propagation is caused by the channel arrival structure and did not thoroughly analyze the channel structure and physical layer communication. On the other hand, Nan [21] and Hua [22] utilized the signal-to-noise ratio to assess the successful reception of physical layer underwater acoustic communication signals and designed an MAC protocol. Nevertheless, in an underwater acoustic environment, channel characteristics such as the signal Doppler shift and multipath expansion structure significantly impact the transmission status of the physical layer. Currently, the propagation characteristics of ocean channels are not considered in underwater acoustic networks. This can lead to inadequate descriptions of interference conflicts, ultimately affecting the performance analysis and protocol design of underwater acoustic networks.

Building upon the aforementioned research, this paper initially analyses the influence of underwater acoustic channels on the bit error rate and packet loss rate of underwater acoustic communication. It develops a decoding conflict model that adheres to the transmission laws of underwater acoustic channels, thus accurately modeling conflicts in underwater acoustic networks. Subsequently, by taking into account physical layer characteristics such as dynamic channels and multimode transmission, the transmission of the network is described using the decoding conflict model. By integrating the Markov decision process, an analysis model for achieving high-precision performance in multimode underwater acoustic networks is formulated. Additionally, a performance analysis method for multimode underwater acoustic networks is proposed. These efforts lay a theoretical foundation for underwater acoustic network design and subsequent protocol research.

The rest of the paper is organized as follows. The system model is introduced in Section 2. In Section 3, the Markov decision process modeling of throughput performance limits of multimode underwater acoustic networks is presented. Additionally, a solution method based on dynamic programming and greedy algorithms is provided. Simulation results are presented and discussed in Section 4 and conclusions are drawn in Section 5.

2. System Model and Assumptions

In [15,16,17,18,19], the authors proposed a throughput performance analysis method based on the traditional interference conflict model. Based on this, this paper proposes a more accurate interference conflict model and throughput analysis method, which are briefly introduced below.

2.1. Integer Delay Matrices and Traditional Network Interference Conflict Model

In a linear acoustic communication network with nonzero propagation delay, the propagation delay between nodes i and j is defined as $D_{ij}^{\prime }$, where ∀$i\ne j$, 1 ≤ i ≤ N. Assume that the channel is symmetric, i.e., $D_{ij}^{\prime }=D_{ji}^{\prime }$. The geometric relationship between network nodes is represented by the delay matrix $D^{\prime }$, which is defined by using a standard unit time slot length.

$$D_{ij}^{\prime }=\frac{del\_Bellhop(w_i,w_j,env)}{\tau }$$

(1)

where $w_i$ represents the spatial location of node i in three-dimensional space; $env$ represents the current network environment parameters, primarily including the sound velocity profile and submarine type; $del\_Bellhop(·,·,·)$ represents the propagation delay calculated by the Bellhop model under the current environment and network node location; and τ represents the length of a time slot. It is important to note that the elements of the delay matrix are propagation delays between links in units of time slot length τ and can be rational numbers, i.e., D can be a noninteger delay matrix. However, with an appropriate choice of time slot length τ, the given noninteger delay matrix can be approximated by an integer delay matrix D as:

$$D_{ij}=⌊\frac{del\_Bellhop(w_i,w_j,env)}{\tau }⌋$$

(2)

where $⌊a⌋$ is the integer nearest to a and less than $a$.

The conventional model of network interference collision proposes that when the timing of two packets reaching the same receiver overlaps, the receiving node is unable to correctly receive any packet.

The network transmission mentioned in this paper is unicast propagation, which means that a sending node corresponds to only one receiving node, and the length of the message $\mu$ is equal to the slot length τ. Assuming normalization of the distance between adjacent nodes $D_{j,j+1}$ = 1 as the standard, and normalization of node distance for communication to RC = 1 and node interference distance to RI = k, this assumption implies that each node can only communicate with its neighbors and that nodes within a distance of k from the sending node will experience interference in reception.

2.2. Schedules and Network Throughput

The slot scheduling matrix for the network with N nodes and a period of T is denoted as S_NT. The element $s_{j,t}$ in the matrix $S_{N\times T}$ represents the state of node j at time slot t. In a half-duplex transceiver configuration for unicast transmission mode, if $s_{j,t}$ = i > 0, it indicates that node j sends data to node i in time slot t; if $s_{j,t}$ = −i < 0, it means that node j receives data from node i at time slot t; and if $S_{j,t}$ = 0, it signifies that node j is silent during time slot t.

Assume the communication distance of the network is denoted as RC = $k_1$ and the interference distance is denoted as RI = $k_2$. If node j transmits information to node i in slot t, and node i receives it in slot D_ij, the conditions for receiving the signal from node j are as follows:

$$s_{j,t}=i\iff s_{i,t+D_{ji}}=-j,D_{ji}<k_1$$

(3)

To ensure successful reception of the signal from node j, it is essential that no signals from other nodes reach node i within the same time slot, as indicated in Formula (4).

$$s_{i,t}=-j\Rightarrow s_{l,t-D_{lt}}\le 0,D_{lt}<k_2,l\ne i$$

(4)

Assume that the transmission rate of the link between nodes i and j is $v$ bps, and the amount of information transmitted in one time slot is $v\mu$ bits. The corresponding block transmission delay μ equals the unit time slot length τ. The average network throughput Y is defined as the total amount of successfully received information by all nodes in the network within a unit of time, normalized by the link rate. The average throughput Y of the scheduling matrix S_NT with period T is represented by the equation

$$Y=\frac{1}{T}{\displaystyle \sum _t{\displaystyle \sum _{j}1\left(s_{j,t}<0\right)}}$$

(5)

where $1\left(·\right)$ is a truth function, where a value of 1 is true and 0 is false. Considering the physical layer link transmission rate, the final network throughput is $Y_f=v\mu *Y$.

In [16], the authors show that the normalized throughput of an N node unicast network does not exceed N/2 when the propagation delay is non-zero.

The following two examples illustrate the fundamental principle of time slot scheduling in the context of long propagation delay. We consider a 3-node equilateral triangle network with nodes positioned at the vertices of equilateral triangles, where the normalized communication distance and interference distance are denoted as RC = 1 and RI = 1, respectively. The transfer time for a message block is given by μ = τ. The delay matrix D and scheduling matrix S for this network are as follows:

$$D=\left[\begin{array}{l}0\hspace{1em}1\hspace{1em}1\\ 1\hspace{1em}0\hspace{1em}1\\ 1\hspace{1em}1\hspace{1em}0\end{array}\right]\hspace{1em}{S}_{3\times 4}=\left[\begin{array}{l}2\hspace{1em}3\hspace{1em}-3\hspace{1em}-2\\ -3\hspace{1em}-1\hspace{1em}1\hspace{1em}3\\ -2\hspace{1em}1\hspace{1em}-1\hspace{1em}2\end{array}\right]$$

(6)

For the three-node network with an equilateral triangle topology, three nodes can successfully transmit 6 units of information in the T = 4τ timeslot. The network throughput is Y = 6ντ/4τ = 1.5ν. When the propagation delay factor is not considered, the three nodes complete data transmission in three-time slots respectively, and the throughput is Y = 1ν. Therefore, the throughput of the three-node network with non-zero propagation delay is 50% higher than that of the network with zero propagation delay.

For the three-node network with an isosceles triangle topology, the delay matrix and scheduling matrix of the network are expressed as

$$D=\left[\begin{array}{l}0\hspace{1em}1\hspace{1em}2\\ 1\hspace{1em}0\hspace{1em}2\\ 2\hspace{1em}2\hspace{1em}0\end{array}\right]\hspace{1em}{S}_{3\times 8}=\left[\begin{array}{l}2\hspace{1em}-2\hspace{1em}-3\hspace{1em}3\hspace{1em}-3\hspace{1em}3\hspace{1em} 2\hspace{1em}-2\\ 1\hspace{1em}-1\hspace{1em} 3\hspace{1em}-3\hspace{1em}3\hspace{1em}-3\hspace{1em} 1\hspace{1em}-1\\ 1\hspace{1em} 2\hspace{1em} 1\hspace{1em} 2\hspace{1em}-2\hspace{1em}-1 -2 -1\end{array}\right]$$

(7)

Three nodes can successfully transmit 12 units of information in the T = 8τ time slot. The network throughput is Y = 12ντ/8τ = 1.5ν.

2.3. Decoding Conflict Model and Multimode Underwater Acoustic Communication

The conventional model for network interference collisions is derived from terrestrial wireless networks and does not consider the characteristics of multipath propagation in underwater acoustic channels. As a result, it may provide an inaccurate description of interference conflicts. This paper presents a conflict decoding model designed to align with the propagation characteristics of underwater acoustic channels to achieve precise modeling of conflicts within underwater networks. The decoding conflict model is based on the bit error rate of receiving node i, which determines whether receiving node j can successfully receive data packets. If the bit error rate of receiving node i is below a certain threshold, then node j can successfully receive data packets from node j; otherwise, node j cannot receive data successfully. It can be expressed by the following formula:

$$\begin{array}{l}b(w_i,w_j,d_{ij},SINR)\ge be{r}_{th}\\ SINR=\frac{P_s}{P_I+Noise}\end{array}$$

(8)

where $SINR$ represents the received signal-to-interference plus noise ratio at receiving node i, $P_s$ is the received signal power, $P_I$ is the interference signal power, $Noise$ is the noise power, $w_i$ and $w_j$ denote the positions of nodes i and j, and $d_{ij}$ stands for the Doppler between transmitting and receiving nodes. Thus, $b(w_i,w_j,d_{ij},SINR)$ is the bit error rate value obtained under the current condition, and $be{r}_{th}$ is the bit error rate requirement of the application layer.

The following formula describes the decoding conflict model through the received signal-to-interference plus noise ratio:

$$SIN{R}_{ij}\ge SN{R}_{th}\left(w_i,w_j,d_{ij},be{r}_{th},f\right)$$

(9)

where $f$ is the processing algorithm of the current communication mode, and $SN{R}_{th}\left(w_i,w_j,d_{ij},be{r}_{th},f_c\right)$ is the minimum received signal-to-interference plus noise ratio to achieve conflict-free transmission under the constraints of the current sending and receiving conditions and the transmission requirements of the application layer.

It is assumed that the network nodes all support three communication modes, namely, PSK, MFSK, and PRSS, and the communication rate is $v_{PSK}>v_{MFSK}>v_{PRSS}$.

3. Throughput Performance Limits of Multimode Underwater Acoustic Networks

In Section 2.2, the communication capability of each link is assumed to be equal, resulting in a normalized throughput upper limit of the network packet of N/2. However, these conclusions may be problematic when considering the varying data transmission capabilities between different network links. For instance, in the scheduling result of an isosceles triangle network described in Section 2.2, where an MFSK communication system is employed at the physical layer with a communication rate of 800 bps, the upper limit of network throughput reaches 1200 bps. However, because the distance between the nodes $l_3$ is relatively close and the propagation loss is small, it can adopt a high-rate communication mode, such as PSK, whose communication rate can reach 1800 bps. The propagation loss $l_1$ and $l_2$ is large, and the MFSK with a low communication rate can still be used to transmit data. At this time, the optimal scheduling result of the three-node network is given according to Formula (5), and its throughput limit is 1/2 × 00 + 1/2 × 800 + 1/2 × 1800 = 1700 bps. However, if only $l_3$ (node 1 and node 2) is used to transmit a PSK signal, its network throughput is 2/2 × 1800 = 1800 bps, which is greater than the maximum throughput of 1700 bps for the three-node network. For multimode networks, the traditional throughput upper limit analysis results have problems and are no longer applicable.

The Markov decision process [23,24,25] can be described by a quintuple: $\left(S,A,P,R,\gamma \right)$. The specific meanings are shown in

Table 1. Parameter meaning in the Markov decision process.

Name	Symbol	Significance
The set of all states of the system	$S$	The stable state of the system over some time
The set of all possible actions of the decision process	$A$	Actions taken by the system in the current state
System state transition probability	$P_a\left(s,s^{\prime }\right)$	The probability of moving to state $s^{\prime }$ by performing action $a\in A$ in the current state $s\in S$
Return function	$R_a\left(s,s^{\prime }\right)$	The reward function or reward function for transferring the system state from $s$ to $s^{\prime }$ under action $a$
Damping coefficient	$\gamma \in \left[0,1\right]$	The discounting or decay of the return function over time

.

The policy $\pi$ is a mapping function from state to action $\pi$: $S\to A$. That is, given the current state $s$, the next action $a=\pi \left(s\right)$ is determined according to the policy $\pi$. For every fixed strategy π, the value function $V_{\pi }\left(s\right)$ satisfies the Bellman equation:

$$\begin{array}{l}{V}_{\pi }\left(s\right)=E\left[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots |s_t=s\right]\\ V_{\pi }\left(s\right)={\displaystyle \sum _a\pi \left(a|s\right)\left(R\left(s,a\right)+\gamma {\displaystyle \sum _{s^{\prime }\in S}{P}_{s\pi \left(s\right)}\left(s^{\prime }\right)V_{\pi }\left(s^{\prime }\right)}\right)}\end{array}$$

(10)

The action value function can be defined considering the value effect of the action $a$ taken

$$\begin{array}{l}{q}_{\pi }\left(s,a\right)=E\left[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\dots |s_0=s,A_t=a\right]\\ q_{\pi }\left(s,a\right)=R\left(s\right)+\gamma {\displaystyle \sum _{s^{\prime }\in S}{P}_{s\pi \left(s\right)}\left(s^{\prime }\right)q_{\pi }\left(s^{\prime },a^{\prime }\right)}\end{array}$$

(11)

The purpose of solving the Markov decision process is to find an optimal action strategy π in the current state $s$, and the corresponding value function of the optimal action strategy is the largest.

In this paper, the throughput performance limit problem of multimode networks is modeled as a Markov decision process, and the greedy algorithm and dynamic programming algorithm are combined to iteratively solve the network throughput and optimal scheduling results. The following is a detailed introduction.

3.1. Modeling of Throughput Performance Limits in Multimode Networks

3.1.1. System State Modeling

In the previous introduction to the network node scheduling model, the states of nodes are denoted as $s_{j,t}$, and the time slot $t$ indicates the current time. In an $N$-node network, each node has $2N-1$ states, which include sending signals to other nodes (the number of states is $N-1$), receiving signals from other nodes (the number of states is $N-1$), and an idle state. The states of the sending and receiving nodes of each link correspond to each other, for example, in link $\left(i,j\right)$, node $i$ transmits signals that reach node $j$ through a propagation delay $D_{ij}$. This corresponding relationship can be utilized to reduce the number of node states and simplify the MDP process. Specifically, only the sending state and idle state are considered individual node states, while the receiving state of network nodes is jointly determined by both the sending state of source nodes and the propagation delay $D_{ij}$. The receiving status of the node is no longer considered, and the status of the node is reduced to $N$.

The propagation delay corresponding to the maximum transmission range in the network is denoted by $G$. The decision of time slot $t$ is related only to the node state of the time slot $t-G$ to $t-1$, so the network state $\widehat{S}$ can be determined by the network node state of the first $G$ moments. Therefore, the state can be modeled as a matrix of $N\times G$, where ${\widehat{s}}_{i,k}$ represents the state of the i-th node at slot t − G – 1 + k.

3.1.2. Action Modeling

Action modeling is similar to state modeling in that the number of states per node remains constant at $N$. Since state $\widehat{S}$ includes the time slot from $t-G$ to $t-1$, the signal transmitted during this period will be received in time slots $t$ to $t-1+G$, and action $A$ will directly impact each node’s ability to successfully receive. To evaluate the return function, network action $A$ must include the actions of all nodes in slots $t$ to $t-1+G$. Therefore, action $A$ can be represented as a matrix $N\times G$, where $a_{i,k}$ indicates the action of node $i$ in slot $t-1+k$. Each node has $2N-1$ actions, including $N-1$ sending actions and idle actions.

3.1.3. Probabilistic Modeling of System State Transition

The system state transition probability $P_a\left(\widehat{s},{\widehat{s}}^{\prime }\right)$ is a deterministic function. The next state $s^{\prime }$ is determined by the current state $s$ and action a.

3.1.4. Reward Function Modeling

The reward function $R_a\left(\widehat{s},{\widehat{s}}^{\prime }\right)$ is represented by the ability of the current state to successfully receive data in the future. Specifically, under the decoding conflict model, whether each node can successfully receive data is analyzed according to the current state $\widehat{s}$ and action $a$.

(1) When the node receives the signal in the sending state, the amount of data successfully received is 0.

(2) If the node does not receive the signal that the destination node is its own, the amount of data received at this time is 0.

(3) In other cases, the amount of successfully received data can be analyzed by the decoding conflict model.

It can be expressed by Formula (12):

$$\begin{array}{l}{r}_{ik}\left(\widehat{s},a_{ik}\right)=\left\{\begin{array}{l}0,\hspace{1em}{a}_{ik}\ne 0\\ 0,\hspace{1em}\exists js.t.\hspace{1em}{\widehat{s}}_{j,G+k-1-D_{ij}}\ne i,\hspace{1em}{a}_{ik}=0\\ f\left(w_j,w_i,d_{ji},SIN{R}_{ji}\right),\hspace{1em}\exists j\hspace{1em}s.t.\hspace{1em}{\widehat{s}}_{j,G+k-1-D_{ij}}=i,a_{ik}=0\end{array}\right.\\ f\left(w_j,w_i,d_{ji},SIN{R}_{ji}\right)=\left\{\begin{array}{l}{v}_1\times \tau ,\hspace{1em}if\hspace{1em}SIN{R}_{ij}\ge SN{R}_{th}\left(w_i,w_j,d_{ij},be{r}_{th},f_1\right)=SN{R}_{th1}and{\widehat{s}}_{j,G+k-1-D_{ij}}=i\\ \cdots \\ v_n\times \tau ,\hspace{1em}elseif\hspace{1em}SIN{R}_{ij}\ge SN{R}_{th}\left(w_i,w_j,d_{ij},be{r}_{th},f_n\right)=SN{R}_{thn}and{\widehat{s}}_{j,G+k-1-D_{ij}}=i\\ 0,\hspace{1em}others\end{array}\right.\\ SIN{R}_{ji}=\frac{P_i}{{\displaystyle \sum _{\begin{array}{l}m=1\\ m\ne i\end{array}}^{N-1}{P}_m}+Noise}\hspace{1em}\exists ms.t.{\widehat{s}}_{m,G+k-1-D_{mi}}\ne 0\end{array}$$

(12)

where ${\widehat{s}}_{j,G+k-D_{ij}}=i$ means that the signal sent by node j in slot $t+k-1-D_{ij}$ reaches node i in slot $t+k-1$, $n$ is the number of supported communication systems, $f_i$ is the processing algorithm of the i-th communication mode and $v_i$ is the rate of the i-th communication system. If ${\widehat{s}}_{m,G+k-D_{mi}}\ne 0$, it indicates that the signal sent by node $m$ in slot $t+k-1-D_{mi}$ reaches node i in slot $t+k-1$, which is the interference signal of node i, $P_m$ is the interference signal energy, and ${\displaystyle \sum _{\begin{array}{l}m=1\\ m\ne i\end{array}}^{N-1}{P}_m}$ is the total interference signal energy.

After that, the amount of data received by each node at each moment is added up as the reward. The transmission scale of the signal in the network is $G$, so it is necessary to consider the benefits of the time scale of $G$.

$$R\left(\widehat{s},a\right)={\displaystyle \sum _{i=1}^N{\displaystyle \sum _{k=1}^G{r}_{ik}\left(\widehat{s},a_{ik}\right)}}$$

(13)

The damping coefficient $\gamma$ is set to the constant 1.

3.2. The Method of Solving the Model

In this paper, we utilize a combination of dynamic programming and greedy algorithms to iteratively solve the Markov decision process, ultimately obtaining the throughput performance of a multimode underwater acoustic network. The key components of our approach include policy evaluation, policy improvement, and policy iteration, which are discussed in detail below.

3.2.1. Policy Evaluation—Estimating State Value

The Bellman equation of the state value function is shown in Equation (10). If according to strategy $\pi$, the value function of each state can be iterated continuously with the Bellman equation

$$V_{k+1}\left(\widehat{s}\right)={\displaystyle \sum _a\pi \left(a|\widehat{s}\right)\left(R\left(\widehat{s},a\right)+\gamma {\displaystyle \sum _{{\widehat{s}}^{\prime }\in \widehat{S}}{P}_{\widehat{s}\pi \left(\widehat{s}\right)}\left({\widehat{s}}^{\prime }\right)V_k\left({\widehat{s}}^{\prime }\right)}\right)}$$

(14)

where $V_{k+1}\left(\widehat{s}\right)$ is the value function of state ${\widehat{S}}_t=\widehat{s}$ after iteration $k+1$, and $V_k\left({\widehat{s}}^{\prime }\right)$ is the value function of state ${\widehat{S}}_{t+1}={\widehat{s}}^{\prime }$ after iteration $k$.

3.2.2. Policy Enhancement—Estimate Action Value

We use policy evaluation to estimate the state value. The current value function is already the optimal value function under the current strategy. However, the current state may not necessarily be the best, so it is necessary to further improve this strategy.

Let us assume that in state $\widehat{s}$, we perform a new action $a^{\prime }$ and run the original policy $\pi$ thereafter, calculating its action value as:

$$q_{\pi }\left(\widehat{s},a^{\prime }\right)=R\left(\widehat{s}\right)+\gamma {\displaystyle \sum _{{\widehat{s}}^{\prime }\in \widehat{S}}{P}_{\widehat{s}\pi \left(\widehat{s}\right)}\left({\widehat{s}}^{\prime }\right)V_{\pi }\left({\widehat{s}}^{\prime }\right)}$$

(15)

The behavior pattern formed by this new action $a^{\prime }$ is called a new strategy ${\pi }^{\prime }$. By comparing the quality of $\pi$ and π′, $V_{\pi }\left(\widehat{s}\right)$ and $V_{{\pi }^{\prime }}\left(\widehat{s}\right)$ are compared, from which the current optimal strategy is selected. This process is strategy promotion, as shown below:

$$\begin{array}{l}{\pi }^{\prime }\left(\widehat{s}\right)=\arg \underset{a}{\max }{q}_{\pi }\left(\widehat{s},a\right)\\ =\arg \underset{a}{\max }\left\{R\left(\widehat{s}\right)+\gamma {\displaystyle \sum _{{\widehat{s}}^{\prime }\in \widehat{S}}{P}_{\widehat{s}\pi \left(\widehat{s}\right)}\left({\widehat{s}}^{\prime }\right)V_{\pi }\left({\widehat{s}}^{\prime }\right)}\right\}\end{array}$$

(16)

3.2.3. Policy Iteration

When a strategy $\pi$ is provided, a value function $V_{\pi }\left(\widehat{s}\right)$ can be derived based on the strategy. Subsequently, a greedy strategy ${\pi }^{\prime }=greedy\left(v_{\pi }\right)$ can be obtained based on the value function. A new value function can then be obtained according to the new strategy ${\pi }^{\prime }$, leading to the generation of a new greedy strategy. This iterative cycle eventually yields the optimal value function ${\pi }_{*}$ and the optimal strategy $V_{{\pi }_{*}}$. The process of policy updating and improvement through cyclic iteration is referred to as policy iteration. The goal of policy iteration is to achieve convergence toward an optimal policy by iteratively calculating the value function. The specific process is outlined below:

$${\pi }_0\stackrel{E}{\to }{v}_{{\pi }_0}\stackrel{I}{\to }{\pi }_1\stackrel{E}{\to }{v}_{{\pi }_1}\stackrel{I}{\to }{\pi }_2\stackrel{E}{\to }\cdots \stackrel{I}{\to }{\pi }_{*}\stackrel{E}{\to }{v}_{{\pi }_{*}}$$

(17)

where $E$ indicates policy evaluation and $I$ indicates policy promotion. The convergence condition is that the optimal strategy obtained in the last two iterations is the same.

3.2.4. Algorithms for Performance Analysis

The throughput performance analysis algorithms of multiregime networks are shown as follows, including Algorithms 1 and 2. Algorithm 1 is used to calculate the optimal action value function, and Algorithm 2 is used to transform the results of Algorithm 1 into throughput performance limits and optimal scheduling schemes for multimode networks.

Algorithm 1 Multimode network throughput performance analysis algorithm

Input: Network node sound source level (SL), propagation loss (TL), delay matrix ($D$), background noise level (NL), communication rate ($v_i$) and decodable SNR threshold ($SN{R}_{thi}$) of multiple communication systems, time slot length τ Output: Optimal policy ${\pi }^{*}$ Procedure:   Set the value function $V_{\pi }\left(\widehat{s}\right)$ of the initial system state $\widehat{s}$ to 0, and the current policy $\pi$ to evenly distribute all actions, While true 1. According to the current state $\widehat{s}$, combined with the parameters of the sonar equation, SL, TL, etc., the SINR of each network node at different times of action $a$ is calculated 1.1 Combined with the reward function model corresponding to Formula (11), the profit value of each action $a$ under the current state $\widehat{s}$ is analyzed successively. 1.2 According to the strategy evaluation Formula (13), update the value function $V_{\pi }\left(\widehat{s}\right)$ of the current state $\widehat{s}$. 2. Repeat step 1 to evaluate the value function of the next state ${\widehat{s}}^{\prime }$ until the value function evaluation of all states is completed. 3. According to the policy promotion Formula (15), the new optimal strategy ${\pi }^{\prime }$ for all states is analyzed in turn. 4. If ${\pi }^{\prime }$ = $\pi$ is true, then 4.1 The optimal policy ${\pi }^{*}={\pi }^{\prime }$ is obtained and output. 4.2 Break. End if End while

Algorithm 2 Throughput and scheduling result analysis algorithm

Input: Optimal policy ${\pi }^{*}$, temporary scheduling matrix $C$, status number $u$ Output: Multimode network throughput performance limit $M^{*}$, optimal scheduling scheme $X^{*}$ Procedure: 1. $u\leftarrow 0$ 2. flag $\leftarrow$ true 3. $\overline{s}\leftarrow \widehat{s}\left(u\right),C\leftarrow \overline{s}$ 4. While flag   4.1 $a_{now}\leftarrow {\pi }^{*}\left(\overline{s}\right)$   4.2 $\overline{s}\leftarrow a_{now}$   4.3 $C\leftarrow [C\hspace{1em}\overline{s}]$   4.4 if the network state saved in C has a loop, then $X\left(u\right)$ is the part of the loop in $C$; Calculate the throughput of $X\left(u\right)$ according to the formula (12), which is $M\left(u\right)$. flag $\leftarrow$ false. End if   End while 5. $u\leftarrow u+1$, return 2 until all states are computed. 6. $n\leftarrow \arg \max \hspace{1em}M$ 7. $M^{*}\leftarrow M\left(n\right),X^{*}\leftarrow X\left(n\right)$

4. Performance Evaluation

We evaluate and compare the performance of the proposed methods in multimode networks via simulations. We compare the throughput of multimode networks under different network topologies and different channel qualities.

In the simulation process, the network nodes all support three communication modes, namely, PSK, MFSK, and PRSS, with a bandwidth of 3.2 k. Other parameters are shown in

Table 2. The simulation parameter table.

Name	Parameter
Depth of water	50 m
Working depth of nodes	10 m
PSK rate	1.8 kbps
MFSK rate	0.8 kbps
PRSS rate	0.2 kbps
SL	183 dB
NL	101.4 dB

.

4.1. Performance Analysis of the Three-Node Single-Mode Network

Under the equilateral triangle network topology as shown in Figure 1a, the proposed method is used to analyze the throughput performance of the single-mode network, and the results are compared with those of traditional methods to verify the correctness of the proposed method.

The results obtained after strategy iteration are shown in the figure below. The diagram illustrates up to three action choices for each state. The optimal scheduling scheme is determined through the analysis of Algorithm 1, as shown in Figure 2. When the initial state sequence number is three, adopting the optimal action (serial number 8) leads to the formation of a new state (serial number 8). Subsequently, the optimal action for this new state is identified, and this process continues until a closed loop exists. The scheduling result in Figure 2 can be represented by Formula (17): a cycle comprises four scheduling time slots with status serial numbers 3, 8, 9, and 19, along with their corresponding specific status values [0;1;0], [0;3;2], [2;0;0], and [3;0;1].

$$s\left(3\right)\stackrel{a\left(8\right)}{\to }s\left(8\right)\stackrel{a\left(9\right)}{\to }s\left(9\right)\stackrel{a\left(19\right)}{\to }s\left(19\right)\stackrel{a\left(3\right)}{\to }{s}_{3\times 3}\Rightarrow \left[\begin{array}{l}0\hspace{1em}0\hspace{1em}2\hspace{1em}3\\ 1\hspace{1em}3\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}2\hspace{1em}0\hspace{1em}1\hspace{1em}\end{array}\right]$$

(18)

There are three optimal scheduling schemes obtained through Algorithm 2, respectively.

$$\left[\begin{array}{l}2\hspace{1em}3\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}0\hspace{1em}1\hspace{1em}3\\ 0\hspace{1em}1\hspace{1em}0\hspace{1em}2\end{array}\right];\left[\begin{array}{l}3\hspace{1em}2\hspace{1em}0\hspace{1em}0\\ 0\hspace{1em}1\hspace{1em}0\hspace{1em}3\\ 0\hspace{1em}0\hspace{1em}1\hspace{1em}2\end{array}\right];\left[\begin{array}{l}0\hspace{1em}3\hspace{1em}0\hspace{1em}2\\ 0\hspace{1em}0\hspace{1em}3\hspace{1em}1\\ 2\hspace{1em}1\hspace{1em}0\hspace{1em}0\end{array}\right]$$

(19)

Only one scheme is given in the paper [14], which is the same as the first scheme in this paper, and all three possible schemes are given in this paper, which verifies the validity of the method in this paper.

4.2. Performance Analysis of Multimode Networks with Different SLs in Arbitrary Topologies

In the following, the throughput performance of the multimode network is analyzed using the method proposed in this paper under an irregular topology. The topology of an irregular triangular network is shown in Figure 3.

The maximum time slot length of the irregular triangle network is G = 2. This network includes three transmission links $l_1$, $l_2$, and $l_3$, with corresponding link lengths rounded to $l_3=2$, $l_2=2$, and $l_1=1$ respectively. The sound source level is measured at 186 dB. The optimal scheduling results are illustrated in Figure 4 where different lines represent different nodes and different columns represent various time slots.

In the scheduling results:

-

“Send_*” indicates that the node is transmitting signals during this time slot; the number corresponding to * represents the destination node of the transmitted signals.

-

“Rcv_*” indicates that the node receives signals during this time slot; the number corresponding to * represents the source node of the received signals.

The corresponding scheduling result consists of eight slots with an upper limit output of 2400 bps per slot. To achieve maximum throughput, two communication modes PSK and MFSK are utilized within the network.

Figure 5 shows a comparison of the throughput analysis results between the proposed method and the traditional method at different emission source levels. The traditional method refers to the method presented in [16]. The throughput performance of the proposed method surpasses that of the method in [26], with a continued improvement observed as the number of iterations increases. After the fourth iteration of the algorithm, convergence is achieved in reaching the upper limit of throughput across all sound source levels. When comparing the throughput performance limits under different sound source levels, the results of the two methods are similar for small and large sound sources. The largest disparity between the two methods appears when the sound source level is 179 dB, at which point the method proposed in this paper achieves a 1.5 kbps per slot higher throughput compared to the method in [16]. On average, the performance is improved by 68%.

Table 3. The upper limit of network throughput at different sound source levels.

Serial Number	SL(dB)	Throughput Upper Limit	Scheduling Matrix	Scheduling Cycle	Communication Link Status
1	>163	0 bits/slot	0	0	0
2	165–169	0.4 bits/slot	$\left[\begin{array}{l}2\\ 0\\ 0\end{array}\right];$	1	PRSS: 1 MFSK: 0 PSK: 0
3	170~176	1.2 bits/slot	$\left[\begin{array}{l}0\hspace{1em}2\\ 0\hspace{1em}1\\ 0\hspace{1em}0\end{array}\right];$	2	PRSS: 0 MFSK: 2 PSK: 0
4	177	1.4 bits/slot	$\left[\begin{array}{l}3\hspace{1em}2\\ 0\hspace{1em}3\\ 0\hspace{1em}0\end{array}\right];$	2	PRSS: 1 MFSK: 2 PSK: 0
5	178	1.6 bits/slot	$\left[\begin{array}{l}2\\ 0\\ 2\end{array}\right];$	1	PRSS: 1 MFSK: 1 PSK: 0
6	179~185	2.1 bits/slot	$\left[\begin{array}{l}3\hspace{1em}2\\ 0\hspace{1em}0\\ 0\hspace{1em}2\end{array}\right];$	2	PRSS: 0 MFSK: 2 PSK: 1
7	186~188	2.4 bits/slot	$\left[\begin{array}{l}0\hspace{1em}2\hspace{1em}0\hspace{1em}3\hspace{1em}0\hspace{1em}3\hspace{1em}0\hspace{1em}2\\ 0\hspace{1em}1\hspace{1em}0\hspace{1em}0\hspace{1em}3\hspace{1em}0\hspace{1em}3\hspace{1em}1\\ 0\hspace{1em}2\hspace{1em}1\hspace{1em}2\hspace{1em}1\hspace{1em}0\hspace{1em}0\hspace{1em}0\end{array}\right];$	8	PRSS: 0 MFSK: 4 PSK: 8
8	>188	2.7 bits/slot	$\left[\begin{array}{l}0\hspace{1em}2\hspace{1em}0\hspace{1em}3\hspace{1em}0\hspace{1em}3\hspace{1em}0\hspace{1em}2\\ 0\hspace{1em}1\hspace{1em}0\hspace{1em}0\hspace{1em}3\hspace{1em}0\hspace{1em}3\hspace{1em}1\\ 0\hspace{1em}2\hspace{1em}1\hspace{1em}2\hspace{1em}1\hspace{1em}0\hspace{1em}0\hspace{1em}0\end{array}\right];$	8	PRSS: 0 MFSK: 0 PSK: 12

presents the results of the algorithm throughput analysis at various sound source levels. It is apparent that there is a direct correlation between the sound source level and the performance limit of network throughput and the communication rate of the associated communication link. When the sound source level is low, only the PRSS mode with the slowest speed can be selected for transmission. However, once the sound source level exceeds 188 dB, all links transition to using the PSK mode with the highest transmission speed. At this threshold, the upper limit of network throughput aligns with the conclusion of Chitre’s paper. It is evident that this paper’s algorithm takes into consideration physical layer characteristics such as link channel quality and multiple modes, resulting in a more accurate analysis of the upper limit of network throughput in an actual underwater acoustic environment.

4.3. Network Throughput under Different Hydrological Environments

Different hydrological conditions in the marine environment correspond to different propagation losses. The network throughput under three typical hydrological conditions is analyzed. The sound velocity profiles of these three conditions are depicted in Figure 6a, and the corresponding transmission losses at different distances are obtained using the Bellhop model, as shown in Figure 6. The different hydrological conditions are associated with different transmission losses, with hydrologic condition 1 having the smallest transmission loss and hydrologic condition 3 having the largest transmission loss. The propagation loss under hydrologic condition 1 is 0.48 dB and 1.31 dB smaller than that under hydrologic conditions 2 and 3, respectively.

The decodable SNR thresholds were analyzed for different systems and transmission distances using Monte Carlo simulation, yielding the specific results illustrated in Figure 7. In the figure, link 3 represents the transmission link between node 1 and node 2, link 2 represents the transmission link between node 1 and node 3, and link 1 represents the transmission link between node 2 and node 3. The figure reveals the following findings:

(1) Among different communication systems, the PRSS mode requires the smallest decoded SNR threshold, while the PSK mode demands the most.

(2) The largest threshold signal-to-noise ratio corresponds to hydrological condition 1, while hydrologic conditions 2 and 3 have smaller threshold signal-to-noise ratios. This difference is due to the complex channel multipath structure under hydrological condition 1, which increases the decoding threshold. On the other hand, hydrologic conditions 2 and 3 have simpler channel structures, resulting in smaller decoding thresholds.

(3) In the PRSS mode, decodable signal-to-noise ratio thresholds show minimal variation across different distances and hydrological conditions. However, significant differences exist within the PSK mode when considering various distances and hydrological conditions. This difference can be attributed to the increased sensitivity of the PSK mode to channel multipath structures compared to that of the PRSS mode.

The analysis of network throughput under different hydrological conditions is presented in Figure 8. It is evident that the throughput performance limits of each sound source level vary under different hydrological conditions, providing results that are more consistent with real-world situations. Similar to Figure 5, the throughput results under different hydrological conditions are consistent for smaller and larger sound source levels. This is because when the sound source level is large enough, all links support multimode communication, whereas when the sound source level is small, no communication is possible. The maximum difference between the throughput upper limits of the three hydrologic conditions is 0.5 kbits/slot. Compared to the traditional method, the average throughput performance is improved by 67.5%, 78.3%, and 67.5%, respectively, under the three hydrological conditions.

5. Conclusions

We have conducted simulations to demonstrate that considering the differences in the communication capabilities of different links can lead to more accurate throughput limits for multimode networks. The signal-to-noise ratio of received signals at different nodes can vary due to factors such as the network node sound source level, propagation loss, marine environment, and interference signals. Higher received signal-to-noise ratios in communication systems can support higher transmission rates, while lower ratios support lower rates. By leveraging this understanding, we remodeled the throughput analysis process and found that the upper limit of the throughput performance for multimode networks is greater than that of traditional methods. Additionally, we developed a decoding conflict model that incorporates the propagation characteristics of actual underwater acoustic channels, providing a more precise assessment of underwater acoustic signal conflicts.

In summary, our proposed methods offer an accurate evaluation of the throughput performance of multisystem underwater acoustic networks. However, it is worth noting that when the network topology becomes more complex or the number of nodes increases, the scale of the state matrix and action matrix becomes very large, rendering the solution algorithm based on the combination of dynamic programming and greedy algorithms impractical. In light of this, we plan to enhance the solution method and propose a new approach that can address the application needs of any network topology. Additionally, we intend to conduct sea tests to apply and validate the method outlined in this paper, with the aim of further refining and improving its effectiveness.