Comprehensive Ocean Information-Enabled AUV Motion Planning Based on Reinforcement Learning

Abstract

Motion planning based on the reinforcement learning algorithms of the autonomous underwater vehicle (AUV) has shown great potential. Motion planning algorithms are primarily utilized for path planning and trajectory-tracking. However, prior studies have been confronted with some limitations. The time-varying ocean current affects algorithmic sampling and AUV motion and then leads to an overestimation error during path planning. In addition, the ocean current makes it easy to fall into local optima during trajectory planning. To address these problems, this paper presents a reinforcement learning-based motion planning algorithm with comprehensive ocean information (RLBMPA-COI). First, we introduce real ocean data to construct a time-varying ocean current motion model. Then, comprehensive ocean information and AUV motion position are introduced, and the objective function is optimized in the state-action value network to reduce overestimation errors. Finally, state transfer and reward functions are designed based on real ocean current data to achieve multi-objective path planning and adaptive event triggering in trajectorytracking to improve robustness and adaptability. The numerical simulation results show that the proposed algorithm has a better path planning ability and a more robust trajectory-tracking effect than those of traditional reinforcement learning algorithms.

1. Introduction

Covering 71% of the Earth’s surface area while being rich in material resources such as minerals, energy and organisms [1], the oceans are extremely important for both the natural ecosystem and human society. Therefore, researchers have invested much effort in researching intelligent, stable and efficient ocean exploration technologies. Among them, autonomous underwater vehicles (AUVs), as important ocean exploration technology tools, have been widely used because they have better flexibility [2] and can accomplish large-scale and wide-ranging underwater ocean monitoring [3].

To enhance the robustness and adaptability of AUV motion, numerous researchers have conducted extensive scientific research in the field of motion planning and achieved remarkable results. Motion planning research encompasses environmental modeling and motion planning algorithms. Environmental modeling necessitates the modeling of both environmental and motion models. An environmental model is utilized to establish a description of the spatial positions of an AUV object within the environment, as well as their corresponding attribute characteristics. Existing environmental modeling methods include the grid method [4], Voronoi diagrams [5], and inverse distance weighting (IDW) [6]. The environmental model established by the grid method is discrete, and actions can only be performed at grid vertices, resulting in systematic errors under actual conditions. The Voronoi diagram method maximizes the distance between the AUV and the obstacles, leading to suboptimal motion planning results. The IDW method, based on the grid method, employs distance-based similarity for weighted interpolation to address the errors caused by the discrete states in the grid method and establish a more realistic environmental model. The motion model defines the state and action outputs of the examined AUV. Traditional methods include the particle method and dynamic method [7]. The particle method simplifies AUV to a particle, while the dynamic method accurately reflects its motion based on performance. This paper uses IDW and dynamic methods for environmental modeling.

It is important to consider the motion-planning algorithms of AUV in addition to environmental modeling [8]. Motion planning algorithms are primarily utilized for path planning and trajectory-tracking tasks. The algorithm’s optimization objective in path planning tasks is to obtain the optimal path to reach the target point. Traditional algorithms include the A* algorithm [9], rapidly exploring random trees (RRT) algorithm [10], rapidly exploring random trees star (RRT*) algorithm [11], particle swarm optimization [12], ant colony optimization [13], and genetic algorithm [14]. The first three methods [9,10,11] are straightforward to implement, but their efficiencies are constrained by the scale of environmental modeling and decrease as the search range expands. The last three approaches are based on intelligent bionic models [12,13,14]. These algorithms have more advantages than the former techniques, but their ability to extract features in more complex environments is limited, resulting in poor generalization ability, a tendency to fall into local optima, and difficulty in meeting practical application requirements. Due to the influence of time-varying ocean currents, the state of the ocean environment is constantly changing, making it very difficult for these algorithms to complete path planning tasks. In trajectory-tracking tasks, the algorithm’s optimization objective is to ensure that the AUV’s motion trajectory remains as consistent as possible with the target trajectory. Traditional algorithms include nonlinear model-based predictive controllers [15,16,17] and proportional–integral–differential (PID) control [18]. Ocean currents introduce uncertainty into the AUV’s state, making it challenging for traditional algorithms to complete trajectory tracking tasks in ocean environments.

With the development of artificial intelligence (AI), increasingly complex AI technologies are being used in AUV motion planning [19], providing more possibilities for solving underwater problems using the Internet of Things [20]. Reinforcement learning (RL), as a new machine learning paradigm, learns through trial and error. The goal of an RL policy is to maximize reward values or achieve specific goals during the process of interacting with the environment [21,22]. This training process endows RL with nonmodality, flexibility, and adaptability, giving it high potential in complex and ever-changing ocean environments. In [23], a value-based Q-learning algorithm was successfully applied to obstacle avoidance tasks. However, this method requires the use of tables to store all operations, consuming considerable storage resources. Mnih et al. [24] incorporated a deep Q network (DQN), which uses deep neural networks to represent state-action value networks and addresses the curse of dimensionality. Chu et al. [25] introduced the DQN algorithm into the AUV path planning task. Zhang et al. [26] incorporated deep deterministic policy gradient (DDPG) [27] into the AUV trajectory-tracking tasks, reducing the induced tracking errors. Du et al. [28] and Hadi et al. [29] used twin delayed deep deterministic policy gradient (TD3) [30] as the AUV state control method to obtain better results, but the exploration process of the algorithm is slow. Xu et al. [31] and Huang et al. [32] introduced soft actor–critic (SAC) [33] for AUV path planning, improving the algorithm’s exploration ability; however, nly single tasks were considered. He et al. [34] proposed asynchronous multithreading proximal policy optimization (AMPPO) based on proximal policy optimization (PPO) [35], which achieved good results in multiple task scenarios, but the designs of the individual environments used the experiment were relatively simple. In present studies [25,26,28,29,31,32,34] on modeling reinforcement learning environments, the real ocean current data are seldom considered, and the overestimation error caused by the ocean current is rarely taken into account in these algorithms, which leads to the low exploration rate, poor adaptability, and robustness in motion planning. To solve these problems, a reinforcement learning-based motion planning algorithm based on comprehensive ocean information (RLBMPA-COI) is proposed to AUV motion planning tasks. The main contributions of this paper are as follows:

• To create a realistic ocean current environment, the AUV motion model and real ocean current data are introduced into the environment modeling based on reinforcement learning. This method effectively minimized the distance between practical applications and the simulations. Therefore, this method brings more significant practical value to AUV motion planning.
• We proposed the RLBMPA-COI AUV motion planning algorithm based on a real ocean environment. By incorporating local ocean current information into the objective function of the action-value network, this algorithm successfully minimizes the overestimation error interference and enhances the efficiency of motion planning. According to the influence of the ocean current, target distance, obstacles, and steps on the motion planning task, we also established a corresponding reward function. This ensures the efficient training of the algorithm, further improving its exploration ability and better adaptability.
• Multiple simulations, including complex obstacles and dynamic multi-objective tasks in path planning, as well as a trajectory-tracking task are designed to verify the performance of the proposed algorithms comprehensively. Compared with state-of-art algorithms, the performance of RLBMPA-COI has been evaluated and proven by numerical results, which demonstrate efficient motion planning and high flexibility for expansion into different ocean environments.

The remainder of this paper is organized as follows. Section 2 introduces the AUV motion model, ocean current model and SAC background information. Then, we describe the specific method implementation details in detail in Section 3. In Section 4, we discuss a variety of experiments to verify the performance of the algorithm. Finally, we conclude our work in Section 5.

2. Background

2.1. AUV Motion Model

As the carrier of the algorithm, the motion model of an autonomous underwater vehicle (AUV) is crucial. An AUV has six degrees of freedom (DOFs). The motion model of an AUV is illustrated in Figure 1, which displays both the ground-fixed coordinate system and the body axis coordinate system of the AUV [7]. Both systems follow the right-hand rule. In the ground-fixed coordinate system, the state vector of an AUV $\eta =[x,y,z,\sigma ,\beta ,\chi ]$ represents its three-axis position and rotation information. In contrast, within the body axis coordinate system, $\mathrm{v}=[u,v,w,g,h,j]$ corresponds to the velocity and angular velocity of each directional component.

The kinematic equations of the AUV are derived by transforming the motion vector $\mathrm{v}$ in the body axis coordinate system into a global state vector $\eta$ in the ground-fixed coordinate system.

$$\eta =\left[\begin{array}{cc}{T}_v& 0\\ 0& T_w\end{array}\right]\mathrm{v}$$

(1)

$$T_v=\left[\begin{array}{ccc}cos\beta cos\chi & -cos\sigma sin\chi +sin\sigma sin\beta cos\chi & sin\sigma sin\chi +cos\sigma sin\beta cos\chi \\ cos\beta sin\chi & cos\sigma cos\chi +sin\sigma sin\beta sin\chi & -sin\sigma cos\chi +cos\sigma sin\beta sin\chi \\ -sin\beta & sin\sigma cos\beta & cos\sigma cos\beta \end{array}\right]$$

(2)

$$T_w=\left[\begin{array}{ccc}1& tan\beta sin\sigma & tan\beta cos\sigma \\ 0& cos\sigma & -sin\sigma \\ 0& sin\sigma {cos}^{-1}\beta & cos\varphi {cos}^{-1}\beta \end{array}\right]$$

(3)

2.2. Ocean Current Modeling

The changing conditions of an ocean environment are influenced by a combination of factors that are time-varying and uncertain. Among them, ocean currents are the key factors affecting AUV motion planning. For this reason, many scholars have conducted research related to ocean current models. One of the more common methods for constructing ocean current models is functional simulation. For example, Ref. [36] described an ocean current model as follows:

$$\left\{\begin{array}{c}Cu{r}_x=k_{1}Cu{r}_{max}sin\left(k_{3}y\right)cos\left(k_{3}y\right)+k_{1}cos\left(2k_{1}t\right)+k_4\hfill \\ Cu{r}_y=-\lambda Cu{r}_{max}cos\left(k_{2}x\right)sin\left(k_{3}y\right)+k_5\hfill \end{array}\right.$$

(4)

In this model, the latitude and longitude velocity components are represented by $Cu{r}_x$ and $Cu{r}_y$, respectively. The variables $k_1$, $k_2$, $k_3$, and $\lambda$ are related to the underwater environment and may change with factors such as tides and depth. The variables $k_4$ and $k_5$ are random variables that reflect the variability of the underwater environment. In an actual ocean environment, the vertical velocity component of a 3D currents is much smaller than the longitudinal component and is often neglected for simplicity.

However, a functional simulation model is limited by its input parameters and may not accurately reflect real ocean currents. Utilizing such a model as an ocean current environment may reduce realism and limit its practical use. The meanings of the model symbols are also shown in

Table A1. Notations.

Name	Description
$\eta$	AUV state vector
$x,\mathrm{y},\mathrm{z}$	Surge, sway and heave in the global coordinate system
$\sigma ,\beta ,\chi$	Roll, pitch and yaw in the global coordinate system
$\mathrm{v}$	Velocity vector of the AUV
$u,v,w$	Velocity components in the body coordinate system
$g,h,j$	Angular velocity components in the body coordinate system
t	Moment of AUV motion
$Cu{r}_{max}$	Maximum change in ocean current
$Cu{r}_x,Cu{r}_y$	The strength of the ocean current components in the global coordinate system
$Cu{r}_{(x_t,y_t)}$	The vector of the ocean current at time t
$k_1,k_2,k_3,\lambda$	Environmental influence parameters
$k_4,k_5$	Stochastic influence parameters
$Di{s}_o$	Distance between the AUV and the obstacle
$Di{s}_t$	Distance between the AUV and the target
$Di{s}_l$	Relative distance between the obstacle and the nearest target
$S_w$	Obstacle warning range
$S_c$	Obstacle collision range
$S_{tar}$	Target perceived distance
${\delta }_{auv}$	The range of angles output by the movement of the AUV
P	The position of the AUV
$P_i^{goal}$	The position of the i-th target point
$k_i^{goal}$	The completion status of the i-th target point
${\theta }_1,{\theta }_2$	State value network parameters
${{\theta }_1}^{\prime },{{\theta }_2}^{\prime }$	Target state value network parameters
$\varphi$	Policy network parameters
${\alpha }_t$	Action entropy value weight
${\epsilon }_{\sigma },{\epsilon }_{\beta }$	Systematic error caused by ocean currents in the longitudinal and transverse directions
$B_{\sigma },b,\Delta \sigma$	The amplitude, frequency and phase of the longitudinal ocean current
$C_{\beta },c,\Delta \beta$	The amplitude, frequency and phase of the transverse ocean current
${\gamma }_x,{\gamma }_y$	The weights of ocean currents for algorithm updates

.

2.3. Soft Actor–Critic

In this study, we apply reinforcement learning (RL) to solve the AUV motion planning problem in a continuous state action space. To implement a deep RL algorithm in the continuous space, we first defined a tuple $\left(S,A,p,r\right)$ based on a Markov decision process. Assuming that the state space S and action space A are continuous, the state transition probability $p(s_t,s_{t+1},a_t):S\times S\times A\to [0,\infty )$ is expressed as the probability density of the next state $s_{t+1}\in S$, given the current state $s_t\in S$ and action $a_t\in A$. Additionally, we use ${\rho }_{\pi }\left(s_t\right)$ and ${\rho }_{\pi }\left(s_t,a_t\right)$ to denote the trajectory distribution of the state and state-action probabilities under policy $\pi \left(a_t|s_t\right)$.

For conventional RL, the optimization goal is to maximize the expected value of the strategy $\pi \left(a_t|s_t\right)$, that is:

$${\pi }_{\mathrm{nor}}^{\ast }=arg\underset{\pi }{max}\sum _t^{}{\mathrm{E}}_{(s_t,a_t)\sim {\rho }_{\pi }}\left[r(s_t,a_t)\right]$$

(5)

However, to ensure that the developed RL algorithm can fully explore the environment and avoid becoming trapped in a local optimal solutions, an entropy term $H(\pi \left(\right|s_t)$ is incorporated into the policy. This improves the exploration capability of the RL algorithm, allowing it to explore the environment more quickly while avoiding local optima. The policy ${\pi }_{E\mathrm{nt}}^{\ast }$ corresponding to the maximum entropy of RL is as follows:

$${\pi }_{E\mathrm{nt}}^{\ast }=arg\underset{\pi }{max}\sum _t^{}{\mathrm{E}}_{(s_t,a_t)\sim {\rho }_{\pi }}[r(s_t,a_t)+{\alpha }_{t}H\left(\pi \left(\right|s_t)\right)]$$

(6)

${\alpha }_t$ is an adaptive temperature parameter that specifies the relative importance of the entropy term in the reward function. This controls the stochasticity of the optimal policy and thus the degree of exploration. The temperature coefficient is initially set to a relatively large value to ensure sufficient exploration during the early stages of training. The adaptive temperature parameter ${\alpha }_t\to 0$ is then adjusted to ensure that the algorithm gradually stabilizes during the later stages of training.

3. Methods

The ocean environment is constantly changing and remains uncertain due to the impact of ocean currents on AUV motion. To achieve improved adaptability and intelligence, the algorithm needs to adjust to changes in the ocean currents. Therefore, we designed a comprehensive ocean information-enabled AUV motion planning model based on RL (RLBMPA-COI) to effectively suppress the disturbances caused by ocean currents in AUV motion planning.

3.1. Marine Environmental Information Model

To balance the simplicity and realism of the environmental simulation and reduce the discrepancy between the simulated and actual environments, this study utilizes real ocean data to construct a time-varying ocean current model and improve the confidence level of the AUV motion planning environment. We focus on the South China Sea (22.25°N to 22.75°N, 115.50°E to 116°E) and use data from the National Data Center for Marine Science [37,38]. We constructed a comprehensive time-varying ocean current model using various information such as temperature, wind speed, wave intensity, wave temperature, and salinity from January to June 2021. The vertical velocity of a 3D ocean current is much smaller than its longitude and latitude components and is usually ignored to reduce the complexity of environmental simulation.

This study constructs an ocean simulation environment based on real ocean environment data using the GYM framework. The GYM framework is a reinforcement learning environment framework developed by the OpenAI company, and it has been widely used in the field of reinforcement learning. First, a grid method is used to create a 100 × 100 scale grid corresponding to the study area. Then, an inverse distance weighted interpolation method is used to interpolate the integrated ocean data model into the grid so that the AUV can obtain the ocean changing current state in a continuous space. The ocean current vector is $Cu{r}_{(x_t,y_t)}={[Cu{r}_{x\left(t\right)},Cu{r}_{y\left(t\right)}]}^T$. The longitudinal current velocity $Cu{r}_x$ and the latitudinal current velocity $Cu{r}_y$ in the plane of the geostationary coordinate system $OXY$ are shown in Figure 2, where t corresponds to the current moment. The variation in the surface currents is shown in Figure 3, where the direction of the black arrows corresponds to the direction of the currents and the distribution of the heatmap corresponds to the current intensity $\left|Cu{r}_{(x_t,y_t)}\right|$. Notably, 10 grid lengths correspond to 0.05°.

3.2. Reinforcement Learning Environment

Based on the ocean current change model, an RL environment is designed, which includes two basic modules: a state transition function and a reward function. The scale of this RL environment is 100 × 100 grids, and all observation parameters and output parameters are scaled by 50 times the scale size.

3.2.1. State Transition Function

The state transition function describes the changes in the environment corresponding to the actions taken by the AUV at each moment. To reduce the computational complexity of the algorithm, we only consider the AUV motion in the horizontal plane. Correspondingly, $\eta$ and $\mathrm{v}$ in the AUV motion model can be simplified to $\eta =[x,y,\chi ]$ and $\mathrm{v}=[u,v,j]$. The change in the position of the AUV is not only affected by its speed, but also related to the present strength of the ocean current $Cu{r}_{(x_t,y_t)}$. Therefore, the change in the state vector of the AUV can be expressed as:

$$\left[\begin{array}{c}\dot{x}\hfill \\ \dot{y}\hfill \\ \dot{\chi }\hfill \end{array}\right]=\left[\begin{array}{c}u{cos}^2\chi +vcos\chi sin\chi +Cu{r}_x\\ ucos\chi sin\chi +v{sin}^2\chi +Cu{r}_y\\ j\end{array}\right]$$

(7)

To meet the needs of motion planning tasks, two features are introduced, namely $Di{s}_o$ and $Di{s}_t$, which represent the distance of the AUV relative to obstacles and the distance relative to the target position, respectively. Overall, the state space of this RL environment can be expressed as $S=[\eta ,\mathrm{v},Cu{r}_{(x_t,y_t)},Di{s}_o,Di{s}_t]$.

Additionally, we designed a continuous action space A so that the algorithm can take more accurate actions. Specifically, the action output consists of speed $\left|\mathrm{v}\right|$ and yaw angle ${\delta }_{auv}$. The range of values for peed is related to the maximum ocean current speed $\left|\mathrm{v}\right|\in [-0.5Cu{r}_{max},2Cu{r}_{max}]$, and the range of values for the yaw angle corresponds to the range of angles at which the hull can turn at the current moment ${\delta }_{auv}\in [-\pi /\phantom{\pi 4}$4,$\pi$/4].

3.2.2. Reward Function

In RL, the model seeks to maximize the expected reward value, which further influences the directional gradient update process of the neural network. By combining the task requirements of path planning and trajectory tracking, this paper carefully designs the corresponding reward functions.

In the path planning task, the target reward suffers from sparsity and abrupt changes. Therefore, a distance reward $r_s$ is designed to guide the training procedure of the algorithm.

$$r_s=-min\left\{\underset{i=1}{\overset{n}{\mathrm{Set}}}\left\{k_i^{goal}Di{s}_t(P,P_i^{goal})\right\}\right\}$$

(8)

$$k_i^{goal}=\left\{\begin{array}{c}10e^4,Di{s}_t(P,P_i^{goal})<S_{tar}\hfill \\ 1,else\hfill \end{array}\right.$$

(9)

$k_i^{goal}$ represents reaching the target, $Di{s}_t(P,P_i^{goal})$ represents the distance of the AUV relative to the target, and $S_{tar}$ represents the criterion for judging whether the target has been reached. To evaluate the ability of the algorithm to adapt to ocean currents changes, an ocean current reward $r_{cur}$ is designed.

$$r_{cur}=\mathrm{sgn}\left({\left[\begin{array}{c}\left(u{cos}^2\chi +vcos\chi sin\chi \right)Cu{r}_x\hfill \\ \left(ucos\chi sin\chi +v{sin}^2\chi \right)Cu{r}_y\hfill \end{array}\right]}^T\right)Cu{r}_{(x_t,y_t)}$$

(10)

When the AUV reaches the target point, it triggers a task completion event and receives a task completion reward $r_{goal}$.

$$r_{goal}=\left\{\begin{array}{c}0,{\displaystyle \sum _{i=0}^n}\left(k_{{}_{i(t+1)}}^{goal}-k_{{}_{i\left(t\right)}}^{goal}\right)=0\hfill \\ 200,else\hfill \end{array}\right.$$

(11)

When the AUV approaches or reaches the obstacle position, it triggers a collision event and receives an obstacle reward $r_{\mathrm{o}}$. The obstacle reward $r_{\mathrm{o}}$ is set in levels as a warning value and a collision value, and is judged based on the obstacle warning distance $S_w$ and the obstacle collision distance $S_c$. The obstacle collision reward $r_{\mathrm{o}}$ is also affected by the completion of task $Di{s}_l$.

$$Di{s}_l=min\left\{\underset{i=1}{\overset{n}{\mathrm{Set}}}\left\{k_i^{goal}Dis(P_{obs},P_i^{goal})\right\}\right\}$$

(12)

$$r_{\mathrm{o}}=\left\{\begin{array}{c}0,Di{s}_o\ge S_w\hfill \\ -5,S_w>Di{s}_o\ge S_c\hfill \\ -30Di{s}_l,Di{s}_o<S_c\hfill \end{array}\right.$$

(13)

Finally, the step reward $r_{step}$ and the total reward r are defined. Specifically, ${\gamma }_{cur}=0.4$, ${\lambda }_{step}=10e^4$. The weight of the ocean current reward is determined by the parameter ${\gamma }_{cur}$ and the distance reward $r_s$, and the weight can be adaptively adjusted to improve the adaptability and robustness of the algorithm to changes in ocean currents.

$$r=r_s+\left|r_s\right|{\gamma }_{cur}{r}_{cur}+r_{goal}+r_{\mathrm{o}}-{\lambda }_{step}/r_{step}$$

(14)

In the trajectory-tracking task, an event trigger reward $r_t$ is defined to judge the overlap of trajectories. At the same time, the distance reward $r_s$ is modified to satisfy the requirements of trajectory tracking. $P^{trace}$ is the target position of the trajectory.

$$r_t=\left\{\begin{array}{c}0,Dis(P,P^{trace})>S_{tar}\hfill \\ 1,else\hfill \end{array}\right.$$

(15)

$$r_s=-Di{s}_t(P,P^{trace})$$

(16)

$$r=r_s+r_t+{\gamma }_{cur}exp(r_t+r_s-1)r_{cur}+r_o$$

(17)

The total trajectory-tracking reward is r. The weight coefficient of the ocean current reward $exp(r_t+r_s-1)$ is adaptively adjusted to improve the robustness of tracking and the adaptability of the algorithm to ocean currents changes.

3.3. RLBMPA-COI Algorithm

In an ocean environment, ocean currents have a significant impact on the motion state of an AUV. Therefore, a motion planning algorithm for an AUV needs to fully consider the impact of ocean currents to reduce the interference caused by changes in ocean currents. To satisfy this requirement, we designed a reinforcement learning-based motion planning algorithm with comprehensive ocean information, the RLBMPA-COI algorithm. The system block diagram of this algorithm is shown in Figure 4.

Overall, the RLBMPA-COI algorithm includes a pair of state-action value networks $Q_{{\theta }_i}^{}(s_t,a_t),i=1,2$, a pair of target state-action value networks $Q_{\theta {}_i^{\prime }}^{}(s_t,a_t),i=1,2$, and a policy network ${\pi }_{\varphi }(s_t,a_t)$. Their respective network parameters are ${\theta }_1,{\theta }_2,{{\theta }_1}^{\prime },{{\theta }_2}^{\prime }$ and $\varphi$. The RLBMPA-COI algorithm enhances the exploration ability of the algorithm by introducing the action entropy ${\alpha }_{t}H(\pi \left(\right|s_t)$, so the optimization goal of the policy network ${\pi }^{\ast }$ can be expressed as:

$${\pi }^{\ast }=arg\underset{\pi }{max}\sum _t^{}{\mathrm{E}}_{(s_t,a_t)\sim {\rho }_{\pi }}[r(s_t,a_t)+{\alpha }_{t}H\left(\pi \left(\right|s_t)\right)]$$

(18)

The state-action value network $Q_{{\theta }_i}(s_t,a_t)$ is used to evaluate the current policy. Furthermore, it is defined as:

$$Q_{{\theta }_i}(s_t,a_t)\stackrel{\Delta }{=}r(s_t,a_t)+\gamma {\mathrm{E}}_{s_{t+1}\sim p}\left[V\left(s_{t+1}\right)\right],i=1,2$$

(19)

$$V\left(s_t\right)\stackrel{\Delta }{=}{\mathrm{E}}_{a_t\sim {\pi }_{\varphi }}[Q_{{\theta }_i}(s_t,a_t)-log{\pi }_{\varphi }\left(a_t\right|s_t)]$$

(20)

$V\left(s_t\right)$ is the soft state-value function, and $\gamma$ is a discount factor. By introducing the target state-action value networks $Q_{{{\theta }_i}^{\prime }}^{}(s_t,a_t)$ to assist with training the state-action value networks $Q_{{\theta }_i}^{}(s_t,a_t)$, the training process is stabilized. In existing methods, a trimmed state-action value function $y_{Qt\mathrm{a}\mathrm{r}}$ is typically used as the target function.

$$y_{Qt\mathrm{a}\mathrm{r}}=r(s_t,a_t)+\gamma \left(\underset{i=1,2}{min}{Q}_{\theta {}_i^{\prime }}(s_{t+1},{\pi }_{\varphi }\left(s_{t+1}\right))-{\alpha }_{t}log{\pi }_{\varphi }\left(a_{t+1}\right|s_{t+1})\right)$$

(21)

However, $y_{Qt\mathrm{ar}}$ only considers the impact of random errors on the update of the $Q_{{\theta }_i}^{}(s_t,a_t)$ network and does not account for the systematic errors caused by the motion of ocean currents, which can lead to overestimation. Therefore, this paper specifically analyzes and improves the impact of ocean current motion on the training process of the $Q_{{\theta }_i}^{}(s_t,a_t)$ network.

To attain improved generalization, the ocean current is simplified to sinusoidal functions $Cu{r}_x$ and $Cu{r}_y$ in the longitudinal and transverse components, respectively. The state-action value networks $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$ can represent the state of the ocean current during sampling as follows:

$$Q_{{\theta }_1}:\left\{\begin{array}{c}Cu{r}_{x1}=B_{\sigma }sin\left(b{\sigma }_x\right)\hfill \\ Cu{r}_{y1}=C_{\beta }sin\left(c{\beta }_y\right)\hfill \end{array}\right.$$

(22)

$$Q_{{\theta }_2}:\left\{\begin{array}{c}Cu{r}_{x1}=B_{\sigma }sin(b{\sigma }_x+\Delta \sigma )\hfill \\ Cu{r}_{y1}=C_{\beta }sin(c{\beta }_y+\Delta \beta )\hfill \end{array}\right.$$

(23)

Different sampling moments result in phase differences between the ocean current in $Q_{{\theta }_1}^{}$ and $Q_{{\theta }_2}^{}$, forming a systematic overestimation error that cannot be eliminated in $y_{Qt\mathrm{ar}}$. Assuming that the phase difference is constant, the existing overestimation error in the longitudinal direction can be defined as ${\epsilon }_{\Delta \sigma }$:

$${\epsilon }_{\Delta \sigma }=\frac{1}{2\pi }{\int }_0^{2\pi }min\{Cu{r}_{x1},Cu{r}_{y1}\}d{\sigma }_x,Cu{r}_{x1}\ge 0,Cu{r}_{y1}\ge 0$$

(24)

Assuming that the sampled phase difference $\sigma$ is uniformly distributed in $[0,2\pi ]$, the average systematic overestimation error caused by the interference of the longitudinal ocean current can be calculated as ${\epsilon }_{\sigma }$:

$${\epsilon }_{\sigma }=\frac{{\int }_0^{2\pi }{\int }_0^{2\pi }min\{Cu{r}_{x1},Cu{r}_{y1}\}d{\sigma }_{x}d\Delta \sigma }{{\left(2\pi \right)}^2},Cu{r}_{x1}\ge 0,Cu{r}_{y1}\ge 0$$

(25)

$$\begin{array}{c}{\epsilon }_{\sigma }=\frac{1}{{\left(2\pi \right)}^2}{\int }_0^{2\pi }{\int }_0^{2\pi }min\{B_{\sigma }sin\left(b{\sigma }_x\right),B_{\sigma }sin(b{\sigma }_x+\Delta \sigma )\}d{\sigma }_{x}d\Delta \sigma \\ =\frac{2B_{\sigma }}{{\left(2\pi \right)}^2}{\int }_0^{2\pi }{\int }_0^{\left|\frac{\pi -\sigma }{2}\right|}sin\left(b{\sigma }_x\right)d{\sigma }_{x}d(\Delta \sigma )\\ =\frac{B_{\sigma }}{b\pi }-\frac{B_{\sigma }}{b^2{\pi }^2}sin\left(b\pi \right),B_{\sigma }sin\left(b{\sigma }_x\right)\ge 0,B_{\sigma }sin(b{\sigma }_x+\Delta \sigma )\ge 0\end{array}$$

(26)

Similarly, in the transverse direction, the average systematic overestimation error caused by the ocean current can be expressed as ${\epsilon }_{\beta }$:

$${\epsilon }_{\beta }=\frac{C_{\beta }}{c\pi }-\frac{C_{\beta }}{c^2{\pi }^2}sin\left(c\pi \right)$$

(27)

Due to the systematic overestimation error caused by ocean currents in the reward function, this paper incorporates an ocean current reward weight coefficient $qu{e}_{cur}$ into the experience tuple, that is, $(s_t,a_t,r,s_{t+1},k_{{}_i}^{goal},qu{e}_{cur})$. At the same time, an ocean current weight queue $Qu{e}_{cur}$ is designed, which can add $qu{e}_{cur}$ to the queue and output smoothed weight values ${\gamma }_x$ and ${\gamma }_y$. The target function $y_{Qt\mathrm{a}\mathrm{r}}$ is redefined as:

$$y_{Qt\mathrm{a}\mathrm{r}}=r(s_t,a_t)+\gamma (1-{\gamma }_x{\epsilon }_{\sigma }-{\gamma }_y{\epsilon }_{\beta })\times \left(\underset{i=1,2}{min}{Q}_{{\theta }_i^{{}^{\prime }}}(s_{t+1},{\pi }_{\varphi }\left(s_{t+1}\right))-{\alpha }_{t}log{\pi }_{\varphi }\left(a_{t+1}\right|s_{t+1})\right)$$

(28)

We calculate the mean squared error (MSE) loss of the state-action value network.

$$\begin{array}{c}{J}_Q\left({\theta }_i\right)={\mathrm{E}}_{(s_t,a_t)\sim D}\left[\frac{1}{2}{(Q_{{\theta }_i}(s_t,a_t)-y_{Qt\mathrm{ar}})}^2\right]\\ ={\mathrm{E}}_{(s_t,a_t)\sim D}\left[\frac{1}{2}{(Q_{\theta }(s_t,a_t)-(r(s_t,a_t)+\gamma (1-{\gamma }_x{\epsilon }_{\sigma }-{\gamma }_y{\epsilon }_{\beta }){\mathrm{E}}_{s_{t+1}\sim p}\left[V_{\overline{\theta }}\left(s_{t+1}\right)\right]))}^2\right]\end{array}$$

(29)

The parameter ${\theta }_i$ is optimized using the random gradient descent method:

$${\widehat{\nabla }}_{{\theta }_i}{J}_Q\left({\theta }_i\right)=\left(Q_{{\theta }_i}(s_t,a_t)-\left(r(s_t,a_t)\right.\right.+\gamma (1-{\gamma }_x{\epsilon }_{\sigma }-{\gamma }_y{\epsilon }_{\beta })\times \left.\left.\left(\underset{i=1,2}{min}{Q}_{{\theta }^{\prime }}(s_{t+1},a_{t+1})-{\alpha }_{t}log\left({\pi }_{\varphi }\left(a_{t+1}\right|s_{t+1})\right)\right)\right)\right){\nabla }_{{\theta }_i}{Q}_{{\theta }_i}(s_t,a_t)$$

(30)

Finally, we update ${\theta }_i$: ${\theta }_i\leftarrow {\theta }_i-{\lambda }_{\theta }{\widehat{\nabla }}_{{\theta }_i}{J}_Q\left({\theta }_i\right),i=1,2$, where ${\lambda }_{\theta }$ is the update weight. We update the target network parameters ${\theta }_i^{{}^{\prime }}$: ${{\theta }_i}^{\prime }\leftarrow {{\theta }_i}^{\prime }+(1-\delta ){{\theta }_i}^{\prime },i=1,2$, where $\delta$ is the soft update weight.

For the strategy network ${\pi }_{\varphi }\left(a_t\right|s_t)$, the loss function is:

$$J_{\pi }\left(\varphi \right)={\mathrm{E}}_{s_t\sim D,{\epsilon }_t\sim N}[\alpha log({\pi }_{\varphi }\left(f_{\varphi }({\epsilon }_t;s_t)\right|s_t)-Q_{\theta }(s_t,f_{\varphi }({\epsilon }_t;s_t))]$$

(31)

The action is reparametrized under the influence of the neural network $a_t=f_{\varphi }({\epsilon }_t;s_t)$. The gradient corresponding to the loss function of the strategy network ${\pi }_{\varphi }\left(a_t\right|s_t)$ is

$${\widehat{\nabla }}_{\varphi }{J}_{\pi }\left(\varphi \right)={\nabla }_{\varphi }{\alpha }_{t}log({\pi }_{\varphi }\left(a_t\right|s_t)+({\nabla }_{a_t}{\alpha }_{t}log\left({\pi }_{\varphi }\left(a_t\right|s_t)\right)-{\nabla }_{a_t}\underset{i=1,2}{min}{Q}_{{\theta }_i}(s_t,a_t)){\widehat{\nabla }}_{\varphi }{f}_{\varphi }({\epsilon }_t;s_t)$$

(32)

Finally, we update the parameter $\varphi$: $\varphi \leftarrow \varphi -{\lambda }_{\varphi }{\widehat{\nabla }}_{\varphi }{J}_{\pi }\left(\varphi \right)$, where ${\lambda }_{\varphi }$ is the update frequency. In addition, the loss function $J\left({\alpha }_t\right)$ dynamically adjusts the entropy weights ${\alpha }_t$:

$$J\left({\alpha }_t\right)={\mathrm{E}}_{a_t\sim {\pi }_{\varphi }}\left[-{\alpha }_{t}log{\pi }_{\varphi }\left(a_t\right|s_t)-{\alpha }_t{\mathrm{H}}_0\right]$$

(33)

where ${\mathrm{H}}_0$ is the target entropy and the corresponding gradient function is ${\widehat{\nabla }}_{{\alpha }_t}J\left({\alpha }_t\right)=-log{\pi }_{\varphi }\left({\alpha }_t\right|s_t)-{\mathrm{H}}_0$, while the entropy weight ${\alpha }_t$ is updated by ${\alpha }_t\leftarrow {\alpha }_t-{\lambda }_{{\alpha }_t}{\widehat{\nabla }}_{{\alpha }_t}J\left({\alpha }_t\right)$, where ${\lambda }_{{\alpha }_t}$ is the update frequency. The algorithm pseudocode is shown in Algorithm 1.    

Algorithm 1: RLBMPA-COI algorithm.
1	Initialize the environment and set the number of episodes $n=0$;
2	Initialize a replay buffer D with a capacity of $B_r$;
3	Randomly initialize the policy network ${\pi }_{\varphi }$ and state-action value networks $Q_{{\theta }_i}^{}(s_t,a_t),i=1,2$ with parameter vectors $\varphi$, ${\theta }_1$, ${\theta }_2$;
4	Initialize the target state-action value networks $Q_{{{\theta }_1}^{\prime }}^{}(s_t,a_t)$ with parameter vectors ${{\theta }_1}^{\prime }\leftarrow {\theta }_1,{{\theta }_2}^{\prime }\leftarrow {\theta }_2$;
5	Define the total training steps T, episode training steps $T_{epi}$;
6	foreach n in set T do
7			Reset the step counter for each episode $t=0$;
8			Clear the event trigger flag $r_{\mathrm{o}}=0$, $r_t=0$;
9			Clear the task completion flag $r_{goal}=0$;
10			Reset the environment, AUV start point $P_0$, initial state $S_0$;
11			while $t<=T_{epi}$ or not $k_{{}_i}^{goal}\equiv 10e^4$ or not $r_{\mathrm{o}}<-5$ do
12					Integrate ocean current information and AUV motion state
					$s_t=[\eta ,\mathrm{v},Cu{r}_{(x_t,y_t)},Di{s}_o,Di{s}_t]$;
13					Based on the AUV motion model and state $s_t$, take action according to
					policy ${\pi }_{\varphi }(s_t,a_t)$;
14					if $Di{s}_o<S_c$ then
15							$r_{\mathrm{o}}=-30Di{s}_l$ Trigger obstacle collision reward, end this episode
16					else
17							if $S_w>Di{s}_o\ge S_c$ then
18									$r_{\mathrm{o}}=-5$, trigger obstacle warning reward
19							else
20									pass;
21							end
22					end
23					if $Di{s}_t<S_{tar}$ then
24							$r_t=1$, trigger target tracking reward
25					end
26					Get an AUV state $S_t=[\eta ,\mathrm{v},Cu{r}_{(x_t,y_t)},Di{s}_o,Di{s}_t]$, update the rewards
					$r_s,r_{step},r_{cur}$, and calculate the total reward r;
27					Integrate to obtain the state $S_{t+1}$ and various rewards of the next moment,
					determining the triggering of the obstacle events and target events, and
					calculate the total reward r;
28					Store the experience $(s_t,a_t,r,s_{t+1},k_{{}_i}^{goal},qu{e}_{cur})$ in D;
29					$t=t+1$;
30			end
31			$n=n+1$;
32			if $n\%f_g\mathrm{is}\mathrm{True}$ then
33					Sample a segment of experience from D;
34					Extract the weight of the ocean current reward $qu{e}_{cur}$ to the queue $Qu{e}_{cur}$;
35					Calculate the average ocean current weight ${\epsilon }_{\sigma },{\epsilon }_{\beta }$ in the queue $Qu{e}_{cur}$;
36					Update $y_{Qt\mathrm{ar}}$;
37					Update ${\theta }_i\leftarrow {\theta }_i-{\lambda }_{\theta }{\widehat{\nabla }}_{{\theta }_i}{J}_Q\left({\theta }_i\right),i=1,2$;
38					Update $\varphi \leftarrow \varphi -{\lambda }_{\varphi }{\widehat{\nabla }}_{\varphi }{J}_{\pi }\left(\varphi \right)$;
39					Update ${\alpha }_t\leftarrow {\alpha }_t-{\lambda }_{{\alpha }_t}{\widehat{\nabla }}_{{\alpha }_t}J\left({\alpha }_t\right)$;
40					Update ${\theta }_i^{\prime }\leftarrow {\theta }_i+(1-\delta ){\theta }_i^{\prime },i=1,2$;
41			end
42	end

4. Experiments and Discussions

4.1. Basic Settings

This study focuses on a section of the South China Sea (from 22.25°N to 22.75°N and from 115.50°E to 116°E). An ocean current model is built using data from the National Ocean Science Data Center. Based on the GYM reinforcement learning framework, the model is imported into a 100 × 100 grid and expanded into a continuous simulation environment using inverse distance-weighted interpolation. Time-varying ocean current values are affected by AUV position. A complete state environment and action output space are constructed by combining the AUV motion model and ocean current model.

This study divides motion planning into path planning and trajectory tracking. We implement algorithms such as SAC, DDPG, TD3, PPO, and AMPPO. The environmental state space is partially observable and the AUV can obtain its state, current state, obstacle observation, and target position. The experiments were performed on Ubuntu 18.04 with 64 GB RAM, NVIDIA GTX3080 GPU and Python 3.8. The specific parameters of the algorithm are shown in

Table A2. Algorithm training parameters.

Parameter	Value
Ocean current weight coefficient ${\gamma }_{cur}$	0.4
Step weight coefficient ${\lambda }_{step}$	$10e^4$
The completion status of the i-th target point $k_i^{goal}$	1 or $10e^4$
Obstacle warning range $S_w$	0.2
Obstacle collision range $S_c$	0.05
Target perceived distance $S_{tar}$	0.05
AUV turning angle range ${\delta }_{auv}$	$[-0.25\pi ,0.25\pi ]$
Learning rate	$3e^{-4}$
Replay buffer D capacity $B_r$	$10e^6$
Current weight queue $u{e}_{cur}$ capacity $B_{que}$	$10e^4$
Total training steps T	$10e^6$
Maximum training steps per episode $T_e$	300
Gradient update frequency $f_g$	1
Update weight of the policy network ${\lambda }_{\varphi }$	0.005
Discount factor $\gamma$	0.99

and

Table A3. Algorithm hyperparameter statistics.

Algorithm Name	DDPG	RLBMPA-COI	PPO	SAC	TD3	AMPPO
Total training steps T	$10e^6$	$10e^6$	$10e^6$	$10e^6$	$10e^6$	$10e^6$
Learning rate	$10e^{-3}$	$3e^{-4}$	$3e^{-4}$	$3e^{-4}$	$10e^{-3}$	$3e^{-4}$
Replay buffer D capacity $B_r$	$10e^6$	$10e^6$	None	$10e^6$	$10e^6$	None
Discount factor $\gamma$	0.99	0.99	0.99	0.99	0.99	0.99
Update weight of the policy network ${\lambda }_{\varphi }$	0.005	0.005	None	0.005	0.005	None
Current weight queue $u{e}_{cur}$ capacity $B_{que}$	None	$10e^4$	None	None	None	None
Target policy noise	None	None	None	None	0.2	None
Number of processes	1	1	1	1	1	2
Generalized advantage estimation (GAE)	None	None	0.95	None	None	0.95

.

4.2. Path Planning Tasks

4.2.1. Path Planning Task with Complex Obstacles

Given the complexity of the real ocean environment, this research concentrates on the path planning task for an AUV in ocean currents. The effectiveness of path planning is evaluated using indicators such as the number of episodes, travel distance per episode, success rate and reward value.

The AUV must navigate to its target location while contending with time-varying ocean currents and complex obstacles. As illustrated in Figure 5, the AUV is initialized at a fixed location at an arbitrary angle. Utilizing its current state information, it selects actions based on its policy and calculates a reward value. The AUV must adapt to changing ocean currents and use event-triggering mechanisms to perceive and avoid dense obstacles while moving toward its target location under the influence of ocean currents to maximize its cumulative rewards.

The entire algorithmic training process is shown in Figure 6. Figure 6a shows the average number of steps taken by each algorithm in each episode, Figure 6b shows the success rate of each algorithm, Figure 6c shows the average reward value of each algorithm, and Figure 6d shows the cumulative distance traveled by each algorithm.

At the start of the training process, the AUV selects actions at random to explore its environment. The number of steps taken in each episode is determined by whether the AUV comes into contact with any boundaries or obstacles. Occasional successes result in higher reward values and these experiences are stored in the experience replay pool. As the algorithm converges, the AUV discovers a stable and optimal solution for its path.

As illustrated in Figure 7, the RLBMPA-COI algorithm converges more quickly due to the action entropy and the reduction in system overestimation errors caused by ocean currents. DDPG and TD3, deterministic policy gradient algorithms, converge to suboptimal solutions when influenced by time-varying ocean currents.

4.2.2. Multi-Objective Path Planning Task

We designed a multi-objective path planning task in which the AUV must navigate around obstacles and reach multiple targets in sequence while being affected by real-time changing ocean currents. The specific goal comprises two fixed targets and one moving target. As illustrated in Figure 8, during each round of training, the AUV is randomly initialized within region 1.

The training process of the algorithm in this environment is shown in Figure 9. The RLBMPA-COI algorithm suppresses the influence of system overestimation errors and takes more accurate actions by incorporating environmental information. Therefore, the algorithm converges faster and maintains a high path planning success rate under the interference of ocean currents, reflecting the strong adaptability of the RLBMPA-COI algorithm. The actual motion paths taken by each algorithm during multi-objective path planning is shown in Figure 10. The RLBMPA-COI algorithm creates a smoother path. When approaching a moving target, its strong robustness enables it to accurately reach the target position, reducing the number of required steps and the cumulative distance.

As shown in

Table 1. Path planning location change table.

Path	Start Point	Terminal Point	AMPPO		SAC		RLBMPA-COI
			Step	Reward	Step	Reward	Step	Reward
1	(55,20)	(25,35), (60,80), (80,25)	107.25	535.4	122.75	548.45	84.4	567.26
2	(65,10)	(25,35), (60,80), (80,25)	118.7	527.76	106	553.25	85	561.58
3	(55,20)	(25,35), (60,65), (80,35)	116.35	533.79	107.32	563.34	82.25	573.46
4	(65,10)	(25,35), (60,65), (80,35)	118.5	529.83	109.25	556.59	87.65	563.05

, the initial position and target point position of the AUV are adjusted to test the path planning ability of the algorithm.

4.2.3. Ocean Current Strength Robustness Experiment

Ocean current intensities are affected by various factors. This study performs a comparative experiment concerning ocean current intensity to evaluate the impacts of different ocean current intensities on path planning algorithms. Specifically, based on multi-objective path planning tasks, all algorithms are trained in actual ocean current environments and tested for robustness under different ocean current intensities by scaling the overall ocean current intensity. Each algorithm is tested for 5000 steps under different current intensities and the average number of steps, average reward and success rate are recorded. The experimental results are shown in Figure 11 and Figure 12.

As the ocean current intensity increases, the average reward and success rate of each algorithm decreased to varying degrees. Among them, only the RLBMPA-COI and TD3 algorithms are able to maintain relative stability under the influence of strong ocean currents. The RLBMPA-COI algorithm had stronger adaptability to ocean current components and can take precise actions with high robustness.

4.2.4. Ocean Current Adaptation Experiment

Extreme weather conditions can change the movement patterns of ocean currents, affecting the updating of state transition functions and yielding new data outside the sample distribution range of the training environment. This poses a great challenge to AUV motion planning algorithms based on reinforcement learning. To address this issue, we designed an ocean current adaptability experiment. Specifically, all algorithms are trained in a real ocean current environment and tested in an environment simulated by an ocean current function [36] to test the robustness and adaptability of the algorithm in an out-of-distribution environment. Firstly, we calculated the extreme and average values of real ocean current data. Furthermore, after adjusting the parameters in Equation (4), we fit the simulation to the real ocean current data in extreme and average values. As shown in

Table 2. Ocean current model parameters.

Parameter	Value
Maximum speed of ocean currents $Cu{r}_{max}$	1.892
Environmental influence parameters of ocean currents $\lambda$	3.018
Environmental influence parameters of ocean currents $k_1,k_2,k_3$	3.122, 3.023, 3.034
Stochastic influence parameters of ocean currents $k_4,k_5$	1.015, 0.987

:

The experimental results, as illustrated in Figure 13 and Figure 14, reveal that in an out-of-distribution ocean current environment, the DDPG, PPO, and AMPPO algorithms were unable to complete the path planning task. In contrast, the RLBMPA-COI algorithm maintained a high success rate, demonstrating its exceptional adaptability to its environment.

4.3. Trajectory-Tracking Task

In a trajectory-tracking task, an AUV must move along a predetermined path, aligning its motion trajectory with the predetermined path as closely as possible to minimize dynamic tracking errors. Since AUV trajectory-tracking tasks implemented in still water do not take environmental factors into account and the experimental backgrounds are overly idealized, this study focuses on trajectory-tracking tasks under the influence of time-varying ocean currents.

Curve Trajectory-Tracking Task

This task uses a cosine curve as the target trajectory and sets obstacles in the target trajectory path. As shown in Figure 15, the AUV needs to align its own motion trajectory with the target trajectory as well as possibly reduce the induced dynamic tracking error.

During the training process of the algorithm, obstacles may be present in the target path, preventing the AUV from moving in the direction of the optimal reward. This requires the algorithm to be capable of sacrificing short-term gains for long-term returns. A changing ocean current can interfere with the recording of the AUV’s motion state, causing errors in the algorithm’s estimation of the AUV’s action value. This can result in an overestimation of the state-action value and ultimately lead to a mismatch between the estimated and actual situations, causing the algorithm to become shortsighted or even incorrect. The training process is illustrated in Figure 16.

As illustrated in Figure 17, due to the interference of obstacles and ocean currents, the PPO, TD3, and AMPPO algorithms obtain the incorrect estimation of the motion value during training and become trapped in local optima. This is reflected in AUV’s inability to navigate around obstacles during the trajectory-tracking process. In contrast, the RLBMPA-COI, DDPG, and SAC algorithms were able to complete the AUV trajectory-tracking task. Compared to the other algorithms, the RLBMPA-COI algorithm is better able to adjust the AUV’s motion state to bring its trajectory closer to the target trajectory, resulting in smaller tracking errors. Additionally, when the tracking errors and ocean currents are accounted for in the reward function, the average return value yielding the AUV trajectory-tracking algorithm based on the RLBMPA-COI algorithm is the highest.

5. Conclusions

In this work, we introduce the RLBMPA-COI algorithm, a multitask motion planning approach for an AUV operating in ocean environments. The algorithm incorporates realtime ocean current data into an environmental model to balance realism and complexity, providing robust support for multitask environments and addressing the limitations exhibited by existing methods. We combine an AUV motion model with environmental data to accurately represent the environmental features and design a reinforcement learning environment for multiple AUV motion planning tasks. The RLBMPA-COI algorithm builds on the SAC algorithm to capture and integrate variations in ocean currents, reducing the overestimation caused by current motion and significantly enhancing the stability and practicality of the algorithm. We evaluate the performance of our algorithm across multiple task environments and demonstrate that the RLBMPA-COI algorithm outperforms other methods in target exploration tasks conducted in scenarios with dense obstacles and multipoint targets. The RLBMPA-COI algorithm also exhibits excellent stability in scenarios with the changing ocean current strength and simulated ocean currents. Furthermore, the RLBMPA-COI algorithm maintains its effective trajectory-tracking effect in complex environments.