Navigation and Obstacle Avoidance for USV in Autonomous Buoy Inspection: A Deep Reinforcement Learning Approach

Abstract

To address the challenges of manual buoy inspection, this study enhances a previously proposed Unmanned Surface Vehicle (USV) inspection system by improving its navigation and obstacle avoidance capabilities using Proximal Policy Optimization (PPO). For improved usability, the entire system adopts a fully end-to-end design, with an angular deviation weighting mechanism for stable circular navigation, a novel image-based radar encoding technique for obstacle perception and a decoupled navigation and obstacle avoidance architecture that splits the complex task into three independently trained modules. Experiments validate that both navigation modules exhibit robustness and generalization capabilities, while the obstacle avoidance module partially achieves International Regulations for Preventing Collisions at Sea (COLREGs)-compliant maneuvers. Further tests in continuous multi-buoy inspection tasks confirm the architecture’s effectiveness in integrating these modules to complete the full task.

1. Introduction

Buoys are widely utilized in meteorological forecasting, marine environment assessment, navigation safety assurance, and scientific research [1,2]. However, as equipment is exposed to harsh marine environments for extended periods, buoys encounter issues such as seawater corrosion and biofouling. Regular inspection and maintenance, therefore, become imperative to ensure the efficient and reliable operation of buoy observation systems [3].

Conventional buoy inspection methods predominantly rely on manual checks, which prove not only costly but also hazardous, particularly under adverse weather conditions. Unmanned Surface Vehicles (USVs), as novel maritime operation platforms, demonstrate significant advantages in automation, all-weather capabilities, and long endurance. When equipped with visual sensors, USVs can effectively detect and evaluate buoy structural integrity, surface corrosion, and abnormal orientation. Previous research by [4] develops a USV system for buoy inspection that achieves dynamic target tracking during circular navigation through Kernelized Correlation Filter (KCF) algorithms, coupled with buoy health assessment using YOLOv7 architecture, as illustrated by Figure 1. Building upon this foundation, this study introduces improvements to USV navigation and obstacle avoidance systems.

To achieve comprehensive detection and evaluation of buoy conditions, a USV must perform stable circumnavigation tasks. Conventional methods typically employ waypoint tracking through integrated Line-of-Sight (LOS) guidance and Proportional-Integral-Derivative (PID) control architectures [5]. These methods approximate circumnavigation trajectories by discretizing the circular path into segments, which, however, could introduce trajectory oscillations [6].

Rooted in Sutton and Barto’s pioneering theoretical framework [7], reinforcement learning (RL) enabled robotic systems to achieve adaptive navigation, dynamic obstacle avoidance, and model-free control. Recent advancements have extended these benefits to USV motion control, demonstrating promising results. Ref. [8] proposes a navigation algorithm using the Deep Q-Network (DQN) for USVs, which demonstrates comparable tracking accuracy to Improved LOS (ILOS) with PD controllers. However, its circular navigation approach still relies on pre-set waypoints and does not eliminate trajectory oscillations. Furthermore, ref. [9] proposes a DQN-based heading control method focusing on a KVLCC2 tanker. Ref. [10] addresses the high computational complexity of Q-learning in USV path tracking through Smoothly-Convergent DRL (SCDRL). While both studies demonstrate the ability to follow curved paths, their reliance on predefined paths presents challenges for dynamic path generation in multi-buoy inspection tasks. Additionally, neither approach explicitly involves a circular path following in their experiments.

As an essential motion control component of the autonomous buoy inspection USV system, this study proposes a Proximal Policy Optimization (PPO)-based circular navigation algorithm. By introducing an angular deviation weighting mechanism, the proposed algorithm enables smooth transitions between linear and circular motions without relying on pre-set waypoints or paths. Another control component, the goal-directed navigation algorithm, is developed as a simplified and modified version of the circular navigation algorithm.

Given the frequent presence of moving objects, such as fishing boats and cargo ships near waterways, an autonomous USV system should be capable of real-time obstacle avoidance. Traditional obstacle avoidance methods, such as the A* algorithm [11] and Dijkstra’s algorithm [12], are mostly effective in static obstacle environments and come with high computational demands, limiting their applicability in time-sensitive scenarios. Methods like the Artificial Potential Field (APF) [13], Velocity Obstacle (VO) [14,15], and Dynamic Window Approach (DWA) [16], while suitable for dynamic environments, often suffer from the risk of getting trapped in local optima [17]. Due to the strong performance of reinforcement learning in complex decision-making tasks, a few reinforcement learning-based obstacle avoidance strategies for USVs have been proposed. Ref. [18] introduces a Deep Deterministic Policy Gradient (DDPG)-based avoidance strategy adhering to COLREGs (International Regulations for Preventing Collisions at Sea), enhanced with a prioritized experience replay mechanism to improve USV learning efficiency. Similarly, ref. [19] applies convolutional neural networks (CNNs) to process temporal state information and train a DQN model, demonstrating higher avoidance success rates compared to traditional VO and APF methods. However, these approaches depend heavily on accurate motion parameters of obstacles, which are often difficult to obtain in real-world inspection tasks. Ref. [17] proposes an end-to-end obstacle avoidance solution using local grid maps by encoding the temporal dynamics of obstacles within a single image. While innovative, this method does not strictly comply with COLREGs, and the process of constructing grid maps from sensor data is not explicitly addressed. Ref. [20] explores avoidance for UAVs using radar sensors but is limited to static obstacle scenarios.

Building upon these foundations, this study designs radar data as image input and adopts the temporal coding strategy from [17]. The proposed approach achieves dynamic obstacle avoidance that partially complies with COLREGs in a fully end-to-end manner, thus bridging the gap between theoretical research and practical implementation of autonomous collision avoidance systems.

The main contributions of this study are as follows:

• An angular deviation weighting mechanism is introduced, enabling a circular navigation algorithm for USVs that demonstrates robust adaptability across different circumnavigation radii, even when trained on a specific radius.
• A novel generalized radar image encoding technique is developed, facilitating an end-to-end USV collision avoidance solution that relies solely on radar sensors for obstacle scenario recognition.
• Inspired by the concept of hierarchical reinforcement learning [21], a decoupled architecture is proposed, separating navigation and obstacle avoidance for USVs. The effectiveness of this architecture is validated in autonomous multi-buoy inspection tasks.

2. Background

2.1. Kinematic Model of USV

This study establishes a three-degree-of-freedom (3-DOF) underactuated vessel mathematical model for algorithm design and validation [22]. Considering only the motion of the vessel in a two-dimensional plane, as illustrated in Figure 2, $\mathit{\eta }=\left[x,y,\psi \right]$ is defined as the vessel’s position and heading angle in the world coordinate system while $\mathit{\upsilon }=\left[u,v,r\right]$ represents the surge velocity, sway velocity, and yaw rate in the body-fixed coordinate frame. The kinematic model of USV can be expressed as:

$$\begin{array}{c}\dot{\mathit{\eta }}=T\left(\psi \right)\end{array}\mathit{\upsilon }$$

(1)

where $T\left(\psi \right)$ is the transformation matrix from the world coordinate system to the body-fixed coordinate system, which can be expressed as:

$$\begin{array}{c}T\left(\psi \right)=\left[\begin{array}{ccc}\cos \psi & -\sin \psi & 0\\ \sin \psi & \cos \psi & 0\\ 0& 0& 1\end{array}\right]\end{array}$$

(2)

2.2. Reinforcement Learning

Reinforcement Learning trains agents to optimize strategies through environment interactions using reward/penalty feedback, aiming to maximize cumulative discounted rewards. It is modeled via Markov Decision Processes (MDPs), which are defined by state/action spaces $\left(\mathcal{S},\mathcal{A}\right)$, transition probability $\mathcal{P}$, reward function $\mathcal{R}$, and discount factor $\gamma$. Each timestep generates a tuple $\langle s_t,a_t,r_{t+1},s_{t+1}\rangle$ where $s_t\in \mathcal{S}$ is the observed state at timestep $t$, $a_t\in \mathcal{A}$ is the action selected at $t$, $s_{t+1}$ is the resulting state determined by transition probability $\mathcal{P}(s_{t+1}|s_t,a_t)$, and $r_{t+1}$ is the feedback reward given by $\mathcal{R}\left(s_t,a_t,s_{t+1}\right)$.

Another critical component is the policy $\pi (a|s)$, which defines the probability distribution over action $a$ given state $s$. The objective of reinforcement learning is to find an optimal policy $\pi *$ that maximizes the expected cumulative discounted reward $R_t$:

$$\begin{array}{c}{R}_t=r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\dots ={\displaystyle {\displaystyle \sum }_{k=0}^{\infty }}{\gamma }^k{r}_{k+t+1}\end{array}$$

(3)

where the discount factor $\gamma \in \left[0,1\right]$ determines the agent’s preference for long-term versus short-term rewards. A lower $\gamma$ favors short-term gains, while a higher $\gamma$ prioritizes long-term benefits.

2.3. Proximal Policy Optimization

Policy gradient methods address traditional RL’s limitations in high-dimensional continuous action spaces by directly optimizing parameterized policies ${\pi }_{\theta }(a|s)$ via gradient ascent to maximize discounted rewards [23]. PPO, an improved Trust Region Policy Optimization (TRPO) variant, ensures stable training through constrained policy updates while maintaining computational efficiency [24]. This study adopts PPO with continuous action spaces as its reinforcement learning framework.

In policy gradient methods, the expected cumulative discounted reward is represented as:

$$\begin{array}{c}J\left(\theta \right)={\mathbb{E}}_{\tau ~{\pi }_{\theta }}\left[{\displaystyle {\displaystyle \sum }_{t=0}^T}{\gamma }^{t}r\left(s_t,a_t\right)\right]\end{array}$$

(4)

where $\tau$ denotes a trajectory $\left(s_0,a_0,r_0,\dots ,s_T\right)$ sampled by the policy ${\pi }_{\theta }$. The expectation is taken over trajectories sampled under the current policy. The policy parameters $\theta$ are updated via gradient ascent to maximize $J\left(\theta \right)$. The policy gradient is derived as:

$$\begin{array}{c}{\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{\tau ~{\pi }_{\theta }}\left[{\displaystyle {\displaystyle \sum }_{t=0}^T}{\nabla }_{\theta }\log {\pi }_{\theta }\left(a_t|s_t\right)A^{\pi }\left(s_t,a_t\right)\right]\end{array}$$

(5)

where $A^{\pi }\left(s_t,a_t\right)$ is the advantage function, which quantifies how much better taking action $a_t$ in state $s_t$ is compared to the average performance under the policy $\pi$.

Typically, the PPO algorithm employs the Generalized Advantage Estimation (GAE) as the advantage function [25]. It is defined as:

$$\begin{array}{c}\left\{\begin{array}{c}{A}_t^{\mathrm{GAE}\left(\gamma ,\lambda \right)}={\delta }_t+\gamma \lambda A_{t+1}^{\mathrm{GAE}\left(\gamma ,\lambda \right)}\\ {\delta }_t=r_t+\gamma V^{\pi }\left(s_{t+1}\right)-V^{\pi }\left(s_t\right)\end{array}\right.\end{array}$$

(6)

where the TD error ${\delta }_t$ serves as an unbiased but high-variance estimate of the advantage function. $V^{\pi }\left(s_t\right)$ denotes the expected cumulative discounted reward an agent will receive starting from state $s_t$ and following policy $\pi$ thereafter.

A key feature of PPO is its ability to avoid resampling trajectories during training by leveraging importance sampling. However, this approach introduces a new challenge: If the policy update step is too large, the importance weight can deviate significantly from its optimal values. To address this, PPO incorporates a clipping mechanism that limits how much the new policy is allowed to diverge from the old policy. These mechanisms lead to the following reformulated objective function:

$$\begin{array}{c}{J}^{CLIP}\left(\theta \right)={\mathbb{E}}_t\left[\min \left({\displaystyle \frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{old}}\left(a_t|s_t\right)}}{A}_t,\mathrm{clip}\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{old}}\left(a_t|s_t\right)}},1-\epsilon ,1+\epsilon \right)A_t\right)\right]\end{array}$$

(7)

where ${\displaystyle \frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{old}}\left(a_t|s_t\right)}}$ is the importance weight, quantifying how much the current policy differs from the old one for each sampled action at each state. $\epsilon$ is a hyperparameter defining the clipping range, ensuring policy updates remain within the interval $\left[1-\epsilon ,1+\epsilon \right]$.

3. Methodology

3.1. Problem Formulation

As shown in Figure 3, the complete inspection task involves the following phases: (1) Navigation from the starting point to the vicinity of the first target buoy; (2) Execution of circumnavigation inspection around the buoy; (3) Proceeding to the next target buoy; (4) Repeating phases (1)–(3) until completing all buoy inspections, followed by return to the starting point. To address the complexity of this task, we decouple the process into three core subtasks:

• Goal-directed navigation: Guides the USV from a start point to a goal point.
• Circular navigation: Enables smooth transition into orbit radius and maintains stable circular motion around the target buoy.
• Obstacle avoidance: Detects and evades moving obstacles during navigation, complying with COLREGs.

In multi-subtask environments, conventional reinforcement learning methods face issues with reward design complexity and model convergence due to conflicting multi-objective optimization [26]. To address these challenges, we adopt a hierarchical reinforcement learning framework and propose a decoupled architecture for navigation and obstacle avoidance, as illustrated in Figure 4.

The core concept involves independent training and deployment of these functional modules to ensure sample efficiency and reusability. Previous studies typically couple navigation with obstacle avoidance, often resulting in high sample requirements and complex reward functions—sparse obstacles lead to insufficient training of avoidance capacity, while dense obstacles cause convergence difficulties and overcautious behaviors. Our approach is based on the intuition that a well-trained navigation module can provide reliable heading references, where obstacle avoidance behaviors can be treated as localized dynamic adjustments to the reference heading. Specifically, when the vessel maintains the correct navigation direction, the independently trained obstacle avoidance module can be activated upon detecting an obstacle to adjust the current heading. Once the obstacle is avoided, the avoidance module is deactivated, and the default navigation module resumes. Based on the proposed architecture, this study independently trains three submodules (goal-directed navigation, circular navigation, and obstacle avoidance), with a decision and encoding unit determining module activation and state transformation.

3.2. Design of Navigation Modules

3.2.1. Environments

To address the environmental differences between navigation and obstacle avoidance tasks, this study designs separate environments for different modules. The goal-directed navigation module and the circular navigation module, introduced in this section, are trained in the same environment but with distinct termination criteria.

As shown in Figure 5, during environment initialization, the USV’s initial position $\left(x_0,y_0\right)$, initial heading angle ${\psi }_0$, goal distance $r_{g0}$, and goal azimuth angle ${\psi }_{g0}$ are randomly assigned:

$$\begin{array}{c}\begin{array}{c}\left(x_0,y_0\right)~\mathcal{U}\left({\left[-12,12\right]}^2\right)\\ {\psi }_0~\mathcal{U}\left(0,2\pi \right)\\ r_{g0}~\mathcal{U}\left(\left[50,70\right]\right)\\ {\psi }_{g0}~\mathcal{U}\left(0,2\pi \right)\end{array}\end{array}$$

(8)

Therefore, the goal position $\left(x_{goal},y_{goal}\right)$ is calculated as:

$$\begin{array}{c}\left(x_{goal},y_{goal}\right)=\left(x_0+r_{g0}\cos {\psi }_{g0},y_0+r_{g0}\sin {\psi }_{g0}\right)\end{array}$$

(9)

For the goal-directed task, the goal serves as the center of the goal area, while for the circular navigation task, the goal is the center of the expected circumnavigation trajectory (which is also the center of the buoy). This randomized task generation enhances the algorithms’ generalization capabilities.

For the goal-directed navigation task, the USV starts from $\left(x_0,y_0\right)$ and navigates toward $\left(x_{goal},y_{goal}\right)$. If the distance to the goal becomes smaller than $E^g$, the USV is considered to have reached the vicinity of the goal, and the task is completed successfully, as shown in the top-right subfigure of Figure 6.

For the circular navigation task, the USV departs from its initial position and approaches a predefined orbital radius $R_r$. It then adjusts its heading to smoothly navigate along the expected circumnavigation trajectory. As shown in main body of Figure 6, during orbiting:

• If the USV’s distance from the buoy exceeds the trajectory deviation tolerance limit $D_t$, the accumulated circumnavigation angle ${\theta }_{total}$ is reset to zero.
• If the USV remains within the tolerance range, ${\theta }_{total}$ is continuously recorded and incremented.

Both clockwise and counterclockwise circumnavigation are considered valid. The task is deemed successful if $\left|{\theta }_{total}\right|\ge 2\pi -E^c$, indicating nearly a full orbit around the buoy.

If the distance between the USV and the buoy becomes smaller than $R_{buoy}$, a collision is triggered, and the episode terminates in failure.

3.2.2. Action Space

In both navigation and obstacle avoidance environments, the USV employs continuous rudder angle control as its action form. By adjusting the rudder angle, the USV can indirectly alter its navigation direction. Considering the hydrodynamic characteristics of the vessel, the effect of the rudder angle on the heading exhibits a delay and cannot instantaneously change the USV’s course. The control range of the rudder angle is set to $\left[-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}\right]$.

3.2.3. State Spaces

The design of the state space directly impacts the agent’s environmental perception capability and training efficiency. This study designs two distinct state spaces based on task complexity and goal specificity.

For navigation tasks with relatively explicit objectives, we adopt a task-oriented state representation that avoids explicit encoding of task-specific goals in the state space, thereby accelerating training convergence.

The state space for circular navigation is represented as:

$$\langle \begin{array}{c}{\theta }_e,d_e,r_{usv},{\theta }_{total}\end{array}\rangle$$

(10)

${\theta }_e$ represents the directed angle between the actual velocity direction of the USV and the line connecting to the goal point:

$$\begin{array}{c}{\theta }_e=\left[{\theta }_v-{\psi }_{goal}+\pi \mathrm{mod} 2\pi \right]-\pi \end{array}$$

(11)

where ${\theta }_v=\arctan \left({\displaystyle \frac{\dot{y}}{\dot{x}}}\right)$ is the velocity direction in the inertial frame, ${\psi }_{goal}=\arctan \left({\displaystyle \frac{x_{goal}-x}{y_{goal}-y}}\right)$ is the goal azimuth relative to the USV. The difference is normalized to $\left[-\pi ,\pi \right]$. This is the most critical element in the state space, directly guiding the agent’s actions.

$d_e$ represents the distance between the USV and the expected circumnavigation trajectory:

$$\begin{array}{c}{d}_e=d_g-R_r\end{array}$$

(12)

where $d_g=\sqrt{{\left(x_{goal}-x\right)}^2+{\left(y_{goal}-y\right)}^2}$ is the distance to the goal. Guided by this element, the agent learns to adjust its heading when $d_e$ is small to ensure a smooth transition to the circumnavigation state and to maintain $d_e$ within $\left[-D_t,D_t\right]$ to accumulate the circumnavigation angle.

$r_{usv}=r$ is the USV’s yaw rate. Considering the hysteresis in rudder control, $r_{usv}$ reflects the USV’s motion trend, preventing oversteering and ensuring stable navigation.

${\theta }_{total}$ represents the accumulated circumnavigation angle. During each state update, this element is updated as follows:

$$\begin{array}{c}{\theta }_{total}^t=\left\{\begin{array}{c}{\theta }_{total}^{t-1}+\Delta {\psi }_{goal}^t, \mathrm{if} \left|d_e^t\right|\le D_t\\ 0, \mathrm{else}\end{array}\right.\end{array}$$

(13)

where $\Delta {\psi }_{goal}^t=\left[{\psi }_{goal}^t-{\psi }_{goal}^{t-1}+\pi \mathrm{mod} 2\pi \right]-\pi$ is the incremental azimuth angle at time $t$. The sign of ${\theta }_{total}$ provides information about the circumnavigation direction (clockwise/counterclockwise), while its absolute value indicates task progress, aiding the agent in making correct decisions.

For the goal-directed navigation, the state space is similar to circular navigation:

$$\langle \begin{array}{c}{\theta }_e,d_g,r_{usv}\end{array}\rangle$$

(14)

3.2.4. Reward Functions

Reward functions critically influence policy optimization efficiency and solution quality as the primary feedback signal [27,28]. In this study, we categorize rewards into high-magnitude terminal rewards and auxiliary rewards. Terminal rewards, with higher magnitudes, are given at task completion, while auxiliary rewards address exploration inefficiencies caused by sparse rewards [29].

Auxiliary rewards are designed using a combination of a negative exponential function and a linear function. The exponential function provides steep gradients with near-optimal behavior to accelerate late-stage training, while the linear function ensures stable early-stage progress and avoids local optima due to vanishing gradients.

To avoid potential bias introduced by reward shaping, we assign positive rewards only when the task is successfully completed. This design ensures the agent remains focused on the main task rather than chasing short-term rewards [30].

The reward functions for the circular navigation task follow these principles: higher auxiliary rewards are given when the USV is closer to the orbital trajectory, has smaller angular errors, and accumulates larger circumnavigation angles. The reward functions are defined as:

$$\begin{array}{c}\left\{\begin{array}{c}{r}_1=e^{-5.0\left|d_e\right|}-1.0-0.5{\displaystyle \frac{\left|d_e\right|}{20}}\\ r_2={\displaystyle \frac{\left|{\theta }_{total}\right|}{2\pi }}-1.0\\ r_3=e^{-5.0\left|{\theta }_e^{desire}\right|}-1.0-0.5{\displaystyle \frac{\left|{\theta }_e^{desire}\right|}{\pi }}\end{array}\right.\end{array}$$

(15)

where ${\theta }_e^{desire}$ represents the deviation between the USV’s velocity direction and the desired direction. We aim for the following behavior (see Figure 7):

• When far outside the orbit, ${\theta }_e$ should approach 0.
• When far inside the orbit, ${\theta }_e$ should approach $\pi$.
• Approaching the orbital boundary, ${\theta }_e$ smoothly transitions to $-{\displaystyle \frac{\pi }{2}}$ or ${\displaystyle \frac{\pi }{2}}$.

To achieve this, a weighted angular error is introduced:

$${\theta }_e^{desire}=\left(1-{\gamma }_e\right){\theta }_{ef}+{\gamma }_e{\theta }_{en}$$

(16)

where ${\gamma }_e$ is a weight coefficient that increases as the distance to the expected circumnavigation trajectory $d_e$ decrease. ${\theta }_{ef}$ represents the desired angular difference for large $d_e$ and ${\theta }_{en}$ represents the desired angular difference for small $d_e$. They are defined as:

$$\begin{array}{c}{\gamma }_e=e^{-5.0\left|d_e\right|}\end{array}$$

(17)

$$\begin{array}{c}{\theta }_{ef}=\left\{\begin{array}{c}\left|\left|{\theta }_e\right|-0\right|, \mathrm{if} d_e>0\\ \left|\left|{\theta }_e\right|-\pi \right|, \mathrm{else}\end{array}\right.\end{array}$$

(18)

$$\begin{array}{c}{\theta }_{en}=\left|\left|{\theta }_e\right|-{\displaystyle \frac{\pi }{2}}\right|\end{array}$$

(19)

The total auxiliary reward for circular navigation is:

$$\begin{array}{c}{r}^c=r_1+2r_2+r_3\end{array}$$

(20)

When the task is successfully completed, a terminal reward $r_v^c=1000$ is provided. If the USV collides with a buoy, a failure reward $r_f^c=-1000$ is given.

For the goal-direct navigation task, with no circumnavigation requirement ($r_2$ removed), rewards focus on minimizing distance $d_g$ and angle error ${\theta }_e$:

$$\begin{array}{c}\left\{\begin{array}{c}{r}_4=e^{-1.0\left|d_g\right|}-1.0-0.5{\displaystyle \frac{\left|d_g\right|}{20}}\\ r_5=e^{-5.0\left|{\theta }_e\right|}-1.0-0.5{\displaystyle \frac{\left|{\theta }_e\right|}{2\pi }}\end{array}\right.\end{array}$$

(21)

The total auxiliary reward for goal-direct navigation is given by:

$$r^g=r_4+r_5$$

(22)

A terminal reward $r_v^g=1000$ is provided only when the USV successfully reaches the vicinity of the goal point. In this environment, there are no failure rewards.

3.2.5. Deep Neural Networks

The PPO algorithm employs Actor-Critic network architecture to implement policy iteration optimization in continuous action space problems. Given a state, the Actor network outputs the parameters of the probability distribution over possible actions. During training, actions are determined by sampling from this distribution, while during testing, the action with the highest probability is selected. The Critic network evaluates the value of a given state, serving as $V^{\pi }\left(s_t\right)$ in Equation (6).

The Beta distribution is adopted to model the estimation of the continuous action range. The Actor network outputs the probability distribution parameters $\left(\alpha ,\beta \right)$ for the rudder angle. The sampled values fall within the interval $\left[0,1\right]$, which are then mapped to the actual action interval $\left[-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}\right]$ via a linear mapping function.

The neural network architecture designed for navigation tasks is illustrated in Figure 8, in which green denotes the Actor’s components, blue represents the Critic’s components, and gray indicates raw input layers. The Actor and Critic are trained independently, each taking the state vector as input. This input passes through two fully connected neural network layers, each with 128 nodes and Tanh activation functions for their outputs. Finally, the Actor outputs the action distribution parameters, while the Critic outputs the state value.

3.3. Design of Obstacle Avoidance Module

3.3.1. Environment

As shown in Figure 9, during the obstacle avoidance task initialization, the USV’s initial position is fixed at the origin of the world coordinate system, with its heading always oriented north, so $\left(x_0,y_0\right)=\left(0,0\right)$, ${\psi }_0={\displaystyle \frac{\pi }{2}}$. Obstacle vessel positions and headings are randomly selected from the four COLREGs-defined encounter scenarios: overtaking scenario, head-on scenario, right-crossing scenario and left-crossing scenario. Besides, the obstacle vessel’s speed, radius, and environmental disturbance are randomly selected within a certain range.

This fixed initialization reduces reward function design complexity (as discussed in Section 3.3.4), narrows the exploration space, and accelerates convergence, albeit at the cost of degraded generalization performance [31]. Consequently, coordinates and heading transformations are required during real-world deployment (see Section 3.4).

The USV is equipped with a radar sensor for obstacle detection. As illustrated in Figure 10, the radar emits $N_{scan}$ beams covering a 360° angular range, with each beam having a detection range of $\left[0,R_{radar}\right]$. The radar is fixed to the USV body, where the $0°$ beam direction aligns with the USV’s bow. Obstacle vessels are modeled as rigid circular bodies with radius $R_{obstacle}$. The USV’s safety zone radius is $R_{usv}$. Let the $i$-th radar beam detect a distance $d_{radar}^i$ at bearing angle ${\theta }_{radar}^i$ relative to the bow. The failure condition is defined as:

$$\begin{array}{c}{d}_{radar}^i<R_{usv}, \exists i\in \left\{1,\dots ,N_{scan}\right\}\end{array}$$

(23)

indicating a collision if any beam distance falls below $R_{usv}$. The success condition is defined as:

$$\begin{array}{c}{d}_{radar}^i=R_{radar}, \forall i\in \left\{1,\dots ,N_{scan}\right\}\end{array}$$

(24)

where all radar beams reach the maximum detection range $R_{radar}$, indicating successful obstacle avoidance.

3.3.2. Action Space

The action space used in the obstacle avoidance environment is identical to that in the navigation environments (see Section 3.2.2).

3.3.3. State Space

The dynamic obstacle avoidance task fundamentally differs from navigation tasks because, as a subtask of path planning, it lacks explicit objectives. Therefore, the state space design should focus on a precise quantitative representation of the USV’s motion state. Given the practical challenges of acquiring real-time velocity and heading information for moving obstacles, this study proposes leveraging radar sensing data as indirect cues. A composite state space is constructed, consisting of a high-dimensional radar data layer and a low-dimensional motion parameter layer.

For the high-dimensional radar data layer, an image-based multi-scale radar data encoding method is proposed. As shown in Figure 11b, we encode four key elements in a 256-level grayscale image through spatial and grayscale stratification: relative position of obstacle vessel, relative motion trend of obstacle vessel, USV safety radius and USV heading angle.

The implementation involves four steps: (1) Convert raw radar polar coordinates $\left(d_{radar}^i,{\theta }_{radar}^i\right)$ to Cartesian coordinates $\left(x_{radar}^i,y_{radar}^i\right)$:

$$\begin{array}{c}\left\{\begin{array}{c}{x}_{radar}^i=d_{radar}^i\cos \left({\theta }_{radar}^i+\psi \right)\\ y_{radar}^i=d_{radar}^i\sin \left({\theta }_{radar}^i+\psi \right)\end{array}\right.\end{array}$$

(25)

where $\left(x_{radar}^i,y_{radar}^i\right)$ are the coordinates of the $i$-th radar beam endpoint in the radar-centric coordinate system. This enables geometric alignment between radar data and the global coordinate system, eliminating heading angle $\psi$ effects on obstacle position representation. (2) Plot radar scan points on the image and sequentially connect them to form obstacle contours. This method also generalizes across radar resolutions, as demonstrated in Figure 12, which includes scans from three different radar resolutions, showing minimal differences in the connected contours.

(3) Inspired by [17], we design an exponential decay fusion algorithm to integrate historical radar frames for motion trend analysis. Let the $t$-th radar image be a matrix $R{I}^t$, with decay coefficient ${\gamma }_{ri}$. The historically fused radar image $R{I}_H^t$ is computed as:

$$\begin{array}{c}R{I}_H^t=\mathrm{clip}\left({\gamma }_{ri}R{I}^t+R{I}_H^{t-1}, 0, 255\right)\end{array}$$

(26)

where pixel values are clipped to $\left[0,255\right]$ for numerical stability. (4) Finally, a safety radius (represented as a central circular region) and a real-time heading vector (a pointer in the central circular region) are overlaid onto the radar image. The entire process is illustrated in Figure 13.

For the low-dimensional motion parameter layer, with a fixed initial position and heading, target-related encoding is unnecessary. The motion state is directly represented by raw kinematic vectors:

$$\langle \begin{array}{c}x,y,\psi ,\dot{x},\dot{y},\dot{\psi }\end{array}\rangle$$

(27)

3.3.4. Reward Function

Due to the decoupling of the obstacle avoidance system and the navigation system, the agent faces a lack of heading decision information during obstacle avoidance, which may result in local optima (e.g., spinning in place to avoid obstacles). To address this, we propose a heading-lock strategy based on the first observation of obstacles. Upon initial obstacle detection, the current heading angle and coordinates of the USV are recorded as reference direction ${\theta }_{ref}$ and reference position $\left(x_{ref},y_{ref}\right)$. Advancing in reference direction will be rewarded positively while deviating from it will be penalized negatively. Based on this principle, the auxiliary reward components are defined as:

$$\begin{array}{c}\left\{\begin{array}{c}{r}_6={\displaystyle \frac{s-s_{max}}{3.0}}\\ r_7=e^{-3.0\left|d_{lat}\right|}-{\displaystyle \frac{\left|d_{lat}\right|}{1.0}}\\ r_8=e^{-5.0\left|\psi -{\theta }_{ref}\right|}-{\displaystyle \frac{\left|\psi -{\theta }_{ref}\right|}{\frac{\pi }{6}}}\\ r_9=\left\{\begin{array}{c}+1, \mathrm{if} \mathrm{follows} \mathrm{COLREGs}\\ -1, \mathrm{else}\end{array}\right.\end{array}\right.\end{array}$$

(28)

where $s$ is the distance the USV has advanced in the reference direction from the reference position and $s_{max}$ denotes the maximum recorded distance historically, updated as $s_{max}^t=\max \left(s_{max}^{t-1},s\right)$. $d_{lat}$ is the lateral offset of the USV relative to the reference path defined by ${\theta }_{ref}$ and $\left(x_{ref},y_{ref}\right)$. As mentioned in Section 3.2.1, the initial coordinates of the USV in the environment are fixed at the origin, the initial heading is fixed north, and the USV encounters an obstacle at the initial moment. Thus, $\left(x_{ref},y_{ref}\right)=\left(0,0\right)$, ${\theta }_{ref}={\displaystyle \frac{\pi }{2}}$, and the remaining variables can be simplified as:

$$\begin{array}{c}s=y\end{array}$$

(29)

$$\begin{array}{c}{d}_{lat}=x\end{array}$$

(30)

$r_6$ provides incentives only when the agent continuously advances while penalizing any backward movement to restrict behaviors such as reversing to avoid obstacles. $r_7$ limits the USV’s velocity angle deviation and $r_8$ restricts the lateral offset distance. $r_9$ encourages the agent to follow COLREGs rules for obstacle avoidance.

The total auxiliary reward is expressed as:

$$\begin{array}{c}{r}^{ao}=r_6+0.5r_7+0.5r_8+r_9\end{array}$$

(31)

When the agent collides with an obstacle, a terminal reward $r_f^{ao}=-250$ is given. When the agent avoids the obstacle, the terminal reward is set to $r_v^{ao}=0$. This is to prevent the agent from prematurely rushing away from the obstacle to obtain the terminal reward while emphasizing the importance of advancing and adhering to rules during obstacle avoidance.

3.3.5. Deep Neural Network

The neural network architecture designed for the obstacle avoidance task is shown in Figure 14. In this network, CNN is used to process radar image input. The input undergoes sequential processing through multiple convolution-pooling layers, progressively reducing its dimension from 256 × 256 to a compact 4 × 4 × 64 feature representation. Notably, this feature extraction module is shared between the Actor and Critic components. The resulting lower-dimensional image features are then concatenated with low-dimensional motion state data. These combined features are then passed through several fully connected layers with decreasing widths, ultimately producing the final output. Due to the large size of the input images, the low-dimensional data is fed into different layers of fully connected layers to accelerate convergence [32].

3.4. Design of Decision and Encoding Unit

As shown in Figure 4, a critical component of this decoupled architecture, in addition to independent modules, is the decision and encoding unit (marked as “decide and encode”), which serves as the centralized coordinator of the entire system. The decision unit determines when to activate specific modules, while the encoding unit bridges the gap between independently trained modules and real-world environments by converting environmental observations into suitable module inputs. By designing tailored decisions and encoding units for different tasks, the architecture achieves versatility. For the multi-buoy inspection tasks in this study, the unit operates as follows.

For the decision unit, the agent first activates the circular navigation module, guiding the USV from the starting point to each buoy for inspection. Once all buoys are inspected, the USV initiates a return-to-home task, triggering a switch to the goal-directed module. The task concludes when the USV reaches its origin. Throughout the process, if radar detects approaching obstacles, the agent immediately switches to the obstacle avoidance module.

For the encoding unit, it is worth noting that when switching to the obstacle avoidance module, a state transformation is required due to the fixed coordinate and heading initialization. As shown in Figure 15a, a local coordinate system is established with the reference position as the origin and the reference heading as the $y$-axis. The motion vector in the world frame, $\langle x,y,\psi ,\dot{x},\dot{y},\dot{\psi }\rangle$, is transformed into the new coordinate frame as $\langle x^{\prime },y^{\prime },{\psi }^{\prime },{\dot{x}}^{\prime },{\dot{y}}^{\prime },{\dot{\psi }}^{\prime }\rangle$:

$$\begin{array}{c}\left\{\begin{array}{c}{x}^{\prime }=\left(x-x_{ref}\right)\cos \left({\displaystyle \frac{\pi }{2}}-{\theta }_{ref}\right)-\left(y-y_{ref}\right)\sin \left({\displaystyle \frac{\pi }{2}}-{\theta }_{ref}\right)\\ y^{\prime }=\left(x-x_{ref}\right)\sin \left({\displaystyle \frac{\pi }{2}}-{\theta }_{ref}\right)+\left(y-y_{ref}\right)\cos \left({\displaystyle \frac{\pi }{2}}-{\theta }_{ref}\right)\\ {\psi }^{\prime }=\left(\psi -{\theta }_{ref}+{\displaystyle \frac{\pi }{2}}\right) mod 2\pi \end{array}\right.\end{array}$$

(32)

$$\begin{array}{c}\left[\begin{array}{c}{\dot{x}}^{\prime }\\ {\dot{y}}^{\prime }\\ {\dot{\psi }}^{\prime }\end{array}\right]=T\left({\displaystyle \frac{\pi }{2}}-{\theta }_{ref}\right)\left[\begin{array}{c}\dot{x}\\ \dot{y}\\ \dot{\psi }\end{array}\right]\end{array}$$

(33)

By substituting the transformed motion vector for the original motion vector in Equations (25)–(27), the input is aligned with the obstacle avoidance module’s training dataset.

4. Training and Validation

4.1. Training Results

Based on the OpenAI Gym framework, this study developed a virtual simulation platform for USV inspection. This platform integrates real-time trajectory monitoring and radar scanning functionalities, enabling the synchronous visualization of both the USV’s navigation status and environmental perception data. For the three subtasks involved in the inspection mission, neural network models are trained independently using the PyTorch deep learning framework (version 2.5.0). All training tasks were conducted on a server equipped with an NVIDIA RTX 4060 Ti GPU, an Intel i5-12500H CPU, and 32 GB of DDR4 memory. The detailed configuration of environment interaction parameters and PPO hyperparameters is provided in

Table 1. Settings of environment interaction parameters.

Parameter	Value
$E^g$	0.5 m
$R_r$	5.0 m
$D_t$	0.5 m
$E^c$	0.01 rad
$R_{buoy}$	0.8 m
$N_{scan}$	360
$R_{radar}$	10.0 m
$R_{obstacle}$	0.75 m
$R_{usv}$	1.0 m
${\gamma }_{ri}$	0.8 m

and

Table 2. Settings of PPO hyperparameters.

Hyperparameter	Value *
Training episodes	2000/8000
Experience buffer size	2048/1024
Max timesteps	500/100
Update epochs	10/10
Learning rate	2 × 10−4/2 × 10−4
$\gamma$	0.99/0.99
$\lambda$	0.95/0.95
$\epsilon$	0.2/0.1

* Value for navigation/value for obstacle avoidance.

.

At the end of each training iteration, the cumulative discounted reward obtained by the agent during that iteration is recorded. To mitigate the impact of random environment initialization, a 200-step moving average is applied. The reward curves for the three subtasks are shown in Figure 16. The results demonstrate that the reward curves for all subtasks exhibit convergence characteristics. The rewards stabilize after a certain number of iterations and eventually reach a steady-state value, indicating that the policies have converged effectively.

4.2. Validation Results

Validation scenarios are independently designed for each of the three subtasks to evaluate the effectiveness of the trained policies. Additionally, a comprehensive multi-buoy inspection scenario is proposed to test the cooperative performance of the obstacle avoidance algorithm alongside the navigation algorithms. In all scenarios, there exists a wind disturbance with a constant speed of 1.0 m/s and a direction consistently oriented to the right.

4.2.1. Validations for Goal-Directed Navigation

To validate the goal-directed navigation algorithm, this study designs two test scenarios: In the first scenario, the initial position of the USV is set 100 m away from the goal point (42.9% beyond the maximum test distance of 70 m in the training set), to validate the algorithm’s generalization capability in unknown states. The second scenario positions the USV’s initial location adjacent to the goal point and with its heading direction facing away from the goal, testing the agent’s autonomous adjustment ability under extreme pose conditions. The test results are shown in Figure 17. The results demonstrate that the goal-directed navigation algorithm successfully reached the destination range in both scenarios. The first scenario proves that the algorithm can adjust its course and proceed toward the target even in unseen situations. The second scenario shows its capability to handle extreme USV attitudes. The smooth paths generated in both cases confirm the algorithm’s excellent control performance.

4.2.2. Validations for Circular Navigation

Two scenarios have been designed for the circular navigation algorithm. In the first scenario, the initial pose of the USV is placed inside the expected circumnavigation trajectory to examine the algorithm’s path recovery capability. In the second scenario, the required orbital radius is set to 10 m (twice the training standard of 5 m) to verify the generalization ability of the algorithm. The test results are shown in Figure 18. The results demonstrate that the circular navigation algorithm can achieve smooth transitions from linear to circular paths and maintain trajectory error within the required threshold, even under task requirements that differ from the training environment, thereby validating the effectiveness of the proposed angular deviation weighting mechanism.

To highlight the advantage of the PPO-based circular navigation algorithm, a comparative experiment is conducted with the traditional LOS navigation algorithm. The parameter configurations of LOS are detailed in

Table 3. Settings of LOS parameters.

Parameter	Value
Lookahead distance	10 m
Number of Waypoints	24
Path Following Tolerance	1.0 m
$K_P$	3.0
$K_D$	1.8

, while the trajectory comparison and lateral error analysis of the two algorithms are shown in Figure 19.

The experimental results indicate that, compared to the traditional LOS algorithm, the PPO-based algorithm exhibits smoother trajectories with fewer trajectory oscillations.

4.2.3. Validations for Obstacle Avoidance

For the obstacle avoidance algorithm, this study validates the performance under four typical scenarios designed in compliance with COLREGs rules.

The visualization results of obstacle avoidance trajectories are presented in Figure 20. The experimental results indicate that, except for the left-crossing scenario, the agent effectively adheres to COLREGs rules and completes avoidance maneuvers in the other three scenarios. Further analysis reveals that in the left-crossing scenario, COLREGs explicitly require the own ship to maintain its course and speed, placing the responsibility of avoidance on the obstacle vessel. However, the obstacle vessel maintains a constant heading, creating a fundamental conflict between the rule requirements and the simulation environment. Faced with this contradiction, to avoid collision risks and associated negative rewards, the agent autonomously opts to violate the rule and execute an avoidance maneuver.

4.2.4. Validation for Multi-Buoy Inspection

To investigate the collaborative performance of independently trained obstacle avoidance and navigation modules, as well as to comprehensively evaluate the autonomous control capabilities of the agent in a complete multi-buoy inspection task, we designed a multi-buoy inspection scenario. This scenario includes two buoys. The USV departs from the starting point, inspects the buoys in a predefined sequence, and returns to the origin after completing the inspection, all while dynamically avoiding appearing obstacles.

The simulation result is shown in Figure 21a. In this test, obstacles are generated just outside the USV’s radar detection range during movement. The first obstacle appears during the USV’s transit to the first buoy, simulating a head-on scenario. The second obstacle appears during transit to the second buoy, simulating a right-crossing scenario where the USV navigates a channel and encounters a crossing obstacle. In both cases, the USV should steer right to avoid collisions in accordance with COLREGs. Experimental results demonstrate that the USV consistently adopted the correct right-steering strategy without exhibiting excessive avoidance maneuvers like course reversal in both encountering scenarios. This confirms that our fixed initialization training, through state transformation, maintains generalization capability.

Different modules are activated to accomplish this task depending on the task progress and obstacle detection, as detailed in Section 3.4. The module-switching workflow is illustrated in Figure 21b, where the red trajectory is generated by the obstacle avoidance module, the blue trajectory by the circular navigation module, and the yellow trajectory by the goal-directed navigation module. The results demonstrate that the module-switching process performs consistently with the theoretical expectations.

5. Conclusions

In this paper, we use reinforcement learning methods to improve the navigation and obstacle avoidance performance of the autonomous buoy inspection USV system. Due to the complexity of the multi-buoy inspection tasks, we designed a decoupled architecture for navigation and obstacle avoidance and trained separate modules for each subtask. For the circular navigation module, we proposed an angle-weighted approach and independent experiments have demonstrated that this module enables smooth transitions. For the obstacle avoidance module, we designed a radar image encoding technique, and independent experiments have shown that it can partially comply with COLREGs for obstacle avoidance. Finally, a module-switching and state-transformation component tailored for multi-buoy inspection tasks is proposed, and integrated module experiments are conducted. The results show that with this component and the modules, the entire buoy inspection process can be fully automated, thereby proving the effectiveness of the proposed architecture.

Moving forward, we will enhance the obstacle avoidance module for full COLREGs compliance. For practical implementation, our algorithm will incorporate variables like obstacle vessel dynamics (speed, heading, dimensions), environmental visibility, and external interference to improve robustness. While initially designed for inspection tasks, the architecture demonstrates strong adaptability. We plan to expand its applications through sensor-specific avoidance modules and the integration of additional navigation modules to handle more complex tasks.