A Comprehensive Review of Deep Learning Techniques in Mobile Robot Path Planning: Categorization and Analysis

Abstract

Deep Reinforcement Learning (DRL) has emerged as a transformative approach in mobile robot path planning, addressing challenges associated with dynamic and uncertain environments. This comprehensive review categorizes and analyzes DRL methodologies, highlighting their effectiveness in navigating high-dimensional state–action spaces and adapting to complex real-world scenarios. The paper explores value-based methods like Deep Q-Networks (DQNs) and policy-based strategies such as Proximal Policy Optimization (PPO) and Soft Actor–Critic (SAC), emphasizing their contributions to efficient and robust navigation. Hybrid approaches combining these methodologies are also discussed for their adaptability and enhanced performance. Additionally, the review identifies critical gaps in current research, including limitations in scalability, safety, and generalization, proposing future directions to advance the field. This work underscores the transformative potential of DRL in revolutionizing mobile robot navigation across diverse applications, from search-and-rescue missions to autonomous urban delivery systems.

1. Introduction

Reinforcement Learning (RL) has emerged as a powerful framework for solving sequential decision-making problems, where an agent interacts with an environment to learn optimal behaviors by maximizing cumulative rewards [1]. This ability to adapt through interaction makes RL particularly suitable for mobile robot path planning in dynamic and uncertain environments. Traditional path planning algorithms, such as A* and Dijkstra’s, excel in static and well-structured settings but falter in applications characterized by uncertainty, unpredictability, and incomplete knowledge of the environment. For instance, search-and-rescue operations in disaster-stricken areas often involve navigating through unknown terrains filled with debris and dynamic obstacles. Similarly, autonomous delivery robots operating in crowded urban environments must make real-time decisions to avoid pedestrians and other obstacles while efficiently reaching their destinations. In such scenarios, RL offers a robust solution by learning policies that directly map sensory inputs to actions, enabling real-time decision-making and adaptability [2].

Deep Reinforcement Learning (DRL), a subset of RL that leverages deep neural networks, has significantly advanced the capabilities of RL by addressing challenges associated with high-dimensional state–action spaces and continuous control. Unlike traditional RL, which struggles to scale in complex environments, DRL employs neural networks to approximate value functions and policies, enabling efficient learning and decision-making in scenarios involving raw sensory inputs like camera images or LiDAR data [3]. This scalability and versatility make DRL an ideal framework for tackling complex navigation problems, such as autonomous navigation in cluttered indoor spaces, multi-agent coordination, and socially aware robot path planning. Techniques, such as Proximal Policy Optimization (PPO) [4], Deep Deterministic Policy Gradient (DDPG) [3], and Soft Actor–Critic (SAC) [2] have demonstrated remarkable performance in such applications, achieving state-of-the-art results across a variety of benchmark environments.

This paper presents a comprehensive review of DRL techniques for mobile robot path planning, categorizing existing methods and analyzing their strengths and limitations. It explores how DRL can address challenges in path planning, including navigation in dynamic and uncertain environments, balancing efficiency with safety, and generalizing to unseen scenarios. The review focuses on the application of DRL in tasks requiring continuous control, adaptive decision-making, and integration with multimodal sensory inputs.

Several reviews of DRL solutions for mobile robot path planning have been presented in the literature, offering broad overviews of existing methods and their applications. However, this work differentiates itself by providing a structured and standardized analysis of DRL techniques through a unified notation framework. Specifically, we categorize various DRL methods and reformulate their mathematical formulations under a consistent notation, facilitating direct comparisons across different approaches. Furthermore, for each category, multiple representative solutions are analyzed, with their key components restructured to maintain uniformity in presentation. A notable contribution of this review is the inclusion of original block diagrams and pseudocode representations for nearly all solutions discussed, offering a clearer understanding of their working principles and implementation details. These enhancements provide a more accessible and structured perspective on DRL-based path planning, allowing researchers and practitioners to better interpret, compare, and apply these methodologies in real-world scenarios.

The rest of this paper is organized as follows: In Section 2, the problem of designing a reinforcement learning solution for mobile robot path planning is formally defined using a differential drive robot as an illustrative example and detailing why reinforcement learning is advantageous. Then, Section 3 explores deep reinforcement learning, and discusses how uncertainties are modeled and handled in DRL with a note on sensor fusion for enhanced path planning using DRL. A categorization of the methods discussed in the next sections is also presented. Section 4 examines value-based DRL approaches such as DQN and D3QN, followed by Section 5 where policy-based methods are introduced. Recent actor–critic type methods, including DDPG, A3C, and PPO, are highlighted as the main developments in DRL for robot path planning, with several contemporary studies reviewed under each sub-category in Section 6. Some emerging trends in DRL for robotics are discussed in Section 7, including transformer-based DRL, metal learning for transfer learning, multi-agent methods, and attention mechanisms. The section also covers the scaling of DRL solutions and the safety of mobile robot exploration using DRL. Section 8 covers the practical considerations for real-world applications, such as the challenges of hybrid methods, optimization for real-time processing, and empirical performance comparisons. Samples of path planning outcomes from selected works are presented and discussed in Section 9, illustrating the benefits of these deep reinforcement learning strategies. The paper concludes by presenting insights and potential future research directions, in Section 10.

2. Problem Statement

2.1. Reinforcement Learning Framework for Control Systems

Reinforcement Learning (RL) is a mathematical framework used to solve sequential decision-making problems, where an agent interacts with an environment to achieve a specific goal by maximizing a cumulative reward signal. These problems are often formulated as a Markov Decision Process (MDP), which is characterized by the 5-tuple:

$$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma ),$$

(1)

where

• $\mathcal{S}$: The state space, representing all possible states of the environment. A state $s_t\in \mathcal{S}$ encodes all relevant information about the environment at a given time t.
• $\mathcal{A}$: The action space, representing all possible actions that the agent can take. An action $a_t\in \mathcal{A}$ represents a decision or control input applied by the agent at time t.
• $\mathcal{P}(s^{\prime }|s,a)$: The state transition probability distribution, which defines the probability of transitioning from a current state $s\in \mathcal{S}$ to a next state $s^{\prime }\in \mathcal{S}$ as a result of taking action $a\in \mathcal{A}$. This function captures the dynamics of the environment and may be deterministic or stochastic.
• $r(s,a)$: The reward function, which assigns a scalar reward to each state–action pair $(s,a)$. This reward quantifies the immediate desirability of taking action a in state s. For example, it might reflect the closeness to a goal or penalties for undesirable events such as collisions.
• $\gamma \in [0,1)$: The discount factor, which determines the relative importance of immediate versus future rewards. A smaller $\gamma$ places more emphasis on immediate rewards, while a value closer to 1 encourages long-term planning.

The goal of the RL agent is to find an optimal policy $\pi (a|s)$, which defines the probability of taking an action a given a state s, to maximize the expected cumulative discounted reward:

$$J(\pi )={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right],$$

(2)

where $s_t$ and $a_t$ represent the state and action at time step t, respectively.

The quality of a policy $\pi$ is often evaluated using the state–value function $V^{\pi }(s)$ or the action–value function $Q^{\pi }(s,a)$, defined as follows:

$$V^{\pi }(s)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)|s_0=s\right],$$

(3)

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)|s_0=s,a_0=a\right].$$

(4)

The optimal policy ${\pi }^{*}$ maximizes these functions and satisfies the Bellman optimality equation:

$$Q^{*}(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim \mathcal{P}}\left[\underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{*}(s^{\prime },a^{\prime })\right].$$

(5)

The Bellman optimality equation is foundational for reinforcement learning, serving as the theoretical basis for many algorithms aimed at finding optimal policies in sequential decision-making problems. It is rooted in the principle of optimality, which states the following: An optimal policy has the property that, regardless of the initial state and initial decision, the subsequent decisions must constitute an optimal policy with regard to the state resulting from the first decision [1]. This principle allows us to decompose the long-term optimization problem into smaller subproblems, leveraging the recursive nature of the problem. For a given state–action pair $(s,a)$, the equation defines the optimal action–value function $Q^{*}(s,a)$ as the immediate reward plus the maximum possible discounted future reward from subsequent states.

The intuition behind the equation lies in the agent’s sequential decision-making process. At each step, the agent selects an action a in state s to maximize the sum of immediate and future rewards. The equation’s recursive nature means the value of the current state–action pair depends on the values of future states, tying local decisions to global optimization.

To derive the Bellman optimality equation, consider the expected cumulative reward under the optimal policy ${\pi }^{*}$ in Equation (4). This can be decomposed into two parts:

• The immediate reward $r(s,a)$ obtained at $t=0$.
• The discounted expected reward from subsequent states, starting at $t=1$.

By definition of the expected value and the Markov property of the environment:

$$Q^{*}(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim \mathcal{P}}\left[{\mathbb{E}}_{{\pi }^{*}}\left[\sum _{t=1}^{\infty }{\gamma }^{t-1}r(s_t,a_t)|s_1=s^{\prime }\right]\right].$$

(6)

Since the future reward depends only on the subsequent state $s^{\prime }$ and action $a^{\prime }$ under the optimal policy, the inner expectation simplifies to ${max}_{a^{\prime }\in \mathcal{A}}{Q}^{*}(s^{\prime },a^{\prime })$. Thus, the Bellman optimality equation is established.

2.2. Reinforcement Learning for Mobile Robot Path Planning

In the context of mobile robot path planning, the goal is to compute a policy that enables the robot to navigate from a starting position to a target position while avoiding obstacles and minimizing path costs. The MDP formulation of this problem is described as follows:

• State Space ($\mathcal{S}$): Each state $s\in \mathcal{S}$ represents the robot’s current status and environmental context. For example, s may include the robot’s position $(x,y)$, orientation $\theta$, velocities, and potentially sensor measurements, such as distances to nearby obstacles and the relative position of the goal.
• Action Space ($\mathcal{A}$): The action $a\in \mathcal{A}$ represents a control input applied to the robot. For instance, a could define discrete motion commands (e.g., MoveForward, TurnLeft), or continuous control signals such as linear velocity v and angular velocity $\omega$.
• State Transition ($\mathcal{P}(s^{\prime }|s,a)$): This models the robot’s motion dynamics and environmental interactions. For example, if the robot is controlled using differential drive kinematics, the transition from s to $s^{\prime }$ after applying a depends on equations of motion. Stochasticity may arise due to wheel slip, sensor noise, or dynamic obstacles.
• Reward Function ($r(s,a)$): The reward function incentivizes behaviors that achieve the desired goal while avoiding undesirable outcomes.
• Discount Factor ($\gamma$): The discount factor determines how much emphasis is placed on immediate versus long-term navigation performance. For instance, $\gamma$ close to 1 ensures that the robot considers the entire trajectory to the goal.

Using reinforcement learning, the agent iteratively interacts with the environment to learn an optimal policy ${\pi }^{*}$ that selects actions to navigate effectively while avoiding obstacles. State-of-the-art deep reinforcement learning algorithms such as Proximal Policy Optimization (PPO) [4] and Soft Actor–Critic (SAC) [2] are particularly suited for this task, as they handle high-dimensional state–action spaces and continuous control effectively.

• Problem Formulation for a Differential Drive Mobile Robot: 

For a differential-drive mobile robot, the state $s\in \mathcal{S}$ is defined as follows: $s={\left[xy\theta v\omega \right]}^{\top },$ where $x,y$ are the Cartesian coordinates of the robot’s position in a 2D plane, $\theta$ is the orientation (heading angle) of the robot relative to the global frame, v is the linear velocity of the robot, and $\omega$ is the angular velocity of the robot. The control input $a\in \mathcal{A}$ is defined as $a={\left[u_v{u}_{\omega }\right]}^{\top },$ where $u_v$ and $u_{\omega }$ are the commanded linear and angular velocity inputs, respectively. The robot’s motion dynamics can be described using the following system of nonlinear differential equations:

$$\begin{array}{ccccccccc}\hfill \dot{x}& =& vcos\theta ,\hfill & \hfill \dot{y}& =& vsin\theta ,\hfill & \hfill \dot{\theta }& =& \omega ,\hfill \\ \hfill \dot{v}& =& -{\displaystyle \frac{1}{{\tau }_v}}(v-u_v)+{\xi }_v,\hfill & \hfill \dot{\omega }& =& -{\displaystyle \frac{1}{{\tau }_{\omega }}}(\omega -u_{\omega })+{\xi }_{\omega },\hfill & \hfill & \end{array}$$

(7)

where ${\tau }_v,{\tau }_{\omega }$ are the time constants for the linear and angular velocity dynamics, and ${\xi }_v,{\xi }_{\omega }$ are the stochastic disturbances representing noise in the robot’s actuators, modeled as Gaussian random variables, ${\xi }_v\sim \mathcal{N}(0,{\sigma }_v^2)$ and ${\xi }_{\omega }\sim \mathcal{N}(0,{\sigma }_{\omega }^2)$. For practical implementation in reinforcement learning, the continuous time dynamics are discretized using a time step $\Delta t$. The discrete-time state transition equations become the following:

$$\begin{array}{ccc}\hfill x_{t+1}& =& x_t+v_{t}cos{\theta }_t\Delta t,\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill y_{t+1}& =& y_t+v_{t}sin{\theta }_t\Delta t,\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill {\theta }_{t+1}& =& {\theta }_t+{\omega }_t\Delta t,\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill v_{t+1}& =& v_t+\left(-{\displaystyle \frac{1}{{\tau }_v}}(v_t-u_v)+{\xi }_v\right)\Delta t,\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\omega }_{t+1}& =& {\omega }_t+\left(-{\displaystyle \frac{1}{{\tau }_{\omega }}}({\omega }_t-u_{\omega })+{\xi }_{\omega }\right)\Delta t.\hfill \end{array}$$

(12)

The robot operates in an environment containing obstacles and goals, which influence the state transition. For example,

• If the robot’s position $(x_{t+1},y_{t+1})$ lies within an obstacle region, the state transition is interrupted, and the robot enters a collision state with a penalty reward.
• Sensor noise can introduce uncertainty in the perceived state, modeled as an additive Gaussian noise $\eta \sim \mathcal{N}(0,\mathrm{\Sigma })$ applied to the observed position and orientation:

  $${\tilde{s}}_{t+1}=s_{t+1}+\eta ,$$

  (13)

  where ${\tilde{s}}_{t+1}$ is the observed state (assuming that the sensing mechanism includes a complete GPS-INS).

For the differential-drive mobile robot example, the deterministic component of the state transition is governed by the discretized nonlinear dynamics described earlier. However, to account for stochastic effects in the real-world environment, the state transition probability density can be expressed as follows:

$$\mathcal{P}(s^{\prime }|s,a)=\mathcal{N}(s^{\prime };f(s,a),\mathrm{\Sigma }),$$

(14)

where

• $f(s,a)$ is the deterministic state transition function derived from the discretized equations of motion,

  $$f(s,a)=\left[\begin{array}{c}{x}_t+v_{t}cos{\theta }_t\Delta t\\ y_t+v_{t}sin{\theta }_t\Delta t\\ {\theta }_t+{\omega }_t\Delta t\\ v_t+\left(-{\displaystyle \frac{1}{{\tau }_v}}(v_t-u_v)+{\xi }_v\right)\Delta t\\ {\omega }_t+\left(-{\displaystyle \frac{1}{{\tau }_{\omega }}}({\omega }_t-u_{\omega })+{\xi }_{\omega }\right)\Delta t\end{array}\right],$$

  (15)
• $\mathrm{\Sigma }$ is the covariance matrix representing the stochastic uncertainties in the state transition, which could arise from sensor noise, actuator variability, or environmental factors.

The covariance matrix $\mathrm{\Sigma }$ is often diagonal under the assumption that the noise in different state dimensions is uncorrelated. For instance,

$$\mathrm{\Sigma }=\mathrm{diag}({\sigma }_x^2,{\sigma }_y^2,{\sigma }_{\theta }^2,{\sigma }_v^2,{\sigma }_{\omega }^2),$$

(16)

where ${\sigma }_x^2,{\sigma }_y^2$ represent the positional uncertainties, ${\sigma }_{\theta }^2$ the orientation uncertainty, and ${\sigma }_v^2,{\sigma }_{\omega }^2$ the variances in the linear and angular velocities due to stochastic disturbances.

The environment in which the robot operates significantly affects the state transition function. In the presence of obstacles or terrain constraints, the deterministic dynamics $f(s,a)$ may no longer accurately describe the transition and $\mathcal{P}(s^{\prime }|s,a)$ must be adjusted accordingly. For example,

• Obstacle Collisions: If the predicted state $s^{\prime }={(x^{\prime },y^{\prime },{\theta }^{\prime },v^{\prime },{\omega }^{\prime })}^{\top }$ lies within an obstacle region, the state transition function could assign zero probability to this transition and redirect the robot to a collision state $s_{\mathrm{collision}}$. Mathematically,

  $$\mathcal{P}(s^{\prime }|s,a)=\left\{\begin{array}{cc}\mathcal{N}(s^{\prime };f(s,a),\mathrm{\Sigma }),\hfill & \mathrm{if}(x^{\prime },y^{\prime })\notin \mathcal{O},\hfill \\ \delta (s^{\prime }-s_{\mathrm{collision}}),\hfill & \mathrm{if}(x^{\prime },y^{\prime })\in \mathcal{O},\hfill \end{array}\right.$$

  (17)

  where $\mathcal{O}$ is the set of obstacle regions, and $\delta (·)$ is the Dirac delta function centered at the collision state.
• Dynamic Obstacles: In environments with moving obstacles, the state transition must account for dynamic constraints imposed by the obstacle’s trajectory. For instance, if the robot is predicted to intersect with an obstacle at time $t+1$, the state transition probability can penalize such trajectories by assigning lower likelihoods:

  $$\mathcal{P}(s^{\prime }|s,a)\propto exp\left(-{\displaystyle \frac{\mathrm{dist}{(s^{\prime },{\mathcal{O}}_t)}^2}{2{\sigma }_{\mathrm{obs}}^2}}\right),$$

  (18)

  where $\mathrm{dist}(s^{\prime },{\mathcal{O}}_t)$ is the distance between the predicted state and the obstacle’s trajectory at time t, and ${\sigma }_{\mathrm{obs}}$ controls the sensitivity to obstacle proximity.

In real-world scenarios, the robot’s sensors introduce noise, leading to uncertainty in the observed states. The state transition probability density can be modified to include the perception model:

$$\mathcal{P}({\tilde{s}}^{\prime }|s,a)=\int \mathcal{P}({\tilde{s}}^{\prime }|s^{\prime })\mathcal{P}(s^{\prime }|s,a)d{s}^{\prime },$$

(19)

where $\mathcal{P}({\tilde{s}}^{\prime }|s^{\prime })$ is the sensor noise model, typically modeled as Gaussian:

$$\mathcal{P}({\tilde{s}}^{\prime }|s^{\prime })=\mathcal{N}({\tilde{s}}^{\prime };s^{\prime },{\mathrm{\Sigma }}_{\mathrm{sensor}}).$$

(20)

The reward function is designed to encourage goal-reaching behavior, penalize collisions, and promote efficient navigation. For the differential drive mobile robot model presented above, such a reward function can be expressed as follows:

$$r(s,a)=\left\{\begin{array}{cc}+R_g,\hfill & {\mathrm{if}\parallel \left[xy\right]}^{\top }-{\left[x_g{y}_g\right]}^{\top }\parallel \le ϵ,\hfill \\ -R_c,\hfill & \mathrm{if}(x,y)\in \mathcal{O},\hfill \\ {-\alpha \parallel \left[xy\right]}^{\top }-{\left[x_g{y}_g\right]}^{\top }{\parallel -\beta \parallel a\parallel }^2,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(21)

where

–

${\left[xy\right]}^{\top }$ is the robot’s current position in the state $s={\left[xy\theta v\omega \right]}^{\top }$,

–

${\left[x_g{y}_g\right]}^{\top }$ is the goal position in the global frame,

–

$ϵ$ is a small threshold to determine if the robot has reached the goal (e.g., within a radius of $ϵ$ around the goal position),

–

$R_g$ is a positive scalar reward assigned when the robot reaches the goal,

–

$R_c$ is a scalar penalty assigned if the robot’s position lies within an obstacle region $\mathcal{O}$,

–

$\alpha$ is a weighting factor penalizing the Euclidean distance between the robot’s current position and the goal,

–

$\beta$ is a weighting factor penalizing the energy expenditure represented by the squared magnitude of the action vector $a={\left[u_v{u}_{\omega }\right]}^{\top }$.

2.3. Why Reinforcement Learning?

Traditional path planning methods, such as A* and Dijkstra’s algorithm, are primarily designed for static environments where obstacles and goals are fixed and known a priori [5]. While extensions to these methods, such as dynamic A* (D*), allow for some level of adaptability [6], they often require substantial computational overhead for replanning. In contrast, RL inherently learns a policy $\pi (a|s)$ that maps states to actions, enabling real-time decision-making without explicit replanning. Indeed, the optimal policy in RL, ${\pi }^{*}(a|s)$, which is obtained by solving

$${\pi }^{*}(a|s)=arg\underset{\pi }{max}{\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right],$$

(22)

directly accounts for environment dynamics through interaction during the training phase [1]. This allows the robot to handle scenarios where the obstacle set ${\mathcal{O}}_t$ or the goal position ${[x_g,y_g]}^{\top }$ varies over time.

Another advantage of RL stems from its optimality in applications involving environments with complex geometries or high-dimensional state spaces. In such applications, traditional methods often struggle due to the exponential growth of the search space [7]. For instance, sampling-based methods like Rapidly-Exploring Random Trees (RRT) can provide feasible paths but are not guaranteed to find optimal paths without significant computational effort. Reinforcement Learning, particularly with Deep RL, excels in such scenarios by approximating the value function $V^{\pi }(s)$ or action–value function $Q^{\pi }(s,a)$ using neural networks [3,8]. Algorithms such as Deep Q-Networks (DQN) or Proximal Policy Optimization (PPO) leverage the Bellman optimality principle (see Equation (5)) to find near-optimal policies efficiently, even in high-dimensional settings [4].

Finally, in real-world applications, robots face significant uncertainties, such as sensor noise, actuator variability, and unpredictable environmental dynamics. While traditional methods can handle some uncertainties through deterministic approximations, they lack the robustness to stochastic effects. In RL, stochasticity is inherently modeled in the state transition function $\mathcal{P}(s^{\prime }|s,a)$ and reward function $r(s,a)$. For example, the dynamics of a differential-drive mobile robot are represented as $\mathcal{P}(s^{\prime }|s,a)=\mathcal{N}(s^{\prime };f(s,a),\mathrm{\Sigma }),$ where $\mathrm{\Sigma }$ captures uncertainties in motion. The RL agent learns to optimize its policy under such probabilistic dynamics, making it robust to real-world noise [2].

Reinforcement Learning is particularly advantageous in several applications and scenarios. In navigation through unknown environments, where the environment is partially known or entirely unknown, RL allows the robot to learn an effective policy through exploration and interaction with the environment [9]. This capability is crucial for tasks such as search-and-rescue operations. In scenarios involving dynamic obstacle avoidance, RL enables the robot to predict and react to dynamic changes in the environment by leveraging the policy $\pi (a|s)$ trained on similar conditions. RL is also effective in multi-objective path planning, allowing for the optimization of complex reward functions that balance competing objectives, such as minimizing energy consumption, maximizing safety, and achieving timely goal completion:

$${r(s,a)=-\alpha \parallel \left[xy\right]}^{\top }-{\left[x_g{y}_g\right]}^{\top }{\parallel -\beta \parallel a\parallel }^2-{\gamma }_{\mathrm{collision}}\mathbb{I}\{(x,y)\in \mathcal{O}\},$$

(23)

where $\mathbb{I}$ is an indicator function penalizing collisions. Additionally, RL can be integrated with imitation learning, enabling robots to learn policies from human demonstrations or expert-generated trajectories, which reduces the need for manual design of path planning algorithms [10].

3. Deep Reinforcement Learning

Deep Reinforcement Learning (Deep RL) leverages deep neural networks as function approximators to address high-dimensional state and action spaces. In scenarios, such as mobile robot path planning, where the state space $\mathcal{S}$ may include sensor data, such as images or LIDAR scans, and the action space $\mathcal{A}$ involves continuous control, Deep RL provides a scalable and effective solution [3,8]. The key elements of Deep RL build directly on the Markov Decision Process (MDP) framework and extend it by introducing function approximation methods.

In Deep RL, the policy $\pi (a|s;\theta )$ is parameterized by a neural network with parameters $\theta$. The policy maps states $s\in \mathcal{S}$ to a probability distribution over actions $a\in \mathcal{A}$. The objective remains to optimize the parameters $\theta$ to maximize the expected cumulative reward (2):

$$J({\pi }_{\theta })={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right],$$

(24)

where $\gamma$ is the discount factor, as defined previously [1].

In addition to the policy, in deep RL, the state–value function $V^{\pi }(s)$ and action–value function $Q^{\pi }(s,a)$, which are critical for evaluating the quality of a policy or action, can be approximated using neural networks:

$$Q(s,a;\varphi )\approx Q^{\pi }(s,a),$$

(25)

where $\varphi$ represents the parameters of the neural network. Algorithms such as Deep Q-Networks (DQN) [8] use this approximation to solve the Bellman equation:

$$Q(s,a;\varphi )=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim \mathcal{P}}\left[\underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime };\varphi )\right].$$

(26)

When deep RL combines a policy network (actor) $\pi (a|s;\theta )$ with a value network (critic) $Q(s,a;\varphi )$, the actor selects actions, while the critic evaluates them. This approach is called the actor–critic paradigm. The policy gradient is computed as follows:

$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }log{\pi }_{\theta }(a|s)Q^{\pi }(s,a)\right],$$

(27)

where $Q^{\pi }(s,a)$ is provided by the critic network [2,4].

Deep RL also addresses the balance between exploration (choosing less certain actions to gather information) and exploitation (choosing actions that maximize known rewards). Techniques such as $ϵ$-greedy policies [1,8] or entropy regularization [2] are commonly used to manage this trade-off. To stabilize learning, deep RL often employs mechanisms such as target networks and experience replay. Target networks provide a stable reference for value updates, and experience replay reuses past experiences for training, improving sample efficiency [8].

The key components of a Deep Reinforcement Learning (Deep RL) framework and their interactions are illustrated in Figure 1. The interaction flow in the diagram starts with the state $s\in \mathcal{S}$, which is processed by both the policy network (actor) and value network. The policy network generates an action $a\in \mathcal{A}$, which interacts with the environment to produce a reward $r(s,a)$ and a new state $s^{\prime }$. The value network evaluates $Q(s,a;\varphi )$ to inform the critic. The critic provides feedback for updating both the value network and the policy network, ensuring iterative improvements in both evaluation and decision-making.

3.1. Modeling and Handling Uncertainty in Deep Reinforcement Learning

Uncertainty is an inherent challenge in real-world robotic applications, arising from sensor noise, dynamic obstacles, and environmental unpredictability. Traditional reinforcement learning models assume a stationary Markov decision process (MDP), which often fails to capture the stochastic nature of real-world interactions. To address these issues, researchers have proposed several uncertainty-aware methods to improve the robustness and adaptability of DRL-based robotic systems.

• Bayesian reinforcement learning  

Bayesian deep reinforcement learning (BDRL) has been widely explored as a means to incorporate model uncertainty into decision-making. Zheng et al. [11] introduced a Bayesian DRL framework that quantifies model uncertainty, aleatoric uncertainty, and reward function uncertainty in robot manipulation tasks. Their findings demonstrated that incorporating Bayesian networks into DRL significantly enhances convergence speed in sparse reward environments.

• Uncertainty-aware policy optimization  

Robust policy optimization techniques have been developed to mitigate the performance degradation caused by uncertain transitions. Zhang et al. [12] proposed an uncertainty set regularization (USR) technique, which formulates an uncertainty-aware policy using an adversarial approach to model unknown transition uncertainties. Their results on real-world reinforcement learning benchmarks showed improved robustness against environmental perturbations.

• Trajectory planning under uncertain constraints  

Trajectory planning in dynamic environments requires DRL agents to anticipate and adapt to uncertain constraints in real-time. Govindarajula1 et al. [13] presented a trajectory planning framework that integrates deep learning models with reinforcement learning to handle dynamic obstacles and unpredictable environments effectively. Their approach demonstrated improved safety and efficiency in both simulated and real-world scenarios.

• Sim-to-real adaptation for handling sensor noise  

Handling sensor noise is crucial for ensuring stable DRL deployment in robotic navigation. Joshi et al. [14] analyzed the impact of sensor noise on UAV navigation and obstacle avoidance using DRL. Their results indicated that denoising techniques such as Kalman filtering and artificially injecting noise during training improved real-world performance. Similarly, Martinez et al. [15] developed a velocity-space-based DRL planner that mitigates discrepancies between simulation and reality, reducing failure rates in highly dynamic environments.

• Deep stochastic Koopman operator for uncertain systems  

A novel data-driven stochastic modeling approach was introduced by Han et al. [16], which utilizes a deep stochastic Koopman operator (DeSKO) for robust learning control in uncertain nonlinear robotic systems. Their framework achieved robust stability in real-world soft robotics applications, demonstrating resilience against unexpected disturbances.

These advancements illustrate the importance of uncertainty-aware DRL techniques in real-world robotics. Future research should focus on improving generalization across diverse operational conditions while maintaining computational efficiency.

3.2. Sensor Fusion for Enhanced Path Planning in DRL

Sensor fusion has become a crucial component in reinforcement learning-based robot navigation, allowing autonomous agents to integrate multiple sensory inputs to improve decision-making in complex environments. Combining LiDAR, camera images, and IMU data enhances depth perception, spatial awareness, and obstacle avoidance, leading to improved policy robustness in deep reinforcement learning (DRL).

Recent studies have demonstrated the effectiveness of LiDAR–camera fusion in DRL-based navigation. Ou et al. [17] introduced a sensor fusion framework that integrates LiDAR and camera data to establish a more reliable environmental representation. Their approach enables mobile robots to navigate unfamiliar environments more efficiently than traditional single-sensor setups. Additionally, Tan [18] designed a lightweight multimodal data fusion network that combines LiDAR and image-based semantic segmentation, bridging the gap between simulated and real-world navigation.

Incorporating IMU data into sensor fusion frameworks further enhances DRL-based path planning by providing motion stability and aiding localization. Xue and Gonsalves [19] proposed a short-term visual-IMU fusion memory agent for drone motion planning, significantly reducing neural network complexity while maintaining robust obstacle avoidance. Similarly, Liu et al. [20] developed a sensor fusion system that integrates IMU with LiDAR-based SLAM to improve 3D color reconstruction in large-scale environments.

Several DRL frameworks now explicitly incorporate sensor fusion to improve real-time navigation. Yan et al. [21] introduced a deep deterministic policy gradient (DDPG)-based approach that fuses laser and vision sensor data, achieving a 10% improvement in navigation success rates in dynamic environments. Furthermore, Jiang et al. [22] developed a multi-sensor fusion framework that preprocesses RGB and depth images from cameras before integrating them with LiDAR data, significantly improving obstacle avoidance in real-world scenarios.

These advancements highlight the growing importance of sensor fusion in DRL-based robotics. Future research should focus on optimizing real-time data processing pipelines and developing lightweight fusion architectures to enhance computational efficiency.

3.3. Categorization

Our investigations have resulted in the categorization of recent DRL solutions for mobile robot path planning into three general categories: value-based method, policy-based methods and hybrid actor–critic methods. In the next three sections, we review each of these categories. Since the last category is more recent and has shown an exceedingly large potential for the guidance and control of mobile robots, we provide an extensive exploration of different Actor–Critic algorithms that in their nature employ a combination of the principles used in value-based and policy-based approaches to DRL for mobile robot path planning. To provide a clearer understanding of the relationships between different DRL methods, Figure 2 presents a conceptual diagram summarizing the categorization of DRL techniques. By structuring the discussion around these categories, we aim to establish a cohesive narrative that logically connects various methods and their applications.

4. Value-Based Methods

In the domain of mobile robot path planning, recent advances in deep reinforcement learning (Deep RL) have introduced both value-based and policy-based methods to solve complex navigation tasks. These methods leverage deep neural networks to approximate functions like the action–value function $Q^{\pi }(s,a)$ or directly parameterize policies $\pi (a|s;\theta )$. This section reviews key value-based methods, such as Deep Q-Networks (DQN), Double DQN (DDQN), and Dueling DQN (D3QN), as well as policy-based approaches, outlining their mathematical foundations and contributions to mobile robot navigation.

Value-based methods in deep RL focus on estimating the action–value function $Q^{\pi }(s,a)$, which represents the expected cumulative reward of taking action a in state s and following the policy $\pi$ thereafter. These methods solve the Bellman equation:

$$Q^{\pi }(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim \mathcal{P}}\left[\underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{\pi }(s^{\prime },a^{\prime })\right],$$

(28)

by using neural networks as function approximators. A well-known value-based deep RL method is the Deep Q-Networks (DQN) [8] which approximates $Q^{\pi }(s,a)$ using a neural network parameterized by $\varphi$, denoted as $Q(s,a;\varphi )$. The network is trained by minimizing the mean squared error between the current Q-value and the target value:

$$L(\varphi )={\mathbb{E}}_{(s,a,r,s^{\prime })}\left[{\left(Q(s,a;\varphi )-y\right)}^2\right],$$

(29)

where the target value y is as follows:

$$y=r(s,a)+\gamma \underset{a^{\prime }\in \mathcal{A}}{max}Q(s^{\prime },a^{\prime };{\varphi }^{-}).$$

(30)

Here, ${\varphi }^{-}$ denotes the parameters of a separate target network, which is updated periodically for stability. The step-by-step pseudocode for DQN is presented in Algorithm 1.

Algorithm 1: Deep Q-Network (DQN) Algorithm

While DQN is effective, it suffers from overestimation bias in Q-value updates. Double DQN (DDQN) [23] addresses this by decoupling the action selection and evaluation processes. The target value is modified as follows:

$$y=r(s,a)+\gamma Q(s^{\prime },arg\underset{a^{\prime }\in \mathcal{A}}{max}Q(s^{\prime },a^{\prime };\varphi );{\varphi }^{-}).$$

(31)

This correction reduces overestimation and improves learning stability. An alternative is the Dueling DQN (D3QN) method [24] which introduces a novel architecture that separates the representation of state–value and advantage functions:

$$Q(s,a;\varphi ,\xi )=V(s;\varphi )+A(s,a;\xi ),$$

(32)

where $V(s;\varphi )$ is the state–value function and $A(s,a;\xi )$ is the advantage function. This architecture helps the network learn which states are valuable independent of the specific actions taken, improving performance in complex environments. The pseudocode shown in Algorithm 2 presents how the D3QN method works in a step-by-step fashion.

Algorithm 2: Dueling Deep Q-Network (D3QN) Algorithm

5. Policy-Based Methods

Value-based methods, such as DQN, DDQN, and D3QN, are effective for discrete action spaces, offering stability and scalability in robotic navigation tasks with well-defined action sets. However, they struggle with continuous control due to the need for discretization. Policy-based methods, on the other hand, are designed for continuous action spaces, making them ideal for controlling mobile robots with differential drive or other complex dynamics.

Policy-based methods directly optimize the policy $\pi (a|s;\theta )$ by maximizing the expected cumulative reward:

$$J({\pi }_{\theta })={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right].$$

(37)

The policy parameters $\theta$ are updated using the gradient of the objective, known as the policy gradient:

$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }log{\pi }_{\theta }(a|s)Q^{\pi }(s,a)\right].$$

(38)

An example of a purely policy-based method is the REINFORCE algorithm [25], which uses Monte Carlo estimates of the return $G_t$ to compute the policy gradient. These methods do not utilize a value function to evaluate actions or guide policy updates, relying solely on sampled returns.

6. Actor–Critic Methods

Actor–critic methods combine a policy-based “actor” with a value-based “critic”. The actor represents the policy $\pi (a|s;\theta )$, while the critic estimates a value function, such as the state–value function $V^{\pi }(s)$ or the action–value function $Q^{\pi }(s,a)$. The critic helps reduce the variance in the policy gradient by replacing the raw return $G_t$ with an advantage estimate:

$$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s).$$

(39)

The policy gradient in actor–critic methods is typically expressed as follows:

$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }log{\pi }_{\theta }(a|s)A^{\pi }(s,a)\right].$$

(40)

The three main categories of actor–critic methods have been applied in mobile robot path planning applications: (1) Deep deterministic policy gradient (DDPG) method and its variants, (2) Asynchronous advantage actor critic (A3C) method and its variants, (3) Proximal policy optimization (PPO) method and its variants, and (4) Soft actor–critic (SAC) method and its variants. Each of these categories are presented in separate sections as follows.

6.1. Deep Deterministic Policy Gradient

The DDPG method is an actor–critic reinforcement learning algorithm designed for continuous action spaces. It extends the Deterministic Policy Gradient (DPG) method [26] by incorporating deep neural networks to parameterize both the policy and value functions. DDPG has been widely used in robotics, including mobile robot path planning, where the control commands often lie in a continuous domain, such as linear and angular velocities. It uses an actor–critic framework where

• The actor $\pi (s;{\theta }^{\pi })$ is a deterministic policy, parameterized by ${\theta }^{\pi }$, that maps states $s\in \mathcal{S}$ to continuous actions $a\in \mathcal{A}$.
• The critic $Q(s,a;{\theta }^Q)$ is a neural network, parameterized by ${\theta }^Q$, that estimates the action–value function:

  $$Q(s,a;{\theta }^Q)\approx Q^{\pi }(s,a)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)|s_0=s,a_0=a\right].$$

  (41)

The policy is updated by maximizing the estimated Q-value:

$${\nabla }_{{\theta }^{\pi }}J(\pi )={\mathbb{E}}_{s\sim \mathcal{D}}\left[{\nabla }_{a}Q(s,a;{\theta }^Q){|}_{a=\pi (s;{\theta }^{\pi })}{\nabla }_{{\theta }^{\pi }}\pi (s;{\theta }^{\pi })\right],$$

(42)

where $\mathcal{D}$ is a replay buffer used to store past transitions $(s,a,r,s^{\prime })$ for sampling during training. The critic is updated by minimizing the mean squared error of the Bellman residual:

$$L({\theta }^Q)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\left(Q(s,a;{\theta }^Q)-y\right)}^2\right],$$

(43)

where the target value y is computed using a target actor ${\pi }^{\prime }(s;{\theta }^{{\pi }^{\prime }})$ and a target critic $Q^{\prime }(s,a;{\theta }^{Q^{\prime }})$:

$$y=r(s,a)+\gamma Q^{\prime }(s^{\prime },{\pi }^{\prime }(s^{\prime };{\theta }^{{\pi }^{\prime }});{\theta }^{Q^{\prime }}).$$

(44)

DDPG uses target networks to stabilize training. The target actor ${\pi }^{\prime }(s;{\theta }^{{\pi }^{\prime }})$ and target critic $Q^{\prime }(s,a;{\theta }^{Q^{\prime }})$ are updated slowly using a soft update mechanism:

$${\theta }^{{\pi }^{\prime }}\leftarrow \tau {\theta }^{\pi }+(1-\tau ){\theta }^{{\pi }^{\prime }},$$

(45)

$${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }},$$

(46)

where $\tau \in [0,1)$ controls the update rate.

To encourage exploration in continuous action spaces, DDPG adds noise to the deterministic policy during training. A common choice is Ornstein–Uhlenbeck (OU) noise, which models temporally correlated random processes:

$$a_t=\pi (s_t;{\theta }^{\pi })+{\mathcal{N}}_t,{\mathcal{N}}_t\sim \mathrm{OU}(\mu ,\sigma ).$$

(47)

Algorithm 3 shows the step-by-step pseudocode for the DDPG algorithm.

In mobile robot path planning, DDPG has been applied to learn navigation strategies in continuous control domains. The actor network generates control commands, such as linear velocity v and angular velocity $\omega$, while the critic evaluates the navigation performance. The state s may include the following:

• The robot’s position $(x,y)$ and orientation $\theta$.
• Sensor inputs, such as LIDAR or depth images, providing information about obstacles.
• The relative position of the goal $(x_g,y_g)$.

The reward function is typically similar to (21), and designed to guide the robot toward the goal while avoiding collisions.

6.1.1. Assisted DDPG (AsDDPG)

In the work by Xie et al. [27], Assisted DDPG (AsDDPG) extends the standard DDPG algorithm by incorporating an external controller to accelerate training and stabilize learning. The external controller acts as a supplementary policy during the early stages of training and is progressively replaced by the learned policy as training converges. The integration of the external controller introduces an adaptive switching mechanism, enabling the algorithm to dynamically choose between the controller and the actor network.

Algorithm 3: Deep Deterministic Policy Gradient (DDPG) Algorithm

The core innovation in AsDDPG is the inclusion of a Critic-DQN network that evaluates both the learned policy and the external controller. At each time step t, the Critic-DQN decides which policy to use based on the Q-value estimates:

$${\sigma }_t=arg\underset{\sigma \in \{\mathrm{policy},\mathrm{controller}\}}{max}Q(x_t,\sigma ;{\theta }^Q),$$

(54)

where ${\sigma }_t$ is the switching action and $Q(x_t,\sigma ;{\theta }^Q)$ represents the Q-value of the state–action pair under the selected policy.

The critic network in AsDDPG consists of two branches: the Critic branch and the DQN branch. The Critic branch evaluates the Q-value of the actor’s actions and computes the policy gradient, while the DQN branch estimates the Q-values of both the external controller and the actor network. The advantage function $A(x,a)$ is computed to improve stability:

$$A(x,a)=Q(x,a;{\theta }^Q)-V(x;{\theta }^V),$$

(55)

where $V(x;{\theta }^V)$ is the state value function learned using a dueling architecture.

The loss function for the Critic-DQN is defined as follows:

$$\begin{array}{c}\hfill L({\theta }^Q,{\theta }^A)={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[{\left(y_i^A-Q(x_i,a_i;{\theta }^A)\right)}^2+{\left(y_i^Q-Q(x_i,{\sigma }_i;{\theta }^Q)\right)}^2\right],\end{array}$$

(56)

where the TD targets $y_i^A$ and $y_i^Q$ are computed as follows:

$$\begin{array}{ccc}\hfill y_i^A& =& r_i+\gamma Q^{\prime }(x_{i+1},{\pi }^{\prime }(x_{i+1};{\theta }^{{\pi }^{\prime }});{\theta }^{A^{\prime }}),\hfill \end{array}$$

(57)

$$\begin{array}{ccc}\hfill y_i^Q& =& r_i+\gamma \underset{{\sigma }^{\prime }}{max}{Q}^{\prime }(x_{i+1},{\sigma }^{\prime };{\theta }^{Q^{\prime }}).\hfill \end{array}$$

(58)

Here, $\gamma$ is the discount factor, $r_i$ is the reward, and $Q^{\prime }$ denotes the target network.

The actor network is updated using the policy gradient derived from the Critic branch:

$$\begin{array}{ccc}\hfill {\nabla }_{{\theta }^{\pi }}J({\theta }^{\pi })& =& {\mathbb{E}}_{x_t\sim \mathcal{D}}\left[{\nabla }_{a}Q(x_t,a;{\theta }^Q){|}_{a=\pi (x_t;{\theta }^{\pi })}{\nabla }_{{\theta }^{\pi }}\pi (x_t;{\theta }^{\pi })\right].\hfill \end{array}$$

(59)

The primary advantage of AsDDPG lies in its ability to leverage the external controller to guide exploration during the early stages of training, reducing the reliance on random exploration. As training progresses, the actor network learns to outperform the external controller, and the reliance on external guidance diminishes–see Figure 3. This adaptive approach ensures faster convergence and more robust learning, particularly in environments with sparse rewards or complex dynamics.

6.1.2. Dynamic Path Planning Using an Improved DDPG Algorithm

The study by Li et al. [28] builds on the DDPG framework by incorporating several enhancements to address its limitations, such as low success rate and slow convergence in dynamic environments. These enhancements include replacing the optimizer, adding prioritized experience replay, introducing transfer learning, and using a curiosity module.

To improve convergence speed and accuracy, the RAdam optimizer is integrated into the DDPG algorithm. The RAdam update rule is given by the following:

$${\theta }_{t+1}={\theta }_t-{\alpha }_t{r}_t{\displaystyle \frac{{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}}},$$

(60)

where

$${\widehat{v}}_t={\displaystyle \frac{v_t}{1-{\beta }_2^t}},v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2,$$

(61)

$${\widehat{m}}_t={\displaystyle \frac{m_t}{1-{\beta }_1^t}},m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,$$

(62)

$$r_t=\sqrt{{\displaystyle \frac{{\rho }_t-4}{{\rho }_{\infty }-4}}·{\displaystyle \frac{{\rho }_t-2}{{\rho }_{\infty }-2}}},$$

(63)

and ${\rho }_t$ and ${\rho }_{\infty }$ are parameters of the RAdam optimization process. This optimizer stabilizes training and enhances convergence.

Prioritized experience replay is employed to prioritize sampling experiences with higher temporal-difference (TD) error. The TD error is defined as follows:

$${\delta }_t=r_{t+1}+\gamma \underset{a}{max}Q(s_{t+1},a;\theta )-Q(s_t,a_t;\theta ),$$

(64)

where Q is the action–value function, $s_t$ is the current state, $a_t$ is the current action, and $\gamma$ is the discount factor. This prioritization accelerates learning by focusing on significant experiences.

To improve exploration in complex environments, a curiosity module is introduced. This module provides intrinsic rewards by minimizing the prediction error of the forward dynamics model:

$$\widehat{\varphi }(s_{t+1})=f(\varphi (s_t),a_t;{\theta }_F),$$

(65)

where $\varphi (s_t)$ is the encoded feature vector of the current state, $a_t$ is the action, and ${\theta }_F$ represents the model parameters. The intrinsic reward is calculated as follows:

$$r_t^i={\displaystyle \frac{\eta }{2}}{\parallel \widehat{\varphi }(s_{t+1})-\varphi (s_{t+1})\parallel }^2,$$

(66)

where $\eta >0$ is a scaling factor.

The total reward combines intrinsic and extrinsic rewards:

$$r_t=r_t^i+r_t^e,$$

(67)

where $r_t^e$ is the external reward from the environment.

Transfer learning is applied to initialize the policy and value networks with pre-trained weights, reducing the required training time. The combined effect of these enhancements significantly improves the convergence speed and success rate of the DDPG algorithm in dynamic environments.

6.1.3. Virtual-to-Real Deep Reinforcement Learning for Mapless Navigation

Tai et al. [29] build on the DDPG framework by introducing asynchronous updates and leveraging low-dimensional sparse range findings for efficient mapless navigation of nonholonomic mobile robots.

To improve sampling efficiency, the sample collection process is separated into an independent thread, creating an asynchronous version of DDPG (ADDPG). This asynchronous mechanism ensures that the agent can collect more samples per iteration compared to traditional DDPG, enhancing convergence. The effectiveness of ADDPG is illustrated by the faster increase in the Q-value during training:

$$Q(s_t,a_t;\theta )=r_t+\gamma \underset{a}{max}Q(s_{t+1},a;\theta ),$$

(68)

where $r_t$ is the immediate reward, $s_t$ is the state, $a_t$ is the action, and $\gamma$ is the discount factor.

The input to the ADDPG model consists of the following:

$$x_t=[\mathrm{laser}\mathrm{range}\mathrm{findings},\mathrm{target}\mathrm{position},\mathrm{previous}\mathrm{velocity}],$$

(69)

where the laser range findings are abstracted into a 10-dimensional sparse representation. These inputs are normalized to the range $(0,1)$.

The output of the actor network consists of continuous control commands, including linear velocity $v_t$ and angular velocity ${\omega }_t$. The actor network updates are computed as follows:

$${\nabla }_{{\theta }^{\mu }}J\approx \mathbb{E}\left[{\nabla }_{a}Q(s,a;{\theta }^Q){|}_{a=\mu (s;{\theta }^{\mu })}{\nabla }_{{\theta }^{\mu }}\mu (s;{\theta }^{\mu })\right].$$

(70)

The critic network uses a three-layer fully connected structure to predict the Q-value. The Q-value is computed using a linear activation function at the output layer:

$$Q(s,a;{\theta }^Q)=w^{T}x+b,$$

(71)

where w and b are the weights and bias, respectively.

The reward function is defined as follows:

$$r(s_t,a_t)=\left\{\begin{array}{cc}{r}_{\mathrm{arrive}},\hfill & \mathrm{if}{d}_t<d_{\mathrm{threshold}},\hfill \\ r_{\mathrm{collision}},\hfill & \mathrm{if}min(x_t)<x_{\mathrm{collision}},\hfill \\ c_r(d_{t-1}-d_t),\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(72)

where $d_t$ is the distance to the target at time t, and $r_{\mathrm{arrive}}$, $r_{\mathrm{collision}}$, and $c_r$ are hyperparameters.

The overall method, visualized in Figure 4 enables end-to-end training of a mapless motion planner capable of transferring knowledge from virtual to real environments without additional fine-tuning, ensuring robust navigation in unseen scenarios.

6.1.4. Goal-Oriented Obstacle Avoidance in Continuous Action Space

The proposed method in [30] extends the DDPG framework by integrating depth-wise separable convolutional layers to handle depth image inputs effectively. This enhancement allows for efficient processing of large-scale sequential depth images, combined with goal-oriented navigation capabilities.

The actor network takes a stack of sequential depth images along with positional information in polar coordinates. The stack of depth images from $t-n$ to t provides a short-term memory, enabling the robot to detect peripheral obstacles that may not be visible in the current input. The depth-wise separable convolutional operation is defined as follows:

$${\mathbf{z}}_i^{\mathrm{depth}}=\sigma \left(\sum _{j=1}^n{k}_{i,j}\ast x_j\right),$$

(73)

where $k_{i,j}$ represents the filter applied to channel j, $x_j$ is the input feature map, and $\sigma$ is the activation function.

The polar coordinates of the robot’s position relative to the goal are concatenated with the processed depth image features. The output of the actor network consists of continuous control commands:

$$[\omega ,v]=\pi (s;{\theta }^{\pi }),$$

(74)

where $\omega$ is the angular velocity, v is the linear velocity, and $\pi (s;{\theta }^{\pi })$ represents the actor policy parameterized by ${\theta }^{\pi }$.

The critic network evaluates the Q-value using the current state and action:

$$Q(s,a;{\theta }^Q)=\mathbb{E}\left[r_t+\gamma \underset{a}{max}Q(s_{t+1},a;{\theta }^Q)|s,a\right],$$

(75)

where $r_t$ is the immediate reward and $\gamma$ is the discount factor.

The reward function is designed to encourage linear motion and penalize unnecessary angular adjustments:

$$r(s_t,a_t)=\left\{\begin{array}{cc}{r}_{\mathrm{goal}},\hfill & \mathrm{if}{d}_t<d_{\mathrm{threshold}},\hfill \\ r_{\mathrm{collision}},\hfill & \mathrm{if}\mathrm{a}\mathrm{collision}\mathrm{occurs},\hfill \\ c_v·v-c_{\omega }·|\omega |,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(76)

where $d_t$ is the distance to the goal, and $c_v$ and $c_{\omega }$ are coefficients for linear and angular velocities.

The target networks for the actor and critic are updated using a soft update mechanism:

$${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime },$$

(77)

where $\tau$ is a small constant controlling the update rate.

This method enables end-to-end learning of a goal-oriented obstacle avoidance policy, trained entirely in simulation and transferable to real-world scenarios without additional fine-tuning. Algorithm 4 shows a step-by-step pseudocode of the method.

Algorithm 4: Pseudocode of the Goal-oriented obstacle avoidance algorithm in continuous action space [30].

6.1.5. DDPG for Path Planning in Cluttered Indoor Spaces

In the study by Gao et al. [31], incremental training, parameter transfer, and a hybrid PRM+TD3 planner are introduced to improve development efficiency, convergence, and generalization in indoor mobile robot path planning. The training process employs an incremental strategy starting in a lightweight 2D simulation environment and gradually transitioning to a complex 3D environment. The training in the 2D environment uses simpler simulations to debug parameters and optimize the reward function before transferring the network to the 3D environment. The transfer of parameters follows:

$${\theta }_{\mathrm{initial}}\leftarrow {\theta }_{2D},$$

(78)

where ${\theta }_{\mathrm{initial}}$ denotes the initial weights transferred from the 2D environment.

The hybrid planner PRM+TD3 combines the Probabilistic Roadmap (PRM) algorithm for global path planning and TD3 for local control. PRM generates intermediate waypoints:

$$W=\{w_1,w_2,\cdots ,w_n\},$$

(79)

which decompose the global path into local sub-paths for TD3 to navigate.

TD3 enhances DDPG by addressing overestimation bias using two Q-networks and delayed updates. The Q-value update is as follows:

$$y(r,s^{\prime },d)=r+\gamma (1-d)\underset{i=1,2}{min}{Q}_{{\varphi }_{\mathrm{target},i}}(s^{\prime },{\pi }^{\prime }(s^{\prime };{\theta }^{\prime })),$$

(80)

where d is the termination flag, r is the reward, and ${\pi }^{\prime }(s^{\prime };{\theta }^{\prime })$ is the target policy.

The critic networks are updated by minimizing the loss:

$$L({\varphi }_i)={\displaystyle \frac{1}{|B|}}\sum _{(s,a,r,s^{\prime },d)\in B}{\left(Q_{{\varphi }_i}(s,a)-y(r,s^{\prime },d)\right)}^2,i=1,2.$$

(81)

The actor network is updated using the policy gradient:

$${\nabla }_{{\theta }^{\pi }}J={\mathbb{E}}_{s\sim B}\left[{\nabla }_a{Q}_{{\varphi }_1}(s,a){|}_{a=\pi (s;{\theta }^{\pi })}{\nabla }_{{\theta }^{\pi }}\pi (s;{\theta }^{\pi })\right].$$

(82)

Target networks for both actor and critic are updated softly:

$${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime },{\varphi }^{\prime }\leftarrow \tau \varphi +(1-\tau ){\varphi }^{\prime },$$

(83)

where $\tau$ controls the rate of update.

The reward function is designed to incentivize collision-free navigation and goal-reaching:

$$r(s_t,a_t)=\left\{\begin{array}{cc}{r}_{\mathrm{goal}},\hfill & \mathrm{if}{d}_t<d_{\mathrm{threshold}},\hfill \\ r_{\mathrm{collision}},\hfill & \mathrm{if}{d}_{\min }<d_t,\hfill \\ \alpha (d_{\mathrm{goal}}-d_t),\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(84)

where $d_t$ is the distance to the target, $d_{\mathrm{threshold}}$ is the goal proximity threshold, and $\alpha$ is a scaling factor.

This approach enhances convergence, reduces training time, and improves generalization to complex environments. See Algorithm 5 for a step-by-step pseudocode of the proposed method.

Algorithm 5: PRM+TD3 with Incremental Training [31].

6.1.6. Learning World Transition Model for Socially Aware Robot Navigation

Cui et al. [32] present an extension of the DDPG framework by integrating a model-based reinforcement learning approach with a deep world transition model, which predicts future states and rewards. This improves sample efficiency and allows socially compliant robot navigation.

The world transition model predicts the evolution of the robot’s environment, including future obstacle maps and rewards. Let the current state $s_t$ be represented as a sequence of obstacle maps, relative goal position, and current velocity. The predicted next state $s_{t+1}$ is given by the following:

$$s_{t+1}=f(s_t,a_t;{\theta }_w),$$

(85)

where f is the world transition model parameterized by ${\theta }_w$.

The obstacle map representation disentangles static and dynamic obstacles using ego-motion transformation. Dynamic features are computed by the following:

$$m_t=o_t-o_{t-1},$$

(86)

where $o_t$ is the obstacle map at time t. Static features are extracted by encoding the last obstacle map.

The reward function incorporates goal-reaching, collision avoidance, and social compliance:

$$R(s_t)=R_g(s_t)+R_c(s_t)+R_s(s_t),$$

(87)

where $R_g(s_t)$ encourages goal-reaching:

$$R_g(s_t)=\left\{\begin{array}{cc}{r}_{\mathrm{arrival}},\hfill & \mathrm{if}\mathrm{goal}\mathrm{reached},\hfill \\ w_1(\parallel p_t-p^{*}\parallel -\parallel p_{t-1}-p^{*}\parallel ),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(88)

Collision avoidance is penalized by $R_c(s_t)$:

$$R_c(s_t)=\left\{\begin{array}{cc}{r}_{\mathrm{collision}},\hfill & \mathrm{if}\mathrm{collision},\hfill \\ w_2\left(1-{\displaystyle \frac{d}{r+1.0}}\right),\hfill & \mathrm{if}d\le r+1.0,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(89)

Social compliance is achieved by penalizing proximity to dynamic obstacles:

$$R_s(s_t)=\left\{\begin{array}{cc}{w}_3\left(1-{\displaystyle \frac{d_{\mathrm{ped}}}{r+1.25}}\right),\hfill & \mathrm{if}{d}_{\mathrm{ped}}\le r+1.25,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(90)

The policy network uses stacked obstacle maps processed with a 3D-CNN to output control commands:

$$a_t=\pi (s_t;{\theta }^{\pi }),$$

(91)

where $\pi$ is the actor policy parameterized by ${\theta }^{\pi }$.

The critic networks estimate the Q-values using the TD3 framework:

$$Q(s,a;{\theta }^Q)=\mathbb{E}\left[r+\gamma \underset{i=1,2}{min}{Q}_{{\theta }^{Q^{\prime }}}(s^{\prime },a^{\prime }){|}_{a^{\prime }={\pi }^{\prime }(s^{\prime };{\theta }^{{\pi }^{\prime }})}\right].$$

(92)

The critic loss is minimized by the following:

$$L({\varphi }_i)={\displaystyle \frac{1}{|B|}}\sum _{(s,a,r,s^{\prime },d)\in B}{\left(Q_{{\varphi }_i}(s,a)-y\right)}^2,i=1,2,$$

(93)

where $y=r+\gamma (1-d){min}_{i=1,2}{Q}_{{\varphi }_i^{\prime }}(s^{\prime },a^{\prime })$.

The actor network is updated using the deterministic policy gradient:

$${\nabla }_{{\theta }^{\pi }}J=\mathbb{E}\left[{\nabla }_a{Q}_{{\varphi }_1}(s,a){|}_{a=\pi (s;{\theta }^{\pi })}{\nabla }_{{\theta }^{\pi }}\pi (s;{\theta }^{\pi })\right].$$

(94)

The target networks are updated softly:

$${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime }.$$

(95)

This method enables efficient policy training using a combination of real and virtual data, leading to socially compliant navigation in dynamic environments. The block diagram shown in Figure 5 shows the process flow of the method.

6.2. Asynchronous Advantage Actor–Critic (A3C) Method

The Asynchronous Advantage Actor–Critic (A3C) method, introduced by Mnih et al. [33], maintains a policy $\pi (a_t|s_t;\theta )$ and an estimate of the value function $V(s_t;{\theta }_v)$. The algorithm operates in the forward view and uses n-step returns to update both the policy and the value function.

The advantage function $A(s_t,a_t;\theta ,{\theta }_v)$ is computed as follows:

$$A(s_t,a_t;\theta ,{\theta }_v)=\sum _{i=0}^{k-1}{\gamma }^i{r}_{t+i}+{\gamma }^{k}V(s_{t+k};{\theta }_v)-V(s_t;{\theta }_v),$$

(96)

where k is bounded by $t_{\max }$, the maximum number of steps. The policy is updated using the gradient:

$${\nabla }_{\theta }J=\sum _{t=t_{\mathrm{start}}}^{t_{\mathrm{end}}}{\nabla }_{\theta }log\pi (a_t|s_t;\theta )A(s_t,a_t;\theta ,{\theta }_v),$$

(97)

where $t_{\mathrm{start}}$ and $t_{\mathrm{end}}$ define the range of steps for the update. The value function is updated by minimizing the loss:

$$L_v({\theta }_v)=\sum _{t=t_{\mathrm{start}}}^{t_{\mathrm{end}}}{\left(R_t-V(s_t;{\theta }_v)\right)}^2,$$

(98)

where $R_t$ is the n-step return:

$$R_t=\sum _{i=0}^{k-1}{\gamma }^i{r}_{t+i}+{\gamma }^{k}V(s_{t+k};{\theta }_v).$$

(99)

Entropy regularization is added to the policy loss to encourage exploration. The entropy of the policy $\pi$ is given by the following:

$$H(\pi (s_t;\theta ))=-\sum _a\pi (a|s_t;\theta )log\pi (a|s_t;\theta ),$$

(100)

and the entropy term is added to the policy gradient with a weight $\beta$:

$${\nabla }_{\theta }{J}_{\mathrm{total}}={\nabla }_{\theta }J+\beta {\nabla }_{\theta }H(\pi (s_t;\theta )).$$

(101)

The updates for the policy and value networks are performed asynchronously across multiple threads. Each thread maintains a local copy of the parameters $\theta$ and ${\theta }_v$, which are periodically synchronized with the global parameters. The local parameters are updated using the following:

$$\theta \leftarrow \theta +\alpha {\nabla }_{\theta }{J}_{\mathrm{total}},{\theta }_v\leftarrow {\theta }_v-\alpha {\nabla }_{{\theta }_v}{L}_v({\theta }_v),$$

(102)

where $\alpha$ is the learning rate.

This asynchronous framework stabilizes training by allowing parallel actor-learners to explore different parts of the state space, reducing the correlation between updates and improving training efficiency. Algorithm 6 presents the step-by-step pseudocode of the A3C method.

Algorithm 6: Asynchronous Advantage Actor–Critic (A3C) Algorithm

6.2.1. A3C Algorithm for Learning to Navigate in Complex Environments

Mirowski et al. [34] have extended A3C by incorporating auxiliary tasks to improve navigation in complex environments. The architecture augments the standard A3C framework with two auxiliary outputs: depth prediction and loop closure detection. These tasks enhance data efficiency and representation learning by providing additional gradient updates during training.

The first auxiliary task, depth prediction, estimates a depth map from RGB inputs. The loss function for depth prediction is implemented as a classification task, dividing depth values into discrete bins:

$$\begin{array}{ccc}\hfill L_{\mathrm{depth}}& =& {\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{j=1}^C{y}_{ij}log({\widehat{y}}_{ij}),\hfill \end{array}$$

(103)

where $y_{ij}$ represents the true probability of the j-th depth bin for the i-th pixel, ${\widehat{y}}_{ij}$ is the predicted probability, N is the number of pixels, and C is the number of depth bins.

The second auxiliary task, loop closure detection, predicts whether the agent is revisiting a previously visited location. The training targets are generated based on position similarity between the current position and earlier positions in the trajectory. The binary classification loss is given by the following:

$$\begin{array}{ccc}\hfill L_{\mathrm{loop}}& =& -{\displaystyle \frac{1}{T}}\sum _{t=1}^T\left(l_{t}log({\widehat{l}}_t)+(1-l_t)log(1-{\widehat{l}}_t)\right),\hfill \end{array}$$

(104)

where $l_t$ is the ground truth loop closure label for time step t, ${\widehat{l}}_t$ is the predicted probability, and T is the total number of steps.

The overall loss function combines the A3C objective with these auxiliary losses. The total loss is as follows:

$$\begin{array}{ccc}\hfill L_{\mathrm{total}}& =& L_{\mathrm{A}3\mathrm{C}}+{\beta }_{\mathrm{depth}}{L}_{\mathrm{depth}}+{\beta }_{\mathrm{loop}}{L}_{\mathrm{loop}},\hfill \end{array}$$

(105)

where ${\beta }_{\mathrm{depth}}$ and ${\beta }_{\mathrm{loop}}$ are coefficients balancing the contributions of the auxiliary tasks.

To integrate auxiliary tasks with the A3C framework, the architecture uses a shared convolutional encoder to process RGB inputs. The shared features are passed to both the policy/value networks and the auxiliary task heads. Specifically, the depth prediction head predicts depth values from the convolutional features, while the loop closure head predicts loop closure probabilities using the hidden states of the LSTM layer. The policy $\pi (a_t|s_t;\theta )$ and value $V(s_t;{\theta }_V)$ networks follow the A3C architecture and are updated with standard A3C gradients.

The depth prediction output is computed from the convolutional features $f_t$ as follows:

$$\begin{array}{ccc}\hfill {\widehat{y}}_{ij}& =& \mathrm{softmax}(W_{\mathrm{depth}}{f}_t+b_{\mathrm{depth}}),\hfill \end{array}$$

(106)

where $W_{\mathrm{depth}}$ and $b_{\mathrm{depth}}$ are the weights and biases of the depth prediction layer.

The loop closure output is computed from the LSTM hidden states $h_t$ as follows:

$$\begin{array}{ccc}\hfill {\widehat{l}}_t& =& \mathrm{sigmoid}(W_{\mathrm{loop}}{h}_t+b_{\mathrm{loop}}),\hfill \end{array}$$

(107)

where $W_{\mathrm{loop}}$ and $b_{\mathrm{loop}}$ are the weights and biases of the loop closure layer.

These auxiliary tasks provide additional supervision signals, accelerating learning and improving the agent’s ability to represent spatial and geometric features critical for navigation. The block diagram shown in Figure 6 presents how the proposed architecture extends A3C by integrating auxiliary tasks for depth prediction and loop closure detection. The shared encoder processes RGB inputs to extract features $f_t$, which are passed to the policy and value networks as well as auxiliary task-specific heads. Depth prediction and loop closure detection contribute to the total loss $L_{\mathrm{total}}$, accelerating training and improving navigation performance.

6.2.2. A3C Algorithm for Robot Navigation in Unknown Rough Terrain

Zhang et al. [35] have extended A3C to address navigation in unknown rough terrain environments by incorporating an elevation map, depth images, and the robot’s 3D orientation as inputs into the policy and value networks. The architecture consists of three separate branches for processing these inputs, which are then merged to compute navigation actions.

The depth image branch processes downsampled depth images $I_d\in {\mathbb{R}}^{84\times 84}$ through a series of convolutional layers, reducing the high-dimensional sensory input into feature embeddings $f_d$. The elevation map branch processes a single-channel 84 × 84 grid $E\in {\mathbb{R}}^{84\times 84}$ that encodes terrain elevation, using convolutional layers to extract features $f_e$. The robot’s 3D orientation $(\alpha ,\beta ,\gamma )$ is passed through a fully connected layer, producing feature embeddings $f_o$ that match the spatial dimensions of $f_e$. These two branches are merged using element-wise addition:

$$\begin{array}{c}\hfill f_{\mathrm{merge}}=f_e+f_o.\end{array}$$

(108)

The outputs from the depth and merged branches are concatenated to form a unified feature representation:

$$\begin{array}{c}\hfill f_{\mathrm{total}}=\mathrm{concat}(f_d,f_{\mathrm{merge}}).\end{array}$$

(109)

This representation is passed through an LSTM layer to capture temporal dependencies in the robot’s navigation states. The output from the LSTM layer is used to compute the policy $\pi (a_t|s_t;\theta )$ and the value $V(s_t;{\theta }_v)$. The policy network computes the probability distribution over navigation actions $a_t$, including moving forward, backward, or turning left or right. The policy is updated using the A3C loss function:

$$\begin{array}{c}\hfill \Delta \theta ={\nabla }_{\theta }log\pi (a_t|s_t;\theta )A(s_t,a_t;{\theta }_v)+\zeta {\nabla }_{\theta }H(\pi (s_t;\theta )),\end{array}$$

(110)

where $A(s_t,a_t;{\theta }_v)$ is the advantage function, computed as follows:

$$\begin{array}{c}\hfill A(s_t,a_t;{\theta }_v)=\sum _{i=0}^{n-1}{\mathrm{\Gamma }}^i{r}_{t+i}+{\mathrm{\Gamma }}^{n}V(s_{t+n};{\theta }_v)-V(s_t;{\theta }_v),\end{array}$$

(111)

and $H(\pi (s_t;\theta ))$ is the entropy of the policy, encouraging exploration, with $\zeta$ as a scaling factor.

The value network minimizes the squared advantage:

$$\begin{array}{c}\hfill \Delta {\theta }_v=A(s_t,a_t;{\theta }_v){\nabla }_{{\theta }_v}V(s_t;{\theta }_v).\end{array}$$

(112)

The reward function is tailored to the rough terrain navigation task. It assigns positive rewards for reaching the goal within a certain tolerance and penalizes undesirable states such as collisions or exceeding the maximum steps per episode:

$$\begin{array}{c}\hfill r_t=\left\{\begin{array}{cc}\mu {\displaystyle \frac{d_{\min }-d_t}{\Delta t}},\hfill & \mathrm{if}{d}_t<d_{\min },\hfill \\ 0.5+r_v,\hfill & \mathrm{if}{d}_t\le d_g,\hfill \\ -0.5,\hfill & \mathrm{if}\mathrm{in}{s}_{\mathrm{tu}},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(113)

Here, $d_t$ is the distance to the goal at time t, $d_{\min }$ is the closest distance observed so far, $d_g$ is the goal tolerance, and $s_{\mathrm{tu}}$ represents undesirable terminal states like flipping or getting stuck.

The network is trained in a simulated environment with high-dimensional sensory inputs. Nine agents operate in parallel, each updating the global policy and value functions asynchronously to accelerate convergence. The block diagram shown in Figure 7 visualizes how architecture extends A3C for navigation in unknown rough terrain by incorporating multiple input branches for depth images, elevation maps, and 3D orientation. These inputs are processed through separate branches to extract features ($f_d$, $f_e$, $f_o$) which are merged and concatenated to form a unified representation ($f_{\mathrm{total}}$). Temporal dependencies are captured using an LSTM layer, and the outputs are used by the policy and value networks to compute navigation actions.

6.2.3. A3C Algorithm for Teaching a Machine to Read Maps

The study by Brunner et al. [36] based Dynamic Obstacle Avoidance andecture that decomposes the navigation task into intermediate subtasks, each handled by a specialized module. These modules are trained either independently or in conjunction with the main A3C policy. The subtasks include visible local map estimation, recurrent localization, map interpretation, and reactive action execution.

The visible local map network processes the 3D RGB visual input and a discretized 3-hot encoded orientation angle $\widehat{\alpha }$. The map excerpt is gated using the visible field estimation:

$$\begin{array}{ccc}\hfill \mathrm{Visible}\mathrm{Local}\mathrm{Map}& =& \mathrm{Map}\mathrm{Excerpt}·\mathrm{Visible}\mathrm{Field},\hfill \end{array}$$

(114)

where · denotes element-wise multiplication. The visible local map output is used to construct an egocentric local map.

The recurrent localization cell integrates the visible local map excerpts into an egocentric local map. It estimates the egomotion as follows:

$$\begin{array}{ccc}\hfill s_t& =& \mathrm{softmax}(f(s_{t-1},\widehat{\alpha },a_{t-1},r_{t-1})+{\mathrm{LM}}_{\mathrm{est},t-1}\ast \tilde{V}),\hfill \end{array}$$

(115)

where $f(·)$ is a two-layer feedforward network, ∗ represents 2D convolution, ${\mathrm{LM}}_{\mathrm{est},t-1}$ is the previous local map estimation, and $\tilde{V}$ is the current visible local map input. The estimated local map is updated as follows:

$$\begin{array}{ccc}\hfill {\mathrm{LM}}_{\mathrm{est},t}& =& {[{\mathrm{LM}}_{\mathrm{est},t-1}\ast s_t+\tilde{V}]}_{-0.5}^{+0.5},\hfill \end{array}$$

(116)

where ${[·]}_{-0.5}^{+0.5}$ denotes clipping to the range $[-0.5,+0.5]$. The updated map is combined with a feedback map to enhance localization:

$$\begin{array}{ccc}\hfill {\mathrm{LM}}_{\mathrm{est}+\mathrm{mfb},t}& =& {[{\mathrm{LM}}_{\mathrm{est},t}+\lambda ·{\mathrm{LM}}_{\mathrm{mfb},t-1}\ast s_t]}_{-0.5}^{+0.5}.\hfill \end{array}$$

(117)

The location probability distribution is calculated by correlating the estimated local map with the global map:

$$\begin{array}{ccc}\hfill {\left\{p_{\mathrm{loc},i}\right\}}_{i=1}^N& =& \mathrm{softmax}(M\ast {\mathrm{LM}}_{\mathrm{est}+\mathrm{mfb},t}),\hfill \end{array}$$

(118)

where M is the global map and N is the number of discrete location cells.

The map interpretation network processes the global map to identify rewarding locations and outputs a short-term target direction (STTD). The STTD at each location is derived using a shortest path algorithm, producing a direction distribution ${\left\{d_{\mathrm{sttd},i}\right\}}_{i=1}^N$.

The reactive agent combines the STTD distribution, the normalized entropy of the location probability $H_{\mathrm{loc}}$, and the estimated target distance to select actions. It maximizes the total reward, which combines extrinsic and intrinsic rewards:

$$\begin{array}{ccc}\hfill I_{\mathrm{explor},t}& =& H_{\mathrm{loc},t-1}-H_{\mathrm{loc},t},\hfill \end{array}$$

(119)

$$\begin{array}{ccc}\hfill I_{\mathrm{exploit},t}& =& {\mathbf{e}}_t^{\top }·{\mathbf{d}}_{\mathrm{sttd},t-1},\hfill \end{array}$$

(120)

where ${\mathbf{e}}_t$ is the egomotion vector and ${\mathbf{d}}_{\mathrm{sttd},t-1}$ is the STTD vector. The agent’s policy is updated using A3C’s policy gradient:

$$\begin{array}{ccc}\hfill \Delta \theta & =& {\nabla }_{\theta }log\pi (a_t|s_t;\theta )A(s_t,a_t;{\theta }_v)+\zeta {\nabla }_{\theta }H(\pi (s_t;\theta )),\hfill \end{array}$$

(121)

where $A(s_t,a_t;{\theta }_v)$ is the advantage function, and $H(\pi (s_t;\theta ))$ is the entropy of the policy.

The proposed architecture improves navigation by integrating modular components, leveraging A3C’s stability, and introducing additional supervision signals through intrinsic rewards and auxiliary tasks. Algorithm 7 presents the step-by-step pseudocode for implementing the proposed architecture.

Algorithm 7: Pseudocode for the navigation algorithm proposed in [36].

6.2.4. A3C Algorithm for Target-Driven Visual Navigation in Indoor Scenes

Zhu et al. [37] extended A3C by introducing a target-driven visual navigation framework based on a deep Siamese actor–critic network. The key innovation is the use of a dual input structure, where both the current state $s_t$ and the target representation g are processed to jointly estimate policy $\pi$ and value V. This allows generalization to unseen targets without re-training.

The network takes as input two images: $s_t$, representing the agent’s current observation, and g, representing the target. Both inputs are processed through Siamese layers that share weights, producing embeddings $\varphi (s_t)$ and $\varphi (g)$ in a shared latent space:

$$\begin{array}{c}\hfill \varphi (s_t)=f_{\mathrm{Siamese}}(s_t),\varphi (g)=f_{\mathrm{Siamese}}(g).\end{array}$$

(122)

The embeddings are concatenated to form a joint representation:

$$\begin{array}{c}\hfill {\varphi }_{\mathrm{joint}}=\mathrm{concat}(\varphi (s_t),\varphi (g)).\end{array}$$

(123)

This joint representation is passed through scene-specific layers to capture environment-specific features. The scene-specific representation is used to compute the policy $\pi (a_t|s_t,g;\theta )$ and the value $V(s_t,g;{\theta }_v)$. The policy network predicts the probability distribution over actions:

$$\begin{array}{c}\hfill \pi (a_t|s_t,g;\theta )=\mathrm{softmax}(W_{\pi }{\varphi }_{\mathrm{joint}}+b_{\pi }).\end{array}$$

(124)

The value network estimates the expected future reward:

$$\begin{array}{c}\hfill V(s_t,g;{\theta }_v)=W_V{\varphi }_{\mathrm{joint}}+b_V.\end{array}$$

(125)

The policy is updated using the A3C policy gradient:

$$\begin{array}{c}\hfill \Delta \theta ={\nabla }_{\theta }log\pi (a_t|s_t,g;\theta )A(s_t,g;{\theta }_v)+\zeta {\nabla }_{\theta }H(\pi (s_t,g;\theta )),\end{array}$$

(126)

where $A(s_t,g;{\theta }_v)$ is the advantage function:

$$\begin{array}{c}\hfill A(s_t,g;{\theta }_v)=r_t+\gamma V(s_{t+1},g;{\theta }_v)-V(s_t,g;{\theta }_v),\end{array}$$

(127)

and $H(\pi (s_t,g;\theta ))$ is the entropy of the policy to encourage exploration.

The value network is updated by minimizing the squared error:

$$\begin{array}{c}\hfill L_V={\displaystyle \frac{1}{2}}{\left(r_t+\gamma V(s_{t+1},g;{\theta }_v)-V(s_t,g;{\theta }_v)\right)}^2.\end{array}$$

(128)

The network leverages asynchronous training, where multiple threads independently interact with the environment. Each thread collects gradients and updates the shared model parameters. Generic Siamese layers are updated across all targets and scenes, while scene-specific layers are fine-tuned for each environment, enabling both inter-target and inter-scene generalization.

The reward function for navigation includes a goal-reaching reward $R_{\mathrm{goal}}$ and a time penalty $R_{\mathrm{time}}$:

$$\begin{array}{c}\hfill R_t=\left\{\begin{array}{cc}10.0,\hfill & \mathrm{if}\mathrm{goal}\mathrm{is}\mathrm{reached},\hfill \\ -0.01,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(129)

This framework improves upon A3C by enabling generalization to unseen targets through the Siamese structure and shared parameter updates, while maintaining data efficiency through asynchronous training. Figure 8 shows a block diagram of the proposed target-driven navigation method based on A3C. The Siamese network processes the current state $s_t$ and target g to generate embeddings $\varphi (s_t)$ and $\varphi (g)$, which are concatenated to form a joint representation ${\varphi }_{\mathrm{joint}}$. Scene-specific layers extract environment-specific features used by the policy and value networks. The policy network outputs navigation actions $a_t$, while the value network evaluates future rewards. Both networks are updated asynchronously using the A3C framework.

Figure 8. Block diagram of the deep reinforcement learning framework proposed in [37].

Figure 8. Block diagram of the deep reinforcement learning framework proposed in [37].

6.2.5. A3C Algorithm for Autonomous Navigation in Indoor Environments

The study by Surmann et al. [38] builds on A3C by using the GA3C framework, which improves efficiency and scalability in parallel training. Instead of maintaining separate network copies for each agent, GA3C centralizes predictions and updates using a global deep neural network (DNN) model. This eliminates the need for synchronization and allows for efficient GPU utilization. The algorithm employs two global queues: a prediction queue for obtaining observations and a learning queue for training updates.

The inputs to the network are the last four laser scans $\{L_{t-3},L_{t-2},L_{t-1},L_t\}$, each with 1081 values, and a one-hot encoded orientation vector ${\mathbf{o}}_t$ representing the direction of the goal. The laser scan values are normalized to the range $[0,1]$, and the orientation is encoded as follows:

$$\begin{array}{c}\hfill {\mathbf{o}}_t=\mathrm{one}\text{-}\mathrm{hot}(g_{\mathrm{dir}})\mathrm{where}{g}_{\mathrm{dir}}\in [0,360].\end{array}$$

(130)

The network begins with two 1D convolutional layers:

$$\begin{array}{c}\hfill \mathrm{Conv}1:1\times 9\times 16,\mathrm{stride}:4,\end{array}$$

(131)

$$\begin{array}{c}\hfill \mathrm{Conv}2:1\times 5\times 32,\mathrm{stride}:2.\end{array}$$

(132)

The outputs are flattened and passed to a dense layer with 256 neurons to generate the shared latent representation $\varphi (s_t)$. The policy network computes the probability distribution over actions $\pi (a_t|s_t;\theta )$:

$$\begin{array}{c}\hfill \pi (a_t|s_t;\theta )=\mathrm{softmax}(W_{\pi }\varphi (s_t)+b_{\pi }),\end{array}$$

(133)

where $a_t$ represents discrete actions corresponding to linear and angular velocities, such as forward motion and turning. The value network estimates the state value $V(s_t;{\theta }_v)$:

$$\begin{array}{c}\hfill V(s_t;{\theta }_v)=W_V\varphi (s_t)+b_V.\end{array}$$

(134)

The policy and value networks are updated using the A3C loss. The policy gradient is computed as follows:

$$\begin{array}{c}\hfill \Delta \theta ={\nabla }_{\theta }log\pi (a_t|s_t;\theta )A(s_t,a_t;{\theta }_v)+\zeta {\nabla }_{\theta }H(\pi (s_t;\theta )),\end{array}$$

(135)

where $A(s_t,a_t;{\theta }_v)$ is the advantage function:

$$\begin{array}{c}\hfill A(s_t,a_t;{\theta }_v)=r_t+\gamma V(s_{t+1};{\theta }_v)-V(s_t;{\theta }_v).\end{array}$$

(136)

The value loss is minimized to improve the value network:

$$\begin{array}{c}\hfill L_V={\displaystyle \frac{1}{2}}{\left(r_t+\gamma V(s_{t+1};{\theta }_v)-V(s_t;{\theta }_v)\right)}^2.\end{array}$$

(137)

The reward function integrates goal-reaching rewards, collision penalties, and intermediate rewards. For goal-reaching, the reward is as follows:

$$\begin{array}{c}\hfill R_{\mathrm{goal}}=20\mathrm{if}\mathrm{the}\mathrm{goal}\mathrm{is}\mathrm{reached}.\end{array}$$

(138)

For collisions:

$$\begin{array}{c}\hfill R_{\mathrm{collision}}=-20\mathrm{if}\mathrm{a}\mathrm{collision}\mathrm{occurs}.\end{array}$$

(139)

Intermediate rewards encourage progress toward the goal:

$$\begin{array}{c}\hfill R_t=\left\{\begin{array}{cc}+1,\hfill & \mathrm{if}\mathrm{the}\mathrm{distance}\mathrm{to}\mathrm{the}\mathrm{goal}\mathrm{decreases},\hfill \\ -1,\hfill & \mathrm{if}\mathrm{the}\mathrm{distance}\mathrm{to}\mathrm{the}\mathrm{goal}\mathrm{increases}.\hfill \end{array}\right.\end{array}$$

(140)

The GA3C framework allows for high-speed parallel training with up to 32 simulation instances running concurrently, ensuring robust and efficient learning. A step-by-step pseudocode of the algorithm is presented in Algorithm 8.

Algorithm 8: GA3C-based Robot Navigation Framework [38]

6.2.6. A3C-Based End-to-End Navigation Strategy

The study by Shi et al. [39] introduced an Intrinsic Curiosity Module (ICM) to address the challenges of sparse rewards in navigation tasks. The ICM generates intrinsic rewards by leveraging the prediction error of a forward model, thereby encouraging exploration in complex environments. Additionally, the network is enhanced with two LSTM layers to capture temporal dependencies, enabling the agent to better utilize previous observations, actions, and rewards.

The policy $\pi (a_t|s_t;{\theta }_p)$ and value $V(s_t;{\theta }_v)$ are represented by neural networks with parameters ${\theta }_p$ and ${\theta }_v$, respectively. The A3C policy gradient and value loss functions are extended to incorporate intrinsic rewards. The policy loss is as follows:

$$\begin{array}{c}\hfill f_{\pi }({\theta }_p)=log\pi (a_t|s_t;{\theta }_p)·(R_t-V(s_t;{\theta }_v)),\end{array}$$

(141)

where the total reward $R_t$ combines extrinsic and intrinsic rewards:

$$\begin{array}{c}\hfill R_t=\sum _{i=0}^{k-1}{\gamma }^i{r}_{t+i}+{\gamma }^{k}V(s_{t+k};{\theta }_v).\end{array}$$

(142)

The value loss minimizes the discrepancy between the predicted and actual returns:

$$\begin{array}{c}\hfill f_v({\theta }_v)={\displaystyle \frac{1}{2}}{(R_t-V(s_t;{\theta }_v))}^2.\end{array}$$

(143)

To calculate intrinsic rewards, the ICM includes an inverse model and a forward model. The inverse model predicts the action $a_t$ given consecutive states $s_t$ and $s_{t+1}$:

$$\begin{array}{c}\hfill {\widehat{a}}_t=g(s_t,s_{t+1};{\theta }_I),\end{array}$$

(144)

where ${\theta }_I$ are the parameters of the inverse model. The loss for the inverse model is as follows:

$$\begin{array}{c}\hfill L_I=\sum _{i=1}^n-P(a_t^i)logq({\widehat{a}}_t^i),\end{array}$$

(145)

with $P(a_t^i)$ as the true action probability and $q({\widehat{a}}_t^i)$ as the predicted action probability. The forward model predicts the next state feature $\varphi (s_{t+1})$ based on the current state feature $\varphi (s_t)$ and action $a_t$:

$$\begin{array}{c}\hfill \widehat{\varphi }(s_{t+1})=f(\varphi (s_t),a_t;{\theta }_F),\end{array}$$

(146)

where ${\theta }_F$ are the parameters of the forward model. The forward model loss is as follows:

$$\begin{array}{c}\hfill L_F={\displaystyle \frac{1}{2}}{\parallel \widehat{\varphi }(s_{t+1})-\varphi (s_{t+1})\parallel }_2^2.\end{array}$$

(147)

The intrinsic reward $r_t^i$ is derived from the forward model’s prediction error:

$$\begin{array}{c}\hfill r_t^i={\displaystyle \frac{\eta }{2}}{\parallel \widehat{\varphi }(s_{t+1})-\varphi (s_{t+1})\parallel }_2^2,\end{array}$$

(148)

where $\eta >0$ scales the intrinsic reward. The total loss function combines policy, value, inverse, and forward losses:

$$\begin{array}{c}\hfill \mathcal{L}=-\lambda {\mathbb{E}}_{\pi (a_t|s_t;{\theta }_p)}\left[\sum _t{r}_t\right]+(1-\beta )L_I+\beta L_F,\end{array}$$

(149)

where $\lambda$ adjusts the weight of the policy gradient, and $\beta$ balances the forward and inverse model losses.

This method significantly enhances A3C by addressing sparse rewards and enabling robust navigation in complex environments. The use of intrinsic rewards, alongside the temporal memory provided by LSTMs, allows the agent to explore effectively and generalize better from simulation to real-world scenarios. The block diagram of Figure 9 illustrates the integration of the Intrinsic Curiosity Module (ICM) with A3C. The input states $s_t$ and $s_{t+1}$ are processed to extract features $\varphi (s_t)$ and $\varphi (s_{t+1})$, which feed into the inverse and forward models. The forward model generates intrinsic rewards $r_t^i$, while the policy and value networks compute the action policy $\pi (a_t|s_t;{\theta }_p)$ and value estimate $V(s_t;{\theta }_v)$. The output is the navigation action $a_t$, optimized to balance exploration and exploitation in sparse-reward environments.

6.2.7. A3C Algorithm for Autonomous Navigation with Landmark Generators

Wang et al. [40] proposed a method that enhances the A3C algorithm by introducing a hierarchical navigation framework that integrates global and local planners connected by a dynamic sub-goal generator. The global planner uses traditional path planning algorithms like A* or Dijkstra to create a global path, while the local planner employs A3C to navigate to dynamically generated sub-goals, optimizing navigation efficiency in dynamic environments.

The local planner is modeled as a partially observable Markov decision process (POMDP) defined as a 5-tuple $(T,S,O,A,R,\gamma )$, where T is the current time, S represents the environment states, O denotes the observed information, A is the action space, R is the reward function, and $\gamma$ is the reward discount factor. To address the challenge of sparse rewards, a reward mechanism is designed as follows:

$$\begin{array}{c}\hfill R_t=r_{ts}+r_{td}+r_{tc}+r_{tst},\end{array}$$

(150)

where

$$\begin{array}{ccc}\hfill r_{ts}& =& \left\{\begin{array}{cc}50,\hfill & \mathrm{if}\mathrm{the}\mathrm{robot}\mathrm{reaches}\mathrm{the}\mathrm{sub}\text{-}\mathrm{goal},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(151)

$$\begin{array}{ccc}\hfill r_{td}& =& \left\{\begin{array}{cc}-0.5,\hfill & \mathrm{if}\mathrm{distance}\mathrm{to}\mathrm{obstacles}<d_{\mathrm{safe}},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(152)

$$\begin{array}{ccc}\hfill r_{tc}& =& \left\{\begin{array}{cc}-20,\hfill & \mathrm{if}\mathrm{collision}\mathrm{occurs},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(153)

$$\begin{array}{ccc}\hfill r_{tst}& =& \left\{\begin{array}{cc}-0.05,\hfill & \mathrm{if}\mathrm{movement}\mathrm{distance}<r_{min},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(154)

The policy $\pi (a|s;\theta )$ and value $V(s;{\theta }_v)$ networks are enhanced with a shared feature extraction backbone and a memory module to retain long-term dependencies. The policy gradient for updating the actor network is as follows:

$$\begin{array}{c}\hfill \Delta \theta ={\nabla }_{\theta }log\pi (a_t|s_t;\theta )A(s_t,a_t;{\theta }_v),\end{array}$$

(155)

where the advantage function $A(s_t,a_t;{\theta }_v)$ is computed as follows:

$$\begin{array}{c}\hfill A(s_t,a_t;{\theta }_v)=R_t+\gamma V(s_{t+1};{\theta }_v)-V(s_t;{\theta }_v).\end{array}$$

(156)

The critic network minimizes the temporal difference (TD) loss:

$$\begin{array}{c}\hfill L_V={\displaystyle \frac{1}{2}}{\left(R_t+\gamma V(s_{t+1};{\theta }_v)-V(s_t;{\theta }_v)\right)}^2.\end{array}$$

(157)

The integration of the sub-goal generator with A3C further reduces the computational burden. The sub-goal generator dynamically identifies intermediate targets based on two types of frontiers, A and B, defined in the robot’s LiDAR data. Sub-goals are generated using the following rules:

$$\begin{array}{ccc}\hfill \mathrm{For}\mathrm{type}A:& & \mathrm{sub}\text{-}\mathrm{goal}\mathrm{at}{d}_{max}\mathrm{along}\mathrm{the}\mathrm{frontier}\prime \mathrm{s}\mathrm{midpoint}.\hfill \end{array}$$

(158)

$$\begin{array}{ccc}\hfill \mathrm{For}\mathrm{type}B:& & \mathrm{adjust}\mathrm{based}\mathrm{on}\mathrm{safe}\mathrm{distance}{d}_{\mathrm{safe}}\mathrm{and}\mathrm{obstacle}\mathrm{edges}.\hfill \end{array}$$

(159)

The hierarchical design of global and local planners ensures robust navigation in dynamic environments, leveraging the memory-enhanced A3C to handle long-term dependencies and optimize navigation efficiency. The block diagram of the proposed hierarchical navigation method, shown in Figure 10, exhibits how the system integrates a global planner with a sub-goal generator to dynamically identify intermediate targets for the local planner. The feature extraction backbone processes the current state $s_t$, and the reward function computes $R_t$ based on proximity, collision, and progress metrics. The policy $\pi (a|s;\theta )$ and value network $V(s;{\theta }_v)$ are optimized using the advantage function $A(s_t,a_t)$, with the final output being the navigation action $a_t$.

6.3. Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO), introduced by Schulman et al. [4], is a widely used reinforcement learning algorithm that strikes a balance between sample efficiency and computational simplicity. PPO builds upon the policy gradient methods and introduces a clipping mechanism to ensure stable and monotonic policy updates.

The goal of PPO is to maximize the expected reward $J({\pi }_{\theta })$ while ensuring that policy updates do not deviate excessively from the current policy. The optimization objective is defined as follows:

$$L^{\mathrm{CLIP}}(\theta )={\mathbb{E}}_t\left[min\left(r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ)A_t\right)\right],$$

(160)

where $r_t(\theta )$ is the probability ratio given by the following:

$$r_t(\theta )={\displaystyle \frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|s_t)}}.$$

(161)

Here, $A_t$ represents the advantage function, $ϵ$ is a hyperparameter that determines the clipping range, and $\mathrm{clip}(·)$ restricts the probability ratio to the interval $[1-ϵ,1+ϵ]$.

The advantage function $A_t$ is computed as follows:

$$A_t=Q^{\pi }(s_t,a_t)-V^{\pi }(s_t),$$

(162)

where $Q^{\pi }(s_t,a_t)$ is the action–value function, and $V^{\pi }(s_t)$ is the state–value function. PPO employs a surrogate objective function to balance exploration and exploitation while avoiding large policy updates, which could destabilize learning.

In addition to the clipping mechanism, the total loss function in PPO combines the policy loss $L^{\mathrm{CLIP}}$, the value function loss $L^{\mathrm{VF}}$, and an entropy term $L^{\mathrm{ENT}}$ to encourage policy exploration:

$$L(\theta )=L^{\mathrm{CLIP}}(\theta )-c_1{L}^{\mathrm{VF}}(\theta )+c_2{L}^{\mathrm{ENT}}(\theta ),$$

(163)

where $L^{\mathrm{VF}}$ is the squared error loss for value function approximation:

$$L^{\mathrm{VF}}(\theta )={\mathbb{E}}_t\left[{\left(V_{\theta }(s_t)-R_t\right)}^2\right],$$

(164)

and $L^{\mathrm{ENT}}$ is the entropy loss:

$$L^{\mathrm{ENT}}(\theta )={\mathbb{E}}_t\left[-\sum _a{\pi }_{\theta }(a|s_t)log{\pi }_{\theta }(a|s_t)\right].$$

(165)

The coefficients $c_1$ and $c_2$ control the relative importance of the value loss and entropy loss, respectively. A step-by-step pseudo code of PPO is presented in Algorithm 9.

Algorithm 9: Proximal Policy Optimization (PPO) Algorithm

   PPO has been applied to various domains, including robotic control, game playing, and autonomous navigation. Its robustness and simplicity make it a preferred choice for continuous and discrete action space problems. By preventing large policy changes through the clipping mechanism, PPO ensures stable learning and improved convergence properties. Some of the recent applications of this algorithm in robotic path planning and navigation are reviewed in the following sections.

6.3.1. Navigation Skills Acquisition for Wheel-Legged Robots

The study by Chen et al. [41] builds upon PPO by introducing hierarchical training strategies with secondary and primary policies, along with domain randomization to improve robustness and generalization. The approach addresses challenges such as reward sparsity, temporal credit assignment, and data inefficiency in high-dimensional navigation tasks.

The hierarchical training begins with secondary policies, each trained for specific behaviors such as moving straight, avoiding obstacles, climbing over obstacles, and squeezing through narrow passages. The secondary policies are trained using PPO with a compound loss function:

$$\begin{array}{c}\hfill L(\theta )=\mathbb{E}[{\lambda }_1{\eta }_{{\pi }_{\theta }}+{\lambda }_2{L}_v],\end{array}$$

(166)

where ${\eta }_{{\pi }_{\theta }}$ is the PPO surrogate function:

$$\begin{array}{c}\hfill {\eta }_{{\pi }_{\theta }}=\mathbb{E}\left[min\left(Z({\psi }_{\theta })A_t,{\psi }_{\theta }{A}_t\right)\right],\end{array}$$

(167)

and $L_v$ is the value function loss:

$$\begin{array}{c}\hfill L_v={\displaystyle \frac{1}{2}}{\left(V_{\theta }(s_t)-R_t\right)}^2.\end{array}$$

(168)

The probability ratio ${\psi }_{\theta }$ is defined as follows:

$$\begin{array}{c}\hfill {\psi }_{\theta }={\displaystyle \frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|s_t)}},\end{array}$$

(169)

and $Z({\psi }_{\theta })$ clips ${\psi }_{\theta }$ to $[1-ϵ,1+ϵ]$, where $ϵ$ is a clipping parameter.

The reward function for secondary policies encourages specific behaviors:

$$\begin{array}{c}\hfill r_t=r_{\mathrm{step}}+r_{\mathrm{progress}}+r_{\mathrm{invalid}},\end{array}$$

(170)

where

$$\begin{array}{ccc}\hfill r_{\mathrm{step}}& =& -0.1,\hfill \end{array}$$

(171)

$$\begin{array}{ccc}\hfill r_{\mathrm{progress}}& =& d_{t-1}-d_t,\hfill \end{array}$$

(172)

$$\begin{array}{ccc}\hfill r_{\mathrm{invalid}}& =& \left\{\begin{array}{cc}-0.1,\hfill & \mathrm{if}\mathrm{collision}\mathrm{or}\mathrm{invalid}\mathrm{action}\mathrm{occurs},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(173)

The primary policy is trained using samples from the domain-randomized batch and real interactions in complex environments. Trajectories from secondary policies are replayed in randomized environments to generate action–observation–return tuples $(s_t,a_t,R_t)$. The advantage function $A_t$ for primary policy training is as follows:

$$\begin{array}{c}\hfill A_t=R_t+\gamma V_{\theta }(s_{t+1})-V_{\theta }(s_t).\end{array}$$

(174)

The policy parameters are updated using stochastic gradient ascent on the compound loss:

$$\begin{array}{c}\hfill L(\theta )=\mathbb{E}[{\lambda }_1{\eta }_{{\pi }_{\theta }}+{\lambda }_2{L}_v].\end{array}$$

(175)

A block diagram showing the flow of computations involved in this method is presented in Figure 11.

Domain randomization improves generalization by altering non-essential aspects of the training environments, such as obstacle appearance, number, and configuration. The randomized environments ensure task-relevant sensory features are preserved while diversifying the training data. The final policy maps height-map images and robot poses to motor actions using a neural network with three convolutional layers followed by fully connected layers.

By decomposing navigation into manageable sub-tasks, utilizing domain randomization, and optimizing with PPO, the proposed method achieves robust and efficient navigation in high-dimensional environments.

6.3.2. PPO-Based Path Planning for Cleaning Robot

Moon et al. [42] propose a PPO method integrated with transfer learning (TL), detection of nearest uncleaned tiles (DNUT), reward shaping (RS), and an elite set (ES) to improve cleaning performance by robots. These additions address challenges such as reward sparsity, inefficient exploration, and data inefficiency in reinforcement learning tasks for cleaning robots.

Transfer learning allows the agent to leverage knowledge from previously trained environments to initialize training in new ones. This ensures improved convergence and robustness. The transferred policy ${\pi }_{\mathrm{TL}}$ initializes the primary policy ${\pi }_{\theta }$:

$$\begin{array}{c}\hfill {\pi }_{\theta }\leftarrow {\pi }_{\mathrm{TL}}.\end{array}$$

(176)

The DNUT technique ensures that the agent focuses on uncleaned tiles. The agent detects the nearest uncleaned tile and adjusts the reward $r_t$ accordingly:

$$\begin{array}{c}\hfill r_t=\left\{\begin{array}{cc}+1,\hfill & \mathrm{if}\mathrm{moving}\mathrm{to}\mathrm{an}\mathrm{uncleaned}\mathrm{tile},\hfill \\ -1,\hfill & \mathrm{if}\mathrm{moving}\mathrm{to}\mathrm{a}\mathrm{cleaned}\mathrm{tile},\hfill \\ -2,\hfill & \mathrm{if}\mathrm{moving}\mathrm{to}\mathrm{an}\mathrm{obstacle}.\hfill \end{array}\right.\end{array}$$

(177)

The reward shaping introduces a stacked value mechanism to encourage consistent positive behavior. The stacked value $R_{\mathrm{stacked}}$ is computed as follows:

$$\begin{array}{c}\hfill R_{\mathrm{stacked}}=\sum _{i=1}^{k}1.5^i,\end{array}$$

(178)

where k is the number of consecutive positive rewards. The total reward becomes the following:

$$\begin{array}{c}\hfill R_t=r_t+R_{\mathrm{stacked}}.\end{array}$$

(179)

The elite set (ES) ensures the agent avoids learning from poor-performance episodes. If the agent takes more than 500 steps in a single episode, the episode is terminated, and the data are excluded from training. Let $E_{\mathrm{elite}}$ denote the set of high-performance episodes:

$$\begin{array}{c}\hfill E_{\mathrm{elite}}=\{e\in E\mid \mathrm{performance}(e)>\mathrm{threshold}\}.\end{array}$$

(180)

The PPO loss function is modified to incorporate these components. The clipped objective is given by the following:

$$\begin{array}{c}\hfill L_{\mathrm{CLIP}}(\theta )={\mathbb{E}}_t\left[min\left(r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ)A_t\right)\right],\end{array}$$

(181)

where $r_t(\theta )={\displaystyle \frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|s_t)}}$ is the probability ratio and $A_t$ is the advantage function:

$$\begin{array}{c}\hfill A_t=R_t+\gamma V_{\theta }(s_{t+1})-V_{\theta }(s_t).\end{array}$$

(182)

The value network minimizes the temporal difference loss:

$$\begin{array}{c}\hfill L_V={\displaystyle \frac{1}{2}}{\left(V_{\theta }(s_t)-(R_t+\gamma V_{\theta }(s_{t+1}))\right)}^2.\end{array}$$

(183)

As presented in Algorithm 10, the agent learns the optimal policy by combining TL for initialization, DNUT for focus, RS for behavior shaping, and ES for data curation. These components significantly enhance the efficiency and robustness of PPO in cleaning environments.

Algorithm 10: The PPO-based Cleaning Method Proposed by [42].

6.3.3. PPO-Based End-to-End Path Planning for Lunar Rovers with Safety Constraints

Yu et al. [43] proposed an end-to-end path planning method based on PPO by integrating a safety-aware reward function, sliding behavior analysis, and curriculum learning to address the challenges of autonomous path planning for lunar rovers. These improvements ensure safety, stability, and adaptability across different terrains. The path planning problem is formulated as a Markov Decision Process (MDP), with the policy ${\pi }_{\theta }$ represented by a deep neural network parameterized by $\theta$. The optimal policy ${\pi }_{\theta }^{*}$ is defined as follows:

$$\begin{array}{c}\hfill u_t={\pi }_{\theta }^{*}(s_t)={\pi }_{\theta }^{*}(s_{t1},s_{t2},s_{t3}),\end{array}$$

(184)

where $s_{t1}$, $s_{t2}$, and $s_{t3}$ represent depth camera data, LiDAR data, and rover/target state information, respectively. The safety-aware reward function incorporates distance and safety considerations:

$$\begin{array}{c}\hfill r_t=r_t^{\mathrm{dis}}+{\alpha }_s·r_t^{\mathrm{safe}},\end{array}$$

(185)

where ${\alpha }_s$ is the safety factor. The distance reward $r_t^{\mathrm{dis}}$ is defined as follows:

$$\begin{array}{c}\hfill r_t^{\mathrm{dis}}=\left\{\begin{array}{cc}10,\hfill & \mathrm{if}{d}_p^t<d_{\mathrm{goal}},\hfill \\ -10,\hfill & \mathrm{if}{d}_o^t<d_{\min },\hfill \\ d_p^{t-1}-d_p^t,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(186)

where $d_p^t$ and $d_p^{t-1}$ are the rover’s distances to the target at time t and $t-1$, and $d_o^t$ is the distance to the nearest obstacle.

The safety reward $r_t^{\mathrm{safe}}$ penalizes sliding and unstable behaviors:

$$\begin{array}{c}\hfill r_t^{\mathrm{safe}}=\left\{\begin{array}{cc}-10,\hfill & \mathrm{if}({\theta }_t>{\theta }_{\max })\vee ({\varphi }_t>{\varphi }_{\max }),\hfill \\ -{\mathrm{slip}}_t·{\beta }_t,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(187)

where ${\theta }_t$ and ${\varphi }_t$ are the pitch and roll angles, ${\mathrm{slip}}_t$ and ${\beta }_t$ are the sliding rate and sliding angle, calculated as follows:

$$\begin{array}{c}\hfill {\mathrm{slip}}_t=A{e}^{B{\theta }_t},{\beta }_t=C{e}^{D{\varphi }_t},\end{array}$$

(188)

with $A,B,C,D$ as constants derived from experimental data. The policy is updated using the PPO objective with a clipped surrogate:

$$\begin{array}{c}\hfill L_{\mathrm{PPO}}(\theta )={\mathbb{E}}_t\left[min\left({\rho }_t(\theta )A_t,\mathrm{clip}({\rho }_t(\theta ),1-ϵ,1+ϵ)A_t\right)\right],\end{array}$$

(189)

where ${\rho }_t(\theta )={\displaystyle \frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|s_t)}}$, and $A_t$ is the advantage function:

$$\begin{array}{c}\hfill A_t=r_t+\gamma V_{\theta }(s_{t+1})-V_{\theta }(s_t).\end{array}$$

(190)

Curriculum learning is applied to improve adaptability by incrementally increasing terrain complexity. Training begins in flat terrains and progresses to more complex environments with adjusted safety factors ${\alpha }_s$. This progressive training ensures robust and efficient learning of safe and optimal paths. Figure 9 shows a block diagram of the method for lunar rover path planning. The input includes depth camera data, LiDAR data, and rover/target state information. Feature extraction is performed using a neural network, followed by safety-aware reward calculations and policy updates using PPO. A curriculum learning strategy incrementally increases terrain complexity, resulting in a safe and efficient navigation policy ${\pi }_{\theta }(a_t|s_t)$. Figure 12 shows a block diagram of the proposed algorithm.

6.3.4. PPO-Based Real-Time Collision Avoidance in Dense Crowds

The study by Liang et al. [44] proposes a method, CrowdSteer, that builds upon PPO by incorporating multi-sensor fusion, a hybrid reward function, and specialized training scenarios to address challenges in dense and dynamic crowd navigation. The approach uses data from a 2-D LiDAR and a depth camera to implicitly model interactions with obstacles and pedestrians, generating smoother and collision-free trajectories. The navigation task is formulated as a Partially Observable Markov Decision Process (POMDP) defined by a 6-tuple $(S,A,P,R,\mathrm{\Omega },O)$, where S represents the state space, A the action space, P the state transition probabilities, R the reward function, $\mathrm{\Omega }$ the observation space, and O the observation probability distribution. The robot’s action at time t is sampled from the policy ${\pi }_{\theta }$ as follows:

$$\begin{array}{c}\hfill a_t\sim {\pi }_{\theta }(a_t|o_t),\end{array}$$

(191)

where $o_t=[o_t^{\mathrm{lid}},o_t^{\mathrm{cam}},o_t^g,o_t^v]$ comprises LiDAR data $o_t^{\mathrm{lid}}$, camera data $o_t^{\mathrm{cam}}$, relative goal location $o_t^g$, and robot velocity $o_t^v$. The hybrid reward function encourages goal-reaching and safe navigation while penalizing collisions and oscillatory behavior:

$$\begin{array}{c}\hfill r_t=r_t^g+r_t^c+r_t^{\mathrm{osc}}+r_t^{\mathrm{safe}},\end{array}$$

(192)

where the components are defined as follows:

$$\begin{array}{ccc}\hfill r_t^g& =& \left\{\begin{array}{cc}{r}_{\mathrm{goal}},\hfill & \mathrm{if}||p_t-p_g||<\delta ,\hfill \\ 2.5(||p_{t-1}-p_g||-||p_t-p_g||),\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(193)

$$\begin{array}{ccc}\hfill r_t^c& =& \left\{\begin{array}{cc}{r}_{\mathrm{collision}},\hfill & \mathrm{if}||p_t-p_{\mathrm{obs}}||<ϵ,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(194)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{osc}}& =& -0.1|{\omega }_t|,\mathrm{if}|{\omega }_t|>0.3,\hfill \end{array}$$

(195)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{safe}}& =& -0.1||R_{\max }-R_{\min }||.\hfill \end{array}$$

(196)

The policy is trained using PPO with a clipped surrogate objective:

$$\begin{array}{c}\hfill L_{\mathrm{PPO}}(\theta )={\mathbb{E}}_t\left[min\left({\rho }_t{A}_t,\mathrm{clip}({\rho }_t,1-ϵ,1+ϵ)A_t\right)\right],\end{array}$$

(197)

where ${\rho }_t={\displaystyle \frac{{\pi }_{\theta }(a_t|o_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|o_t)}}$ is the probability ratio, and $A_t$ is the advantage function:

$$\begin{array}{c}\hfill A_t=r_t+\gamma V_{\theta }(s_{t+1})-V_{\theta }(s_t).\end{array}$$

(198)

The policy ${\pi }_{\theta }$ is represented by a neural network with separate branches for processing LiDAR and camera data, followed by fusion layers that integrate these features with goal and velocity observations. Lidar data $o_t^{\mathrm{lid}}$ and camera data $o_t^{\mathrm{cam}}$ are processed as follows:

$$\begin{array}{ccc}\hfill o_t^{\mathrm{lid}}& =& \{l\in {\mathbb{R}}^{512}:0<l_i<4\},\hfill \end{array}$$

(199)

$$\begin{array}{ccc}\hfill o_t^{\mathrm{cam}}& =& \{C\in {\mathbb{R}}^{150\times 120}:1.4<C_{ij}<5\}.\hfill \end{array}$$

(200)

During training, the policy is initialized in static environments and progressively trained in dynamic scenarios with occluded obstacles and high pedestrian densities. This curriculum learning approach improves the robot’s ability to generalize to real-world dense crowd navigation while ensuring smooth and efficient trajectories.

6.3.5. Self-Learned Vehicle Control Using PPO

Canal and Taschin [45] proposed a method that enhances PPO for vehicle control by combining minimal path planning with reinforcement learning to achieve efficient and generalizable vehicle control in obstacle-rich environments. The key innovation lies in using PPO to learn vehicle control policies that follow a pre-computed approximate path while adapting to unforeseen obstacles and optimizing for faster completion times.

The environment is framed as a POMDP, where the state $s_t$ includes relative positions of the next n checkpoints, vehicle velocity, and simulated LiDAR data. The action $a_t$ consists of continuous control variables, such as throttle and steering for cars, and x-axis and z-axis movements for drones. The policy ${\pi }_{\theta }$ is parameterized by a neural network, mapping the observed state $s_t$ to an action $a_t$:

$$\begin{array}{c}\hfill a_t\sim {\pi }_{\theta }(a_t|s_t).\end{array}$$

(201)

The reward function balances three objectives: checkpoint progress, collision avoidance, and minimizing time. The total reward $r_t$ at time t is defined as follows:

$$\begin{array}{c}\hfill r_t=r_t^{\mathrm{chk}}+r_t^{\mathrm{time}}+r_t^{\mathrm{col}},\end{array}$$

(202)

where

$$\begin{array}{ccc}\hfill r_t^{\mathrm{chk}}& =& \left\{\begin{array}{cc}1+\sum _{i=1}^k{\delta }_i,\hfill & \mathrm{if}\mathrm{checkpoint}i\mathrm{is}\mathrm{passed},\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(203)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{time}}& =& -\beta ·\Delta t,\hfill \end{array}$$

(204)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{col}}& =& \left\{\begin{array}{cc}-3,\hfill & \mathrm{if}\mathrm{collision}\mathrm{occurs},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(205)

The policy is trained using PPO with a clipped surrogate objective:

$$\begin{array}{c}\hfill L_{\mathrm{PPO}}(\theta )={\mathbb{E}}_t\left[min\left({\rho }_t{A}_t,\mathrm{clip}({\rho }_t,1-ϵ,1+ϵ)A_t\right)\right],\end{array}$$

(206)

where ${\rho }_t={\displaystyle \frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|s_t)}}$ is the probability ratio, and $A_t$ is the advantage function:

$$\begin{array}{c}\hfill A_t=r_t+\gamma V_{\theta }(s_{t+1})-V_{\theta }(s_t).\end{array}$$

(207)

The training process employs curriculum learning, progressively increasing the complexity of generated maps by adjusting the proportion of obstacles and randomizing the initial orientation of the vehicle. Early training stages use simpler maps with fewer obstacles and optimal starting orientations. The difficulty gradually increases as the agent improves, enhancing generalization to unseen environments.

Additionally, the method introduces a fractional factorial design to optimize hyperparameters and state representations. This approach ensures the efficient evaluation of different configurations without exhaustive testing, reducing overall computational costs during training. By combining minimal path planning, curriculum learning, and PPO-based control, the proposed method eliminates the need for detailed motion planning while achieving high performance in generalizable navigation tasks.

6.4. Soft Actor–Critic (SAC)

Soft Actor–Critic (SAC), introduced by Haarnoja et al. [2], is an off-policy reinforcement learning algorithm that combines maximum entropy reinforcement learning with actor–critic methods. The primary goal of SAC is to optimize a stochastic policy by maximizing both the expected cumulative reward and the entropy of the policy, encouraging exploration and robustness to model uncertainty.

The objective of SAC can be expressed as follows:

$$J(\pi )={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t\left(r(s_t,a_t)+\alpha \mathcal{H}(\pi (·|s_t))\right)\right],$$

(208)

where $\mathcal{H}(\pi (·|s_t))=-{\mathbb{E}}_{a\sim \pi }[log\pi (a|s_t)]$ is the entropy of the policy, $\alpha$ is the temperature parameter that balances exploration (via entropy maximization) and exploitation (via reward maximization), and $\gamma$ is the discount factor.

SAC employs two key networks: a stochastic policy $\pi (a|s;\theta )$ parameterized by $\theta$, and two Q-value functions $Q_{{\varphi }_1}(s,a)$ and $Q_{{\varphi }_2}(s,a)$ parameterized by ${\varphi }_1$ and ${\varphi }_2$, respectively. The use of two Q-value functions addresses the overestimation bias typically observed in value-based reinforcement learning methods. The target Q-value is computed as follows:

$$y_t=r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}\sim \mathcal{P}}\left[min(Q_{{\varphi }_1}(s_{t+1},a_{t+1}),Q_{{\varphi }_2}(s_{t+1},a_{t+1}))-\alpha log\pi (a_{t+1}|s_{t+1})\right].$$

(209)

The Q-value networks are updated by minimizing the Bellman residual:

$$L_Q({\varphi }_i)={\mathbb{E}}_{(s_t,a_t,r_t,s_{t+1})\sim \mathcal{D}}\left[{\left(Q_{{\varphi }_i}(s_t,a_t)-y_t\right)}^2\right],$$

(210)

where $\mathcal{D}$ is a replay buffer storing transitions.

The policy $\pi (a|s;\theta )$ is optimized to maximize the expected entropy-augmented reward:

$$J_{\pi }(\theta )={\mathbb{E}}_{s_t\sim \mathcal{D},a_t\sim {\pi }_{\theta }}\left[\alpha log\pi (a_t|s_t)-Q_{\varphi }(s_t,a_t)\right].$$

(211)

The temperature parameter $\alpha$ can be adjusted dynamically to control the trade-off between exploration and exploitation. The temperature is updated by minimizing the objective:

$$L_{\alpha }(\alpha )={\mathbb{E}}_{a_t\sim {\pi }_{\theta }}\left[-\alpha log\pi (a_t|s_t)-{\mathcal{H}}_{\mathrm{target}}\right],$$

(212)

where ${\mathcal{H}}_{\mathrm{target}}$ is the desired minimum entropy. The step-by-step pseudocode for the SAC algorithm is presented in Algorithm 11.

Algorithm 11: Soft Actor–Critic (SAC) Algorithm

   SAC has been shown to outperform traditional reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3) in tasks involving high-dimensional continuous action spaces. Its ability to encourage exploration through entropy maximization makes it particularly effective in environments with sparse rewards and high uncertainty.

The combination of off-policy learning, entropy regularization, and a stochastic actor enables SAC to achieve robust performance in a wide range of robotic control tasks, including locomotion, navigation, and manipulation. The recent advancements related to robotic path planning are reviewed in the following sections.

6.4.1. SAC Algorithm in Dynamic Path Planning for Mobile Robots

Yang et al. [46] improved the original SAC algorithm to address the challenges of mobile robot path planning in environments with static and dynamic obstacles. They introduced an enhanced reward function, state dynamic normalization, and a prioritized replay buffer to improve robustness and decision-making efficiency.

The SAC framework is retained, where the policy ${\pi }_{\theta }$ maps the current state $s_t$ to a stochastic action $a_t$:

$$\begin{array}{c}\hfill a_t\sim {\pi }_{\theta }(a_t|s_t),\end{array}$$

(213)

with the actor network optimized to maximize the entropy-regularized objective:

$$\begin{array}{c}\hfill J_{\pi }(\theta )={\mathbb{E}}_{s_t\sim \mathcal{D},a_t\sim {\pi }_{\theta }}\left[\alpha H({\pi }_{\theta }(·|s_t))-Q^{\pi }(s_t,a_t)\right],\end{array}$$

(214)

where $H({\pi }_{\theta })$ represents the entropy term, balancing exploration and exploitation, and $\alpha$ is the entropy temperature coefficient.

The critic networks are updated by minimizing the temporal difference (TD) error for the value function $Q^{\pi }$:

$$\begin{array}{c}\hfill L_Q(\varphi )={\mathbb{E}}_{(s_t,a_t,r_t,s_{t+1})\sim \mathcal{D}}\left[{\left(Q_{\varphi }(s_t,a_t)-\widehat{Q}(s_t,a_t)\right)}^2\right],\end{array}$$

(215)

where the target $\widehat{Q}(s_t,a_t)$ includes the entropy term:

$$\begin{array}{c}\hfill \widehat{Q}(s_t,a_t)=r_t+\gamma {\mathbb{E}}_{a_{t+1}\sim {\pi }_{\theta }}\left[Q^{\pi }(s_{t+1},a_{t+1})-\alpha log{\pi }_{\theta }(a_{t+1}|s_{t+1})\right].\end{array}$$

(216)

To enhance the reward structure, a hybrid reward function $r_t$ is defined to guide robot behavior:

$$\begin{array}{c}\hfill r_t=r_t^{\mathrm{angle}}+r_t^{\mathrm{target}}+r_t^{\mathrm{collision}}+r_t^{\mathrm{termination}},\end{array}$$

(217)

where

$$\begin{array}{ccc}\hfill r_t^{\mathrm{angle}}& =& cos({\mathbf{l}}_{\mathrm{vector}},{\mathbf{v}}_{\mathrm{robot}}),\hfill \end{array}$$

(218)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{target}}& =& -0.4{\displaystyle \frac{d_t}{500}}+1.0+\left({\displaystyle \frac{d_t^{\mathrm{prev}}-0.95d_t}{24}}\right),\hfill \end{array}$$

(219)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{collision}}& =& \left\{\begin{array}{cc}-1,\hfill & \mathrm{if}{d}_{\mathrm{obs}}<5\mathrm{m},\hfill \\ 0.5,\hfill & \mathrm{if}\mathrm{exiting}\mathrm{danger}\mathrm{zone},\hfill \end{array}\right.\hfill \end{array}$$

(220)

$$\begin{array}{ccc}\hfill r_t^{\mathrm{termination}}& =& \left\{\begin{array}{cc}100,\hfill & \mathrm{if}\mathrm{goal}\mathrm{reached},\hfill \\ -6,\hfill & \mathrm{if}\mathrm{collision}\mathrm{occurs},\hfill \\ -5,\hfill & \mathrm{if}\mathrm{border}\mathrm{exceeded}.\hfill \end{array}\right.\hfill \end{array}$$

(221)

State dynamic normalization ensures stable training by normalizing the state vector to follow a normal distribution:

$$\begin{array}{c}\hfill {\widehat{s}}_t={\displaystyle \frac{s_t-\mu }{\sigma }},\end{array}$$

(222)

where $\mu$ and $\sigma$ are the mean and standard deviation of the state distribution. The prioritized replay buffer selects samples with higher importance for training, weighted by the following:

$$\begin{array}{c}\hfill P(i)={\displaystyle \frac{p_i^{\beta }}{{\sum }_k{p}_k^{\beta }}},\end{array}$$

(223)

where $p_i$ is the priority of sample i and $\beta$ controls the influence of priority on sampling.

This combination of techniques enables SAC to efficiently navigate in complex environments, outperforming the original SAC and PPO algorithms in terms of stability and cumulative rewards. A block diagram of the proposed SAC-based method for mobile robot navigation is shown in Figure 13. The input comprises state information $s_t$, including LiDAR, velocity, and target observations. Normalized states are processed by the actor and critic networks. The prioritized replay buffer improves sampling, and the entropy adjustment refines exploration. The final policy update combines outputs to produce the optimized policy ${\pi }_{\theta }$, which ensures efficient navigation actions.

6.4.2. Sim-to-Real Transfer with Incremental Environment Complexity

The study by Chaffre et al. [47] proposed a method that builds on SAC by introducing incremental environment complexity and an improved reward shaping mechanism to achieve robust depth-based mapless navigation for mobile robots. The main enhancements address the sim-to-real transfer problem by combining domain randomization with structured environment transitions. The SAC framework is used, where the policy ${\pi }_{\theta }(a_t|s_t)$ maps states $s_t$ to actions $a_t$, optimizing an entropy-regularized objective:

$$\begin{array}{c}\hfill J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=1}^T\left(r(s_t,a_t)+\alpha H({\pi }_{\theta }(·|s_t))\right)\right],\end{array}$$

(224)

where $H({\pi }_{\theta })$ is the entropy of the policy:

$$\begin{array}{c}\hfill H({\pi }_{\theta })=-\sum _{a\in A}{\pi }_{\theta }(a|s)log{\pi }_{\theta }(a|s),\end{array}$$

(225)

and $\alpha$ is a temperature parameter balancing exploration and exploitation.

The Q-value and value functions are defined as follows:

$$\begin{array}{ccc}\hfill Q_w(s_t,a_t)& =& r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}\sim {\rho }_{\pi }}\left[V_{\psi }(s_{t+1})\right],\hfill \end{array}$$

(226)

$$\begin{array}{ccc}\hfill V_{\psi }(s_t)& =& {\mathbb{E}}_{a_t\sim {\pi }_{\theta }}\left[Q_w(s_t,a_t)-\alpha log{\pi }_{\theta }(a_t|s_t)\right].\hfill \end{array}$$

(227)

The reward function is shaped to guide the robot’s behavior effectively during training. The reward $r(s_t,a_t)$ is defined as follows:

$$\begin{array}{c}\hfill r(s_t,a_t)=\left\{\begin{array}{cc}C·d{R}_t·V_r,\hfill & \mathrm{if}d{R}_t>0,\hfill \\ r_{\mathrm{recede}},\hfill & \mathrm{if}d{R}_t\le 0,\hfill \\ r_{\mathrm{reached}},\hfill & \mathrm{if}{d}_t<d_{\min },\hfill \\ r_{\mathrm{collision}},\hfill & \mathrm{if}\mathrm{collision}\mathrm{detected},\hfill \end{array}\right.\end{array}$$

(228)

where $d{R}_t=d_{t-1}-d_t$ is the change in distance to the target, and $V_r$ is a velocity reduction factor:

$$\begin{array}{c}\hfill V_r=\left(1-max(v_t,0.1)\right)·{\displaystyle \frac{1}{max(d_t,0.1)}}.\end{array}$$

(229)

The constants $r_{\mathrm{reached}}$, $r_{\mathrm{collision}}$, and $r_{\mathrm{recede}}$ are set to encourage task completion while penalizing collisions and ineffective movements.

Incremental complexity is introduced by progressively training the policy in three simulated environments (empty room, static obstacles, and mixed static and dynamic obstacles). Transitions between environments are governed by success rates $S_{\mathrm{rate}}$ over recent episodes. If $S_{\mathrm{rate}}>\alpha$, the agent progresses to a more complex environment; if $S_{\mathrm{rate}}<\beta$, it returns to a simpler environment. The policy update is performed by minimizing the KL divergence:

$$\begin{array}{c}\hfill {\pi }^{\mathrm{new}}=arg\underset{{\pi }^{\prime }}{min}{\mathrm{D}}_{\mathrm{KL}}\left({\pi }^{\prime }(·|s_t)\parallel exp\left(Q_{{\pi }^{\mathrm{old}}}(s_t,·)-log{Z}_{{\pi }^{\mathrm{old}}}(s_t)\right)\right).\end{array}$$

(230)

Here, $Z_{{\pi }^{\mathrm{old}}}(s_t)$ is a partition function that normalizes the distribution.

This approach ensures smooth transitions between simulation and real-world scenarios by gradually increasing task complexity and refining the policy through domain randomization and reward shaping.

A block diagram of the SAC-based method is presented in Figure 14. The input includes state $s_t$ with LiDAR, velocity, and target observations. State normalization refines inputs for the actor and critic networks. The actor generates a policy ${\pi }_{\theta }(a_t|s_t)$, while the critic evaluates value functions. The hybrid reward function and entropy adjustment guide the training process. The policy update refines ${\pi }_{\theta }$, producing optimized navigation actions.

Figure 14. Block diagram of the sim-to-real transfer method of [47].

Figure 14. Block diagram of the sim-to-real transfer method of [47].

6.4.3. Dynamic Obstacle Avoidance and Path Planning Integration

Choi et al. [48] developed an SAC-based framework for decentralized dynamic obstacle avoidance and path planning integration. The enhancements to the SAC algorithm include a tailored state representation, a hybrid reward function, and a combination of reinforcement learning and classical path planning for improved navigation. The state $s_t$ is formulated as follows:

$$\begin{array}{c}\hfill s_t=\{s_t^{\mathrm{lidar}},s_t^{\mathrm{goal}},s_t^{\mathrm{speed}}\},\end{array}$$

(231)

where $s_t^{\mathrm{lidar}}=[o_{t-2}^l,o_{t-1}^l,o_t^l]$ represents LiDAR measurements from three consecutive time steps to capture the temporal dynamics of obstacles. The goal state is $s_t^{\mathrm{goal}}=[x,y]$, the relative position of the robot to the target, and the speed state $s_t^{\mathrm{speed}}=[v,\omega ]$, consisting of forward velocity v and rotational velocity $\omega$. The action $a_t=[v,\omega ]$ is defined in a continuous space:

$$\begin{array}{c}\hfill v\in [0,0.55],\omega \in [-0.60,0.60].\end{array}$$

(232)

The reward function $R(s_t,a_t)$ combines terms for goal achievement, collision avoidance, and control efficiency:

$$\begin{array}{c}\hfill R(s_t,a_t)=R_g+R_c+R_{\omega },\end{array}$$

(233)

where

$$\begin{array}{ccc}\hfill R_g& =& \left\{\begin{array}{cc}10,\hfill & \mathrm{if}{\mathrm{dist}}_{\mathrm{curr}}<0.5,\hfill \\ {\mathrm{dist}}_{\mathrm{prev}}-{\mathrm{dist}}_{\mathrm{curr}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(234)

$$\begin{array}{ccc}\hfill R_c& =& \left\{\begin{array}{cc}-10,\hfill & \mathrm{if}\mathrm{collision},\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(235)

$$\begin{array}{ccc}\hfill R_{\omega }& =& \left\{\begin{array}{cc}-0.1|\omega |,\hfill & \mathrm{if}|\omega |>0.6,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(236)

The SAC objective is augmented to optimize this hybrid reward while maintaining exploration through entropy regularization:

$$\begin{array}{c}\hfill J(\theta )=\mathbb{E}\left[\sum _{t=0}^T\left(r(s_t,a_t)+\alpha H(\pi (a_t|s_t))\right)\right],\end{array}$$

(237)

where $H(\pi (a_t|s_t))$ is the entropy of the policy, encouraging diverse action exploration.

To address navigation inefficiencies and conflicts between obstacle avoidance and target-reaching, a classical path planning component is integrated. The global path planner generates a trajectory using an A* algorithm, which provides a “look-ahead point” $p_{\mathrm{lookahead}}$ for the SAC agent:

$$\begin{array}{c}\hfill p_{\mathrm{lookahead}}=p_{\mathrm{path}}(d_{\mathrm{lookahead}}),\end{array}$$

(238)

where $d_{\mathrm{lookahead}}$ is the distance parameter guiding the point selection along the path. The relative position of the look-ahead point is incorporated into the RL agent’s input:

$$\begin{array}{c}\hfill s_t^{\mathrm{goal}}=[p_{\mathrm{lookahead}}-p_{\mathrm{robot}}].\end{array}$$

(239)

The actor and critic networks are designed with convolutional layers to process LiDAR data and fully connected layers for speed and goal components. The critic evaluates the Q-value as follows:

$$\begin{array}{c}\hfill Q(s_t,a_t)=r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}\sim \pi }\left[V(s_{t+1})\right],\end{array}$$

(240)

and the value function $V(s_t)$ is as follows:

$$\begin{array}{c}\hfill V(s_t)={\mathbb{E}}_{a_t\sim \pi }[Q(s_t,a_t)-\alpha log\pi (a_t|s_t)].\end{array}$$

(241)

By combining SAC with classical path planning and integrating dynamic obstacle prediction into the state representation, the method achieves robust and efficient navigation in complex environments. Algorithm 12 presents a step-by-step pseudocode for the method.

Algorithm 12: The SAC-based Dynamic Obstacle Avoidance and Path Planning of [48].

6.4.4. Robot Navigation in Constrained Pedestrian Environments

The study by Perez-D’Arpino et al. [49] proposed a method that extends SAC by integrating a sampling-based motion planner with a reactive sensorimotor policy to achieve dynamic navigation in constrained pedestrian environments. The problem is formulated as a Partially Observable Markov Decision Process (POMDP) defined as follows:

$$\begin{array}{c}\hfill M=(S,A,O,T,R,\gamma ),\end{array}$$

(242)

where S is the state space, A is the action space, O is the observation space, $T(s^{\prime }|s,a)$ is the state transition model, $R(s,a)$ is the reward function, and $\gamma$ is the discount factor. Observations $o_t\in O$ include $o_t=\{\mathrm{goal},\mathrm{lidar},\mathrm{waypoints}\}$, where

• goal represents the 2D coordinates of the target in the robot’s frame,
• lidar consists of 128 range measurements from a 1D LiDAR sensor, and
• waypoints are derived from a global planner using a 2D map.

The policy ${\pi }_{\theta }(a_t|o_t)$ outputs actions $a_t=[v_x,v_y,\omega ]$, where $v_x$ and $v_y$ are the linear velocities, and $\omega$ is the angular velocity.

The reward function $R(s,a)$ incorporates terms to encourage goal-reaching, minimize time, avoid collisions, and follow global waypoints:

$$\begin{array}{c}\hfill R=R_{\mathrm{goal}}+R_{\mathrm{timestep}}+R_{\mathrm{collision}}+R_{\mathrm{potential}}+R_{\mathrm{waypoint}}.\end{array}$$

(243)

Here,

$$\begin{array}{ccc}\hfill R_{\mathrm{goal}}& =& +1\mathrm{if}{\mathrm{dist}}_{\mathrm{goal}}<d_g,\hfill \end{array}$$

(244)

$$\begin{array}{ccc}\hfill R_{\mathrm{timestep}}& =& -0.001,\hfill \end{array}$$

(245)

$$\begin{array}{ccc}\hfill R_{\mathrm{collision}}& =& -1\mathrm{if}\mathrm{collision}\mathrm{occurs},\hfill \end{array}$$

(246)

$$\begin{array}{ccc}\hfill R_{\mathrm{potential}}& =& f({\mathrm{dist}}_{\mathrm{waypoint}}),\hfill \end{array}$$

(247)

where $d_g$ is the goal tolerance distance, and $f({\mathrm{dist}}_{\mathrm{waypoint}})$ is a potential-based reward inversely proportional to the distance to the next waypoint.

The SAC algorithm optimizes the expected sum of rewards while maximizing the entropy of the policy:

$$\begin{array}{c}\hfill J(\theta )=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(R(s_t,a_t)+\alpha H(\pi (a_t|o_t))\right)\right],\end{array}$$

(248)

where $\alpha$ is the entropy temperature and $H(\pi (a_t|o_t))$ is the policy entropy:

$$\begin{array}{c}\hfill H(\pi (a_t|o_t))=-\int \pi (a_t|o_t)log\pi (a_t|o_t)d{a}_t.\end{array}$$

(249)

The policy and value networks are updated using SAC’s off-policy gradient rules. The value function is computed as follows:

$$\begin{array}{c}\hfill V(s_t)={\mathbb{E}}_{a_t\sim \pi }\left[Q(s_t,a_t)-\alpha log\pi (a_t|o_t)\right],\end{array}$$

(250)

and the critic is updated by minimizing the Bellman error:

$$\begin{array}{c}\hfill L_Q=\mathbb{E}\left[{(Q(s_t,a_t)-(r(s_t,a_t)+\gamma V(s_{t+1})))}^2\right].\end{array}$$

(251)

The policy receives high-level waypoints from a motion planner and uses RL to navigate while avoiding collisions and adapting to dynamic pedestrian behavior. Training involves multi-layout environments (e.g., corridors, doors, intersections), allowing the policy to generalize to unseen, compositional environments.

6.4.5. Velocity Range-Based SAC Reward Shaping for Mapless Navigation

A novel SAC reward shaping method was proposed by Lee and Jeong [50] in which a “Velocity Range-Based Reward Shaping” technique is devised to improve map-less navigation for autonomous mobile robots. This technique adjusts the reward function to prioritize efficient and safe movements by incorporating velocity-based evaluations into the reinforcement learning framework. The robot’s state $s_t$ is defined as follows:

$$\begin{array}{c}\hfill s_t=f(x_t,p_t),\end{array}$$

(252)

where $x_t$ represents LiDAR-based distance data, and $p_t$ is the relative target coordinate.

The action $a_t=[v,\omega ]$ consists of the linear velocity v and angular velocity $\omega$, with velocity bounded by the following:

$$\begin{array}{c}\hfill v\in [-v_{\max },v_{\max }],\omega \in [-{\omega }_{\max },{\omega }_{\max }].\end{array}$$

(253)

The reward function $r(s_t,a_t)$ incorporates velocity-based evaluation and consists of the following:

$$\begin{array}{c}\hfill r(s_t,a_t)=r_{\mathrm{goal}}+r_{\mathrm{step}}+r_{\mathrm{velocity}},\end{array}$$

(254)

where

$$\begin{array}{ccc}\hfill r_{\mathrm{goal}}& =& \left\{\begin{array}{cc}{r}_{\mathrm{arrive}},\hfill & \mathrm{if}{d}_t<d_{\mathrm{goal}},\hfill \\ r_{\mathrm{collide}},\hfill & \mathrm{if}{\min }_x<c_{\mathrm{collision}},\hfill \end{array}\right.\hfill \end{array}$$

(255)

$$\begin{array}{ccc}\hfill r_{\mathrm{step}}& =& w_1{d}_t+w_2{\theta }_t,\hfill \end{array}$$

(256)

$$\begin{array}{ccc}\hfill r_{\mathrm{velocity}}& =& e_t,\hfill \end{array}$$

(257)

with $d_t$ being the distance to the target, ${\theta }_t$ the orientation error, $e_t$ the velocity-based evaluation score, and $w_1,w_2$ the importance weights.

The velocity evaluation score $e_t$ is determined by dividing the velocity range $[-v_{\max },v_{\max }]$ into n segments and assigning evaluation scores $e_1,e_2,\cdots ,e_n$ to each range:

$$\begin{array}{c}\hfill v_1<v_2<\cdots <v_n,e_1<e_2<\cdots <e_n.\end{array}$$

(258)

The policy ${\pi }_{\theta }(a_t|s_t)$ is optimized using the SAC objective, modified to include the velocity-based reward:

$$\begin{array}{c}\hfill J(\pi )=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(r(s_t,a_t)+\alpha H(\pi (a_t|s_t))\right)\right],\end{array}$$

(259)

where $\alpha$ is the entropy temperature, and $H(\pi (a_t|s_t))$ is the entropy of the policy:

$$\begin{array}{c}\hfill H(\pi (a_t|s_t))=-\int \pi (a_t|s_t)log\pi (a_t|s_t)d{a}_t.\end{array}$$

(260)

The value function $V(s_t)$ and Q-function $Q(s_t,a_t)$ are trained using the following updates:

$$\begin{array}{ccc}\hfill V(s_t)& =& {\mathbb{E}}_{a_t\sim \pi }\left[Q(s_t,a_t)-\alpha log\pi (a_t|s_t)\right],\hfill \end{array}$$

(261)

$$\begin{array}{ccc}\hfill L_Q& =& \mathbb{E}\left[{\left(Q(s_t,a_t)-\left(r(s_t,a_t)+\gamma V(s_{t+1})\right)\right)}^2\right].\hfill \end{array}$$

(262)

By incorporating velocity-based evaluation into the reward function and leveraging the entropy-regularized objective, the method achieves stable and efficient map-less navigation in dynamic environments. A block diagram of the process flow of the method is presented in Figure 15. The input includes state information $s_t$, derived from LiDAR data and the target position. State normalization prepares inputs for the actor and critic networks. The actor generates the policy ${\pi }_{\theta }(a_t|s_t)$, while the critic evaluates the Q-function. A hybrid reward function and entropy adjustment guide the training process. The replay buffer stores transitions for policy updates. The final policy update combines outputs, producing optimized navigation actions.

7. Emerging Trends in Reinforcement Learning for Robotics

Recent advancements in reinforcement learning have introduced new techniques that improve learning efficiency, generalization, and decision-making in robotics. This section explores emerging trends, including transformer-based RL, attention mechanisms, meta-learning, and transfer learning.

7.1. Transformer-Based DRL

Transformers, originally developed for natural language processing, have shown promise in reinforcement learning by improving long-term sequence modeling and decision-making. Hu et al. [51] provided a comprehensive survey on transformer-based RL, highlighting its applications in robotic manipulation, autonomous driving, and multi-agent coordination. These architectures enable better credit assignment over long time horizons and mitigate the “deadly triad” problem of RL. Similarly, Zhou et al. [52] introduced a transformer-based approach for autonomous driving, demonstrating a 41% reduction in collision rates compared to conventional reinforcement learning algorithms.

7.2. Meta-Learning and Transfer Learning in Robotics

Meta-learning aims to enable RL agents to adapt rapidly to new environments by learning how to learn. Ren et al. [53] developed a meta-RL framework that leverages human preference feedback for few-shot adaptation, allowing robots to learn complex tasks with minimal demonstrations. Furthermore, Liu and Ahmad [54] proposed a multi-task RL framework that reuses previously learned policies for efficient adaptation in continuous control tasks, reducing sample complexity and improving policy generalization.

Transfer learning has also gained traction in RL-based robotics. Jiang et al. [55] introduced a transformer-based multi-agent RL approach for cross-domain transfer in urban environments, significantly improving generalization across different cities. These methods enable RL agents to transfer knowledge across different environments, reducing training times for real-world deployment.

7.3. Advancements in Multi-Agent Reinforcement Learning

Multi-agent RL (MARL) has witnessed significant progress in handling complex coordination problems. Gabler and Wollherr [56] proposed a decentralized MARL framework with dual critics, enabling robots to optimize joint team rewards while considering individual constraints. Additionally, Song et al. [57] introduced a hierarchical reinforcement learning method for large-scale multi-agent path planning, enhancing sample efficiency through spatiotemporal abstraction.

7.4. Attention Mechanisms in Robot Navigation

Attention mechanisms have been increasingly applied to RL-based navigation tasks. Escudie et al. [58] developed an attention graph-based RL framework for multi-robot social navigation, improving interaction modeling in dense human environments. Their results demonstrated enhanced cooperation and safety in real-world scenarios.

These emerging trends illustrate the growing capabilities of RL in robotics and highlight promising directions for future research. Integrating these advanced methodologies will enable more efficient, robust, and scalable autonomous systems.

7.5. Scaling DRL for Large-Scale Robotic Systems and Multi-Agent Environments

Deploying deep reinforcement learning (DRL) in large-scale robotic systems, such as urban autonomous fleets or search-and-rescue missions, presents unique challenges related to computational efficiency, coordination, and policy transferability. Recent studies have explored various methods to improve the scalability of DRL-based approaches for multi-agent environments.

One approach is the development of hierarchical DRL techniques that enable more efficient decision-making in large agent populations. Liu et al. [59] provided an extensive survey on scalability challenges in multi-agent DRL (MADRL), highlighting the role of hierarchical structures in managing state–action space complexity. Their study emphasized that scalable algorithms must incorporate robustness mechanisms to handle dynamic environments effectively.

Graph-based learning methods have also proven effective for large-scale coordination. Zhang et al. [60] introduced a graph-based DRL technique for multi-robot task allocation, leveraging graph normalization to enhance scalability. Their approach demonstrated improved task efficiency and adaptability to varying team sizes, outperforming conventional heuristic-based task assignment methods.

Distributed and federated learning have also been explored as viable solutions for scalability. Pramuk et al. [61] proposed a federated DRL framework that enables multi-robot systems to share learned policies without centralized computation, reducing communication overhead while maintaining performance in large-scale navigation tasks. Similarly, Chen et al. [62] introduced an open-source framework, MultiRoboLearn, designed to bridge the gap between multi-agent DRL simulations and real-world multi-robot applications, ensuring better generalization across different environments.

Another critical consideration in scalability is decentralized policy learning. Liu et al. [63] presented an Independent Soft Actor–Critic (ISAC)-based method that enhances decentralized training in large-scale multi-agent path planning. Their results showed improved success rates and computation efficiency compared to centralized learning approaches.

Overall, these advancements illustrate the growing feasibility of DRL in large-scale robotic systems, paving the way for scalable, efficient, and adaptive multi-agent coordination in real-world applications.

7.6. Safe Exploration in DRL for Robotics

Ensuring safe exploration is a critical challenge in deep reinforcement learning (DRL) for robotic applications. While traditional RL methods rely on trial-and-error exploration, real-world deployment necessitates strategies that prevent unsafe actions during training and execution. Recent advancements in safe DRL have introduced several approaches, including constrained optimization, risk-aware policy learning, and uncertainty-aware exploration techniques.

One approach to safe exploration is through constrained optimization, where policies are trained under explicit safety constraints. Zhao et al. [64] introduced Safe Set Guided State-wise Constrained Policy Optimization (S-3PO), a method that ensures zero training violations by employing a safety monitor with black-box dynamics. Their approach improves sample efficiency while preventing unsafe transitions. Similarly, Xu et al. [65] proposed an Extra Safety Budget (ESB-CPO) framework that balances exploration efficiency with safety compliance, dynamically adjusting constraints throughout training.

Hierarchical reinforcement learning has also been explored for safe navigation. Roza et al. [66] developed a constrained hierarchical RL framework that integrates a safety layer to modify low-level policies, reducing collision rates while maintaining high performance. Additionally, Marchesini et al. [67] introduced a Safety-Oriented Search technique that biases policy optimization towards safer actions, leveraging an evolutionary cost optimization framework.

Uncertainty-aware policies have been proposed to address model inaccuracies in dynamic environments. Ramírez and Yu [68] developed a Monte Carlo dropout-based method to estimate uncertainty in state–action pairs, enabling safer decision-making in robotic control. Their approach significantly reduces risk while maintaining policy efficiency. Furthermore, Jayant and Bhatnagar [69] introduced a model-based constrained RL approach that leverages an ensemble of neural networks to estimate uncertainty and ensure robust constraint satisfaction.

Beyond policy optimization, risk-aware exploration strategies have been studied to enhance safety. Kim et al. [70] proposed SafeTAC, a Tsallis entropy-regularized safe RL method that improves exploration without violating safety constraints. Their method successfully mitigates overly conservative behaviors while optimizing performance in high-dimensional robotic tasks.

8. Practical Considerations for Real-World Applications

8.1. Real-World Applications and Challenges of Hybrid DRL Methods

Hybrid DRL approaches that integrate value-based and policy-based reinforcement learning have gained significant attention due to their potential to combine the sample efficiency of value-based methods with the stability of policy-gradient techniques. Recent studies highlight both the advantages and the existing challenges of these methods in mobile robot path planning.

One key advantage of hybrid approaches is their improved learning efficiency and generalization in real-world environments. Jiang et al. [71] introduced a goal-conditioned hybrid model that integrates model-based and model-free reinforcement learning, demonstrating superior learning efficiency and generalization performance in complex robotic navigation tasks. Similarly, Pinosky et al. [72] proposed a systematic approach to combining predictive models with experience-based policy learning, leading to improved sample efficiency and motor skill acquisition.

However, hybrid methods also pose challenges related to computational complexity and stability. Wang et al. [73] examined the integration of Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) in autonomous robot navigation, reporting that while hybrid models offer enhanced adaptability in unknown environments, they require careful tuning of reward functions and policy updates to prevent instability during training. Additionally, Li [74] highlighted the challenges of applying hybrid DRL techniques to underwater navigation, where continuous control tasks demand high sample efficiency and robust convergence properties.

Despite these challenges, hybrid methods continue to evolve with innovative frameworks. Jiang et al. [75] introduced the HOPE framework, which fuses reinforcement learning with trajectory-based planning, achieving higher success rates in diverse parking scenarios. Meanwhile, Feng et al. [76] developed a hybrid DRL-based navigation strategy for social robot interaction, demonstrating improved compliance with human navigation patterns in real-world environments.

Overall, while hybrid DRL methods present promising solutions for mobile robot path planning, further research is required to optimize their computational efficiency and stability for deployment in dynamic and unpredictable settings.

8.2. Optimizing DRL for Real-Time Processing in Robotics

Real-time decision-making is a fundamental requirement in robotic applications, where delays in processing sensor data and executing control commands can significantly impact performance and safety. Traditional deep reinforcement learning (DRL) methods, while effective in simulation, often struggle with real-time execution due to computational overhead, sample inefficiency, and high-dimensional state–action spaces. Recent advancements have focused on optimizing DRL for real-time robotic applications by improving computational efficiency, reducing latency, and enabling on-device processing.

One approach to achieving real-time DRL execution is on-device learning and optimization. Li et al. [77] introduced the ${\mathrm{R}}^3$ framework, which co-optimizes timing, memory, and algorithm performance for DRL on embedded robotic platforms. Their method dynamically adjusts batch sizes and replay buffer sizes to balance computational efficiency with learning stability, significantly reducing latency in real-time robotic control.

Parallel and asynchronous processing techniques have also been explored to improve DRL efficiency. Bohm et al. [78] proposed a non-blocking asynchronous DRL training architecture that decouples action execution from learning updates, allowing robots to make faster decisions while continuously refining policies in the background. Their approach significantly outperforms conventional blocking architectures, particularly in real-world robotic control tasks with variable communication delays.

Another strategy is to leverage lightweight neural architectures for real-time inference. Sain et al. [79] discussed the use of compressed deep networks and network quantization techniques to accelerate DRL policy execution on robotic hardware. Their findings suggest that reducing model complexity without sacrificing policy performance is key to achieving real-time responsiveness.

Additionally, model-based reinforcement learning (MBRL) has been proposed as an alternative to traditional model-free DRL approaches. Sacks et al. [80] introduced Deep Model Predictive Optimization (DMPO), which integrates model predictive control (MPC) with reinforcement learning to improve planning efficiency in high-speed robotic tasks. By incorporating predictive models, DMPO enables robots to make faster and more accurate decisions, reducing the reliance on large sample collections.

8.3. Empirical Performance Comparisons of DRL Methods

To complement the theoretical discussions, we summarize recent empirical evaluations of DRL-based mobile robot path-planning techniques.

Table 1. Performance Comparison of DRL Methods in Mobile Robot Path Planning.

Study	DRL Method	Success Rate (%)	Training Time (h)	Robustness to Noise
[81]	DQN, DDQN	78	24	Moderate
[53]	PPO, SAC	88	12	High
[82]	DDPG, SAC	91	10	High
[83]	DQN variants	85	18	Moderate
[84]	Improved DQN	82	14	High
[85]	SAC (Improved)	95	9	Very High

presents key performance metrics across different DRL methods.

Liu et al. [81] evaluated Deep Q-Networks (DQN) and its variants (DDQN) for robot navigation in cluttered environments, reporting an average success rate of 78%. However, these value-based methods required longer training times (24 h on average) and exhibited moderate robustness to sensor noise. Conversely, Ren et al. [53] found that policy-based techniques like Proximal Policy Optimization (PPO) and Soft Actor–Critic (SAC) achieved success rates above 88% while converging in just 12 h.

Further studies Ruqing et al. [82], Gök [83] showed that actor–critic methods (DDPG, SAC) outperformed value-based methods, achieving a 91% success rate with better generalization in dynamic environments. Notably, Zhao et al. [85] demonstrated that an improved SAC algorithm yielded the highest success rate (95%) and exhibited superior robustness, making it a promising approach for real-world deployment.

9. Samples of the Results

9.1. Evaluation of PPO in Robotic Path Planning

In their work on Proximal Policy Optimization (PPO), Schulman et al. [4] evaluated the algorithm across several simulated robotic tasks using the MuJoCo physics engine. The tasks included challenging continuous control problems, such as HalfCheetah, Hopper, and Walker2d. Each task was run for one million timesteps, and the experiments used a fully connected neural network with two hidden layers of 64 units for the policy representation. Key hyperparameters, including the clipping threshold $ϵ=0.2$, were tuned to maximize performance. The authors compared PPO against other algorithms like Trust Region Policy Optimization (TRPO), vanilla policy gradient methods, and Advantage Actor Critic (A2C), demonstrating that PPO consistently achieved superior performance in terms of sample efficiency and stability. Figure 16 presents a comparative performance analysis of PPO against various algorithms across seven MuJoCo environments. This figure illustrates the training curves for each method, highlighting the robustness of PPO in achieving higher cumulative rewards over the training period. It effectively showcases PPO’s advantage in continuous control tasks, making it a crucial inclusion for understanding its contributions to robotic path planning.

9.2. DDPG for Continuous Robotic Control

Lillicrap et al. [3] proposed the DDPG algorithm to address challenges in continuous action domains, employing an actor–critic framework with deterministic policy gradients. The experimental evaluation covered various challenging physical control tasks simulated using MuJoCo, including classic problems, such as cartpole swing-up, pendulum balance, and locomotion with cheetah-like agents. The study involved training on both low-dimensional state descriptions (e.g., joint angles) and high-dimensional pixel inputs, showcasing DDPG’s ability to learn policies “end-to-end”. The experiments used consistent hyperparameters and network architectures across tasks, with reward functions tailored to encourage task-specific behaviors such as smooth locomotion or object manipulation. Policies were evaluated periodically without exploration noise to assess stability and convergence. Figure 17 presents performance curves for a variety of environments. This figure demonstrates the effectiveness of DDPG, highlighting the impact of using components like target networks and batch normalization. Notably, the inclusion of these components led to more stable learning and better overall performance, especially when training on pixel-based observations.

9.3. Accelerated Learning in Robotic Navigation with Assisted DDPG

Xie et al. [27] introduced the Assisted Deep Deterministic Policy Gradient (AsDDPG) framework, which incorporates an external controller, such as a PID controller, to enhance the training efficiency of Deep Reinforcement Learning (DRL) for robotic navigation. The framework initially uses the external controller to guide policy exploration, enabling the agent to learn basic behaviors like moving in a straight line. Over time, the actor network surpasses the controller’s performance and operates independently, while the controller is gradually discarded. The experimental setup involved training in simulation environments (Stage and Gazebo) with three levels of complexity: empty, simple, and complex obstacle layouts. Performance was evaluated using metrics like collision rates, goal-reaching frequency, and learning curves under varying network configurations and reward functions.

Figure 18 shows the smoothed learning curves of AsDDPG and standard DDPG under different network hyperparameters, demonstrating that AsDDPG not only accelerates training but is also more robust across hyperparameter variations. AsDDPG achieves higher average performance compared to DDPG, even under challenging conditions. This result underscores the effectiveness of leveraging an external controller to stabilize and speed up learning in DRL-based navigation tasks.

9.4. Enhancing Sample Efficiency with Conservative Model-Based Actor–Critic

Wang et al. [86] introduced the Conservative Model-Based Actor–Critic (CMBAC), a reinforcement learning framework designed to address the challenge of model inaccuracies in complex environments. CMBAC employs an ensemble of probabilistic neural networks to approximate the posterior distribution of Q-values, using the average of the bottom-k estimates to optimize policies conservatively. This approach ensures that the agent avoids unreliable promising actions that arise from overestimation biases. The experiments were conducted on a range of MuJoCo environments, including Walker2d, HalfCheetah, and Humanoid, to evaluate sample efficiency and robustness. Metrics such as cumulative reward and learning curves were used to compare CMBAC with state-of-the-art algorithms like MBPO, MOPO, and SAC.

A notable result is presented in Figure 19, which illustrates the average returns of CMBAC and baseline methods across six benchmark tasks. The figure demonstrates that CMBAC consistently outperforms competing algorithms in sample efficiency, achieving comparable or superior performance with fewer environment interactions. The shaded regions in the figure represent the standard deviation across multiple random seeds, highlighting the robustness of CMBAC even in noisy scenarios.

9.5. Leveraging Auxiliary Tasks for Enhanced Navigation in Dynamic Mazes

Mirowski et al. [34] proposed a navigation framework leveraging auxiliary tasks, including depth prediction and loop closure detection, to improve the training efficiency and performance of reinforcement learning agents. Using the A3C algorithm, the authors trained agents in visually rich, 3D mazes with dynamic goals, such as random goal mazes and an I-maze inspired by rodent navigation experiments. The auxiliary tasks, predicting depth and detecting loop closures, provided additional learning signals that accelerated the development of spatial reasoning and navigation capabilities. The experiments involved static and dynamic mazes of varying sizes and complexities, with performance measured by cumulative rewards, area under the learning curve, and position decoding accuracy. Key findings demonstrated that incorporating auxiliary tasks improved data efficiency, robustness, and stability, enabling agents to approach human-level performance in some scenarios.

Figure 20 compares the learning curves of agents with and without auxiliary tasks across different maze configurations. The figure highlights how agents augmented with depth prediction and loop closure detection achieve faster convergence and higher final performance. This result underscores the value of auxiliary tasks in enabling more efficient reinforcement learning in complex navigation domains.

9.6. Real-World Autonomous Navigation Using DRL

Surmann et al. [38] proposed a DRL framework for the autonomous navigation of mobile robots in real-world, unstructured indoor environments. The system uses fused data from a 2D laser scanner and an RGB-D camera as input, while the output consists of linear and angular velocity commands generated by an asynchronous advantage actor–critic (GA3C) network. The robot is trained in a custom simulation environment capable of parallel processing, which accelerates learning by simulating multiple environments simultaneously. Key features of the framework include the fusion of 2D and 3D sensor data for robust obstacle detection, the use of a simplified network architecture to optimize for hardware constraints, and a reward function tailored to encourage goal-directed navigation while minimizing collisions. The learned policy is deployed on a real robot (Turtlebot 2), demonstrating its capability to navigate efficiently without requiring pre-built maps or external planners.

A particularly illustrative result is presented in Figure 21, which compares training results for 1D and 2D convolutional network architectures over 20,000 episodes. The figure highlights that 1D convolutions achieve comparable navigation performance while reducing training time significantly, making them more practical for real-world applications. Additionally, the red-circled area in the figure indicates a reduction in environmental diversity during later training stages, focusing on challenging scenarios to refine policy performance.

9.7. Collision Avoidance in Dense Crowds Using Hybrid Sensors and DRL

Liang et al. [44] proposed a novel algorithm called CrowdSteer, designed for real-time collision avoidance in densely crowded environments. The approach uses hybrid sensor data from a 2-D LiDAR and a depth camera, fused via a deep reinforcement learning (DRL) framework to enable robust navigation. The problem is modeled as a Partially Observable Markov Decision Process (POMDP) and solved using Proximal Policy Optimization (PPO). High-fidelity simulations, including complex pedestrian interactions, were used to train the policy, ensuring a smooth sim-to-real transfer. The CrowdSteer algorithm generates collision-free trajectories in scenarios with varying crowd densities, narrow passages, and occlusions. Evaluations were conducted on both Turtlebot and Jackal robots, demonstrating superior performance compared to state-of-the-art methods like Dynamic Window Approach (DWA) and Fan et al.’s approach. A key visualization of the results is presented in Figure 22, which compares the trajectories generated by CrowdSteer and Fan et al.’s method in static and occluded pedestrian scenarios. The figure highlights that CrowdSteer produces smoother trajectories and avoids oscillations, resulting in significantly improved success rates in dense environments. These results emphasize the effectiveness of combining LiDAR and depth camera data in achieving robust, real-world collision avoidance.

9.8. Robot Navigation in Constrained Pedestrian Environments Using Reinforcement Learning

Pérez-D’Arpino et al. [49] proposed a reinforcement learning-based approach for enabling robots to navigate efficiently around pedestrians in constrained indoor environments, such as corridors and intersections. Their method combines a global motion planner with a reinforcement learning (RL) agent that handles local interactions with moving pedestrians. The RL policy is trained in a compositional multi-layout regime, where canonical layouts, such as corridors and doorways, are used to develop robust navigation strategies. These policies generalize well to unseen, more complex layouts and 3D reconstructions of real-world environments like a supermarket and an apartment. The experiments were conducted using the Interactive Gibson Simulator and the Soft Actor–Critic (SAC) algorithm to optimize the navigation policies. The evaluation metrics included success rate, collision rate, and personal space overlap, highlighting the efficacy of the multi-layout training approach. A key result is depicted in Figure 23, which compares the success rate of the RL-based approach with a traditional motion planner under varying pedestrian densities. The figure shows that the proposed method significantly outperforms the planner, particularly in high-density scenarios, by avoiding the “freezing robot problem” and enabling smoother navigation. This demonstrates the robustness and scalability of the approach in dynamic and constrained environments.

9.9. Further Case Studies and Real-World Applications of DRL

Deep reinforcement learning (DRL) has been successfully applied to various real-world robotic tasks, enabling autonomous agents to perform complex decision-making and control operations in dynamic environments. This section highlights notable case studies that illustrate the practical deployment of DRL in robotics.

One prominent application of DRL is in robotic manipulation tasks. Krishnan [87] demonstrated the use of DRL for robotic arm control in medical applications, such as instrument-assisted surgery and object manipulation in constrained environments. Their approach maps raw pixel inputs from a camera to control commands using a deep reinforcement learning neural network, enabling precise manipulation with minimal manual intervention.

Another area of successful DRL implementation is autonomous robot navigation. Farooqui [88] applied DRL-based techniques to real-time mobile robot path planning, improving navigation efficiency in dynamic environments. By combining visual perception with reinforcement learning policies, their approach enabled robots to make real-time adjustments in response to obstacles, significantly reducing collision rates.

DRL has also been integrated into multi-agent robotic systems to enhance coordination and collaboration. Orr and Dutta [89] explored multi-agent deep reinforcement learning (MADRL) for multi-robot applications, demonstrating its effectiveness in cooperative tasks such as warehouse automation and swarm robotics. Their findings indicate that DRL-based policies enable robots to learn from each other’s experiences, improving task efficiency and adaptability.

Additionally, DRL has proven effective in motion control for industrial and agricultural robotics. Liu [90] showcased DRL-based trajectory optimization in robotic manipulators for automated fruit picking, achieving significant improvements in efficiency and precision. Their study also highlighted the challenges of transferring learned policies from simulations to real-world deployments, emphasizing the importance of domain adaptation techniques.

10. Conclusions and Future Directions

This review provides a comprehensive analysis of the application of Deep Reinforcement Learning (DRL) in mobile robot path planning, showcasing its potential to tackle dynamic and uncertain environments where traditional methods often fall short. By leveraging neural networks to handle high-dimensional state–action spaces, DRL enables mobile robots to make real-time decisions with enhanced efficiency and adaptability. The categorized exploration of value-based methods, policy-based approaches, and hybrid frameworks demonstrates the breadth and versatility of DRL techniques in addressing complex navigation challenges. Key advancements, such as Proximal Policy Optimization (PPO), Soft Actor–Critic (SAC), and hybrid frameworks, have delivered a state-of-the-art performance in scenarios ranging from indoor navigation to multi-agent coordination. Despite this progress, limitations persist in scalability, safety, interpretability, and generalization to unseen environments. Addressing these challenges is essential for fully harnessing the transformative potential of DRL in autonomous navigation.

The future of DRL in mobile robot path planning lies in overcoming these limitations and expanding its applicability to real-world scenarios. One critical area is scalability, as current DRL models often struggle with large and complex environments. Distributed and hierarchical learning frameworks could address this by decomposing navigation tasks into subproblems. For instance, a warehouse robot might use a hierarchical approach to first navigate to a specific aisle and then locate a target object within it. Real-time performance improvements will also be essential, especially for logistics robots in high-demand environments like fulfillment centers.

Ensuring safety in navigation is another crucial direction, particularly for robots operating in human-centric environments. By integrating DRL with safety constraints and verification mechanisms, robots can learn to avoid risky behaviors. Autonomous delivery robots in crowded urban areas, for example, could benefit from policies that account for pedestrian movements and dynamic obstacles while maintaining a reliable path to their destination.

The gap between simulation and reality remains a significant challenge. While simulation environments are invaluable for training DRL models, real-world deployment often suffers from discrepancies in environmental conditions. Techniques like domain adaptation and transfer learning could help bridge this gap. For example, autonomous vehicles trained on simulated highways could adapt their policies to real-world conditions involving variable weather, road textures, and traffic dynamics.

Coordination among multiple robots presents another promising avenue for DRL. Multi-agent reinforcement learning frameworks could promote collaboration and efficient resource use in shared environments. Search-and-rescue missions, for instance, could benefit from teams of drones and ground robots working together to navigate and locate survivors in disaster-stricken areas. Similarly, such frameworks could optimize fleets of warehouse robots for collaborative tasks, reducing bottlenecks and improving throughput.

Socially aware navigation is another frontier where DRL could play a transformative role. Incorporating human preferences and social norms into reward functions can help robots operate seamlessly in public spaces. Service robots in airports, shopping malls, or hospitals must navigate crowded areas while respecting personal space and pedestrian flow patterns. These considerations could enhance their acceptance and utility in everyday human environments.

Energy efficiency is an often-overlooked aspect of path planning but is critical for battery-powered mobile robots. Reward functions that factor in energy consumption and optimize for sustainable operation could extend the operational range of robots in applications like agricultural fieldwork or remote inspection tasks. For instance, agricultural robots managing tasks like planting or weed removal would benefit from energy-efficient navigation strategies that maximize battery life across large fields.

The exploration of unstructured terrains presents additional challenges and opportunities for DRL. Robots operating in forests, disaster sites, or underwater environments require policies capable of handling incomplete environmental data and complex dynamics. Exploration-based DRL approaches, which use curiosity-driven rewards, could enable robots to navigate unknown terrains effectively. For example, underwater robots conducting coral reef surveys could leverage these techniques to explore and document fragile ecosystems while avoiding obstacles.

Deep reinforcement learning presents significant computational challenges when deployed on real-world robotic systems, particularly due to the high-dimensional sensory inputs required for robust decision-making. LiDAR and vision-based sensing are widely used in mobile robot path planning, but processing these inputs in real-time demands specialized hardware configurations.

Gautier et al. [91] highlighted that multi-robot systems relying on DRL for task allocation face severe computational bottlenecks, particularly when cloud resources are unavailable. Their study demonstrated that distributing computation across local robotic clusters improves efficiency but introduces coordination challenges. Similarly, Kwon et al. [92] explored Hyperdimensional Computing (HDC) as a lightweight alternative to deep learning for sensorimotor control, achieving a 14.2× improvement in computational efficiency while maintaining comparable accuracy.

For autonomous navigation, high-dimensional image data processing is another major concern. Vijetha and Geetha [93] proposed a state abstraction technique to transform raw vision data into compact representations, significantly reducing the computational overhead of DRL policies running on resource-constrained devices. Additionally, Xiang et al. [94] introduced RMBench, a benchmarking framework for DRL-based robotic manipulation, revealing that Soft Actor–Critic (SAC) achieves the best trade-off between computational cost and task performance.

Another critical challenge is balancing computational efficiency with training stability. Han et al. [16] proposed a deep stochastic Koopman operator (DeSKO) to model complex robot dynamics, improving decision-making efficiency while mitigating the impact of computational constraints. Their approach demonstrated robust performance in real-world soft robotic systems.

Given these considerations, future research should focus on optimizing DRL architectures for edge computing and developing hybrid approaches that integrate learning-based methods with classical control techniques to alleviate hardware limitations.

Finally, advancements in integrating vision and multimodal sensory systems could significantly enhance DRL’s robustness in real-world applications. Combining inputs from cameras, LiDAR, and other sensors could enable robots to build richer environmental models. Autonomous drones delivering packages in urban areas, for instance, could utilize multimodal inputs for precise obstacle detection and landing.

As DRL continues to evolve, its potential to generalize across unseen scenarios will become increasingly important. Approaches like meta-reinforcement learning and few-shot learning could enable robots to adapt rapidly to new tasks or environments with minimal additional training. For example, autonomous robots in industrial settings could quickly learn to navigate newly reconfigured layouts or operate in entirely different facilities.

By addressing these challenges and directions, DRL could become a cornerstone of mobile robot navigation, paving the way for its widespread deployment in diverse applications. These advancements will not only improve the adaptability and efficiency of robots but also enhance their safety and reliability, enabling them to meet the demands of real-world scenarios effectively.