Deep-Reinforcement-Learning-Based Motion Planning for a Wide Range of Robotic Structures

Abstract

The use of robot manipulators in engineering applications and scientific research has significantly increased in recent years. This can be attributed to the rise of technologies such as autonomous robotics and physics-based simulation, along with the utilization of artificial intelligence techniques. The use of these technologies may be limited due to a focus on a specific type of robotic manipulator and a particular solved task, which can hinder modularity and reproducibility in future expansions. This paper presents a method for planning motion across a wide range of robotic structures using deep reinforcement learning (DRL) algorithms to solve the problem of reaching a static or random target within a pre-defined configuration space. The paper addresses the challenge of motion planning in environments under a variety of conditions, including environments with and without the presence of collision objects. It highlights the versatility and potential for future expansion through the integration of OpenAI Gym and the PyBullet physics-based simulator.

1. Introduction

The significance of robot manipulators in engineering applications and scientific research has increased substantially in recent years. This surge can be attributed to the more frequent application of technologies such as autonomous robotics, physics-based simulation, etc. [1,2], within Industry 4.0 [3], along with the utilization of artificial intelligence techniques. As technology continues to advance, the convergence of robotics and artificial intelligence within the concept of Industry 4.0 holds immense promise. This requires a comprehensive exploration and understanding of the transformative potential inherent in these advancements.

The motion planning problem, shown schematically in Figure 1, is a fundamental and challenging research topic in industrial robotics. It involves generating a collision-free path or trajectory for various robotic structures to perform specific tasks while adhering to constraints and avoiding obstacles. The main objective is to determine an efficient path for the robotic manipulator $\mathcal{R}$ to move from its initial configuration ${\theta }_s$ to the desired final configuration ${\theta }_f$ and generate reference inputs for the motion control system. Traditionally, formulations of the motion planning problem for robotic manipulators rely on the concept of configuration space ($\mathcal{C}$ or C-space) [4,5], representing all possible transformations based on the robot’s kinematics model.

The traditional task of planning trajectories for robots with a high number of degrees of freedom (DoF), such as industrial robots (typically four to seven DoFs), has undergone a significant revolution in motion planning with the development of sampling-based algorithms [6,7,8]. Representative examples of these algorithms include Rapidly Exploring Random Trees (RRTs) and Probabilistic Roadmaps (PRMs). Another approach to solving the path planning problem, which was tested in complex scenarios such as a social environment with a human as a dynamic obstacle [9], was developed based on fuzzy logic. The motion planning problem has also been addressed using evolutionary computation techniques, as discussed in the comprehensive review [10]. Such problems were even used to create benchmarking instances for evolutionary computation methods [11,12,13]. Another group of approaches ti solving the motion planning problem involves using reinforcement learning (RL) algorithms [14] to complete partial tasks within a specific area. In contrast to traditional methods, which often rely on pre-defined maps and algorithms, RL has emerged as a groundbreaking paradigm, revolutionizing path planning by allowing autonomous agents to learn and adapt their strategies through experience.

In recent years, the combination of deep neural networks (DNN) [15] and reinforcement learning, termed deep reinforcement learning (DRL) [16], has become a popular choice for end-to-end control in robotics research, where sequential actions are learned directly from raw input observations. This statement is supported by several review articles that address the research problem [17,18,19]. These articles highlight various research gaps that merit further investigation. One of the identified gaps is the lack of versatility of robotic structures used to teach specific skills. In addition, issues related to reliability and accuracy are frequently discussed in the articles. Future research directions include the exploration of safe navigation for robots, where they must autonomously navigate their environment without causing harm to themselves or encountering obstacles within the configuration space.

One of the fundamental areas of research in motion planning that utilizes RL algorithms involves navigating in structured environments with the objective of reaching predetermined [20,21] or randomly positioned targets [22,23,24,25,26]. However, these environments lack the presence of collision objects. Among the most commonly used algorithms for the presented problem are actor–critic model-free algorithms such as the Deep Deterministic Policy Gradient (DDPG) [27], the Twin Delayed Deep Deterministic Policy Gradient (TD3) [28], and the Soft Actor–Critic (SAC) [29], often used with an experience replay buffer [30]. Mobile platforms extended with robotic arms are used in addition to robotic arms to solve problems in this area. The research described in [31] presents the generation of robot base motions for mobile platform systems. However, collision detection within the environment is not considered in the approach presented in the article. Another article describes the use of RL algorithms, specifically DRL, to solve inverse kinematics problems in 7-DoF robotic manipulators [32]. The method employs Products of Exponentials as a forward kinematics computation tool and the Deep Q-Network as an inverse kinematics solver. The article notes a limitation of the proposed method in the inclusion of self-collision detection. Another approach integrates DRL to explore the length-optimal path in Cartesian space and to derive the energy-optimal solution to inverse kinematics [33]. This solution combines a so-called traditional path planner, which includes collision detections, and DRL to find the optimal solution for redundant robotic manipulator. The other one uses the base DRL to avoid collisions with humans in the working space of the robotic arm. The approach described in [34] proposes the safety shield in scenarios with a high probability of a collision.

DRL algorithms are also used in the field of robotics to solve complex tasks, such as grasping, pushing, and sliding objects within the environment. The approach described in [35] presents on-policy and off-policy RL algorithms for object manipulation tasks. The comparison indicates that when obtaining new experience is costly and computation power is not a concern, the off-policy method SAC may be preferable. Conversely, if data collection is less expensive, such as in simulations, the on-policy method Proximal Policy Optimization (PPO) can achieve better results. In another article, as described in [36], an effective approach is presented to use RL for robotic manipulation based on simulated locomotion demonstrations. The proposed approach has been evaluated on 13 tasks using the 7-DoF Fetch robotic arm. One limitation of the method is its inability to avoid potential collisions with objects. On the other hand, the approach described in [37] presents a new automated method for complex object manipulation in environments with obstacles. The method is based on hindsight goal generation and its extensions. Valuable hindsight goals are selected by balancing graph-based diversity and proximity metrics. The results are demonstrated using the Kuka collaborative robotic manipulator with 7-DoF. DRL algorithms are also used as tools in robotics research to enable closed-loop dynamic control for soft robotic manipulators [38]. This adoption of DRL algorithms reflects a wider trend in robotics research, where various construction structures, including conventional “standard” robotic manipulators, low-budget, or even self-made robots, are increasingly being used for control tasks. This trend demonstrates the increasing recognition of the effectiveness of DRL in managing complex control issues across diverse robotic platforms, promoting innovation, and expanding the availability of advanced control methods within the field.

The primary motivation of this paper is to improve the versatility of DRL methods in robotic applications to a wide range of robotic structures, which was identified as one of the major research gaps in contemporary reviews [17,18,19]. Although the use of DRL methods in the field of industrial robotics is growing in popularity, the implementation of these methods on various robotic structures, as well as reaching targets in environments with the presence of collision objects, represents a relatively unexplored area of research. The novelty of the proposed solution lies in the reliability and accuracy of the solution for the safe navigation of a wide range of robotic structures in a predefined environment.

To demonstrate the versatility of the proposed method in this paper, robotic structures that are part of a robotic laboratory Industry 4.0 Cell [39] were used. The laboratory contains a variety of robotic structures, as shown in Figure 2, which represents a set of the most common geometric representations used for experiments in robotic research, that is, an industrial robot ABB IRB 120 [40] with six degrees of freedom, the same robot extended by a linear axis providing seven DoFs, a SCARA robot Epson LS3-B401S [41] with four DoFs, a dual-arm collaborative robot ABB IRB14000 [42] with seven DoFs on each arm, and finally, a collaborative robot Universal Robots UR3 [43] with six DoFs.

The structure of this paper is as follows. Section 2 delves into reinforcement learning methods, providing an overview of the fundamental concept of reinforcement learning. Additionally, it discusses the integration of deep learning techniques into traditional reinforcement learning methods, known as deep reinforcement learning. Section 3 is dedicated to Deep-Reinforcement-Learning-Based planning algorithms, specifically designed for tasks aimed at achieving randomly generated targets within a predefined configuration space. Finally, Section 4 summarizes the main findings and insights presented throughout this paper. It also offers reflections on the implications of the study and suggests potential avenues for future research.

2. Fundamental Concept of Reinforcement Learning

Reinforcement learning [14] is an area of machine learning in which an intelligent agent learns how to achieve specific behavior by maximizing rewards from the environment through the actions it chooses to execute, much like a human being, through the trial and error method. A higher reward encourages the agent to choose the selected action in the future, while a lower reward, also known as a penalty, has the opposite effect. The idea is that the agent will learn how to achieve its behavior within the specific environment in an effective method that converges on an optimal solution.

The problem of reinforcement learning is formally defined in terms of optimal control of a Markov Decision Process (MDP). Figure 3 illustrates its basic structure in an iteration loop. A characteristic property of MDPs is that each state is only dependent on the previous state, i.e., a memoryless property where each state contains all the information necessary to predict the next state.

A Markov Decision Process is a discrete-time stochastic control process and can be represented as the tuple (S, A, R, p), where S is the continuous multidimensional state space, and A is the continuous multidimensional action space. The state transition probabilities are defined by the function $p:S\times S\times A\to [0,1]$, which represents the probability density of the next state $s^{\prime }\in S$ based on the current state $s\in S$ and action $a\in A$.

$$p(s^{\prime }\mid s,a)=\mathbb{P}\left\{S_{t+1}=s^{\prime }\mid S_t=s,A_t=a\right\}$$

(1)

The reward function, defined by $R:S\times S\times A\to \mathbb{R}$, represents the immediate reward that the agent receives after the transition from $s\in S$ to $s^{\prime }\in S$ using an action $a\in A$.

$$R(s,a,s^{\prime })=\mathbb{E}\left\{R_t\mid S_t=s,A_t=a,S_{t+1}=s^{\prime }\right\}$$

(2)

The objective of the process is for the agent to learn optimal behavior, also known as a policy denoted as $\pi =\pi (a\mid s)$, which maximizes its discounted expected return. A discount factor $\gamma$ in the range $(0,1]$ is usually introduced when calculating the long-term reward, resulting in the following expression for the continuous ongoing process:

$$G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots =\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+1+k},$$

(3)

where $\gamma$ determines the priority of long-term future rewards.

In the field of reinforcement learning, the value function is a crucial concept that guides decision-making processes. It serves as a quantitative measure of the expected cumulative reward that an agent can anticipate while occupying a specific state or following a particular action. This metric helps the agent assess the desirability of states and actions, facilitating the learning of an optimal policy.

More formally, a state value function can be defined to determine the expected return when following a policy $\pi$ for a particular state s, with the value function $V^{\pi }\left(s\right)$.

$$V^{\pi }\left(s\right)={\mathbb{E}}_{\pi }[G_t\mid S_t=s]$$

(4)

The action value function, which represents the expected discounted return when starting in state s and initially taking action a but then following policy $\pi$, can be defined as follows:

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }[G_t\mid S_t=s,A_t=a].$$

(5)

The primary goal of the agent in reinforcement learning is to find the optimal policy ${\pi }^{*}$, which is better than or equal to all other policies. This can be achieved by estimating the corresponding optimal functions for the action value function, defined as follows

$$Q^{*}(s,a)=\underset{\pi }{\max }{Q}^{\pi }(s,a).$$

(6)

The optimal action value function is satisfied by the Bellman equation, a fundamental concept in reinforcement learning. The Bellman optimality equation explains that the value of a state is equal to the expected return when the best action $a^{\prime }$ is taken in that state. The equation for $Q^{*}$ is defined as follows:

$$Q^{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma \underset{a^{\prime }}{\max }{Q}^{*}(s^{\prime },a^{\prime })\mid S_t=s,A_t=a].$$

(7)

The Bellman optimality equation is a fundamental formula in reinforcement learning theory. After finding the optimal action value function $Q^{*}(s,a)$, the optimal policy ${\pi }^{*}$ can be followed by selecting the optimal action $a^{*}$ in each state s, defined as

$$a^{*}\left(s\right)=\underset{a^{\prime }}{\mathrm{argmax}}{Q}^{*}(s,a^{\prime }).$$

(8)

It is important to note that the optimization of the value function is often performed off-policy and can, therefore, utilize experience replay.

The well-known and straightforward algorithm for this type of problem is Q-learning, as described in [44]. The Q-learning algorithm belongs to the category of Temporal Difference (TD) learning [45]. TD learning is a form of a value-based approach in which the value function is optimized by minimizing the TD error, denoted as $\delta$.

$${\delta }_t=R_t+\gamma \underset{a^{\prime }}{\max }{Q}^{\pi }(s^{\prime },a^{\prime })-Q^{\pi }(s_t,a_t)$$

(9)

The Q-learning update rule for the estimation of $Q^{*}(s_t,a_t)$ becomes an optimization problem described as follows

$$Q^{*}(s_t,a_t)\leftarrow Q^{\pi }(s_t,a_t)+\alpha {\delta }_t=Q^{\pi }(s_t,a_t)+\alpha [R_t+\gamma \underset{a^{\prime }}{\max }{Q}^{\pi }(s^{\prime },a^{\prime })-Q^{\pi }(s_t,a_t)],$$

(10)

where $\alpha \in [0,1)$ is the learning rate.

It is important to note that the standard form of Q-learning is inefficient for large environments because it must consider every possible state–action pair to determine the optimal $Q^{*}(s,a)$, for example, by using a tabular approach. To address this inefficiency, a general approximator based on deep neural networks can be used instead of large tables.

2.1. Deep Reinforcement Learning

Deep reinforcement learning [16] is an extension of reinforcement learning (RL) that uses deep neural networks [15] as function approximators for value functions or policies. RL, at its core, involves training agents to make sequential decisions in environments, and DRL enhances this process by employing deep neural networks to handle high-dimensional data.

DRL has significant applications in robotics, especially in addressing challenges related to path planning. In this context, several DRL algorithms, also known as actor–critic model-free RL algorithms, such as the Deep Deterministic Policy Gradient [27], the Twin Delayed Deep Deterministic Policy Gradient [28], and the Soft Actor–Critic [29], often used with an experience replay buffer [30], have proven effective. These algorithms optimize both the policy and the value functions with the overarching goal of maximizing cumulative rewards, enabling them to converge on optimal results. To enhance the learning process, an experience replay buffer is commonly employed to store transitions during interactions with the environment. Figure 4 depicts the actor-critic architecture.

The integration of Deep Reinforcement Learning into robotics and path planning demonstrates its capability to handle complex and high-dimensional data, facilitating the effective learning of various tasks [22,23,24]. In the following sections, the details of the DDPG, TD3, and SAC algorithms are briefly described.

2.1.1. Deep Deterministic Policy Gradient

The Deep Deterministic Policy Gradient (DDPG) [27] is an off-policy reinforcement learning algorithm designed for continuous action spaces that uses a deterministic policy. It utilizes two neural networks: the actor and the critic. The actor network predicts the optimal action to take in a given state, while the critic network evaluates the quality of the actor’s chosen action.

The DDPG algorithm learns the optimal policy and Q-function simultaneously. Initially, it uses the Bellman equation and off-policy memory to learn the Q-function. Subsequently, the algorithm utilizes the acquired Q-function to learn the policy. The primary objective is to determine the optimal action value, denoted as $a^{*}$, for each state in a continuous control environment, similar to the Q-learning method (see Equation (7)).

The Bellman equation is utilized to estimate the optimal Q-function through a set of artificial neural networks, denoted as $Q_{\theta }(s,a)$, where $\theta$ represents the network’s parameter. The principle of the DDPG algorithm is to minimize the Mean Squared Bellman Error (MSBE) function defined for the set D of transitions (s, a, R, $s^{\prime }$, d). The equation for the MSBE function can be described as follows:

$$L(\theta ,D)={\mathbb{E}}_D[(Q_{\theta }(s,a)-{(R_t+\gamma (1-d)\underset{a^{\prime }}{\max }{Q}_{\theta }(s^{\prime },a^{\prime }))}^2],$$

(11)

where d indicates whether the terminal condition is satisfied or not, and the term $(R_t+\gamma (1-d)\underset{a^{\prime }}{\max }{Q}_{\theta }(s^{\prime },a^{\prime }))$ is called the target.

The problem is that both $Q_{\theta }$ and the target depend on the same parameter $\theta$. Due to this dependency, the minimization of the MSBE function becomes unstable. Therefore, another neural network, $Q_{{\theta }_{\mathrm{targ}}}$, referred to as the target network, is used to compute the error function. Since the action space is infinite, it is possible to use another neural network, ${\mu }_{\mathrm{targ}}$, known as the target policy network, to approximate the function $\max Q_{\theta }(s^{\prime },a^{\prime })$. The target networks are then updated using the parameters of the main networks using the Polyak averaging formula [46].

Based on the above information, Equation (11) will be rewritten as follows:

$$L(\theta ,D)={\mathbb{E}}_D[(Q_{\theta }(s,a)-{(R_t+\gamma (1-d)\underset{a^{\prime }}{\max }{Q}_{{\theta }_{\mathrm{targ}}}(s^{\prime },{\mu }_{\mathrm{targ}}(s^{\prime })))}^2].$$

(12)

2.1.2. Twin Delayed Deep Deterministic Policy Gradient

The Twin Delayed Deep Deterministic Policy Gradient (TD3) [28] is an off-policy reinforcement learning algorithm designed for a continuous action space. It is an extension of the DDPG algorithm that aims to overcome some of its limitations by implementing several significant improvements.

Firstly, the action is selected using so-called target policy smoothing, as follows:

$$a^{\prime }(s^{\prime })=\mathrm{clip}({\mu }_{\mathrm{targ}}(s^{\prime })+\mathrm{clip}(ϵ,-c,c),a_{\mathrm{Low}},a_{\mathrm{High}}),$$

(13)

where $ϵ$ is a random variable with a normal distribution and c is a hyperparameter.

Secondly, the algorithm uses clipped double Q-learning [47], which involves learning two Q-functions concurrently instead of one and selecting the smaller one to calculate the target term in the Bellman equation. This approach enables TD3 to more effectively capture the uncertainty in the environment and reduce the overestimation of value estimates that may occur in DDPG.

The TD3 algorithm includes two primary networks, denoted as $Q_{{\theta }_0}$ and $Q_{{\theta }_1}$, and two target networks, denoted as $Q_{{\theta }_{0,\mathrm{targ}}}$ and $Q_{{\theta }_{1,\mathrm{targ}}}$. The general form of the MSBE function for the TD3 algorithm is described in Equation (14).

$$L({\theta }_i,D)={\mathbb{E}}_D[(Q_{{\theta }_i}(s,a)-{(R_t+\gamma (1-d)\underset{i\in 0,1}{\min }{Q}_{{\theta }_{i,\mathrm{targ}}}(s^{\prime },{\mu }_{\mathrm{targ}}(s^{\prime })))}^2]$$

(14)

2.1.3. Soft Actor–Critic

The Soft Actor–Critic (SAC) [29] algorithm is an off-policy reinforcement learning algorithm designed for a continuous action space. It differs from DDPG and TD3 in its approach, although it incorporates some techniques from these algorithms, such as the clipped double Q-trick. The most significant aspect of the SAC algorithm is the entropy regularization feature. This algorithm maximizes the expected return and entropy during policy training, eliminating improper convergence.

The concept of entropy, which evaluates the randomness of a variable x using its density function P, can be defined as follows:

$$H(P)={\mathbb{E}}_P[-logP(x)].$$

(15)

Taking into account the effect of entropy, where the agent receives a bonus reward proportional to the entropy of the policy at the time step, the optimal policy can be defined as follows:

$${\pi }^{*}=\underset{\pi }{\mathrm{argmax}}{\mathbb{E}}_{\pi }[\sum _{t=0}^{\infty }{\gamma }^t(R(s_t,a_t,s_{t+1})+\alpha H(\pi (a_t\mid s_t)))],$$

(16)

where $\alpha >0$ is a hyperparameter called the trade-off coefficient.

In addition to the optimal policy, two Q-functions, $Q_{{\theta }_0}$ and $Q_{{\theta }_1}$, are optimized by the SAC algorithm. It uses a clipped double Q-learning trick and updates the target networks using Polyak averaging. The MSBE function for the SAC algorithm is described in Equation (17).

$$L({\theta }_i,D)={\mathbb{E}}_D[{(Q_{{\theta }_i}(s,a)-y(r,s^{\prime },d))}^2],$$

(17)

where the target term, denoted $y(r,s^{\prime },d)$, is expressed as

$$y(r,s^{\prime },d)=R_t+\gamma (1-d)(\underset{i\in 0,1}{\min }{Q}_{{\theta }_{i,\mathrm{targ}}}(s^{\prime },a^{\prime })-\alpha log\pi (a^{\prime }\mid s^{\prime })),$$

(18)

where $\pi$ is the current learned policy and $a^{\prime }\sim \pi$. To optimize the policy, the SAC algorithm uses a reparameterization trick to determine the action as follows

$$a^{\prime }=f_{\theta }(ϵ,s),$$

(19)

where $f_{\theta }$ is a deterministic function and $ϵ$ is random noise with normal distribution.

3. Deep-Reinforcement-Learning-Based Motion Planning

The combination of deep neural networks and reinforcement learning, known as deep reinforcement learning, represents a relatively unexplored area of research in robotics and motion planning. The complex approach of learning sequential actions directly from raw input observations, especially in the problem of reaching a static [20,21] or random target [22,23,24,25,26] within the robot configuration space, shows potential in this area. Among the most commonly used algorithms for the presented problem, as described in the previous part, are actor–critic model-free algorithms such as DDPG, TD3, and SAC, often used with experience replay.

In consideration of the research objectives, which prioritize the implementation of a modular approach to DRL-based motion planning solutions across a variety of robotic structures, each algorithm was compared in a specific experiment. The methods highlighted in the above-mentioned literature face challenges related to accuracy, independence from the type of robotic structure, the potential occurrence of self-collision, or collision avoidance in general. The research aims to address these challenges, eliminate shortcomings, and achieve greater versatility.

3.1. Comparison of Actor–Critic Algorithms

The comparison of DRL algorithms was focused mainly on the specific robotic arm Universal Robots UR3, considering that it was part of previous research [20]. The solved problem was divided into two parts (see Figure 5), with both parts focusing on reaching the target in a pre-defined configuration space. The first part, defined by the environment ${\mathcal{E}}_1$, was focused on reaching the target within the configuration space without any external collision. The second part, defined by the environment ${\mathcal{E}}_2$, was focused on the same problem, but with the presence of a statically positioned external collision object.

In both cases, the task was to move the robotic arm from the initial position to the desired final position within the pre-defined configuration space without collisions, including self-collisions. An effective approach to avoiding collisions was to approximate the individual parts of the robot’s structure, as well as potential external collision objects, using so-called bounding boxes [48,49]. The static objects that form the base of the robot and potential external objects were approximated with Axis-Aligned Bounding Boxes (AABBs), while dynamic objects such as joints were approximated with Oriented Bounding Boxes (OBBs). The movement was executed by generating the trajectory using linear segments with parabolic blends (LSPB). The Levenberg–Marquardt method [50] was used to define the motion using a numerical inverse kinematics approach. The learning process was performed through the integration of OpenAI Gym [51] and PyBullet [52] physics-based simulator.

The observation space consists of the position of the end effector in the current iteration and the desired end effector position. The desired position, known as the target, was randomly generated in each episode to enhance the learning model. The action space consists of the end effector movement command in three-dimensional Euclidean space. The reward function, as a critical part of the learning process, was defined for the ${\mathcal{E}}_1$ environment as follows:

$$R_i=-(\parallel {\mathbf{p}}_d-{\mathbf{p}}_i\parallel ),$$

(20)

and for the ${\mathcal{E}}_2$ environment as

$$R_i=-(\parallel {\mathbf{p}}_d-{\mathbf{p}}_i\parallel +\frac{{\gamma }_{\mathcal{O}}}{1+\parallel {\mathbf{p}}_{\mathcal{O}}-{\mathbf{p}}_i\parallel }).$$

(21)

In Equation (20), the ${\mathcal{E}}_1$ environment was determined by the simple Euclidean distance between the current position of the end effector, denoted as ${\mathbf{p}}_i$, and the desired position of the end effector, denoted as ${\mathbf{p}}_d$. Equation (21) for the ${\mathcal{E}}_2$ environment contains an extension of the aforementioned Equation (20) with a penalty term related to the collision object. This penalty is defined based on the position of the collision object, denoted as ${\mathbf{p}}_{\mathcal{O}}$; the current position of the end-effector, denoted as ${\mathbf{p}}_i$; and by the threshold parameter ${\gamma }_{\mathcal{O}}$ (empirically set to the value 0.01). This threshold parameter affects the significance of the penalty within the learning process. It is important to note that while the reward function for the ${\mathcal{E}}_1$ environment adheres to conventional methodologies for the given task, the reward calculation function for the ${\mathcal{E}}_2$ environment introduces a novel approach.

The learning process is interrupted when the robot encounters an external object or self-collision. Conditions for process interruption include the singularity check, along with evaluating the feasibility of the inverse kinematics solution. Successful completion of the learning episode is achieved when the current end-effector position of the robotic arm aligns closely with the randomly generated target, defined by the desired end-effector position.

The training process, with the parametrization shown in Appendix A, using the DDPG, SAC, and TD3 algorithms, both with and without Hindsight Experience Replay (HER), is illustrated in Figure 6 for environment ${\mathcal{E}}_1$ and Figure 7 for environment ${\mathcal{E}}_2$. The comprehensive results of the evaluation process are described in

Table 1. The table presents experimental results comparing various DRL algorithms within the ${\mathcal{E}}_1$ environment. The required minimum success rate to meet the specified criteria was set at 0.98.

Algorithm	Success Rate	Percentage of Successful Targets	Mean Reward per Episode	Mean Episode Length
DDPG	0.98–1.0	95.86%	−0.388	5.299
DDPG + HER	0.98–1.0	89.83%	−0.387	5.431
SAC	0.98–1.0	96.52%	−0.408	5.682
SAC + HER	0.98–1.0	94.85%	−0.407	5.701
TD3	0.98–1.0	96.61%	−0.386	5.298
TD3 + HER	0.98–1.0	78.73%	−0.395	5.720

for environment ${\mathcal{E}}_1$ and

Table 2. The experimental results comparing various DRL algorithms within the ${\mathcal{E}}_2$ environment. The required minimum success rate to meet the specified criteria was set at 0.8.

Algorithm	Success Rate	Percentage of Successful Targets	Mean Reward per Episode	Mean Episode Length
DDPG	0.8–0.97	95.01%	−0.710	5.866
DDPG + HER	0.8–0.96	86.72%	−0.711	5.944
SAC	0.8–0.85	5.048%	−0.709	5.684
SAC + HER	0.8–0.88	19.07%	−0.713	5.844
TD3	0.8–0.96	89.59%	−0.711	5.901
TD3 + HER	0.8–0.94	81.99%	−0.718	6.146

for environment ${\mathcal{E}}_2$. The results underscore the importance of selecting appropriate algorithms based on the characteristics of the environment. Specifically, the TD3 algorithm proved to be effective in environments without external collisions, while the DDPG algorithm was well suited for scenarios featuring statically positioned external collision objects. This strategic approach improved the adaptability of the chosen methods for a diverse range of robotic arms in future applications.

3.2. Application of Selected Algorithms to a Wide Range of Robotic Structures

The application of suitable algorithms for a specific case, obtained from the comprehensive experiments described in the previous section, was used for a wide range of robotic arms in both the ${\mathcal{E}}_1$ environment, as described in Figure 8, and the ${\mathcal{E}}_2$ environment, as described in Figure 9. Specifically, the TD3 algorithm was employed in environments without external collisions, while the DDPG algorithm was utilized for scenarios featuring statically positioned external collision objects.

The results of the training process, which was evaluated for specific robot arms, namely ABB IRB 120 with and without an extended linear axis, Epson LS3-B401S, dual-arm collaborative robot ABB IRB14000, and Universal Robots UR3, are presented in

Table 3. The table presents experimental results comparing a wide range of robotic structures using the TD3 algorithm within the ${\mathcal{E}}_1$ environment. The minimum success rate required to meet the specifications was set at 0.98.

Robot Type	Success Rate	Percentage of Successful Targets	Mean Reward per Episode	Mean Episode Length
Universal Robots UR3	0.98–1.0	96.61%	−0.386	5.298
ABB IRB 120	0.98–1.0	84.32%	−0.602	6.073
ABB IRB 120 Ext.	0.98–1.0	87.65%	−0.637	6.647
ABB IRB 14000 (L)	0.98–1.0	94.51%	−0.256	4.593
ABB IRB 14000 (R)	0.98–1.0	91.69%	−0.256	4.608
Epson LS3-B401S	0.98–1.0	94.39%	−0.064	2.474

for environment ${\mathcal{E}}_1$ and

Table 4. The table presents experimental results comparing a wide range of robotic structures using the DDPG algorithm within the ${\mathcal{E}}_2$ environment. The minimum success rate required to meet the specifications was set at 0.8.

Robot Type	Success Rate	Percentage of Successful Targets	Mean Reward per Episode	Mean Episode Length
Universal Robots UR3	0.8–0.97	95.01%	−0.710	5.866
ABB IRB 120	0.8–0.98	95.26%	−0.874	6.609
ABB IRB 120 Ext.	0.8–0.98	96.83%	−1.031	8.041
ABB IRB 14,000 (L)	0.8–0.97	94.85%	−0.503	5.356
ABB IRB 14,000 (R)	0.8–0.96	92.14%	−0.499	5.227
Epson LS3-B401S	0.8–0.99	98.19%	−0.396	5.040

for environment ${\mathcal{E}}_2$.

To underscore the universality of the proposed method, an additional prediction test was conducted for a portfolio of robotic arms. The results are presented in

Table 5. The table presents experimental results comparing a wide range of robotic structures using the TD3 algorithm within the ${\mathcal{E}}_1$ environment for one hundred randomly generated targets. The position error is given in meters.

Robot Type	Success Rate	Mean Reward per Episode	Mean Episode Length	Mean Absolute Position Error
Universal Robots UR3	1.0	−0.365	4.94	0.0024
ABB IRB 120	1.0	−0.501	5.26	0.0055
ABB IRB 120 Ext.	1.0	−0.502	5.16	0.0061
ABB IRB 14,000 (L)	1.0	−0.233	4.21	0.0024
ABB IRB 14,000 (R)	1.0	−0.228	4.13	0.0028
Epson LS3-B401S	1.0	−0.059	2.32	0.0030

for environment ${\mathcal{E}}_1$ and

Table 6. The table presents experimental results comparing a wide range of robotic structures using the DDPG algorithm within the ${\mathcal{E}}_2$ environment for one hundred randomly generated targets. The position error is given in meters.

Robot Type	Success Rate	Mean Reward per Episode	Mean Episode Length	Mean Absolute Position Error
Universal Robots UR3	1.0	−0.711	6.04	0.0058
ABB IRB 120	1.0	−0.859	6.45	0.0023
ABB IRB 120 Ext.	1.0	−0.945	6.98	0.0065
ABB IRB 14,000 (L)	1.0	−0.471	5.15	0.0040
ABB IRB 14,000 (R)	1.0	−0.470	5.12	0.0032
Epson LS3-B401S	1.0	−0.374	4.79	0.0027

for environment ${\mathcal{E}}_2$. The results clearly demonstrate the high efficiency and universality of the algorithms in each environment across a diverse range of robotic structures.

4. Conclusions

In this paper, the development of a method for planning motion across a diverse range of robotic structures based on innovative algorithms was presented. The combination of deep neural networks and reinforcement learning, known as deep reinforcement learning, was represented as a relatively unexplored area of research in robotics and motion planning, especially in the problem of reaching a static or random target within the robot configuration space. The methods highlighted in the literature review faced challenges related to accuracy, independence from the type of robotic structure, potential occurrences of self-collision, or collision avoidance in general. The proposed method effectively addresses the aforementioned shortcomings and showcases the high efficiency and universality of the algorithms across a diverse range of robotic structures. To substantiate this claim, a comprehensive comparison of actor–critic model-free algorithms was conducted, including Deep Deterministic Policy Gradient, Twin Delayed Deep Deterministic Policy Gradient, and Soft Actor–Critic, both with and without Hindsight Experience Replay. The problem was divided into two parts, with both parts focusing on reaching the target in a pre-defined configuration space. The first part, defined by the environment ${\mathcal{E}}_1$, focused on reaching the target within the configuration space without any external collision. The second part, defined by the environment ${\mathcal{E}}_2$, focused on the same problem, but with the presence of a statically positioned external collision object. The movement was executed by generating the trajectory using linear segments with parabolic blends. The results underscored the importance of selecting appropriate algorithms based on the characteristics of the environment. Specifically, the TD3 algorithm proved to be effective in environments without external collisions, while the DDPG algorithm was well suited for scenarios featuring statically positioned external collision objects. This strategic approach improved the adaptability of the methods chosen for a diverse range of robotic arms, as applied to the portfolio of presented robotic arms.

It is important to note that in practical scenarios involving real robots, the execution of movements may not always be realized through the LSPB method as demonstrated in the simulation experiments. In practical scenarios, for an individualized approach to robot control, it is necessary to use special methods that enable the robot to read the trajectory profile in real-time or near real-time, instead of the simple target position. These special methods can be, for example, EGM (Externally Guided Motion) for the ABB robot and UR-RTDE (Real-Time Data Exchange) for the UR robot. Otherwise, it is necessary to rely on the trajectory generation methods defined by the manufacturer of the respective robot.

Outlook for Future Research

Future investigations into Deep-Reinforcement-Learning-Based motion planning have the potential to address more complex challenges, including the avoidance of collisions with multiple static or dynamic objects, multi-robot motion planning, human–machine collaboration, planning with energy consumption or path efficiency optimization, intelligent grasping of diverse objects, and in general multi-objective environments for both simulation and real-world applications. One avenue for tackling these challenges involves enhancing DRL algorithms through the integration of multi-agent structures, such as the Multi-Agent Deep Deterministic Policy Gradient [53] or Multi-Agent TD3 [54]. These frameworks offer the potential to optimize collaborative decision-making among multiple agents, thereby improving the efficiency and robustness of motion planning tasks. Another potential research objective is to use a more sophisticated physics-based simulation tool, such as Unity3D or NVIDIA Isaac Sim, instead of the PyBullet tool that was utilized.