Integrated Intelligent Control of Redundant Degrees-of-Freedom Manipulators via the Fusion of Deep Reinforcement Learning and Forward Kinematics Models

Abstract

Redundant degree-of-freedom (DOF) manipulators offer increased flexibility and are better suited for obstacle avoidance, yet precise control of these systems remains a significant challenge. This paper addresses the issues of slow training convergence and suboptimal stability that plague current deep reinforcement learning (DRL)-based control strategies for redundant DOF manipulators. We propose a novel DRL-based intelligent control strategy, FK-DRL, which integrates the manipulator’s forward kinematics (FK) model into the control framework. Initially, we conceptualize the control task as a Markov decision process (MDP) and construct the FK model for the manipulator. Subsequently, we expound on the integration principles and training procedures for amalgamating the FK model with existing DRL algorithms. Our experimental analysis, applied to 7-DOF and 4-DOF manipulators in simulated and real-world environments, evaluates the FK-DRL strategy’s performance. The results indicate that compared to classical DRL algorithms, the FK-DDPG, FK-TD3, and FK-SAC algorithms improved the success rates of intelligent control tasks for the 7-DOF manipulator by 21%, 87%, and 64%, respectively, and the training convergence speeds increased by 21%, 18%, and 68%, respectively. These outcomes validate the proposed algorithm’s effectiveness and advantages in redundant manipulator control using DRL and FK models.

1. Introduction

Redundant degree-of-freedom (DOF) manipulators have additional degrees of freedom, endowing the system with enhanced obstacle avoidance capabilities, superior singularity avoidance, and higher fault tolerance [1]. However, while solving the forward kinematics (FK) of such manipulators is relatively straightforward, their inverse kinematics (IK) present innumerable possible solutions. Therefore, solving the IK of redundant DOF manipulators poses significant complexity and challenges [2]. Numerical methods are the prevailing approach for solving the IK of manipulators, but they typically yield only locally optimal solutions [3]. Furthermore, these methods are hampered by slow computational speeds and high costs [4], which can adversely affect the effectiveness of subsequent motion control for the manipulator.

The analytical method [5] establishes explicit expressions for the joint angles of manipulators, thus offering high computational efficiency and precision. Crane [6] proposed a technique to fix one joint of a 7-DOF manipulator, obtaining the analytical solution for the remaining 7-DOF subchain, which facilitates the derivation of the kinematic inverse solution for the manipulator. Xavier et al. [7] introduced the application of Grobner basis theory to solve the IK of 7-DOF manipulators and validated the accuracy of this method. However, the analytical approach requires the IK equations to be rederived based on the specific structural configuration of each manipulator [8], which limits its generalizability.

With the development of artificial intelligence, scholars have increasingly utilized genetic algorithms [9] and neural networks [10] to solve the IK of manipulators with redundant DOF. However, genetic algorithms are often criticized for their slow iterative solution process, while neural networks require extensive training data. Additionally, both methods are prone to converging on local optima, which can compromise the precision of the solutions.

Reinforcement learning (RL) involves intelligent agents learning optimal strategies through interactions with the environment to maximize cumulative rewards [11]. This approach offers several advantages for intelligent control of manipulators, including the elimination of the need to establish kinematic models, robust handling of complex problems, and self-learning capabilities. Consequently, RL has emerged as a promising method for intelligent manipulator control. Perrusquia et al. [12] proposed a multi-agent RL method to solve the IK problem of redundant DOF manipulators. Lee et al. [13] developed a 7-DOF manipulator posture control algorithm based on RL and neural networks. Ramirez et al. [14] introduced an RL method combined with human demonstrations and applied it to the control of redundant DOF manipulators. However, RL algorithms exhibit inefficiencies in exploring high-dimensional spaces and are prone to getting trapped in local optima in complex tasks.

To improve the performance of RL algorithms, researchers have begun integrating deep learning with reinforcement learning, leading to the development of deep reinforcement learning (DRL) [15] methods. Li et al. [16] proposed a general framework that integrates DRL into the motion planning of redundant DOF manipulators, optimizing path planning and deriving the energy-optimal solution to IK. Carlos et al. [17] introduced a DRL framework for controlling manipulators in simulated environments. Zheng et al. [18], focusing on a 6-DOF manipulator, proposed a trajectory-planning method based on DRL, which improves convergence by introducing dynamic action selection strategies and combinatorial reward functions.

Although DRL has partially addressed the shortcomings of RL, it still faces issues such as slow training convergence, poor stability, and limited scalability when dealing with intelligent control tasks for high DOF or redundant DOF manipulators. To address these challenges, this paper proposes an efficient convergence FK-DRL algorithm for the intelligent control of redundant DOF and 4-DOF manipulators. The algorithm integrates the easily established FK model of the manipulator into the DRL algorithm’s training framework. The FK model effectively guides the agent in exploring optimal control strategies, thereby significantly improving the training convergence speed and stability of the algorithm. Furthermore, the algorithm only requires modifications to the FK model of the manipulator to be applicable to the intelligent control of other manipulators, demonstrating excellent scalability.

In the remainder of this article, the proposed FK-DRL algorithm is introduced in Section 2. The comparison simulations are conducted to illustrate the performance of the proposed algorithm in Section 3. Section 4 further validates the performance of the proposed algorithm through actual experiments with a 4-DOF manipulator. Furthermore, the conclusions and future work are summarized in Section 5.

2. Design of the FK-DRL Control Algorithm

This section introduces the design of the FK-DRL algorithm. We provide a detailed explanation of the process of modeling the manipulator’s control task using the Markov decision process (MDP) model [19] and the construction of the manipulator’s FK model. Additionally, we elucidate how to integrate the FK model into the DRL algorithm to form the FK-DRL algorithm.

2.1. Control Problem and MDP Modelling

The task of manipulator control can be characterized as follows: A target block is randomly placed within the workspace of the manipulator. The agent rotates the various joints of the manipulator to ensure that the end-effector [20] is positioned at the desired location and orientation.

The control task of a manipulator can be modeled as an MDP, with the objective of identifying an optimal policy that enables the manipulator’s end-effector to reach the target pose accurately with the minimum number of steps.

The control task of a manipulator can be modelled as an MDP, with the objective of identifying an optimal policy that enables the end-effector to reach the target pose accurately with the least number of steps.

Define the state space $S$:

$$\forall s_t\in S,s_t=[j,p_e,p_g]$$

(1)

where $j$ denotes the $n$ joint angles $({\theta }_1,{\theta }_2,\dots {\theta }_n)$ of the manipulator, $p_e$ is the current position $(x_e,y_e,z_e,{\alpha }_e,{\mathsf{\beta }}_e,{\gamma }_e)$ of the end-effector, and $p_g$ is the target position $(x_g,y_g,z_g,{\alpha }_g,{\mathsf{\beta }}_g,{\gamma }_g)$.

Define the action space $A$:

$$\forall a_t\in A,a_t=[{\theta }_1,{\theta }_2,\dots {\theta }_n]$$

(2)

where action $a_t$ is the target angle of the $n$ joints of the manipulator.

Define the reward function $r_t$:

$$\begin{array}{l}{r}_{\mathrm{a}}=-d\\ =-\sqrt{\begin{array}{l}{\left(x_e-x_g\right)}^2+{\left(y_e-y_g\right)}^2+{\left(z_e-z_g\right)}^2+\\ {({\alpha }_e-{\alpha }_g)}^2+{\left({\mathsf{\beta }}_e-{\mathsf{\beta }}_g\right)}^2+{\left({\gamma }_e-{\gamma }_g\right)}^2\end{array}}\\ r_b=\left\{\begin{array}{c}1d^{\prime }>d\hspace{0.33em}\hspace{0.33em}\\ 0\hspace{0.33em}{d}^{\prime }<d\hspace{0.33em}\end{array}\right.\\ r_c=\left\{\begin{array}{c}0d>\hspace{0.33em}\epsilon \\ \eta \hspace{0.33em}d<\epsilon \end{array}\right.\\ r_t=r_a+r_b+r_c\end{array}$$

(3)

where $r_a$ defines a reward value that is negatively correlated with the distance between $p_e$ and $p_g$, $r_b$ defines a directional reward function, $d$ denotes the distance between $p_e$ and $p_g$ at the current moment, $d’$ denotes the distance between $p_e$ and $p_g$ at the previous moment, and when $p_e$ is closer to $p_g$ than at the previous moment it receives a reward value of 1; otherwise, it receives a reward value of 0. $r_c$ denotes that $p_e$ and $p_g$ receive a reward value of 0 when the distance between them is greater than $\epsilon$ of the reward value; otherwise, the task is considered completed and a reward value of $\mathsf{\eta }$ is obtained. The total reward value $r_t$ is the sum of $r_a$, $r_b$, and $r_c$. The episode ends when the time step equals $n$ unless the task is completed within $n$ time steps.

The sum of episode reward values $R_t$ is defined as:

$$R_t={\displaystyle \sum _{t=1}^n{r}_t}$$

(4)

In manipulator control tasks, the target pose is randomly initialized within the manipulator’s workspace at the beginning of each episode. The agent identifies the optimal strategy based on the current environmental state, which yields the joint angles necessary for the manipulator’s end-effector to reach the target pose as swiftly as possible.

2.2. Modelling of the FK of the Manipulator

The FK model of the n DOF manipulator is established according to the D-H (Denavit–Hartenberg) [21] parametric method, and the relative positions between joints $\{i-1\}$ and $\left\{i\right\}$ in its joint coordinate system are shown in Figure 1.

According to the D-H parameters, the transformation matrix ${}_{i }{}^{i-1}T$ between joints $\{i-1\}$ and $\left\{i\right\}$ is obtained as:

$${}_{i }{}^{i-1}T=\left[\begin{array}{cccc}\cos {\theta }_i& -\sin {\theta }_i& 0& a_{i-1}\\ \sin {\theta }_i\cos {\alpha }_{i-1}& \cos {\theta }_i\cos {\alpha }_{i-1}& -\sin {\alpha }_{i-1}& -\sin {\alpha }_{i-1}{d}_i\\ \sin {\theta }_i\sin {\alpha }_{i-1}& \cos {\theta }_i\sin {\alpha }_{i-1}& \cos {\alpha }_{i-1}& \cos {\alpha }_{i-1}{d}_i\\ 0& 0& 0& 1\end{array}\right]$$

(5)

where ${\alpha }_{i-1}$, $a_{i-1}$, $d_i$, and ${\theta }_i$ are the D-H parameters of the manipulator.

Its base joint to end-effector joint coordinate system transformation matrix ${}_n{}^{0}T$ is:

$${}_n{}^{0}T{=}_1^{0}T{\cdot }_2^{1}T\cdot \dots {\cdot }_n^{n-1}T=\left[\begin{array}{cccc}{n}_x& o_x& a_x& p_x\\ n_y& o_y& a_y& p_y\\ n_z& o_z& a_z& p_z\\ 0& 0& 0& 1\end{array}\right]$$

(6)

where ${[\begin{array}{ccc}{n}_i& o_i& a_i\end{array}]}_{3\times 3}(i=x,y,z)$ denotes the rotation matrix, and each column is the attitude vector coordinates of the manipulator end-effector; the fourth column ${[\begin{array}{ccc}{p}_x& p_y& p_z\end{array}]}^T$ denotes the position vector of the manipulator end-effector.

The manipulator end-effector position $p$ is:

$$p=(p_x,p_y,p_z,\alpha ,\beta ,\gamma )$$

(7)

where $\alpha ,\beta ,\gamma$ denote the heading, pitch, and roll angles of the manipulator end-effector in Euler angles, respectively.

Denoting the FK model of the n DOF manipulator as $M$, and the mapping relation from the joint ${\theta }_1,{\theta }_2,\dots {\theta }_n$ mapping to the manipulator end-effector position $p$ as $f$, then:

$$M:f({\theta }_1,{\theta }_2,\dots {\theta }_n){=}_n^{0}T$$

(8)

2.3. FK-DRL Algorithm

DRL algorithms can be classified into value-based and policy-based algorithms. Value-based algorithms select the action with the highest value as the optimal policy each time [22]:

$${\pi }^{*}(a|s)=\underset{a}{\arg \max }{q}^{*}(s,a)$$

(9)

where ${\pi }^{*}(a|s)$ represents the optimal policy, and $q^{*}(s,a)$ represents the optimal action value function.

Policy-based RL algorithms, on the other hand, directly parameterize the policy, denoted as $\pi (a|s,\theta )$, and represent it as a function whose objective is to find the optimal parameter ${\theta }^{*}$ that maximizes the expectation of the cumulative reward [23]. Policy-based RL algorithms need to find the gradient ${\nabla }_{{\theta }^{\mu }}{L}_A({\theta }^{\mu })$ of the policy’s objective function $L({\theta }^{\mu })$, defined as

$${\nabla }_{{\theta }^{\mu }}{L}_A({\theta }^{\mu })={\displaystyle \frac{1}{N}}{\displaystyle \sum _i{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{a=\pi (s_i)}}{\nabla }_{{\theta }^{\mu }}\pi (s|{\theta }^{\mu })$$

(10)

Current mainstream actor–critic algorithms such as deep deterministic policy gradient (DDPG) [24], twin delayed deep deterministic policy gradient (TD3) [25], and soft actor–critic (SAC) [26] can estimate value functions with relative precision and guide policy learning. However, these methods necessitate persistent interaction between the agent and the environment to enrich the data within the experience replay buffer [27]. This process is both time-consuming and resource-intensive, leading to slow convergence rates and suboptimal control precision during training.

To address the aforementioned issues, this paper proposes the FK-DRL algorithm for controlling manipulators with redundant degrees of freedom. The specific workflow is as follows:

Step 1: Define the FK model M, the highest number of training episodes E, the highest number of interactions with the environment per episode T1, the number of dynamic planning per episode P, and the maximum number of interactions with the FK model M in one dynamic planning episode T2.

Step 2: In each training episode, the intelligent agent begins to update the Actor and Critic networks after each interaction with the environment. After the T1 interactions, an episode of dynamic planning is initiated:

(1)

Take one unfinished task from the experience pool $transition (s_n,a_n,r_n,s_{n+1})$.

(2)

For the sake of differentiation, the state $s_n$ is denoted as $s_k^M$ and used as the initial state in an episode of planning. At the same time, OU (Ornstein Uhlenbeck) noise is added to the action $a_n$ [28] and is denoted as $a_k^M=[{\theta }_1^k,{\theta }_2^k,\dots {\theta }_n^k]$.

(3)

Inputting the action $a_k^M$ into the model M, the pose of the end-effector $p_e^k=(x_e^k,y_e^k,z_e^k,{\alpha }_e^k,{\beta }_e^k,{\gamma }_e^k)$ is obtained. The next state $s_{k+1}^M=[j^{k+1},p_e^{k+1},p_g^{k+1}]$ is obtained from the current state $s_k^M=[j^k,p_e^k,p_g^k]$ after the input action $a_k^M$. The reward value $r_k^M$ is calculated to get the $transitio{n}_k^M (s_k^M,a_k^M,r_k^M,s_{k+1}^M)$, which is stored in the experience pool.

(4)

Input $s_{k+1}^M$ into the Actor network, which then outputs an action. Add OU noise to this action to obtain action $a_{k+1}^M$ and repeat step (3) to obtain $transitio{n}_{k+1}^M (s_{k+1}^M,a_{k+1}^M,r_{k+1}^M,s_{k+2}^M)$, which is then stored in the experience pool. Extract experiences from the pool to update the Actor and Critic networks.

(5)

Continue to interact with model M until the number of interactions reaches T2 and the Actor and Critic networks, indicating the end of this dynamic planning episode;

(6)

Repeat steps (1)–(5) P times.

Step 3: Repeat Step 2 until the maximum number of episodes E is reached to obtain the optimal strategy $\pi (a|s,\theta )$.

The pseudocode for the FK-DRL algorithm is shown in Algorithm 1.

Algorithm 1. FK-DRL

Given that the network update processes of algorithms such as TD3, SAC, and DDPG are essentially the same, this paper uses FK-DDPG as an example to demonstrate the overall framework of the FK-DRL algorithm, as shown in Figure 2.

3. Simulation

To evaluate the efficacy of the proposed algorithms, control tasks involving a 7-DOF manipulator and a 4-DOF manipulator were conducted in a simulation environment. The agents were trained using both the proposed FK-DDPG, FK-TD3, and FK-SAC algorithms, as well as the conventional DDPG, TD3, and SAC algorithms. Subsequently, the performance of the manipulators was evaluated post-training.

3.1. Simulation Environment

In this study, we utilized CoppeliaSim (formerly known as V-REP) [29] as the simulation software. Neural network training was conducted using PyTorch 1.9.0 on a single GPU (NVIDIA GeForce RTX 3090) with CUDA version 11.2.

The manipulator model, designed in SolidWorks, was imported into CoppeliaSim. Blocks with randomly initialized positions were added to serve as the target poses for the manipulator’s end-effector in each simulation episode. Figure 3 shows the 3D simulation model of the 7-DOF manipulator and its workspace for the end-effector. The workspace of the end-effector can be represented as a rectangular region defined as the coordinate range from (0.3, −0.1) to (0.5, 0.1). Within CoppeliaSim, we can acquire data on the joint angles, the pose of the end-effector, and the coordinates of the blocks.

As shown in Figure 3, Joint 1 rotates around the Z-axis of the world coordinate system, while Joints 2 through 7 are all rotational joints. The end-effector of the manipulator is considered a fixed Joint 8. The D-H parameters of this manipulator are presented in

Table 1. Parameters of 7-DOF manipulator D-H.

$\mathit{i}$	${\mathit{\alpha }}_{\mathit{i}\mathbf{-}1}(\mathbf{°})$	${\mathit{a}}_{\mathit{i}\mathbf{-}1}(\mathit{m})$	${\mathit{d}}_{\mathit{i}}(\mathit{m})$	${\mathit{\theta }}_{\mathit{i}}(°)$
1	0	0	0.2039	${\theta }_1$
2	90	0	0	${\theta }_2$
3	−90	0	0.2912	${\theta }_3$
4	90	0	0	${\theta }_4$
5	−90	0	0.3236	${\theta }_5$
6	90	0	0	${\theta }_6$
7	−90	0	0.0606	${\theta }_7$
8	0	0	0.1006	$0$

.

The FK model M of the manipulator is:

$$M:f({\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5,{\theta }_6,{\theta }_7){=}_8^{0}T$$

(11)

Figure 4 illustrates the 3D simulation environment of the 4-DOF manipulator and its end-effector workspace. The workspace of the end-effector can be represented as a rectangular region defined as the coordinate range from (0.2, −0.15) to (0.5, 0.15). The D-H parameters of the 4-DOF manipulator are shown in

Table 2. Parameters of 4-DOF manipulator D-H.

$\mathit{i}$	${\mathit{\alpha }}_{\mathit{i}\mathbf{-}1}(\mathbf{°})$	${\mathit{a}}_{\mathit{i}\mathbf{-}1}(\mathit{m})$	${\mathit{d}}_{\mathit{i}}(\mathit{m})$	${\mathit{\theta }}_{\mathit{i}}(\mathbf{°})$
1	0	0	0.0445	${\theta }_1$
2	90	0.0025	0	${\theta }_2$
3	0	0.081	0	${\theta }_3$
4	0	0.0775	0	${\theta }_4$
5	0	0.126	0	$0$

.

The FK model M of this manipulator is:

$$M:f({\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4){=}_5^{0}T$$

(12)

The network architectures of the DDPG, TD3, FK-DDPG, and FK-TD3 algorithms are identical. The network model for a 7-DOF manipulator control task is illustrated in Figure 5.

The SAC and FK-SAC algorithms employ stochastic policy approaches. Their Actor networks consist of four fully connected neural layers, each with 256 neurons. Following each fully connected layer, a ReLU activation function is applied. The output of the Actor network includes one layer that produces the mean $\mu$ of a normal distribution after passing through the first two fully connected layers. Another output layer, also following the first two layers, generates the exponent of the standard deviation $\sigma$ of the normal distribution. The Critic network structure is consistent with that of DDPG.

The network hyperparameters [30] of each algorithm are shown in

Table 3. Training hyperparameter setting.

Parameter	Value
Learning rate of Actor network ${\alpha }_A$	1 × 10−4
Learning rate of Critic network ${\alpha }_C$	5 × 10−4
Discount rate $\gamma$	0.9
Size of the replay buffer capacity	10,000
Target network soft update factor $\tau$	0.005
Batch size	32
Episode	120,000
Step of interactions with the environment per episode ${\mathrm{T}}_1$	16
Step of interactions with M in dynamic programming ${\mathrm{T}}_2$	16
Step of dynamic programming $P$	5

.

3.2. Analysis of Simulation Results of 7-DOF Manipulator

This study presents an evaluation of the convergence curves for FK-DDPG, FK-TD3, and FK-SAC algorithms, along with the classical DDPG, TD3, and SAC algorithms, by averaging the results of 10 random training runs. Figure 6, Figure 7 and Figure 8 depict the training outcomes. In each figure, the central line represents the average reward per episode, calculated over 10 random training sessions, while the shaded area indicates the standard deviation. The horizontal axis denotes the number of episodes during training, while the vertical coordinate represents the cumulative reward value for the current round divided by the current number of rounds.

It is evident that integrating the FK model of a manipulator with any of the DDPG, TD3, or SAC algorithms resulted in superior training performance compared to employing the DDPG, TD3, or SAC algorithms independently. Notably, the algorithms that combine FK—FK-DDPG, FK-TD3, and FK-SAC—demonstrated smaller shaded areas in their training curves, indicating that the incorporation of FK with DDPG, TD3, or SAC led to a more stable training process.

The FK-DDPG algorithm converged after 80,000 episodes, marking a 27% reduction in the number of episodes required for convergence compared to the standard DDPG algorithm, which converged after 110,000 episodes. Additionally, the average reward value at the completion of training for FK-DDPG exceeded that of DDPG by approximately 5 units.

The FK-TD3 algorithm achieved convergence at 85,000 episodes, whereas the standard TD3 algorithm failed to converge within the same number of episodes. Similarly, the FK-SAC algorithm converged after 90,000 episodes, demonstrating a 25% reduction in episodes compared to the SAC algorithm, which only began to show a convergence trend after 120,000 episodes. The average reward for FK-SAC upon training completion was about 8 units higher than that of SAC. The slower convergence rate of the SAC algorithm can be attributed to its stochastic policy approach based on the maximum entropy principle. During training, the SAC algorithm maintains a policy distribution and seeks to maximize the entropy of the policy to enhance exploration and improve the extent of exploration across different actions. This increased exploratory behavior results in a slower convergence rate during training.

The FK-DDPG, FK-TD3, and FK-SAC algorithms all exhibited shorter convergence times and higher control precision. This is because these algorithms incorporate FK models during dynamic planning, enabling the agent to leverage exploration and learning further. Such integration facilitates efficient data augmentation, generating a greater variety of grasping scenarios and samples to enrich the training data. Consequently, agents can rapidly acquire an understanding of actions, which better guides their exploration and decision-making processes, allowing for precise control of the manipulator to reach target poses in fewer episodes.

Training time is a more accurate measure of the time expended during the process of training agents. In this paper, it specifically refers to the time elapsed from the start of training until convergence is achieved or the maximum number of episodes is reached.

As illustrated in Figure 9, the training time required for the FK-DDPG, FK-TD3, and FK-SAC algorithms was significantly less than that of their respective classical DRL counterparts. For instance, the FK-DDPG algorithm reduced training time by 21%, while the FK-TD3 and FK-SAC algorithms achieved reductions of 18% and 68%, respectively. The incorporation of the FK model not only decreased the number of episodes required for convergence but also substantially shortened the total convergence time.

To further assess the effectiveness of the algorithms, we selected average reward (total reward obtained during testing divided by the number of tests) and success rate (number of tasks completed divided by the number of tests) as metrics. The average rewards and success rates of the FK-DDPG, FK-TD3, and FK-SAC algorithms compared with the DDPG, TD3, and SAC algorithms over 100 simulation trials are depicted in Figure 10 and Figure 11.

Figure 10 demonstrates that during the testing phase, the average rewards for the FK-DDPG, FK-TD3, and FK-SAC algorithms were consistently higher, and their reward variances were substantially lower than those of the corresponding classical DRL algorithms. This is in line with observations made during the training phase. For example, the FK-DDPG algorithm achieved an average reward of 9.96, in contrast to the DDPG algorithm’s average reward of 4.45.

As illustrated in Figure 11, the success rates of the FK-DDPG, FK-TD3, and FK-SAC algorithms were significantly higher than those of the classical DRL algorithms. For instance, the FK-TD3 algorithm achieved a testing success rate of 99%, in stark contrast to the mere 12% success rate of the classical TD3 algorithm. This discrepancy may be ascribed to the reliance of classical DRL algorithms on a pure trial-and-error methodology for acquiring control strategies for manipulators, without the benefit of pre-existing kinematic model knowledge to inform the learning process. This absence of guidance makes the learning process more challenging and unstable, consequently reducing the success rate of the task.

3.3. Analysis of Simulation Results of 4-DOF Manipulator

Based on the existing experimental conditions, in Section 4, we utilize the trained FK-DRL algorithm and the classic DRL algorithm to control a real-world 4-DOF manipulator for intelligent control task testing. To compare with the experimental results in the real world and thereby comprehensively evaluate the performance of the FK-DRL and DRL algorithms, this study also conducted simulation training and testing on a 4-DOF manipulator intelligent control task. We set an early stop condition during the simulation process. The training ended when the reward curve of the algorithm converged and remained stable. The training processes for the DDPG, FK-DDPG, TD3, FK-TD3, SAC, and FK-SAC algorithms are depicted in Figure 12, Figure 13 and Figure 14, respectively.

Figure 15, Figure 16 and Figure 17 present the training duration, average reward, and task success rates for the FK-DDPG, FK-TD3, and FK-SAC algorithms versus the DDPG, TD3, and SAC algorithms. The results indicate that when controlling a 4-DOF manipulator, the FK-DDPG, FK-TD3, and FK-SAC algorithms maintain a significant advantage over the DDPG, TD3, and SAC algorithms in terms of convergence speed and control precision. This demonstrates the applicability of the proposed algorithms for intelligent control of manipulators with varying degrees of freedom.

However, the performance improvement of the FK-DDPG, FK-TD3, and FK-SAC algorithms was not as pronounced when controlling a 4-DOF manipulator as it was with a 7-DOF manipulator. This is attributed to the exponential growth in the state and action spaces as the number of degrees of freedom of the manipulator increases. The FK-DRL algorithm possesses an advantage in guiding policy exploration; therefore, the algorithm achieves greater improvements in convergence speed and control precision with higher DOF manipulators.

4. Experiments and Analysis

After validating the effectiveness of the proposed algorithm through simulations, this section deploys the trained FK-DRL algorithm onto a real-world experimental platform and compares it with classical DRL algorithms.

4.1. Experimental Platform

To evaluate the practical effectiveness of the algorithm presented in this paper, a four-degrees-of-freedom manipulator, identical to the one used in the simulation environment (see Figure 18), was employed in a physical setting. The gripper of the manipulator was configured as a fixed joint, with each of the four joints being driven by UART serial bus servos, each rated at a torque of $4.5N\cdot m$.

Prior to conducting the experiment, camera calibration was performed using Zhang’s calibration method [31,32] to establish a coordinate transformation matrix between the pixel coordinate system and the manipulator’s spatial coordinate system. In this study, the camera and the manipulator are not integrated, meaning the camera operates in an “eye-to-hand” configuration, and its pose and position remain constant throughout the experiment. During the experiment, the joint coordinates of the manipulator were obtained through the servo control program, the pose of the end effector was determined using Equation (3), and the target pose was calculated from the pixel coordinates of the blocks captured in the photographs taken by the camera through the established coordinate transformation matrix.

4.2. Analysis of Experimental Results

The model trained was transferred to a real-world environment featuring a 4-DOF manipulator, and the process of the manipulator reaching the target pose was observed. Taking the FK-DDPG algorithm as an example, the sequence of the manipulator’s movement towards the target pose is illustrated in Figure 19a–f, with the entire process taking 9 s.

To evaluate the performance of the proposed FK-DRL algorithm in a physical setting, we also conducted 100 trials and used the average reward and success rate as metrics to assess the experimental outcomes. The results are presented in Figure 20 and Figure 21.

Similar to the simulation results, the FK-DRL algorithm achieved higher average rewards and success rates compared to the classical DRL algorithms. Specifically, the task completion success rates for FK-DDPG, FK-TD3, and FK-SAC were 96%, 98%, and 94%, respectively, while the success rates for the conventional DDPG, TD3, and SAC algorithms were 89%, 44%, and 72%, respectively. However, there was a noticeable decrease in both average rewards and success rates in the physical environment compared to the simulation environment. This discrepancy can be attributed to real-world factors such as joint friction and manufacturing errors, which create inconsistencies between the simulation and the physical environments. This issue can typically be mitigated by further training the agent in the physical environment for a number of episodes, as suggested in the literature [33].

By comparing the experimental results of a 4-DOF manipulator in the real world with those in a simulated environment, we found that, compared to classical DRL algorithms, the task success rates of FK-DDPG, FK-TD3, and FK-SAC in the simulation increased by 6%, 48%, and 20%, respectively. In the real world, the success rates increased by 7%, 54%, and 22%, respectively. The results indicate that the proposed FK-DRL algorithms exhibited more significant stability improvements in the real world. This is attributed to the presence of various interferences in the real world, where classical DRL algorithms demonstrate poor adaptability in such dynamic environments. This further validates the stability and superiority of the proposed FK-DRL algorithms.

5. Conclusions

To address the issues of slow training convergence, low exploration efficiency, and poor stability in classic DRL algorithms when controlling redundant DOF manipulators, an efficient FK-DRL intelligent control algorithm is proposed for the intelligent control tasks of redundant DOF and 4-DOF manipulators. The control problem of the manipulator is transformed into an MDP problem. To improve the convergence and stability of DRL-based controllers, the easily established FK model of the manipulator is integrated into the training framework of the baseline DRL algorithm.

During the training process, the agent initially interacts with a 3-D simulation environment, continuously optimizing its decision-making strategy. After a period of pre-training, failed samples are extracted from the experience pool. The agent then inputs actions with added disturbance noise into the FK model for rapid trial-and-error learning, quickly optimizing the decision-making strategy. The FK-DRL algorithm effectively reduces the number of interactions between the agent and the 3-D simulation environment while obtaining a large number of successful samples during interactions with the FK model, significantly improving the training convergence speed and stability of the intelligent control algorithm.

For simulation verification, a joint simulation framework using CoppeliaSim and Python was constructed, and extensive simulation training and testing were conducted on 7-DOF and 4-DOF manipulators. To compare with the latest technological solutions, we selected the widely recognized DDPG, TD3, and SAC algorithms in various fields and compared them with the proposed FK-DDPG, FK-TD3, and FK-SAC algorithms. Compared to classic DRL algorithms, the FK-DDPG, FK-TD3, and FK-SAC algorithms significantly shortened training time and greatly improved task completion rates. Specifically, in the simulation experiments of the 7-DOF manipulator, the training times of FK-DDPG, FK-TD3, and FK-SAC algorithms were reduced by 21%, 18%, and 68%, respectively, and the task completion rates were increased by 21%, 87%, and 64%, respectively. In the simulation experiments of the 4-DOF manipulator, these algorithms reduced training time by 20%, 40%, and 65%, respectively, and increased task completion rates by 6%, 48%, and 20%, respectively.

Furthermore, we conducted experimental validation on a real 4-DOF manipulator and performed multiple experimental comparisons of the aforementioned algorithms. The results show that the task completion rates of the FK-DDPG, FK-TD3, and FK-SAC algorithms were improved by 7%, 54%, and 22%, respectively, further demonstrating the effectiveness and stability of the FK-DRL algorithm. Simulations and experiments indicate that integrating the FK model into the DRL algorithm’s training framework can not only significantly improve training convergence speed and exploration efficiency but also enhance the adaptability and stability of the algorithm in intelligent control tasks. Additionally, the FK-DRL algorithm avoids the complex processes of dynamic modeling and inverse kinematics solving, exhibits strong scalability, and achieves excellent control performance across manipulators with different DOFs by merely adjusting the easily established FK model of the manipulator, laying a theoretical foundation for more efficient and intelligent control strategies for redundant DOF manipulators.

However, due to the limitations of experimental conditions, this study only conducted control tests on a 4-DOF manipulator in the real world. In the future, we plan to expand the scope of experiments to include a 7-DOF physical manipulator and other physical robotic systems with different DOFs and structures to further validate the performance of the FK-DRL algorithm in real-world scenarios. This will help in more comprehensively assessing the scalability and stability of the FK-DRL algorithm.