Efficient Robot Manipulation via Reinforcement Learning with Dynamic Movement Primitives-Based Policy

Abstract

Reinforcement learning (RL) that autonomously explores optimal control policies has become a crucial direction for developing intelligent robots while Dynamic Movement Primitives (DMPs) serve as a powerful tool for efficiently expressing robot trajectories. This article explores an efficient integration of RL and DMP to enhance the learning efficiency and control performance of reinforcement learning in robot manipulation tasks by focusing on the forms of control actions and their smoothness. A novel approach, DDPG-DMP, is proposed to address the efficiency and feasibility issues in the current RL approaches that employ DMP to generate control actions. The proposed method naturally integrates a DMP-based policy into the actor–critic framework of the traditional RL approach Deep Deterministic Policy Gradient (DDPG) and derives the corresponding update formulas to learn the networks that properly decide the parameters of DMPs. A novel inverse controller is further introduced to adaptively learn the translation from observed states into various robot control signals through DMPs, eliminating the requirement for human prior knowledge. Evaluated on five robot arm control benchmark tasks, DDPG-DMP demonstrates significant advantages in control performance, learning efficiency, and smoothness of robot actions compared to related baselines, highlighting its potential in complex robot control applications.

1. Introduction

Describing the environment as a Markov Decision Process (MDP), reinforcement learning (RL) is a potential branch of machine learning that focuses on iteratively learning tasks through direct interactions without human prior knowledge [1,2]. It has been widely implemented in both academia and industry, achieving significant success in areas such as games [3,4,5], unmanned systems [6,7,8], intelligent manufacturing [9,10,11], and robot control [12,13,14]. In contrast, implementing the unrestricted exploration behaviors of RL and neural network-based control policies in real-world engineering scenarios is extremely costly, particularly when dealing with noise and disturbances [15], which present substantial challenges for implementing learning algorithms to control sophisticated unmanned systems [16]. As a result, developing more sample-efficient methods remains a crucial challenge in the field of RL. The machine learning community has explored various approaches to enhance the sampling efficiency of RL. These included employing entropy and relative entropy to constraint the policy update to improve the agent’s efficiency of exploration in high-dimensional states [17,18], and generating pseudo samples from learned models to boost the training process [19,20]. However, these methods still largely depend on traditional control policies that use neural networks to approximate and determine one-step actions based on the current state, lacking a mechanism for implementing efficient and smooth control trajectories. Overall, current state-of-the-art RL approaches have primarily enhanced learning capabilities by improving actor–critic network update mechanisms and building models for target tasks. However, research on improving learning efficiency and control performance from the perspective of the forms of control actions and their smoothness remains limited.

Unlike the direct control policy based on neural networks, Dynamic Movement Primitives (DMPs) [21,22] offer an appealing solution for designing robot manipulation trajectories while featuring high non-linearity and real-time performance. DMPs are widely used in robot control scenarios including pouring water without spillage [23,24] and deformable object manipulation [25,26,27] owing to their robustness, guaranteed convergence, and easy spatiotemporal scaling. Inspired by DMPs’ advantages in parameterizing smooth and efficient robot motion trajectories, the feasibility of adaptively learning DMP-based control policies in RL on the robot platform has been explored. As a pioneering work that integrated Path Integrals and DMPs within one RL framework, Policy Improvement with Path Integrals (PI2) first investigated the effectiveness of DMPs as the control policy in RL on a simulated quadruped robot with 12 degrees of freedom [28]. Based on this work, a DMP policy incorporating not only robot control trajectories but also contact force signal trajectories was proposed to enhance the performance of PI2, achieving superior performance on a KUKA robot arm in a real-world wood planing scenario [29]. Neural Dynamic Policies (NDPs) [30] attempted to embed DMPs into the actor networks under the modern RL’s actor–critic network framework. They employed neural network-based policies to determine the parameters of the DMP controller, which generated smooth and efficient robot control trajectories. Evaluated by several control tasks built on a simulated Kinova Jaco robot arm, NDPs outperformed Proximal Policy Optimization (PPO) [31] in learning capability and control performance. Although NDPs can be considered a state-of-the-art RL approach using the DMP-based policy, the input states and output actions of their DMP-based policy must be consistent. Otherwise, a manually designed inverse controller—which is usually costly—is required. Therefore, it is difficult to support torque control tasks when the observed states are only related to joint angles. This limitation significantly restricts NDPs’ potential applications in the field of robot control.

To address the generalization issue of NDPs in complex robot control scenarios, particularly in robot arms with multiple degrees of freedom, we propose a novel RL approach DDPG-DMP that combines DMP-based control policy with the actor–critic RL framework from Deep Deterministic Policy Gradient (DDPG) [32]. Our method seamlessly integrates a DMP controller, whose parameters are determined by the actor networks, into the modern RL’s actor–critic structure to generate smooth and efficient robot control trajectories. DDPG-DMP develops an adaptive inverse controller that can be learned through the RL process to flexibly transfer the output trajectories of the DMP controller to various robot control signals while preserving trajectory continuity. This enables the DMP-based policy to generate control action trajectories that differ from the input states. Therefore, the proposed method enjoys a smooth control of various control signals including torque, angle, and end-effector position without requiring human prior knowledge. Evaluated on various robot arm simulation tasks based on torque, angle, and end-joint displacement control, DDPG-DMP achieved significant improvements in learning efficiency and control performance compared to the original DDPG without DMPs and the related work on NDPs. The relationship between DDPG-DMP and other related works is presented in Table 1 where ◯ and × indicate the involved and uninvolved terms, N/A indicates the unapplicable term. The contributions of this work can be summarized as follows:

• Developing a novel RL approach specific for robot control tasks by integrating DMP-based policy and DDPG’s on-policy actor–critic framework. It enhanced the learning capability and the sample efficiency compared to the related work on NDPs [30], which inherits the off-policy characteristics of PPO.
• Designing an adaptive inverse controller that learns the hidden relationships between DMP outputs and various robot control actions. This overcomes the limitation of NDPs [30], which rely on human prior knowledge for designing inverse controllers. As a result, it significantly expands the applicability of DMPs in the RL domain.
• Evaluated on various simulated robot arm scenarios, the proposed method achieved superior control performances compared with NDPs and the original DDPG, highlighting the potential of RL with a DMP-based policy in complex robot control applications.

Table 1. Comparison of FPMPC and the related MBRL approaches.

Approach	Off-Policy	Policy Type	Adaptive Inverse Controller
DDPG [32]	◯	Neural Networks	N/A
SAC [33]	◯	Neural Networks	N/A
TD3 [34]	◯	Neural Networks	N/A
PPO [35]	×	Neural Networks	N/A
NDP [30]	×	DMP	×
DDPG-MPC (ours)	◯	DMP	◯

Table 1. Comparison of FPMPC and the related MBRL approaches.

Approach	Off-Policy	Policy Type	Adaptive Inverse Controller
DDPG [32]	◯	Neural Networks	N/A
SAC [33]	◯	Neural Networks	N/A
TD3 [34]	◯	Neural Networks	N/A
PPO [35]	×	Neural Networks	N/A
NDP [30]	×	DMP	×
DDPG-MPC (ours)	◯	DMP	◯

The remainder of this paper is organized as follows. The background of NDPs and actor–critic networks is introduced in Section 2. Section 3 details the proposed approach. The experimental results are presented in Section 4. The conclusion is presented in Section 5.

2. Preliminaries

2.1. Markov Decision Process

Traditional RL describes the target task as an MDP [1] with elements $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma )$. $\mathcal{S}=\{{\mathit{s}}_1,{\mathit{s}}_2,...,{\mathit{s}}_n\}$ and $\mathcal{A}=\{{\mathit{a}}_1,{\mathit{a}}_2,...,{\mathit{a}}_n\}$ denote the finite sets of states and actions. The transition from one state $\mathit{s}$ to another one ${\mathit{s}}^{\prime }$ under a certain action $\mathit{a}$ follows unknown dynamics ${\mathcal{P}}_{\mathit{s}{\mathit{s}}^{\prime }}^{\mathit{a}}=p\left({\mathit{s}}^{\prime }\right|\mathit{s},\mathit{a})$, and the corresponding task-related reward function is defined as ${\mathcal{R}}_{\mathit{s}{\mathit{s}}^{\prime }}^{\mathit{a}}$. The RL agent aims to search an optimal policy $\pi \left(\mathit{a}\right|\mathit{s})$ to maximize the reward from the current state $\mathit{s}$:

$$\begin{array}{c}\hfill r_{{\mathit{s}}_t}=\sum _{{\displaystyle \genfrac{}{}{0pt}{}{\mathit{a}\in \mathcal{A}}{{\mathit{s}}_{t+1}\in \mathcal{S}}}}\pi \left(\mathit{a}\right|{\mathit{s}}_t){\mathcal{P}}_{{\mathit{s}}_t{\mathit{s}}_{t+1}}^{\mathit{a}}{\mathcal{R}}_{{\mathit{s}}_t{\mathit{s}}_{t+1}}^{\mathit{a}}.\end{array}$$

(1)

Define a discount factor $\gamma \in (0,1)$; the value function is defined as the expected discounted total reward from $\mathit{s}$:

$$\begin{array}{c}\hfill V_{\pi }\left(\mathit{s}\right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_{{\mathit{s}}_t}|{\mathit{s}}_0=\mathit{s}\right].\end{array}$$

(2)

The maximized value function satisfies a Bellman equation:

$$\begin{array}{c}\hfill V^{*}\left(\mathit{s}\right)=\underset{\pi }{max}\sum _{{\displaystyle \genfrac{}{}{0pt}{}{\mathit{a}\in \mathcal{A}}{{\mathit{s}}^{\prime }\in \mathcal{S}}}}\pi \left(\mathit{a}\right|\mathit{s}){\mathcal{P}}_{\mathit{s}{\mathit{s}}^{\prime }}^{\mathit{a}}\left({\mathcal{R}}_{\mathit{s}{\mathit{s}}^{\prime }}^{\mathit{a}}+\gamma V^{*}\left({\mathit{s}}^{\prime }\right)\right).\end{array}$$

(3)

It can be translated to a function of state–action pairs $Q(·)$ that is more widely used:

$$\begin{array}{c}\hfill Q^{*}(\mathit{s},\mathit{a})=\underset{\pi }{max}\sum _{{\mathit{s}}^{\prime }\in \mathcal{S}}{\mathcal{P}}_{\mathit{s}{\mathit{s}}^{\prime }}^{\mathit{a}}\left({\mathcal{R}}_{\mathit{s}{\mathit{s}}^{\prime }}^{\mathit{a}}+\gamma \sum _{{\mathit{a}}^{\prime }\in \mathcal{A}}\pi \left({\mathit{a}}^{\prime }|{\mathit{s}}^{\prime }\right)Q^{*}\left({\mathit{s}}^{\prime },{\mathit{a}}^{\prime }\right)\right).\end{array}$$

(4)

2.2. Deep Deterministic Policy Gradient

DDPG [32] introduced actor–critic networks to separately learn the estimated Q function and the policy $\pi$ by critic neural networks $Q_{\mathit{\theta }}$ and actor networks ${\pi }_{\mathit{\varphi }}$ with independent parameters $\mathit{\theta }$ and $\mathit{\varphi }$. At the t-th step of interaction, the agent of DDPG decides the control action with exploration noise $\mathit{\omega }$ following Gaussian distribution or the Ornstein–Uhlenbeck process from its actor networks ${\mathit{a}}_t={\pi }_{\mathit{\varphi }}\left({\mathit{s}}_t\right)+\mathit{\omega }$. After executing ${\mathit{a}}_t$, the agent observes the next step state ${\mathit{s}}_{t+1}$ and a reward ${\mathcal{R}}_{{\mathit{s}}_t{\mathit{s}}_t^{\prime }}^{{\mathit{a}}_t}$. The whole transition $({\mathit{s}}_t,{\mathit{a}}_t,{\mathcal{R}}_{{\mathit{s}}_t{\mathit{s}}_t^{\prime }}^{{\mathit{a}}_t},{\mathit{s}}_t^{\prime })$ is stored in the memory replay buffer $\mathcal{D}$ as one sample.

The training procedure of DDPG was designed mainly based on the one of Deep Q Networks (DQN) [36]. For a mini-batch with N samples from the replay buffer $\mathcal{D}$, the critic networks are updated by minimizing the following loss function:

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathit{\theta }\right)& ={\displaystyle \frac{1}{N}}\sum _{n=1}^N{\left({\mathcal{R}}_{{\mathit{s}}_n{\mathit{s}}_n^{\prime }}^{{\mathit{a}}_n}+\gamma Q_{{\mathit{\theta }}^{-}}({\mathit{s}}_n^{\prime },{\pi }_{{\mathit{\varphi }}^{-}}\left({\mathit{s}}_n^{\prime }\right))-Q_{\mathit{\theta }}\left({\mathit{s}}_n,{\mathit{a}}_n\right)\right)}^2.\hfill \end{array}$$

(5)

The actor networks are updated via a sampled policy gradient as follows:

$$\begin{array}{c}\hfill {\nabla }_{\mathit{\varphi }}\leftarrow {\displaystyle \frac{1}{N}}\sum _{n=1}^N{\nabla }_{{\mathit{a}}_n^{\prime }}{Q}_{\mathit{\theta }}\left({\mathit{s}}_n,{\mathit{a}}_n^{\prime }\right){|}_{{\mathit{a}}_n^{\prime }={\pi }_{\mathit{\varphi }}\left({\mathit{s}}_n\right)}{\nabla }_{\mathit{\varphi }}{\pi }_{\mathit{\varphi }}\left({\mathit{s}}_n\right)\end{array}$$

(6)

where $Q_{{\mathit{\theta }}^{-}}$ and ${\pi }_{{\mathit{\varphi }}^{-}}$ are the target networks of the critic and actor with parameters smoothly updated from $\mathit{\theta }$ and $\mathit{\varphi }$ under the control of parameter $\eta$:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{\theta }}^{-}& \leftarrow \eta \mathit{\theta }+(1-\eta ){\mathit{\theta }}^{-},\hfill \\ \hfill {\mathit{\varphi }}^{-}& \leftarrow \eta \mathit{\varphi }+(1-\eta ){\mathit{\varphi }}^{-}.\hfill \end{array}\end{array}$$

(7)

2.3. Dynamic Movement Primitives

DMP [21] was proposed to stably represent complex motor action trajectories that can be flexibly adjusted without manual parameter tuning in robot control. Define ${\alpha }_y$ and ${\beta }_y$ as constants; y, g as the state variable and the target position. One motor action trajectory is encoded by DMP as follows:

$$\begin{array}{c}\hfill \ddot{y}\left(t\right)=\left({\alpha }_y\left({\beta }_y\left(g-y\left(t\right)\right)-\left({\displaystyle \frac{\dot{y}\left(t\right)}{\tau }}\right)\right)+f\left(x_t\right)\right){\tau }^2\end{array}$$

(8)

where $\tau$ is a time constant. Let input x follow a canonical system $\dot{x}=-{\alpha }_{x}x\tau$ with initial value $x_0=1$; the forcing function $f\left(x_t\right)$ is approximated by M Gaussian basis functions ${\psi }_i(·)$ and one $M\times 1$ weights vector $\mathit{w}$:

$$\begin{array}{c}\hfill f\left(x_t\right)={\displaystyle \frac{{\sum }_{i=1}^M{\psi }_i\left(x_t\right)w_i{x}_t}{{\sum }_{j=1}^M{\psi }_j\left(x_t\right)}}=\varphi {\left(x_t\right)}^T\mathit{w},\end{array}$$

(9)

$$\begin{array}{c}\hfill {\psi }_i\left(x_t\right)=exp\left(-{\displaystyle \frac{1}{2{\sigma }_i^2}}{(x_t-c_i)}^2\right).\end{array}$$

(10)

${\sigma }_i$ and $c_i$ are the parameters of the i-th basis function.

DMP encodes a T step trajectory $\mathit{y}={[y\left(t\right),\dot{y}\left(t\right),\ddot{y}\left(t\right)]}_{t=1:T}^T$ by modifying its weight vector $\mathit{w}$. The corresponding forcing function at the t-th step can be calculated from Equation (9):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill f\left(x_t\right)={\displaystyle \frac{1}{{\tau }^2}}\ddot{y}\left(t\right)-{\alpha }_y\left({\beta }_y\left(g-y\left(t\right)\right)-{\displaystyle \frac{\dot{y}\left(t\right)}{\tau }}\right).\end{array}\end{array}$$

(11)

Let $\mathit{f}=\mathbf{\Phi }\mathit{w}$, with basis functions $\mathbf{\Phi }={[\varphi \left(x_1\right),...,\varphi \left(x_T\right)]}^T$ and $\mathit{f}={[f\left(x_1\right),...,f\left(x_T\right)]}^T$; the weights vector $\mathit{w}$ can be determined via least squares regression:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathit{w}={({\mathbf{\Phi }}^T\mathbf{\Phi })}^{-1}{\mathbf{\Phi }}^T\mathit{f}.\end{array}\end{array}$$

(12)

3. Approach

3.1. Neural Dynamic Policies

An NDP [30] attempts to employ the DMP-based policy into the actor–critic structure of the traditional RL approach PPO. Its principle and workflow are illustrated in Figure 1. Receiving the input state ${\mathit{s}}_t$, the actor networks output the parameters of DMP rather than the action signal ${\mathit{a}}_t$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill [g,\mathit{w}]={\pi }_{\mathit{\varphi }}\left({\mathit{s}}_t\right)\end{array}\end{array}$$

(13)

where g and $\mathit{w}$ are the target position and the weights vector of a DMP-based policy, and $\mathit{\varphi }$ is the parameter of the actor networks. Let ${\mathit{y}}_0$ represent the variables related to each action in the observed state ${\mathit{s}}_t$, and assuming that the information of previous $\mathit{y}$ is known (the acceleration and velocity at the initial moment could be initialized as zero and iterated), the NDP generates one trajectories with $m=40$ steps based on the DMP dynamics in Equation (8):

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\ddot{\mathit{y}}}_j& =\alpha (\beta (g-{\mathit{y}}_{j-1})-{\dot{\mathit{y}}}_{j-1}+f\left(x_j\right),\hfill \\ \hfill {\dot{\mathit{y}}}_j& ={\dot{\mathit{y}}}_{j-1}+{\ddot{\mathit{y}}}_{j-1}dj,\hfill \\ \hfill {\mathit{y}}_j& ={\mathit{y}}_{j-1}+{\dot{\mathit{y}}}_{j-1}dj\hfill \end{array}\end{array}$$

(14)

where $dj=0.025$ is set following [30]. At this point, the NDP successfully generates a continuous control action trajectory of length $m=40$ steps based on the input state ${\mathit{s}}_t$. The trajectory ${[{\mathit{y}}_j,{\dot{\mathit{y}}}_j,{\ddot{\mathit{y}}}_j]}_{j=1}^m$ includes position, velocity, and acceleration.

An NDP employs an inverse controller to sample the position information from the generated trajectory ${[{\mathit{y}}_j,{\dot{\mathit{y}}}_j,{\ddot{\mathit{y}}}_j]}_{j=1}^m$ at equal intervals to reconstruct a five-step robot control action trajectory $\mathit{\mu }=[{\mathit{\mu }}_t,{\mathit{\mu }}_{t+1},...,{\mathit{\mu }}_{t+K-1}]$, where $K=5$, which matches the time steps of state and action spaces for the target task. At the next step, the NDP samples five steps’ exploration actions $\mathit{a}=[{\mathit{a}}_t,{\mathit{a}}_{t+1},...,{\mathit{a}}_{t+K-1}]$ based on $\mathit{\mu }$ and sequentially interacts with the external environment using these five-step actions. The five-step interactive samples ${\{{\mathit{s}}_{t+k},{\mathit{a}}_{t+k},{\mathcal{R}}_{{\mathit{s}}_{t+k}{\mathit{s}}_{t+k}^{\prime }}^{{\mathit{a}}_{t+k}},{\mathit{s}}_{t+k}^{\prime },v\left({\mathit{s}}_{t+k}\right)\}}_{k=0}^{K-1}$ are sequentially collected and stored in the memory buffer $\mathcal{D}$. The element $v\left({\mathit{s}}_t\right)$ is the output of critic networks (e.g., the estimated value function), which is required in the update of PPO. Please note that an NDP’s inverse controller conducts a simplified mapping that discards velocity and acceleration information from the initial trajectory ${[{\mathit{y}}_j,{\dot{\mathit{y}}}_j,{\ddot{\mathit{y}}}_j]}_{j=1}^m$. As a result, the output control trajectory $\mathit{a}$ only follows the discrete positions of the DMP trajectory, compromising its quality and smoothness. This characteristic hinders an NDP’s capability to generate DMP trajectories of physical variables absent from the observed states, such as generating torque trajectories from joint states. It is highlighted that such a conversion would require manually designed inverse controllers based on human prior knowledge [30], which significantly constrains an NDP’s applicability in robotics.

The update of actor and critic networks in NDPs is mainly similar to PPO, but instead of purely randomly sampling a mini-batch from the memory buffer $\mathcal{D}$, it selects the entire $K=5$ steps, continuous trajectory to ensure that the continuity of the generated trajectory is fully considered during training. For one training sample $\{{\mathit{s}}_t,{\mathit{a}}_t,{\mathcal{R}}_{{\mathit{s}}_t{\mathit{s}}_t^{\prime }}^{{\mathit{a}}_t},{\mathit{s}}_t^{\prime },v\left({\mathit{s}}_t\right)\}$, the loss function of the actor networks is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathcal{L}\left(\mathit{\varphi }\right)=min\left(c\left(\mathit{\varphi }\right)·A\left({\mathit{s}}_t,{\mathit{a}}_t\right),\mathrm{clip}\left(c\left(\mathit{\varphi }\right),1-ϵ,1+ϵ\right)·A\left({\mathit{s}}_t,{\mathit{a}}_t\right)\right),\end{array}\end{array}$$

(15)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\left(\mathit{\varphi }\right)={\displaystyle \frac{{\pi }_{\mathit{\varphi }}\left({\mathit{a}}_t\right|{\mathit{s}}_t)}{{\pi }_{{\mathit{\varphi }}^{-}}\left({\mathit{a}}_t\right|{\mathit{s}}_t)}}\end{array}\end{array}$$

(16)

where $c\left(\mathit{\varphi }\right)$ is the ratio between the current and previous policies, $ϵ$ is the parameter for clipping overlarge $c\left(\mathit{\varphi }\right)$, and $A({\mathit{s}}_t,{\mathit{a}}_t)$ is the estimated advantage function based on the current critic networks $V_{\mathit{\theta }}$ within the whole rollout with N steps:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill A({\mathit{s}}_t,{\mathit{a}}_t)=\sum _{l=0}^N{\left(\gamma \lambda \right)}^l·\left({\mathcal{R}}_{{\mathit{s}}_{t+l}{\mathit{s}}_{t+l}^{\prime }}^{{\mathit{a}}_{t+l}}+\gamma ·V_{\mathit{\theta }}\left({\mathit{s}}_{t+l}^{\prime }\right)-V_{\mathit{\theta }}\left({\mathit{s}}_{t+l}\right)\right),\end{array}\end{array}$$

(17)

$\lambda$ is the corresponding hyperparameter. The critic networks are updated using the loss function below:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathcal{L}\left(\mathit{\theta }\right)={\left({\mathcal{R}}_{{\mathit{s}}_t{\mathit{s}}_t^{\prime }}^{{\mathit{a}}_t}+V_{\mathit{\theta }}\left({\mathit{s}}_t^{\prime }\right)-V_{\mathit{\theta }}\left({\mathit{s}}_t\right)\right)}^2.\end{array}\end{array}$$

(18)

3.2. DDPG with DMP-Based Policy and Adaptive Inverse Controller

In this subsection, we introduce the proposed method DDPG-DMP as a novel framework to integrate the DMP-based policy into the traditional actor–critic RL-approach DDPG. The principle and the workflow of the proposed method are illustrated in Figure 2. Compared to an NDP, which employs policy networks and value function critic networks, the proposed method is mainly based on the traditional actor–critic framework where the critic networks estimate the q function in order to inherit the off-policy characteristic from DDPG. Given the input state as ${\mathit{s}}_t$, the actor networks in DDPG-DMP output two sets of parameters: $\mathit{\varphi }$ and $\mathit{\rho }$. The first set $\mathit{\psi }=[g,\mathit{w}]$ is consistent with the NDP and presents the target position and the weights vector required by the DMP. The second set $\mathit{\rho }$ is simultaneously passed to an adaptive inverse controller as its neural networks’ parameters.

Using this setup, the DMP generates an m-step trajectory ${[{\mathit{y}}_j,{\dot{\mathit{y}}}_j,{\ddot{\mathit{y}}}_j]}_{j=1}^m$ with position, velocity, and acceleration information based on parameter set $\mathit{\psi }$ using Equation (14). The adaptive inverse controller then converts it into a discrete control trajectory $\mathit{a}=[{\mathit{a}}_t,{\mathit{a}}_{t+1},...,{\mathit{a}}_{t+K-1}]$ with the consideration of the information related to velocity and acceleration. Compared with the inverse controller in an NDP, the adaptive inverse controller proposed in DDPG-DMP enhances the capability of the RL framework in fine-tuning and translating the trajectories generated by the DMP, especially when the control variables are not included in the observed states (e.g., adaptively mapping from joint angles to torque or end-effector position). It contributes to the superior feasibility of the RL and DMP-based policy in a wide range of robot systems without requiring additional human prior knowledge. In practice, the adaptive inverse controller is implemented as an additional neuron (with a tanh activation function) at the end of the actor networks, with its weights and the bias of this neuron determined by $\mathit{\rho }$. Unlike an NDP, where the DMP and inverse controller are external processes of the actor networks, the DDPG-DMP considers ${\pi }_{\mathit{\varphi }}$, DMP, and adaptive inverse controller as three parts of the entire actor network, outputting a control action trajectory with $K=5$ steps based on the input state ${\mathit{s}}_t$.

3.3. Update of DDPG-DMP

Consistent with DDPG, the proposed method adds exploration noises to the generated trajectory $[{\mathit{a}}_t,{\mathit{a}}_{t+1},...,{\mathit{a}}_{t+K-1}]$ and interacts with the external environment using this trajectory. The corresponding samples ${\{{\mathit{s}}_{t+k},{\mathit{a}}_{t+k},{\mathcal{R}}_{{\mathit{s}}_{t+k}{\mathit{s}}_{t+k}^{\prime }}^{{\mathit{a}}_{t+k}},{\mathit{s}}_{t+k}^{\prime }\}}_{k=0}^{K-1}$ within $K=5$ steps are sequentially obtained and stored to the memory buffer $\mathcal{D}$. Define the size of a mini-batch as N; DDPG-DMP randomly samples N complete trajectories (each one has K steps) during its update. Denote ${\{{\mathit{s}}_{t+k},{\mathit{a}}_{t+k},{\mathcal{R}}_{{\mathit{s}}_{t+k}{\mathit{s}}_{t+k}^{\prime }}^{{\mathit{a}}_{t+k}},{\mathit{s}}_{t+k}^{\prime }\}}_{k=0}^{K-1}$ as one sampled trajectory; we only select the first step state ${\mathit{s}}_t$ and the first step of the output control trajectory ${\mathit{a}}_t$ as the input of critic networks $Q_{\mathit{\theta }}(·)$ in order to satisfy the update formulas of the original DDPG. Therefore, we only use the first step’s state ${\mathit{s}}_t$, the next step’s state, ${\mathit{s}}_t^{\prime }$ and action ${\mathit{a}}_t$ for each sampled trajectory. To address this operation’s neglect of information related to subsequent actions in the sampled trajectory and their corresponding states, we aggregate the reward functions in all K steps as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{R}}_t^{\mathrm{traj}}=\sum _{k=0}^{K-1}{\mathcal{R}}_{{\mathit{s}}_{t+k}{\mathit{s}}_{t+k}^{\prime }}^{{\mathit{a}}_{t+k}}.\end{array}\end{array}$$

(19)

The processed sample $[{\mathit{s}}_t,{\mathit{a}}_t,{\mathcal{R}}_t^{\mathrm{traj}},{\mathit{s}}_t^{\prime }]$ is finally used to update both actor and critic networks in DDPG-DMP. The learn and update processes of the proposed method are summarized in Algorithm 1. Please note that there is one significant limitation of the proposed DDPG-DMP: beyond the first step, the remaining control actions and their corresponding states in the K-step trajectory generated by the actor networks are almost invisible to the agent during the update process. They only influence the accumulated reward ${\mathcal{R}}_t^{\mathrm{traj}}$. This characteristic substantially restricts the learning efficiency of DDPG-DMP. Addressing this limitation will be a crucial focus of our future work.

Algorithm 1: Learning and Update Processes of the Proposed DDPG-DMP

4. Experimental Results

4.1. Simulation Settings

In this section, we report the evaluation of the proposed method DDPG-DMP using five robot arm manipulation scenarios from two different benchmark environments. The first three tasks Thrower-v0, Pusher-v0, and Reacher-v1 were from OpenAI Gym (https://www.gymlibrary.dev/environments/mujoco/pusher/, accessed on 15 November 2024) and aimed to control a robot to throw, push the target object, or move the end-effector to the randomly generated target positions. We further selected the PandaReach-v2 task from the Panda robot arm simulation environment panda-gym (https://github.com/qgallouedec/panda-gym, accessed on 15 November 2024), where the agent directly controls the end-effector position of a Panda robot arm to reach the randomly generated target positions. To additionally demonstrate the advantages of our method in directly controlling robot joints, we extended the PandaReach-v2 task PandaReachJoints-v2 where the agent can directly control all joints of the Panda robot arm. The state–action spaces of the five tasks and the corresponding parameters for DDPG-DMP are shown in

Table 2. Common hyperparameter settings of compared environments.

	Reacher-v1	Pusher-v0	Thrower-v0	PandaReach-v1	PandaReachJoints-v1
State Dimension	11	23	23	20	20
Action Dimension	2	7	7	3	7
Critic Learning Rate	$3\times {10}^{-4}$	$3\times {10}^{-4}$	$3\times {10}^{-4}$	$3\times {10}^{-4}$	$3\times {10}^{-4}$
Actor Learning Rate	$5\times {10}^{-4}$	$5\times {10}^{-4}$	$5\times {10}^{-4}$	$3\times {10}^{-4}$	$3\times {10}^{-4}$
Target Update Rate ($\tau$)	$5\times {10}^{-3}$	$5\times {10}^{-3}$	$5\times {10}^{-3}$	$5\times {10}^{-3}$	$5\times {10}^{-3}$
Networks Structure	$(256,256)$	$(256,256)$	$(256,256)$	$(256,256)$	$(256,256)$
Batch Size (J)	256	256	256	256	256
Discount Factor ($\gamma$)	0.99	0.99	0.99	0.95	0.95
Steps per Update (K)	5	5	5	5	5
Memory Buffer Size	${10}^6$	${10}^6$	${10}^6$	${10}^5$	${10}^5$
Warmup Steps	${10}^4$	${10}^4$	${10}^4$	2000	2000
Control Modes	Torque	Torque	Torque	End-Effector Position	Joint Angle

. Considering that all experimental environments supported the standard OpenAI Gym API, the proposed method can be directly applied to any real physical robot system compliant with OpenAI Gym API and ROS communication API, such as panda_robot package (https://github.com/rvl-lab-utoronto/panda_robot, accessed on 15 November 2024), which is adapted for Panda robot arm hardware, and robo-gym package (https://github.com/jr-robotics/robo-gym, accessed on 15 November 2024), which supports the UR5 robot arm hardware. Most parameters, including the neural network structure, the learning rate of critic networks, and the target update rate, were set uniformly across tasks. We only adjusted the learning rate of actor networks and the steps for warm-up to meet the specific requirements of each task. In terms of the compared baselines, we selected several widely applied RL algorithms: DDPG [32], Twin Delayed Deep Deterministic Policy Gradient (TD3) [34], and Soft Actor Critic (SAC) [33], as well as the NDP [30], which is based on a DMP policy. All compared methods were developed using PyTorch (DDPG, TD3, and SAC were implemented following open source code: https://github.com/Lizhi-sjtu/DRL-code-pytorch (accessed on 15 November 2024), and the NDP was implemented following its official code: https://github.com/shikharbahl/neural-dynamic-policies, accessed on 15 November 2024). NDPs can be seen as the state-of-the-art RL approach using the DMP policy. DDPG, TD3, and SAC were selected as the traditional model-free RL approaches using the actor–critic framework. We did not consider other newer model-freeRL methods as baselines because our proposed DDPG-DMP can be directly applied to any method with an actor–critic framework. In this paper, we primarily focused on the improvement in learning performance when employing the DMP-based policy compared to the NDP and other similar approaches that do not use the DMP-based policy. All experiments were conducted in five independent trials on a computational server with Intel Xeon W2295 CPU, NVIDIA GeForce RTX 3090 GPU, 64GB memory, and Ubuntu 20.04 OS (Santa Clara, CA, USA).

The experimental results are detailed in the following order. The learning capability of all compared methods in OpenAI robot arm scenarios is evaluated in Section 4.2, where the learning curves are shown in Figure 3, and the average rewards are compared in

Table 3. Average return over three OpenAI robot arm scenarios.

	DDPG-DMP	DDPG	TD3	SAC	NDP
Reacher-v1	$-7.36\pm 0.54$	$-6.58\pm 0.54$	$-7.89\pm 0.57$	$-6.10\pm 0.61$	$-15.56\pm 1.47$
Pusher-v0	$-33.37\pm 1.73$	$-44.23\pm 8.82$	$-32.06\pm 1.41$	$-31.87\pm 4.01$	$-46.81\pm 2.16$
Thrower-v0	$-62.74\pm 9.78$	$-180.37\pm 24.61$	$-111.52\pm 23.74$	$-95.33\pm 27.72$	$-74.90\pm 18.27$

. The corresponding results of the Panda robot arm scenarios are demonstrated in Figure 4 and

Table 4. Success rates over two Panda simulation control scenarios.

	DDPG-DMP	DDPG	TD3	SAC	NDP
PandaReach-v2	$0.99\pm 0.00$	$0.41\pm 0.24$	$0.99\pm 0.00$	$0.99\pm 0.01$	$0.00\pm 0.00$
PandaReachJoints-v2	$0.99\pm 0.01$	$0.00\pm 0.00$	$0.92\pm 0.01$	$0.85\pm 0.41$	$0.00\pm 0.00$

. Figure 5, Figure 6 and Figure 7 illustrate the control behaviors of DDPG-DMP and the baseline in OpenAI and Panda robot arm scenarios as a case study.

4.2. OpenAI Robot Arm Scenarios

The learning performances of the proposed method and other baselines in three tasks in the OpenAI Gym environment, Reacher-v1, Pusher-v0, and Thrower-v0 are analyzed in this subsection. The learning curves of all compared methods in these tasks are presented in Figure 3, where the shaded region represents the corresponding standard deviation over five independent trials with different random seeds. The final average returns obtained by each method over five trials are compared in Table 3. Both results demonstrate the significant superiority of our method in terms of not only learning efficiency and control performance.

In the Reacher-v1 task, DDPG, TD3, SAC, and the proposed DDPG-DMP ultimately converged to similar average returns according the learning curves. Regarding the convergence speed, DDPG-DMP achieved an approximately $-10$ average return in about 26,000 steps, while DDPG required around 33,000 steps, indicating an $11.7\%$ improvement for our method. As a comparison, SAC required more interaction steps to converge in this task from a relatively low initial average return due to its inherent random exploration policy. Only TD3 achieved a learning efficiency comparable to DDPG-DMP. Please also note that DDPG-DMP only updates after executing a DMP trajectory segment every $K=5$ steps while traditional RL methods like DDPG, TD3, and SAC update the agent at every step. This further showcases the significant advantage of our method in updating efficiency. Although the average return calculated over five-step samples resulted in a slight disadvantage for DDPG-DMP as shown in Table 3, this had no significant impact on actual control performance. In contrast, the NDP, which was also based on a DMP policy, struggled to leverage DMP’s smooth control advantages in the this task. Its learning curve shows no significant improvements, resulting in our method outperforming the NDP by up to $52.7\%$ in the final converged average return.

In Pusher-v0, which has larger state and action spaces, the proposed DDPG-DMP outperformed the DDPG and NDP in the final converged average (with $43.7\%$ and $26.7\%$ higher average return, respectively) while achieving the best learning efficiency. Our method quickly converged to over $-40$ average return within 30,000 steps. In the most challenging task Thrower-v0, it was clearly observed that the original DDPG failed to learn a proper policy to finish this task. Other traditional RL approaches like TD3 and SAC could not maintain their average return and finally converged to poor control performances with very high standard deviations. As a comparison, the NDP showcased the advantages of a DMP-based policy in control performance for the given task, which requires high coordination between trajectory smoothness and control actions across seven degrees of freedom. It consistently achieved a suboptimal average return, significantly outperforming DDPG, SAC, and TD3. However, NDP took about 150,000 steps to converge, which was relatively slow. In contrast, the proposed algorithm not only enhanced the advantages of the DMP-based policy in control performance but also maximized its learning efficiency. By generating smooth control action trajectories independently for all seven joints simultaneously, our method demonstrated improvements of $65.1\%$, $43.8\%$, $34.2\%$, and $16.2\%$ compared to DDPG, TD3, SAC, and NDP, respectively. Remarkably, DDPG-DMP surpassed the NDP’s peak average return using only one-third of the interactions, exhibiting superior learning efficiency. The NDP’s poor performances in three torque-based tasks were reasonable. Its inverse controller failed to generate proper torque trajectories when the observed states lacked torque information. Moreover, its on-policy characteristic inherited from PPO also contributed to a lower learning efficiency. In contrast, our method employed an adaptive inverse controller (it was implemented by one additional neuron in the actor networks) to learn the mapping from joint information to torque trajectories, demonstrating significant advantages in both control performances and learning efficiency. Furthermore, the off-policy inherent from DDPG contributed to superior sample efficiency than the NDP conducted with on-policy learning following PPO. We should focus on the improvements of DDPG-DMP over the original DDPG, as our algorithm can be freely applied to TD3 and SAC with almost no modifications, allowing us to benefit from these algorithms’ enhanced learning capabilities compared to DDPG.

4.3. Panda Robot Arm Scenarios

In this subsection, we evaluate the proposed method by more complex robot arm control scenarios: PandaReach-V2 and PandaReachJoints-V2 on a Panda robot arm simulation environment panda-gym (https://github.com/qgallouedec/panda-gym, accessed on 15 November 2024). The first task directly controls the three-dimensional position of the end-effector via internal inverse kinematics. The second task is extended from the first one to further control the angle of each joint to further investigate the capability of the proposed method in the Panda robot arm. The reward is designed to reach the randomly generated end-effector position as closely as possible. The observation states of two tasks were designed as a 20-dimensional vector including the position and velocity of the end-effector. The learning curves of all compared approaches are illustrated in Figure 4. Table 4 compares the final converged results of all methods. We selected success rate as the evaluation metric following the original paper of panda-gym [37].

In the PandaReach-V2 task, DDPG struggled to learn and exhibited an unstable learning process, evidenced by a high standard deviation in the success rate. TD3 learned at a nearly $100\%$ success rate, albeit at a bit slower convergence. Among traditional RL methods, only SAC achieved quick and stable convergence. In terms of the approaches using DMP-based policy, the NDP completely failed to learn this task. As a comparison, the proposed method converged to an almost $100\%$ success rate with efficiency second only to SAC. It is worth noting that while our method was slightly less efficient than the neural network policy-based SAC in this task where directly controlling the end-effector position created a relatively simple scenario, it demonstrated superior smoothness in robot trajectories when controlling robot arm joint angles. We explore this advantage in the next case.

In the PandaReachJoints-V2 task, the agent attempts to move the end-effector to reach the randomly generated target position by controlling the seven joint angles in a Panda robot arm. This task implicitly requires learning the unknown inverse kinematics of a Panda robot, making it significantly more complex than the previous task. Such an increased challenge results in all comparison methods failing to reach the target successfully in the first 50,000 steps. Among all traditional RL approaches based on the neural network policy, the DDPG showed no signs of learning the task, even after about 200,000 interaction steps. TD3 slowly learned to a success rate of around $92\%$. SAC, despite its advantages in the previous task, underperformed TD3 in this task. It demonstrated good learning efficiency within the first 120,000 steps but turned to unstable afterward, finally converging to a low success rate ($85\%$) with a high standard deviation. For the approaches using the DMP-based policy, the NDP continued to struggle in this task, while the proposed method demonstrated the significant advantages of DMP’s smooth trajectories in both task success rate and learning efficiency. DDPG-DMP not only stably converged to a $99\%$ success rate but also enjoyed markedly superior learning efficiency compared to other baselines. These results indicate that the adaptive inverse controller is crucial to employ a DMP-based policy into RL for general applications in robotic systems. Under the proper setting in DDPG-DMP, the DMP-based policy contributes to significant advantages in the learning efficiency.

4.4. Case Study

In this subsection, we describe conducted case studies on the OpenAI and Panda robot arm scenarios to demonstrate the advantages of the proposed DDPG-DMP over other baselines. We focus particularly on comparing it to the original DDPG, SAC, and TD3 that employ neural network-based policies, in terms of their impacts on robotic arm control behaviors. Figure 5 shows the screenshots of a robot arm controlled by the trained DDPG and DDPG-DMP in one complete rollout (100 steps) in the Pusher-v0 task, depicting the robot’s state at steps $0, 25, 50, 75$, and 100 from left to right. It was clearly observed that DDPG’s torque control lacked proper coordination between multiple joints, causing excessive robotic arm movement. Between steps 25 and 50, the robot end-effector pushed the object far away from the target position before reversing direction to complete the task. In contrast, the proposed DDPG-DMP generated gentler control actions, smoothly guiding the robot arm’s end-effector towards the object from steps 0 to 75, then steadily pushing it to the target within the final 25 steps. Figure 4 illustrates the trajectories of joint angle positions and angular velocities for three crucial joints (wrist3, wrist4, wrist5) using DDPG and DDPG-DMP. Owing to the characteristic of the DMP, our method controlled each joint smoothly in terms of both angle and angular velocity. In contrast, the original DDPG without employing the DMP-based policy not only exhibited larger joint angle fluctuations but also produced severe jitter in joint angular velocities, further leading to excessively large robot arm control actions and violent swings in the end-effector position.

Furthermore, we analyzed the superiority of DDPG-DMP in directly generating joint angle control trajectories in the PandaReachJoints-v2 task compared to TD7 and SAC (DDPG and NDP were not included in this comparison due to their poor control performance). Figure 7 compares the trajectories of all seven joint angle signals in one complete rollout for one target position after 175,000 steps’ training using the aforementioned methods (joints are represented by indices 0 to 6, from the base to the joint closest to the end-effector). It was observed that in controlling the base joints (index 0, index 1) and shoulder joint (index 2), all three methods generated relatively stable control trajectories while DDPG-DMP achieved relatively smoother and smaller-amplitude joint angle trajectories. For the joints responsible for controlling the first segment of the robotic arm (index 3 and 4), TD3 and SAC exhibited noticeable jitter, negatively impacting the coordination of all seven joints. In contrast, DDPG-DMP consistently generated smooth trajectories, effectively avoiding joint jitter. For the last two joints (indices 5 and 6), our method’s control trajectories demonstrated clear superiority in both appropriate joint movement range and smooth angle changes. This led to superior control performance and a high success rate.

5. Conclusions and Discussion

In this work, we introduce DDPG-DMP, a novel and effective RL approach that employs a DMP to generate robot control actions. Unlike traditional RL methods that use neural networks to directly output control signals, our method generated proper robot control trajectories with consideration of the velocity and acceleration information, resulting in smooth and effective robot control behaviors. Compared to a related RL approach using DMP, DDPG-DMP leveraged the off-policy actor–critic framework for enhanced learning efficiency and achieved superior feasibility across a wide range of control signals (e.g., torque, joint angle, and end-effector position) without requiring additional human knowledge by proposing an adaptive inverse controller. The experimental results of robot manipulation tasks on OpenAI Gym and Panda simulation demonstrated significant advantages in control performance, learning efficiency, and smoothness of robot actions over related baselines, indicating the potential of our method in the field of robot control.

However, the current DDPG-DMP had limitations in fully utilizing trajectory information. Beyond accumulating the rewards over a K-step trajectory, it lacked a mechanism to fully consider the entire trajectory’s states and actions (as discussed in Section 3.3). This presents a crucial direction for our future work. We also plan to explore integrating the DDPG-DMP framework with state-of-the-art model-free RL methods such as Continuous Dynamic Policy Programming (CDPP) [18], TD7 [38], and CrossQ [39]. Furthermore, incorporating DMP-based policies into model-based RL methods like Dropout-based Probabilistic Ensembles with Trajectory Sampling (DPETS) [40] and optimizing the initial trajectories of its model predictive controller offers an interesting research direction. From the perspective of engineering implementation, we aim to investigate the potential of DDPG-DMP in various real-world scenarios, including real physical systems such as Panda and UR5 robot arms, quadruped robots, and dexterous hands.