Combined Reinforcement Learning and CPG Algorithm to Generate Terrain-Adaptive Gait of Hexapod Robots

Abstract

Terrain adaptation research can significantly improve the motion performance of hexapod robots. In this paper, we propose a method that combines reinforcement learning with a central pattern generator (CPG) to enhance the terrain adaptation of hexapod robots in terms of gait planning. The hexapod robot’s complex task presents a high-dimensional observation and action space, which makes it challenging to directly apply reinforcement learning to robot control. Therefore, we utilize the CPG algorithm to generate the rhythmic gait while compressing the action space dimension of the agent. Additionally, the proposed method requires less internal sensor information, which exhibits strong applicability. Finally, we conduct experiments and deploy the proposed framework in the simulation environment. The results show that the terrain adaptation policy trained in our framework enables the hexapod robot to move more smoothly and efficiently on rugged terrain compared to the traditional CPG method.

1. Introduction

The movement path of the legged robot is formed by discrete landing points. In uneven terrain environments, legged robots have a stronger motion potential than regular wheeled or tracked mobile robots and can meet task requirements in more complex environments. Among them, the good motion stability of the hexapod robot makes it have greater advantages and application value in the fields of rescue, transportation exploration, and so on [1,2,3]. However, redundant actuators increase the difficulty of modeling and motion planning. In order to make robots move smoothly on irregular ground, complex control systems and algorithms usually need to be designed [4,5,6]. At present, the research on adaptive motion control technology of hexapod robots mainly includes (1) model-based motion optimization [7,8,9,10,11,12,13]; (2) bio-inspired methods [14,15,16,17,18]; and (3) machine learning [19,20,21,22].

The hexapod robots in [7,8,10] use external sensing sensors, such as depth sensors [8], stereo cameras [7], and LIDAR [10], to model the surrounding, so that the robot can determine a reasonable landing point position and leg trajectory through planning algorithms and realize the crossing of obstacles and balancing the body. Such methods can significantly improve the motion performance of robots but require expensive sensors and large computational resources to ensure efficiency. The method proposed in [9,13] only relies on the legs of the robot to sense the environment and adjusts the joint positions through the balance algorithm to maintain the body posture. This class of algorithms has low implementation cost, but it requires a large amount of prior knowledge, and the improvement in robot motion performance is limited. In [18], a new biologically inspired adaptive neuroendocrine control approach was proposed, which combined the three modules of CPG control, an artificial hormone network, and unsupervised input-related learning so as to make effective adjustment of legs in continuous motion. Moreover, the method does not require a robot model and is therefore general. In [21,22], the authors adopted the hierarchical control framework and reinforcement learning method to improve the adaptive motion ability of the hexapod robot. The policy network obtained by iteratively training a hexapod robot in a stochastic simulation environment has a certain generalization capability, which is an effective approach to solve the control problem of complex nonlinear systems in an unknown environment.

Motion control based on CPG offers the advantages of simple design and low computational complexity. However, the performance of CPG-based motion control is poor in rough terrain. At the same time, directly controlling the joint angles of hexapod robots with reinforcement learning to achieve terrain-adapted walking is challenging. On the one hand, this is because the hexapod robot has many joints, which means that as the dimension of the output layer of the policy network increases, there are more parameters to learn in the network. On the other hand, we need the hexapod robot to converge to a regular walking pattern instead of swinging its legs in a haphazard manner, which requires a lot of effort in the design of the reward function. Imitation learning and reinforcement learning are usually combined to make the robot learn a specific behavior pattern.

Therefore, this paper focuses on designing a novel terrain-adapted gait generation method for hexapod robots by combining CPG and reinforcement learning, with leg contact feedback as the primary means of environment sensing. The signal coupling of oscillators simplifies gait design and reduces the action space dimension of reinforcement learning, making it easier to converge. First, a network comprising of six Hopf oscillators generates the basic rhythmic gait signal. Then, the mapping function transforms the signal into the foot coordinates. Finally, the inverse kinematic (IK) module computes the angle of motion of the joints. The key point of this approach is that the parameters of the mapping function, corresponding to the oscillator of each leg, are variable and their values are determined by the policy network trained by reinforcement learning. The parameters are modified to change the trajectory of the foot, and the body pose adjusts during the walk to improve the motion performance in rough terrain.

The rest of the paper is organized as follows: Section 2 introduces the hexapod robot platform and the method of generating rhythmic gait using the CPG network. Section 3 proposes a terrain-adaptive gait generation framework that fuses reinforcement learning and CPG networks (RL-CPG) and describes the design and training details of the policy network. Section 4 presents experiments to validate the effectiveness and feasibility of the proposed framework. Finally, the conclusions and avenues for future research are presented in Section 5.

2. Rhythmic Gait Generation by the CPG Network

In this section, we briefly introduce the used hexapod robot platform. Then, a network composed of Hopf oscillators for generating rhythmic gaits is described in detail.

2.1. Hexapod Robot

The work in this paper uses a hexapod robot model built in Pybullet [23], as shown in Figure 1. The six legs are evenly distributed around the body and each has three joints: the hip, knee, and ankle joints. The robot is blind because it does not carry external sensors such as radar and camera, and the robot can only sense its own state information, body attitude angle, acceleration, foot-tip collision, etc. External precision sensors have a high probability of damage, and in the absence of such sensor information, we would like the hexapod robot to still have some terrain adaptation capability; hence, we initiated a related study on blind hexapod robot platforms.

2.2. CPG Network

The CPG algorithm is used to output a stable periodic signal. Classical oscillation models include the Matsuoka neuron oscillator model [24,25], the Van der Pol (VDP) relaxation oscillator [26,27], the Kuramoto phase oscillator [28], etc. The Hopf harmonic oscillator model [29,30] has the advantages of high stability, few parameters, clear physical meaning, and easy tuning. Therefore, the Hopf oscillator is chosen as the signal source in this paper and its basic mathematical model is as follows:

$$\left\{\begin{array}{c}\dot{x}=\alpha \left(\mu -x^2-y^2\right)x-\omega y\\ \dot{y}=\beta \left(\mu -x^2-y^2\right)y+\omega x\end{array}\right.,$$

(1)

where $x$, $y$ are the output of the oscillator; $\alpha$, $\beta$ are convergence rate coefficients; $\mu$ is the square of the output signal amplitude; $\omega$ is the frequency of the oscillator. When the hexapod robot walks with different gaits, the support time and swing time of the legs are not the same. In order to make the rising section of the oscillator output correspond to the swing phase of the leg and the falling section correspond to the support phase of the leg, the coverage coefficient $\epsilon$ is added to adjust the value of $\omega$, which has the following relationship:

$$\left\{\begin{array}{c}\omega =\frac{{\omega }_{st}}{e^{-ay}+1}+\frac{{\omega }_{sw}}{e^{-ay}+1}\\ {\omega }_{st}=\frac{1-\epsilon }{\epsilon }{\omega }_{sw}\end{array}\right.,$$

(2)

where ${\omega }_{st}$ and ${\omega }_{sw}$ are stance phase frequency and swing phase frequency, respectively; $a$ is a positive coefficient ($a=100$ in this paper) that determines the rate of transition of the frequency ω between ${\omega }_{st}$ and ${\omega }_{sw}$. The coefficient of common gait on land occupancy is shown in the

Table 1. Common gaits corresponding to the coverage coefficient.

Gait	Tripod	Tetrapod	Wave
$\epsilon$	1/2	2/3	5/6

below.

In this paper, each leg is controlled using signals generated by a single oscillator unit, and the whole robot uses six oscillator units to form a coupled network to generate a rhythmic gait. As shown in Figure 2, the network is performed by bidirectional coupling of neighboring oscillatory units.

In order to realize coupling, the signals generated by different oscillators, the coupling term ${\mathsf{\Delta }}_{ji}$ is added to Equation (1), and the final mathematical model is given by

$$\left\{\begin{array}{c}{\dot{x}}_i=\alpha \left(\mu -x_i^2-y_i^2\right)x_i-{\omega }_i{y}_i\\ {\dot{y}}_i=\beta \left(\mu -x_i^2-y_i^2\right)y_i+{\omega }_i{x}_i+\lambda {\displaystyle {\displaystyle \sum }_j}{\Delta }_{ji}\\ x_i=\left(x-{\mathrm{offset}}_{x_i}\right)\\ y_i=\left(y-{\mathrm{offset}}_{y_i}\right)\end{array}\right.,$$

(3)

where $\lambda$ represents the coupling strength coefficient between oscillators ($\lambda =0.1$ in this paper), and if its value is too large, it will cause the curve to appear blurred; the output signal compensation ${\mathrm{offset}}_{x_i}$ and ${\mathrm{offset}}_{y_i}$ are used to adjust the center position of the output signal to meet the needs of different gaits. The coupling way of the network determines the expression of the coupling term as follows:

$${\Delta }_{ji}=\left\{\begin{array}{c}\left(y_j\cos {\theta }_{ji}-x_j\sin {\theta }_{ji}\right) \left|j-i\right|\le 1;i,j=1,2,\dots ,6\\ 0 \left|j-i\right|>1;i,j=1,2,\dots ,6\end{array}\right.,$$

(4)

where ${\theta }_{ji}$ represents the phase difference between oscillators, and the phase difference between legs corresponding to common gaits is shown in Figure 3.

The transition from the output signal of the oscillator to the control signal of the robot joint requires a mapping that transforms the signal into a foot trajectory and then computes the joint angle via inverse kinematics. The mapping function is defined as follows:

$$\left\{\begin{array}{c}{f}_{ix}=f_{i{x}_0}+k_0{x}_i\sin \varphi \\ f_{iy}=f_{i{y}_0}+k_0{x}_i\cos \varphi \\ f_{iz}=\left\{\begin{array}{c}{f}_{i{z}_0}+k_1{y}_i+b y_i\ge 0\\ f_{i{z}_0}+k_2{y}_i+b y_i<0\end{array}\right.\end{array}\right.,$$

(5)

where $\left(f_{i{x}_0},f_{i{y}_0},f_{i{z}_0}\right)$ is the initial position of the foot; $k_0$, $k_1$, $k_2$ are the scaling coefficients, depending on the workspace of the specific robot; $b$ is the z-axis compensation and $\varphi$ is the direction of the stride. In summary, Equations (2)–(5) form the basic part of the adaptive gait generator in this paper, and with appropriate gait parameters, the hexapod robot will move with a rhythmic gait generated by the CPG algorithm. Figure 4 shows the robot moving in a tripod gait on flat ground with the mapping parameters set to $k_0=0.02$, $k_1=0.04$, and $k_2=0$.

3. Terrain-Adaptive Gait Generation

3.1. Problem Description

Here we consider the robot interacting with the environment in discrete time steps. At each time step, the robot state (position, attitude, velocity, angular velocity, angular acceleration, torque, etc.) and environmental information (terrain height, ground friction coefficient, etc.) can be completely described by a set of state vectors ${\mathit{s}}_{\mathit{t}}\in \mathcal{S}$. By definition, this vector contains the information needed to select the operation at the current time step. The robot obtains the information ${\mathit{o}}_{\mathit{t}}\in \mathcal{O}$ from ${\mathit{s}}_{\mathit{t}}$ and performs an action ${\mathit{a}}_{\mathit{t}}\in \mathcal{A}$ to interact with the environment. At the next time step, the state vector is updated and state-transition distribution is denoted by $\mathcal{T}\left(s_t, s_{t-1}\right)$. The reward function $r_t\in \mathcal{R}\left(a_t,s_t\right)$ evaluates the quality of each state-action transition.

The above process can be described as a Markov decision process (MDP) and satisfies the basic Markov property; that is, in a sequential process, the state at time $t+1$ only depends on the state at time $t$ and has nothing to do with the state before time $t$. When the state ${\mathit{s}}_{\mathit{t}}$ is not fully observable, that is ${\mathit{o}}_{\mathit{t}}\subset {\mathit{s}}_{\mathit{t}}$, we refer to the process as a partially observable MDP (POMDP).

In this paper, it is obvious that the robot cannot observe all the information about the environment, so we treat the learning problem as a POMDP. The agent selects the appropriate action by the policy network ${\pi }_{\theta }:\mathcal{O}\to \mathcal{A}$. The parameter $\theta$ denotes the weights of the neural network. Our goal is to find policy parameters such that the reward is maximized over a finite time horizon $T$ using the observation ${\mathit{o}}_{\mathit{t}}$, according to the following objective:

$${\theta }^{\prime }={\arg }_{\theta }\max \mathbb{E}\left[{\displaystyle \sum }_{t=0}^{T-1}{\gamma }^t{r}_t\right],$$

(6)

where $0<\gamma \le 1$ is a discount factor, $\mathbb{E}$ is the expected value, and ${\theta }^{\prime }$ is the optimal policy parameters. The architecture of the terrain-adaptive gait generator designed in this paper is shown in Figure 5.

In our framework, the terrain-adaptive gait generator with RL-CPG algorithm adjusts the position of the foot by using different mapping parameters. This is a simple and practical way to adjust the body balance, regardless of which rhythmic gait the robot is walking with. As a result, in the reinforcement learning task, we adopt an action ${\mathit{a}}_{\mathit{t}}=\left\{\vartheta , \mathit{b}\right\}$ that is a 7-D vector. $\vartheta \in \left[-1,1\right]$ represents the turning direction of the robot, where the robot turns by adjusting the stride difference between the legs on both sides of the body. The larger the absolute value of $\vartheta$, the larger the stride difference. $\mathit{b}$ is the variable in the mapping function of each leg.

As mentioned above, our research is based on the blind hexapod robot platform. This means that there is a limit to what the robot can observe. Considering future deployments on real robots, we choose the observation ${\mathit{o}}_{\mathit{t}}=\left\{\mathit{\phi },\mathit{\omega },\mathit{v},\mathit{S},\mathit{\tau },\dot{\mathit{\theta }}\right\}$. $\mathit{\phi }$ is the Euler angle of the body. $\mathit{\omega }$, $\mathit{v}$ are the angular and linear velocities of the body, respectively. $\mathit{S}$ is a 6-D Boolean vector representing the touchdown of each leg. $\mathit{\tau }$ and $\dot{\mathit{\theta }}$ are the joint torques and angular velocities, respectively.

3.2. Reward Function

The reward design encourages the robot to adjust the position of its foot according to the terrain and its own motion state to keep the fuselage as smooth as possible while maintaining the forward direction. Therefore, we design the following rules of reward:

(1)

In our work, the robot does not have sensors to sense the terrain ahead, so it cannot make a plan in advance. However, the robot is able to level the tilted body by compensating for joint positions, and this behavior deserves a reward. Thus, the expression for $r_{balance}$ is given by

$$r_{balance}=\left|\min \left(\mathit{\phi }\circ \mathit{\omega }, 0\right)\right|,$$

(7)

where $\mathit{\phi }\circ \mathit{\omega }$ means that each element of the two vectors is multiplied separately. $r_{balance}$ encourages actions that help the robot recover its balance.

(2)

In order to avoid the robot taking advantage of the first rule to obtain a large number of rewards and behaving in an undesired way, we need to design a negative reward as follows:

$$r_{rotation}=-\left({\delta }_1\sum \left|\mathit{\phi }\right|+{\delta }_2\sum \left|\mathit{\omega }\right|\right),$$

(8)

where ${\delta }_1$ and ${\delta }_2$ are the weighting factors.

(3)

The reward $r_{forward}$ for robot motion is necessary, and we expect the robot to keep moving forward in rough terrain. The expression is

$$r_{forward}=v+d,$$

(9)

where $v$ is the forward speed of the robot, and $d$ is the forward distance of the robot.

(4)

In this paper, we expect to perturb the gait trajectory as little as possible. The energy consumption penalty $r_{energy}$ is the penalty for gait motion efficiency and energy consumption. The expression we designed is

$$r_{energy}={{\displaystyle \sum }}_{i=1}^{18}{\tau }_i·{\dot{\theta }}_i,$$

(10)

where ${\tau }_i$ is the joint torque, and ${\dot{\theta }}_i$ is the joint angular velocity.

3.3. Termination Condition

Certain termination conditions are set to avoid the policy network falling into local minima and improve the sampling efficiency. When the following condition occurs, the agent will terminate the training and start again from the initial state.

(1)

The robot is involved in a self-collision.

(2)

The pitch, roll, or yaw of the base exceeds the allowed range.

(3)

The base height is less than the set threshold.

(4)

The distance the robot moves exceeds the set threshold.

3.4. Policy Training

In this paper, the augmented random search (ARS) [31] method is used to train a policy to maximize the expected reward return as shown in Equation (6). ARS uses a random parameter search policy optimization technique which is more simplified than the current reinforcement learning algorithm and can improve the sample efficiency. We chose a linear policy so that it could reduce the application cost in the future. Thus, for each ARS optimization step, we deploy 8 rollouts per iteration with a parameter learning rate of 0.015 and parameter exploration noise of 0.05. During the training process, a new rugged ground is generated by a random seed at each episode. The height of the ground varies by a maximum of 75% of the height of the raised leg.

After the training is completed, we can obtain a policy network with fixed parameters. The flow chart of the robot for terrain-adapted walking using the RL-CPG algorithm is shown in Figure 6. First, the coupled rhythmic signals with specified gait parameters, which are the basis for the hexapod robot to move in a regular pattern, are output by the oscillator network. Then, the currently obtained observations of the robot are fed into the policy network to obtain the mapping parameters and bring them into the mapping function. Next, the signal is mapped to the foot-end trajectory using the mapping function. Finally, the IK module calculates the desired angle for each joint and sends them to the joint actuators to move the hexapod robot.

4. Experiments and Results

4.1. Motion Verification

In this experiment, we validate the trained policy network. The height of the robot is initialized to be 0.147 m. We set the positive y-axis direction of the world coordinate frame as the forward direction. The robot is set to walk with a tripod gait, and the strategy network slightly adjusts the mapping parameters in real time. Figure 7 shows the process of the hexapod robot walking on rough ground. From the results, we can see that when a leg lands on the raised ground, the output of the policy network for that leg is large to compensate for the height of the leg landing earlier. The movement of the hexapod robot performs well on rough ground.

During motion, the roll and pitch angles reflect the degree of body sway. Figure 8 records the roll and pitch angles during the movement of the robot using RL-CPG algorithm. The results show that the shaking amplitude of the robot is about 3 degrees, and the performance is smooth. This also indicates that the proposed terrain-adapted gait generator is feasible and capable of making the hexapod robot move in rough terrain.

4.2. Motion Performance Comparison

In order to compare the direct control of robot joint motion using RL baseline algorithm and the gait generation method using RL-CPG algorithm and CPG algorithm, we performed 100 experiments, and each experiment randomly generated rugged terrain. The fixed time step is set to 5000. The motion states of the hexapod robot with these different algorithms are compared under the same terrain, as shown in

Table 2. Statistics of the experimental data of RL-CPG algorithm and traditional CPG algorithm.

	$\mathbf{Avg} \left({\mathit{d}}_{\mathit{y}}\right)$	$\mathbf{Avg} \left({\mathit{d}}_{\mathit{x}}\right)$	$\mathbf{Mean} \left(\left|{\mathit{\phi }}_{\mathit{x}}\right|\right)$	$\mathbf{Mean} \left(\left|{\mathit{\phi }}_{\mathit{y}}\right|\right)$	Died
RL-CPG	2.63	0.41	0.020	0.019	15
CPG	2.15	0.46	0.021	0.020	32
RL	0.96	0.73	0.034	0.035	18

.

We compared the average forward distance, the average offset, the mean of the absolute value of the roll and pitch angle, and the number of early terminations. We expect the robot to walk in the forward direction as much as possible. It is found that the method of using RL baseline to directly control the joint motion of the hexapod robot fails to make the robot learn a regular motion pattern and walk on rough ground under the same training conditions. From the results, it can be seen that the robot using RL-CPG algorithm walks farther in the y direction than that using CPG algorithm, and the offset in the x direction is smaller; that is, the walking efficiency is higher. In addition, the robot using the RL-CPG algorithm terminates significantly fewer times early.

However, the degree of body sway is not obvious from the mean comparison. The absolute values of the body roll angle and pitch angle of the robot at each time step in all experiments were counted, and their distribution was observed as shown in Figure 9. It can be seen that with the RL-CPG algorithm, the robot body has a smaller amplitude range and its motion is more stable.

5. Conclusions

In this paper, a terrain-adaptive gait generation framework with RL-CPG algorithm is proposed to improve robot terrain adaptation. The CPG algorithm reduces the difficulty of applying reinforcement learning to the hexapod robot, and the application of reinforcement learning makes the CPG algorithm more flexible and suitable for walking on rough ground. In addition, the algorithm relies on fewer sensors, so the algorithm has more stable performance and strong versatility. Despite the low deployment cost of the proposed algorithm, the current study only considers rugged terrain, and it is the focus of future work to consider more complex terrain and apply the algorithm to real robots. However, the study of terrain adaptation in this paper is only for rugged ground, and it will be our main work in the future to consider more complex ground situations.