Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion

Abstract

In this paper, a novel reinforcement learning method that enables a humanoid robot to learn bipedal walking using a simple reference motion is proposed. Reinforcement learning has recently emerged as a useful method for robots to learn bipedal walking, but, in many studies, a reference motion is necessary for successful learning, and it is laborious or costly to prepare a reference motion. To overcome this problem, our proposed method uses a simple reference motion consisting of three sine waves and automatically sets the waveform parameters using Bayesian optimization. Thus, the reference motion can easily be prepared with minimal human involvement. Moreover, we introduce two means to facilitate reinforcement learning: (1) we combine reinforcement learning with inverse kinematics (IK), and (2) we use the reference motion as a bias for the action determined via reinforcement learning, rather than as an imitation target. Through numerical experiments, we show that our proposed method enables bipedal walking to be learned based on a small number of samples. Furthermore, we conduct a zero-shot sim-to-real transfer experiment using a domain randomization method and demonstrate that a real humanoid robot, KHR-3HV, can walk with the controller acquired using the proposed method.

1. Introduction

Humanoid robots are capable of human-like movements and can therefore be useful in assisting and/or replacing humans in environments designed for humans. However, enabling humanoid robots to walk on two legs is a challenging task, due to the complex dynamics. There has been a long history of research on the design of bipedal gait controllers using mathematical models and dynamics, such as the inverted pendulum [1,2] and zero-moment point (ZMP) [3,4], but these approaches require sophisticated controller designs involving accurate mathematical models of the robot and its environment.

Recently, methods using reinforcement learning have been extensively studied (e.g., [5,6,7,8,9]) as a promising alternative approach. The reinforcement-learning-based methods are useful because they do not depend on a sophisticated controller designed by humans. The controller can be automatically acquired through trial and error.

However, learning bipedal walking from scratch by reinforcement learning (e.g., [10,11]) requires much time for the trial-and-error learning process and often results in unnatural gait patterns. In many studies, this problem has been addressed by providing the robot with a reference motion (e.g., [5,7,12,13]), which is an example motion that helps the robot to quickly learn a natural gait pattern.

Although the use of a reference motion enables the quick learning of a natural gait pattern, preparing the reference motion can be laborious or costly. For example, in some studies [7,12], expert controllers were used to generate a reference motion, but the preparation of an expert controller requires significant human effort. In other studies [5,13], reference motions were created by capturing human motion, but this approach requires costly motion capture equipment.

Therefore, it is highly desirable in the reinforcement learning of bipedal walking to use a simple reference motion that can easily be prepared. Since the purpose of using a reference motion is to help a robot to learn to walk, the reference motion does not need to be so elaborate that the robot can walk using the reference motion alone (if such an elaborate reference motion is available, there are other useful approaches, such as tracking control methods that guarantee prescribed performance [14,15]). However, little research has been conducted on the use of simple reference motions. An exception is the study by Wu et al. [16] on a robot with bird-like legs (see Section 2 for details), but their method is not readily applicable to a humanoid robot.

To address this issue, in this paper, we propose a novel reinforcement learning method that enables a humanoid robot to learn bipedal walking using only a simple reference motion. This is the first contribution of this paper. The reference motion that we use is a simple motion consisting of three sine waves, and the parameters that determine their waveforms are automatically set using Bayesian optimization. Thus, our reference motion can be prepared with minimal human involvement. To enable quick learning while using the simple reference motion, we introduce the following two means. First, we use inverse kinematics (IK). As will be described in Section 3, our simple reference motion only specifies the robot’s positions in relation to the torso. Thus, we use IK to identify all the angles of the leg joints that realize the specified sole positions. We expect that the reference motion expressed by the joint angles will provide the robot with more direct clues in learning how to walk. Second, we use the reference motion as a bias for the action determined by reinforcement learning, rather than as an imitation target. In many previous studies [5,7,12,13], the robot is given a reward function that encourages it to imitate a given reference motion. Although this approach is suitable when the goal of learning is to imitate the reference motion, such a reward function disturbs learning when the goal is not imitation itself. In the present study, by using the reference motion as a bias rather than as the imitation target, we allow our robot to focus on the true objective, i.e., learning bipedal walking. We conduct numerical experiments and show that our proposed method enables bipedal walking to be learned using a small number of samples.

Furthermore, we apply our method to a real robot. This is the second contribution of this paper. We use a small and inexpensive humanoid robot, KHR-3HV (ver. 2) manufactured by Kondo Kagaku Co., Ltd., Tokyo, Japan. We show that, through a sim-to-real approach using domain randomization, we can enable the real robot to walk in zero shots. In other words, the real robot can walk with the controller acquired using our proposed method in simulations, without any additional learning. To the best of our knowledge, this is the first successful example of a reinforcement-learning-based gait controller implemented on a real KHR-3HV robot.

The remainder of this paper is structured as follows. In Section 2, we discuss related work. In Section 3, we describe the proposed method. In Section 4, we explain the simulation experiments. In Section 5, we explain the experiments on a real robot. In Section 6, we state the conclusions.

2. Related Work

Similar to the present study, Wu et al. [16] proposed a reinforcement learning method that uses only a simple reference motion. In their study, the reference motion was represented by only two sine waves. They used a Cassie robot and showed that bipedal walking was successfully learned in a simulator. However, as described in Section 4, their method did not succeed in learning bipedal walking when applied on our humanoid robot. This may be due to the difference in the shape of the robots; the Cassie robot has bird-like legs and can walk without moving its hips to the left or right, whereas our robot is humanoid and needs to move its hips to the left or right while walking. In Section 4, we show that our proposed method is superior for our humanoid robot. Another difference is that our proposed method automatically searches for the sine wave parameters, whereas these must be provided by humans when using the method of Wu et al. Additionally, we apply our method to a real robot, whereas their study was conducted with a simulator only.

Various methods have been studied to assist the reinforcement learning of bipedal walking. Other than the reference-motion-based methods described in Section 1, the following methods have also been studied. Shi et al. [17] proposed a reference-free reinforcement learning method in which the robot is helped by the application of external forces. However, it is necessary for humans to set up a well-designed learning curriculum, such as where to initially apply and then how to gradually reduce the external forces so that the robot can eventually walk without them. Singh et al. [18] used a well-designed hand-crafted reward function and realized the learning of a natural bipedal walking pattern without using a reference motion. The reward function encourages the robot to realize, e.g., desirable foot–ground reaction forces depending on the phase of a gait cycle. However, it is necessary for humans to create such a reward function.

The approach of using IK with reinforcement learning has been adopted in several existing studies; for example, Kaymak et al. [8] proposed a gait controller that combined IK and reinforcement learning and applied it to DARwIn-OP, a small humanoid robot. However, in their study, some patterns of toe movements with parameters such as the stride length had to be prepared in advance, and the reinforcement learning method only optimized these parameters within a limited range. IK was used to convert the movement patterns into joint angles. In contrast, our proposed method optimizes the angle of every joint. Tutsoy et al. [19] proposed a method that involved learning to balance a NAO robot, using an approach similar to [8], where low-dimensional actions optimized by reinforcement learning are converted into joint angles using IK.

In the present study, the KHR-3HV robot (ver. 2) is used as the target robot. This robot is an inexpensive (less than USD 100), small humanoid robot and has been used for studies on various topics, including gait generation using an evolutionary algorithm [20], balancing on inclined surfaces [21], energy efficiency analysis for bipedal walking [22], and multimodal locomotion [23]. However, to the best of our knowledge, our study is the first in which a bipedal gait controller for a real robot is acquired using reinforcement learning. Compared to the Cassie robot, which has been extensively studied for bipedal walking by reinforcement learning [12,16,24,25], enabling the KHR-3HV robot to learn to walk might be more difficult. One reason is that, as mentioned above, Cassie has bird-like legs and the KHR-3HV is humanoid. Another reason is that Cassie’s servo motors are concentrated at the top of the legs, making the moments of the legs small, whereas the KHR-3HV’s servo motors are located inside the legs, making the leg moments large. We emphasize, however, that discussions of the superiority or inferiority of robots are beyond the scope of this paper.

3. Proposed Method

An overview of the proposed method is shown in Figure 1. In the proposed method, a neural network is trained using reinforcement learning. The environment of the robot is assumed to be a Markov decision process (MDP), which is defined by a 5-tuple $\{S,A,\mathcal{T},\mathcal{R},\gamma \}$, where S is the state space, A is the action space, $\mathcal{T}$ is the state transition function, $\mathcal{R}$ is the reward function, and $\gamma \in [0,1)$ is the discount factor. The details are explained in what follows.

Figure 2 shows the structure of the left leg of the KHR-3HV robot, and

Table 1. DH parameters of the left leg.

Joint	${\mathit{a}}_{\mathit{i}}$ [mm]	${\mathit{\alpha }}_{\mathit{i}}$ [rad]	${\mathit{d}}_{\mathit{i}}$ [mm]	${\mathit{\phi }}_{\mathit{i}}$ [rad]
1	0	$\frac{\pi }{2}$	23	${\phi }_1$ *
2	41	$-\frac{\pi }{2}$	0	${\phi }_2-\frac{\pi }{2}$
3	65	0	0	${\phi }_3$
4	65	0	0	${\phi }_4$
5	41	$\frac{\pi }{2}$	0	${\phi }_5$
6	25	0	0	${\phi }_6$

* In the present study, ${\phi }_1$ is fixed at 0 [rad]. Only the angles ${\phi }_2$ to ${\phi }_6$ are controlled.

shows its Denavit–Hartenberg (DH) parameters [26]. We use these parameters to solve the IK problem (Section 3.5). We note that these parameters are readily available when reinforcement learning is performed on a physics simulator, because such parameters are required by a physics simulator to simulate the robot’s motion.

3.1. Reference Motion

In the proposed method, a humanoid robot acquires a bipedal walking controller via reinforcement learning using a simple reference motion.

To keep the reference motion simple, we consider only three variables: the heights of the left and right soles and the lateral displacement of the soles relative to the torso. As the reference motion, we use the motion of walking in place, rather than walking forward. In the reference motion, the left and right soles are alternately raised; the torso is displaced in the right direction when the left sole is raised, and vice versa. It is not necessary for the robot to actually walk in place without falling down when controlled by the reference motion alone.

The reference motion is defined by three equations as

$$\begin{array}{ccc}\hfill y_t^{\mathrm{ref}}& =& \mathrm{sign}(sin\left(2\pi \frac{t}{T}\right))min\left(\right|wsin\left(2\pi \frac{t}{T}\right)|+\Delta w,w),\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill z_t^{\mathrm{ref},\mathrm{left}}& =& max(0,hsin\left(2\pi \frac{t}{T}\right)-\Delta h),\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill z_t^{\mathrm{ref},\mathrm{right}}& =& max(0,hsin(2\pi \frac{t}{T}+\pi )-\Delta h),\hfill \end{array}$$

(3)

where $t(=0,1,2,\dots )$ is the time step after an episode has started and T is a given period of the reference motion. An episode starts with the robot in the initial posture shown in Figure 1 and ends either when the robot falls down or when a certain number of steps has elapsed (in this study, we set it to 500).

Figure 3 shows example waveforms of these equations. The variable $y_t^{\mathrm{ref}}$ in Equation (1) is the lateral displacement of the soles of both feet relative to the torso. Equation (1) represents the movement of the torso to the left and right in accordance with the movement of the soles, as depicted in Figure 3. The variable $z_t^{\mathrm{ref},\mathrm{left}}$ in Equation (2) is the height at which the sole of the left foot is raised. Equation (2) represents the motion of raising the left foot during the first half of each walking cycle (Phase 3 in Figure 3). Similarly, $z_t^{\mathrm{ref},\mathrm{right}}$ in Equation (3) is the height at which the sole of the right foot is raised, and Equation (3) represents the motion of raising the right foot in the second half of each walking cycle (Phase 7 in Figure 3).

Equation (1) has two parameters, w and $\Delta w$. The parameter w determines the maximum amount of lateral displacement. The larger the fraction $\frac{\Delta w}{w}$, the larger the proportion of time in each walking cycle during which the lateral displacement is at its maximum value. In addition, there are two parameters, h and $\Delta h$, in Equations (2) and (3). The value of $h-\Delta h$ determines the maximum height of the soles in the reference motion (see Phase 3 in Figure 3). The larger the fraction $\frac{\Delta h}{h}$, the larger the proportion of time spent in the double support phase (i.e., the phase in which the two soles are touching the ground) in each walking cycle. This is because Equations (2) and (3) indicate that the heights of the soles, $z_t^{\mathrm{ref},\mathrm{left}}$ and $z_t^{\mathrm{ref},\mathrm{right}}$, are both zero when time t satisfies $sin\left(2\pi \frac{t}{T}\right)\le \frac{\Delta h}{h}$ and $sin(2\pi \frac{t}{T}+\pi )\le \frac{\Delta h}{h}$.

These four parameters, $w,\Delta w,h,$ and $\Delta h$, are automatically optimized using Bayesian optimization during learning and thus do not need to be set by humans. Details of the Bayesian optimization are explained in Section 3.6.

As will be described in Section 3.5, we calculate all the angles of the robot’s leg joints that realize sole positions specified by $y_t^{\mathrm{ref}},z_t^{\mathrm{ref},\mathrm{left}},$ and $z_t^{\mathrm{ref},\mathrm{right}}$. We use IK to calculate such angles, and we denote this by

$$a_t^{\mathrm{IK}}={\pi }_{\mathrm{IK}}(y_t^{\mathrm{ref}},z_t^{\mathrm{ref},\mathrm{left}},z_t^{\mathrm{ref},\mathrm{right}}),$$

(4)

where $a_t^{\mathrm{IK}}$ denotes the angles of all the joints. We call $a_t^{\mathrm{IK}}$ a reference bias because we use it as a bias for the reinforcement learning output (Equation (5)). $a_t^{\mathrm{IK}}$ is an N-dimensional vector when there are N servo motors to be controlled. In our robot, $N=10$ because we control 10 servo motors equipped within the two legs of the robot.

3.2. Action and Target Joint Angle

At each time t, we obtain output $a_t^{\mathrm{RL}}$ of the reinforcement learning and add it to the reference bias $a_t^{\mathrm{IK}}$ calculated from the reference motion as in the previous section. We let the result be the target joint angles $u_t^{\mathrm{target}}$. Specifically,

$$u_t^{\mathrm{target}}=a_t^{\mathrm{IK}}+a_t^{\mathrm{RL}}$$

(5)

(Figure 1), where $a_t^{\mathrm{RL}}$ is also an N-dimensional vector like $a_t^{\mathrm{IK}}$. We assume that each joint of the robot has a servo motor that changes its angle by PID control so that the given target angle $u_t^{\mathrm{target}}$ is approached (Section 4 and Section 5).

Determining $u_t^{\mathrm{target}}$ in this way means that all the angles of the target servo motors can be directly optimized by reinforcement learning. The role of $a_t^{\mathrm{IK}}$ is simply to provide a bias for the reinforcement learning output. This is in contrast to the previous studies [8,19] mentioned in Section 2, in which IK was used to calculate the target angles for all servo motors from low-dimensional parameters or actions optimized by reinforcement learning.

We obtain the output of the reinforcement learning, $a_t^{\mathrm{RL}}$, using the following procedure. First, when the current state $s_t$ (see Section 3.3 for details) is input to the neural network, the action $a_t$ is determined according to a probabilistic policy ${\pi }_{\theta }\left(a_t\right|s_t)$ represented by the neural network. Here, $\theta$ denotes the parameters of the neural network. The action $a_t$ is an N-dimensional vector, and each element of $a_t$ takes a value between $-1$ and 1.

Next, $a_t$ is passed through a low-pass filter (LPF) to improve the smoothness [27]. Specifically, $a_t^{\prime }$ is calculated from $a_t$ as

$$a_t^{\prime }=b{a}_t+(1-b)a_{t-1}^{\prime },$$

(6)

where $b(0<b\le 1)$ is a smoothing factor, and $a_{t-1}^{\prime }$ is the smoothed value one step ago. We set $a_{-1}^{\prime }$, which is a variable that is necessary to calculate Equation (6) when $t=0$, to 0. We set $b=0.2$ in all the experiments in this study. The smoothed action $a_t^{\prime }$ is also an N-dimensional vector, and each element of it takes a value between −1 and 1.

Finally, $a_t^{\prime }$ is multiplied by a constant k to obtain the output of the reinforcement learning, $a_t^{\mathrm{RL}}$. Specifically,

$$a_t^{\mathrm{RL}}=k{a}_t^{\prime },$$

where k is a parameter that determines the range of possible values of $a_t^{\mathrm{RL}}$, i.e., the range of search by the reinforcement learning (Figure 1). In our experiments, we set $k=0.5{\omega }_{\max }dt$, where ${\omega }_{\max }$ is the maximum angular velocity of the servo motors and $dt$ is the time interval of one step.

3.3. State

The state $s_t$ consists of the observations obtained from the environment in the most recent five steps $\{o_t,o_{t-1},o_{t-2},o_{t-3},o_{t-4}\}$ and also of a phase vector (Figure 1).

The observation $o_t$ at each time step is a 16-dimensional vector consisting of the 10 joint angles of the robot’s legs and the 3-dimensional linear acceleration and 3-dimensional angular velocity of the robot’s torso. Thus, the five observations $\{o_t,o_{t-1},o_{t-2},o_{t-3},o_{t-4}\}$ are represented by an 80-dimensional vector. The reason for using the past observations from $o_{t-1}$ to $o_{t-4}$ is to provide the policy with more information to infer the dynamics of the environment [28].

The phase vector is a 2-dimensional vector consisting of $\{sin\left(\frac{2\pi t}{T}\right),cos\left(\frac{2\pi t}{T}\right)\}$, where T is the period of the walking motion to be learned and t is the time step since the start of the episode. It is known that learning may be facilitated by providing such a phase vector, since walking is a cyclic movement [29].

In total, the state $s_t$ is an 82-dimensional vector. It is input to the neural network (Figure 1).

3.4. Reward

The task reward $r^{\mathrm{T}}$, which evaluates how well the walking task is achieved, is defined as follows:

$$\begin{array}{c}{r}^{\mathrm{T}}=w_1{r}^{\mathrm{o}}+w_2{r}^{\mathrm{v}}+w_3{r}^{\mathrm{s}}+w_4{r}^{\mathrm{a}}+r^{\mathrm{f}},\hfill \\ w_1+w_2+w_3+w_4=1,\hfill \end{array}$$

(7)

where $w_1,w_2,w_3,$ and $w_4$ are the weights that balance each term. We did not assign a weight for $r^{\mathrm{f}}$ because it is a penalty term that takes a negative value, whereas the other terms take positive values. The details of each term are described below. The values of $w_1$ to $w_4$ and of the parameters within each term were determined through numerical experiments.

In many other studies that use a reference motion, an imitation reward $r^{\mathrm{I}}$ that evaluates the similarity between the reference motion and the robot’s motion is introduced, and the sum of the task reward $r^{\mathrm{T}}$ and the imitation reward $r^{\mathrm{I}}$ is maximized. In our proposed method, however, only the task reward is considered so that the reinforcement learning can focus on task accomplishment rather than on the imitation of the reference motion. The reference motion is only added to the output of the reinforcement learning as a bias (Equation (5)). In our experiments (Section 4.2 and Section 4.3), we compare our proposed method with two other methods that use imitation rewards.

3.4.1. Orientation Reward $r^{\mathrm{o}}$

To encourage the robot to maintain an orientation consistent with the direction in which it should walk, we use

$$r^{\mathrm{o}}=exp\left[-75({\varphi }^2+{\theta }^2)\right],$$

(8)

where $\varphi ,\theta$ are the roll and pitch angles of the robot’s torso. Thus, Equation (8) encourages the values of $\varphi$ and $\theta$ to become zero.

3.4.2. Velocity Reward $r^{\mathrm{v}}$

To encourage the robot’s speed to approach the target speed, we define the velocity reward, as follows, based on [10]:

$$\begin{array}{ccc}\hfill r^{\mathrm{v}}& =& \left\{\begin{array}{cc}exp(-l)\hfill & \mathrm{if}\mathrm{sign}\left(v_x^{\mathrm{I}}\right)>0\mathrm{and}{r}^{\mathrm{o}}>0.8,\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \\ \hfill l& =& K_x\left[{(v_x^{\mathrm{cmd}}-v_x^{\mathrm{b}})}^2+{(v_x^{\mathrm{cmd}}-v_x^{\mathrm{I}})}^2\right]\hfill \\ & & +K_y\left[{(v_y^{\mathrm{cmd}}-v_y^{\mathrm{b}})}^2+{(v_y^{\mathrm{cmd}}-v_y^{\mathrm{I}})}^2\right]\hfill \\ & & +K_{\omega }{({\omega }_z^{\mathrm{cmd}}-{\omega }_z)}^2,\hfill \end{array}$$

(9)

where $v_x^{\mathrm{b}}$ and $v_y^{\mathrm{b}}$ are the linear velocities of the robot’s torso along the x and y axes in the world coordinate system, respectively. $v_x^{\mathrm{I}}$ and $v_y^{\mathrm{I}}$ are the linear velocities of the torso along the x and y axes in the local coordinate system of the torso, respectively. ${\omega }_z$ is the angular velocity around the z axis in the world coordinate system. $v_x^{\mathrm{cmd}}$, $v_y^{\mathrm{cmd}}$, and ${\omega }_z^{\mathrm{cmd}}$ are the corresponding target values.

We note that in Equation (9), we have two types of torso velocities, i.e., $v_x^{\mathrm{b}}$ and $v_y^{\mathrm{b}}$, which are represented in the world coordinate system, as well as $v_x^{\mathrm{I}}$ and $v_y^{\mathrm{I}}$, which are represented in the local coordinate system of the torso. In Equation (9), the terms involving $v_x^{\mathrm{b}}$ and $v_y^{\mathrm{b}}$ encourage the robot to move in the target direction in the world coordinate system, and the terms involving $v_x^{\mathrm{I}}$ and $v_y^{\mathrm{I}}$ encourage the robot to move forward in the local coordinate system. Without the former, the robot is not penalized for moving in a different direction. Without the latter, the robot is not penalized for walking in a strange manner, such as a sideways walk [10].

In all the experiments in this study, we set $v_x^{\mathrm{cmd}}=0.15$ [m/s], $v_y^{\mathrm{cmd}}=0$ [m/s], and ${\omega }_z^{\mathrm{cmd}}=0$ [rad/s]. The value of 0.15 m/s for $v_x^{\mathrm{cmd}}$ is faster than the maximum speed that the robot can physically achieve but was used to encourage the robot to walk as quickly as possible [30]. $K_x,K_y,$ and $K_{\omega }$ are weight parameters. In all the experiments in this study, we set $K_x=15$, $K_y=50$, and $K_{\omega }=25$.

3.4.3. Symmetry Reward $r^{\mathrm{s}}$

In order for the robot to keep its upper body upright, the left and right legs must be tilted approximately equally in the coronal plane, as depicted in Phase 2 in Figure 3. To encourage such tilts, we use

$$r^{\mathrm{s}}=exp\left[-\frac{{\sum }_{i=1}^2{(q_i^{\mathrm{left}}-q_i^{\mathrm{right}})}^2}{0.01}\right],$$

where $q_1^{\mathrm{left}}$ and $q_1^{\mathrm{right}}$ are the joint angles of the left and right hips ($q_1^{\mathrm{left}}$ is equal to ${\phi }_2$ in Figure 2), respectively, and $q_2^{\mathrm{left}}$ and $q_2^{\mathrm{right}}$ are the joint angles of the left and right ankles ($q_2^{\mathrm{left}}$ is equal to ${\phi }_6$ in Figure 2), respectively.

3.4.4. Survival Reward $r^{\mathrm{a}}$

To encourage the robot not to fall down, we use a survival reward $r^{\mathrm{a}}$. It is set to 1 when the robot has not fallen over and to 0 when the robot falls over. A fall of the robot is detected when the height of the torso is less than 170 mm, where the torso’s height in the initial posture (Figure 1) is 318 mm.

If the robot falls over, the episode ends, and the robot returns to the initial posture shown in Figure 1. The initial posture is a posture where the legs are slightly bent so that the robot’s center of gravity is 2 cm lower than that of the upright posture. The purpose of using this posture is to make the robot less likely to fall down on its first step. The displacements specified by the reference motion ($y_t^{\mathrm{ref}},z_t^{\mathrm{ref},\mathrm{left}},$ and $z_t^{\mathrm{ref},\mathrm{right}}$ in Equations (1)–(3)) are measured with regard to this initial posture.

3.4.5. Torque Penalty $r^{\mathrm{f}}$

To encourage the smooth motion of the robot by punishing excessive movements of its joints, we use

$$r^{\mathrm{f}}=-0.01\sum _{i=1}^N\left|f_i\right|,$$

where $f_i$ is the torque produced by the servo motor of the i-th joint.

3.5. Inverse Kinematics

As described in Section 3.1, at each time t, the positions of the left and right soles with respect to the torso are specified by the reference motion. In our reference motion, the soles are not rotated; they remain in the rotation of the initial posture. Thus, at each time t, the pose (i.e., both position and rotation) of each sole is specified.

For each of the left and right legs, we find the joint angles that place the sole in the specified pose (Equation (4)). This is an IK problem, and there are various types of methods to solve it. Here, we briefly explain the solution method that we used for the left leg. The solution for the right leg was obtained in the same way.

Let p be the specified pose of the left sole and q be the vector of the joint angles to be controlled, which are the angles from ${\phi }_2$ to ${\phi }_6$ shown in Figure 2 and Table 1. Given the DH parameters in Table 1, the forward kinematics function f that maps q to p is determined.

To find q for the given p, we numerically solved the following minimization problem

$$minE\left(q\right),$$

(10)

where $E\left(q\right)$ is the error function defined as

$$E\left(q\right)=\frac{1}{2}{e}^{\mathrm{T}}{W}_{e}e,$$

in which $e=p-f\left(q\right)$.

We used the Gauss–Newton method to solve Equation (10). Specifically, we calculated

$$\begin{array}{ccc}\hfill q_{k+1}& =& q_k+\beta {\left(J{\left(q_k\right)}^{\mathrm{T}}{W}_{e}J\left(q_k\right)\right)}^{-1}{g}_k,\hfill \\ \hfill g_k& =& J{\left(q_k\right)}^{\mathrm{T}}{W}_e{e}_k\hfill \end{array}$$

iteratively with $k=0,1,2,\dots$. Here, $J\left(q_k\right)$ is the Jacobi matrix, $q_0$ is the initial joint angle that we set to the zero vector, and $\beta$ is a step length that we set to $0.5$. The iteration continues until the value for the error function $E\left(q_k\right)$ becomes less than a small constant $ϵ$ or until k becomes 30. We set $ϵ={10}^{-6}$.

3.6. Optimizing the Parameters of the Reference Motion

To determine the values of the parameters $w,\Delta w,h,$ and $\Delta h$ of the reference motion, we use Bayesian optimization [31], which is a method that efficiently searches for the maximum or minimum value of a function whose shape is unknown. We used the TPE algorithm [32] implemented in Optuna [33].

We optimize $w,\Delta w,h,$ and $\Delta h$ so as to maximize the following function

$$\begin{array}{c}\hfill L(w,\Delta w,h,\Delta h)=\frac{1}{H}\sum _{e=1}^H\left(\frac{1}{T_e}\sum _{t=0}^{T_e}{r}_{e,t}\right),\end{array}$$

(11)

where $r_{e,t}$ is the value of the reward $r^{\mathrm{T}}$ in Equation (7) obtained at step t of episode e, $T_e$ is the number of time steps of episode e, and $\frac{1}{H}{\sum }_{e=1}^H(·)$ takes the average over the most recent H episodes, in which $w,\Delta w,h,$ and $\Delta h$ were set to the values specified on the left-hand side of this equation.

Before the beginning of each episode, the TPE algorithm fixes the parameters $w,\Delta w,h,$ and $\Delta h$ at specific values. After the end of the episode, the rewards obtained during the episode are used to update $L(w,\Delta w,h,\Delta h)$ through Equation (11). Using the updated $L(w,\Delta w,h,\Delta h)$, the algorithm sets the parameters to new values for the next episode. This is repeated many times, and, based on the collected values of $L(w,\Delta w,h,\Delta h)$, the algorithm gradually becomes able to estimate the optimal values for $w,\Delta w,h,$ and $\Delta h$.

This Bayesian optimization is performed in parallel with the reinforcement learning process that updates policy ${\pi }_{\theta }$. The changing of the parameters through Bayesian optimization was performed until the 4500-th episode, after which the parameters were fixed at the optimal values found.

3.7. Optimizing ${\pi }_{\theta }$

In this study, we use the Proximal Policy Optimization (PPO) algorithm [34] for reinforcement learning. It has been suggested that PPO, compared with other deep reinforcement learning algorithms, is more suitable for the learning of bipedal gaits (e.g., [35]). PPO improves the policy ${\pi }_{\theta }\left(a_t\right|s_t)$ by using a stochastic gradient ascent method, limiting abrupt changes in the policy when its parameter $\theta$ is updated. The clipped surrogate objective, which is the loss function of PPO, is defined as

$$L^{\mathrm{CLIP}}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t)\right],$$

(12)

where

$$r_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)},$$

${\widehat{A}}_t$ is the estimated advantage at time t, and $ϵ$ is the clip parameter. We refer the reader to the original paper [34] for details.

4. Numerical Experiments

4.1. Simulation Setups

In this study, we used a 3D robot simulator, Webots [36], to perform the simulations required for reinforcement learning and Bayesian optimization. The Webots simulator performed the physical simulations at 50 Hz.

The target robot is a small humanoid robot, KHR-3HV (ver. 2). This robot has 22 degrees of freedom. To simplify the learning of walking, we controlled 10 joints of the legs. The angles of the other joints were fixed. Each joint angle was controlled by the PID controller of a servo motor, and the PID parameters were set to $K_{\mathrm{p}}=18.5$, $K_{\mathrm{I}}=0$, and $K_{\mathrm{D}}=0.009$. Webots has two parameters regarding the joints, which are dampingConstant and staticFriction, and we set them to $0.002$ and $0.025$, respectively.

The values for $u_t^{\mathrm{target}}$, $a_t^{\mathrm{IK}}$, and $a_t^{\mathrm{RL}}$ in Equation (5) were calculated at 25 Hz, i.e., every 40 ms. We set the walking cycle T to 50 steps. Thus, one period of the reference motion was $40\times 50=2000$ ms.

For Bayesian optimization to determine the four parameters of the reference motion, as mentioned in Section 3.6, we used the TPE algorithm [32] implemented in Optuna [33]. We used the algorithm with its default parameter values. We set H to 8. We set the search space of each reference motion parameter to be integers within $10\le w\le 20$ [mm], $0\le \Delta w\le 10$ [mm], $20\le h\le 30$ [mm], and $0\le \Delta h\le 10$ [mm].

To implement the environment of the reinforcement learning, we used OpenAI Gym [37]. To implement the neural network and PPO algorithm, we used Stable-Baselines3 [38].

The neural network was a fully connected four-layer feedforward network consisting of an input layer of 84 neurons, two hidden layers of 64 neurons each, and an output layer of 20 neurons, with $\tanh$ activation functions. The outputs of the neural network were sampled from Gaussian distributions whose means and standard deviations were determined by the activation values of the neurons of the output layer. Sampled values below −1 or above 1 were clipped to −1 or 1, respectively.

Table 2. PPO hyperparameters.

Hyperparameter	Value
Time steps per actor batch	2048
Batch size	128
Optimization epochs	10
Learning rate	$3\times {10}^{-4}$
Discount factor $\gamma$	$0.99$
GAE $\lambda$	$0.95$
Clip parameter	$0.2$
Entropy coefficient	$0.0$

lists the values for the PPO hyperparameters. The parameter $\lambda$ in Table 2 is a parameter that is used to calculate the estimated advantage ${\widehat{A}}_t$ in Equation (12) with the Generalized Advantage Estimation (GAE) method (see [34] for details).

The weights to balance each term of the reward function were set to $w_1=0.1,w_2=0.75,$$w_3=0.1,$ and $w_4=0.05$. These values were empirically determined through several experiments.

We used a personal computer with an Intel Core i7-11700K CPU and an NVIDIA GeForce GTX1650 GPU for the simulation experiments. The simulation took approximately one hour per ${10}^6$ steps.

4.2. Methods for Comparison

To evaluate the proposed method, we compared it with three other methods. Each of these methods is explained below.

4.2.1. RL Only

The first method for comparison is a method that does not use a reference motion and uses only reinforcement learning. In this method, the target angles of the robot’s joints were determined as

$$u_t^{\mathrm{target}}=q_t+a_t^{\mathrm{RL}},$$

(13)

where $q_t$ is the vector of the current joint angles, which is a part of observation $o_t$ (Section 3.3), and $a_t^{\mathrm{RL}}$ is the output of the reinforcement learning, which is calculated in exactly the same way as in the proposed method (Section 3.2).

4.2.2. Imitation Learning 1

As mentioned in Section 2, Wu et al. [16] also proposed a reinforcement learning method using a simple reference motion. They successfully trained a Cassie robot in a simulator, although they did not control a real robot. For comparison, we prepared a similar method. Here, we note that it was impossible to implement exactly the same method as theirs because of differences in the robots (e.g., their method uses information regarding shin springs and pelvis, which is unavailable in our robot). Thus, we replaced their task reward with our $r^{\mathrm{T}}$ in Equation (7) because $r^{\mathrm{T}}$ in Equation (7) worked well for our robot in our proposed method. Additionally, we modified the parameter values in Equations (15) and (16) below to maximize the reward obtained after learning. The method that we prepared is as follows.

First, following their method [16], we used only the heights of the left and right soles for the reference motion. Specifically, $z_t^{\mathrm{ref},\mathrm{left}}$ and $z_t^{\mathrm{ref},\mathrm{right}}$ of Equations (2) and (3) were used, and $y_t^{\mathrm{ref}}$ in Equation (1) was not used. In this method, $z_t^{\mathrm{ref},\mathrm{left}}$ and $z_t^{\mathrm{ref},\mathrm{right}}$ were used as the targets for imitation.

We did not perform Bayesian optimization of h and $\Delta h$. This is because, as will be explained below, we used the reference motion to define an imitation reward term. By not performing Bayesian optimization, we avoided changing the imitation reward term so that we could plot and compare their values in later figures. h and $\Delta h$ were fixed at $h=25$ [mm] and $\Delta h=5$ [mm]. These are the same parameter values that enable our proposed method to successfully learn bipedal gaits, as will be shown in Section 4.3.

Second, the target angles $u_t^{\mathrm{target}}$ of the robot’s joints were determined using Equation (13), without IK. Instead, as in Wu et al. [16], an imitation reward term was added to the reward function so that the reward was higher when the robot’s motion was closer to the reference motion. Specifically, the reward function r was defined as

$$\begin{array}{c}r=w^{\mathrm{I}}{r}^{\mathrm{I}}+w^{\mathrm{T}}{r}^{\mathrm{T}},\hfill \\ w^{\mathrm{I}}+w^{\mathrm{T}}=1,\hfill \end{array}$$

(14)

where $r^{\mathrm{I}}$ is the imitation reward defined below, $r^{\mathrm{T}}$ is the task reward defined in Equation (7), and $w^{\mathrm{I}}$ and $w^{\mathrm{T}}$ are the weights that balance these rewards.

The imitation reward $r^{\mathrm{I}}$ was defined as in [16],

$$\begin{array}{ccc}\hfill r^{{\mathrm{I}}_{*}}& =& exp\left[-\frac{1}{0.{015}^2}\sum _{i=1}^2{(h_{{\mathrm{foot}}_i}^{\mathrm{ref}}-h_{{\mathrm{foot}}_i})}^2\right],\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill r^{\mathrm{I}}& =& \frac{r^{{\mathrm{I}}_{*}}-B_{\mathrm{lower}}}{B_{\mathrm{upper}}-B_{\mathrm{lower}}},\hfill \end{array}$$

(16)

where $h_{{\mathrm{foot}}_1}^{\mathrm{ref}}$ and $h_{{\mathrm{foot}}_2}^{\mathrm{ref}}$ are the heights of the soles in the reference motion and are equal to $z_t^{\mathrm{ref},\mathrm{left}}$ and $z_t^{\mathrm{ref},\mathrm{right}}$ in Equations (2) and (3), respectively. ${h_{\mathrm{foot}}}_1$ and ${h_{\mathrm{foot}}}_2$ are the current heights of the robot’s left and right soles above the ground, respectively. Thus, this imitation reward encourages the robot to change the heights of its soles in the same way as $z_t^{\mathrm{ref},\mathrm{left}}$ and $z_t^{\mathrm{ref},\mathrm{right}}$ in Equations (2) and (3). The parameters in Equation (16) were set to $B_{\mathrm{lower}}=0.4$ and $B_{\mathrm{upper}}=1.0$. These values were empirically determined though several experiments.

4.2.3. Imitation Learning 2

As we will show in the next section, a walking motion could not be learned using Imitation Learning 1, introduced in the previous section. Therefore, we also prepared another imitation-reward-based method that was similar to Imitation Learning 1 but was closer to our proposed method. This method is the same as Imitation Learning 1 except for the following two aspects.

First, regarding the reference motion, we used $y_t^{\mathrm{ref}}$, $z_t^{\mathrm{ref},\mathrm{left}}$, and $z_t^{\mathrm{ref},\mathrm{right}}$ in Equations (1)–(3) instead of using only $z_t^{\mathrm{ref},\mathrm{left}}$ and $z_t^{\mathrm{ref},\mathrm{right}}$. The parameters of the reference motion were set to $w=20$ [mm], $\Delta w=5$ [mm], $h=25$ [mm], and $\Delta h=5$ [mm]. These are the same parameter values that enable our proposed method to successfully learn bipedal gaits, as will be shown in Section 4.3.

Second, we used the same IK as the one used in our proposed method (Section 3.5) so that we could determine the joint angles $a_t^{\mathrm{IK}}$ that realized the reference motion. In this method, $a_t^{\mathrm{IK}}$ was used as the target for imitation. To encourage the robot to imitate $a_t^{\mathrm{IK}}$, the imitation reward term $r^{\mathrm{I}}$ in Equation (14) was defined as

$$r^{\mathrm{I}}=exp\left[-3\sum _{i=1}^{10}{(q_i^{\mathrm{ref}}-q_i)}^2\right],$$

where $q_i^{\mathrm{ref}}$ is the i-th element of $a_t^{\mathrm{IK}}$, i.e., the imitation target angle for the i-th joint, and $q_i$ is the current angle of the same joint. We used the imitation reward term $r^{\mathrm{I}}$, following [7].

4.3. Experimental Results

Figure 4 shows the learning curves of the proposed method (Section 3) and the RL-only method (Section 4.2.1).

In the figure, the result of the proposed method without Bayesian optimization of the reference motion parameters is also plotted. In this case, we set the reference motion parameters to $w=20$ [mm], $\Delta w=5$ [mm], $h=25$ [mm], and $\Delta h=5$ [mm]. These parameter values were prepared for later use in the experiments of Imitation Learning 1 and 2 shown below. They are the parameter values at which the robot walks in place without falling down when controlled solely by the reference motion, and they were prepared through trial-and-error experiments performed by humans.

The proposed method, both with and without Bayesian optimization, resulted in greater rewards than the RL-only method. Figure 5 shows the motions learned by $5\times {10}^6$ steps using the proposed method (with Bayesian optimization) and using the RL-only method. With the proposed method, the robot was able to walk forward, in contrast to the RL-only method. These results show that the proposed method enabled faster learning (i.e., learning based on a smaller number of samples) than the RL-only method.

Figure 6 shows the results of the two methods that use imitation rewards, i.e., Imitation Learning 1 in Section 4.2.2 and Imitation Learning 2 in Section 4.2.3. The reference motion parameters were fixed at $w=20$ [mm], $\Delta w=5$ [mm], $h=25$ [mm], and $\Delta h=5$ [mm], which are the values that allowed learning bipedal walking using the proposed method, as shown in the video in the Supplementary Materials. Figure 6 shows the results in the following four conditions:

• $w^{\mathrm{I}}=0.2,w^{\mathrm{T}}=0.8$;
• $w^{\mathrm{I}}=0.4,w^{\mathrm{T}}=0.6$;
• $w^{\mathrm{I}}=0.6,w^{\mathrm{T}}=0.4$;
• $w^{\mathrm{I}}=0.8,w^{\mathrm{T}}=0.2$.

Note that the case of $w^{\mathrm{I}}=0.0$ and $w^{\mathrm{T}}=1.0$ was excluded from the experimental conditions above because it was the same as the RL-only condition. The case of $w^{\mathrm{I}}=1.0$ and $w^{\mathrm{T}}=0.0$ was also excluded because, in this case, the learning was based only on the imitation reward term and the optimal policy was to simply imitate the reference motion that allowed the robot to walk in place.

Figure 6a shows the results of Imitation Learning 1 (Section 4.2.2). It can be seen that the task reward, shown on the right side of the figure, reached a maximum when $w^{\mathrm{I}}=0.2$ and $w^{\mathrm{T}}=0.8$. However, the task reward was lower than that of the proposed method shown in Figure 4. The experiment was conducted up to $1\times {10}^7$ steps, twice as many as in Figure 4, but the robot could not walk, as shown in the video in the Supplementary Materials.

Figure 6b shows the results of Imitation Learning 2 (Section 4.2.3). It can be seen that, as with Imitation Learning 1, the task reward, shown on the right side, reached a maximum when $w^{\mathrm{I}}=0.2$ and $w^{\mathrm{T}}=0.8$. Although the task reward was higher than that shown in Figure 6a, it was still lower than that of the proposed method shown in Figure 4. The robot could not walk after $1\times {10}^7$ steps, as shown in the video in the Supplementary Materials.

These results indicate that the learning of bipedal walking occurred faster when using the proposed method rather than the imitation-reward-based methods. There are many parameters other than $w^{\mathrm{I}}$ and $w^{\mathrm{T}}$ in Imitation Learning 1 and 2, and it was not practical to examine all cases, but, at least in our experiments, bipedal walking was not learned when using these methods.

The reason for this result may be that, since the imitation learning methods maximize a reward function consisting of two terms, i.e., the imitation reward $r^{\mathrm{I}}$ and the task reward $r^{\mathrm{T}}$, it is more difficult to find the optimal solution than with our proposed method, in which only the task reward $r^{\mathrm{T}}$ is maximized.

Comparing the results of Imitation Learning 1 (Figure 6a) and Imitation Learning 2 (Figure 6b) with $w^{\mathrm{I}}=0.2$ and $w^{\mathrm{T}}=0.8$, we can see that the task reward was higher in Imitation Learning 2. This result is expected because Imitation Learning 2 has more information than Imitation Learning 1 due to the inclusion of lateral movements in the reference motion and also the use of IK. We can also see that the task reward was higher for Imitation Learning 2 (Figure 6b right) than the RL-only method (Figure 4), suggesting that the reference motion or IK had at least some benefit.

Table 3. Walking performance of the learned controllers.

Method	Walking Distance in 750 Steps (i.e., 30 s) [cm]	Speed [cm/s]
Proposed method	$53.8\pm 6.9$	$1.79\pm 0.23$
Proposed method w/o Bayesian optimization	$33.7\pm 2.0$	$1.12\pm 0.07$
RL only	-	-
Imitation Learning 1	-	-
Imitation Learning 2	-	-

shows the performance of the controllers learned with each method. The proposed method with Bayesian optimization resulted in faster walking than the method without Bayesian optimization. Using the RL-only and Imitation Learning 1 and 2 methods, the robot could not move forward, although it did not fall down.

In Figure 4, comparing the cases with and without Bayesian optimization, we can see that the rewards were almost the same, but the rewards were slightly higher in the case with Bayesian optimization. As seen in Table 3, the performance was also higher in the case with Bayesian optimization. In the case with Bayesian optimization, the parameter values of $w,\Delta w,h,$ and $\Delta h$, chosen as a result of the optimization, were values that would cause the robot to fall down if the robot was controlled using only the reference motion. Since it is difficult for humans to set such values in advance, it was not only found that Bayesian optimization was useful in automatically setting the parameters but also that it could sometimes set better values than those set by humans.

5. Real-Robot Experiments

5.1. Real-Robot System

Figure 7 shows the overview of the real-robot system.

The humanoid robot KHR-3HV (ver. 2) used in this study has 22 servo motors. Each servo motor has an encoder inside to measure the rotation angle. Each servo motor was wired to a microcomputer, Teensy 4.0, via an ICS conversion board (Product No. 03121, Kondo Kagaku Co., Ltd., Tokyo, Japan). Teensy 4.0 acquired the data of every joint angle at approximately 250 Hz.

An Inertial Measurement Unit (IMU), BMI088, was attached inside the backpack on the back of the robot to measure the 3-dimensional accelerations and 3-dimensional angular velocities of the torso. The IMU was wired to a microcomputer, a Raspberry Pi 4, which acquired the data at approximately 400 Hz using the I2C protocol. A one-dimensional Kalman filter algorithm [39] was used to reduce noise in the acquired data.

Teensy 4.0 was connected to a host PC via a USB cable, and the host PC received the data of the current joint angles at 250 Hz. The Raspberry Pi 4 was connected to the host PC via a LAN cable, and the host PC received the IMU data at 400 Hz. The host PC used the acquired data to create the input to the neural network. Since the neural network required the input at 25 Hz (Section 4.1), the measurement frequencies of 250 Hz and 400 Hz were sufficiently high. Every time the host PC created the input data, it used the most recently acquired data.

The host PC was equipped with an Intel Core i7-6700K CPU and no GPU. The neural network was not trained in this real-robot experiment; the action $a_t^{\mathrm{RL}}$ was determined using the neural network that had already been trained with our proposed method in the simulation environment. The target angles, $u_t^{\mathrm{target}}$ in Equation (5), were sent to the servo motors via Teensy 4.0.

All communications between the host PC, Teensy 4.0, and Raspberry Pi 4 were performed using a Robot Operating System (ROS) [40].

5.2. Domain Randomization

To cope with the gap between the simulation and real environments, we performed domain randomization [41] during learning in the simulation environment. Specifically, random external forces and noises were added during simulation using the parameters shown in

Table 4. Domain randomization parameters.

Parameter	Value
Applied force $\left[\mathrm{N}\right]$	$U(0,3)$
Direction of the applied force $\left[\mathrm{deg}\right]$	$\{0,90,180,270\}$
Interval between force applications $\left[\mathrm{ms}\right]$	$U(50,90)$
Motor noise $\left[\mathrm{rad}\right]$	$U(0,0.0087)$
Motor offset $\left[\mathrm{rad}\right]$	$U(-0.0175,0.0175)$
Gyro noise $[\mathrm{rad}/\mathrm{s}]$	$\mathcal{N}(0,0.005)$
Gyro offset $[\mathrm{rad}/\mathrm{s}]$	$U(-0.0175,0.0175)$
Accelerometer noise (x and y axis) $[\mathrm{m}/{\mathrm{s}}^2]$	$\mathcal{N}(0,0.08)$
Accelerometer noise (z axis) $[\mathrm{m}/{\mathrm{s}}^2]$	$\mathcal{N}(0,0.12)$
Accelerometer offset $[\mathrm{m}/{\mathrm{s}}^2]$	$U(-0.29,0.29)$

. For example, a random external force whose strength was sampled from a uniform distribution, ranging from 0 to 3 [N], was applied to the robot’s center of mass from a direction randomly sampled from the four directions along the x- or y-axis in the x–y plane. The duration of each force application was fixed at 50 ms (not shown in Table 4). The motor noise and offset in Table 4 are the noise and offset that occurred when the rotation angle of each servo motor was measured. The other noises and offsets in Table 4 are those that occurred when the angular velocities and accelerations were measured by the IMU.

The effectiveness of domain randomization has been verified in many papers [10,13,24,27], and many parameters, such as robot friction, action delays, and motor torques, have been randomized. However, since excessive randomization leads to increased learning times and conservative policies, we performed randomization for as few parameters as possible in this study.

The parameter values regarding the external force were determined empirically through several experiments. The values for the motor noise, motor offset, and IMU noises were determined through hardware experiments in which we measured actual values. The IMU offsets were determined based on the data sheet [42].

5.3. Experimental Results

The policy learned in the simulation environment using the proposed method was used to control the real robot, and the robot successfully walked in zero shots, i.e., with no additional learning. The robot’s gait is shown in Figure 8 and in the video in the Supplementary Materials.

The walking speed of the real robot was 1.9 cm/s, which is almost the same as the speed of $1.79\pm 0.23$ cm/s observed in the simulation (Table 3). The stride was 3.8 cm. The single support phase (SSP), i.e., the phase in which only one sole was touching the ground, took 1.92 s of the whole 2.00 s cycle. It was reasonable for the robot to have such a long SSP in this experiment because the target walking speed $v_x^{\mathrm{cmd}}$ in the velocity reward (Section 3.4.2) was set to a large value (0.15 m/s), and the robot learned to walk as quickly as possible.

6. Conclusions

In this study, we proposed a reinforcement learning method to acquire a bipedal walking controller using only a simple reference motion. The reference motion was a simple motion consisting of three sine waves, and its parameters could be automatically set using Bayesian optimization. To facilitate reinforcement learning, we introduced two means: the use of inverse kinematics (IK) and the use of the reference motion as a bias. Numerical experiments showed that, with the proposed method, a bipedal walking controller can be learned using a smaller number of samples, compared with a method using reinforcement learning only and with methods using imitation rewards. Furthermore, experiments using a real robot, KHR-3HV (ver. 2), showed that, using a sim-to-real technique using domain randomization, a real robot could walk with the controller acquired in a simulator using the proposed method.

Since the present study was limited to controlling a small humanoid robot, an important future work would be to apply the proposed method to other robots, such as human-sized robots and robots of different shapes. In the present study, the previous method [16], which was successfully trained on a robot with bird-like legs, was not successful on our humanoid robot. Thus, it would be important to investigate the range of applicability of our proposed method.

Another line of future work is to apply the proposed method to the learning of more complex gaits. The present study was limited to a case in which the robot was encouraged to walk as quickly as possible. Learning various walking patterns, such as stopping after a certain distance or walking slowly, remains an issue for future work. Using additional sensors may also be useful to achieve a variety of walking behaviors. For example, combinations with vision sensors utilizing advanced processing methods (e.g., [43]) may be used to enable walking on stairs or in places with various obstacles.

Another future issue is to provide some guarantees for walking performance. There are recent studies on tracking control that can theoretically guarantee prescribed performance (e.g., [14,15]). It would be desirable to provide such guarantees in the reinforcement learning of bipedal walking.

Additionally, although the learning of bipedal walking was possible based on a relatively small number of samples when using our proposed method, the further acceleration of learning—for example, by parallelizing the learning with multiple simulations [44]—could be a future challenge.