Online Reinforcement-Learning-Based Adaptive Terminal Sliding Mode Control for Disturbed Bicycle Robots on a Curved Pavement

Abstract

The reaction wheel is able to help improve the balancing ability of a bicycle robot on curved pavement. However, preserving good control performances for such a robot that is driving on unstructured surfaces under matched and mismatched disturbances is challenging due to the underactuated characteristic and the nonlinearity of the robot. In this paper, a controller combining proximal policy optimization algorithms with terminal sliding mode controls is developed for controlling the balance of the robot. Online reinforcement-learning-based adaptive terminal sliding mode control is proposed to attenuate the influence of the matched and mismatched disturbance by adjusting parameters of the controller online. Different from several existing adaptive sliding mode approaches that only tune parameters of the reaching controller, the proposed method also considers the online adjustment of the sliding surface to provide adequate robustness against mismatched disturbances. The co-simulation experimental results in MSC Adams illustrate that the proposed controller can achieve better control performances than four existing methods for a reaction wheel bicycle robot moving on curved pavement, which verifies the robustness and applicability of the method.

1. Introduction

Compared with four-wheeled vehicles and other multi-wheeled vehicles, the long and narrow shape of bicycle robots (BRs) with only two ground contact points can provide greater flexibility and higher energy efficiency, which has the great potential application value in rugged and narrow terrain [1]. However, most existing studies focus on the balance control of BR on flat roads. To realize the unmanned driving of BR on rough roads, it is necessary to study the balance control of BR on curved pavements.

As for developed mechanisms for the balance control of BR, they can be divided into two categories. The first category is based on steering control. For instance, Zhang [2] presented BR dynamics with an accurate steering mechanism model and analyzed its balance control. Zhao [3] designed a steering controller to balance BR at a low speed (e.g., 0.58 m/s). Sun [4] proposed a polynomial controller to achieve the balance of BR with time-varying forward velocities. However, steering control cannot provide sufficient resistance to disturbances for BR with low forward speeds. The second category is controlling the auxiliary balancing mechanism, such as control moment gyroscopes (CMGs), mass balancers (MBs) and reaction wheels (RWs). Chen [5] designed a stabilizing assistant system for BR by using CMG. Zheng [6] combined steering and CMG to improve the performance of balance control. BR with MB, such as a mass pendulum, can shift its center of gravity corresponding to its attitude [7]. Reaction wheel bicycle robots (RWBRs) have also drawn considerable attention [8,9]. Compared with MB and RW, CMG has more complex structures and more weight, which is not suitable for BR on curved pavement. MB usually needs to be able to provide both large torque and high instantaneous speed, so it is only suitable for lightweight BR or BR under small perturbations. Therefore, in order to ensure the practicability of BR on a curved pavement, RWBR is selected as the research object in this paper.

The existing control methods for BR with auxiliary balancing mechanisms mainly include linear control, nonlinear control, and intelligent control. The linear control, such as proportional-integral-differential (PID) [10] and linear quadratic regulator (LQR) [11,12], can achieve balance control in the range of small roll angles by approximating the linearization around the equilibrium point. However, for the balance control of RWBR on a curved pavement, external disturbances and unmodeled characteristics can lead to the degradation of the control performance of linear controllers and even the instability of the system. In this regard, some scholars studied the nonlinear control for BR. In [13], a sliding mode control and low-pass filtering were used to realize balance control on a flat road on a real motorcycle, which showed better anti-interference abilities than the linear controller. In [14], a fuzzy sliding mode controller was designed to deal with impulse disturbance and system uncertainty, but the determination of fuzzy rules was rather complicated. An adaptive law was designed for the reaching control part of the sliding mode controller, and its coefficients were tuned monotonically in [9]. With respect to intelligent control, taking reinforcement learning (RL) as an example, the neural network is used to fit the system’s model or control strategy, and the control strategy is tuned via the continuous interactions between the system and the environment to maximize the expected return. As RL achieved remarkable results in a series of tasks [15,16], some scholars have also explored the application of RL in BR tasks [17,18,19]. However, due to the difficulty of RL in sampling efficiency, interpretability, and stability proof, RL is limited in robot motion control.

To solve the aforementioned problems of the intelligent control method, combining the stability guarantee of traditional feedback controls (FC) with the optimization ability of RL is attractive. Intuitively, this combination can be divided into parallel and serial strategies. The parallel one (Figure 1a) superimposes the outputs of RL and conventional feedback control. Taking [20] as an example, the residual reinforcement learning (RRL) method is adopted to weaken the influence of unmodeled characteristics on system stability, and the complex control problem is decomposed into two parts, one of which can be effectively solved by the traditional feedback control method and the other can be solved by RL. The method is successfully applied to a complex operation task of a physical manipulator without modeling contact and collision. The serial one (Figure 1b) uses the optimization capabilities of RL to tune the parameters or structures of FC. The serial strategy may have the following advantages: (1) From the view of FC, the adaptive change in FC parameters based on RL is beneficial for attenuating the influence of unmodeled characteristics and periodic disturbances on the performance of FC. (2) From the perspective of RL, the introduction of FC as a prior controller framework can greatly improve the learning speed of RL. Thus, the series of FC and RL for the balance control of RWBR is necessary and meaningful.

In this paper, the serial controller of the terminal sliding mode control (TSMC) [21] and proximal policy optimization algorithms (PPO) [22] is proposed to solve the balance control task of RWBR while driving on curved pavements. The control problem considers matched disturbances (the dry friction of inertial wheel, etc.) and composite unmatched disturbances (including unmodeled wheel–ground contact, gust disturbances, and periodic disturbances induced by terrain, etc.). TSMC is a nonlinear FC method with strong robustness relative to system uncertainties and disturbances. PPO is a type of policy gradient training that alternates between sampling data via environmental interaction and optimizing a clipped surrogate objective function using stochastic gradient descent. With its excellent versatility and stability, it is used as a benchmark algorithm for RL at present. The proposed controller, called PPO-TSMC, explores the environment via output sampling with Gaussian distributions. The scalar value of the BR state quantity in disturbed environments is estimated by using a multi-layer neural network. The parameters of TSMC are tuned online by gradient descent under reasonable optimization objectives, and an adaptive terminal sliding mode controller based on RL is finally formed, which is used for the balance control of RWBR.

The contribution of this paper is to implement a PPO-TSMC controller that adaptively adjusts the parameters of TSMC online based on the RL method of PPO and to apply it to the balance control problem of RWBR while driving on a curved pavement. Specifically, a simplified numerical model of RWBR is derived, and a feedback transformation is designed for the simplified RWBR model. Then, a test scenario with multiple disturbances was constructed in MSC Adams, and the controller’s performances of RWBR using TSMC, adaptive integral terminal sliding mode (AITSM) [9], PPO, RRL, and PPO-TSMC were compared under different rear-wheel velocities and transverse periodic disturbance amplitudes of BR. The effectiveness and robustness of the proposed PPO-TSMC in task scenarios are verified. In addition, to the best knowledge of the authors, few studies considered the balance control of BR under both matched and unmatched disturbances on curved pavements.

This paper is organized as follows: The RWBR is described in Section 2, and a dynamical model referring to the inertial wheel pendulum is established using some assumptions and simplifications. In Section 3, the simplified dynamical model is transformed into chain form via feedback transformation. On this basis, the PPO-TSMC adaptive controller is designed. The TSMC Net., Critic Net., and the calculation flow of the adaptive adjustment of controller parameters are described in detail. In Section 4, the simulation environment in MSC Adams is built, and four test cases were designed; then, the performance of five different controllers are compared. Finally, in the Section 5, a conclusion is reached.

The video of the experiment is available at the following website: https://github.com/ZhuXianjinGitHub/PPO-TSMC, accessed on 25 October 2022.

2. Dynamics

In order to illustrate the motion of RWBR on a curved pavement, the reference frames are defined in Figure 2. The inertia frame is defined as $o_g{x}_g{y}_g{z}_g$, and the body-fixed reference frame is defined as $o_c{x}_c{y}_c{z}_c$, where the center $o_C$ is located at the center of gravity of the BR. The BR consists of four rigid bodies, namely, a rear wheel, body frame, reaction wheel, and front wheel (simplified as R, B, W, and F, respectively), as shown in Figure 2. In addition, the assumptions in the dynamical model are made as follows: (1) The thickness of the rear and front wheels is negligible, and the contacts between wheels and the ground are regarded as point contacts. (2) These four rigid bodies are symmetrical with respect to the plane of the rear and front wheels, so the center of mass of these bodies are in the same plane.

Consider the body frame and rear and front wheels as one unit P, and the reaction wheel as another W in Figure 3. The BR can be converted to an inertia wheel pendulum system [23]. The body fixed reference frames $o_1{x}_1{y}_1{z}_1$ and $o_2{x}_2{y}_2{z}_2$ are defined on P and W. The mass and inertia matrix of the two parts with respect to the body-fixed reference frame are $m_1$, $I_1$, $m_2$, and $I_2$. $L_1$ and $L_2$ represent the distance between the centroids of P and W and the connection between front- and rear-wheel ground points on a flat road:

$$\begin{array}{c}J{\ddot{q}}_1+I_2{\ddot{q}}_2-Mgcos\left(q_3\right)sin\left(q_1\right)=d_1\hfill \\ I_2\left({\ddot{q}}_1+{\ddot{q}}_2\right)=\tau +d_2\hfill \end{array}$$

(1)

where $J=m_1{L}_1^2+m_2{L}_2^2+I_1+I_2$, $M=m_1{L}_1+m_2{L}_2$, $q_1$ is the roll angle, $q_2$ is the reaction wheel angel, $q_3$ is the pitch angle of the robot, $\tau$ is the input torque of the reaction wheel’s motor, and $d_1$ and $d_2$ are mismatched/matched disturbances.

3. Design of Controller

First, the feedback transformation of an equivalent inertia wheel pendulum is introduced in this section. Upon this transformation, a terminal sliding mode controller is constructed. Then, the actor-critic framework, TSMC Net., and the key components of the PPO-TSMC are presented. The process and convergence of the PPO-TSMC are described. An analysis of the PPO optimization process for TSMC is performed.

3.1. Feedback Transformation and Terminal Sliding Mode Control

Feedback transformation is a common approach in the control of the nonlinear systems. Based on the Olfati-Saber transformation mentioned in [24,25], the following state variables and feedback transformation are defined (1).

$$\begin{array}{cc}\hfill & x_1=J{\dot{q}}_1+I_2{\dot{q}}_2\hfill \\ \hfill & x_2=Mgcos\left(q_3\right)sin\left(q_1\right)\hfill \\ \hfill & x_3=Mgcos\left(q_3\right)cos\left(q_1\right){\dot{q}}_1\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill u=& -\frac{1}{J-I_2}Mgcos\left(q_3\right)cos\left(q_1\right)\tau -Mgcos\left(q_3\right)sin\left(q_1\right){\dot{q}}_1^2+\hfill \\ & {\left(Mgcos\left(q_3\right)\right)}^2\frac{I_2}{J{I}_2-I_2^2}cos\left(q_1\right)sin\left(q_1\right)\hfill \end{array}$$

(3)

The system can be expressed as follows:

$$\begin{array}{cc}\hfill & {\dot{x}}_1=x_2+d_1\hfill \\ \hfill & {\dot{x}}_2=x_3\hfill \\ \hfill & {\dot{x}}_3=u+\frac{1}{J-I_2}Mgcos\left(q_3\right)cos\left(q_1\right)d_2\hfill \end{array}$$

(4)

where $\left|\frac{1}{J-I_2}Mgcos\left(q_3\right)cos\left(q_1\right)\right|\le \frac{1}{J-I_2}Mg$ is bounded.

The TSMC is designed according to the method in [26]. First, define the recursive sliding surfaces as follows:

$$\begin{array}{cc}\hfill & s_0=x_1\hfill \\ \hfill & s_1={\dot{s}}_0+{\alpha }_0{s}_0+{\beta }_0{\left|s_0\right|}^{p0}\mathrm{sign}\left(s_0\right)\hfill \\ \hfill & s_2={\dot{s}}_1+{\alpha }_1{s}_1+{\beta }_1{\left|s_1\right|}^{p1}\mathrm{sign}\left(s_1\right)\hfill \end{array}$$

(5)

where ${\alpha }_0,{\alpha }_1,{\beta }_0,{\beta }_1>0,0<p_0<1,0<p_1<1$ and the sign function [27] is defined as follows:

$$\begin{array}{c}\hfill \mathrm{sign}\left(s\right)=\left\{\begin{array}{cc}1,\hfill & s>0\hfill \\ -1,\hfill & s<0\hfill \end{array}\right.\end{array}$$

(6)

and $\mathrm{sign}\left(0\right)\in [-1,1]$. Next, based on the sliding surfaces (5), the following output of the mean part of the TSMC Net. is defined as follows:

$$u_{smc}=-\left(u_{eq}+u_r\right)$$

(7)

where $u_{eq}$ and $u_r$ are the equivalent and reaching controllers with the following expressions:

$$\begin{array}{cc}\hfill & u_{eq}={\alpha }_0{s}_0^{\left(2\right)}+{\alpha }_1{s}_1^{\left(1\right)}+{\beta }_0\frac{d^2}{d{t}^2}\left(|s_0{|}^{p_0}\mathrm{sign}\left(s_0\right)\right)+{\beta }_1\frac{d}{dt}\left(|s_1{|}^{p_1}\mathrm{sign}\left(s_1\right)\right)\hfill \\ \hfill & u_r=\phi s_2+\vartheta \left(|s_2{|}^{p_2}\mathrm{sign}\left(s_2\right)\right)\hfill \end{array}$$

(8)

where $\phi ,\vartheta >0$ and ${\theta }_1:={\left[{\alpha }_0,{\alpha }_1,{\beta }_0,{\beta }_1\right]}^T$, ${\theta }_2:={\left[\phi ,\vartheta \right]}^T$.

3.2. PPO-TSMC

In order to adopt reinforcement learning to optimize the TSMC, the above system needs to be discretized and meets the following assumptions.

Assumption 1.

The above system satisfies the Markov property, which means that the state at time t only depends on the state at the time $t-1$ and the corresponding action, independent of other historical states and inputs.

The actor-critic framework for the optimization of a Markov decision process includes two time-scale algorithms in which the critic uses temporal difference learning with a linear approximation architecture, and the actor is updated in an approximate gradient direction based on information provided by the critic. The actor-critic framework combined the advantages of actor-only and critic-only methods. PPO with the actor-critic style is one of the most popular on-policy RL algorithms. It simultaneously optimizes a stochastic policy as well as an approximator for the neural network value function. The main reason for choosing PPO in PPO-TSMC is that PPO uses conservative policy iterations based on an estimator of the advantage function to guarantee the monotonic improvement for general stochastic policies. The monotonic improvement guarantee for general stochastic policies can be found in [28].

PPO-TSMC aims at combing the interference rejection ability of TSMC and the monotonic improvement ablity of PPO. Specifically, in order to adaptively adjust the coefficients of TSMC using automatic differential software, the TSMC controller is represented as a neural network. The weights of the neural networks represent the coefficients of the TSMC. Then, the TSMC controller represented by a neural network is used to replace the mean part of the PPO’s actor network. Finally, the optimization framework based on PPO with an actor-critic is explored to adaptively adjust the coefficients of TSMC.

Remark 1.

In this paper, the reinforcement learning method is introduced to tune parameters ${\alpha }_0,{\beta }_0,{\alpha }_1,{\beta }_1,\phi$, and θ of the TSMC controller (7)–(8). The main motivation of the PPO-TSMC control scheme is to improve the robustness with respect to matched and mismatched disturbances. Indeed, the online adaptation of $\phi ,\theta$ can attenuate the matched disturbance $\frac{1}{J-I_2}Mgcos\left(q_3\right)cos\left(q_1\right)d_2$, and well-adjusted parameters such as ${\alpha }_0,{\beta }_0,{\alpha }_1$, and ${\beta }_1$ can lead to a more robust sliding surfaces against mismatched disturbance $d_1$. The adaptive terminal sliding mode was also designed for RWBR in [9] , but the existing method only considered the online regulation of $\phi ,\theta$, so this method cannot actively compensate mismatched disturbances. Moreover, the adaptive gains of [9] are monotonic, which may cause more serious chattering in practice.

TSMC Net. (Figure 4) is used to map states $x_t={(x_1,x_2,x_3)}_t\in R^3$ to actions $u_t\in R$, $u_t\sim {\pi }_{\theta }\left(x_t\right)$ in which $\theta$ represents the parameters of the policy $\pi$. In this paper, strategy $\pi$ is a normal distribution comprising the mean part of TSMC and the standard deviation part of a non-negative output from a neural network.

The mean part: From the literature [24,29], state $x_1$ in (4) is the generalized momentum conjugate relative to $q_1$ in (1). Therefore, the control objective of RWBR can be equivalently considered as the stabilization of the transformed system (4).

The standard deviation part: The value of the standard deviation is output by the neural network, as shown in Figure 4. The structure of the neural network is set up by a hyperbolic tangent function $y=tanh\left(x\right)$, a fully connected layer, and a softplus activation function $y=log\left(1+e^x\right)$. The fully connected $\left(i\right)$ in Figure 4 represents the fully connected layer and i represents the number of neurons in the neural network. The weights of the neural network are independent of the states, such as the methods in [22].

$${\sigma }^2\left(x_t\right)=log\left(1+e^{W_1^{T}tanh\left(x_t\right)+a}\right)$$

(9)

The output of the TSMC Net. is as follows:

$$u\sim N\left(u_{smc}\left({\theta }_1,{\theta }_2\right),{\sigma }^2\left({\theta }_3\right)\right)$$

(10)

where ${\theta }_3:={\left[W_1,a\right]}^T$ denotes the weights of the neural network in the standard deviation part. $\theta ={\left[{\theta }_1,{\theta }_2,{\theta }_3\right]}^T$ represents adjustable parameters.

Remark 2.

The normal distribution can be seen as a diagonal Gaussian policy with one dimension. The Gaussian policy is one of the most common type of stochastic policies in deep RL and a type of policy used in continuous action space. The standard deviation in Gaussian policies controls the exploratory behavior during policy training. Different implementations of the standard deviation are discussed in [30].

The critic Net. implements a value function approximator that is used to map state $x_t$ to scalar value $V_{\mu }\left(x_t\right)$, in which $\mu$ means the parameter in the critic Net. The scalar value represents the predicted discounted cumulative long-term reward when the agent starts from the given state and takes the best possible action. The critic Net. of this paper is shown in Figure 5, which is composed of a deep neural network and an ReLU [31] nonlinear activation function. The gradient descent calculation of the critic Net. is shown in the next section.

To facilitate the description of the composition and calculation process of PPO-TMSC, the framework of PPO-TSMC is shown in Figure 6. The process can be divided into three steps. In the first step, Gaussian noise is added on the output of the TSMC so that the RWBR can conduct some interactive exploration with the environment. The second step is to randomly sample the stored state and action sequence. The state value and advantage function under the finite-horizon estimators are calculated, and the parameters in critic Net. are updated. The third step is to update TSMC coefficients based on the information provided by critic Net.

Step 1: $q_t$ can be obtained from the interaction of RWBR and the environment, $x_t$ can be calculated by transformation (2), $u_t$ is based on strategy ${\pi }_{\theta }$ (10), and the actual output of the controller can be obtained by the transformation (3). Then, $q_{t+1}$ and $x_{t+1}$ can be obtained by interactions with the environment. Reward $r_{t+1}$ can be calculated by (11). Then, $\left\{x_t,u_t,r_{t+1},x_{t+1}\right\}$ is stored as tuples:

$$\begin{array}{c}\hfill r_t=\left\{\begin{array}{cc}0\hfill & r_t<0,\hfill \\ 1-\varphi \overline{X}{\left(t\right)}^2\hfill & 0<r_t<1\hfill \\ 1\hfill & r_t>1\hfill \end{array}\right.\end{array}$$

(11)

where $\varphi =\left[a_1,a_2,a_3,a_4\right],a_i>0$ and $\overline{X}={\left[x_1,x_2,x_3,u\right]}^T$.

Step 2: Sample from the stored tuples and update the parameters $\mu$ of the critic Net. Sample a sequence of N tuples $\Omega =\left\{x_t,u_t,r_{t+1},x_{t+1},\cdots ,x_{t+N-1},u_{t+N-1},r_{t+N},x_{t+N}\right\}$. The generalized advantage estimator (GAE) ${\hat{A}}_t$ shown by (13) is estimated based on temporal difference errors ${\delta }_t$:

$${\delta }_t=r_t+\gamma V_{\mu }\left(x_{t+1}\right)-V_{\mu }\left(x_t\right)$$

(12)

$${\hat{A}}_t={\delta }_t+\left(\gamma \lambda \right){\delta }_{t+1}+{\left(\gamma \lambda \right)}^2{\delta }_{t+2}+\cdots +{\left(\gamma \lambda \right)}^{U-t+1}{\delta }_{U-1};\gamma ,\lambda \in \left[0,1\right],$$

(13)

where $\gamma$ is the discounted factor for future rewards and $\lambda$ is the smoothing factor for GAE. GAE (13) is beneficial for obtaining a better balance between bias and variance [32]. Update critic parameters $\mu$ by minimizing loss $L^V$ across all sampled mini-batch data:

$$L^V\left(\mu \right)=\frac{1}{M}\sum _{i=1}^M{A_i}^2$$

(14)

$$\mu =\mu -{\eta }_{\mu }\nabla L^V\left(\mu \right)$$

(15)

where ${\eta }_{\mu }$ is the learning rate of critic Net.

Step 3: Update TSMC Net. by minimizing the loss function $L^{CLIP}$:

$$L^{CLIP}\left(\theta \right)=\frac{1}{M}\sum _{i=1}^M\left[-min\left(\frac{{\pi }_{\theta }\left(u_i|x_i\right)}{{\pi }_{\theta old}\left(u_i|x_i\right)}{\hat{A}}_i,clip\left(\frac{{\pi }_{\theta }\left(u_i|x_i\right)}{{\pi }_{\theta old}\left(u_i|x_i\right)},1-\epsilon ,1+\epsilon \right){\hat{A}}_i\right)+\omega H\left(\theta ,x_i\right)\right]$$

(16)

$$H\left(\theta ,x_i\right)=\frac{1}{2}ln\left(2\pi ·e·{{\sigma }_i}^2\right)$$

(17)

$$\theta =\theta -{\eta }_{\theta }\nabla L^{CLIP}\left(\theta \right)$$

(18)

where ${\pi }_{\theta }\left(u_i|x_i\right)$ represents the probability of taking action $u_i$ for a given state x, ${\pi }_{\theta old}\left(u_i|x_i\right)$ represents the corresponding probability of policy parameter $\theta$ before updates, and the $\epsilon$ clip ensures that each update will not fluctuate too much. $H\left(\theta ,x_i\right)$ is the entropy loss that is used to encourage the agent’s exploration, $\omega$ is the entropy loss weight factor, and ${\eta }_{\theta }$ represents the learning rate of TSMC Net.

Remark 3.

The global convergence proof for PPO is challenging since it uses deep neural networks, policies that become greedy, and previous policies for the trust region method. The authors of [33] provided an overview of a convergence proof. In [34], the two-time-scale stochastic approximation theory was employed to prove that PPO guarantees local convergence.

In TSMC-PPO, ${\theta }_1>0$ and ${\theta }_2>0$ can be ensured in the TSMC Net. The local convergence of PPO can ensure that the number of ${\theta }_3$ approaches 0. Therefore, the asymptotically stable TSMC-PPO can be guaranteed in practice.

4. Simulation Experiment

4.1. Simulation Platform

In order to verify the performance of the controller proposed in this paper, the simulation environment is built in MSC Adams, as shown in Figure 7.

In the simulation environment, an RWBR is placed on a curved pavement. According to the physical parameters of the BR, the parameters in Formula (1) are calculated as follows: $M_1=0.5376$ kg, $I_1=8.4614\times {10}^{-6}$ kg $·{\mathrm{m}}^2$, $M_2=1.0137$ kg, $I_2=8.2354\times {10}^{-6}$ kg $·{\mathrm{m}}^2$, and $L_1=L_2=0.05$ m. The maximum length of the robot is $0.3$ m. The function relationship between the height of the curved pavement and the direction of the $x_g$ of the inertia frame is $y=1.5\times \left(sin\right(\pi /12\times x)-1)$, and the total length of the pavement is 60 m.

In MSC Adams, the contact force model includes the normal positive pressure based on collision [35] and the tangential friction force model based on velocity [36]. The function expression of collision positive pressure is listed as follows:

$$F_c=max\left(0,k·{x_d}^e-step\left(x_d,0,0,d,c_{max}\right)·{\dot{x}}_d\right)$$

(19)

where $x_d$ is the penetration depth, k is the contact stiffness coefficient, e is force exponent, and d is the penetration depth when damping reaches the maximum. $c_{max}$ is the maximum damping coefficient. As shown in the Figure 8, the collision force has two parameters, the stiffness coefficient and the damping coefficient. Thus, the collision model can be used when assuming that the tire and the ground are made of rubber.

As shown in Figure 9, the frictional force model based on velocity defined in (20) is an effective method for calculating tangential contact forces:

$$\mu \left({\nu }_{\tau }\right)=\left\{\begin{array}{c}-\mathrm{sign}\left(v_{\tau }\right){\mu }_d\\ \mathrm{sign}\left(v_{\tau }\right)step\left(\left|v_{\tau }\right|,v_s,-{\mu }_s,v_d,-{\mu }_d\right)\\ -step\left(v_{\tau },-v_s,-{\mu }_s,v_s,{\mu }_s\right)\end{array}\right.\begin{array}{c}\left|v_{\tau }\right|>v_d\\ v_s<\left|v_{\tau }\right|⩽v_d\\ -v_s⩽v_{\tau }⩽v_s\end{array}$$

(20)

where $v_{\tau }$ is the relative velocity of contact points, $v_s$ and $v_d$ represent the stiction transition velocity and friction transition velocity, respectively, and ${\mu }_s$ and ${\mu }_d$ represent the static friction coefficient and dynamic friction coefficient, respectively.

The step function expression in MSC Adams is defined as follows:

$$step\left(x,x_0,h_0,x_1,h_1\right)=\left\{\begin{array}{c}{h}_0\\ h_0-a·{\Delta }^2\left(3-2\Delta \right)\\ h_1\end{array}\right.\begin{array}{c}x⩽x_0\\ x_0<x<x_1\\ x⩾x_1\end{array}$$

(21)

where $a=h_0-h_1$, $\Delta =\left(x-x_0\right)/\left(x_1-x_0\right)$.

The parameters used in the contact force model can be chosen while following the instruction of previous studies [37,38], which provide recommended ranges of the related parameters for different materials. Based on those recommendations, the values of parameters are finally determined after multiple simulations in Adams.

Four test cases are formed by the combination of different rear-wheel angle velocities, v, and periodic lateral disturbances (for simulating wind gusts), d, as shown in

Table 1. The parameters of four test cases.

	Case 1	Case 2	Case 3	Case 4
$v (\mathrm{rad}/\mathrm{s})$	1.5	1.5	3	3
$d \left(\mathrm{Nm}\right)$	$0.2sin(t/\pi )$	$0.3sin(t/\pi )$	$0.2sin(t/\pi )$	$0.3sin(t/\pi )$

. In addition, a vertical upward perturbation $y=0.1sin(0.5t/\pi )\mathrm{Nm}$ is added at the center of the RWBR.

In each case, performances of TSMC, AITSM, PPO, RRL, and PPO-TSMC are compared. In all cases, the control frequency is set to 1 kHz. All cases in the experiments are set to 400 s. The initial coefficients of all controllers are tuned to obtain an acceptable control performance. Four existing controllers are implemented as follows. (1) The implementation of TSMC is shown in Section 3.1 in addition to parameters ${\theta }_1\left(0\right)=[1,1,1,1]$ and ${\theta }_2\left(0\right)=[5,6]$. (2) PPO is implemented by replacing the TSMC Net. in Section 3.2 with the actor Net. The actor Net. in PPO is shown in Figure 10. (3) The output of the RRL is the output of TSMC multiplied by 0.7 plus the output of PPO multiplied by 0.3, as shown in Figure 1a. (4) The implementation of the AITSM controller is as follows:

$$u_r=\left({\gamma }_0+{\gamma }_1\left|s_1\right|+{\gamma }_2\left|{\dot{s}}_1\right|\right)\mathrm{sign}\left(s_2\right)+\epsilon {\left|s_1\right|}^a\mathrm{sign}\left(s_2\right)+k{\left|s_2\right|}^{b·\mathrm{sign}\left(\left|s_2\right|-1\right)}{s}_2$$

(22)

and ${\gamma }_0,{\gamma }_1$, and ${\gamma }_2$ are generated by the adaptation laws as follows:

$${\dot{\gamma }}_0=\left\{\begin{array}{c}{\eta }_0\left|s_2\right|if\left|s_2\right|>\xi \hfill \\ 0else\hfill \end{array}\right.{\dot{\gamma }}_1=\left\{\begin{array}{c}{\eta }_1\left|s_2\right|\left|s_1\right|if\left|s_2\right|>\xi \hfill \\ 0else\hfill \end{array}\right.{\dot{\gamma }}_2=\left\{\begin{array}{c}{\eta }_2\left|s_2\right|\left|{\dot{s}}_1\right|if\left|s_2\right|>\xi \hfill \\ 0else\hfill \end{array}\right.$$

(23)

where $\epsilon =0.01,a=0.5,k=0.05,b=0.2,\xi =0.8,{\eta }_0=1,{\eta }_1=1,{\eta }_2=1$, ${\gamma }_0\left(0\right)=5,{\gamma }_1\left(0\right)=5,$ and ${\gamma }_2\left(0\right)=5$.

4.2. Experimental Results

In this section, we present the experimental results of the controllers, including TSMC, AITSM, RRL, RL, and PPO-TSMC. The following conclusions can be drawn from experiments. Firstly, the RWBR with PPO-TSMC is the only controller that completes the full 400 s test scenario in all four cases (Figure 11), which shows that it has the best robustness over others. Secondly, it is straightforward to observe that the PPO-TSMC obtained the smallest roll angle tracking error from

Table 2. Performance comparisons of five controllers.

Case	Controller	MAX (rad)	MEAN (rad)	RMS (rad)	Time (s)
1	TSMC	-	-	-	196
	AITSM	0.215	−0.0229	0.0625	-
	PPO	-	-	-	22
	RRL	-	-	-	361
	PPO-TSMC	0.0869	−0.0053	0.0371	-
2	TSMC	-	-	-	168
	AITSM	0.231	−0.0353	0.0735	-
	PPO	-	-	-	14
	RRL	-	-	-	274
	PPO-TSMC	0.132	−0.0078	0.0539	-
3	TSMC	0.224	−0.0199	0.0883	-
	AITSM	-	-	-	389
	PPO	-	-	-	11
	RRL	0.321	−0.0171	0.0931	-
	PPO-TSMC	0.137	−0.0082	0.0489	-
4	TSMC	0.269	−0.0194	0.0858	-
	AITSM	-	-	-	265
	PPO	-	-	-	8
	RRL	0.314	0.0046	0.0774	-
	PPO-TSMC	0.211	−0.0072	0.0540	-

; thus, PPO-TSMC has the best control performance among the five controllers. This is mainly due to the fact that both TSMCs without information about the bound of disturbance and AITSM with a monotonically increasing adaptation law (23) cannot maintain a good control performance for the balance control of RWBR with matched disturbances, mismatched disturbances, and unmodeled characteristics. Moreover, the reinforcement learning method is difficult to directly apply to the online balance control task. In order to adaptively adjust the parameters of TSMC, PPO-TSMC explores the influence of various disturbances through TSMC Net. and then fits an estimator of value function through a sequence of states, actions, and rewards. This value function can effectively guide the update of policy parameters based on gradients.

Since the RWBR with RL easily falls, to rule out this low-efficiency controller, Figure 11 only presents the tracking performance of the remaining four controllers. Figure 12, Figure 13 and Figure 14 illustrate the variety of critic Net., weights, the mean parameters of the TSMC Net., and the std of TSMC Net. The gain value of AITSM is shown in Figure 15. In case 1 and case 2, the RWBR with the TSMC controller falls down in less than 200 s, and the RWBR with the RRL controller falls down at about 380 s and 300 s. In case 3 and case 4, the RWBR with the AITSM controller falls down at 390 s and 260 s, respectively. The RWBR with a PPO-TSMC controller completed 400 s simulation in all four cases.

In order to present the adaptive change process of the parameters of the proposed controller. In Figure 12, neural network weights of the middle layer of Critic Net. in four cases are described over time. As can be seen from Figure 12, at about 200 s, the weight of the Critic N. neural network becomes stable. In Figure 13 and Figure 14, the weight of TSMC Net, namely the coefficient of the sliding mode control, shows a relatively clear trend with respect to changes after 200 s. The adaptive parameter tunement method of AITSM (Figure 15) leads to the phenomenon of high-amplitude chattering, or even instability in Figure 11c, due to the monotonically increasing adaptation law (23).

For further data comparisons, the maximum (MAX), the root mean square (RMS), and the mean (MEAN) values of the roll are adopted, which are defined as follows.

$$\begin{array}{cc}\hfill \mathrm{MAX}\left(q_1\left(i\right)\right)& =max\left(\left|q_1\left(i\right)\right|\right)\hfill \\ \hfill \mathrm{MEAN}\left(q_1\left(i\right)\right)& =\sum _{i=1}^N\frac{q_1\left(i\right)}{N}\hfill \\ \hfill \mathrm{RMS}\left(q_1\left(i\right)\right)& =\sqrt{\sum _{i=1}^N\frac{q_1^2\left(i\right)}{N}}\hfill \end{array}$$

(24)

In Table 2, for the BR that fell in the 400 s simulation test, no relevant comparisons were made between the system and the machine. The comparison of control performances under five controllers in four cases is listed. The simulations with PPO-TSMC exhibit the smallest MAX, MEAN, and RMS. The PPO controller behaves the worst performance. Although the TSMC controller can run the entire distance in case 1 and case 4, it performs poorly in case 1 and case 2, while AITSM and RRL controllers are better than TSMC but still worse than the proposed control. It can be concluded that the proposed PPO-TSMC controller achieves superior balancing performances compared to the other four proposed controllers.

5. Conclusions

In this paper, a PPO-TSMC controller is developed for the balancing purpose of an RWBR system on curved pavement with matched disturbances (the dry friction of inertia wheels, etc.) and composite mismatched disturbances (including unmodeled wheel–ground contact and gust disturbances and topographically introduced periodic disturbances, etc.). By connecting PPO and TSMC in series, a random action strategy based on a normal distribution is constructed in the framework of an initial TSMC controller. The controller parameters under disturbances and unmodeled characteristics are explored by using the strategy, and the online adaptive adjustment of TSMC parameters based on PPO algorithm is realized. The comparison between the proposed PPO-TSMC and TSMC, PPO, AITSM, and RRL illustrates stronger robustness and better control performance.

This study is different from existing related research studies about the reaction wheel bicycle robot from the perspective of the task, the method of the control, and the simulation test. Previous studies have not considered the influence of an unstructured curved pavement on the balance control of RWBR. For example, in [9] (2021), an adaptive integral terminal sliding mode control scheme was developed for the balancing purpose of the RWBR on a flat road with uncertainties and unmodelled dynamics by designing adaptive laws. In [39] (2022), an extreme-learning-machine scheme was designed as a compensator for estimating lumped uncertainties of the RWBR on a flat road. From the perspective of control methods, the existing studies [9,39] only considered reducing the influence of matched disturbances but not mismatched disturbances. However, not only the matched perturbation term but also the influence of the mismatched disturbances are considered in this work. Namely, parameters of the sliding mode surface and reaching control are adjusted simultaneously. In terms of simulation test settings, this work sets four different test cases in MSC Adams via different vehicle speeds and lateral disturbances. The comparison between the proposed PPO-TSMC and TSMC, PPO, AITSM, and RRL illustrates stronger robustness and better control performances. It shows that the PPO-TSMC in this work has application prospects.

On the other hand, improving the sample efficiency of the proposed PPO-TSMC will be considered as the main future research direction of this paper. From Figure 12 and Figure 13, the critic network does not become stable until about 200 s. Moreover, the convergence of actor Net. occurs later than that of critic Net. It is not very ideal for the practicability of the algorithm and limits the effectiveness of PPO-TSMC in a rapidly changing environment or when there are more random disturbances. Therefore, our future work aims to replace the critic deep neural network with Gaussian process regression [40], radial basis functions [41], or some other components of model-based reinforcement learning [42] to reduce the demand for data samples. Next, discretizing the output of the controller [43] is another method for reducing sample requirements. Finally, a physical deployment and pilot study are planned.