Driving Torque Distribution Strategy of Skid-Steering Vehicles with Knowledge-Assisted Reinforcement Learning

Abstract

Due to the advantages of their drive configuration form, skid-steering vehicles with independent wheel drive systems are widely used in various special applications. However, obtaining a reasonable distribution of the driving torques for the coordinated control of independent driving wheels is a challenging problem. In this paper, we propose a torque distribution strategy based on the Knowledge-Assisted Deep Deterministic Policy Gradient (KA-DDPG) algorithm, in order to minimize the desired value tracking error as well as achieve the longitudinal speed and yaw rate tracking control of skid-steering vehicles. The KA-DDPG algorithm combines knowledge-assisted learning methods with the DDPG algorithm, within the framework of knowledge-assisted reinforcement learning. To accelerate the learning process of KA-DDPG, two assisted learning methods are proposed: a criteria action method and a guiding reward method. The simulation results obtained, considering different scenarios, demonstrate that the KA-DDPG-based torque distribution strategy allows a skid-steering vehicle to achieve high performance, in tracking the desired value. In addition, further simulation results, also, demonstrate the contributions of knowledge-assisted learning methods to the training process of KA-DDPG: the criteria action method speeds up the learning speed by reducing the agent’s random action selection, while the guiding reward method achieves the same result by sharpening the reward function.

1. Introduction

Due to their advantages of simple structure, sustainability, and flexible control methods, skid-steering vehicles have been widely used in civil and military applications, including agricultural vehicles [1], combat vehicles [2,3], robots [4,5], and so on. As shown in Figure 1, skid-steering vehicles are typically equipped with four independent driving wheels, forming an overdrive system with more control output than control input. Unlike car-like vehicles [6,7,8], skid-steering vehicles do not possess a dedicated steering system; therefore, a reasonable torque distribution is key to skid-steering vehicle control. With the development of optimal problem solvers, through reinforcement learning (RL) techniques in complex systems, new insights have been provided for coordinated torque distribution in skid-steering vehicles. Therefore, the development of torque distribution strategies based on RL has become a promising research direction for future skid-steering vehicles.

The conventional control methods for skid-steering vehicles, mainly, include kinematics and dynamics methods [9]. In kinematics methods, the desired control value is transformed into a reference speed for each wheel, based on the vehicle kinematics, following which each driving wheel generates driving torque, to track its own reference speed for vehicle control [10,11,12]. Kinematics methods are subject to significant slippage and skidding, when precisely following the wheel reference speed, which seriously impacts the control performance of the vehicle. In [13], the authors proposed a speed-adaptive control algorithm for six-wheeled differential steering vehicles. To enhance the vehicle handing and lateral stability performance, reference wheel speeds are, individually, generated for each wheel, based on their respective slipping and skidding status. In [14], a hierarchical controller was designed for a skid-steering vehicle, based on the need of dynamic control. In the controller, a reference wheel speed generator was designed to calculate the wheel speed, and a wheel speed follower is used to reach the target wheel speed. In [15], in order to mitigate the effects of wheel slip on accurate control, the authors proposed a new kinematics model for skid-steering vehicles, which has the ability to predict and compensate for the slippages in the forward kinematics.

To improve maneuverability and stability, recent studies have proposed dynamics methods for the control of skid-steering vehicles. Dynamics methods can address the problem of torque distribution for skid-steering vehicles, through optimization theory [16,17,18]. In [19], the authors proposed a hierarchical torque distribution strategy for a skid-steering electric unmanned ground vehicle (EUGV), in order to control the longitudinal speed and yaw rate. The objective function, consisting of the longitudinal tire workload rates and weight factors of the tires, was established by considering inequality constraints, including actuator, road adhesion, and tire friction circle constraints. In [20], a hierarchical control framework for a six-wheel independent drive (6WID) vehicle was proposed, and optimization theory was employed to distribute the driving torques. The control strategy was used to realize the real-time torque distribution, having the ability to maintain wheel failure tolerance and limit wheel slip.

To date, RL algorithms have been successfully implemented in robots [21] and unmanned aerial vehicles (UAVs) [22,23] as well as energy [24], transportation [25], and other complex systems [26,27]. The DDPG algorithm has been successfully used to deal with decision-making problems involving continuous action spaces in various vehicle control applications [28], such as trajectory planning [29], automatic lane changing [30], and optimal torque distribution [31,32,33]. To overcome the shortcomings of the original DDPG in the training process, many learning tricks have been proposed to make the training more efficient and the convergence more stable. In [34], the authors proposed a DDPG-based controller that allows UAVs to fly robustly in uncertain environments. Three learning tricks—delayed learning, adversarial attack, and mixed exploration—were introduced, to overcome the fragility and volatility of the original DDPG, which greatly improved the convergence speed, convergence effect, and stability. In [35], a control approach based on Twin Delayed DDPG (TD3) was proposed, to handle the model-free attitude control problem of On-Orbit Servicing Spacecraft (OOSS), under the guidance of a mixed reward system. The Proportional–Integral–Derivative (PID) guide TD3 algorithm effectively increased the training speed and learning stability of the TD3 algorithm, through the use of prior knowledge. In [36], an improved energy management framework that embeds expert knowledge into the DDPG was proposed for hybrid electric vehicles (HEVs). By incorporating the battery characteristics and the optimal brake specific fuel consumption of HEVs, the proposed framework not only accelerated the learning process, but also obtained better fuel economy. In [37], the authors proposed a knowledge-assisted DDPG for the control of a cooperative wind farm, by combining knowledge-assisted methods with the DDPG algorithm. Three analytical models were utilized to speed up the learning process and train robustly.

To summarize the above work, kinematic methods are incapable of overcoming unexpected wheel slip, while dynamics methods can, theoretically, effectively control wheel slip within a limited range. However, the implementation of dynamics methods requires complicated functions, in order to estimate the vehicle model and the wheel–ground interactions, which are difficult to apply in practice [38]. The RL algorithm iteratively explores the optimal control strategy of the system in the training process, and the associated neural networks can approximate the dynamics model and wheel–ground interactions. Therefore, an RL-based torque distribution strategy for skid-steering vehicles is incorporated into our work.

In this study, we propose a KA-DDPG-based driving torque distribution method for skid-steering vehicles, in order to minimize the tracking error, with respect to the desired value. We, first, analyze a dynamics model of skid-steering vehicles for torque distribution. Then, the agent of the KA-DDPG algorithm is designed for vehicle control. Based on the torque distribution strategy of KA-DDPG, we can achieve longitudinal speed and yaw rate tracking control for skid-steering vehicles. The main contributions of this study can be summarized as follows: (1) a KA-DDPG-based torque distribution strategy is proposed for skid-steering vehicles, in order to minimize the errors of the longitudinal speed and yaw rate, thus realizing the tracking control of the desired value; (2) considering the learning efficiency of the KA-DDPG algorithm, the knowledge-assisted RL framework is proposed by combining two knowledge-assisted learning methods with the DDPG algorithm; and (3) a dynamics model for skid-steering vehicles is constructed, in order to evaluate the performance of the proposed method.

The remainder of this paper is organized as follows. Section 2 consists of two parts: one addressing the dynamics model of skid-steering vehicles, and another focused on the DDPG algorithm. Section 3 presents our KA-DDPG-based torque distribution method for skid-steering vehicles. The settings of the simulation environment are detailed in Section 4. In Section 5, the performance of the KA-DDPG-based torque distribution strategy and the contributions of the assisted learning methods in KA-DDPG are illustrated. Section 6 concludes this work and discusses possible future work.

2. Preliminaries

2.1. Dynamics Model

The skid-steering vehicle investigated in this study is shown in Figure 1, which is equipped with four independently driven wheels, forming a differential steering system. $OXY$ is the reference coordinate system, $oxy$ is the fixed coordinate system attached to the vehicle, and o represents the origin of the fixed coordinate system. B is the width of the vehicle. The distances from the center of gravity (CG) to the front and rear wheels are $L_f$ and $L_r$, respectively. As shown in Figure 1, the motion of the skid-steering vehicle is caused by the friction forces, which are generated by the wheel–ground interactions. The dynamics model is formulated as follows:

$$\begin{array}{cc}\hfill m\left(v_{xa}^{\prime }-v_{ya}{\omega }_{\varphi a}\right)& =\sum _{i=1}^{i=4}{f}_{x_i}+d_x,\hfill \\ \hfill m\left(v_{ya}^{\prime }+v_{xa}{\omega }_{\varphi a}\right)& =\sum _{i=1}^{i=4}{f}_{y_i}+d_y,\hfill \\ \hfill J{\omega }_{\varphi a}^{\prime }& =\sum _{i=1}^{i=4}{\tau }_{{\mathbf{f}}_i}+d_{\varphi },\hfill \end{array}$$

(1)

where $f_{x_i}$ is the longitudinal force, and $f_{y_i}$ is the lateral force on the $i\mathrm{th}$ wheel, m is the mass of the vehicle, J is the inertia of the vehicle around CG, $v_{xa}$ is the longitudinal speed, $v_{ya}$ is the lateral speed, ${\omega }_{\varphi a}$ is the yaw rate, ${\tau }_{{\mathbf{f}}_i}$ is the torque caused by the traction force ${\mathbf{f}}_i=f_{x_i}\widehat{x}+f_{y_i}\widehat{y}$ (where $\widehat{x}$ is the unit vector in the longitudinal direction, and $\widehat{y}$ is the unit vector in the lateral direction), and $d_x$, $d_y$, and $d_{\varphi }$ are disturbances in the different directions, respectively.

From Equation (1), it can be seen that the vehicle motion is determined by $f_{x_i}$ and $f_{y_i}$ resulting from the wheel–ground interaction. A large number of semi-empirical complex friction models have been used to describe the wheel–ground interaction, such as the LuGre and Dugoff models. Most such models are very complex and are not necessary for the design of RL-based control strategies. In this work, we choose an appropriate friction model, described in [39], where $f_{x_i}$ and $f_{y_i}$ are defined as follows:

$$\begin{array}{cc}\hfill f_{x_i}& ={\mu }_f{N}_i{S}_f\left(s_i\right)\frac{s_{x_i}}{s_i},\hfill \\ \hfill f_{y_i}& ={\mu }_f{N}_i{S}_f\left(s_i\right)\frac{s_{y_i}}{s_i},\hfill \\ \hfill s_i& =\sqrt{s_{x_i}^2+s_{y_i}^2},\hfill \end{array}$$

(2)

where $N_i$ is the force of the driving wheel in the vertical direction, ${\mu }_f$ is the friction coefficient, $S_f\left(s_i\right)=\frac{2}{\pi }\times arctan\left(90s_i\right)$ is the dynamic feature of the friction force, $s_{x_i}$ is the longitudinal slip, and $s_{y_i}$ is the lateral slip. The slips in different directions are defined as follows:

$$\begin{array}{cc}\hfill s_{x_{lf}}=r{\omega }_{r_{lf}}-v_x+\frac{B}{2}{\omega }_{\varphi a},& s_{y_{lf}}=v_y+L_f{\omega }_{\varphi a},\hfill \\ \hfill s_{x_{lr}}=r{\omega }_{r_{lr}}-v_x+\frac{B}{2}{\omega }_{\varphi a},& s_{y_{lr}}=v_y-L_r{\omega }_{\varphi a},\hfill \\ \hfill s_{x_{rf}}=r{\omega }_{r_{rf}}-v_x-\frac{B}{2}{\omega }_{\varphi a},& s_{y_{rf}}=v_y+L_f{\omega }_{\varphi a},\hfill \\ \hfill s_{x_{rr}}=r{\omega }_{r_{rr}}-v_x-\frac{B}{2}{\omega }_{\varphi a},& s_{y_{rr}}=v_y-L_r{\omega }_{\varphi a},\hfill \end{array}$$

(3)

where ${\omega }_{r_i}$ is the wheel rotation speed. As shown in Figure 2, the dynamics model for each driving wheel can be described as:

$$j{\omega }_{r_i}^{\prime }=T_i-r{f}_{x_i}-T_{z_i}+d_{r_i},i=lf,lr,rf,rr,$$

(4)

where j is the wheel inertia, $T_i$ is the driving torque, $T_{z_i}$ is the rolling resistance torque, r is the wheel radius, and $d_{r_i}$ is the disturbance from the external environment.

Based on the definitions of the wheel dynamics model and wheel slip, the dynamics model of the wheel slip in the longitudinal direction can be described as:

$$s_{x_i}^{\prime }=\frac{r}{j}\left(T_i-r{f}_{x_i}-T_{z_i}+d_{w_i}\right)-v_{xa}^{\prime }.$$

(5)

The lateral slip speed $v_{ya}$ is very small, relative to the longitudinal speed $v_{xa}$. From an engineering perspective, we simplify the problem by disregarding the lateral motion of the vehicle. Based on the vehicle dynamics model introduced above, the objective of this paper is to propose an RL-based torque distribution strategy for skid-steering vehicles, which can minimize the errors of longitudinal speed and yaw rate, such that the desired value can be tracked well.

2.2. DDPG Algorithm

The DDPG algorithm [40] is a mature RL algorithm that can be used to deal with continuous action control. The idea of experience replays and random sampling are incorporated into the DDPG algorithm, and the process of establishing the target network is, also, considered. The DDPG is an actor–critic algorithm, where the basic framework includes actor networks and critic networks. The actor networks are used for policy search, while the critic networks are used for value function approximation. As shown in Figure 3, the DDPG algorithm includes two actor networks and two critic networks.

The online actor network, u, is used to output an action $a=u\left(s\mid {\theta }^u\right)$, according to the current state s. The online critic network, Q, is used to evaluate the action and obtain a Q-value $Q\left(s,a\mid {\theta }^Q\right)$. In the above, ${\theta }^u$ and ${\theta }^Q$ represent the parameters of the online actor network and online critic network, respectively. In addition, to follow the network update process, the DDPG also includes a target actor network $u^{\prime }$ and a target critic network $Q^{\prime }$.

The critic network is updated, by minimizing the loss function between the target and online networks. The loss function is given as follows:

$$L\left({\theta }^Q\right)=\frac{1}{N}\sum _t^N{\left(y_i-Q\left(s_i,a_i\mid {\theta }^Q\right)\right)}^2,$$

(6)

where N is the number of mini-batches, and $y_i$ is the target value, defined as

$$y_i=r_i+\tau Q^{\prime }\left(s_{i+1},u^{\prime }\left(s_{i+1}\mid {\theta }^{u^{\prime }}\right)\mid {\theta }^{Q^{\prime }}\right),$$

(7)

where $\tau$ is the discount factor, and ${\theta }^{u^{\prime }}$ and ${\theta }^{Q^{\prime }}$ are the parameters of the target actor network and target critic network, respectively. The actor networks are updated according to the following formula:

$$\begin{array}{c}\hfill {\nabla }_{{\theta }^u}u{\mid }_{s_t}\approx \frac{1}{N}\sum _t^N\left[{\nabla }_{a_t}Q\left(s_t,a_t\mid {\theta }^Q\right){\nabla }_{{\theta }^u}u\left(s_t\mid {\theta }^u\right)\right].\end{array}$$

(8)

Further, the target networks are updated as follows:

$$\begin{array}{c}\hfill {\theta }^{Q^{\prime }}\leftarrow \lambda {\theta }^Q+\left(1-\lambda \right){\theta }^{Q^{\prime }},\\ \hfill {\theta }^{u^{\prime }}\leftarrow \lambda {\theta }^u+\left(1-\lambda \right){\theta }^{u^{\prime }},\end{array}$$

(9)

where $\lambda$ is a configurable constant coefficient, such that $\lambda \ll 1$.

During the learning process of DDPG, the agent can only select actions based on the randomly generated policy, then optimize the policy through trial and error. This action may lead to sub-optimal policy actions in the overdrive system. Therefore, the agent requires the help of knowledge-assisted learning methods, to guide the learning direction in the learning process.

3. KA-DDPG-Based Direct Torque Distribution Method

3.1. Framework of KA-DDPG

The KA-DDPG algorithm is, mainly, composed of the original DDPG algorithm and knowledge-assisted methods. The considered knowledge-assisted methods include a low-fidelity controller that generates criteria actions and an evaluation method that generates guiding rewards. In this study, the low-fidelity controller is a simplified and inaccurate torque distribution method for skid-steering vehicles, which may be replaced by a torque distribution method based on optimal control theory. The evaluation method is an estimation value function, based on the ideal torque distribution feature, which was constructed through expert knowledge. The framework of KA-DDPG is shown in Figure 4.

After receiving a state s, the agent generates an action through its own policy, named the agent action $a_a$. The low-fidelity controller also generates an action based on the state s, named the criteria action $a_c$. The action that the agent actually performs in the environment is called the execution action $a_e$, which is produced by adjustment of the agent action $a_a$, with respect to the criteria action $a_c$. The reward that the agent gathers from the environment is called the observed reward $R_o$, while the reward gathered from the evaluation method is called the guiding reward $R_g$. The updating reward $R_u$ is generated by combining the observation reward $R_o$ and the guiding reward $R_g$. Through continuous interaction with the environment, the agent can quickly learn to make reasonable actions in the direction of high reward, with the help of knowledge-assisted learning methods. The help of knowledge-assisted learning methods is, gradually, eliminated over the training period.

3.2. KA-DDPG-Based Driving Torque Distribution

We introduce the KA-DDPG-based torque distribution method in this section, including the state space, action space, and reward function. The criteria action method is elaborated in the action space part, while the guiding reward method is elaborated in the reward function part.

3.2.1. State Space

The state space of a vehicle is valuable information that the agent can attain before the decision-making step, which is used to evaluate the vehicle’s situation. In this study, the control objective of the torque distribution strategy for the skid-steering vehicle is to minimize the error in desired value tracking, including minimizing the longitudinal speed error and the yaw rate error. Thus, the longitudinal speed error $v_{delta}$ and the yaw rate error ${\omega }_{delta}$ are included in the agent’s state space, defined as follows:

$$\begin{array}{cc}\hfill v_{delta}& =v_{xd}-v_{xa},\hfill \\ \hfill {\omega }_{delta}& ={\omega }_{\varphi d}-{\omega }_{\varphi a},\hfill \end{array}$$

(10)

where $v_{xd}$ and ${\omega }_{\varphi d}$ are the desired values of the longitudinal speed and yaw rate of vehicle, respectively, which are illustrated in Figure 1. The sign of the value indicates the direction of motion. In addition, the longitudinal acceleration $v_{xa}^{\prime }$ and the angular acceleration ${\omega }_{\varphi a}^{\prime }$ have a significant impact on the convergence performance of the desired value tracking error. Thus, both are included in the state space. In summary, the state space is defined as follows:

$$\mathrm{state}=\left\{v_{delta},{\omega }_{delta},v_{xa}^{\prime },{\omega }_{\varphi a}^{\prime }\right\}.$$

(11)

3.2.2. Action Space

For the drive configuration of the skid-steering vehicle described in this study, the vehicle forms a differential drivetrain, through four independently driven wheels. The wheels work in torque drive mode, and their control output is the driving torque of the wheel. Thus, the action space can be expressed as follows:

$$\mathrm{action}=\left\{T_{fl},T_{rl},T_{fr},T_{rr}\right\}.$$

(12)

During the learning process, the agent learns from previous experience, by continuously interacting with its environment, and generates reasonable actions through the powerful fitting ability of the neural network. At the beginning of learning, the actions are randomly generated by the agent, which means that actions with low rewards are likely to be generated. Although the control performance of the low-fidelity controller cannot meet the requirements, it still greatly outperforms the control of randomly selected actions. Thus, the low-fidelity controller is used to assist the agent to learn quickly, by generating criteria actions; that is, it acts like a teacher when the agent has just started learning, guiding the direction of learning. The core idea of the criteria action method is to assist the agent in searching for actions at the beginning of learning, reducing randomly selected actions. The framework of the criteria action method is shown in Figure 5.

The execution action $a_e$ is generated by adjusting the agent action $a_a$, with the help of the criteria action $a_c$. The criteria action needs to be eliminated after assisting the agent for a period of time, such that the agent can learn to select reasonable actions independently. Thus, the execution action $a_e$ is defined as follows:

$$a_e=\left(1-{\gamma }^i\right)·a_a+{\gamma }^i·a_c,$$

(13)

where $\gamma$ is the discount factor, and i is the training episode. The execution action defined in this way is the same as training the agent with a low-fidelity controller at the beginning, then slowly reducing the assistance over time, until the assistance method has no effect on the agent’s action selection, such that the agent selects the action independently.

The low-fidelity controller helps to reduce randomly selected actions for the agent and provides a direction of learning for action selection. The low-fidelity controller defined in this study is expressed by Equation (14), which may be replaced by a suitable dynamics-based torque distribution method.

$$\begin{array}{c}\hfill T_{lf}=T_{lr}=K_p\left(m·\frac{v_{delta}}{dt}-J·\frac{{\omega }_{delta}}{dt}\right),\\ \hfill T_{rf}=T_{rr}=K_p\left(m·\frac{v_{delta}}{dt}+J·\frac{{\omega }_{delta}}{dt}\right),\end{array}$$

(14)

where $K_p$ is the proportional coefficient, and $dt$ is the step size. The controller imparts the same driving torque to the wheels on the same side, which does not meet the requirements of actual control, but it is far superior to randomly selected actions.

3.2.3. Reward Function

The reward is the only feedback signal available for the agent’s learning, which is used to evaluate the action taken by the agent. The reward function designed in this paper combines a dense reward with a sparse reward, for resolving the problem of driving torque distribution. Three types of excitations are considered in the dense reward: (1) a reward $r_1$, regarding the desired value tracking errors; (2) a reward $r_2$, regarding the convergence of tracking errors; (3) and a reward $r_3$, which encourages reducing the driving torques. The specific rewards are defined as follows:

$$r_1=v_{delta}^2+5·{\omega }_{delta}^2,$$

(15)

$$r_2={\left(v_{xa}^{\prime }\right)}^2+{\left({\omega }_{\varphi a}^{\prime },\right)}^2,$$

(16)

$$r_3=T_{fl}^2+T_{rl}^2+T_{fr}^2+T_{rr}^2.$$

(17)

The dense reward part of the learning process is defined as follows:

$$r_{dense}=w_1·r_1+w_2·r_2+w_3·r_3.$$

(18)

The dense reward is a punishment that helps the agent minimize the tracking error of the desired value. In addition, the sparse reward is used as a positive reward, rewarding the agent for controlling the vehicle in an ideal state. The sparse reward is defined as follows:

$$r_{sparse}=\left\{\begin{array}{cc}3000,\hfill & \mathrm{if}\left|v_{delta}\right|<0.01\times \left|v_{xd}\right|\mathrm{and}\left|{\omega }_{delta}\right|<0.01\times \left|{\omega }_{\varphi d}\right|\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(19)

The final observed reward is the sum of the dense reward $r_{dense}$ and the sparse reward $r_{sparse}$, expressed as follows:

$$R_o=r_{dense}+r_{sparse}.$$

(20)

Unlike the original DDPG algorithm, which directly uses the observed reward as the updating reward to update the policy directly, in the KA-DDPG algorithm, we adjust the observed reward, with the help of the evaluation method, to sharpen the updating reward, which can accelerate the agent’s learning process. The guiding reward method is proposed to adjust the updating reward, and its framework is shown in Figure 6.

The core idea behind the guiding reward method is to sharpen the updating reward that the agent uses to update the policy. The guiding reward, like the criteria action, needs to eliminate the effect after a period of training. Thus, by combining the guiding reward $R_g$ and the observed reward $R_o$, the updating reward $R_u$ is defined as follows:

$$R_u=\left(1-{\gamma }^i\right)·R_o+{\gamma }^i·R_g.$$

(21)

The distance between the agent action and criteria action is used as the guiding reward, which is a simple but effective evaluation method. The agent action and criteria action are both four-dimensional vectors. The guiding reward is defined as:

$$R_g=-\sqrt{{\left(a_a-a_c\right)}^T\left(a_a-a_c\right)}.$$

(22)

The KA-DDPG algorithm is summarized in Algorithm 1.

Algorithm 1. KA-DDPG algorithm.

1: Initialize critic network $Q\left(s,a\mid {\theta }^Q\right)$ and actor network $u\left(s\mid {\theta }^u\right)$; 2: Initialize target network $Q^{\prime }$ and $u^{\prime }$ with same weights; 3: Initialize replay buffer $RB$; 4: for  $episode=1,\cdots ,M$  do 5:        Receive initial observation state $s_0$; 6:        for $t=1,\cdots ,T$ do 7:             Select agent action $a_{a,t}=u\left(s_t\mid {\theta }^u\right)+N_t$; 8:             Get criteria action $a_{c,t}$; 9:             Calculate execution action: $a_{e,t}=\left(1-{\gamma }^{episode}\right)·a_{a,t}+{\gamma }^{episode}·a_{c,t}$; 10:           Get observed reward $R_{o,t}$; 11:           Get guiding reward $R_{g,t}$; 12:           Calculate updating reward: $R_{u,t}=\left(1-{\gamma }^{episode}\right)·R_{o,t}+{\gamma }^{episode}·R_{g,t}$; 13:           Store transition $〈s_t,a_{e,t},R_{u,t},s_{t+1}〉$ in $RB$; 14:           Sample a random mini-batch of N transitions from $RB$; 15:           Set $y_i=r_i+\gamma Q^{\prime }\left(s_{i+1},u^{\prime }\left(s_{i+1}{\theta }^{u^{\prime }}\right)\mid {\theta }^{Q^{\prime }}\right)$; 16:           Update critic by minimizing the loss: $L\left({\theta }^Q\right)=\frac{1}{N}{\sum }_t^N{\left(y_i-Q\left(s_i,a_i\mid {\theta }^Q\right)\right)}^2$; 17:           Update the actor policy, using the sampled policy gradient:            ${\nabla }_{{\theta }^u}u{\mid }_{s_t}\approx \frac{1}{N}{\sum }_t^N\left[{\nabla }_{a_t}Q\left(s_t,a_t\mid {\theta }^Q\right){\nabla }_{{\theta }^u}u\left(s_t\mid {\theta }^u\right)\right]$; 18:           Update the target networks $Q^{\prime }$ and $u^{\prime }$.

4. Simulation Environment Settings

In the simulations, we considered the dynamics model of skid-steering vehicles with four independent driving wheels, which includes vehicle body dynamics, wheel dynamics, and the wheel–ground interaction model. It is important to note that we consider the vehicle as running on flat ground with friction coefficient ${\mu }_f=0.85$.

Table 1. Vehicle dynamics parameters.

Description	Symbol	Value
Vehicle mass	m	$2360\mathrm{kg}$
Vehicle width	B	$2.4\mathrm{m}$
Distance from the front axle to CG	$L_f$	$2.5\mathrm{m}$
Distance from the front axle to CG	$L_r$	$2.5\mathrm{m}$
Wheel radius	r	$0.2\mathrm{m}$
Vehicle inertia around CG	J	$4050\mathrm{kg}·{\mathrm{m}}^2$
Friction coefficient	${\mu }_f$	$0.85$
Rotation moment of wheel	j	$0.85\mathrm{kg}·{\mathrm{m}}^2$
Rolling resistance torque	$T_{Z_i}$	$11.8\mathrm{N}·\mathrm{m}$
Max torque of wheel	$T_{max}$	$100\mathrm{N}·\mathrm{m}$

provides the detailed vehicle dynamics settings used in the simulations.

The KA-DDPG algorithm was implemented on the PyCharm IDE with Python 3.7, and ran on an Intel Core i5 computer. Based on the definition of the agent, the actor network and its target network were constructed, using two $4\times 512\times 512\times 4$ fully connected neural networks, and the critic network and its target network were constructed using two $8\times 512\times 512\times 1$ fully connected neural networks. The structures of the neural networks are shown in Figure 7, and the detailed parameters of the KA-DDPG algorithm are listed in

Table 2. Hyperparameters of the KA-DDPG algorithm.

Parameters	Value
Random seed	2
Max episode	5000
Max steps per episode	100
Step size	$0.1$
Memory capacity	1,000,000
Batch size	512
Actor network learning rate	$0.0003$
Critic network learning rate	$0.001$
Exploration noise scale	$0.1$
Soft update rate	$0.01$
Discount factor	$0.999$
Proportional coefficient	$0.05$

.

Based on the simulation environment design described above, training was carried out for a total of 5000 episodes, and the designed agent learned the torque distribution strategy. During training, the vehicle state was randomly initialized in each episode. Due to the two newly introduced knowledge-assisted learning methods, the training of the KA-DDPG converged quickly and stably. After completing the learning, only the parameters of the actor network were retained, which then received the current state of the vehicle in real-time and generated the optimal distributing action, in order to realize control of the vehicle.

5. Results and Discussion

Simulations were first conducted to demonstrate the control performance of the KA-DDPG-based torque distribution method. The vehicle behaviors are discussed under different scenarios, including a straight scenario and a cornering scenario. Furthermore, we verified the contributions of the knowledge-assisted learning methods in the learning process of the KA-DDPG algorithm, through three different cases.

5.1. Effectiveness of KA-DDPG

In order to verify the control performance of the KA-DDPG-based torque distribution method for skid-steering vehicles, the simulations were designed with two different scenarios: a straight scenario and a corning scenario. The low-fidelity controller defined in Section 3.2.2 was considered as the baseline for comparative experiments, which is a controller based on physical knowledge. We introduced an evaluation method that is, commonly, used to quantitatively evaluate longitudinal speed and yaw rate tracking performance. This evaluation method uses the integral of the quadratic function of the tracking error of the longitudinal speed and yaw rate, to evaluate the tracking performance. The integrals $J_1$ and $J_2$ are, respectively, expressed as follows [41]:

$$\begin{array}{c}\hfill J_1={\int }_0^t{v}_{delta}^{2}d\tau ,\\ \hfill J_2={\int }_0^t{\omega }_{delta}^{2}d\tau .\end{array}$$

(23)

In the straight scenario, the desired yaw rate was always kept at 0 rad/s, in order to identify the drift of the skid-steering vehicle. The desired longitudinal speed changed during the simulation, which was set to $3.8$ m/s at the beginning of the simulation, to $1.8$ m/s and $3.2$ m/s at 200 s and 400 s, respectively, and, then, kept at $3.2$ m/s until the end. The simulation results for the straight scenario are shown in Figure 8.

Figure 8a shows a comparison of longitudinal speed tracking, for the different torque distribution methods in the straight scenario. To more clearly demonstrate the tracking performance for longitudinal speed, Figure 8b shows the corresponding tracking error. The tracking error of the baseline method fluctuated greatly, and the longitudinal speed tracking performance was significantly worse than that of KA-DDPG. The tracking error of the baseline frequently exceeded $0.2$ m/s, while the maximum error of KA-DDPG was always stably kept within $0.2$ m/s. Figure 8c shows a comparison of the yaw rate tracking performance in the straight scenario. It can be emphasized that the target yaw rate is a constant value in the straight scenario, which is always 0 rad/s. The tracking results show that KA-DDPG kept the tracking error of the yaw rate within $0.2$ rad/s, while the maximum tracking error of the baseline exceeded $0.2$ rad/s. As shown in Figure 8d, the heading of the baseline was larger than that of KA-DDPG, indicating that the vehicle controlled by the baseline suffered from greater drift. These simulation results demonstrate that the tracking performance of the KA-DDPG algorithm, for the longitudinal speed and yaw rate in the straight scenario, was better than that of the baseline.

Figure 9 shows the torque distribution curves for each wheel in the straight scenario, and the torque distributions of the two different methods had similar trends. However, the driving torques of the KA-DDPG algorithm reflect the vehicle’s situation more quickly than the baseline, which explains why the KA-DDPG algorithm presented better tracking performance than the baseline.

To verify the cornering ability of the KA-DDPG-based torque distribution method, the simulations were conducted in a cornering scenario. Compared to the simulations in the straight scenario, the tracking performance of the proposed torque distribution method was more rigorously verified in the cornering scenario. The longitudinal speed of the vehicle was set to $1.8$ m/s at the beginning of the simulation, $3.7$ m/s at 150 s, $2.5$ m/s at 350 s, and, then, held at $2.5$ m/s until the end. The yaw rate was no longer a constant value, which was set to $3.2$ rad/s at the beginning of the simulation, $2.5$ rad/s at 150 s, $3.7$ rad/s at 350 s, and, then, held at $3.7$ rad/s until the end of the simulation. The tracking performances of the different methods in the cornering scenario are shown in Figure 10. Figure 10a shows the longitudinal speed tracking results in the cornering scenario, and it can be seen, intuitively, that the longitudinal speed tracking performance did not deteriorate significantly, compared with the straight scenario. Figure 10b shows the tracking errors of the longitudinal speed in the cornering scenario. The maximum error of the baseline exceeded $0.2$ m/s, while the maximum error of the KA-DDPG method was kept within $0.2$ m/s. The yaw rate tracking performance of different methods is shown in Figure 10c, while Figure 10d shows the tracking errors of the yaw rate in the cornering scenario. The KA-DDPG method kept the yaw rate error within $0.2$ rad/s, which was significantly smaller than that of the baseline. In the cornering scenario, the tracking performance of the KA-DDPG method was still better than that of the baseline, which is the same as the conclusion in the straight scenario, indicating that the tracking performance did not deteriorate in a more complex scenario.

Figure 11 shows the torque distribution curves, for each wheel in the cornering scenario. The torque distributions of the different methods in the cornering scenario showed the same trend as in the straight scenario, and the driving torques of the KA-DDPG method changed more quickly than those of the baseline method.

To quantitatively illustrate the tracking performance more specifically, Equation (23) was introduced, to evaluate the tracking performance of the longitudinal speed and yaw rate. The quantitative evaluation results for the simulations in the straight scenario and the cornering scenario discussed above are displayed in Figure 12.

Figure 12a shows the quantitative evaluation in the straight scenario. In 500 s of straight driving, $J_1$ in the KA-DDPG method was reduced by $71.36\%$, compared to the baseline. Similarly, $J_2$ of the KA-DDPG method was reduced by $87.33\%$, compared to the baseline. The quantitative evaluation demonstrated that the KA-DDPG method has better tracking performance for the longitudinal speed and yaw rate than the baseline in the straight scenario, consistent with the results of the above analysis. The quantitative evaluation in the cornering scenario is shown in Figure 12b. Compared with the evaluation results of the baseline, in the simulation under the cornering scenario, $J_1$ of the KA-DDPG method was reduced by $67.87\%$, and $J_2$ was reduced by $80.99\%$, when compared to the baseline. These quantitative evaluations in different scenarios demonstrate that the KA-DDPG method has better desired value tracking performance than the baseline.

Based on the analysis of the simulation results, the tracking performance of skid-steering vehicles, based on the KA-DDPG method, was investigated. Compared with the baseline, the KA-DDPG-based torque distribution method showed better tracking performance in different scenarios. Although the baseline is a low-fidelity controller, it was sufficient to illustrate that the KA-DDPG method can be successfully applied to the torque distribution problem of skid-steering vehicles.

5.2. Contributions of Knowledge-Assisted Learning Methods

To verify the contributions of the knowledge-assisted learning methods in the learning process of KA-DDPG, we trained the KA-DDPG algorithm, with three cases. The configurations of these cases were as follows: (1) the KA-DDPG, including the criteria action and guiding reward methods (i.e., the algorithm proposed in this work); (2) the KA-DDPG, only including the guiding reward method; and (3) the KA-DDPG, only including the criteria action method. As the skid-steering vehicle studied in this paper is an overdrive system, applying the original DDPG may cause the agent to search in a wrong direction, leading to a sub-optimal solution. Therefore, we do not consider the torque distribution method, based on the original DDPG, in this section. In each case, the KA-DDPG was trained for 5000 episodes. During training, the vehicle state was randomly initialized in each episode. To verify the stability of the convergence, the training in each case was repeated five times.

Figure 13 shows the total rewards during the learning process of the first case, which is the case of KA-DDPG with both of the assisted learning methods. As shown in Figure 13a, the rewards converged quickly and smoothly in each learning process. Such learning processes indicate that the agent can still stably learn a reasonable torque distribution strategy, to ensure the control performance. With the assistance of knowledge-assisted learning methods, the agent no longer randomly generates actions with low rewards during the learning process, and has the ability to rapidly increase the cumulative rewards, which not only accelerated the convergence of the learning process, but also ensured that the convergence was smooth. As shown in Figure 13b, on average, the KA-DDPG-based torque distribution method outperformed the baseline in cumulative rewards after training for 200 episodes, which means that KA-DDPG not only learns control policies from the low-fidelity controller, but also explores and learns policies for better control performance. At about 300 episodes, each learning process converged smoothly.

Then, we trained the KA-DDPG, including only the guiding reward method and without the criteria action method, in order to illustrate the contribution of the criteria action method. Figure 14 shows the learning process of the KA-DDPG, without the assistance of the criteria action method. Without the assistance of criteria actions, execution actions were searched randomly at the beginning of the learning process, making the rewards more volatile. As the number of episodes increased, the volatility of the total rewards decreased, and the learning process converged stably. Figure 14a shows the total rewards in each learning process, and Figure 14b shows the average of the total rewards. Without the assistance of the criteria action method, the KA-DDPG needed about 800 episodes to perform better than the baseline, while the first case (including both assisted learning methods) only took about 200 episodes to achieve the same result. Similarly, the convergence of KA-DDPG, without the criteria action method, was, also, slower than the first case, taking about 1200 episodes to converge. Such results illustrate that the learning cost of KA-DDPG is greatly increased without the assistance of the criteria action method. The above simulation results demonstrate that the criteria action method reduces the learning cost, by helping the agent to reduce the randomly selected actions in the learning process.

Finally, we trained the KA-DDPG, including only the criteria action method, in order to illustrate the contribution of the guiding reward method. Figure 15 shows the learning process of the KA-DDPG algorithm, without sharpening the reward through use of the guiding reward method. Compared to the first case, KA-DDPG, without the guiding reward method, had a larger reward at the beginning of the learning process, but it grew more slowly and showed more volatility. The simulation results demonstrated that the guiding reward method in KA-DDPG led the learning process to converge faster, by sharpening the reward function. Figure 15a shows the total rewards of KA-DDPG, without the guiding reward method in each learning process, while Figure 15b shows the average of the total rewards. Without the assistance of the guiding reward method, KA-DDPG needed about 500 episodes to outperform the baseline, while only about 200 episodes were needed to achieve the same control performance, when using both assisted learning methods. Similarly, KA-DDPG, without the guiding reward method, needed about 800 episodes to converge, which was also slower. These simulation results demonstrate that the guiding reward method, also, has the ability to reduce the learning cost. However, unlike the criteria action method, which reduces the learning cost by reducing randomly selected actions, the guiding reward method achieves this purpose by sharpening the reward function.

To summarize, the simulation results presented above demonstrate that both of the assisted learning methods proposed in this paper can reduce the learning cost of the KA-DDPG algorithm. The guiding reward method accelerates the learning process, by sharpening the updating reward with the assistance of the evaluation method, whereas the criteria action method achieves this through providing the learning direction by criteria actions, thus reducing the randomly searched actions.

5.3. Discussion

To verify the KA-DDPG based torque distribution strategy, we conducted the desired value tracking performance evaluation in two different scenarios, by comparing with the controller used for assisted learning. The contribution of the assisted learning methods was, also, verified by comparing the learning process of different cases of the KA-DDPG algorithm.

For the desired value tracking performance evaluation, quantitative evaluations in different scenarios show that the KA-DDPG method had smaller desired value tracking errors than the baseline. In other words, the KA-DDPG method had a better desired value tracking performance. The great improvements in the tracking performance come from the fact that the KA-DDPG algorithm not only learned from the knowledge-assisted learning methods but also explored the better distribution strategy, through the exploration ability of RL.

For the contribution of knowledge-assisted learning methods evaluation, we trained the KA-DDPG algorithm, with different configurations on three cases, namely the KA-DDPG algorithm with two assisted learning methods and the KA-DDPG algorithm with only one of the assisted learning methods. The result is the KA-DDPG with two assisted learning methods converged with less learning time than the other cases. The great improvements in the learning process come from the knowledge-assisted learning methods. The guiding reward method accelerates the learning process, by sharpening the updating reward with the assistance of the evaluation method, whereas the criteria action method achieves this by providing the learning direction through criteria actions, reducing the randomly searched actions.

From all the simulation results, we conclude that the KA-DDPG algorithm, which combines the knowledge-assisted learning methods with the DDPG algorithm, can be successfully applied to the torque distribution of skid steering vehicles. This work lays a foundation for the use of RL technologies to directly distribute the torque of skid-steering vehicles, in future practical applications. The knowledge-assisted RL framework proposed in this work provides a powerful weapon for applying RL technologies in overdrive systems, such as skid-steering vehicles. However, the verification part of this paper was carried out in a simulation environment, so challenges do still exist to transfer the proposed method to a real application. For future research, we will consider reducing the reality gap in a real application.

6. Conclusions

In this study, a KA-DDPG-based torque distribution strategy for skid-steering vehicles was proposed, in order to minimize the tracking error of the desired value, which included the longitudinal speed and yaw rate, making the considered problem a dual-channel control problem. The KA-DDPG algorithm combines the DDPG algorithm with knowledge-assisted learning methods, constructing a knowledge-assisted learning framework that combines analytical methods with an RL algorithm. Two knowledge-assisted methods were introduced into the KA-DDPG algorithm: a criteria action method and a guiding reward method.

In order to validate the proposed strategy, simulations were first conducted in different scenarios, including a straight scenario and a cornering scenario. The tracking performance simulation results demonstrated that the proposed torque distribution strategy had an excellent control performance in different scenarios. In addition, simulations were conducted to verify the contributions of the assisted learning methods in the learning process of KA-DDPG. The simulation results illustrated that both of the proposed assisted learning methods helped to accelerate the agent’s learning process. The criteria action method provides the learning direction for the agent and speeds up the learning speed, by reducing the random search actions, while the guiding reward method achieves the same result by sharpening the reward function.

This work opens an exciting path for the use of RL algorithms in the problem of torque distribution in skid-steering vehicles. However, there are still some areas that require more in-depth study. The verification section of this research was carried out in a simulation environment, without any experiments being in a real environment. Wheel slip limits, which are important for skid-steering vehicle control, were not considered in this work. The lateral movement of the vehicle, which is unavoidable in actual applications, was, also, not considered in this work. At present, we are exploring these unsolved problems to extend this work.