A DDQN Path Planning Algorithm Based on Experience Classification and Multi Steps for Mobile Robots

Abstract

Constrained by the numbers of action space and state space, Q-learning cannot be applied to continuous state space. Targeting this problem, the double deep Q network (DDQN) algorithm and the corresponding improvement methods were explored. First of all, to improve the accuracy of the DDNQ algorithm in estimating the target Q value in the training process, a multi-step guided strategy was introduced into the traditional DDQN algorithm, for which the single-step reward was replaced with the reward obtained in continuous multi-step interactions of mobile robots. Furthermore, an experience classification training method was introduced into the traditional DDQN algorithm, for which the state transition generated by the mobile robot–environment interaction was divided into two different types of experience pools, and experience pools were trained by the Q network, and the sampling proportions of the two experience pools were updated through the training loss. Afterward, the advantages of a multi-step guided DDQN (MS-DDQN) algorithm and experience classification DDQN (EC-DDQN) algorithm were combined to develop a novel experience classification multi-step DDQN (ECMS-DDQN) algorithm. Finally, the path planning of these four algorithms, including DDQN, MS-DDQN, EC-DDQN, and ECMS-DDQN, was simulated on the OpenAI Gym platform. The simulation results revealed that the ECMS-DDQN algorithm outperforms the other three in the total return value and generalization in path planning.

1. Introduction

Path planning for mobile robots refers to planning an optimal or near-optimal collision-free path from the start point to the endpoint according to the given obstacle environment, which is a complex problem of decision optimization with little prior information [1,2]. The common path planning algorithms include Dijkstra’s algorithm [3], A* algorithm [4], D* algorithm and genetic algorithm [5], particle swarm optimization [6], and neural network algorithm [7]. However, as the work environment of mobile robots changes, it becomes an urgent demand for robots to adapt to the complex and ever-changing environment, so certain disadvantages of these algorithms are exposed in this situation [8,9].

As one of the methodologies of machine learning, reinforcement learning is used to describe and solve the problem of how agents learn and optimize strategies during their interaction with the environment. Its strength lies in solving the path planning problem of mobile robots with limited a priori information [10]. However, traditional reinforcement learning algorithms, such as Q-learning, which are limited by action space and sample space, cannot be applied to practical scenes with a large state space and continuous action space [11]. Deep learning is a major branch of machine learning in artificial intelligence. After training, it is able to identify the regular information of sample data and help explain the internal correlation of data. Deep reinforcement learning obtained by combining deep learning with reinforcement learning can solve the problems faced by Q-learning and other algorithms in path planning [12]. In the literature [13], a Deep Q Network (DQN) algorithm was proposed for the improvement of learning strategies based on different requirements for the depth and breadth of experience in different learning stages of intelligent path planning. However, the modeling of its reward function faces certain limitations when it is used in a complex environment. To overcome this problem, the initialization of robots in each round of training was redefined and an improved double DQN (DDQN) algorithm was developed to solve the problem of insufficient experience [14]. In terms of dynamic and complex environments, the DDQN algorithm was used for path planning of unmanned aerial vehicles (UAVs), reducing the energy consumed during UAV flight and computing operations [15]. Given the survival probability of UAVs in an enemy-threatened dynamic environment, a fast evaluation model was built and a deep reinforcement learning method for UAV path planning based on global situation information was put forward by plotting the situation map corresponding to the overall threat value of each enemy to the UAV [16]. Aiming at the instability of the training phase and the sparsity of the environment state space, a specific training method and its reward function were designed, effectively solving the non-convergence problem due to the reward sparsity in the state space [17]. For higher accuracy of the target Q value during the training process, heuristic knowledge was introduced into reinforcement learning, thereby improving the efficiency of path planning and obstacle avoidance of the DQN algorithm [18]. Through the network parameter training based on the dynamic fusion of the a priori knowledge of DDQN and average DQN, a dynamic fusion DDQN algorithm, dynamic target DDQN (DT-DDQN), was designed, narrowing the gap between the Q value output by the network and the real one and reducing the influences of the estimation on the action choices of robots [19]. The evaluation function of DQN was improved by the correction function to increase the evaluation accuracy of the value function of the algorithm [20]. Deep reinforcement learning was applied to path planning of mobile robots in unknown dynamic environments [21], where targeting the problem of mutual collision triggered by abnormal rewards due to the relative motion of obstacles and robots, two reward thresholds were set to modify the abnormal rewards, thus realizing optimal decision learning. The experience playback mechanism was combined with Q-learning [22] and the soft actor–critic (SAC) algorithm [23], respectively, improving the sample utilization rate, the learning efficiency and convergence speed of the algorithm in three-dimensional (3D) path planning of UAVs and path planning of multi-arm robots. As for the problem of uniform sampling with moderate probability during experience playback, an improved SAC algorithm, prioritized experience replay (PER)-SAC algorithm was designed, increasing the convergence speed of the algorithm [24]. The DQN algorithm was improved by DDQN with PER, where the degree of intelligent ship driving was raised to accelerate the learning process [25].

According to the above analysis, the reinforcement learning algorithm was improved in the literature aforementioned mainly based on the temporal difference reinforcement learning error of state transition in experience or the network with the approximate value function. However, the influences of the experience characteristics on algorithm training were ignored. In this paper, algorithm improvement and fusion were completed from the two perspectives of value function error and experience, generating an experience classification multi-step DDQN (ECMS-DDQN) path planning algorithm that combines multi-step guidance and experience classification to improve the adaptivity of path planning of mobile robots.

Firstly, based on the DDQN algorithm, the single-step reward was replaced with the reward obtained by the continuous multi-step interaction of mobile robots to enable these robots to obtain a more accurate target state during training. Secondly, based on the DDQN algorithm, the state transition generated by the interaction between mobile robots and the environment was divided into two different types of experience pools, which were sampled and stitched in different proportions. The stitched state was trained by the Q network, and the sampling proportions of the two experience pools were updated through the training loss to improve the training speed and experience utilization of mobile robots. Then, the above two improvements were combined to develop a novel ECMS-DDQN path planning algorithm. Finally, the effectiveness of the proposed algorithm was validated through the path planning experiments of mobile robots designed in two different environments.

The rest of the paper is structured as below: The theoretical basis of DDQN is illustrated in Section 2. A detailed description of the MS-DDQN algorithm and EC-DDQN algorithm, as well as the ECMS-DDQN algorithm formulated by combining the two, is given in Section 3. Section 4 is focused on different experimental environments built to validate and analyze the performance of the proposed algorithm. Finally, the paper is concluded in Section 5.

2. DDQN Algorithm

2.1. DQN Algorithm

In the DQN algorithm, the input of Q network is the state feature $\varphi \left(S\right)$ corresponding to the state information $S$ observed by agents. The output is the estimated value of action value $Q(s,a)$ of state when all available actions $a\in A$ are executed in the same state. Generally, lots of data are needed in convergence of the neural network, which requires the convergence of Q network to reach the optimal state, as the action value function does. Therefore, the DQN algorithm cannot update every step like the Q-learning algorithm. Concerning this problem, experience replay techniques were introduced into the DQN algorithm to realize the convergence of the value function [26], which is also one of the core characteristics of the DQN algorithm.

Two networks with the same structure, namely, critic network Q and target network $\widehat{Q}$, were used in the DQN algorithm. Critic network Q was used to select actions and update parameters, while target network $\widehat{Q}$ was used to calculate the target Q value. The parameters of critic network Q were updated through iterations, while parameters of target network $\widehat{Q}$ were updated by copying parameter $w$ of critic network Q to parameter $w^{-}$ of target network $\widehat{Q}$ at intervals. For state transition $\left(\varphi \left(S\right),A,R,\varphi \left(S^{\prime }\right),is\_done\right)$, target value $Q_{target}\left(\varphi \left(S\right),A\right)$ equals the outcome of multiplying the selected maximum action value $Q\left(\varphi \left(S^{\prime }\right),A^{\prime },w\right)$ in the state $S$ by the sum of instant reward of post-state $S$ and the product of expectation attenuation parameters $\gamma$. The target value used in parameter updating as shown in Equation (1):

$$Q_{target}\left(\varphi \left(S\right),A\right)=\{\begin{array}{cc}R& is\_done\mathrm{is}\mathrm{true}\\ R+\gamma {\max }_{a^{\prime }}\widehat{Q}\left(\varphi \left(S^{\prime }\right),A^{\prime },w^{-}\right)& is\_done\mathrm{is}\mathrm{false}\end{array}$$

(1)

In Equation (1), $a^{\prime }\in A$ refers to the action that maximizes the value of post-state $Q\left(\varphi \left(S^{\prime }\right),A^{\prime },w\right)$ and $is\_done$ refers to whether the current state is the terminal state.

Mean-square deviation cost function is usually used in the DQN algorithm to calculate the network error. The expression of cost function is shown in Equation (2):

$$J(\omega )=\frac{1}{2M}{\displaystyle \sum _{t=1}^M{\left[r_t-y_t\right]}^2}$$

(2)

In Equation (2), $r_t$ is target value, and $y_t$ is output value through network. Plug the Equation (1) into Equation (2), and the network error between target value and output value through the network can be calculated, as shown in Equation (3):

$$J(\omega )=\frac{1}{2M}{\displaystyle \sum _{t=1}^M{\left[R_t+\gamma Q({S^{\prime }}_t,{A^{\prime }}_t,w)-Q(S_t,A_t,w)\right]}^2}$$

(3)

Through back propagation with gradient, parameter $w$ of neural network was updated and Q network was converged in the end. The parameter $w$ was updated, as shown in Equation (4):

$$w_{t+1}=w_t+\alpha \left[R_{t+1}+\gamma \underset{a}{\max Q(S_{t+1},a,w_t)-Q(S_t,A_t,w_t)}\right]{\nabla }_{w_t}Q(S_t,A_t,w_t)$$

(4)

The pseudo form of DQN algorithm is shown in Algorithm 1.

Algorithm 1: Pseudo form of the DQN algorithm

Input: ${\epsilon }_{\mathrm{init}}$, ${\epsilon }_{\min }$, $\mathrm{T}$, $\alpha$, $\gamma$, $m$, $N$, $n$, $A$, $Q(w)$, $Q_{\mathrm{target}}(w)$, $C$

for t = 1 to T do

Initialization: Initialize the first state $S$ in the present state sequence and acquire its eigenvector $\varphi (S)$

Update the basic exploration factor ${\epsilon }_t$ based on the number of iteration episodes $t$

for the each initialized first state $S$ do

(a) Select an action $A={\mathrm{argmax}}_{a}Q(\varphi (S),A,w)$ as per the probability $1-{\epsilon }_t$, or randomly select an action $A$

(b) Execute the action $A$ under the state $S$ to acquire an instant reward $R$, and obtain a new state $S^{\prime }$ and its eigenvector $\varphi (S^{\prime })$. $is\_done$ refers to whether the state is terminated

(c) Store the quintuple $\left\{\varphi (S),A,R,\varphi (S^{\prime }),is\_done\right\}$ in an experience container D

(d) $S=S^{\prime }$

(e) Randomly collect $m$ state transition samples (sample size >$N$) from the experience container D, and calculate the present $Q_{\mathrm{target}}(y_j)$: $$y_j=\{\begin{array}{cc}{R}_j& is\_don{e}_j\mathrm{is}\mathrm{true}\\ R_j+\gamma {\max }_{a^{\prime }}\widehat{Q}(\varphi (S^{\prime }),{A^{\prime }}_j,w^{-})& is\_don{e}_j\mathrm{is}\mathrm{false}\end{array}$$

(f) Use the mean square error loss function $\frac{1}{m}{{\displaystyle \sum _{j=1}^m\left(y_i-Q(\varphi (S),A_j,w)\right)}}^2$ to update the network weight $w$ via the small-batch gradient descent method

Update the target network parameter $Q_{\mathrm{target}}(w^{\prime })$ every $C$ steps

Until $S$ ends in an is_done state

end for

Until reaching the maximum number of iteration episodes $\mathrm{T}$

end for

Output: $Q_{\mathrm{target}}(w^{\prime })$

2.2. DDQN Algorithm

Like the Q-learning algorithm, the target value in the DQN algorithm can be directly calculated through the greedy algorithm. Although Q value can quickly be closer to the possible optimization target in this way, the target Q value will be overestimated and the deviation will be large in the algorithm model. This problem can be solved in the DDQN algorithm [27].

In the DDQN algorithm, overestimation can be eliminated by decoupling the action selection and calculation of the target Q value. The key is the full use of the double network structure. In updating parameters of the target Q network through the DDQN algorithm, the critic Q network was firstly used to select the action $a^{ma{x}^{\prime }}$ with an optimal value, as shown in Equation (5):

$$a^{\max }(S^{\prime },w)=arg\underset{a^{\prime }}{\max }Q(\varphi (S^{\prime }),a,w)$$

(5)

This action is used to calculate the target Q value in target network $\widehat{Q}$, as shown in Equation (6):

$$Q_{target}(\varphi (S),A)=\begin{array}{cc}R+\gamma \widehat{Q}(\varphi (S^{\prime }),a^{\max }(S^{\prime },w),w^{-})& (is\_done\mathrm{is}\mathrm{false}\end{array})$$

(6)

For the state transition $\left(\varphi \left(S\right),A,R,\varphi \left(S^{\prime }\right),is\_done\right)$, the target value is used to update parameters through the DDQN algorithm, as shown in Equation (7):

$$Q_{target}\left(\varphi \left(S\right),A\right)=\{\begin{array}{cc}R& is\_done\mathrm{is}\mathrm{true}\\ R+\gamma \widehat{Q}\left(\varphi \left(S^{\prime }\right),arg\underset{a^{\prime }}{max}Q\left(\varphi \left(S^{\prime }\right),a,w\right),w^{-}\right)& is\_done\mathrm{is}\mathrm{false}\end{array}$$

(7)

The pseudo form of DDQN algorithm is shown in Algorithm 2.

Algorithm 2: Pseudo form of the DDQN algorithm

Input: ${\epsilon }_{\mathrm{init}}$, ${\epsilon }_{\min }$, $\mathrm{T}$, $\alpha$, $\gamma$, $m$, $N$, $n$, $A$, $Q(w)$, $Q_{\mathrm{target}}(w)$, $C$

for t = 1 to T do

Initialization: Initialize the first state $S$ in the present state sequence and acquire its eigenvector $\varphi (S)$

Update the basic exploration factor ${\epsilon }_t$ based on the number of iteration episodes $t$

for the each initialized first state $S$ do

(a) Select an action $A={\mathrm{argmax}}_{a}Q(\varphi (S),A,w)$ as per the probability $1-{\epsilon }_t$, or randomly select an action $A$

(b) Execute the action $A$ under the state $S$ to acquire an instant reward $R$, obtain a new state $S^{\prime }$ and its eigenvector $\varphi (S^{\prime })$. $is\_done$ refers to whether the state is terminated

(c) Store the quintuple $\left\{\varphi (S),A,R,\varphi (S^{\prime }),is\_done\right\}$ in an experience container D

(d) $S=S^{\prime }$

(e) Randomly collect $m$ state transition samples from the experience container D if the total sample size in the container is greater than $N$, and calculate the present $Q_{\mathrm{target}}(y_j)$: $$y_j=\{\begin{array}{cc}{R}_j& is\_don{e}_j\mathrm{is}\mathrm{true}\\ R_j+\gamma \widehat{Q}(\varphi (S^{\prime }),arg{\max }_{a^{\prime }}Q(\varphi (S^{\prime }),a,w),w^{-})& is\_don{e}_j\mathrm{is}\mathrm{false}\end{array}$$

(f) Use the mean square error loss function $\frac{1}{m}{{\displaystyle \sum _{j=1}^m\left(y_i-Q(\varphi (S),A_j,w)\right)}}^2$ to update the network weight $w$ via the small-batch gradient descent method

Update the target network parameter $Q_{\mathrm{target}}(w^{\prime })$ every $C$ steps

Until $S$ ends in an $is\_done$ state

end for

Until reaching the maximum number of iteration episodes $\mathrm{T}$

end for

Output: $Q_{\mathrm{target}}(w^{\prime })$

3. DDQN-ECMS Algorithm

3.1. Multi Step DDQN Algorithm

In the DDQN algorithm, the instant reward $R$ obtained in current action and estimated value in the next moment was used to calculate the target Q value. In the early stage when the deviation of network parameters was large, the target Q value also showed a large deviation, so the speed of training was slow. The idea of multi-step was introduced for paper to improve the performance of the DDQN algorithm. Specifically, instant reward $R^{\left(n\right)}$ obtained in successive MS was used to substitute the instant reward $R$ ($n$ is a step-size parameter) in the DDQN algorithm to obtain the more accurate target Q value in the training. This algorithm is named the multi step DDQN (MS-DDQN) algorithm, $R^{\left(n\right)}$, as shown in Equation (8):

$$R^{(n)}={\displaystyle \sum _{i=0}^{n-1}{\gamma }_t^{(i)}{R}_{t+i+1}}$$

(8)

For n sequences of state transition $\left(\varphi \left(S_i\right),A_i,R_i,\varphi \left(S_i^{\prime }\right),is\_done\right) \left(i=t,t+1,\dots ,t+n\right)$, the target value was used by the algorithm in parameter updating, as shown in Equation (9):

$$Q_{target}(\varphi (S),A)=\{\begin{array}{cc}{R}_t^{(n)}& is\_don{e}_{t+n}\mathrm{is}\mathrm{true}\\ R_t^{(n)}+{\gamma }_t{}^{(n)}{Q}_{\arg \max }& is\_don{e}_{t+n}\mathrm{is}\mathrm{false}\end{array}$$

(9)

In the of Equation (9), the value of step-size parameter $n$ should be set based on the real situation, which can effectively improve the training speed of the algorithm.

The pseudo form of MS-DDQN algorithm is shown in Algorithm 3.

Algorithm 3: Pseudo form of the MS-DDQN algorithm

Input: ${\epsilon }_{\mathrm{init}}$, ${\epsilon }_{\min }$, $\mathrm{T}$, $\alpha$, $\gamma$, $m$, $N$, $n$, $n_s$, $A$, $Q(w)$, $Q_{\mathrm{target}}(w)$, $C$

for t = 1 to T do

Initialization: Initialize the first state $S$ in the present state sequence and acquire its eigenvector $\varphi (S)$

Update the basic exploration factor ${\epsilon }_t$ based on the number of iteration episodes $t$

for the each initialized first state $S$ do

(a) Select an action $A={\mathrm{argmax}}_{a}Q(\varphi (S),A,w)$ as per the probability $1-{\epsilon }_t$, or randomly select an action $A$

(b) Execute the action $A$ under the state $S$ to acquire an instant reward $R$, obtain a new state $S^{\prime }$ and its eigenvector $\varphi (S^{\prime })$. $is\_done$ refers to whether the state is terminated

(c) Store the quintuple $\left\{\varphi (S),A,R,\varphi (S^{\prime }),is\_done\right\}$ in a multi-step guidance queue container D. (When len(D) = $n_s$, the first quintuple in the queue pops up firstly)

(d) When len(D) = $n_s$, store the quintuple $\left\{\varphi (S{)}^{\prime },A^{\prime },R^{\prime },\varphi (S^{\prime }{)}^{\prime },is\_don{e}^{\prime }\right\}$ into the experience container M. $\varphi (S{)}^{\prime }$, $A^{\prime }$ and $is\_don{e}^{\prime }$ stand for $\varphi (S)$, $A$ and $is\_done$ of the first quintuple, respectively, in the queue D, $\varphi (S^{\prime }{)}^{\prime }$ refers to the non-$is\_done$ state after the subsequent maximum number of iteration episodes, and $R^{\prime }$ represents the cumulative sum of subsequent damping instant rewards ended in a $is\_done$ state.

(e) $S=S^{\prime }$

(f) When the total sample size in the container is greater than $N$, randomly select $m$ state transition samples from the experience container M and calculate the present $Q_{\mathrm{target}}(y_j)$: $$y_i=\{\begin{array}{cc}R& is\_done\mathrm{is}\mathrm{true}\\ R+\gamma \widehat{Q}(\varphi (S^{\prime }),\arg {\max }_{a^{\prime }}Q(\varphi (S^{\prime }),a,w),w^{-})& is\_done\mathrm{is}\mathrm{false}\end{array}$$

(g) Use the mean square error loss function $\frac{1}{m}{{\displaystyle \sum _{j=1}^m\left(y_i-Q(\varphi (S),A_j,w)\right)}}^2$ to update the network weight $w$ via the small-batch gradient descent method

Update the target network parameter $Q_{\mathrm{target}}(w^{\prime })$ every $C$ steps

Until $S$ ends in an $is\_done$ state

end for

Until reaching the maximum number of iteration episodes $\mathrm{T}$

end for

Output: $Q_{\mathrm{target}}(w^{\prime })$

3.2. Experience Classification DDQN Algorithm

3.2.1. Experience Classification Training Method

For the current experience replay techniques of the DDQN algorithm, the state transition is just simply stored in order, where the differences and similarities between different state transitions are not considered. As a result, although a large number of collisions occur in the early stage, the state transition for collisions may not be selected, so the speed of convergence will become slow. In this paper, a training method—experience classification (EC)—was proposed to improve the utilization of experience when agents collide with the obstacle or stay close to the obstacle in the early stage.

In the EC method, state transitions can be stored in several different experience replay buffers and storage conditions and sampling formulas of experience replay buffers can be designed according to the real demands. In the application of path planning, with complexity of algorithm and real demands taken into consideration, experience replay buffers will be divided into Memory0 and Memory1 according to whether a state transition belongs to a certain range of obstacles. A sampling batch is kept in every experience replay buffer and sampling in two experience replay buffers is based on the size of sampling batches. Then, a new sampling batch is obtained from the combination of the two and sent to a Q network for parameter training and updating. After the training, the average loss $l_0$ and $l_1$ of two experience replay buffers reflect the value of different types of experience for the current training stage. Therefore, sampling proportion, that is, the size of sampling batches in two experience replay buffers, was updated according to the demands of different experience replay buffers in various training stages and the returning loss of two experience replay buffers. The sketch map of EC training method is shown in Figure 1, where batch0 and batch1 are samples based on the sampling proportion of different experience replay buffers and loss0 and loss1 are average loss of training of different experience replay buffers.

3.2.2. Experience Classification Training Method

For the DDQN algorithm, all network parameters are the random value in the initial stage. All the samples should have the chance to be trained to help agents find the probable task demands as soon as possible. Therefore, in this paper, the EC method was used in auxiliary training. Within certain episodes in the initial stage, experience within a certain range of obstacles was sampled more, but the training trend of total samples was not changed. This algorithm is called the experience classification DDQN (EC-DDQN) algorithm.

To ensure algorithm’s stability in the initial stage of training, the state transitions whose number is less than that of minimum exploration samples should be stored in two experience replay buffers, while the state transitions whose number is more than that of minimum exploration samples should be selectively stored according to the features of state transitions. In this paper, Memory0 was designed for auxiliary training, so state transitions within a certain range of obstacles would be stored in Memory0, and all the state transitions would be stored in Memory1. The sampling proportions of Memory0 and Memory1 are expressed by $P_0$ and $P_1$.

In the EC-DDQN algorithm, three parameters, auxiliary training coefficient $\beta$, initial sampling probability $p_0$, and dynamic sampling probability $p_1$, were safeguarded to realize the design. Meanwhile, the sampling proportion of auxiliary experience replay buffers was designed into two parts. In the first part, initial sampling probability $p_0$ decreases as training goes. The $p_0$ cannot be too large when it is set; otherwise, it will influence the stability of the algorithm. In the second part, dynamic sampling probability $p_1$ is dynamically distributed to two experience replay buffers according to their training loss $l_0$ and $l_1$. The updating formula of sampling proportion $p_0$ is specified as shown in Equation (10):

$$P_0=\{\begin{array}{cc}{p}_0\times {\epsilon }_t+p_1\times \frac{l_0}{l_0+l_1}& t\le \beta T\\ 0& else\end{array}$$

(10)

The pseudo form of the EC-DDQN algorithm is shown in Algorithm 4.

Algorithm 4: Pseudo form of the EC-DDQN algorithm

Input: ${\epsilon }_{\mathrm{init}}$, ${\epsilon }_{\min }$, $\mathrm{T}$, $\alpha$, $\gamma$, $m$, $p_0$, $p_1$, $\mathsf{\Phi }$, $N$, $n$, $A$, $Q(w)$, $Q_{\mathrm{target}}(w)$, $C$

for t =1 to T do

Initialization: Initialize the first state $S$ in the present state sequence and acquire its eigenvector $\varphi (S)$

Update the basic exploration factor ${\epsilon }_t$ based on the number of iteration episodes $t$

for the each initialized first state $S$ do

(a) Select an action $A={\mathrm{argmax}}_{a}Q(\varphi (S),A,w)$ as per the probability $1-{\epsilon }_t$, or randomly select an action $A$

(b) Execute the action $A$ under the state $S$ to acquire an instant reward $R$, obtain a new state $S^{\prime }$ and its eigenvector $\varphi (S^{\prime })$. $is\_done$ refers to whether the state is terminated

(c) Store the quintuple $\left\{\varphi (S),A,R,\varphi (S^{\prime }),is\_done\right\}$. Simultaneously store M0 and M1 when len(M1) < N. Store M1 when len(M1) >= N, and simultaneously store M0 if $\varphi (S{)}^{\prime }\in \Phi$.

(d) $S=S^{\prime }$

(e) Update the sampling proportion of the auxiliary experience pool:

$P_0=\{\begin{array}{cc}{p}_0\times {\epsilon }_t+p_1\times \frac{l_0}{l_0+l_1}& t\le \beta T\\ 0& else\end{array}$

(f) When the total sample size in the container is greater than $N$, randomly select $m$ state transition samples from the experience containers M0 and M1 at sampling proportions of $P_0$ and $P_1$, respectively, and calculate the present $Q_{\mathrm{target}}(y_j)$: $$y_i=\{\begin{array}{cc}R& is\_done\mathrm{is}\mathrm{true}\\ R+\gamma \widehat{Q}(\varphi (S^{\prime }),\arg {\max }_{a^{\prime }}Q(\varphi (S^{\prime }),a,w),w^{-})& is\_done\mathrm{is}\mathrm{false}\end{array}$$

(g) Use the mean square error loss function $\frac{1}{m}{{\displaystyle \sum _{j=1}^m\left(y_i-Q(\varphi (S),A_j,w)\right)}}^2$ to update the network weight $w$ via the small-batch gradient descent method, and return the average losses $l_0$ and $l_1$ of different experience pools.

Update the target network parameter $Q_{\mathrm{target}}(w^{\prime })$ every $C$ steps

Until $S$ ends in an $is\_done$ state

end for

Until reaching the maximum number of iteration episodes $\mathrm{T}$

end for

Output: $Q_{\mathrm{target}}(w^{\prime })$

3.3. DDQN Algorithm Based on Experience Classification and Multi Steps

The advantages of MS and EC are applied in the DDQN algorithm, so a new DDQN-ECMS algorithm is figured out. Its pseudo form is shown in Algorithm 5.

Algorithm 5: Pseudo form of the DDQN-ECMS algorithm

Input: ${\epsilon }_{\mathrm{init}}$, ${\epsilon }_{\min }$, $\mathrm{T}$, $\alpha$, $\gamma$, $m$, $p_0$, $p_1$, $\Phi$, $N$, $n$, $n_s$, $A$, Q(w), C

for t = 1 to T do

Initialization: To get the eigenvector $\varphi \left(S\right)$ of the first state in state sequence $S$.

Update the basic exploration factor ${\epsilon }_t$ according to the number of iteration episodes $t$.

for the each initialized first state S do

(a) Select action $A={\mathrm{argmax}}_{a}Q\left(\varphi \left(S\right),A,w\right)$ based on probability $1-{\epsilon }_t$, otherwise action $A$ will be randomly selected.(b) Execute action

(b) Execute action $A$ in the state $S$ and get the instant reward $R$. New state $S^{\prime }$ and its eigenvector $\varphi \left(S^{\prime }\right)$ will be obtained. $is\_done$ refers to whether the state is terminated.

(c) Store quintet $\left\{\varphi \left(S\right),A,R,\varphi \left(S^{\prime }\right),is\_done\right\}$ in container for MS queue D. (If len(D) = $n_s$, the first quintet in the queue will pop up)

(d) If len(D) = $n_s$, store quintet $\left\{\varphi {(S)}^{\prime },A^{\prime },R^{\prime },\varphi {(S^{\prime })}^{\prime },is\_don{e}^{\prime }\right\}$ in the experience container. If len(M1) < N: store M0 and M1; if len(M1) >= $N$: store M1. If $\varphi {(S)}^{\prime }\in \Phi$, store M0. $\varphi {(S)}^{\prime }$, $A^{\prime }$ and $is\_don{e}^{\prime }$ are $\varphi \left(S\right)$, $A$ and $is\_done$ in the first quintet of the M3 queue. $\varphi {(S^{\prime })}^{\prime }$ is subsequent maximum number of steps that is not in $is\_done$ state. $R^{\prime }$ is cumulative sum with damping instant reward subsequently ended in $is\_done$ state.

(e) $S=S^{\prime }$

(f) Update the sampling proportion of auxiliary experience replay buffers: $$P_0=\{\begin{array}{cc}{p}_0\times {\epsilon }_t+p_1\times \frac{l_0}{l_0+l_1}& t\le \beta T\\ 0& else\end{array}$$

(g) If the total number of samples in container is more than N, m samples of state transitions are randomly selected from experience containers M0 and M1 according to sampling proportion $P_0$ and $P_1$ to calculate the current $Q_{\mathrm{target}}(y_j)$: $$y_i=\{\begin{array}{cc}R& is\_done\mathrm{is}\mathrm{true}\\ R+\gamma \widehat{Q}\left(\varphi \left(S^{\prime }\right),\arg \underset{a^{\prime }}{max}Q\left(\varphi \left(S^{\prime }\right),a,w\right),w^{-}\right)& is\_done\mathrm{is}\mathrm{false}\end{array}$$

(h) Use mean squared deviation loss function: $$\frac{1}{m}{\displaystyle \sum }_{j=1}^m{\left(y_i-Q\left(\varphi \left(S\right),A_j,w\right)\right)}^2$$

Update network weight parameter $w$ through the small batch gradient descent method. The average loss is $l_0$ and $l_1$ while returning to different experience replay buffers.

Update target network parameters $Q_{\mathrm{target}}(w^{\prime })$ every C step.

Until the $S$ as terminal stateend for

Until the maximum number of iteration episodes $T$.

end for

Output: $Q_{\mathrm{target}}(w^{\prime })$

4. Test and Analysis

4.1. Setting of the Experimental Simulation Environment

1.

Design of reward function

The directional control of the agent was discretized into four directions, namely left turn for 30°, left turn for 15°, right turn for 15°, and right turn for 30°. More detailed reward feedback was adopted in the design of reward function to improve the efficiency of interaction between the agent and the environment and make it better utilize rewards to adjust the action plan. A reward of −0.01 was obtained in the case of a direction change. A reward of −0.05× number of turning times was gained in the event of continuous turning. A reward of −0.1 was obtained when the agent was within a certain range of an obstacle. A reward of −1 was obtained in case of a collision. When the agent was near the goal position, a final reward of 10 was obtained. A base reward $10\times \Delta dis$ was obtained for each step, and $\Delta dis$ was the normalized value of the distance difference between two time steps with the diagonal of the scene as the unit length. The reward function was specifically designed as shown in Equation (11):

$$r=\{\begin{array}{cc}10\times \Delta dis& \mathrm{base}\_\mathrm{reward}\\ -0.01& \mathrm{direction}\_\mathrm{changed}\\ -0.05\times \mathrm{vibration}\_\mathrm{times}& \mathrm{continus}\_\mathrm{direction}\_\mathrm{changed}\\ -0.1& \mathrm{near}\_\mathrm{the}\_\mathrm{obstacles}\\ -1& \mathrm{collision}\\ 10& \mathrm{reach}\_\mathrm{the}\_\mathrm{goal}\end{array}$$

(11)

In Equation (11), “base_reward” means the base reward when the agent is near or far away from the obstacle, “direction_changed” means that the agent direction is changed, continus_direction_changed indicates that the agent will be subjected to a penalty of −0.05× number of turning times if changing directions for three consecutive times. ”near_the_obstacles” means that the agent is within a certain range of the obstacle, “collision” means that the agent collides with the obstacle, and “reach_the_goal” means that the agent reaches the goal position.

2.

Design of value fitting network Q

The designed neural network was a four-layer deep neural network. The input layer was composed of 10 neurons. The input layer was fully connected to the hidden layers. Each of two hidden layers was composed of 64 neurons, with the activation function of ReLU. The hidden layers were fully connected to the output layer, and the output layer was composed of five neurons.

3.

Parameter setting

The initial exploration factor ${\epsilon }_{init}$ and the minimum exploration factor ${\epsilon }_{min}$ facilitate agents to select between the optimal-value action and a random action. To be specific, agents will select the optimal-value action at a probability of ${\epsilon }_{init}-{\epsilon }_{min}$, and randomly select an action at a probability of ${\epsilon }_{min}$, thus ensuring that they will explore different states at a certain probability and preventing them from the failure of acquiring new experiences due to the constant selection of the optimal-value action. Given that the agent initially knew nothing about the environmental information, ${\epsilon }_{init}$ and ${\epsilon }_{min}$ were set as 1 and 0.01, respectively, to better explore the environmental information. When the learning rate $\alpha$, a parameter used to update the DDQN weight, was too low, the convergence process would be rather slow. At an excessively high $\alpha$, however, the gradient might fluctuate back and forth near the minimum value, and even the convergence might fail. By reference to the literature [26], $\alpha$ was set as 0.001 and the step interval for network parameter updating as C = 100 in this study. The single-episode maximum number of steps $Ste{p}_{max}$ means the maximum number of action times that can be done by the agent in a single loop, and the maximum number of iteration episodes $T_{max\_episodes}$ denotes the total amount of DDQN network parameter training. In this study, $Ste{p}_{max}$ and $T_{max\_episodes}$ were set as 1000 and 100, respectively, by combining the designed 2D path planning simulation experience. The reward damping factor $\gamma$, which was used to balance the current reward and the future reward, was set as 0.9 according to the literature [28]. In addition, the maximum capacity $M_{\mathrm{capacity}}$ of an experience playback container, which decided the number of state transition sequences that could be stored, was set as 20,000. As for deep neural network (DNN) training parameters, the sampling batch size $M_{\mathrm{capacity}}$ was determined as 128 through repeated experiments. In the section of the experience partitioning method, the minimum number of explorations $N_{\mathrm{explore}}$ was a threshold for judging whether to conduct experience partitioning. Moreover, the sampling proportions of two experience pools were co-decided by the initial exploration probability $p_0$, the dynamic sampling probability $p_1$, and the auxiliary exploration coefficient $\beta$. To keep the algorithms stable in the initial training phase, $N_{\mathrm{explore}}$, $p_0$, $p_1$ and $\beta$ were set as 500, 0.2, 0.2 and 0.4, respectively. An exponential damping factor of 0.95 was adopted for the exploration factor $\epsilon$ of all algorithms, expecting to improve the utilization rate of experience [12]. The related parameters are listed in

Table 1. Experimental training parameters’ settings.

Parameter	Value
initial exploration factor $\left({\epsilon }_{\mathrm{init}}\right)$	1
minimum exploration factor $\left({\epsilon }_{min}\right)$	0.01
learning rate $\left(\alpha \right)$	0.001
network parameter of updated step interval $\left(C\right)$	100
maximum step of single episode $\left(Ste{p}_{max}\right)$	1000
maximum number of each iteration episodes $\left(T_{max\_episodes}\right)$	100
reward attenuation factor $\left(\gamma \right)$	0.9
maximum capacity of experience playback container $\left(M_{\mathrm{capacity}}\right)$	20,000
minimum exploration number $\left(N_{\mathrm{explore}}\right)$	500
the sampling batch size $\left(N_{\mathrm{batchSize}}\right)$	128
initial exploration probability $\left(p_0\right)$	0.2
dynamic sampling probability $\left(p_1\right)$	0.2
auxiliary exploration coefficient $\left(\beta \right)$	0.4

.

4.2. Experiments and Result Analysis

Two experiments were designed to verify the algorithms proposed in this chapter. One experiment was performed in an environment with the fixed initial position of the agent, goal position, and obstacle positions, and the other was done in an environment with the random initial position of the agent, goal position, and obstacle positions. The performances of related algorithms in both experimental environments were compared on the OpenAI Gym platform.

4.2.1. Experiment in the Environment with Fixed Positions and Result Analysis

With the size of $7\times 7{\mathrm{m}}^2$, the experimental scene was visualized under the pixels of $700\times 700$. The visual scene is shown in Figure 2. The dark grey polygons of different sizes in the scene represent different obstacles, and there were five positions, and all the obstacles were fixed. The x-coordinates, y-coordinates, and radiuses of obstacles were set, from down to top, as [1.75, 1.75, 0.21], [1.75, 5.25, 0.28], [3.5, 3.5, 0.28], [5.25, 1.75, 0.21] and [5.25, 5.25, 0.35], respectively. The blue car was the agent in the experiment, and the goal position range was represented by a circular area with a red flag in the middle. The initial direction and initial position of the agent can be set arbitrarily, and the motion mode is uniform motion with variable direction. In the experiment in the environment with fixed positions, the initial direction of the agent was fixed to the front, and the goal position was fixed to the upper right. The task of the agent was to successfully reach the goal position without collision with the obstacles through direction control. For the agent, it was assumed that the direction and relative distance of the goal position were known, and, in addition, the agent could detect the distance of an object within a fan-shaped range of 180° in front, and the detection radius d could be arbitrarily set.

Four different algorithms were used for path planning in the scene of the experiment with fixed positions. In theory, the larger the MS value is set, the larger the calculation amount of the algorithm is, and the more accurate the estimation is. However, the case is not better if this value is bigger in practical applications. Therefore, in order to figure out a reasonable MS value, this value was firstly set as 3 in this paper, and it was determined to select the MS value $n_s=4$ of the MS-DDQN and DDQN-ECMS algorithms through experiments. In the experiments, the agent could reach the goal position successfully without any collision in the later stage of training (around the last 10 episodes) in all four of the algorithms. Through training with each of the four algorithms, the return curve was obtained, as shown in Figure 3. In this figure, the abscissa represents the current number of iteration episodes, and the ordinate represents the total return for each episode. The steps consumed in the four algorithms are shown in Figure 4. In this figure, the abscissa represents the current number of iteration episodes and the ordinate represents the total number of steps consumed by each episode.

Figure 3 shows that four algorithms can converge to the optimal reward value in the later stage. At the initial stage of training, the total return values of the four algorithms all rose rapidly, and there were certain fluctuations in the whole medium stage. For the EC-DDQN algorithm, the rise of the reward at the initial stage was the fastest but the most unstable in the middle and latter stages. In general, the DDQN-ECMS algorithm was more stable. Figure 4 shows that there were some fluctuations in the total steps of the four algorithms, and they were all within 200 steps in the 20 episodes after gradual stabilization. The difference between the four algorithms was not large. The path given by the final episode of single training in each of the four algorithms is shown in Figure 5. In the figure, the black, red, green, and blue circles represent the trajectories of the DDQN, MS-DDQN, EC-DDQN, and DDQN-ECMS algorithms, respectively.

Figure 5 shows that the path given by the DDQN algorithm was the smoothest, but the overall distance to the three obstacles was relatively close. In the path given by the MS-DDQN algorithm, the agent was close to the obstacles twice, and the steering adjustment was not timely. In the path given by the EC-DDQN algorithm, the direction was adjusted twice due to obstacles, and the direction change angle was relatively large in the first direction adjustment. The path given by the DDQN-ECMS algorithm was relatively far away from the obstacles, and the steering adjustment was relatively smooth when it was close to the obstacles, so it is safer overall.

However, under the single training, the quality of the performances of the algorithms cannot be determined because of the randomness. Therefore, the four algorithms were run 100 times each, and the performances of the four algorithms were counted. The counted items included the average time spent by the algorithm, the average return of the last 10 episodes of each training, the average number of steps, and the average reward of each step of the last 10 episodes of each training. The reason why the average value of the last 10 episodes was calculated is that the exploration rate of the last 10 episodes was stably below 0.01 under the exponential attenuation of 0.95. At this time, it was close to complete convergence for the algorithm, and some random interference can also be eliminated, which can better reflect the performance of the algorithm. The performance of each algorithm in the environment with fixed positions is shown in

Table 2. Performance comparison of four different algorithms in a fixed position scene.

Algorithm	DDQN	MS-DDQN	EC-DDQN	DDQN-ECMS
The average time spent (s)	129.732	122.244	122.587	124.709
The average return values	8.203	8.825	8.275	8.930
The average number of steps	157.815	160.267	158.132	157.330
The average reward per step	0.052	0.055	0.052	0.057

.

According to Table 2, compared to the time spent by the DDQN algorithm, the average time spent by the MS-DDQN and EC-DDQN algorithms was about 7 s less, and the average time spent by the DDQN-ECMS algorithm was about 5 s less. The average return values of both the MS-DDQN and DDQN-ECMS algorithms were significantly larger than those of the DDQN and EC-DDQN algorithms, and there was little difference between the latter two algorithms. The average number of steps of the DDQN-ECMS algorithm was the smallest, while that of the MS-DDQN algorithm was the largest. In terms of the average reward per step, the score of the DDQN-ECMS algorithm was the highest, followed by the MS-DDQN algorithm, while the DDQN and EC-DDQN algorithms share the lowest score. With the score of the DDQN algorithm as the reference, there was no increase for the EC-DDQN algorithm, while the score of the MS-DDQN algorithm increased by 5.77% and that of the DDQN-ECMS algorithm by 9.62%, with obvious advantages.

4.2.2. Experiment in the Environment with Random Positions and Result Analysis

The experimental scene size in the environment with random positions was the same as the experimental scene size in the experiment with fixed positions, and the number of obstacles was also 5, and the blue car was also the agent in the experiment, and the goal position range was also represented by a circular area with a red flag in the middle.. However, the goal position, positions, and sizes of obstacles in a certain range and the positions and direction of the agent were randomly set in each episode. That is to say, all the initial conditions in each episode were random in the training with the maximum number of iteration episodes as 100. An example of a random scene is provided as shown in Figure 6.

Four different algorithms were also applied for comparison of path planning in the environment with random positions. In addition, for the setting of MS value in this environment, the MS values $n_s=3$ of the MS-DDQN and DDQN-ECMS algorithms were determined through experiments. Through training with each of four algorithms, the return curve was obtained, as shown in Figure 7, and the number of steps of single training was also obtained, as shown in Figure 8. As aforementioned, the abscissa represents the number of iteration episodes, and the ordinate represents the total return value and the total number of steps.

Figure 7 shows that the total return values of the four algorithms fluctuated to some extent during the whole training process. Generally, as the number of iteration episodes increased, the fluctuations became smaller and closer to convergence. The MS-DDQN algorithm fluctuated largely in the medium stage, while the other three algorithms were obviously more stable than the former. Figure 8 shows that the number of steps in all four algorithms fluctuated largely. In terms of stability, the curve of the MS-DDQN algorithm fluctuated the most, while the difference of the other three algorithms was not obvious.

The paths were planned in the environment with random positions by using the network parameters of the four algorithms completed in this training in order to verify the generalization performances of the algorithms. The path planning results of the four algorithms under four environments with random positions are shown in Figure 9. As aforementioned, the black, red, green, and blue circles represent the trajectories of the DDQN, MS-DDQN, EC-DDQN, and ECMS-DDQN algorithms, respectively.

Figure 9 shows that, in this training, the agent started more often from the left and got close to the goal in the MS-DDQN algorithm, the agent started more often from the right and got close to the goal in the DDQN, EC-DDQN and DDQN-ECMS algorithms. However, the deflection angles were smaller for the EC-DDQN and DDQN-ECMS algorithms. The agent could reach the goal position in all four of the algorithms. However, the planned paths of the EC-DDQN and DDQN-ECMS algorithms were smoother with fewer direction changes.

The agent was trained for 100 times in the environment with random positions with each of the four algorithms, and the average performances of the four algorithms were counted, as shown in

Table 3. Performance comparison of four different algorithms in a random position scene.

Algorithm	DDQN	MS-DDQN	EC-DDQN	DDQN-ECMS
The average time spent (s)	73.292	74.436	67.913	74.013
The average return values	8.886	8.644	8.902	9.121
The average number of steps	71.684	72.767	69.888	71.102
The average reward per step	0.124	0.119	0.127	0.128

.

Table 3 shows that the time consumed in the EC-DDQN algorithm was the shortest, while there was little difference in the time consumed in the other three algorithms, about 1 s. The average return value of the DDQN-ECMS algorithm is the highest, while that of the MS-DDQN is the lowest. The average number of steps of the EC-DDQN algorithm is the smallest, followed by that of the DDQN-ECMS algorithm. In terms of the average reward per step that best reflects the performance of an algorithm, the performances of MS-DDQN, EC-DDQN, and DDQN-ECMS Algorithms decreased by 4.03%, 2.42%, and 3.26%, respectively, with the DDQN algorithm as the reference, and the performance of DDQN-ECMS algorithm showed the best comprehensive performance.

5. Conclusions and Future Directions

In this study, a mobile robot path planning algorithm referred to as ECMS-DDQN was proposed to improve the estimation accuracy of DDQN algorithm for the target Q value during the training process and the experience utilization efficiency during the experience playback process. Next, the path planning simulation verification was performed on an OpenAI Gym platform. Then, the superiority of the proposed algorithm was verified by comparing it with the DDQN algorithm, the MS-DDQN algorithm, and the EC-DDQN algorithm. Finally, the following conclusions were drawn through the theoretical study and the simulation experiment:

Based on the DDQN algorithm, on the one hand, the multi-step guidance method was introduced to replace the instant reward at a single moment with the instant reward of continuous multi-step interaction to improve the estimation accuracy of DDQN for the target Q value during training. On the other hand, the experience was classified according to state transition features and stored in different experience pools according to the experience type. Then, the sampling proportion was dynamically adjusted on the basis of the average training loss of different experience pools. Finally, the two improvements were fused to form a novel ECMS-DDQN algorithm, whose superiority was verified through the path planning simulation experiment of mobile robots.

Despite certain advantages in the total return value and the generalization performance, the proposed ECMS-DDQN algorithm shows no evident advantages in the average time consumption, the average return value, or the average number of steps over the DDQN algorithm, the MS-DDQN algorithm, and the EC-DDQN algorithm. Therefore, how to further improve the superiority of this algorithm will be key research content in the future.

Compared with the actual environment, the random environment set up in this study is too simple despite a certain complexity. Algorithms under environments with dynamic obstacles and dynamic target positions should be investigated in the follow-up research. Meanwhile, only simulation verification was performed in this study, and thus such verification should be done by establishing a corresponding actual experimental environment. In addition, the parameters of related algorithms given in this study remain to be further optimized, which will be key research content in the future.