GR(1)-Guided Deep Reinforcement Learning for Multi-Task Motion Planning under a Stochastic Environment

Abstract

Motion planning has been used in robotics research to make movement decisions under certain movement constraints. Deep Reinforcement Learning (DRL) approaches have been applied to the cases of motion planning with continuous state representations. However, current DRL approaches suffer from reward sparsity and overestimation issues. It is also challenging to train the agents to deal with complex task specifications under deep neural network approximations. This paper considers one of the fragments of Linear Temporal Logic (LTL), Generalized Reactivity of rank 1 (GR(1)), as a high-level reactive temporal logic to guide robots in learning efficient movement strategies under a stochastic environment. We first use the synthesized strategy of GR(1) to construct a potential-based reward machine, to which we save the experiences per state. We integrate GR(1) with DQN, double DQN and dueling double DQN. We also observe that the synthesized strategies of GR(1) could be in the form of directed cyclic graphs. We develop a topological-sort-based reward-shaping approach to calculate the potential values of the reward machine, based on which we use the dueling architecture on the double deep Q-network with the experiences to train the agents. Experiments on multi-task learning show that the proposed approach outperforms the state-of-art algorithms in learning rate and optimal rewards. In addition, compared with the value-iteration-based reward-shaping approaches, our topological-sort-based reward-shaping approach has a higher accumulated reward compared with the cases where the synthesized strategies are in the form of directed cyclic graphs.

1. Introduction

Motion planning has been used in robotics research to make movement decisions under certain movement constraints. Reinforcement learning has served as one approach to solve this sequential decision-making problem. As most motion planning scenarios involve continuous state representations, Deep Reinforcement Learning (DRL) approaches have dealt with these continuous cases. However, the typical DRL approaches suffer from reward sparsity and overestimation. It is also challenging to train the agents to deal with complex task specifications under deep neural network approximations and sparse feedback from the environment. Hierarchical DRL was proposed to integrate hierarchical value functions at different temporal scales, which decompose a complicated task into sub-tasks and guide the agents to solve the sub-tasks with policies over atomic actions [1]. However, the goal of these sub-tasks could be myopic, and the strategies trained with hierarchical DRL approaches could be locally optimal [2].

Linear temporal logic (LTL) is a compelling logic to express temporal properties, which can be used as valuable abstractions to guide robotics to interact with the physical world and take proper actions. LTL also provides specifications such as safety requirements, liveness requirements, and reactive requirements. The synthesized strategies of LTL could be used to program the reward functions of DRL and train the artificial agents efficiently [3]. However, it is of double exponential complexity to synthesize the control strategy of LTL. So, integrating DRL with LTL synthesis is not scalable. This paper considers one of the fragments of LTL, Generalized Reactivity of rank 1 (GR(1)), as a high-level reactive temporal logic to guide robots in learning efficient movement strategies under a stochastic environment. GR(1) is comprised of constraints for environmental assumptions and system guarantees with safety propositions and fairness assumptions, whose strategy can be synthesized with polynomial time complexity [4]. Many motion planning cases use GR(1) as high-level specifications, such as collision avoidance [5] and multi-agent motion planning [6]. However, less research has focused on integrating GR(1) with DRL.

There are three main contributions to this paper. We first use the synthesized GR(1) strategy to construct a potential-based reward machine where high-level experiences would be saved per state. These experiences can then be used to update the value functions of DRL. In this work, we integrate GR(1) with DQN, double DQN and dueling double DQN. We also observe that the synthesized strategies could be in the form of Directed Cyclic Graphs (DCG), and potential values should be calculated based on the topological order to aid in the learning of DRL. Thus, we develop a topological-sort-based reward-shaping approach to calculate the potential value of the reward machine. Finally, we integrate the reward machine synthesized from GR(1) specifications with a Dueling Double Deep Q-Network (3DQN) to reduce overestimation. Experiments on multi-task learning show that the proposed approach outperforms the state-of-art algorithms in learning rate and optimal rewards. Compared with the value-iteration-based reward-shaping approaches, our topological-sort-based reward-shaping approach has a higher accumulated reward compared with the cases where the synthesized strategies are in the form of DCGs. In addition, an ablation study shows that the 3DQN architecture performs better than the Double Deep Q-Network (2DQN) and the Deep Q-Network(DQN).

The rest of the paper is structured as follows. Section 2 describes the related work that integrates LTL with DRL to train the agents to satisfy multi-task specifications. Section 3 reviews the preliminaries of GR(1) synthesis and DRL and reward shaping. In Section 4, we define the MDP with a potential-based reward machine from the strategy synthesized from GR(1) formulas, then integrate the potential-based reward machine with DRL. In addition, we provide a topological-sort-based reward-shaping algorithm to deal with the cases where the synthesized strategies are in the form of DCGs. Section 5 uses a stochastic environment as a case study to demonstrate the approach. We first show that the proposed approach outperforms the state-of-art algorithms under three different GR(1) specifications. Then, we use the same GR(1) specification under different maps to show that the topological-sort-based reward-shaping approach could help to improve the training efficiency when the synthesized strategies are in the form of DCGs. Section 6 concludes our work and describes future work.

2. Related Work

Motion planning in real-world applications usually involves complex tasks. These high-level tasks could be described with some formal languages such as LTL [7]. Formal specifications are needed to integrate the expert knowledge and task specification to generate more detailed task descriptions to guide the motion specification [8]. Recently, research has been working on integrating motion planning with temporal logic. Fu et al. proposed an extension of RL to the temporal-logic-constrained control of unknown stochastic systems [9]. Jothimurugan. et al. developed a compositional learning approach incorporating reinforcement learning to learn neural network policies for high-level plans with a Dijkstra-style planning algorithm [10]. Li et al. proposed using temporal logic to specify complex rules that robots should follow and integrate domain knowledge into learning. The strategy obtained by this method in terms of robustness is better than that using heuristic reward learning [11]. Bozkurt et al. proposed a reinforcement learning framework for synthesizing control strategies in positional environments from LTL specifications in unknown stochastic environments [12]. The strategy learned through this framework can maximize the probability of satisfying the LTL formula without learning the transition. Kantaros et al. proposed the STyLuS*, a highly scalable asymptotic optimal control synthesis algorithm, to guide motion planning in large-scale, multi-agent systems [13].

Although LTL can be used as high-level specifications to describe the temporal requirements, it is computationally intractable to translate the LTL specifications when the formula is complicated [14,15]. Several previous works have investigated the approaches to constructing the product of LTL and MDP to obtain an optimal control policy while capturing temporal properties. The main category of automata studied so far includes Limit-Deterministic Büchi Automata (LDBA) and Deterministic Finite Automata (DFA). An LDBA could decrease the size of the MDP product because of its simplicity [16]. Hasanbeig et al. converted the given property into an LDBA and shaped the reward function so that an RL algorithm can satisfy the corresponding LTL [17].

There is also research on using DFA to construct the reward machine, which supports the specification of reward functions while exposing the reward function structure [3]. Icarte et al. proposed a QRM approach, which uses mealy machines to construct reward machines and decompose the whole task into sub-tasks. Then, traditional Q-learning is used to learn the sub-tasks [18]. Based on this approach, Camacho et al. added reward shaping with value iteration to calculate the potential functions for different states in the reward machine [19]. The Counterfactual Experiences for Reward Machine (CRM) approach is proposed by Icarte et al. to add the counterfactual experiences as the experience replay to be applied to offline reinforcement learning algorithms such as Q-learning, DQN and DDPG [3]. The results show that CRM performs better in different simulation scenarios than QRM despite its larger neural network structure.

3. Preliminaries

In this section, we mainly introduce some preliminaries of temporal logic specifications and the background of deep reinforcement learning. Temporal logic is a formalism introduced by Pnueli to express the temporal properties of state-based systems. As one of the most commonly used temporal logics, LTL contains standard logical operators and temporal operators [20]. An LTL formula $\phi$ could be interpreted over infinite traces as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \phi ::=& p\mid \neg \phi \mid {\phi }_1\wedge {\phi }_2\mid {\phi }_1\vee {\phi }_2\mid {\phi }_1\Rightarrow {\phi }_2\mid {\phi }_1\iff {\phi }_2\hfill \\ \hfill & \mid {\phi }_1\mathcal{U}{\phi }_2\mid ◯{\phi }_2\mid \square \phi \mid ♢\phi \hfill \end{array}\end{array}$$

Here, p is the atomic proposition. Given the trace $\pi$ and a position i, we use the following inductive rules on the structure of $\phi$ to show that $(\pi ,i)\models \phi$, which means the trace $\pi$ satisfies the LTL formula $\phi$ at position i:

• $(\pi ,i)\models p$ iff $p\in {\pi }_i$;
• $(\pi ,i)\models \neg \phi$ iff $(\pi ,i)⊭\phi$;
• $(\pi ,i)\models {\phi }_1\wedge {\phi }_2$ iff $(\pi ,i)\models {\phi }_1$ and $(\pi ,i)\models {\phi }_2$;
• $(\pi ,i)\models {\phi }_1\vee {\phi }_2$ iff $(\pi ,i)\models {\phi }_1$ or $(\pi ,i)\models {\phi }_2$;
• $(\pi ,i)\models {\phi }_1\Rightarrow {\phi }_2$ iff $(\pi ,i)\models \neg {\phi }_1$ or $(\pi ,i)\models {\phi }_2$;
• $(\pi ,i)\models {\phi }_1\mathcal{U}{\phi }_2$ iff $\exists k·k\ge i,(\pi ,k)\models {\phi }_2\wedge \forall j·i\le j<k,(\pi ,j)\models {\phi }_1$;
• $(\pi ,i)\models ♢\phi$ iff $\exists k·k\ge i,(\pi ,k)\models \phi$;
• $(\pi ,i)\models \square \phi$ iff $\forall k·k\ge i,{(\pi ,k)}^i\models \phi$.

Based on the above rules, the trace $\pi$ satisfies $\phi$ if $(\pi ,0)\models \phi$. The temporal operator □ expresses the requirement that the property $\phi$ is always true on the trace. The ♢ operator denotes that the property $\phi$ holds at some point on the trace. The $\mathcal{U}$ operator takes two arguments, ${\phi }_1$ and ${\phi }_1$, such that there exists a position k such that $(\pi ,k)$ satisfies ${\phi }_2$ and $(\pi ,j)$ satisfies ${\phi }_1$ for all $i<k$. The synthesis of LTL aims to generate a strategy that satisfies the LTL specification interpreted over infinite traces. However, the LTL synthesizing problem is known to be 2EXPTIME-complete [4]. GR(1) is a fragment of LTL, whose control strategies can be synthesized in polynomial time [21]. In this paper, we mainly examine the synthesized strategies of GR(1) specifications to guide the agents in learning under stochastic environments.

3.1. GR(1) Synthesis

The GR(1) specifications can be formalized in Formula (1), which consists of:

• ${\mathsf{\phi }}_I$: a temporal formula characterizing the initial state;
• ${\bigwedge }_i\square {\psi }_i$: the conjunction of LTL formulas that denote the system invariants, where ${\psi }_i$ is a temporal formula;
• ${\bigwedge }_i\square {\phi }_i$: the conjunction of LTL formulas that denote the liveness goals, where ${\phi }_i$ is a temporal formula.

$$\varphi =({\mathsf{\phi }}_I^e\wedge \underset{i}{\bigwedge }\square {\psi }_i^e\wedge \underset{i}{\bigwedge }\square ♢{\phi }_i^e)\to ({\mathsf{\phi }}_I^s\wedge \underset{i}{\bigwedge }\square {\psi }_i^s\wedge \underset{i}{\bigwedge }\square ♢{\phi }_i^s)$$

(1)

In Equation (1), ${\mathsf{\phi }}_I^e,{\psi }_i^e,{\phi }_i^e$ denote the assumptions on the environment, and ${\mathsf{\phi }}_I^s,{\psi }_i^s,{\phi }_i^s$ characterize the behavior specifications of the system. Equation (1) requires that if the environment satisfies the assumptions presented as temporal formulas, the system is supposed to satisfy the behavior specifications. The GR(1) specification could be synthesized to a transducer in polynomial time by constructing a GR(1) game structure and solving the game with $\mu$-calculus [21]. The transducer is in the form of $\langle E,e_0,F,{\delta }_e,{\delta }_a\rangle$, where E is a finite set of states; $e_0\in E$ is the initial state; $F\subset E$ is the set of the accepting state; ${\delta }_e\in E\times 2^P\to E$ is the transition function between states; and ${\delta }_a\in E\to 2^P$ is the state output function which outputs the atomic propositions.

3.2. Deep Reinforcement Learning with Reward Shaping

3.2.1. Deep Q-Network

Reinforcement Learning (RL) is one of the artificial intelligence techniques used to train an agent to learn the optimal strategies to maximize the reward while exploring the environment [22]. In RL, the environment is modeled as an MDP in the form $\mathcal{M}=\langle S,s_0,A,P,R,\gamma \rangle$, where S stands for a finite set of states, $s_0\in S$ denotes the initial states, A denotes a finite set of actions, $P\in S\times A\to Dist\left(S\right)$ denotes the transition probability distribution, $R\in S\times A\times S\to \mathbb{R}$ denotes the rewards assigned to the transitions between states and $\gamma$ denotes the discount factor. The policy of the agent $\lambda \in S\to Dist\left(A\right)$ is the probability distribution of the action $a\in A$ given the current state $s\in S$. At each step, the agent selects the action $a\sim \lambda \left(s\right)$ based on the current s and receives the reward $R(s,a,s^{\prime })$, where $s^{\prime }\sim P(s,a)$. The goal of the agent is to find the optimal policy ${\lambda }^{*}$ that maximizes the expected discounted reward.

Q-learning is a model-free reinforcement learning approach to find the optimal policy ${\lambda }^{*}$ using the Bellman equation and value iteration [23]. However, the real environment often has a continuous state space, which makes it impractical to use Q-learning to train the agents. The DRL approach uses two neural networks as function approximators to estimate the Q function, namely, the Q-network $Q(s,a;\theta )$ and the target network $\widehat{Q}(s,a;{\theta }^{-})$ [24]. Here, $\theta$ and ${\theta }^{-1}$ denote the network parameters. The Q-network and target network are initialized so that $\widehat{Q}=Q$. At each step t, the agent under state $s_t\in S$ selects the action $a_t\in A$, enters the state $s_{t+1}$ and obtains the reward $R(s_t,a_t,s_{t+1})$. The transition $<s_t,a_t,s_{t+1},R_t>$ is then saved to the buffer area $\mathcal{D}$. Then, some random mini-batch of the transition $<s_i,a_i,s_{i+1},R_i>$ is sampled from $\mathcal{D}$, which is used to train the target value in Equation (2). The Q-network with parameter $\theta$ can then be trained by minimizing the loss function shown in Equation (3) [24]. After $\tau$ steps, the parameters of the Q-network are copied to the target network:

$$y_i=R_i+\gamma \underset{a_{i+1}}{max}\widehat{Q}(s_{i+1},a_{i+1};{\theta }_i^{-1})$$

(2)

$$L_i\left({\theta }_i\right)=\mathbb{E}\left[{\left(R_i+\gamma \underset{a_{i+1}}{max}\widehat{Q}(s_{i+1},a_{i+1};{\theta }_i^{-1})-Q\left(s,a;{\theta }_i\right)\right)}^2\right]$$

(3)

3.2.2. Double Deep Q-Network

Thrun and Schwartz found that the random errors in actions would cause overestimation in Q-learning [25], which is then proved to introduce upward bias in DQN. Van et al. proposed Double DQN (2DQN) to reduce overestimation by splitting the max operator into action selection and action evaluation [26]. The target network is used for the evaluation of the policy of the Q-network. Instead of using the target network to obtain the action a that maximizes the Q value, 2DQN uses the Q-network to obtain the action a that maximizes the Q value, and the loss function in Equation (3) is changed to Equation (4):

$$L_i\left({\theta }_i\right)=\mathbb{E}\left[{\left(R_i+\gamma \widehat{Q}(s_{i+1},argmaxQ(s_{i+1},a;{\theta }_i);{\theta }_i^{-1})-Q\left(s,a;{\theta }_i\right)\right)}^2\right]$$

(4)

3.2.3. Dueling Double Deep Q-Network

With the development of DRL, much research has focused on convolutional networks to deal with continuous state spaces. Wang et al. proposed a dueling network architecture on top of DRL to represent two separate estimators, namely, the state value function $\mathcal{V}$ and the state-dependent action advantage function $\mathcal{A}$ [27]. The Q-network can then be presented as Equation (5):

$$Q(s,a;\theta ,\alpha ,\beta )=\mathcal{V}(s;\theta ,\beta )+\mathcal{A}(s,a;\theta ,\alpha )-\underset{a^{\prime }\in \mid \mathcal{A}\mid }{max}\mathcal{A}(s,a^{\prime };\theta ,\alpha )$$

(5)

Here, $\theta$ denotes the network parameters of the convolutional layers, $\alpha$, and $\beta$ represents the network parameters of fully connected layers. The advantage function $\mathcal{A}$ is forced to have zero advantage at the chosen action.

3.2.4. Reward Shaping

Recent research has shown that it is difficult to specify the rewards when designing the MDP. Usually, a reward is assigned only if the agent has accomplished the tasks, but this approach does not suit scenarios involving multiple sub-tasks and specifications. Reward shaping has been proven effective in incorporating domain knowledge into RL [28]. Potential functions $\mathsf{\varphi }\in S\to \mathbb{R}$ could be assigned to each state s, and the reward function could be designed as Equation (6) to avoid the transition from the state with high potential values to those with low potential values:

$$R(s_t,a_t,s_{t+1})\leftarrow R(s_t,a_t,s_{t+1})+\gamma \mathsf{\varphi }\left(s_{t+1}\right)-\mathsf{\varphi }\left(s_t\right)$$

(6)

4. Multi-Task Deep Reinforcement Learning with Generalized Reactivity Specifications

It is challenging to design reward functions of DRL for multi-task motion planning for several reasons. Firstly, the continuous state space should be extended to keep track of the sub-tasks that have been accomplished, which vastly decreases the learning efficiency while increasing the computation complexity. In addition, the sub-tasks might have different constraints, making it challenging to design the intermediate rewards for the agents to reach sub-goals. Take the scenario in Figure 1a as an example. There are two kinds of robotics on the map. The circles represent swarm robotics, which collects samples from the map. Each agent within the swarm moves in one direction at a constant speed and collects one category of sample on the map. The other kind is the processing robot, which collects samples from the swarm robotics with some temporal order. The processing robot can move in four directions with different speeds.

Example 1.

Take the GR(1) specification in Figure 2 as an example. The agents within the swarm are marked as a,b,c and d. The agents within the processing robot group, marked as $agen{t}_1$ and $agen{t}_2$, should cooperate to complete the specification of tasks. Initially, $agen{t}_1$ has not reached a, and $agen{t}_2$ has not reached $b,c,d$. The system constraints of the two agents can be expressed as:

• $agen{t}_1$ should reach a if it has not reached a;
• $agen{t}_2$ should reach b only when $agen{t}_1$ has reached a and perform the collecting task;
• $agen{t}_2$ should not reach c and d if $agen{t}_1$ has not reached a;
• $agen{t}_2$ should not reach $b,c,d$ simultaneously.

Under these system constraints, the goal of the two agents can be expressed as:

• $agen{t}_1$ should reach a infinitely often;
• $agen{t}_2$ should reach $b,c,d$ infinitely often if $agen{t}_1$ has reached location a infinitely often.

The synthesized strategy of the example is shown in Figure 1b.

Figure 2. The GR(1) specification used in the example.

Figure 2. The GR(1) specification used in the example.

4.1. Framework

The overall framework is shown in Figure 3. We first construct GR(1) specifications from the high-level system requirements, which are used to synthesize the strategy and construct the Potential-Based Reward Machine (PBRM). The GR(1) synthesis tool gr1c [29] is used to construct the synthesized strategy in Figure 2. The synthesized strategy abstracts the environment states and guides the transitions between abstract states. This helps the agent keep track of the abstract states and follow the synthesized strategy to finish the tasks under the required specifications. We then integrate the PBRM with MDP to construct a new MDP, which is trained with a Dueling Double Deep Q-Network.

4.2. Integrating Deep Reinforcement Learning with the GR(1) Strategy

We first extend the synthesized strategy $\mathcal{S}$ to the potential-based reward machine $\mathcal{N}$ in Definition 1, which introduces the reward function ${\delta }_r$ and potential function $\mathsf{\varphi }$ into $\mathcal{S}$. Given the current state e, as well as the atomic proposition ${\delta }_a\left[e\right]$, ${\delta }_r\left[e\right]\left[{\delta }_a\left[e\right]\right]$ and $\mathsf{\varphi }\left[e\right]$ are calculated based on the status of e. In the cases that the next state in the reward machine is one of the accepting states, formally presented as ${\delta }_e\left[e\right]\left[{\delta }_a\left[e\right]\right]\in F^{\prime }$, then ${\delta }_r\left[e\right]\left[{\delta }_a\left[e\right]\right]$ is assigned with a constant reward $\mathcal{R}$, and $\mathsf{\varphi }\left[e\right]$ is also assigned the same constant reward $\mathcal{R}$. Here, the reward of the state transition and the potential function is under the same scalar. The accepting states could be used to indicate that the task specifications have been satisfied. In the cases that the next state does not belong to the accepting states, then ${\delta }_r\left[e\right]\left[{\delta }_a\left[e\right]\right]$ is assigned with 0 and $\mathsf{\varphi }\left[e\right]$ is assigned with the value within the range $(0,\mathcal{R})$, which is shown in Equation (7).

Definition 1

(Potential-based Reward Machine). Given the synthesized strategy

$$\mathcal{S}=\langle E,e_0,F,{\delta }_e,{\delta }_a\rangle ,$$

the potential-based reward machine is defined as

$$\mathcal{N}=\langle E^{\prime },e_0^{\prime },F^{\prime },{\delta }_e^{\prime },{\delta }_r,\mathsf{\varphi }\rangle ,$$

where $E^{\prime }$ is a finite set of states; $e_0^{\prime }\in E^{\prime }$ is the initial state; $F^{\prime }\subset E^{\prime }$ is the set of accepting states; ${\delta }_e^{\prime }\in E^{\prime }\times 2^P\to E^{\prime }$ is the transition function between states; ${\delta }_r\in E^{\prime }\times 2^P\to \mathbb{R}$ is the rewards function of the state with the transition function; and $\mathsf{\varphi }\in E^{\prime }\to \mathbb{R}$ is the potential function of the states. $\mathcal{S}$ and $\mathcal{N}$ satisfy:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill & E^{\prime }=E,e_0^{\prime }=e_0,F^{\prime }=F,{\delta }_e^{\prime }={\delta }_e\hfill \\ \hfill & {\delta }_r\left[e\right]\left[{\delta }_a\left[e\right]\right]=\left\{\begin{array}{cc}0\hfill & {\delta }_e\left[e\right]\left[{\delta }_a\left[e\right]\right]\notin F^{\prime }\hfill \\ \mathcal{R}\hfill & {\delta }_e\left[e\right]\left[{\delta }_a\left[e\right]\right]\in F^{\prime }\hfill \end{array}\right.\hfill \\ \hfill & \left\{\begin{array}{cc}\mathsf{\varphi }\left[e\right]\in (0,\mathcal{R})\hfill & {\delta }_e\left[e\right]\left[{\delta }_a\left[e\right]\right]\notin F^{\prime }\hfill \\ \mathsf{\varphi }\left[e\right]=\mathcal{R}\hfill & {\delta }_e\left[e\right]\left[{\delta }_a\left[e\right]\right]\in F^{\prime }\hfill \end{array}\right.\hfill \end{array}\right.\end{array}$$

(7)

Then, we extend the MDP with $\mathcal{N}$ to reward the agents in Definition 2. $\mathcal{M}$ and $\mathcal{N}$ share the same set of atomic propositions $2^P$. Given that the agent is in the state $\langle s,e\rangle$ and performs action a to move from state s to $s^{\prime }$ in the MDP with the labeling function $\mathcal{L}\in S\times A\times S\to 2^P$, if $\langle e,\mathcal{L}(s,a,s^{\prime })\rangle \in \left({\delta }_e\right)$, then $\mathcal{N}$ would move from e to ${\delta }_e(e,\mathcal{L}(s,a,s^{\prime }))$; otherwise, the reward machine will stay in the state e. The transition probability distribution of ${\mathcal{M}}_{\mathcal{N}}$ is the same as $\mathcal{M}$. In $\mathcal{M}$, $R=\mathcal{R}$ when the task specifications are satisfied, and $R=0$ otherwise. As shown in Equation (8), the reward function is updated with the potential functions as ${\delta }_r(e,\mathcal{L}(s,a,s^{\prime }))+\gamma \mathsf{\varphi }\left(e^{\prime }\right)-\mathsf{\varphi }\left(e\right)$ if the next state is the accepting state; otherwise, the reward is ${\delta }_r(e,\mathcal{L}(s,a,s^{\prime }))$. Here, ${\delta }_r$ and $\mathsf{\varphi }$ are in the same scalar based on Definition 1 to balance the impact of $\mathcal{M}$ and $\mathcal{N}$.

Definition 2

(MDP with Potential-based Reward Machine). Given a potential-based reward machine $\mathcal{N}=\langle E,e_0,F,{\delta }_e,{\delta }_a,\mathsf{\varphi }\rangle$ and the MDP $\mathcal{M}=\langle S,s_0,A,P,R,\gamma \rangle$, where $\mathcal{M}$ and $\mathcal{N}$ share the labeling function $\mathcal{L}$. The MDPPBRM is defined as ${\mathcal{M}}_{\mathcal{N}}=\langle \widehat{S},\widehat{s_0},A,\widehat{P},\widehat{R},\gamma \rangle$, where:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill & \widehat{S}=S\times E\hfill \\ \hfill & \widehat{s_0}=s_0\times e_0\hfill \\ \hfill & \widehat{P}(\langle s^{\prime },e^{\prime }\rangle \mid \langle s,e\rangle ,a)=P(s^{\prime }\mid s,a)\hfill \\ \hfill & \widehat{R}(\langle s,e\rangle ,a,\langle s^{\prime },e^{\prime }\rangle )=\left\{\begin{array}{cc}{\delta }_r(e,\mathcal{L}(s,a,s^{\prime }))\hfill & e=e^{\prime }\hfill \\ {\delta }_r(e,\mathcal{L}(s,a,s^{\prime }))+\gamma \mathsf{\varphi }\left(e^{\prime }\right)-\mathsf{\varphi }\left(e\right)\hfill & e\ne e^{\prime }\hfill \end{array}\right.\hfill \\ \hfill & {\delta }_r\mathit{and}\mathsf{\varphi }\mathit{are}\mathit{in}\mathit{the}\mathit{same}\mathit{scalar}\hfill \end{array}\right.\end{array}$$

(8)

Based on ${\mathcal{M}}_{\mathcal{N}}$, we develop a Dueling Double Deep Q-Network in Algorithm 1 to aid the agents in learning motion planning strategies under GR(1) specifications. Algorithm 1 takes in the extended MDP ${\mathcal{M}}_{\mathcal{N}}$; the Q-network parameters $\theta ,\alpha ,\beta$; the target network parameter ${\theta }^{-1}$; the empty replay buffer $\mathcal{D}$; batch size $N_b$; and target network update frequency $\tau$. For each step t of each episode, Algorithm 1 follows the standard routine of DQN that uses the policy derived from $Q(\langle s,e\rangle ,a;\theta ,\alpha ,\beta )$ with the $ϵ$-greedy approach to select actions $a_t$. The agent then takes action $a_t$ and observes the next state $\langle s^{\prime },e^{\prime }\rangle$. The reward is calculated based on Equation (8). If the next state is the accepting state, then ${\delta }_r(e,\mathcal{L}(s,a,s^{\prime }))+\gamma \mathsf{\varphi }\left(e^{\prime }\right)-\mathsf{\varphi }\left(e\right)$; otherwise, the the reward is ${\delta }_r(e,\mathcal{L}(s,a,s^{\prime }))$. Then, the transition and the reward pair $\langle s_t,a_t,e_t,s_t^{\prime },e_t^{\prime },{\widehat{R}}_t\rangle$ are saved to the replay buffer $\mathcal{D}$ for sampling. During the training step, a mini-batch of $N_b$ is sampled. The loss function $L\left({\theta }_t\right)$ is calculated based on the mini-batch data in Equation (9). In the case that $e^{\prime }$ is in the accepting state, then the target value is $R_t$; otherwise, the target value is $R_t+\gamma \widehat{Q}(s_{t+1},argmaxQ(s_{t+1},a;{\theta }_t);{\theta }_t^{-1})$. Then, the gradient descent optimization is used to set the weight values of $\theta ,\alpha ,\beta$. Q-network parameters Q are copied to the target network $\widehat{Q}$ every $\tau$ step:

$$\begin{array}{c}\hfill L\left({\theta }_t\right)=\left\{\begin{array}{cc}\mathbb{E}\left[{\left(R_t-Q\left(s,a;{\theta }_t,{\alpha }_t,{\beta }_t\right)\right)}^2\right]\hfill & e^{\prime }\in Terminal\hfill \\ \mathbb{E}\left[{\left(R_t+\gamma \widehat{Q}(s_{t+1},argmaxQ(s_{t+1},a;{\theta }_t);{\theta }_t^{-1})-Q\left(s,a;{\theta }_t,{\alpha }_t,{\beta }_t\right)\right)}^2\right]\hfill & e^{\prime }\notin Terminal\hfill \end{array}\right.\end{array}$$

(9)

Algorithm 1: Dueling Double Deep Q-Network on ${\mathcal{M}}_{\mathcal{N}}$ synthesized from GR(1) formula.

4.3. Topological-Sort-Based Reward Shaping

In typical DRL algorithms, the reward would only be assigned when the agent has accomplished the whole task, leading to sparse reward issues. It becomes challenging for the agents to learn the correct actions during intermediate steps. Reward shaping can provide intermediate rewards to the agent so that the agent knows which action to take under the current state. Potential-based Reward Shaping (PBRS) uses the difference in the potential values of different states to guarantee policy invariance [30]. Based on PBRS, work in [3] proposed using value iteration to approximate the cumulative discounted reward in the reward machine. However, the potential values calculated with this approach highly depend on the discount factor ${\gamma }_r$. Given that the shortest distance between state s to the terminal state is d, then $\mathsf{\varphi }\left(s\right)={\gamma }_r^d$. Given that the initial state $s_0$ is close to the terminal state, then $\mathsf{\varphi }\left(s_0\right)$ would be close to the potential value of the terminal state, making the training process less effective. In addition, we observe that the value iteration approach is less practical in the cases of reward machines with Strongly Connected Components (SCC). Based on the reward machine constructed from the synthesized strategy from GR(1) specifications, we develop a topological-sort-based reward-shaping approach in Algorithm 2 to calculate the potential values of reward machines with DCG.

Algorithm 2 takes the synthesized strategy from GR(1) specifications as the input and outputs the potential values of the reward machine states, which can be integrated with the strategy to construct the potential-based reward machine $\mathcal{N}$. We first transform the strategy $\mathcal{S}$ to a DCG $\mathcal{G}$. In the transformed graph, the vertices represent the states of $\mathcal{S}$, and the edges represent the transitions of $\mathcal{S}$. We then apply topology sort on the graph $\mathcal{G}$ and obtain all the SCCs within the graph. Here, we use $sorted\_scc$ to denote the set of SCCs. Let $tp\in sorted\_scc[1,n]$ be the topological order of SCC, which is a bijection function from $sorted\_scc$ to the order of SCC. Here, n denotes the number of SCCs in the graph. Let $w\in sorted\_scc\to \mathbb{R}$ be the potential value of SCC and $size\in sorted\_scc\to [1,m]$ be the number of elements of the specific SCC, where m denotes the number of vertices in the graph. Then, w is calculated based on Clause (10) (line 4–5). All the elements e within the same SCC shares the same potential value $w\left[s\right]$ (line 8). In summary, Algorithm 2 calculates the SCC in $\mathcal{S}$ and assigns the weight of each SCC based on its topological order and the number of elements:

$$w\left[s\right]=\left(\sum _{i=1}^{i<tp\left[s\right]}size\left[t{p}^{-1}\left[i\right]\right]\right)+size\left[s\right],\forall s\in sorted\_scc$$

(10)

Algorithm 2: Topological-sort-Based Reward-Shaping Algorithm.

5. Experimental Section

This section presents empirical evaluations of the proposed algorithms in two experimental settings. We used three GR(1) specifications to specify the reactive multi-task specifications in different stochastic environments, whose synthesized strategies are shown in Figure 4. In this setting, we assumed that two agents were trying to reach the swarm robotics and finish their collecting tasks, one of which was the initiative agent and the other being the passive agent. The initiative agent was specified with the environment assumptions, and the passive agent was specified under the system guarantees in the GR(1) specification. The specifications of the passive agents are required to be satisfied only if the specifications of the initiative agents are satisfied. The algorithms are tests on the initiative agent and passive agent. We assume that each agent has ten sensors pointing in different directions, and each direction observes five variables: the range (x,y direction), the category of the sensed robot and the velocity of the sensed swarm robot (x,y direction). In addition, there are two additional sensors for their value in both the x and y directions. All these values are real numbers. So, the state is a 52-dimensional continuous state space. Moreover, action is a four-dimensional discrete space, which moves in four different directions.

5.1. Experiment Setup

In this section, we compare the GR(1)-guided Dueling Double DQN with Topological-Sort-based Reward Shaping (GR(1)-3DQN-TSRS), GR(1)-guided Double DQN with Topological-Sort-based Reward Shaping (GR(1)-2DQN-TSRS) and GR(1)-guided DQN with Topological-Sort-based Reward Shaping (GR(1)-DQN-TSRS) against three state-of-art baselines algorithms: GR(1)-guided DQN (GR(1)-DQN), GR(1)-guided DQN with Value-Iteration-based Reward Shaping (GR(1)-DQN-VIRS) and Hierarchical RL for Reward Machines (HRL) [3]. The HRL approach decomposes the multiple tasks into options, all of which would be learned simultaneously through off-policy learning. We first implemented Algorithms 1 and 2 under the OpenAI Gym environment [31] with the stochastic environment settings shown in Figure 1a. We compared the learning rate and the optimal reward of these algorithms with two different settings:

• Different GR(1) specifications under the same environment;
• Different environments under the same GR(1) specification.

The hyper-parameter settings of the experiment are shown in

Table 1. The Hyper-parameter Settings of the Experiments.

Hyper-Parameter	Definition	Value
$\mathcal{D}\_size$	replay buffer size	50,000
$layer\_num$	number of layers	4
$activation$	activation function	Relu
$N_b$	batch size	32
$\tau$	target network update frequency	100
$steps\_per\_episode$	number of steps per episode	1000
$total\_episode$	total number of episodes	2000

. For each experiment, we calculated the learning curve for the average reward per step. In addition, we use the mean value, standard deviation and maximum value to show the performance of different algorithms.

5.2. Performance Evaluation under Different GR(1) Specifications

In the first experiment, we compare the four algorithms, namely, HRM, GR(1)-DQN, GR(1)-DQN-VIRS and GR(1)-DQN-TSRS under three GR(1) specifications, whose synthesized strategies are in different formats. This experiment aims to show that topological-sort-based reward shaping works better in situations where the synthesized strategies are in the form of DCGs. The synthesized strategies of ${\phi }_1$, ${\phi }_2$ and ${\phi }_3$ are presented in Figure 4. In the experiment settings, the synthesized strategies of ${\phi }_1$ are in the form of Directed Acyclic Graphs (DAG), and those of ${\phi }_2$ and ${\phi }_3$ are in the form of DCGs.

Figure 5, Figure 6 and Figure 7 show the comparison of the learning curves of the algorithms using different GR(1) specifications under the same environment. The results show that the GR(1)-DQN, GR(1)-DQN-VIRS and GR(1)-DQN-TSRS learn faster than HRM under all GR(1) specifications. Figure 8, Figure 9 and Figure 10 show the box plot of the learning curve for different GR(1) specifications.

Table 2. The mean, max and standard deviation of learning curves from specification ${\phi }_1$, ${\phi }_2$ and ${\phi }_3$ starting from training step 1600.

		HRM	GR(1)-DQN	GR(1)-DQN-VIRS	GR(1)-DQN-TSRS
	mean	0.126	0.982	0.981	0.974
${\mathbf{\phi }}_{\mathbf{1}}$	std	0.0001	0.0003	0.0001	0.0001
	max	0.148	0.996	0.991	0.991
	mean	0.092	0.284	0.251	0.411
${\mathbf{\phi }}_{\mathbf{2}}$	std	0.0005	0.0004	0.0003	0.0017
	max	0.140	0.313	0.290	0.500
	mean	0.181	0.722	0.676	0.752
${\mathbf{\phi }}_{\mathbf{3}}$	std	0.0002	0.0004	0.0004	0.0009
	max	0.203	0.756	0.714	0.820

shows the mean, maximum and standard deviation of the learning curves of different models. As can be seen from the values, GR(1)-DQN, GR(1)-DQN-VIRS and GR(1)-DQN-TSRS have higher rewards per step compared with HRM in all the three cases. Furthermore, in the cases where the synthesized strategy is in the form of DAGs, GR(1)-DQN, GR(1)-DQN-VIRS and GR(1)-DQN-TSRS have similar mean rewards. For example, GR(1)-DQN obtained 0.982 per step, GR(1)-DQN-VIRS obtained 0.981 per step and GR(1)-DQN-TSRS obtained 0.974 per step. However, in the cases that the synthesized strategy is in the form of DCGs, GR(1)-DQN-TSRS has higher mean rewards and maximum rewards compared with GR(1)-DQN and GR(1)-DQN-VIRS. For example, in the ${\phi }_2$ and ${\phi }_3$ specifications, the GR(1)-DQN-TSRS model obtained the highest mean and maximum reward in both cases. So, the experiment results concluded that the GR(1)-DQN-TSRS approach could obtain a higher accumulated reward compared with HRM, GR(1)-DQN and GR(1)-DQN-VIRS.

5.3. Ablation Study under Different Environment Settings

We use ${\phi }_2$ as the GR(1) specification in the second experiment and perform an ablation study with three different stochastic environments. Different stochastic environment settings will also have an impact on the learning efficiency of agents, for example, if the task specification requires the agent to reach swarm agent b after reaching swarm agent a, where a and b are close on the map. In this case, it would be easy for the agents to learn the policy to move from a to b. However, if a and b are far from each other in another stochastic map, then it would be inefficient for the agents to learn the policy for motion planning under complicated task specifications.

In this ablation study, we mainly examine the performances of six algorithms in three different environments, namely, HRM, GR(1)-DQN, GR(1)-DQN-VIRS, GR(1)-DQN-TSRS, GR(1)-2DQN-TSRS and GR(1)-3DQN-TSRS. Figure 11, Figure 12 and Figure 13 show the learning curve of these six algorithms under different environment settings. Figure 14, Figure 15 and Figure 16 show the box plot of different algorithms in three environments.

Table 3. The mean, max and standard deviation of learning curves from maps M1, M2 and M3 starting from training step 1600.

		HRM	GR(1) -DQN	GR(1) -DQN -VIRS	GR(1) -DQN -TSRS	GR(1) -2DQN -TSRS	GR(1) -3DQN -TSRS
	mean	0.091	0.248	0.251	0.411	0.450	0.458
M1	std	0.0005	0.0004	0.0003	0.0017	0.0010	0.0004
	max	0.140	0.313	0.290	0.500	0.497	0.497
	mean	0.082	0.328	0.261	0.358	0.445	0.461
M2	std	0.0002	0.0004	0.0004	0.0002	0.0006	0.0005
	max	0.110	0.365	0.305	0.388	0.497	0.500
	mean	0.080	0.270	0.320	0.404	0.408	0.522
M3	std	0.0002	0.0006	0.0008	0.0003	0.0004	0.0004
	max	0.115	0.323	0.368	0.443	0.450	0.557

shows the mean, maximum and standard deviation of the learning curves of different models under different maps. Firstly, we observe that the GR(1)-DQN-TSRS, GR(1)-2DQN-TSRS and GR(1)-3DQN-TSRS performs better than HRM, GR(1)-DQN and GR(1)-DQN-VIRS in the three maps. GR(1)-3DQN-TSRS obtained the highest mean and maximum value in M2 and M3. GR(1)-DQN-TSRS obtained the maximum reward in M1. All the approaches have similar variances. From the learning curve and the table, we could observe the following conclusions:

• HRM obtains the lowest reward compared with the other five algorithms;
• GR(1)-DQN and GR(1)-DQN-VIRS have similar performances, but their reward per step is smaller than the topological-sort-based reward-shaping approaches;
• It is evident that topological-sort-based reward shaping could help GR(1)-DQN obtain more rewards during training;
• Overall GR(1)-2DQN-TSRS is better than GR(1)-DQN-TSRS, and GR(1)-3DQN-TSRS is better than GR(1)-2DQN-TSRS.

Figure 11. Comparison of learning curves for multi-task reinforcement learning algorithms with ${\phi }_2$ under stochastic environment setting 1.

Figure 11. Comparison of learning curves for multi-task reinforcement learning algorithms with ${\phi }_2$ under stochastic environment setting 1.

Figure 12. Comparison of learning curves for multi-task reinforcement learning algorithms with ${\phi }_2$ under stochastic environment setting 2.

Figure 12. Comparison of learning curves for multi-task reinforcement learning algorithms with ${\phi }_2$ under stochastic environment setting 2.

Figure 13. Comparison of learning curves for multi-task reinforcement learning algorithms with ${\phi }_2$ under stochastic environment setting 3.

Figure 13. Comparison of learning curves for multi-task reinforcement learning algorithms with ${\phi }_2$ under stochastic environment setting 3.

Figure 14. Box plot of learning curve for ${\phi }_2$ under the environment setting 1.

Figure 14. Box plot of learning curve for ${\phi }_2$ under the environment setting 1.

Figure 15. Box plot of learning curve for ${\phi }_2$ under the environment setting 2.

Figure 15. Box plot of learning curve for ${\phi }_2$ under the environment setting 2.

Figure 16. Box plot of learning curve for ${\phi }_2$ under the environment setting 3.

Figure 16. Box plot of learning curve for ${\phi }_2$ under the environment setting 3.

6. Conclusions

This paper considers one fragment of LTL, GR(1), as a high-level reactive temporal logic to guide robots in learning efficient movement strategies under stochastic environments. The GR(1) specifications used in this paper, whose control policy can be synthesized in polynomial time, are applicable in industrial applications. Taking the synthesized strategies of GR(1) specifications as the potential-based reward machine, we extend MDP with a topological-sort-based reward-shaping approach to improve learning efficiency. The reward machine is then integrated with the dueling double deep network to guide motion planning under continuous state space. Experiments on multi-task learning under GR(1) specifications show that the proposed approach outperforms the state-of-art algorithms in learning efficiency and optimal rewards per step. In addition, compared with the value-iteration-based reward-shaping approaches, our topological-sort-based reward-shaping approach could handle the cases where the synthesized strategies are in the form of DCGs.

There still exist some research challenges for future work. For example, the initial GR(1) specification could be incomplete and unrealizable. We could develop a counter-strategy-based framework to refine the GR(1) specifications with realizable strategies. In addition, this paper mainly focuses on value-based DRL approaches. In future work, we might explore the possibilities of integrating GR(1) specifications with policy-based approaches such as PPO.