Research on Multi-Robot Formation Control Based on MATD3 Algorithm

Abstract

This paper investigates the problem of multi-robot formation control strategies in environments with obstacles based on deep reinforcement learning methods. To solve the problem of value function overestimation in the deep deterministic policy gradient (DDPG) algorithm, this paper proposes an improved multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm under the CTDE framework combined with the twin delayed deep deterministic policy gradient (TD3) algorithm, which adopts a prioritized experience replay strategy to improve the learning efficiency. For the problem of difficult obstacle avoidance for a robot formation, a hybrid reward mechanism is designed to use different formation maintenance strategies in obstacle areas and obstacle-free areas to achieve the control goal of obstacle avoidance by reasonably changing the formation. The simulation experiments verified the effectiveness of the multi-robot formation control strategy designed in this paper, and comparative simulations verified that the algorithm has a faster convergence speed and more stable performance.

1. Introduction

The fields of formation control, cooperative control, and swarm control of multi-agent systems have been rapidly developing in recent decades [1]. The research on formation control focuses on making multi-agent systems generate and maintain a predetermined formation structure to perform the corresponding scenario tasks when moving toward a target area. Multi-robot formation control has been widely used in military and firefighting applications, such as cruise reconnaissance, collaborative search and rescue, and transportation [2,3,4,5,6]. In the early stages of formation control research, researchers applied traditional control methods such as predictive control [7] and sliding mode variable structure control [8] to solve formation problems and developed formation control strategies based on the leader-following method [9], the virtual structure method [10], and the behavior-based method [11]. All of the above traditional control methods require accurate system modeling to calculate the control law, which significantly increases the complexity of the system design.

Obstacle avoidance in formation navigation for multiple robots is a fundamental and important capability. The work [12] proposed a distributed control method for the formation navigation task of robot teams that combines constrained nonlinear optimization with consistency to achieve obstacle avoidance in formations. However, this approach needs to avoid the risk of deadlock. The work [13] combined state transition techniques with model predictive control (MPC) to propose a decentralized controller for the formation reconfiguration of a UAV system in an obstacle environment. The experimental results showed that, based on this method, the UAV system can successfully learn an unknown dynamics model and solve the optimal control problem using MPC.

Over the years, reinforcement-learning-based solutions to formation control problems have received much attention, with the advantages of avoiding cumbersome dynamics modeling and coping well with changes in unknown environments through self-learning capabilities [14]. The work [15] used the GQ($\lambda$) algorithm to learn leader-following control, which improved the path-planning capability of the followers and was experimented on a physical robot platform. However, only a linear formation structure could be considered. The work [16] improved the actor–critic algorithm by adding an adaptive discriminator to estimate the unknown system dynamics and performed stability analysis. The work [17] improved the DDPG algorithm by applying the advantage function to a multi-USV formation generation and maintenance strategy. The work [18] used the DDPG algorithm to solve the USV formation path-tracking problem by adding a random braking mechanism after action selection, which allowed the USV team to correctly control the velocity when it approached the reference position and prevented it from falling into a local optimum. The work [19] improved the VDN algorithm by proposing a C2VDN algorithm with cognitive consistency based on the fish-like robot formation control problem, where all robots share state-value network parameters and achieved circular formation control based on a decentralized controller. The work [20] used the DDPG algorithm for formation motion planning and control tasks for TWODMR robot teams and performed comparison experiments with the artificial potential field algorithm. We also refer to [21,22,23,24] for more related research on learning-based methods for different applications.

All of the literature above studied and improved the formation control problem in their respective scenarios using reinforcement learning algorithms. However, most of the literature does not consider the detection capability of the agents for obstacles, and there is less research on multi-agent formation navigation strategies in environments with obstacles. On the other hand, most of the reinforcement learning algorithms used in the existing literature are DDPG algorithms [25], which originated from the DQN algorithm [26], which is an off-policy approach. The DDPG algorithm updates the target value function during training using the action that is currently considered to have the largest Q value, which can lead to the problem of Q overestimation. However, the overestimation is usually non-uniform, leading to unreliable decisions made by the algorithm. Based on these problems, this paper investigated multi-robot formation navigation in environments with obstacles using deep reinforcement learning in conjunction with the leader-following method. Based on the twin delayed deep deterministic policy gradient (TD3) algorithm [27] combined with the CTDE framework [28], an improved multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm is proposed. The main contributions of this paper are as follows: (1) To address the overestimation problem in the DDPG algorithm, we used the TD3 algorithm in combination with the CTDE framework to implement multi-robot formation control. (2) We designed a hybrid reward mechanism that allows the robot formation to change its formation reasonably when facing an obstacle area and thus pass. (3) The prioritized experience replay (PER) mechanism [29] was also added to make full use of high-quality samples for training and improve the learning efficiency. At the same time, we set up both guided rewards and sparse rewards to facilitate robot learning. The simulation results showed that the algorithm can perform the formation navigation task in environments with obstacles well.

The rest of the paper is organized as follows: In Section 2, we provide the problem description and system modeling. Section 3 describes the reinforcement learning algorithm used in this paper and the improvements made. Section 4 describes the reinforcement-learning-based formation control scheme, including the state space design and reward function design. In Section 5, we conduct the simulation experiments and analyze the experimental results. Finally, Section 6 concludes the whole paper.

2. Problem Description and Modeling

2.1. System Model

In a two-dimensional environment, there exist N omnidirectional robots with $i=1,\dots ,N$; when $i=lea$, this denotes the leader robot ${\mathit{r}}_{lea}$, and when $i=1,\dots ,N-1$, this denotes the following robot ${\mathit{r}}_i$. The position and velocity of ${\mathit{r}}_{lea}$ are denoted by $p_{lea}$ and $v_{lea}$, respectively, and the position and velocity of ${\mathit{r}}_i$ are denoted by $p_i$ and $v_i$, respectively. There exist M circular obstacles of radius $R_{obs}$. ${\mathit{o}}_j$ denotes the $jth$ obstacle, where $j=1,\dots ,M$. The position of ${\mathit{o}}_j$ is denoted as $p_{oj}$, and there exists one target point whose position is denoted as $p_g$. The dynamics model of the robot ${\mathit{r}}_i$ is described as follows:

$$\left\{\begin{array}{c}{a}_i^t=F_i^t/m\hfill \\ v_i^t=v_i^{t-1}+a_i^t·\Delta t\hfill \\ p_i^t=p_i^{t-1}+v_i^t·\Delta t\hfill \end{array}\right.$$

(1)

where $i=1,\dots ,N-1$ and $i=lea$. $a_i^t$, $v_i^t$, and $p_i^t$ all belong to ${\mathbb{R}}^2$, denoting the acceleration, velocity, and position of robot ${\mathit{r}}_i$ at moment t; $\Delta t$ denotes the system sampling period; m is the mass of the robot. Let $u_i^t=F_i^t$ denote the control input, where $F_i^t\in {\mathbb{R}}^2$ represents the force vector of robot ${\mathit{r}}_i$.

2.2. Formation Problem Description

The multi-robot formation scenario is shown in Figure 1, where there are some obstacles and robots in a finite plane area, and the robots are divided into leader and followers. The formation navigation task requires the robots to collectively form a default formation and reach the target area while avoiding collisions.

As shown in Figure 2, at the beginning, the robots move the shortest path from their respective initial positions to form a default formation.

For the formation avoidance problem, we considered the following scheme: when the leader robot leads the formation into the obstacle area (we assumed that the leader robot knows the area range of the obstacle distribution), the formation changes into a chain structure when it passes the obstacle area, and after passing it, it returns to the default formation and continues to move towards the target point, as shown in Figure 3.

In this paper, we assumed that the leader robot can globally perceive the location of the target point. Define the detection distance of the robot as ${\rho }_d>0$. When the distance between the robot and the obstacle is less than ${\rho }_d$, the robot can sense the obstacle’s position. During the formation movement, the robot can obtain the status information of itself and other robots in real-time, which includes velocity information $v_i^t$ and position information $p_i^t$. The control target of formation navigation is divided into three parts: maintaining the default formation, avoiding collision, and reaching the target position.

(1) Maintaining the default formation: The formation members start from their respective starting positions and move to form a default formation, and the default spacing should be maintained between all robots. Therefore, the control objective is to minimize the distance error between robots:

$$min\left|d_i^j-d_i^{j^{*}}\right|,i,j\in \left[1,\dots ,N\right],i\ne j$$

(2)

where $d_i^{j^{*}}$ is the desired relative distance between robot ${\mathit{r}}_i$ and robot ${\mathit{r}}_j$ and $d_i^j$ is the actual relative distance between robot ${\mathit{r}}_i$ and robot ${\mathit{r}}_j$.

(2) Collision avoidance: During the formation movement, the distance between robots and between robots and obstacles should satisfy $d_i^j\ge {\rho }_r$ and $dis\left({\mathit{r}}_i,{\mathit{o}}_j\right)\ge {\rho }_o$, where $dis(·)$ denotes the Euclidean distance in two dimensions and ${\rho }_r$ and ${\rho }_o\left({\rho }_d>{\rho }_o\right)$ are the safe distance between robots and the safe distance between robots and obstacles, respectively.

(3) Reaching the target position: During the formation movement, our goal is to minimize the distance between the leader robot and the target point, as follows:

$$min{∥p_{lea}-p_g∥}_2$$

(3)

3. Multi-Robot Formation Control Based on Multi-Agent Reinforcement Learning

3.1. Multi-Agent Reinforcement Learning

In single-agent reinforcement learning, the environment is stable for the agent. However, in the multi-agent system, each agent will interact with the environment, which will cause the environment to become unstable for each agent. To solve this problem, the centralized training and decentralized execution framework (CTDE) [28] is often used in research. At the beginning, all agents participate in the training simultaneously to jointly improve the parameters of the policy network and the value network. After the training, the agents make decisions and take actions according to their respective policy networks.

The Markov decision process (MDP) is defined as the process of the interaction between an agent and its environment [30]. The Markov decision process for a multi-robot formation can be represented explicitly by a multivariate group $\left\{S,A,R,O,P\right\}$, where S denotes the state space of the environment and $s^t\in S$ denotes the state at moment t. $A=\left\{A_1,\dots ,A_N\right\}$ denotes the joint action space of all robots, and the joint action of all robots is denoted as $a=\left\{a_1,\dots ,a_N\right\}$, where $a_i\in A_i$ denotes the action of robot ${\mathit{r}}_i$. $R=\left\{R_1,\dots ,R_N\right\}$ denotes the set of reward functions of all robots, and $r_i=R_i\left(s^t,a_i\right)$ denotes the reward value of robot ${\mathit{r}}_i$ when it performs action $a_i$ in state $s^t$. The observation space of all robots can be denoted as the set $O=\left\{O_1,\dots ,O_N\right\}$, and the observations of all robots are denoted by $o=\left\{o_1,\dots ,o_N\right\}$. The state transfer probability function is denoted as $P\left(s^{t+1}\mid s^t,a_i\right)$, that is the probability distribution specifying when the environmental state $s^t$ changes to the state $s^{t+1}$ at the next moment when the robot ${\mathit{r}}_i$ performs the action $a_i$ in state $s^t$.

3.2. MATD3

Overestimation bias is a common problem in reinforcement learning, where the action value function is maximized and used to learn the optimal policy, which can cause the learned policy to become inaccurate [31]. The TD3 algorithm updates the target value function by learning two critic networks and choosing a smaller estimated Q value to approach the true Q value to alleviate the overestimation problem. Furthermore, TD3 uses a policy delayed update approach, where the critic network is updated for a certain number of steps before the actor network is updated [27]. The MATD3 algorithm combines the implementation process of TD3 and the “centralized training, distributed execution” architecture. The algorithm structure of MATD3 is designed as shown in Figure 4. MATD3 uses two sets of actor–critic structures, and each robot has its own independent actor network $\mu \left(o_i;{\theta }_i^{\mu }\right)$ with parameter ${\theta }_i^{\mu }$, which calculates action $a_i$ based on the current observation $o_i$ of robot ${\mathit{r}}_i$. The two critic networks are trained in a centralized manner, respectively, network $Q_1\left(o_1,\dots ,o_N,a_1,\dots ,a_N;{\theta }_i^{Q_1}\right)$ with parameter ${\theta }_i^{Q_1}$ and network $Q_2\left(o_1,\dots ,o_N,a_1,\dots ,a_N;{\theta }_i^{Q_2}\right)$ with parameter ${\theta }_i^{Q_2}$, which take the observations and actions of all robots as the input, and the network output is the estimated Q value to evaluate the goodness of action $a_i$, thus improving the performance of the actor network. For stable learning, the target actor network and two target critic networks are constructed with parameters ${\widehat{\theta }}_i^{\mu }$, ${\widehat{\theta }}_i^{Q_1}$, and ${\widehat{\theta }}_i^{Q_2}$, respectively, and the parameter update law of the target network is:

$${\widehat{\theta }}_i^{\mu }=\tau {\theta }_i^{\mu }+(1-\tau ){\widehat{\theta }}_i^{\mu }$$

(4)

$${\widehat{\theta }}_i^{Q_1}=\tau {\theta }_i^{Q_1}+(1-\tau ){\widehat{\theta }}_i^{Q_1}$$

(5)

$${\widehat{\theta }}_i^{Q_2}=\tau {\theta }_i^{Q_2}+(1-\tau ){\widehat{\theta }}_i^{Q_2}$$

(6)

where $\tau$ is the soft update parameter. To alleviate the overestimation problem, a smaller Q value is chosen to calculate the target Q value, so that the estimated Q value can be close to a more realistic and reliable target Q value, which in turn guides the parameter update of the actor network. The target Q value of robot ${\mathit{r}}_i$ can be defined as follows:

$$y_i^t=r_i^t+\gamma ·min\left(Q_j\left(o_1^{t+1},\dots ,o_N^{t+1},{\tilde{a}}_1^{t+1},\dots ,{\tilde{a}}_N^{t+1};{\widehat{\theta }}_i^{Q_j}\right)\right);j=1,2$$

(7)

where $\gamma$ is the discount factor and ${\tilde{a}}_i^{t+1}$ is the target action of the clipping, which is the output of the target actor network with the following expression.

$${\tilde{a}}_i^{t+1}=\mu \left(o_i^{t+1};{\widehat{\theta }}_i^{\mu }\right)+clip\left(\mathcal{N}(0,\sigma ),-c_i,c_i\right)$$

(8)

where $\mathcal{N}(0,\sigma )$ denotes a random noise satisfying a normal distribution and $c_i$ is the noise edge value.

Define the two critic loss functions of the robot ${\mathit{r}}_i$ as $J\left({\theta }_i^{Q_1}\right)$ and $J\left({\theta }_i^{Q_2}\right)$, as follows:

$$J\left({\theta }_i^{Q_1}\right)={\mathbb{E}}_{o^t,a^t,r^t,o^{t+1}}\left[{\left(Q_1\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_1}\right)-y_i^t\right)}^2\right]$$

(9)

$$J\left({\theta }_i^{Q_2}\right)={\mathbb{E}}_{o^t,a^t,r^t,o^{t+1}}\left[{\left(Q_2\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_2}\right)-y_i^t\right)}^2\right]$$

(10)

The two critic network parameters ${\theta }_i^{Q_1}$ and ${\theta }_i^{Q_2}$ are updated by gradient descent, specifically in Equations (11) and (12).

$${\theta }_i^{Q_1}={\theta }_i^{Q_1}-{\alpha }_C^1{\nabla }_{{\theta }_i^{Q_1}}J\left({\theta }_i^{Q_1}\right)$$

(11)

$${\theta }_i^{Q_2}={\theta }_i^{Q_2}-{\alpha }_C^2{\nabla }_{{\theta }_i^{Q_2}}J\left({\theta }_i^{Q_2}\right)$$

(12)

where ${\alpha }_C^1$ and ${\alpha }_C^2$ are the learning rates of the two critic networks, respectively. The formula for calculating the actor gradient of the robot ${\mathit{r}}_i$ is defined as follows:

$${\nabla }_{{\theta }_i^{\mu }}J\left({\theta }_i^{\mu }\right)={\mathbb{E}}_{o^t,a^t}\left[{\left.{\nabla }_{{\theta }_i^{\mu }}{\mu }_i\left(o_i^t;{\theta }_i^{\mu }\right){\nabla }_{a_i}Q\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^Q\right)\right|}_{a_i={\mu }_i\left(o_i^t;{\theta }_i^{\mu }\right)}\right]$$

(13)

Then, the update law of the actor network parameter ${\theta }_i^{\mu }$ is:

$${\theta }_i^{\mu }={\theta }_i^{\mu }-{\alpha }_A{\nabla }_{{\theta }_i^{\mu }}J\left({\theta }_i^{\mu }\right)$$

(14)

where ${\alpha }_A$ is the learning rate of the actor network.

3.3. Combining Prioritized Experience Replay

In order to reuse historical experience data, the experience replay mechanism was first introduced in the DQN algorithm to store historical data in the experience replay buffer, which greatly improves the experience utilization [26]. However, the random sampling approach ignores the importance of the experience data, which may lead to the uneven quality of the sampled experience data and, thus, reduce the learning efficiency. To solve the above problem, the prioritized experience replay (PER) method is proposed, and the PER method prioritizes each experience datum according to its importance and is able to select the more important experiences more times when sampling, thus increasing the learning efficiency [29]. The PER method performs well in single-agent algorithms such as DQN and DDPG. To address the characteristics of the multi-agent CTDE framework, this paper used a centralized experience buffer to store the joint information of all the agents and, then, introduced the prioritized experience replay mechanism to prioritize the sampling experience according to the experience quality.

On the other hand, TD-error is expressed as the difference between the estimated Q value and the target Q value, which is usually used to update the estimate of the critic network, so it can be used as a correction to the estimate, so the TD-error can reflect how well the algorithm assesses the true value of the experience. Samples with larger TD-error values are given higher priority, and conversely, samples with smaller TD-error values are given lower priority. The TD-error derived by robot ${\mathit{r}}_i$ for sample k through two critic networks is shown in Equations (15) and (16).

$${\delta }_{i,1}^k=y_i^t-Q_1\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_1}\right)$$

(15)

$${\delta }_{i,2}^k=y_i^t-Q_2\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_2}\right)$$

(16)

In this paper, the larger value between the absolute values of two TD-errors was chosen as the priority of sample k, as shown in Equation (17).

$$p_k=max\left(\left|{\delta }_{i,1}^k\right|,\left|{\delta }_{i,2}^k\right|\right)$$

(17)

Then, the sampling probability of sample k can be defined as:

$${\tilde{p}}_k=\frac{{\left(p_k\right)}^{\alpha }}{{\sum }_l{\left(p_l\right)}^{\alpha }}$$

(18)

where ${\tilde{p}}_k$ is the sampling probability of sample k. ${\left(p_k\right)}^{\alpha }$ denotes the priority of sample k in the replay buffer. $\alpha$ is a hyperparameter from 0 to 1 that affects the priority, where $\alpha$ is 0, the sampling criterion is random sampling, and l denotes the number of samples in the replay buffer.

Using the PER method would have the problem of changing the state distribution. Before the PER method was proposed, the replay buffer existed to address the data correlation and make the sample data independently and identically distributed. However, using the PER would more frequently replay experiences with higher TD-error values, which would inevitably introduce bias. In order to avoid bias leading to oscillations or even dispersion during training, the importance sampling method is introduced to appropriately reduce the weights of good samples so that the impact of each sample in the gradient descent is the same during training, thus ensuring the convergence of the results. The sampling weights for sample k are defined as:

$${\omega }_k=\frac{1}{l^{\beta }·p_k^{\beta }}$$

(19)

where $\beta$ is a 0 to 1 hyperparameter that determines the impact of improving the PER on the convergence result. The larger $\beta$ is, the greater the sampling weight of the lower-priority samples will be. To improve the stability of training, we normalized the weights and derived them as follows:

$${\omega }_k=\frac{{(l\ast {\tilde{p}}_k)}^{-\beta }}{{max}_m\left({\omega }_m\right)}=\frac{{\left(l\ast {\tilde{p}}_k\right)}^{-\beta }}{{max}_m\left({(l\ast {\tilde{p}}_m)}^{-\beta }\right)}=\frac{{\left({\tilde{p}}_k\right)}^{-\beta }}{{max}_m\left({\left({\tilde{p}}_m\right)}^{-\beta }\right)}$$

(20)

where m is the sample ordinal number with the largest weight. Therefore, the two critic network loss functions in the MATD3 algorithm can be rewritten as:

$$J\left({\theta }_i^{Q_1}\right)={\mathbb{E}}_{o^t,a^t,r^t,o^{t+1}}\left[{\omega }_k\ast {\left(Q_1\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_1}\right)-y_i^t\right)}^2\right]$$

(21)

$$J\left({\theta }_i^{Q_2}\right)={\mathbb{E}}_{o^t,a^t,r^t,o^{t+1}}\left[{\omega }_k\ast {\left(Q_2\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_2}\right)-y_i^t\right)}^2\right]$$

(22)

where ${\omega }_k$ is the sampling weight of sample k. $y_i^t$ is the target Q value of robot ${\mathit{r}}_i$, and $Q_j\left(o_1^t,\dots ,o_N^t,a_1^t,\dots ,a_N^t;{\theta }_i^{Q_j}\right);j=1,2$ is the estimated Q value of the critic network.

The flow of the MATD3 algorithm based on prioritized experience replay is shown in Algorithm 1. Steps 1 and 2 initialize the network parameters of each robot and the corresponding target network parameters. Step 3 initializes the training parameters, including the conditioning coefficients $\alpha$ and sampling weight coefficients $\beta$, the experience buffer D, the number of experiences K for each extraction, and the network update frequency d. Steps 4 to 26 introduce the framework process of the MATD3 algorithm for the PER, as shown in Section 3.1. o denotes the set of all robot observations. $a_i$ denotes the action of robot ${\mathit{r}}_i$, where $\mathcal{N}(0,\sigma )$ is the random noise in the training. MATD3 calculates the priority $p_k$ and sampling weight ${\omega }_k$ for each experience during sampling according to the formula in Section 3.3 and subsequently updates the critic network and actor network. Finally, a soft update of the parameters of the target network is performed.

Algorithm 1 MATD3 with prioritized experience replay.

1: Create actor network $\mu$ and two critic networks $Q_1$, $Q_2$ by using ${\theta }_i^{\mu }$, ${\theta }_i^{Q_1}$, and ${\theta }_i^{Q_2}$  2: Initialize target $\mu$, target $Q_1$, and target $Q_2$ by using ${\widehat{\theta }}_i^{\mu }$, ${\widehat{\theta }}_i^{Q_1}$, and ${\widehat{\theta }}_i^{Q_2}$  3: Initialize priority parameters $\alpha$, $\beta$; replay buffer D; minibatch K; update policy frequency d  4: for episode=1 to total episode do  5:     Generate random noise for action exploration  6:     Acquire original observations $o=(o_1,...,o_N)$  7:     for step=1 to max episode length do  8:            for each robot i, select actions $a_i={\mu }_i\left(o_i\right)+\mathrm{clip}(\mathcal{N}(0,\sigma ),-c_i,c_i)$  9:            Execute actions $(a_1^t,...,a_N^t)$; observe $r_i^t$ and $o_i^{t+1}$ 10:          Store $(o^t,a_1^t,...,a_N^t,r^t,o^{t+1})$ in D 11:          $s^t=s^{t+1}$ 12:          for robot $i=1$ to N do 13:             for $j=1$ to K do 14:                  Sample experience k with probability $p_k$ from D 15:                  Compute weight ${\omega }_k$ and TD-error ${\delta }_k$ 16:                  Update the priority of k according to $|{\delta }_k|$ 17:             end for 18:             Calculate target critic Q value via (7) 19:             Update critic by minimizing the loss via (21), (22) 20:             if $t\%d=0$ then 21:                  Update policy ${\mu }_i$ with gradient via (13) 22:             end if 23:          end for 24:          Use (4) and (6) soft update target network parameters ${\widehat{\theta }}_i^{\mu }$, ${\widehat{\theta }}_i^{Q_1}$, and ${\widehat{\theta }}_i^{Q_2}$ 25:     end for 26: end for

4. Multi-Robot Formation Strategy Design Based on Improved MATD3

4.1. Robot Formation MDP Design

We designed a Markov decision process for robot formation based on a leader-following strategy. Using the leader-following strategy allows for a better description of the relationships between the formation members and makes it easier to implement and adjust the formation. In addition, to distinguish the relationship between the leader and followers and to make the formation more scalable, each robot has its own MDP. Robots in the formation are divided into three different categories: leader robot, left-following robots, and right-following robots, where:

(1) The leader needs to find an optimal strategy to plan the shortest path to reach the goal point while avoiding obstacles.

(2) The followers need to maintain the formation by keeping a predetermined distance from other robots.

4.2. State and Action Space Design

The state space is defined according to the specific types of robots in the formation. For the leader robot ${\mathit{r}}_{lea}$, its main goal is to find an optimal path to reach the target point, so the state information includes its own position $S_{lea}^p$ and velocity $S_{lea}^v$, the target point position $S_g^p$, and the detection information of the obstacle position $S_{lea}^o$, specifically expressed as:

$$\begin{array}{c}{S}_{leader}=\left[S_{lea}^p,S_{lea}^v,S_g^p,S_{lea}^o\right]=\left[p_{lea}^x,p_{lea}^y,v_{lea}^x,v_{lea}^y,p_{lea}^x-p_g^x,p_{lea}^y-p_g^y,p_{lea}^x-p_{oj}^x,p_{lea}^y-p_{oj}^y\right],j=1,\dots ,M\hfill \end{array}$$

(23)

For the following robots, the left-following robots and the right-following robots share the same state space. The state information of the following robot ${\mathit{r}}_i$ includes its own position $S_i^p$ and velocity $S_i^v$, the other robots’ positions $S_l^p$, and the detection information of the obstacle position $S_i^o$, specifically expressed as:

$$\begin{array}{c}{S}_{follower}^i=\left[S_i^p,S_i^v,S_l^p,S_i^o\right]=\left[p_i^x,p_i^y,v_i^x,v_i^y,p_l^x,p_l^y,p_i^x-p_{oj}^x,p_i^y-p_{oj}^y\right],j=1,\dots ,M\hfill \end{array}$$

(24)

where $i\in [1,\dots ,N-1],l\in [1,\dots ,i-1,i+1]$.

The action space of robot ${\mathit{r}}_i$ is the magnitude of the force, i.e.,

$$A_i={\left[F_i^x,F_i^y\right]}_{i=1,\dots ,N}$$

(25)

4.3. Reward Function Design

The reward function of the robot is designed according to the formation control problem of multiple robots. To avoid the problem of low training efficiency due to sparse rewards, we used a combination of guided rewards and sparse rewards. In the process of robot–environment interaction, if reaching the target point, the collision between robots, the collision between robots and obstacles, or reaching the preset spacing between robots occurs, then the sparse reward is used; if none of them occurs, then the guided reward is used.

Different types of robots have different roles, and for the leader robot, the design of the guided reward function:

$$R_G^{leader}=-{∥p_{lea}-p_g∥}_2$$

(26)

and sparse reward:

$$R_S^{leader}=\left\{\begin{array}{c}{R}_1,\mathrm{if}{∥p_{lea}-p_g∥}_2\le {\rho }_g\hfill \\ R_4,\mathrm{if}{∥p_{lea}-p_{oj}∥}_2\le {\rho }_o\hfill \\ R_5,\mathrm{if}{∥p_{lea}-p_i∥}_2\le {\rho }_r\hfill \end{array}\right.$$

(27)

where $i=1,\dots ,N-1$, $R_1$ is the leader robot’s reward for reaching the target point, $R_4$ is the penalty for the collision of the leader robot with the obstacle, $R_5$ denotes the penalty for the collision of the leader robot with the other robots, ${\rho }_g$ denotes the target point radius, ${\rho }_o$ denotes the obstacle safety distance, and ${\rho }_r$ denotes the collision safety distance.

When the leader robot enters the obstacle area, the formation will change to a more passable one. Specifically, the formation strategy will be changed when the leader robot satisfies $p_{o,min}^x<p_{lea}^x<p_{o,max}^x$ and $p_{o,min}^y<p_{lea}^y<p_{o,max}^y$, where $p_{o,min}^x$, $p_{o,max}^x$, $p_{o,min}^y$, $p_{o,max}^y$ is the range of the obstacle area.

We set up a hybrid reward mechanism for the following robot. When there is no obstacle area, the robot teams form a V-shaped formation to perform the formation task, and each following robot needs to maintain a preset distance from all its neighboring robots, as shown in Figure 2; when entering the obstacle area, each following robot only needs to maintain a preset distance from a specific neighboring robot in order to form a chain formation structure, as shown in Figure 5. Set the follower guide reward function:

$$R_G^{follower}=\left\{\begin{array}{c}-\mid \begin{array}{c}{d}_i^j-d_i^{j^{*}}{\mid }_{}\end{array},\mathrm{if}\mathrm{beyond}\mathrm{the}\mathrm{obstacle}\mathrm{area}\hfill \\ -\mid \begin{array}{c}{d}_n^m-d_n^{m^{*}}\mid \end{array},\mathrm{others}\hfill \end{array}\right.$$

(28)

and sparse reward:

$$R_S^{follower}=\left\{\begin{array}{c}{R}_2,\mathrm{if}\mathrm{beyond}\mathrm{the}\mathrm{obstacle}\mathrm{area}\mathrm{and}{d}_i^j=d_i^{j^{*}}\hfill \\ R_3,\mathrm{if}\mathrm{not}\mathrm{beyond}\mathrm{the}\mathrm{obstacle}\mathrm{area}\mathrm{and}{d}_n^m=d_n^{m^{*}}\hfill \\ R_4,\mathrm{if}{∥p_i-p_{oj}∥}_2\le {\rho }_o\hfill \\ R_5,\mathrm{if}{∥p_i-p_j∥}_2\le {\rho }_r\hfill \end{array}\right.$$

(29)

where $d_i^j$ and $d_i^{j^{*}}$ denote the actual relative distance and the desired relative distance of any two robots in the obstacle-free area, respectively. $d_n^m$ and $d_n^{m^{*}}$ denote the actual relative distance and desired relative distance between the left-side $mth$ following robot and the right-side $nth$ following robot when in the obstacle area, respectively. $R_2$ and $R_3$ denote the reward for maintaining the formation when the formation is in the non-obstacle and obstacle areas, respectively. $R_4$ and $R_5$ denote the penalty for collision between any robot and any obstacle and the penalty for collision between any two robots, respectively.

5. Simulation Experiments

5.1. Experimental Environment Setup

The experimental hardware environment of this paper was an Intel Core i7-10700K CPU and NVIDIA GeForce RTX 3090 GPU, and a multi-robot formation simulation training environment was built based on the OpenAI platform. The simulation environment was a square two-dimensional plane with a side length of 8 m. Five robots existed in the environment, and one of them was the formation leader; the radius of each robot was set to 0.1 m, and the maximum speed limit was 0.85 m/s. Four obstacles and one target point existed in the environment; the radius of the obstacles was randomly distributed between (0.4 m, 0.6 m), and the radius of the target point was 0.05 m. At the beginning of the training, the obstacles and target point were randomly generated within a fixed range.

5.2. Training Parameters’ Setting

This experiment used the MADDPG algorithm, the MATD3 algorithm, and the improved MATD3 algorithm for training. The network structure was the same for all three algorithms, and each network contained two hidden layers with 128 nodes in each layer. The activation function used was ReLU, and the optimizer was Adam. The maximum number of training episodes was 60,000; the maximum number of time steps per episode was 150; the learning rate was set to 0.001; the network parameters were updated every 100 steps; the discount factor $\gamma$ was 0.95; the parameter $\lambda$ was set to 0.2; the replay buffer size was ${10}^6$; the training batch size was 1024; parameters $\alpha$ and $\beta$ were set to 0.6 and 0.5, respectively; the robot detection distance ${\rho }_d$ was set to 0.2, and the collision safety distance ${\rho }_r$, as well as the obstacle safety distance ${\rho }_o$ were set to 0.1; the obstacle area parameters $p_{o,min}^x$, $p_{o,max}^x$, $p_{o,min}^y$, $p_{o,max}^y$ were set to −3, 3, −2, 2, respectively; the locations of the target points were randomly generated within the range $(3,3.5)$.

5.3. Experimental Results and Analysis

The goal of reinforcement learning is to maximize the cumulative discounted reward, so the cumulative episode average reward is commonly used to measure the goodness of the training results, and we used the average reward of the last 1000 episodes as the final episode average reward value. Training was performed using the three algorithms described above, respectively. The episode average rewards are shown in Figure 6. It can be seen that the three algorithms can learn better policies around 20,000 episodes and gradually reach convergence. In comparison, the improved MATD3 algorithm had the highest average reward after convergence and had better stability compared to the other two algorithms.

On the other hand, to measure the ability of the robot formations to avoid collisions, we set two metrics, including the collision rate between robots (CBRS) and the collision rate between robots and obstacles (CBRO). They were obtained based on whether the distance between the robots was less than ${\rho }_r$ and whether the distance between the robots and obstacles was less than ${\rho }_o$, as shown in Equation (30).

$$\left\{\begin{array}{c}CBRS=\frac{\mathrm{Number}\mathrm{of}\mathrm{collisions}\mathrm{between}\mathrm{robots}}{\mathrm{Number}\mathrm{of}\mathrm{time}\mathrm{steps}\mathrm{per}\mathrm{episode}}\hfill \\ CBRO=\frac{\mathrm{Number}\mathrm{of}\mathrm{collisions}\mathrm{between}\mathrm{the}\mathrm{robot}\mathrm{and}\mathrm{the}\mathrm{obstacle}}{\mathrm{Number}\mathrm{of}\mathrm{time}\mathrm{steps}\mathrm{per}\mathrm{episode}}\hfill \end{array}\right.$$

(30)

The experimental results are shown in Figure 7. Obviously, the improved MATD3 algorithm had better performance in formation collision avoidance.

The path-planning ability of the leader robot directly determined the success of the formation navigation task, so we used the final distance between the leader robot and the target point (UDBLT) as one of the algorithm metrics. The experimental results are shown in Figure 8. All three algorithms can converge to near 0, indicating that the leader robot can successfully plan the path to the target point.

Formation stability error (RFSE) is an important indicator of robot formation navigation, which we define according to the following equation.

$$e\left(\mathit{r}\right)=\frac{1}{T}\sum _{t=0}^T\sum _{i,j}^N\left|d_i^j-d_i^{j^{*}}\right|,i\ne j$$

(31)

where $d_i^j$ and $d_i^{j^{*}}$ denote the actual relative distance and the desired relative distance between robot ${\mathit{r}}_i$ and robot ${\mathit{r}}_j$, respectively. The experimental results are shown in Figure 9. It can be seen that the improved MATD3 algorithm had a lower formation stability error compared to the other algorithms, indicating that the multiple robots can maintain the preset formation well.

Table 1. Detailed comparison of experimental results.

Algorithm	CBRO (%)	UDBLT	RFSE
MADDPG	3.72	0.15	2.28
MATD3	4.29	0.05	1.09
PER-MATD3	2.37	0.08	0.36

shows the comparative data of the above metrics in more detail. According to Table 1, the improved MATD3 algorithm performed better in terms of collision avoidance and formation stability errors in the obstacle environment.

The trained multi-robot formation path is shown in Figure 10, where the robot team formed a preset formation at Step 10 and was moving toward the target area. At Step 35, the robot formation completed the formation change and entered the obstacle area. At Step 95, the robot formation passed the obstacle area, resumed the preset formation, and continued to move towards the target area. At the end, the robot formation successfully navigated to the target area. As can be seen, due to the hybrid reward mechanism, the robot formation can reasonably change its formation when facing the obstacle area and, thus, achieve obstacle avoidance more easily.

6. Conclusions

This paper investigated the multi-robot formation navigation problem in environments with obstacles using reinforcement learning methods, solved the overestimation problem using the TD3 algorithm, and made the algorithm training more efficient by introducing the prioritized experience replay method. The formation control strategy was proposed in combination with the leader-following method. Facing the random distribution of environmental obstacles, a hybrid reward mechanism was designed to change the formation shape to collaboratively avoid the obstacles. The final experimental results showed that the improved algorithm in this paper had a lower collision rate and better formation stability. How to apply the algorithm to dynamic environments with obstacles and reduce the training time comprises our future research direction. It would also be interesting to integrate the physics-based obstacle-detecting method into our learning-based framework [32,33,34,35,36,37,38,39].