Multi-AUVs Cooperative Target Search Based on Autonomous Cooperative Search Learning Algorithm

Abstract

As a new type of marine unmanned intelligent equipment, autonomous underwater vehicle (AUV) has been widely used in the field of ocean observation, maritime rescue, mine countermeasures, intelligence reconnaissance, etc. Especially in the underwater search mission, the technical advantages of AUV are particularly obvious. However, limited operational capability and sophisticated mission environments are also difficulties faced by AUV. To make better use of AUV in the search mission, we establish the DMACSS (distributed multi-AUVs collaborative search system) and propose the ACSLA (autonomous collaborative search learning algorithm) integrated into the DMACSS. Compared with the previous system, DMACSS adopts a distributed control structure to improve the system robustness and combines an information fusion mechanism and a time stamp mechanism, making each AUV in the system able to exchange and fuse information during the mission. ACSLA is an adaptive learning algorithm trained by the RL (Reinforcement learning) method with a tailored design of state information, reward function, and training framework, which can give the system optimal search path in real-time according to the environment. We test DMACSS and ACSLA in the simulation test. The test results demonstrate that the DMACSS runs stably, the search accuracy and efficiency of ACSLA outperform other search methods, thus better realizing the cooperation between AUVs, making the DMACSS find the target more accurately and faster.

1. Introduction

With the development of technology and oceanic applications, AUV (autonomous underwater vehicle) has played an important role in marine applications. Compared with HOV (human occupied vehicle) and ROV (remotely operated vehicle), AUV is an unmanned agent without cables and can accomplish the work independently, safely and efficiently [1]. Relying on these advantages, AUV has been widely used in minefield search, reconnaissance, and anti-submarine, marine exploration, marine rescue, marine observation [2,3,4,5], etc. However, limited operational capability and sophisticated mission environments make a single AUV unable to meet the requirements of high efficiency and large-scale missions. MAS (multi-AUVs system) provides a new method to overcome these difficulties due to the great efficiency and high reliability brought by the space–time distribution and redundant configuration [6]. Compared with a single AUV, MAS has the following characteristics: (1) distribution, including spatial distribution and functional distribution. Spatial distribution is reflected in the fact that AUVs can be distributed in different areas for the operation to improve operational efficiency. The functional distribution shows that AUVs can carry different sensors and actuators to complete sophisticated missions through cooperation [7]. (2) Redundancy: MAS is redundant in quantity. When an AUV cannot work anymore, other AUVs in the MAS can replace it, ensuring that the mission is not interrupted [8]. The above characteristics make MAS show great advantages in applicability, economy, robustness, scalability, and is particularly beneficial to the larger missions such as underwater target search. The characteristic of the underwater target search is that the mission areas are large, and the targets are randomly distributed within the mission area, which requires the AUV to search the whole area in the shortest time to determine the location of the target [9]. Most MAS adopt centralized control structure. Although this control structure is relatively simple, once the central node fails, the entire system will be paralyzed [10,11]. Then, most search methods at this stage are pre-planning algorithms. This kind of algorithm first rasterizes the mission area and then makes the AUV scan all the grids according to a pre-planned path [12]. Although these search algorithms can accurately locate the target, they cannot produce effective cooperation behavior between AUVs, which will reduce mission efficiency. This shortcoming is unacceptable for some urgent search missions, such as shipwrecked ship searching and locating, anti-mine, and anti-submarine. In addition, limited detection, communication, endurance capability, and changing water depth will cause unexpected situations for AUVs during the mission. The pre-planning algorithm cannot adjust the search strategy based on the real-time situation [13]. Hence, for the current problem, we establish the DMACSS (distributed multi-AUVs collaborative search system) and propose the ACSLA (autonomous collaborative search learning algorithm) that is integrated into the DMACSS. This article mainly has the following contributions:

1. DMACSS and system modeling: we establish the DMACSS and build the dynamic detection and update model of DMACSS with the sophisticated environment (changing depth of water, randomly distributed targets, the uncertainty of the sensors, limited communication capability) based on the probability map model. Using this model, the search environment and the entire search process of the DMACSS are reasonably abstracted. The basis for the rationality of modeling is also provided.

2. Information communication and fusion: a local information fusion mechanism is proposed, which can use the information collected by each AUV in DMACSS to make up for the lack of perception ability of a single AUV. The information fusion mechanism combined with specially designed time stamp mechanism, that can make the information fusion process more effectively under limited communication condition, improve the search efficiency and reduce the error rate.

3. ACSLA: we propose ACSLA and integrate it with DMACSS so that AUVs in the system can get cooperative search strategies. The ACSLA is trained based on RL algorithms [14,15], it has specially designed state information, reward functions, and a new distributed training framework called SASF (single asynchronous sharing framework) to make the training process more stable and easy to converge, which is essential for search performance of DMACSS.

2. Background

In many studies, the problem of multi-agent cooperative search has become a research hotspot in academia and industry and receives widespread attention. Although some researches do not directly use AUV as the research object, it also provides a basis and reference for research on multi-AUVs.

Rajnarayan, D.G.; et al. [16] discuss the application of cooperation theory in the search mission of the multi-agents and use Radner’s decentralized cooperation theory to make the optimality of cooperation between agents the same as the global optimality. The CS (centralized strategy) and DCS (distributed collaboration strategy) are all derived. Wang, X.; et al. [17] assign multiple sensors to a set of discrete search units to find hidden targets. To solve the sensor’s uncertainty in the detection process, interference is added to the traditional discrete search formula to establish a new mathematical model. Finally, the greedy algorithm is used to optimize and solve this model. Hong, S.P.; et al. [18] combine the Markov chain with the concept of minimizing the undiscovered probability and propose a fast hybrid heuristic intelligent algorithm. Experimental results show that this heuristic algorithm can complete the search path decision in a short time. Singh [19] et al. propose a hybrid framework for guidance and navigation of swarm of unmanned surface vehicles (USVs) by combining the key characteristics of formation control and cooperative motion planning. Under this framework, a combination of offline planning and online planning is applied to the marine environment. In order to enable the USV to avoid dynamic obstacles, based on the USV maneuvering response time, using the A* algorithm based on offline optimal path planning and safe distance constraints, Mina [20] and others propose a general multi-USV navigation, guidance, and control framework. Thi, H.A.L.; et al. [21] study a hierarchical search planning model, which divides the search area space into several subspaces, and then conducts the second search planning in the search subspace. This secondary search planning method makes the entire search process efficient and precise.

Specific to AUV, Healey, et al. [22] studied the problem of complete coverage in the cooperative search process of multi-AUVs. In order to ensure complete coverage of the search area in the event of AUV loss in the formation, an effective cooperative strategy was designed and tested. Welling, et al. [23] used the multi-AUVs system to perform the cooperative search and target clear missions. They discussed the mission assignment problem and compared the two assignment strategies based on the closest distance and fuzzy logic from a time-consuming perspective. Shafer, et al. [24] studied the multi-AUVs cooperative adaptive search behavior under sophisticated environments and proposed a cooperative strategy that can make the multi-AUVs system achieve multiple missions in parallel. In addition to theoretical research, some multi-AUVs systems have already been applied in practical missions. Since REMUS (Remote Environmental Monitoring Underwater System) underwater robots have played an important role in minefield detection operations during the Iraq War, the ONR (Office of Naval Research) continuously funded several scientific research institutions to develop unmanned underwater systems. The MIT (Massachusetts Institute of Technology) carried out a research project called GOATS (Generic Oceanographic Array Technology System). The GOATS project used multi-AUVs equipped with underwater acoustic equipment to form a mobile underwater detection network which was used to search for mines in coastal waters [25,26,27,28]. Based on the GOATS project, the research team consisting of NURC (National Undersea Research Center) and MIT launched a project called Generic Littoral Interoperable Network Technology (GLINT) in 2008. The multi-AUVs system in the project is equipped with various sensors to complete the missions of automatic detection, positioning and tracking of specific targets [29]. In Europe, from 2012 to 2015, relevant research institutes in Italy, Estonia, the United Kingdom, Spain, and Turkey jointly launched a research project called ARROWS (Archaeological Robot System for the World’s Seas) [30], which aimed to use multi-AUVs systems to improve the submarine scanning efficiency and research on the mission allocation strategy [31] and underwater communication [32]. At this stage, most MACSS systems adopt a centralized control structure and use a pre-planned method without considering real-time motion characteristics of AUV. To this end, we established the DMACSS system based on the distributed control structure and proposed the ACSLA algorithm that can realize real-time planning of search paths to adapt to complex environments.

3. Modeling

In order to establish DMACSS, we must reasonably simplify and abstract the mission environment according to the actual mission situation and effectively model the mission environment, each part of the system, and the system working process.

3.1. Environment Model

In order to effectively search for targets in the mission environment, DMACSS must keep updating the environment state in terms of limited target information. To this end, the probability map model is used. As we all know, the probability map model is used to model the uncertainty of the mission environment. We assume that the target is on the seafloor. Since the AUV navigates in the water, the complex terrain of the seabed has no effect on the navigation of the AUV, so we project the AUV onto a flat region, as shown in Figure 1. First, we establish an inertial coordinate system, where $\alpha$ is the coordinate origin, and the positive direction of the $x$ axis points to the east. The mission area of $A\in {\mathrm{ℝ}}^2$ is divided into $L_x\times L_y$ grids, each grid is called a target area $g$, $g=\left(m,n\right)$, $m\in \left\{1,2,3\cdots ,L_x\right\},n\in \left\{1,2,3\cdots L_y\right\}$. Let ${\theta }_g=1$ denote that there is a target in the $g$, otherwise ${\theta }_g=0$ indicates that there is no target in the $g$. Similarly, ${\mathcal{P}}_{g,k}$ can be expressed as the probability of target existence in the grid $g$ at time $k$, ${\mathcal{P}}_{g,k}\in \left[0,1\right]$. Where ${\mathcal{P}}_{g,k}=1$ indicates that there must be a target in the $g$, and ${\mathcal{P}}_{g,k}=0$ indicates that there must be no target in the $g$. Before the start of the mission, each grid has an a priori initial probability, ${\mathcal{P}}_g=0.5$. Set the threshold ${\mathcal{P}}^{+}$ and ${\mathcal{P}}^{-}$ as the upper and lower limits of the ${\mathcal{P}}_{g,k}$, that is, ${\mathcal{P}}_{g,k}>{\mathcal{P}}^{+}$ means that there is a target in the $g$. ${\mathcal{P}}_{g,k}<{\mathcal{P}}^{-}$ means that there is no target in the $g$. The coordinates of each AUV are expressed as ${\mu }_{i,k}={\left[p_{i,k}^T,h_{i,k}\right]}^T\in {\mathrm{ℝ}}^3 \left(i=1,2,\cdots N\right)$. $p_{i,k}^T$ are the coordinates of the AUV projection on the $A$. $h_{i,k}$ is the depth where the AUV is located, $T$ is the conversion operation, $N$ is the total number of AUVs. The kinematics model of AUV is shown in Appendix A.

3.2. Sensor Model

At this stage, sonar is the most commonly used sensor for AUV to obtain underwater targets and other environmental information. The active sonar system and passive sonar system are two commonly used sonar systems. In the target search mission, an active sonar system is more suitable as the sensor of the system. Since the detection accuracy of the sonar system will directly affect the update of the probability graph model, in this article, we use sonar as a modeling object and focus on the relationship between the sonar accuracy and the mission environment.

The working principle of the active sonar system makes it susceptible to non-linear interference problems caused by the water media or other external factors. In addition, obstacles between the sonar and the target will also affect the accuracy of sonar. Formula (1) shows the forward-looking sonar detection model after adding constraints of nonlinear noise and obstacle:

$$I_{i,target}=\left\{\begin{array}{c}0 d_{i,target}>D_{max}\\ 0 d_{i,target}<D_{min}\\ 0 There is an obstacle between sonar and target\\ h_{target}+d_{i,target}\zeta D_{min}<d_{i,target}<D_{max}\end{array}\right.$$

(1)

Among them, $I_{i,target}$ represents the target information collected by the $AU{V}_i$, $D_{max}$ and $D_{min}$ are the maximum and minimum detection distance of the sonar, $h_{target}$ represents the sonar detection function under noise-free conditions, and $d_{i,target}$ represents the distance between the target and the $AU{V}_i$, $\zeta$ is nonlinear interference. Formula (1) indicates that when the $d_{i,target}$ not in $\left[D_{min},D_{max}\right]$, or there exist the obstacle between the sonar and target, the target information cannot be obtained. $\zeta$ has an important impact on the correctness of the $I_{i,target}$. The larger value of the $\zeta$, the lower the correctness of $I_{i,target}$. We set the probability of $AU{V}_i$ detecting the correct $I_{i,target}$ as $p_{i,d}$, and the probability of $AU{V}_i$ detecting the wrong $I_{i,target}$ as $p_{i,f}$:

$$p_{i,d}=P\left(O_{i,g,k}=1\left|{\theta }_g=1\right.\right) 1-p_{i.d}=p\left(O_{i,g,k}=0\left|{\theta }_g=1\right.\right)$$

(2)

$$p_{i,f}=P\left(O_{i,g,k}=1\left|{\theta }_g=0\right.\right) 1-p_{i,f}=p\left(O_{i,g,k}=0\left|{\theta }_g=0\right.\right)$$

(3)

As shown in Formulas (2) and (3), $p_{i,d}$ is the probability that the $AU{V}_i$ detects the target when there is a target in the grid $g$. $p_{i,f}$ is the probability that the $AU{V}_i$ detects the target when there is not a target in the grid $g$. $O_{i,g,k}$ is the observation value of $g$ by $AU{V}_i$ at time $k$. It can be seen from Formula (1) that the value of the $\zeta$ is related to the $d_{target}$, so we can get Formulas (4) and (5):

$$p_{i,d}=f_1\left(d_{i,target}\right)$$

(4)

$$\left\{\begin{array}{c}\check{p_{i,d}} d_{i,target}<D_{min}\\ f_1\left(x\right)=K_1{e}^{-K_1{\left(d_{i,target}-D_{min}\right)}^2} D_{min}<d_{i,target}<D_{max}\\ 0.5 d_{i,target}>D_{max}\end{array}\right.$$

where $f_1^{\prime }\left(d_{i,target}\right)<0$ for $d_{i,target}\in \left(D_{min},D_{max}\right)$ and $1>f_1\left(D_{min}\right)=\check{p_{i,d}}>f_1\left(D_{max}\right)=0.5$.

$$p_{i,f}=f_2\left(d_{i,target}\right)$$

(5)

$$\left\{\begin{array}{c}\widehat{p_{i,f}} d_{i,target}<D_{min}\\ f_2\left(x\right)=K_1{e}^{K_2{\left(d_{i,target}-D_{min}\right)}^2} D_{min}<d_{i,target}<D_{max} \\ 0.5 d_{i,target}>D_{max}\end{array}\right.$$

where $f_2^{\prime }\left(d_{i,target}\right)>0$ for $d_{i,target}\in \left(D_{min},D_{max}\right)$ and $0<f_2\left(D_{min}\right)=\widehat{p_{i,f}}<f_2\left(D_{max}\right)=0.5$.

Remark 1. Due to the sophisticated marine environment (for example, the temperature of seawater, salinity, and ocean currents etc.), the model may change with different locations. In this article, we do not consider this situation, so the sensor model does not change in the whole mission area.

When the targets are located on the plane of the seabed, from Figure 2, we can get the relationship between the $d_{i,target}$, $h_{i,k}$ and $H$ (the water depth at its location) as shown in Formula (6):

$$d_{i,target}=\frac{\left(H-h_{i,k}\right)}{cos\left(\frac{\beta }{2}\right)}$$

(6)

where $\beta$ is the opening angle of the sonar.

3.3. Environment Update Model

As the mission continues, the system’s perception of the environment is constantly changing, so we need to update the environment model based on the latest detection results. ${\mathcal{P}}_{i,g,k}$ represents the probability that the grid $g$ has a target at time $k$ which is detected by $AU{V}_i$. Let $m_g$ represent the number of times the $g$ is searched, then we can build Formula (7):

$${\mathcal{P}}_{i,g,k}=\left\{\begin{array}{c}1 if m_g\to \infty and {\theta }_g=1\\ 0 if m_g\to \infty and {\theta }_g=0\end{array}\right.$$

(7)

Formula (7) indicates that when there is a target in the $g$ and the number of times that the $g$ is searched tends to infinity, ${\mathcal{P}}_{i,g,k}\to 1$, that is, there must be a target in the $g$. When there is no target in the $g$ and the number of times that the $g$ is searched tends to infinity, ${\mathcal{P}}_{i,g,k}\to 0$, that is, there must be no target in the $g$. Next, we update ${\mathcal{P}}_{i,g,k}$. According to the Bayesian empirical formula, the update formula of ${\mathcal{P}}_{i,g,k}$ is obtained:

$$\begin{array}{ll}{\mathcal{P}}_{i,g,k}& =\frac{p(O_{i,g,k}|{\theta }_g=1){\mathcal{P}}_{i,g,k-1}}{p\left(O_{i,g,k}|{\theta }_g=1\right){\mathcal{P}}_{i,g,k-1}+p(O_{i,g,k}|{\theta }_g=0)(1-{\mathcal{P}}_{i,g,k-1})}\\ & =\left\{\begin{array}{cc}\hfill \frac{p_{i,d}{\mathcal{P}}_{i,g,k-1}}{p_{i,d}{\mathcal{P}}_{i,g,k-1}+p_{i,f}(1-{\mathcal{P}}_{i,g,k-1})},& if O_{i,g,k}=1\hfill \\ \hfill \frac{\left(1-p_{i,d}\right){\mathcal{P}}_{i,g,k-1}}{\left(1-p_{i,d}\right){\mathcal{P}}_{i,g,k-1}+\left(1-p_{i,f}\right)(1-{\mathcal{P}}_{i,g,k-1})},& if O_{i,g,k}=0\hfill \\ \hfill {\mathcal{P}}_{i,g,k-1}& otherwise\hfill \end{array}\right.\end{array}$$

(8)

where ${\mathcal{P}}_{i,g,k-1}$ represents its a priori probability. Convert Formula (8) from nonlinear to linear form based on the Formula (9) to facilitate the calculation for more efficient:

$${\mathcal{L}}_{i,k}\triangleq \ln n\left(\frac{1}{{\mathcal{P}}_{i,g,k}}-1\right)$$

(9)

The simplified update formula of probability map is shown in Formula (10):

$${\mathcal{L}}_{i,g,k}={\mathcal{L}}_{i,g,k-1}+{𝓆}_{i,g,k}$$

(10)

$${𝓆}_{i,g,k}=\left\{\begin{array}{cc}\hfill ln\left(\frac{p_{i,f}}{p_{i,d}}\right),& if O_{i,g,k}=1\hfill \\ \hfill ln\left(\frac{1-p_{i,f}}{1-p_{i,d}}\right),& if O_{i,g,k}=0\hfill \end{array}\right.$$

(11)

According to Formula (10), it can be proved that Formula (7) still holds and the proof process is shown in Appendix B. In Appendix C, we deduce the relationship between $p_{i,d}$, $p_{i,f}$, and the convergence rate of the probability map.

3.4. Search Information Fusions

Due to the limitation of sensor performance, each AUV can only observe a limited area, so it can only update the probability map of the area within the observation radius $R_s$, and because the basic probability map model does not integrate the state information of other AUVs, each AUV cannot grasp the global probability map information. This will negatively affect the search speed of the system. Therefore, to compensate for the limited detection capability of each AUV, the probability map model must be improved by building an information fusion mechanism that makes each AUV in the system faster converge to the same probability map reflecting the target location.

The communication between AUVs is the basis for building the information fusion mechanism. Same as the observation capability of AUV, the communication capability of AUV is also restricted by equipment and environment, the $AU{V}_i$ can only communicate with AUVs within a communication radius $R_c$. Therefore, we use the ${{\rm N}}_{i,k}$ represent the neighbor of the $AU{V}_i$ at time $k$:

$${{\rm N}}_{i,k}=\left\{j|\left\{i,j\right\}\in \Vert {\mu }_{i,k}-{\mu }_{j,k}\Vert \le R_c\right\}\cup \left\{i\right\}$$

(12)

According to the number of AUVs in the ${{\rm N}}_{i,k}$, we can divide ${{\rm N}}_{i,k}$ into different levels as $d_{i,k}=\left|{{\rm N}}_{i,k}\right|$. Using $d_{i,k}$, we can calculate the fusion matrix $w$: $w_{i,i,k}=1-\left(\left(d_{i,k}-1\right)/N\right)$, $w_{i,j,k}=\left(1/N\right)$ for $j\in {{\rm N}}_{i,k}$ $\left(j\ne i\right)$ and $w_{i,j,k}=0$ for $j\notin {{\rm N}}_{i,k}$. $AU{V}_i$ searches at time $k$ and stores ${\mathcal{L}}_{i,g,k}$, then transmit ${\mathcal{L}}_{i,g,k}$ to neighbors and use Formula (13) to update ${\mathcal{L}}_{i,g,k}$:

$${\mathcal{L}}_{i,g,k}={\mathcal{L}}_{i,g,k-1}+{{\displaystyle \sum }}_{j\in {{\rm N}}_{i,k}}{q}_{j,g,k}$$

(13)

Next, through the $w_{i,j,k}$, the updated ${\mathcal{L}}_{i,g,k}$ is merged with the ${\mathcal{L}}_{j,g,k}$ of other AUVs, as shown in Formula (14):

$${\mathcal{H}}_{i,g,k}={{\displaystyle \sum }}_{j\in {{\rm N}}_{i,k}}{w}_{i,j,k}{\mathcal{L}}_{j,g,k}$$

(14)

For the entire system, we first define the parameters for the DMACSS:

$${{\rm Y}}_{g,k}\triangleq {\left[{\mathcal{H}}_{1,g,k},{\mathcal{H}}_{2,g,k},\cdots ,{\mathcal{H}}_{N,g,k}\right]}^T$$

(15)

$${\Phi }_{g,k}\triangleq {\left[q_{1,g,k},q_{2,g,k},\cdots ,q_{N,g,k}\right]}^T$$

(16)

$${\left[w_k\right]}_{i,j}=w_{i,j,k}$$

(17)

Then, we can obtain the update rules of information fusion mechanism as shown in Formula (18):

$${{\rm Y}}_{g,k}=\left({\displaystyle \prod }_{t=1}^k{w}_t\right){{\rm Y}}_{g,0}+{\displaystyle \sum }_{l=1}^k\left({\displaystyle \prod }_{t=l}^k{w}_t\right){\Phi }_{g,k}$$

(18)

3.5. Time Stamp

In order to improve the efficiency of information fusion, in addition to the information fusion mechanism, we also propose a time stamp mechanism. Specifically, when each AUV fuses the probability map, it is also necessary to transmit a timestamp map. The timestamp can prevent the AUV from repeatedly fusing some information about the determined search grid, and improve search efficiency. Set the timestamp as $t_{i,g}$, which means that the $AU{V}_i$ has updated the latest probability search map of the grid $g$. We establish three rules for time stamp mechanism:

(1) When the $AU{V}_i$ detects the grid at the current time, the update of ${\mathcal{P}}_{i,g,k}$ comes from its detection behavior. At this time, the current grid $g$ and the timestamp $t_{i,g}$ are updated to the current time $k$ as $t_{i,g,k}.$.

(2) When the observation area of the AUV in the system overlaps, the $AU{V}_i$’s probability map update comes from the information fusion within its communication range, update the timestamp $t_{i,g}$, which is the timestamp closest to the current $AU{V}_i$’s position.

(3) When the AUV fuses information, it not only transmits the probability map information but also needs to interact with the information of the timestamp. When the AUV encounters different timestamps, information fusion will happen.

4. Search Method

In this section, we will introduce the search method that is integrated with DMACSS, so that each AUV in DMACSS can get a cooperative control strategy that can make the AUV obtain the best search trajectory based on the value of real-time detection and maximize system search capability.

Through the tailored design of state information, reward function, and training framework, ACSLA is proposed for the multi-AUVs cooperative target search. In order to compare the effects of different RL algorithms (the RL algorithm is introduced in Appendix D) when training ACSLA, we used the deep Q-network (DQN) algorithm based on value iteration and the deep deterministic policy gradient algorithm (DDPG) algorithm based on policy gradient to train ACSLA, respectively. Below we will describe in detail state information, reward function, and training framework.

4.1. State Information

For the agent, the essence of the strategy is a mapping from state to action, therefore the state information must be able to fully reflect the state of the agent at each step so that the agent can choose the correct action. Using the model we built in the previous chapter, the state information of each AUV in the DMACSS include two pieces of information, one is the target information ${\mathcal{H}}_{i,k}$ and another is cooperation information $C_{i,k}$:

$$S_{i,k}=\left\{{\mathcal{H}}_{i,k},C_{i,k}\right\}$$

(19)

${\mathcal{H}}_{i,k}$ represents the target information which is the probability map of the whole area obtained by ${\mathcal{H}}_{i,g,k}$:

$${\mathcal{H}}_{i,k}={\displaystyle \sum }_{g\in \left\{1,2,\cdots L_x\times L_y\right\}}{\mathcal{H}}_{i,g,k}$$

(20)

$C_{i,k}$ represents the coordinate information of other AUVs in the $2R_c$ of $AU{V}_i$. The role of this information is to make the AUVs have the ability to the cooperative. However, we cannot directly input AUV’s coordinate information as cooperation information into the ACSLA, because the coordinate information will be interfered with by the probability map information, the ACSLA cannot extract the corresponding feature information. To this end, we have processed the coordinate information accordingly, that is, converting the coordinate information into a cooperation map of size $2R_c\times 2R_c$:

$$C_{i,k}={\displaystyle \sum }_{g\in \Vert g-{\mu }_{i,k}\Vert \le 2R_c}{c}_{i,g,k}$$

(21)

where $c_{i,g,k}$ is the Gaussian distribution centered on the coordinates of the neighboring of $AU{V}_i$. Using the branch structure of the convolutional neural network, the features of the two parts of state information can be extracted separately for the agent to learn.

4.2. Reward Function

The reward function is the core part of the ACSLA. Appropriate reward function can guide agents to learn appropriate strategies, without making the strategy unable to converge or fall into a local optimum. To design a reasonable reward function, we must first clarify the goal of the mission: (1) accurately locate all the targets in the mission area; (2) under the premise of accurately finding all the targets, the search time can be reduced as soon as possible. Among the two indicators, the former reflects the accuracy of the system, and the latter reflects the efficiency of the system. Our reward functions are designed according to these two goals.

4.2.1. Target Reward

We stipulate that when $AU{V}_i$ finds a target during the search process (only when ${\mathcal{P}}_{i,g,k}>\overline{\mathcal{P}}$, it is considered that the $AU{V}_i$ has found the target accurately at grid $g$, otherwise, it is considered that there is no target found at grid $g$), the $AU{V}_i$ will get a reward. The reward function of the initial target reward $r_{i,k}^{t1}$ is shown in Formula (22):

$$r_{i,k}^{t1}={{\displaystyle \sum }}_{g\in A}{1}_{{\mathcal{P}}_{i,g,k\ge \overline{\mathcal{P}} and {\mathcal{P}}_{i,g,k-1<\overline{ \mathcal{P}}}}}$$

(22)

When there are multiple targets in the search area, and the system finds all the location of the targets, each AUV in the DMACSS will receive a final target reward $r^{t2}$:

$$r_{i,k}^{t2}={{\displaystyle \prod }}_{g\in \left\{g\in A:{\theta }_g=1\right\}}{1}_{{\mathcal{P}}_{i,g,k\ge \overline{\mathcal{P}}}}$$

(23)

If the system does not find all the targets, it will be punished accordingly, that is, the system will receive a certain negative reward $r^{t3}$. The penalty value is related to the number of missed targets. The greater number of the missing target, the greater the penalty value. $r^{t3}$ is shown in Formula (24):

$$r_{i,k}^{t3}={{\displaystyle \sum }}_{g\in A:{\theta }_g=1}1-{{\displaystyle \sum }}_{g\in A}{1}_{{\mathcal{P}}_{i,g,k\ge \overline{p} and {\mathcal{P}}_{i,g,k-1<\overline{p}}}}$$

(24)

Finally, the target reward of $AU{V}_i$ at time $k$ is composed as:

$$r^t={\beta }_1{r}_{i,k}^{t1}+{\beta }_2{r}_{i,k}^{t2}+{\beta }_3{r}_{i,k}^{t3}$$

(25)

where ${\beta }_1,{\beta }_2,{\beta }_3$, are weight coefficients.

4.2.2. Dispersed Reward

In order to expand the search range, the AUV should be dispersed as much as possible in the search process to avoid the occurrence of multiple AUVs concentrated in a small area, resulting in a waste of search resources. To this end, we set up a dispersed reward $r_{i,k}^d$. The reward is based on the number of AUVs within the $AU{V}_i$’s communication radius to give the $AU{V}_i$’s different rewards. The larger the AUV is distributed in the search area, the more dispersed rewards the AUV gets. The distribution information of AUV is obtained by $C_{i,k}$:

$$r_{i,k}^d={\beta }_4\frac{1}{C_{i,k}}$$

(26)

In addition, in order to avoid collisions between AUVs, when the distance between $AU{V}_i$ and other AUVs is less than the obstacle avoidance radius $R_a$, the $AU{V}_i$ will receive a larger negative reward $r_{i,k}^{d_a}$:

$$r_{i,k}^{d_a}=\left\{\begin{array}{c}-{\beta }_5\Vert {\mu }_{i,k}-{\mu }_{j,k}{\Vert }^2, \exists j \Vert {\mu }_{i,k}-{\mu }_{j,k}\Vert \le R_o, j\in {{\rm N}}_{i,k} \left(j\ne i\right) \\ 0, other\end{array}\right.$$

(27)

where ${\beta }_4,{\beta }_5$ are weight coefficients.

4.2.3. Time Consumption Reward

The time-consuming reward $r_{i,k}^c$ is to evaluate strategy from the perspective of time. $r_{i,k}^c$ is designed in the form of a piecewise function to enable the AUV to obtain corresponding time consumption rewards in different stages of training:

$$r_{i,k}^c={\beta }_6\left(k_{max}-k_e+{\gamma }_{1}max\left(k_1-k_e, 0\right) + {\gamma }_{2}max\left(k_2-k_e, 0\right)\right)$$

(28)

We divide the $r_{i,k}^c$ obtained during the search process into three parts. $k_{max}-k_e$ represents the consumption rewards obtained for the entire segment of the episode. ${\gamma }_{1}max \left(k_1-k_e, 0\right)$ represents the consumption rewards of the first segment of the episode, $k_2-k_e$ represents the consumption rewards of the second segment of the episode, where ${\gamma }_1$ and ${\gamma }_2$ are the weight coefficients of different segments, $k_{max}$ is the preset maximum number of episode steps. $k_e$ is the number of the episode step of $AU{V}_i$, $k_1,k_2$ are the preset segmentation points of the step. $r_{i,k}^c$ is an overall reward, so it is the same for every AUV in the system.

4.2.4. Sparse Reward

In the initial stage of training, the AUV cannot obtain sufficient rewards to learn the search strategy because the AUV cannot find the target immediately. Therefore, to make the AUV have better learning ability in the early stage of training and make the learned strategy converge faster, we set up sparse rewards $r_{i,k}^s$:

$$r_{i,k}^s={\beta }_7\frac{{\eta }_{i,k-1}-{\eta }_{i,k}}{{\eta }_{i,k-1}}$$

(29)

where ${\beta }_4$ is the weight coefficient. Formula (29) rewards the AUV according to the uncertainty of each $g$ in the search area. The uncertainty is defined as: ${\eta }_{i,k}\triangleq \frac{1}{L_x\times L_y}{{\displaystyle \sum }}_{g\in A}{\eta }_{i,g,k}$, where ${\eta }_{i,g,k}=e^{-{\beta }_{\eta } \left|{\mathcal{L}}_{i,g,k}\right|}$, ${\beta }_{\eta }$ is a constant. ${\eta }_{i,k}$ will continue to decrease as the search progresses, so it will not affect the later stage of the training.

Finally, the reward function of $AU{V}_i$ at time $k$ is composed as:

$$r_{i,k}=r^t+r_{i,k}^d+r_{i,k}^{d_a}+r_{i,k}^c+r_{i,k}^s$$

(30)

4.3. Training Framework

An unstable training process is a common problem in MARL (multi-agent reinforcement learning). This is because each agent is part of the environment, so for any agent, the training environment is always changing. To overcome this problem, we design a new distributed training framework called SASF (single asynchronous sharing framework) for DMACSS. The schematic diagram of the SASF is shown in Figure 3. When we train the search strategy of $AU{V}_i$, other AUVs in the system use the same strategy ${\tau }_i^{\theta }$ as $AU{V}_i$ to sample the environment to provide $AU{V}_i$ with information fusion data, but we only update ${\tau }_i^{\theta }$ in real-time, the strategies used by other AUVs unchanged over a period of steps, so that the training environment can be relatively stable for some time. After a certain number of training steps, $AU{V}_i$ shares the updated strategy with other AUVs, so that the entire system can improve the search capability together. Compared with the traditional training framework, our training framework can make the training process converge stably, and will not fall into the local optimal trap during the training process. Whether it is a value-based iterative or a gradient-based RL method can use this framework to train ACSLA. The training process of ACSLA for $AU{V}_i$ is given in Algorithm 1 (trained by DQN) and Algorithm 2 (trained by DDPG). However, this training structure also has a significant shortcoming, that is, because the experience of other AUVs is not fully utilized, it takes long training time to achieve satisfactory results.

Algorithm 1 Training Process of the DQN

Initialize Q-network $Q^{\theta }$ and T-network $Q^{{\theta }^{\prime }}$ with ${\theta }^{\prime }$ = $\theta$ for the each AUV, replay buffer $\mathcal{D}$, discount factor $\gamma ,$ number of samples in minibatch $m,$ number of episodes $\overline{e},$ the maximum episode step $T$

For $episode=1,\overline{e}$ do

Initialize sequence $s_{i,k}$ and preprocessed sequence $o_{i,k}=\varphi \left(s_{i,k}\right)$

For $k=1,T$ do

select a random action $a_{i,k}$ with probability $\epsilon$

Otherwise select action $a_k$ according to $a_{i,k}=argma{x}_a{Q}^{\theta }\left(o_{i,k},a\right)$

Execute action $a_{i,k}$, receive reward $r_{i,k}$ and get next state $s_{i,k+1}$

Get observation $o_{k+1}=\varphi \left(s_{i,k+1}\right)$

Store transition $\left(o_{i,k},a_{i,k},r_{i,k},o_{i,k+1}\right)$ in $\mathcal{D}$

Sample random minibatch of transitions $\left(o_{i,j},a_{i,j},r_{i,j},o_{i,j+1}\right)$ from $\mathcal{D}$

$y_{i,j}=\left\{\begin{array}{c} r_{i,j} if \mathrm{if} \mathrm{episode} \mathrm{terminastes} \mathrm{at} \mathrm{step} j+1\\ r_{i,j}+\gamma ma{x}_{a^{\prime }}{Q}^{{\theta }^{\prime }}\left(o_{i,k+1},a^{\prime }\right) otherwise\end{array}\right.$

Perform a gradient descent step on $Q^{\theta }$ $${\left(y_{i,j}-Q^{\theta }\left(o_{i,j},a_{i,j}\right)\right)}^2$$

Every C steps reset $Q^{{\theta }^{\prime }}=Q^{\theta }$

End for

End for

Algorithm 2 Training Process of the DDPG

Initialize Actor-network $A\left(s|{\theta }^A\right)$ and critic network $C\left(s,a|w^C\right)$ for each AUV with weights ${\theta }^A$,$w^C$. Initialize target network $A^{\prime }$ and $C^{\prime }$ with weights ${\theta }^{A^{\prime }}\leftarrow {\theta }^A, w^{C^{\prime }}\leftarrow w^C$. Replay buffer $\mathcal{D}$, discount factor $\gamma ,$ update factor $\tau ,$ number of samples in minibatch $m,$ $w$ update frequency $f,$ random noise function $\mathcal{N},$ number of episodes $\overline{e}$, the maximum episode step $T$

for $episode=1,\overline{e}$ do

Initialize the environment, state$s_0$ and time $k$

while not (𝑘 ≥ $\overline{k}$ or targets are found) do

for $i\in 1,N in parallel$ do

Receive observation $o_{i,k}=\varphi \left(s_{i,k}\right)$

for $k=1,T$ do

Select action $a_{i,k}$ according to $a_{i,k}=A\left(o_{i,k}|{\theta }^A\right)+\mathcal{N}$

Execute action $a_{i,k}$, receive reward $r_{i,k}$ and get next state $s_{i,k+1}$

Get observation $o_{i,k+1}=\varphi \left(s_{i,k+1}\right)$

Store transition $\left(o_{i,k},a_{i,k},r_{i,k},o_{i,k+1}\right)$ in $\mathcal{D}$

$m$ transitions $\left(o_{i,j},a_{i,j},r_{i,j},o_{i,j+1}\right)$ are sampled from $\mathcal{D}$

Calculate the target-Q value $y_{i,j}$:

$y_j=\left\{\begin{array}{c} r_j if k\ge T\\ r_{i,j}+\gamma C^{\prime }\left(o_{i,k+1},A^{\prime }\left(o_{i,k+1}|{\theta }^{A^{\prime }}\right)|w^{C^{\prime }}\right) if k<T\end{array}\right.$

Update $w^C$:

$\frac{1}{m}{\displaystyle {\displaystyle \sum }_{j=1}^m}{\left(y_{i,j}-C\left(s,a|w^C\right)\right)}^2$

Update ${\theta }^A$:

$\frac{1}{m}{\displaystyle {\displaystyle \sum }_{j=1}^m}{\nabla }_{a}C\left(s,a|w^C\right)\left|s=s_{i,j},a=A\left(s_{i,j}\right)\right.{\nabla }_{{\theta }^A}A\left(s|w^C\right)\left|s_{i,j}\right.$

Update $w^C$ and ${\theta }^A$ if $k \% f=1$:

$w^{C^{\prime }}\leftarrow \tau w^C+\left(1-\tau \right)w^{C^{\prime }}$

${\theta }^{A^{\prime }}\leftarrow \tau {\theta }^A+\left(1-\tau \right){\theta }^{A^{\prime }}$

end for

end for

5. Simulation Test

In all simulations, the size of the entire surveillance region is set to $\left[0,25\right]\times \left[0,25\right]m^2$. The initial position of each AUV is relatively close, ensuring that all AUVs are in a state where they can communicate with each other. The speed of each AUV is uniform and the maximum steering angle of each AUV is $\theta$. We set the $R_s=6$, $R_c=8$,$R_a=2$. The depth change of the search area is shown in Figure 4.

We use Google’s TensorFlow [33] to build a simulation environment based on python. In this article, in order to improve the calculation accuracy of ACSLA, we use a neural network structure called dueling-network which is shown in Figure 5a–c. Dueling-network structure can reduce the calculation error of the algorithm. In terms of parameter update of the neural network, the optimizer is the Adam optimization method [34], the learning rate is set to $2.5\times {10}^{-5}$. The replay buffer only stores 10,000 data, the batch size is set to 40, that is, 40 samples are required for one training. The exploration strategy in training adopts the ε-greedy method, in which the probability of exploration is selected as 0.2. Figure 5d shows the reward changes of the ACSLA that are trained by different RL methods during the training process.

Environmental information entropy as an evaluation index of algorithm performance. Environmental information entropy can reflect the convergence speed of the probability map, and its calculation formula is as follows:

$$H\left(k\right)=-{\displaystyle \sum }_{g=1}^{L_x\times L_y}\left(1-\overline{{\mathcal{P}}_{g,k}}\right)\ln \left(1-\overline{{\mathcal{P}}_{g,k}}\right)$$

(31)

$$\overline{{\mathcal{P}}_{g,k}}=\frac{{{\displaystyle \sum }}_{i=1}^N\left({\mathcal{P}}_{i,g,k}\right)}{N}$$

(32)

Scenario 1: This scenario is used to compare the effects when ACSLA is trained by different RL methods (DQN and DDPG algorithms). To better reflect the changes in the probability map and the target distribution in the test, we use the three-dimensional histogram to show the probability map. It can be seen from Figure 6a,b that the final probability maps obtained by the ACSLA (trained by DQN and DDPG algorithms) can accurately reflect the target distribution. Figure 6c is the change graph of the environmental information entropy, it can be seen from Figure 6c that ACSLA trained by the DDPG algorithm can make the probability map converge faster. This is because DDPG adopts the Actor–Critic architecture, an actor-network is used to fit the strategy function and directly output actions. DQN is the algorithm based on the value function, which outputs the Q-value of the action instead of directly outputting the action. The agent also selects the corresponding action according to the Q-value of the action. Therefore, under the same mission environment, the convergence speed of DDPG is faster than DQN, especially when the action space is large, this is more obvious.

Scenario 2: This test is mainly to compare the effect of different numbers of AUVs on ACSLA. First, we ensure that other parameters in DMACSS remain unchanged, and set the number of AUVs in the system to 3, 4, 5, 6, and 7, respectively. The ACSLA is trained by the DDPG. It can be seen from the test results in Figure 7, when the number of AUVs increases from 3 to 4, the convergence speed of the probability map has been significantly improved. When the number of AUVs increases from 4 to 6, the convergence speed of the probability map does not change significantly. But it is worth noting that when the number of AUVs is increased from 6 to 7, the convergence speed of the probability map has a significant decrease. This is because, although the increasing number of AUVs enhances the system’s ability to search for the area, this makes the calculation of the system also significantly increase, which will affect the convergence speed of the algorithm.

Scenario 3: In this test, we mainly verify the performance of ACSLA through comparative tests. Here we select some classic search methods such as the random algorithm and coverage control algorithm as the comparison algorithm of ACSLA. In the test, other parameters remain unchanged, the number of AUVs is 4 and and the number of targets increases sequentially. The ACSLA is trained by the DDPG. Each algorithm performs 500 complete episodes, using average search accuracy and average search steps to evaluate the performance of different search methods. The test results are shown in

Table 1. Comparative tests.

	Setup	Search Time (Search Accuracy)
AUVs	Targets	ACSLA	Coverage Control	Random
4	2	131 (100%)	235 (98.3%)	410 (79.4%)
4	3	144 (99.6%)	243 (96.3%)	456 (73.3%)
4	4	161 (99.7%)	265 (94.0%)	790 (67.2%)

and Figure 8. It can be seen from Table 1 and Figure 8 that the system using the ACSLA algorithm has the best search performance regardless of the number of targets. The average convergence speed of the ACSLA in the test is 145 steps, and the average search accuracy reaches 99.77%. Compared with the other two algorithms, it has greater advantages in search accuracy and convergence speed.

6. Conclusions

In this paper, we establish the DMACSS and propose the ACSLA that is integrated into the DMACSS. Compared with the previous system, DMACSS adopts a distributed control structure to improve the system robustness, and combines an information fusion mechanism and a time stamp mechanism, so that each AUV in the system can exchange and fuse information during the mission, improving the operating efficiency of the system. ACSLA is an adaptive learning algorithm trained by the RL method with a tailored design of state information, reward function, and training framework. We test DMACSS in simulation experiments and compared ACSLA with other cooperative search methods. The test results show that DMACSS runs stably and the search accuracy and efficiency of ACSLA are higher than other search methods, meaning it can better realize the cooperation between AUVs, make DMACSS more accurate and faster to find the target. At the same time, our research still has some problems and shortcomings. DMACSS lacks the ability to search for dynamic targets, and the ACSLA requires a long training time to achieve stable performance. In addition, the fact that the system cannot perform tasks in sophisticated environments (obstacles and other interference) is also a problem we need to solve. We hope that we can gradually resolve these problems in future research.