Development of a Cascade Intelligent System for Path Planning of the Group of Marine Robotic Complexes

Abstract

Artificial Intelligence (hereinafter referred to as AI) systems have recently found great application and use in various industries, such as data processing, data analysis, and the operation control of marine robotic complexes, etc. In view of the ever-increasing degree of complexity of the missions assigned to marine robotic systems, it is becoming obvious that the AI technologies should be used as combined systems which can provide control of marine robotic complexes (hereinafter referred to as MRCs), their navigation in sea, logic formation of MRC behaviour in uncertain environments, path planning, and processing optimization of the received MRC payload data. All the areas mentioned above are within the field of MRC development, and currently do not have a general solution. This article discusses the development process of an intelligent system for path planning of a group of marine robotic complexes. The architecture of the intelligent system is based on a cascade approach, which includes the consistent use of functional modules designed as various “organs of perception” of the system. A detailed description of the development of each module and mathematical modelling of the presented algorithms are provided in this paper, and the main results of the conducted full-scale experiments are demonstrated.

1. Introduction

Currently, intelligent systems (hereinafter referred to as ISs) have a very wide range of applications thanks to AI development that extend human skills and capabilities.

The term “Artificial intelligence” is widely used at present to refer to applications for complex missions that previously could only be performed by humans. AI makes it possible to reproduce and improve on what we learn from the world around us and what can be done with such information. This feature of AI underlies innovation. AI is based on various machine learning technologies, which recognize data patterns and generate predictions. In view of this particular feature, AI is an intermediate link that often lies at the junction of several technologies, which allows the potential of various systems to be increased and the best results to be achieved more effectively.

Today, AI technologies, machine learning (hereinafter referred to as ML), and deep learning (hereinafter referred to as DL) technologies are used in the field of marine robotics. Developers increasingly use these methods to implement planning, control, and internal diagnostic systems in the field of airborne [1,2], ground [3,4], and marine robotic complexes [5,6,7], including underwater gliders [8,9,10] and surface gliders [11], as well as for the tasks of monitoring and patrolling potentially dangerous underwater objects. Intelligent systems provide the implementation of adaptation to uncertain environments [12,13] and ability to evaluate and simulate the current state [14,15,16], and can perform goal-setting functions and action planning for air, ground, and sea robotics [17,18,19,20], including vehicle basing (issues related to AUV docking) [21].

Currently, a transition is underway from group control to practical implementation of group applications for homogeneous groups of autonomous vehicles (hereinafter referred to as AVs) [22,23,24] and heterogeneous groups of AVs [25,26]. Developers use existing groundwork from video games to train neural networks in order to set up and train multi-agent teams [27,28,29,30,31,32,33]. Progressive development of models and technologies for group application of marine robotic complexes (and MRCs as well) poses new challenges for developers related to the planning and operation of MRC groups in uncertain marine environments [34,35]. Marine robotics usually involves performing tasks without preliminary detailed mapping in large open water areas with different depths, bottom topography, and constantly changing periodic and aperiodic effects. In addition, the underwater environment imposes significant restrictions on the possibility of using the sensor, communication, and navigation systems of AVs, leading to additional requirements to consider the power and structural features of the controlled object as well as the conditions of the area for the planned job. To take into account such functional features, developers often use methods based on intelligent neural networks and ML, which makes it possible to increase the autonomy of the MRC. In previous papers [36,37,38,39], authors have demonstrated the use of neural networks to solve the problems of positioning and navigation of MRC as well as to calibrate the automatic control system [40,41,42,43,44,45,46].

Analysis has shown that while AI systems are widely used to solve data analysis and processing problems [47,48,49,50,51,52,53,54,55,56,57], they are generally not used in motion-planning systems for groups of AVs. To date, there have been papers presenting the use of AI for motion planning of a single marine robotic complex [58,59] and as an auxiliary decision-making system when objects move in sophisticated environments with obstacles [60,61,62,63,64]. However, these papers do not consider features of the interaction of AVs with each other or features of group operation when moving in a formation, when it is necessary to solve the tasks of rebuilding or reconfiguring the formation. It should be noted that developers are currently considering centralized principles in this direction to build up a group management system. Such systems have a distinctive feature in the relatively low “survivability” of the system in case of unforeseen situations (e.g., failure of the leader). Moreover, assumptions are made upon MRC operation taking the preliminary cartography into account.

In this paper, we have taken up the task of developing an intelligent system for governing group interaction of MRCs. The proposed system should analyze data available to the group, make a recommendation as to the next action taken by each agent, and form an intragroup interaction policy. This allows the system to conduct subsequent self-learning in order to find adaptive solutions and ways to rebuild and reconfigure the formation in the most optimal way. A key distinguishing feature of intelligent MRC operation with an adaptive control and planning system is that the sequence of actions necessary to achieve the goal is not determined in advance, as information on future states of the environment is generally not available. The advantage of the developed “cascade” planning method over other group motion planning systems is the use of DL methods for each agent to ensure that the decentralized control concept is complied with. The developed “cascade” method of group motion planning has a modular architecture for easy system scaling. The presented method combines various algorithms for different tasks, for instance, to form a group interaction area, process group sensory data, and combine reinforcement learning methods for intra-group interaction of MRCs.

Today, reinforcement learning methods are widely used in motion planning systems for robots and groups of robots [65,66,67,68]. However, the results of related studies and papers have mostly described the use of Artificial Neural Networks in certain deterministic conditions with known hard-coded values for the area of operation. These sources present the use of centralized AI systems for the optimal distribution of a group of robots over an area and the organization of group interaction to solve a global task in front of a group, for example, using the smart factory pattern implementation [69]. In particular, many methods are based on the analysis of information obtained from preliminary cartography, which is not available in real conditions in general. Our analysis revealed that currently available methods and algorithms cannot explicitly solve the problems of monitoring, patrolling, and surveillance of the required territories and objects, which include, for example, potentially dangerous underwater objects. Thus, it is necessary to consider reconfiguration and adaptive changes to address the presence of such objects within the structure of the formation while avoiding both static and moving obstacles. A new algorithm based on graph-analytical and neural network methods is presented in our study. The combination of reinforcement learning and graph-based algorithms is advantageous, as it allows robotic complexes and group(s) to quickly identify possible research areas and use their knowledge to maximize rewards. For example, graph-based algorithms can calculate the global situation and trajectory as well as the future states of each individual robot or group (which may lead to high rewards) and the best ways to change the state. Reinforcement learning can use this information to realize improve performance.

In this article, Section 2 considers the formal statement of the problem and the problems associated with the use of the MRC groups in uncertain environments with obstacles. This section provides a workflow for developing the intragroup MRC interaction algorithms. Next, Section 3 proposes the development workflow of a neural network cascade system for path planning of an MRC group, including the development workflow for its architecture. Our full-scale experiment is described in detail in Section 4. Finally, in the conclusion we summarize the key results of this research and propose future work for the development of MRC group path-planning intelligent systems.

2. Problem Statement

The primary goal of the present study is to develop a neural network system for path planning of the MRC group that allows agents to move in an uncertain environment with fixed and moving obstacles while taking into account the restrictions imposed on maintaining the structure of the MRC group formation.

The following tasks must be resolved in order to implement the cascade system of intelligent path planning of the MRC group:

• Development of algorithms for the local path planner of each agent in the group and the global approach of the group motion;
• Implementation of the logical approach for the intragroup policy operation for the interaction of agents in a group, which is based on the communication range and the vision system parameters, which in turn are functions of the current and predicted parameters that describe environmental conditions, including data on the location of agents and their current state;
• Development of an architecture for the intelligent interaction of agents based on the existing intragroup policy.

The general structure of the proposed cascade approach for the intelligent interaction planning of the MRC group in an uncertain environment with obstacles is presented in the Figure 1.

The figure shows the following main stages of the intelligent planning system operation:

• Formation of an area of group interaction ${ð}_{group}$. This module implements the construction of the area space, in which all agents of the group must be located.
• Global planning algorithm development based on the RRT* method. This algorithm provides real-time trajectory formation of the group $H_{group}$ without preliminary area mapping.
• Vision system (hereinafter referred to as VS) of each agent in the group, which forms a dataset ${\mathsf{\Omega }}_n$ of the existing obstacles in the AV’s n field of view. This algorithm integrates the data received from the VS of each AV into a single field of view ${\mathsf{\Omega }}_{group}(R_1,R_2,R_3)$ (in which case $n=3$), which provides the subsequent analysis and a recommendation at the next stage of the planning algorithm operation.
• Formation of a policy for the internal interaction of agents in a group $P_{agents}$. This module is required for the analysis of all events occurring in the external space around the agent $E_a$ and its reaction to these events $A_n$ based on conditions that determine the current state of other agents in the group $S_{agents}(S_{a1},S_{a2})$, taking the group task into account.
• Development of a neural network path planner for each agent in the group, which manages the local trajectory of the agent’s motion and is based on the data received from the built-in VS systems. In addition, this manages the agent’s position in the group interaction area (hereinafter referred to as GIA) and the group interaction policy parameter. This method is based on the DQN approach [61,70] with a modernized error function.
• Development of a group neural network planner, which is aimed at ensuring the synergy of data received from each agent’s individual neural network. This module calculates the group reward policy $R_{group}$, and affects the subsequent actions of each agent in the group.

2.1. Development of the GIA Formation Algorithm

Use of MRC groups in real sea conditions is always associated with the use of high-precision technologies to ensure proper positioning and the necessary parameters for radio and hydroacoustic communication. However, it is not always possible for developers to take into consideration the various indicators of marine environment stratification, the presence of multidirectional currents, and other non-stationary processes, especially when the MRC is used in 3D marine environments [71,72,73]. An aspiration to ensure high accuracy in the field of MRC groups’ application leads to the high complexity of algorithm development in this context, and, on the other hand, to the high complexity of implementation and application in real-world conditions [27,34,74]. Therefore, we have decided to deviate from the generally accepted principles of constructing a “hard-coded system” of the MRC formation and implement a method for constructing an MRC group based on the GIA. This method can be described by the following set of parameters (under the conditions of applying the algorithm in a 2D formulation): $dis{t}_r$—given distance between agents; $r_{con}$—range of the communication system (e.g., radio communication systems during the MRC surface group operation). To calculate the boundaries of the GIA ${ð}_{group}$, the following expression is used (1):

$$\begin{array}{c}{ð}_1={\mathsf{\Xi }}_1\cap {\mathsf{\Xi }}_2\\ {ð}_2={ð}_1\cap {\mathsf{\Xi }}_3\\ {ð}_{group}={ð}_1\cap {ð}_2\end{array}$$

(1)

where ${\mathsf{\Xi }}_1(r_{co{n}_1},dis{t}_r), {\mathsf{\Xi }}_2(r_{co{n}_2},dis{t}_r), {\mathsf{\Xi }}_3(r_{co{n}_3},dis{t}_r)$ are the bounding areas of operation for each agent of the group depending on the distance between agents and the allowable communication range. An example of the formation of the GIA is shown in Figure 2.

Output values after the algorithm operation are the variables $[R,E_{dist},E_{con}]\in {ð}_{group}$ describing the states between all agents in group (2) and the group’s estimated center $\nabla \in {ð}_{group}$:

$$\begin{array}{c}{E}_{dist}=\left[\begin{array}{c}|R_1-R_2|-dis{t}_r\\ |R_1-R_3|-dis{t}_r\\ |R_2-R_3|-dis{t}_r\end{array}\right]E_{con}=\left[\begin{array}{c}{r}_{con}-|R_1-R_2|\\ r_{con}-|R_1-R_3|\\ r_{con}-|R_2-R_3|\end{array}\right]\end{array}$$

(2)

This set of parameters is used for subsequent training of the group neural network, which is responsible for analysis and recommending further actions, either as a single agent in the group or the entire group.

2.2. Development of an Algorithm for the MRC Group’s Global Trajectory Planning

The designed system for the global planning trajectory formation is based on the real-time RRT* method. The primary task of the MRC group is to move from the starting point to the end point. To do this, we propose building up a map of random tree branches to cover the entire considered area. This process does not take very much time, and can be performed during the pre-launch phase.

The following input parameters are used: R is the functioning area’s size; $S_p$ is a starting point of the group; $F_p$ is an end point of the group; $\epsilon$ is a simulation step; $r_r$ is a coefficient describing the radius within which the search for neighbouring nodes of the tree is performed. All calculated points are saved to the array $T\left(i\right)=(t_1,t_2)$ of i × 2 dimension, where i is the number of algorithm iterations.

During each iteration, we take a random point from $Ran{d}_p=N_{new1},N_{new2},\dots$, which is a random sequence. The distance from $N_{new}$ to all other points in the tree is calculated and an array $D=(Dist,id)$ is formed, where $Dist$ is the distance and $id$ is the number of points in the tree of $i\ast 2$ dimension with i being the number of algorithm iterations (3).

$$Dist=\sqrt[2]{{(n_1-t_1\left(i\right))}^2+{(n_2-t_2\left(i\right))}^2}$$

(3)

The closest point to $N_{new}$ is selected, i.e., $N_{near}=min\left(D\right)$. Thus, $N_{new}$ moves closer to the selected point, meaning that $N_{new}$ is epsilon away from the selected point (4).

$$\begin{array}{c}{N}_{new\left(1\right)}=t_1+(-1)\ast e^{(}{t}_1-n_1)/Dist\\ N_{new\left(2\right)}=t_2+(-1)\ast e^{(}{t}_2-n_2)/Dist\end{array}$$

(4)

Next, points located within the circle of radius r and its the center in $N_{new}$ are selected as $Neighbours=D<r$. The edge distance from F to neighbours is the $Cost$. The total cost is a sum of $Cost$ and $Dist$ (5).

$$TotalCost=Cost+Dist$$

(5)

The point with the lowest $TotalCost$ is selected as $ParentNode=min\left(TotalCost\right)$. After drawing an edge from $ParentNode$ to $N_{new}$ to produce a rewiring node and lower the cost of the neighbours’ nodes, we say that $N_{new}$ is now the Parent node for its neighbours. Costs are compared, which may lead to tree rebuild in the case of the new cost being lower (6).

$$TotalCost\left(Neighbours\right)>TotalCost\left(N_{new}\right)+Dist.$$

(6)

The process scheme used for node rewiring (finding the nearest and selecting the “best” parent) is shown in Figure 3. R is a rewiring range; blue points are tree nodes, red points are reward added nodes; green point $N_{new}$ is a new added point after rewiring.

An example of the algorithm ’s operation is one robot moving, as presented in the Figure 4. The AV starts moving from the start point to the end point by following a parent node trajectory.

Figure 4 demonstrates an agent’s motion in an unknown environment with a static obstacle; the blue dots are the nodes of the RRT* tree branches; the black line is the actual path traveled by the agent; the green line is the current calculated path along which the agent is moving; the red dotted zone is the area of obstacle detection by the VS (circular LIDAR of MRC’s VS ${\mathsf{\Omega }}_n$); and the red areas are the detected obstacle boundaries.

The system recalculates the current trajectory in real time according to the information received from the VS module. The center point is the center of the GIA $\nabla \in {ð}_{group}$ moving along a calculated target trajectory $H_{group}$. A branch rebuilding function works only within a certain radius of the agent, which in the considered simulation equals the radius of the VS visibility zone. The blue nodes of the branches remain in the memory of the AV in case the device encounters a complex obstacle structure and needs to search for other ways around.

Additionally, the motion of the tree root has been implemented along with the center of the group, ensuring that the path remains relevant for each recalculation. In order to find the most profitable path in our research, path search was implemented using a breadth-first search. The cost of hitting the exact node and distance from that node to the end point was calculated for each node. Thus, these additions significantly improve the underlying RT-RRT* algorithm.

Using internal libraries of the ROS framework, the points received from the VS are recalculated taking into account the heading angle and the global coordinate system. When using the RRT algorithm, the distance of the graph points to the obstacle points is checked, ensuring that inaccessible graph points are blocked.

An example of the algorithm’s operation is group of robots moving, as presented in Figure 5.

In the Figure 5, red lines are the obstacle points defined by the VS. We used 600 iterations (nodes) to build the tree, which is represented by the yellow lines. The blue branches represent possible trajectories obtained as a result of the rewiring process. At output, the algorithm produces 600 nodes, links between them, and arrays containing information about neighbor, parent, and child nodes and the cost of each node and the point of the path found. The path is represented by the green line.

Input data for the described RRT algorithm consist of the map dimension ($n\ast m$), number of iterations (samples), and a constantly updated array of points obtained from the depth camera.

2.3. Development of an Algorithm for the Computer VS of an MRC Agent and Group Field of View (GFOV)

To simulate VS operation, we used an Astra Pro Realsence camera model (available in ROS). During operation, the depth camera produces an array of points with a resolution of 640 × 320, which gives 204,800 points. These points are converted to the PointCloud2 type and transferred to the RRT module. To increase the processing speed, it is possible to reduce the resolution of the depth camera. An example of the process by which images are obtained from a camera is presented in Figure 6.

The YOLOv5 pre-trained neural network is responsible for distribution into classes [75], and is able to successfully resolve this task. This neural network is trained on user data by loading an additional dataset with the selected “obstacles” and “agent” classes.

Based on the data by the sensory system, the agent is able to obtain information about the external environment. The VS is based on two main functional modules, namely, an obstacle detection module and an object identification module. The Obstacle detection module allows to generate a cloud of points in the visibility zone of the VS $P_v=P_v{}_1\dots P_v{}_n$ (where n is the number of rays emitted by the VS if LIDAR is used), then sends the appropriate command to the actuators to recalculate the local trajectory of the AV $H_r{}_k$, where k is the number of AVs in the group. The Object identification module provides identification of an object located in the agent’s visibility zone, and is an auxiliary module that provides additional data to the neural network of the AV. To simplify the task, only two classes are formed, namely, “obstacles” and “agent”.

Because the presented work describes the planning of a system design process for a 2D setting, available TurtleBot3 models with their own set of actions were used in order to simplify the development of a software simulation complex.

Consequently, a software simulation complex has been developed based on the ROS framework and its Gazebo and RVIZ software components [76]. The collected array of custom data is based on a group motion simulation of TurtleBot3 AVs [77,78] and a camera installed on their platform, as seen in the Figure 7. In addition, the database consisted of 270 images, on which the corresponding classes of “obstacles” and “neighbouring agent” were been marked and the necessary labels created. Training was carried out using the following set of parameters: $epoch=500$, $batchsize=16$.

The result of algorithm operation is shown in the Figure 6.

The figures above demonstrate how the system returns the class $\mathcal{D}$ of a certain object; 0 is the obstacle, while 1 is the agent in the group. The operation of this module as a part of GFOV allows for aggregation of an array of data on certain classes in the field of view of an object $\mathcal{D}$, which outputs detection on each frame $d_i,\dots ,d_N$. Each $d_i$ is a set containing tuples $b=(x,y,w,h,c)$. It denotes the detected bounding box and its confidence $0\le c\le 1$ that the box corresponds to an instance of an object of class $\mathcal{L}$. Here, we use $b\in \mathcal{L}$ to denote the case of a box being a member of a given class. The intersection condition $\mathcal{D}\left(\mathcal{L}\right)\in P_v$ allows an agent in the group to determine both the class of an object ahead and the distance to it.

The second step is to obtain data on the shared field of view of the group and determine the percentage of obstruction with respect to the VS data for each agent of the group. A detailed description of this algorithm is considered in [79].

It should be noted that in this case a simplification is introduced, in that LIDARs were used for the TurtleBot3 3D model with a viewing angle of 45 degrees.

Hereinafter, GPOV data ${\xi }_n$ and data from the obstacle class identification module $\mathcal{D}$ are transmitted to the input of the intelligent neural network of each agent in the group.

3. Development of the Cascade Architecture of the Neural Network Planning Module for Agents of the MRC Group

The proposed method for forming an interacting group of robots permits data exchange between agents in the group based on their respective vision systems. The global planning module based on the information of the GFOV module determines a global motion trajectory. The GIA generation module allows the best way of rebuilding the group to be chosen while avoiding obstacles and taking into account the distance necessary to maintain communication between the agents. The architecture of the developed intelligent cascade planning system, considering both the input and output parameters, is presented in Figure 8. To generate control signals for the actuators of each agent of the group, the internal planning logic was implemented based on an artificial neural network and a DL method for the agents.

Operation of the algorithm on the basis of the cascade intelligent planner, shown in Figure 8, is as follows. First, the available GPOV, GIA, and RRT* modules receive input data from the navigation systems and VS of each agent in the group. As presented above, the result of the GPOV operation is a data array describing the current state of the shared field of view, including the classes of objects. The GIA algorithm provides data on the current state of the formation in the form of ${\epsilon }_n$ and $\mathcal{D}$. The auxiliary trajectory planning module calculates the recommended trajectory in the form of $H_{group}$. This dataset is sent to the neural system for path planning of each agent of the group, which is responsible for the action of each agent in the group during learning. Thus, each agent has its own environment (the agent’s local environment ${\mathbb{E}}_{agent}$), from which it receives feedback in the form of an updated state $S^{\prime }$ and reward R. At the same time, the agent is in constant interaction with other agents in the group, meaning that its local environment is part of the overall group environment ${\mathbb{E}}_{agent}\in {\mathbb{E}}_{group}$.

In order to make adjustments with group-wide impact, the last module of the cascade architecture to be introduced is the Overall group environment analysis module. This module determines the intragroup state of each agent and the group-wide reward for agents for selected actions A.

The resulting decentralized group planning system is easily implemented and scalable. The system implements a cascade of algorithms, which together prepare and generate data for the agents’ onboard neural network training.

3.1. Development of a Neural Network Agent Action Planner Module in the MRC Group

The data used as input for the agents’ neural network result from either the operation itself or calculations carried out by the modules described above. A planner module is implemented on a fully connected neural network basis. The DQN approach is used as the main training method [61,70]. The agent has its own memory of (M) = 20,000 elements, storing a set of values $(S,R,S^{\prime },A,Q)$ that describe the previous state, reward, current state, and recommended maximum best value of the selected action. The agent model is presented in Figure 9. The rule that forms the basis for training the agent’s neural network planner is shown in Formula (7).

$$r_t+\gamma ·\underset{a!}{max}Q·\left(S_{t+1},a^{\prime };{\theta }^{-}\right)-Q·{\left(S_t,a_t;\theta \right)}^2$$

(7)

The target network and Q-network (shown in the figure below) have an identical structure.

The architecture of the developed neural network is shown in Figure 10. The neural network has 364 input values, which include arrays of distances between the agents of the group, the distance and direction angle of the current agent in relation to the target, data from the VS, the identifier of the detected object (an obstacle or an agent of the group), and the agent’s current status. This information enters the internal block of the neural network, which consists of three dense layers of 64 neurons each and two dropout layers with an exclusion parameter equal to 0.2. The RMSProp optimizer with parameters $learningrate=0.00025$, $\rho =0.9$, and $\epsilon =1^{-6}$ was chosen as the activation function.

The agent’s neural network module provides the error calculation function based on the Huber loss (8):

$$\mathrm{Huber}\mathrm{loss}=\left\{\begin{array}{cc}\frac{1}{2}{\left(y-y_p\right)}^2,& \left|y-y_p\right|\le \delta \\ \delta \left|y-y_p\right|-\frac{1}{2}{\delta }^2,& \left|y-y_p\right|>\delta \end{array},\right.$$

(8)

where y is the current value of the AV states, $y_p$ is the predicted values of the AV states, and $\delta$ is the error function tuning factor, which in this case is $\delta =1$. A previously developed software simulation complex was used as an environment to conduct training. The interaction of the agent with the environment occurs on the basis of five possible actions, which are presented in Figure 11.

When interacting with the environment, an AV changes its state from (S) to ($S^{\prime }$) based on a reward $R_r$. The expression used to describe the internal policy is responsible for the AV’s reward function is as follows (9):

$$\begin{array}{c}{R}_r=Y_r\left(A\right)\ast 5\ast D_{\mathrm{rate}}\\ D_{\mathrm{rate}}=2^{D_{\mathrm{cur}}/D_{\mathrm{goal}}}\\ Y_r\left(A\right)=1-4|0.5-0.5\ast \alpha |\\ \alpha =\mathrm{heading}\end{array}$$

(9)

where $heading$ is the agent’s current heading, $D_{\mathrm{rate}},Y_r\left(A\right)$ is the component of the reward based on the distance and the agent’s current heading, and $D_{\mathrm{goal}}$ is the agent’s distance to the target.

3.2. Development of a Neural Network Correction Module for MRC Group Action Planning

The neural network correction module for planning the actions of the MRC group is responsible for evaluating the actions performed by each agent in the group. The primary objective of this module is to form an intragroup policy of behaviour through the analysis of states and rewards $(S_n,R_{r}n)$ for all agents in the local ${\mathbb{E}}_{agent}$ and overall group ${\mathbb{E}}_{group}$ environments. The function that describes the intragroup interaction policy of agents, which is responsible for the group reward component formation, is provided by expression (12) below:

$$R_{g13}=\left\{\begin{array}{cc}-e^{D_{13}-2.1}& \mathrm{if}{D}_{13}<2.1\\ -e^{1.9-D_{13}}& \mathrm{if}{D}_{13}<1.9\\ 3& \mathrm{other}\mathrm{cases}\end{array}\right.$$

(10)

$$R_{g12}=\left\{\begin{array}{cc}-e^{D_{12}-(\sqrt{2}+0.1)}& \mathrm{if}{D}_{12}>\sqrt{2}+0.1\\ -e^{(\sqrt{2}-0.1)-D_{12}}& \mathrm{if}{D}_{12}<\sqrt{2}-0.1\\ 3& \mathrm{other}\mathrm{cases}\end{array}\right.$$

(11)

$$R_{g1}=R_r+R_{g12}+R_{g13}$$

(12)

where several scenes in the software simulation complex were implemented in order to train a group of three agents: a scene without obstacles, one with stationary obstacles, and one with dynamically changing obstacles. The scenes are shown in Figure 12. It should be noted that at this stage an initial structure consisting of a wedge-type formation was considered.

Training was conducted on a PC equipped with an NVIDIA Geforce 3060 video card with 12 Gb RAM (considering the pre-installed CUDA 11.7), 64 Gb DDR4 RAM, and Intel Core i9 4016 Gz processor. The elapsed training time varied for each scene: 2 h for the first scene, 10 h for the second scene, and about 15 h for the third scene.

The graphs presented below in Figure 13 describe the learning results for the three generated scenes. An example of training a group on the scene without obstacles with the GIA display and its changes during motion is shown in Figure 14.

In the Figure 13, the plots (a) and (c) demonstrate the results of group training in the environment without obstacles. As can be seen in the figures, the developed group neural network successfully achieves (with 90% probability) accident-free group motion after 400 training epochs (episodes), which we considered to fulfill the conditions specified by the GIA and shared field of view algorithms for reaching the target coordinates. The plots (b) and (c) demonstrate the results of group training in the environment with dynamic obstacles. After 800 epochs (episodes) of training, the neural network successfully copes (with 75% probability) with the task of group motion to reach the target coordinates while avoiding obstacles.

4. Full-Scale Trials of the Developed Modules’ Operation for Planning and Intragroup Interaction Using the Example of a Displacement MRC

In this section, we present full-scale tests of modules based on the proposed method. In future works, we plan to use the results of full-scale trials for hardware integration of a neural network planning system with the mathematical models of robotic systems. The set of actions of TurtleBot3 is comparable to the set of actions of mini-boats, as in both cases there are two engines.

To test the developed algorithms, experimental models of crewless boats were implemented. A schematic layout of the considered crewless boat’s equipment is shown in Figure 15.

• Module: Transceiver, RF, FSK, GFSK, LORA, OOK, 868 MHz, TTL, UART/HOPERF
• Antenna ant gps GPS900-1 SMA-m 3M ZOGLAB
• Lithium 3v battery
• Two Collector engine R540- 33110 12 V (540 class)
• UNO R3 ATMEGA328A-AU CH340G board with USB cable
• Imu sensor with 10 degrees of freedom (troyka module)
• Raspberry PI 4 model b+ 2 Gb microcomputer
• D-Link DWA-137/C1A USB 2.0 Wi-Fi Network Adapter
• Astra Pro Realsense RGBD Depth Camera

In our systemRGBD depth cameras based on Structured light technology are used as the main devices for obtaining information on obstacles, as it is shown in the Figure 16. The option of using Depth from stereo technology is under consideration for future stages. There are issues with structured light cameras, among which a few should be mentioned in this context. The main issue is that they produce a noisy depth map in open air, as solar radiation contains an infrared spectrum. Another issue is that they provide a large number of reflections when used in water.

Operation trials of the group management system were carried out; it was necessary to test the RRT algorithm as a global planning system, the system for depth map generation, and the speed of group interaction.

Due to the unaccounted-for influence of sunlight on the data obtained from the depth camera, we had to abandon the system test with the camera; these tests will need to be carried out in future research.

A formalized statement of the mission assigned to a group of robots can be represented as follows: it is necessary to ensure the movement of a group of marine robotic systems through the key points of the route, taking into account the preservation of the group formation (in the full-scale tests, the movement of the group was in a “front”-type formation) while taking into account obstacle avoidance and maintaining the necessary distance to maintain a communication channel between the robots. In thie experiment, the target distance between the robots was 2 m and the tolerance corridor between robots was 0.5 m.

The full-scale tests included the following steps:

• Study of the local and global planning performance under real conditions and in two cases (without obstacles and with obstacles).
• Operability validation of the technical vision system and the information exchange module (volumes of transmitted data) in order to build up a common field of view of the group.
• Formation of a database from the VS system, which is subsequently loaded into the neural network block in order to classify group agents.
• Integration of data obtained from the vision system into the neural network planning system implemented on the hardware of crewless boats.

The goal of the experiment was to move the group from point A to point B. To do this, ten observed measurements were used. The direct distance from point A to point B was 25 m. The RRT tree building time for each experiment is presented in

Table 1. Time and path length required to build the RRT tree.

Sample	1	2	3	4	5	6	7	8	9	10
$\begin{array}{c}\mathrm{Tree}\mathrm{building}\mathrm{time},\sec \end{array}$	0.433	0.451	0.442	0.432	0.442	0.412	0.421	0.412	0.423	0.411
$\begin{array}{c}\mathrm{Path}\mathrm{length}\mathrm{for}1\mathrm{AV},\mathrm{m}\end{array}$	26.82	27.112	28.023	25.93	29.12	26.84	26.93	30.03	26.34	28.53
$\begin{array}{c}\mathrm{Path}\mathrm{length}\mathrm{for}2\mathrm{AV},\mathrm{m}\end{array}$	27.23	26.91	28.93	27.26	27.42	25.81	25.96	27.34	27.45	27.53
$\begin{array}{c}\mathrm{Path}\mathrm{length}\mathrm{for}3\mathrm{AV},\mathrm{m}\end{array}$	26.12	29.14	27.34	26.32	26.75	28.21	28.96	26.78	28.44	29.53

.

The data transmitted from each crewless boat were as follows:

• Service information, navigation information, and 3D camera data = 1.2 mb;
• Service information, navigation information, and LIDAR data = 1.5 kb.

The software of the crewless boat was implemented on the ROS in order to simplify the integration and porting of existing automatic control system (hereinafter referred to as ACS) and planning modules. The full-scale experiment was carried out in the inner water areas of St. Petersburg, Russia and Chelyabinsk, Russia. The result of group operation of the crewless boats is shown in Figure 17.

The conducted experiment made it possible to test the application of the developed algorithms in a real environment, considering the use of the implemented crewless boat hardware and software platform. An operator GUI was implemented to display the necessary information about the crewless boat motion, target coordinates, and actual trajectories of the agents in the group. In addition, the GUI was able to show the operation of the RRT* algorithm in real time.

To ensure VS operation in the full-scale trials, we decided to replace the Astra Pro Realsense 3D camera with the RPLIDAR S2 30 m TOF LIDAR, which has a visibility range of 30 m.

It should be noted that the visibility of the LIDAR was limited in the experiment by its visibility sector of +/− 30 degrees from the device bow. Wave motion was compensated for by matrix transformations using the 3D compass data.

The tests consisted of system operation series with a typed obstacle. The key task of the group, as in the simulation, was to bypass a static obstacle. The distance from the starting point to the end point was approximately 30 m, which can be considered a small distance taking into account the possibility of LIDAR action and the speed of the boats. We conducted fifteen launches, three launches of which were unsuccessful. Failure of the LIDAR and of the point cloud merging system were detected during the first and second unsuccessful launches, respectively. During the third unsuccessful launch, there was a problem with the GPS, causing the robots to crash into an obstacle. The RRT tree building time for each experiment is presented in

Table 2. RRT tree building time and path length.

Sample	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
$\begin{array}{c}\mathrm{Tree}\mathrm{building}\\ \mathrm{time},\sec \end{array}$	0.433	0.451	0.442	0.432	0.442	0.412	0.421	0.412	0.423	0.411	0.413	0.411	0.441	0.421	0.403
$\begin{array}{c}\mathrm{Path}\mathrm{length}\mathrm{for}1\\ \mathrm{robot},\mathrm{m}\end{array}$	38.79	46.46	46.75	39.92	39.1	39.84	42.88	47.5	40.13	45.42	42.77	43.8	40.9	45.57	45.83
$\begin{array}{c}\mathrm{Path}\mathrm{length}\mathrm{for}2\\ \mathrm{robot},\mathrm{m}\end{array}$	44.38	41.67	40	37.78	39.02	44.26	37.58	44.35	41.87	40.09	43.05	44.78	47.72	41.56	37.49
$\begin{array}{c}\mathrm{Path}\mathrm{length}\mathrm{for}3\\ \mathrm{robot},\mathrm{m}\end{array}$	47.38	46.4	38.74	41.57	45.94	40.35	41.06	42.93	46.13	41.84	44.03	47.48	37.78	37.22	41.45

.

The full-scale trials with an obstacle is shown in Figure 18.

In Figure 18, Figure 18a,b demonstrate group motion and avoidance of the detected obstacle, while Figure 18c,d demonstrate the operator’s GUI, showing the motion trajectories of a group of robots (the operation of the global planning algorithm is displayed). Figure 19 demonstrates the full-scale trials results in RVIZ in real time; the red, yellow, and green circles are the GPS coordinates of the agents, while the red solid line is the obstacle detected by the RPLIDAR S2.

Our experiment demonstrates the possibility of using the developed complex for further study of path construction algorithms, as well as for further study of neural network technologies for group interaction of MRCs. The database of images accumulated during the experiment, which includes models of the crewless boat and planned obstacles, makes it possible to train the developed neural network (a part of the GPOV module). We intend to carry out training of the neural network on real objects and use these results in our subsequent studies on full-scale models of the proposed complex.

5. Conclusions

The present article provides a detailed example of a cascade intelligent system implementation for path planning of a marine AV group. The primary design stages and basic modules of the developed architecture are described.

The proposed system is based on modules that combine neural network approaches and machine learning methods, modules that form a group’s single field of visibility representation, an area of intragroup interaction, and an auxiliary global RRT* planner module.

The developed combination of reinforcement learning algorithms and graphs allows more efficient exploration of the space of possible solutions. The agent can explore both deterministic and probabilistic policies of interaction with the environment.

The available dataset enables deep learning by the neural network of each agent in the group. In addition, constant data exchange with respect to the states of neighbours in the group and subsequent analysis allow a scalable and decentralized system structure to be implemented for intragroup interaction. An additional policy for intragroup state evaluation involves a corrective impact on the training of the cascaded neural network action planner.

A software simulation complex based on the ROS, RVIZ, Gazebo, and ML libraries was developed, making it possible to conduct research on the developed models and planning methods. The obtained simulation results demonstrate the performance of the proposed neural network planning algorithms.

All source code was written in the Python programming language, chosen for this research due to its ease of use and because of the large number of available auxiliary libraries. On the other hand, Python is an interpreted language that is considered slow. When a fully working prototype of this system is ready (using the full-scale models), we plan to migrate to C++ in order to increase the speed of the entire system.

The presented methods and algorithms for group interaction, along with the practical cases, demonstrate the possibility of using an intelligent system as part of the MRC in 2D environments. The existing architecture and the modular principle of building up the planning system make it possible to carry out additional adaptation in order to use this system as part of the AUV, considering the existing limitations and features of AUV operation in a 3D environment.

The simulation model demonstrated a 90% probability of successful group motion mission completion when applying the algorithms in an environment without obstacles and a 75% probability in an environment with moving obstacles. An important issue is to improve the quality of the neural network in environments containing obstacles. One possibility is to consider the use of convolutional layers in the architecture of the neural network of the agent.

Furthermore, we carried out software and hardware implementation to lay further groundwork. An alpha version of a GUI for monitoring and managing a group of MRCs based on the ROS system was implemented. Models of crewless boats were selected as the objects of control, on which the necessary software and hardware, communication, and navigation modules were installed. The conducted full-scale experiment demonstrates the efficiency of the developed algorithms in real-world application conditions. It should be noted that the propulsion parameters of the crewless boats must be improved to ensure greater maneuverability. To permit the subsequent training of the VS module’s neural network and for the developed neural network’s full operation during full-scale experiments, a database was generated using the data collected from the VS.

Further work is aimed at developing the architecture of the cascade intelligent path planner system, specifically, by introducing data on branches from the RRT* auxiliary planning module into the neural network group module and forming an “advantage” system for the agent’s neural network. The modified architecture is shown in Figure 20.

The “advantage” system is to be based on a policy responsible for analyzing each training epoch for a group of agents, the actions they have taken, and the states they have received, as well as the rewards. In addition, analysis of the constructed and used nodes of the RRT* tree should be carried out by this module. We expect the introduction of such a system to lead to an increase in the quality of group neural network training [80].