Design and Implementation of Simulation System for Multi-UAV Mission

Abstract

Multi-UAV (unmanned aerial vehicle) mission collaboration is one of the current research hotspots in automation, artificial intelligence, and other fields. The difficulty and high cost of real flight verification have led to the problem that related knowledge learning focuses on theoretical derivation and ignores technical practice. In this paper, a multi-UAV mission simulation system is designed to show the greatest advantage of its collaborative mission planning results simulated and tested in 3D scenarios. It simultaneously reflects and records changes in UAV position, velocity, status, and other state values. The simulation program can be directly applied to real flight with only a few settings. First, the general system framework is presented. Second, according to the simulation requirements, the key modules involved in the software in the loop are given. Finally, through a path planning test for two UAVs, the effect of the system is demonstrated. The results show that the simulation verification based on the system is consistent with the real flight results in terms of functional implementation. It is applicable to the teaching process of related majors and can also provide support for the realization of many complex tasks in the future.

1. Introduction

UAVs have the advantages of flexible deployment, ease of maintenance and operation, and simple take-off and landing requirements. They can be used in civilian application areas such as urban logistics [1], aerial photography [2], line inspection [3,4], etc.; in wartime, they can also be used for ground reconnaissance [5,6,7,8,9], strike missions [10,11,12,13,14,15,16,17], dynamic target tracking, and other military tasks. However, a single UAV has many shortcomings in performing complex missions limited by its load and power constraints. If multiple UAVs with different functions are coordinated according to a relevant logic to give them a certain degree of autonomy and intelligence, the effect of mission accomplishment can be greatly improved. This is also one of the current hot research problems in the field of automation and artificial intelligence [18,19,20,21,22,23].

In the multi-UAV research field, collaborative mission verification and testing plays an important role. Common verification methods include traditional numerical simulations [24,25], hardware-in-the-loop simulations [26,27], and real flight tests [28]. The numerical simulation is performed using MATLAB and other software, and the model is solved by constructing mathematical models and writing relevant algorithms on a computer basis. Finally, the calculated data and change curves of the simulation object are obtained. Hardware-in-the-loop simulation refers to the real-time simulation in which part of the physical objects are included in the simulation loop. Compared to numerical simulation, hardware-in-the-loop simulation is closer to real flight results. However, the hardware-in-the-loop simulation system for multi-UAV missions is more complex to build, which may result in longer test cycles and higher equipment costs. Real flight testing is the most direct and effective method of verifying multi-UAV missions. For example, the U.S. Office of Strategic Capabilities has conducted 103 UAVs with independent decision-making and online formation flight demonstrations [29], and the China Electronics Technology Group has also conducted 67, 119, and 200 fixed-wing UAV flight tests in recent years [30]. However, there are many risk points in real flight testing of multi-UAV missions, specifically the high property damage caused by hardware and software problems, the accidental injuries to personnel caused by UAV failures or human operator errors, and the increased development time and cost because of frequent test flights. Therefore, in the algorithm research and verification phase, a simulation test is more feasible than a real flight. One of the problems that should be solved by UAV simulation is how to make the theoretical results, including algorithms, a technical application value, especially to support verification in real flight.

Researchers have achieved many results in studying the simulation of UAV missions. In [31], the simulation tool JADE is used to test large multi-UAV systems, and the use of the simulation environment and tools in agent communications is described. In [32], PixHawk flight control and peripherals are used to evaluate a LIDAR sensor and its ability to navigate and avoid obstacles. In [33], an interactive simulation platform for a UAV swarm based on MATLAB/Simulink is developed. It can realize 3D simulation visualization of UAVs and human–computer interaction functions such as voice and gestures. In [34], a modular co-formation software system for a UAV swarm is designed, and it is applicable for engineering practice. Currently, the multi-UAV mission simulation platform has some shortcomings, which are mainly reflected in the singularity of the function; it contains:

(1)

Programs written with software such as MATLAB are mainly used to test the algorithms themselves, with insufficient attention paid to technical implementation.

(2)

Open-source simulation software such as FlightGear focuses on flight operations training, leaving autonomy and intelligence to be desired.

(3)

The Gazebo 3D simulation environment provides good support for open-source flight controls but requires secondary development in specific modes.

In this paper, a multi-UAV mission simulation system is designed to complete the simulation of cooperative missions. The results show that the simulation verification based on the system has some certain agreement with the real flight results in terms of function realization. The system is applicable to the teaching process of related subjects and can also help in the realization of many complex tasks in the future.

2. General System Framework

Considering the mission simulation requirements of multi-UAV, the general framework of the simulation system designed in this paper is shown in Figure 1 below.

It contains six major parts: collaborative planning module, ground station module, on-board logical task module, execution module, scene display module, and flight parameter setting/state display module. The system is based on the idea of hierarchical progressive control and modular design, with the overall architecture divided into a decision layer, a control layer, an execution layer, and a display layer. It has the characteristics of centralized management, centralized decision-making, and decentralized control, which can meet the basic principle of adhering to the task-oriented upward and compatibility with heterogeneous platforms with different dynamic characteristics downward.

The decision layer is the collaborative planning module that can plan and generate task parameters based on specific task requirements. The control layer is the ground station module and the on-board logical mission module, which are mainly responsible for monitoring the status of the multi-UAV system, sending control commands, and reading, loading, and interacting with mission parameters. The execution layer is the open-source flight control PX4 firmware and virtual external sensors, which are used to track the desired state of the multiple UAVs sent by the control layer and to execute the relevant actions of the external sensors. The display layer is a Gazebo 3D physical simulation platform and the QGC (QGroundControl) virtual ground station; in addition to completing two major functions of loading different models of virtual UAV types and quantities, and real-time status display, the QGC virtual ground station can also be used to set the speed, attitude, power, etc. of the UAV platform.

3. System Design

3.1. Hardware Components

Both the collaborative planning module and the ground station module run on the ground station computer, which is a HUAWEI MateBook X Pro laptop with Windows 10 installed and equipped with an eighth-generation i7 processor, 16 GB RAM, and a 1 TB hard drive. The remaining modules run on a RADEN small-size computer host with an Ubuntu 18.04 operating system, equipped with an eighth-generation i7 processor and 32 GB RAM. The ground station computer and the Ubuntu host computer communicate via a serial line, and the system hardware composition is shown in Figure 2.

3.2. Software Design

(1)

Collaborative planning module

The collaborative planning module is developed based on MATLAB under the GUI framework, and mainly includes target assignment, formation control, path planning, conflict detection, and other functions. The module is able to establish multi-UAV typical application scenarios such as search, penetration and tracking, and complete path planning and waypoint binding based on mission planning algorithms. In addition to MATLAB, the module also supports programming languages such as Python and C++.

(2)

Ground station module

The ground station has mission start/termination, command generation/sending, status monitoring/display, emergency disposal/control, etc. functions. As the sending center of mission commands, the ground station has mission calculation functions, including parameter setting, resource allocation, command generation, and process planning. The ground station is also the data collection center for multi-UAV missions, collecting heartbeat packets sent by all UAVs over the data transmission link and decoding them to obtain flight parameters for real-time display. All flight data, mission instructions, and other data are backed up in the background, generating flight logs for later review and replay of the entire mission process. The ground station software interface is written in Qt, as shown in Figure 3.

(3)

On-board logic task module

The on-board logic task module program is the core for multi-UAV mission decision and cooperative control and is deployed on the Ubuntu host. A variety of functions such as mission calculations, image recognition, and control command resolution is realized by configuring the development environment of the ROS (robot operating system), MAVROS (MAVLINK function package in ROS), OpenCV (image processing library), Python, and C/C++ on the host computer. The on-board program architecture design is shown in Figure 4.

The onboard program development makes full use of the highly flexible software architecture of ROS and designs the entire program as a task control node (Control Node) and an image processing node (Image Node), which uses multiple threads to process multiple data internally and achieves the required functions through the communication mechanism between the nodes and the data interaction between the threads. For the task control node, the functions of the four threads are as follows:

Thread 1: subscribing to flight parameters and image data.

The mission control node uses the topic communication and service communication mechanisms of ROS to obtain the relevant data from the flight control node and the image processing node, respectively, according to the specified frequency. Among them, the absolute coordinates, relative coordinates, attitude, speed, acceleration, flight mode, starting point, altitude, and power of the aircraft are subscribed by the flight control node; the target identification flag and the relative position information of the target are subscribed by the image processing node.

Thread 2: sending heartbeat packets in real time.

Heartbeat packets are like the beating of the UAV’s “heart” and are sent continuously at a specific frequency as long as the on-board program is running. The packets contain the ID number, type, task performed, and flight parameters, etc. The heartbeat packets are sent to the ground station via the serial port by calling the serial send function.

Thread 3: receiving mission orders in real time.

Mission instructions are sent to each UAV through the ground station. The ground station sends different instructions depending on the mission phase, and the on-board computer invokes the corresponding cooperative control actions upon completion of the analysis.

Thread 4: control commands resolving in real time.

This thread designs the planning and switching between different mission phases, calls the corresponding cooperative control algorithm according to the current mission phase, brings in the mission parameters, and resolves the flight control commands. After resolving the control commands, the MAVSDK in the MAVROS function package is used to encapsulate the control instructions according to the MAVLINK protocol and send them to the PX4 flight controller for execution.

The image processing node has an image recognition function and is deployed in a single thread. The target identification is performed using the image captured by the monocular camera to create the target identification mark, and once the target is identified, the positioning algorithm is invoked in the thread to obtain the relative coordinates of the target in the airframe coordinate system. The target identification mark and the relative position coordinates of the target are encapsulated according to the ROS message format and entered into a specific topic subscribed by the task control node.

In the ROS environment, the software architecture is modular, and the functions are flexibly extended by developing working nodes. The subsequent replacement and addition of sensors only requires the modification or addition of nodes. At the same time, the on-board software development makes full use of the strong secondary development capability of the open source PX4, with adjustable flight parameters and fewer constraints, reducing dependence on commercial flight control and enhancing mission autonomy.

(4)

Execution module

The execution module is the open-source PX4, version 1.11.2. The source code is available via the Github project and can be downloaded locally and compiled for use in the simulation environment.

(5)

Scene display module

The scene display module is the Gazebo 3D physical simulation software. The Gazebo model of the UAV is provided in the source code of PX4, and a UAV model of IRIS in the Gazebo environment can be called directly from the launch file, as shown in Figure 5. By modifying the configuration file, we can also set the type and number of UAVs.

(6)

Flight parameter setting/state display module

The flight parameter setting/state display module is a virtual QGC ground station, through which the UAV operation status can be monitored in real time and the UAV trajectory can be displayed for comparison with the planning results.

4. Path Planning Simulation Test for Two UAVs

Take the path planning of two quadrotor UAVs as an example to illustrate the whole working process of the simulation system.

4.1. Mission Scenario Description

The scenario is as follows: after a geological disaster in a city where a large area of buildings has collapsed and communications have been disrupted, two UAVs need to be dispatched to search and identify the area where the casualties may exist. At this point, the general location of the target area is known, and the mission environment is divided into a 10 × 10 grid on the map. The specific reconnaissance areas of the two UAVs are designated by the superior, and the routes are crossed, so the planning module needs to make further judgment, as shown in Figure 6. Under the condition of setting velocity, if two UAVs arrive at the intersection point of the planned routes at different times, it means that there will be no collision, and the planning result is valid. Otherwise, re-planning is required.

4.2. Path Planning for Two UAVs Based on Ant Colony Algorithm

Ant colony algorithms are inspired by the foraging behavior of ant populations, in which each ant in the colony leaves pheromones on its movement path to communicate information to the colony. The shorter the movement path, the higher the pheromone concentration, and the more likely the path will be selected by the ants. This forms a positive feedback mechanism for the ants to find the shortest path. The basic principle of the ant colony algorithm is as follows [35,36,37,38,39].

The ants select paths according to the pheromone concentration and the heuristic function, and the probability that ant k moves from grid i to grid j can be expressed as follows:

$$P_{ij}^k=\{\begin{array}{r}\frac{{\tau }_{ij}^{\alpha }(t){\eta }_{ij}^{\beta }(t)}{{\displaystyle \sum _{s\in A_k}{\tau }_{is}^{\alpha }(t){\eta }_{is}^{\beta }(t)}},s\in A_k\\ 0,s\notin A_k\end{array}$$

(1)

where α is the pheromone heuristic factor and β is the distance expectation function factor, which affect the importance of pheromone and distance heuristic function respectively. Ak indicates the next destinations ants can reach. τ_ij(t) is the pheromone concentration on the moving path at time t; η_ij(t) represents the distance heuristic function, where

$${\eta }_{ij}(t)=\frac{1}{d_{ij}}$$

(2)

$$d_{ij}=\sqrt{{(x_i-x_j)}^2+{(y_i-y_j)}^2}$$

(3)

Each ant will leave a certain amount of pheromone when moving. Therefore, when the algorithm iterates continuously, the pheromone content in the path gradually accumulates and volatilizes at the same time. When all ants in the population complete each round of iteration, the pheromone content on the path will be updated according to the following rules:

$${\tau }_{ij}(t+1)=(1-\rho ){\tau }_{ij}(t)+\mathsf{\Delta }{\tau }_{ij}(t)$$

(4)

$$\mathsf{\Delta }{\tau }_{ij}(t)={\displaystyle \sum _{k=1}^m\mathsf{\Delta }{\tau }_{ij}^k(t)}$$

(5)

$$\mathsf{\Delta }{\tau }_{ij}^k=\{\begin{array}{l}\frac{Q}{L_k}\\ 0\end{array}$$

(6)

where ρ is the pheromone volatilization coefficient; Δτ_ij represents the sum of pheromones released on the path between the two nodes; $\mathsf{\Delta }{\tau }_{ij}^k$ denotes pheromone increment; L_k is the path length of ant k, and Q is the pheromone enhancement coefficient.

By smoothing the planned curves, the obtained UAV flight trajectory is shown in Figure 7. The algorithm can discriminate the intersection point, and the two UAVs will not reach the intersection at the same time. Therefore, the planned path is effective and feasible.

4.3. Simulation Results

After the ground station starts the mission, the UAVs fly according to the pre-planned path, and the states at different simulation stages are shown in Figure 8.

To facilitate comparison and verification, the planned paths are flipped in a certain order, and they are shown in Figure 9.

The path of the UAV recorded by the QGC virtual ground station is shown in Figure 10, and the comparison shows that the flight data from the simulation system matches the planning path well.

4.4. Real Flight Validation

To verify the results of the path planning simulation, two multi-rotor UAVs are used to verify the mission in real flight. The software needs to delete the state-switching program in the on-board logic task module and replace it with remote-control-mode switching. In addition, the UAV used in the simulation is IRIS, which is built in the PX4 firmware. For the real flight, an in-house-developed quadrotor UAV is used for demonstration, which needs to be sensor calibrated before the mission. The actual flying UAV and flight scenes are shown in Figure 11 and Figure 12.

Figure 13 shows the position data of the real flight of UAV1 and UAV2, respectively, with the discounted line is the planned path and the straight line is the return path. Due to the complexity of the real flight environment and the differences between the UAVs themselves in terms of mechanical structure and dynamic behavior, the simulation data and the real flight results do not match exactly under normal circumstances. The error data varies greatly each time.

Figure 14 shows the velocity change curves of the real flight process. The real flight uses the position control mode, and both UAVs have a more obvious deceleration before reaching the desired waypoint each time, which in turn determines whether they have reached the target point. The velocity change curves show a zeroing process. In both simulation of this system and real flight, the single position control makes the flight process incoherent, a defect that cannot be reflected in simple traditional numerical simulation and needed to be further improved in subsequent engineering applications.

5. Conclusions

In this paper, a multi-UAV mission simulation system is designed. The main advantage of the system is that the results of collaborative mission planning are simulated and tested in a 3D scenario. The changes in position, velocity, attitude, and other state variables of the UAVs are reflected and recorded simultaneously. The simulation program can be applied directly to the real flight with only a few settings. Each subsystem adopts modular design, which is convenient for promotion and replacement. In terms of function realization, the simulation verification based on the system has a certain agreement with the real flight results.

The next step is to improve the system from two aspects. One is to enrich the Gazebo simulation scene, and the other is to equip the virtual UAV with more sensors to improve the simulation capability of the system for autonomous and intelligent UAVs.