1. Introduction
Advances in science and technology in the past decade have brought increased interest towards the development of autonomous UAVs. Swarms are common in nature, such as fish schools, ants and honeybee swarms. The study of UAV swarms based on this biologically inspired concept has become an important trend in UAV development [
1,
2]. This makes autonomous UAVs a very promising technology for activities such as geological mapping, surveillance, resource gathering and rescue missions, among others. Maintaining, changing and reconstructing formation control is necessary for UAV swarm missions. A fundamental component of UAV swarm missions is a navigation system capable of providing navigation and relative positioning. Currently, most UAV formation controls are based on integrated navigation systems (INS), which include an inertial measurement unit (IMU) and a GNSS receiver. The IMU can accurately estimate the attitude and position of the UAV over a short period of time. However, due to random drift errors in inertial devices, the pose error accumulates over time. A GNSS such as the global positioning system (GPS) is more accurate over longer periods of time. Therefore, an IMU and GNSS are usually combined through extended Kalman filters (EKF) or unscented Kalman filters (UKF) to achieve high-precision pose information over a long time. Generally, UAV swarms can achieve high-precision relative positioning by sharing their locations through internetworking. However, in constantly evolving, high-intensity adversarial environments, relying solely on a GNSS to establish reliable spatial position and velocity measurements relative to geographic coordinate systems has become difficult [
3]. With the development of the GNSS, the competition and control over the GNSS and its operating environment are becoming increasingly fierce. Currently, countermeasures such as interference, deception and attacks against GNSS signals and systems are emerging one after another, and the operating environment of satellite navigation systems is gradually deteriorating. The potential threats to the GNSS system mainly include the following: (1) directly killing or interfering with navigation satellites through anti-satellite weapons; (2) the use of radio interference, information security attacks and other means to target ground operation and control stations and other facilities, which makes it difficult for these ground operation and control facilities to function properly, resulting in the inability of the entire navigation system to function properly; (3) using various methods such as suppression interference and deception to directly interfere with the navigation terminal, making it unable to output or mistakenly output navigation information, thereby interfering with the normal use of the navigation system. On the other hand, GNSS signals are affected when passing through environmental objects such as trees, buildings and rocks. These obstacles can cause signal reflection, scattering, refraction, etc., resulting in longer signal propagation paths, weakened signal strengths, phase delays, etc. These impacts can lead to a decrease in the reception quality of GNSS signals, thereby affecting the accuracy of position calculation [
4]. Therefore, navigation methods that do not rely on communication under GNSS-denied environments have become increasingly important for UAV swarm applications. Due to the complexity of outdoor environments, UAV formation flying requires a large number of personnel, which leads to high development costs and large time requirements for navigation algorithms. Currently, open-source simulation systems such as Promethues [
5], Xtdrone [
6,
7] and Rflysim [
8,
9] are more concerned with single control or swarm control, and, in the early stages of developing relative navigation algorithms, it is necessary to verify them in simulation systems to achieve the rapid iteration and validation of the algorithms.
Visual navigation calculates the pose of UAVs relative to ground features to achieve positioning and navigation. Visual navigation has the advantage of minimal environmental impact and high accuracy and has been widely used in GNSS-denied environments [
10]. Benefiting from the low cost and high-precision positioning, the SMN system possesses key potential in the GNSS-denied environment [
11]. In [
12], a novel SMN is proposed, and an optimized factor of the homography matrix is used to reduce the projection errors. In the experiment, the UAV flies at a 150 m height and the flight speed is 12 m/s. The results show that the time consumption for on-matching is 0.6 s and the position error is 4.6 m. In an SMN under GNSS-denied environments, the navigation accuracy is related to the flight altitude. Generally, a navigation error to flight altitude ratio of less than 1% is considered to reflect high precision. Due to factors such as changes in flight altitude and attitude during the flight process, the real-time aerial photos taken by drones exhibit complex geometric distortions, resulting in severe rotation, scaling and even deformation compared to the pre-stored reference images of drones. The inertial information from the IMU is not affected by external factors, which is beneficial. Inertial information can be used to solve the perspective transformation relationship between images [
13].
In terms of relative navigation, the work in [
14], which investigates onboard visual relative localization in GNSS-denied environments, detects the IDs through black and white colors and achieves localization through the diameter of the circles. Another approach [
15] presents a novel onboard relative localization method for swarms of multirotor UAVs. The method uses a new smart sensor called UVDAR, as part of a UVDAR system, which can be used to obtain both the relative position and orientation from a modified camera. Blinking ultraviolet markers are used to identify the ID of the UAV and the orientation. In [
16], the author utilizes UWB for peer-to-peer localization.
The simulator integrates the world environment, sensor physical models, dynamic models of fixed-wing drones and multi-rotor drones, controllers of the drones, a formation controller, an ISMN module, relative navigation based on UWB and a vision module. The contributions of this study are as follows:
- (1)
A real-time simulator for navigation in GNSS-denied environments is developed in order to improve the iteration efficiency of navigation algorithms;
- (2)
A novel scene matching navigation algorithm called ISMN is proposed; based on the simulator, the ISMN algorithm is validated;
- (3)
A relative navigation method that does not rely on inter-communication is proposed.
The remainder of this article is organized as follows.
Section 2 describes the architecture of the simulation, which focuses on scene matching navigation and relative navigation.
Section 3 describes the sensor model used in the simulation.
Section 4 demonstrates the simulation result. In
Section 5, the conclusions are summarized.
2. Architecture
As shown in
Figure 1, the simulator contains the world, model, controller, ISMN, formation controller and relative navigation modules. In the world module, various environments, such as deserts, grasslands and canyons, can be simulated, which can be used to verify the robustness of the relative navigation. On the other hand, Google Earth containing GPS information is used for the SMN. Sensor models are strongly related to state estimation. In the simulator, a gyroscope, accelerometer, magnetometer, barometer, camera, UWB, etc., can be imported as plugins to supply accurate sensor models for the simulation platform. The controller and formation controller serve for control and swarm motion. Inertia, altitude and images are used for absolute navigation in GNSS-denied environments in the ISMN module. The relative navigation module receives images from cameras and distances from UWB and calculates the relative position for the formation controller.
In the simulation, the flight environment is set through the world module, and the UAV flies in the environment with sensors such as accelerometers, gyroscopes, magnetometers, UWB and cameras and so on. The ISMN module collects the information of the accelerometer, gyroscope, magnetometer and camera and calculates the absolute pose of the UAV. The pose information is sent to the controller module, which can stabilize the UAV through closed-loop control. The relative navigation based on UWB and vision module collects the images of adjacent drones from the camera and the distances of adjacent drones from UWB and calculates the relative position of the UAV. The relative position is sent to the formation controller module, which can implement swarm control.
2.1. Inertial-Aided Scene Matching Navigation (ISMN)
SMN is an effective solution for localization in unknown environments when the GNSS is disturbed. We have developed a new localization method that uses local feature points extracted by the Accelerated Robust Feature (SURF) to match the current drone’s captured map image and onboard satellite image, thereby obtaining the position information of the current UAV. In order to improve the efficiency of SMN, the inertial information is obtained for assistance. The ISMN procedure is shown in
Figure 2. It uses IMU propagation to determine the scope of the reference satellite map to improve the real-time performance. Considering that the real-time images from the onboard camera and the reference images from the satellite map are not in the same frame, the larger the obliqueness, the higher the probability of mismatching. To solve this problem, we use the attitude from the IMU to correct the real-time input images. This approach enhances the accuracy of image matching in large viewing angles and, consequently, improves the precision of SMN.
2.1.1. Reference Map from INS Propagation
IMU data are used to narrow the range of satellite images that we need to analyze to perform localization. The propagator is based on the relative acceleration and velocity of the INS model, which is accurate over a short time. As shown in
Figure 3, the position of the UAV at time k is
. After a short time, the position of the UAV at time
k + 1 is
. The distance at which the UAV flies forward is
, which can be obtained by INS propagation. The deviation
caused by the drone attitude can be obtained through parameters such as the drone flight attitude and altitude. The specific derivation process is as follows.
The velocity update equation with respect to the East, North, Up (ENU) frame in the simplified inertial navigation algorithm is shown in Equation (1).
is the transformation from the body frame (b frame) to the local geographic frame (n frame), represents the velocity in the n frame, is the specific force vector as measured by accelerometers in the b frame, is the rotation rate of the Earth expressed in the n frame, is the transport rate, which is only related to the longitude and latitude, and is the local gravitational acceleration vector.
The position update equation in longitude and latitude is shown in Equation (2).
where
represents longitude,
represents latitude,
is the Earth’s radius,
is the eccentricity of an ellipsoid and
is the oblateness of the ellipsoid shape.
According to the current longitude, latitude, INS propagation, attitude and height of the drone, we can predict the next feature position in the base map.
The next drone position can be obtained as follows:
Concerning the attitude of the drone, the positions of images captured by the camera are derived as
where
is the drift from obliqueness, is a transform from the NED coordinate to the WGS-84 coordinate and is the altitude of the UAV.
According to , we can obtain the feature position in the base map, so the number of images that we need to analyze is significantly reduced. We use this method to increase the efficiency of the SMN.
2.1.2. Inertial-Aided Georeference
SMN has a mismatching problem, because the attitude changes during UAV flight will cause differences between the base map and the input images. However, due to the immunity of inertial information to external influences, the inertial information can be utilized to project the real-time image to the reference image coordinates. The matching method aided by inertial information is shown in
Figure 4. The attitude of the camera can be estimated from the mechanization results of the IMU. The transformation from the body frame to the navigation frame is
[
17]:
where
,
and
denote yaw, pitch and roll, respectively.
The conversion relationship between the camera coordinate system and the IMU coordinate system (body coordinate system) can be expressed as
where
refer to the installation angle between the camera and IMU.
The camera’s optical axis taken from the reference image is almost perpendicular to the ground,
is considered to be perpendicular to the ground downwards,
and
are in the same direction and
and
are in the same direction, Therefore, there are
The conversion relationship between the reference maps and real-time maps can be obtained from the above:
According to the above conversion relationship and camera parameters, the squint image can be converted to an orthophoto:
where
, and
is the z-direction coordinate component of
in the camera frame.
denotes the internal parameters of the camera, which can be obtained by calibration.
2.2. Relative Navigation Based on Vision and UWB
In this work, we propose the use of local visual information and UWB to perform relative localization. Here, we aim to utilize the onboard sensors including cameras and UWB for relative navigation. In this way, there is no need for inter-communication and localization infrastructure.
2.2.1. Detection and Tracking of Adjacent UAVs
YOLOv3-tiny [
18] is used to detect the adjacent UAVs. YOLOv3-tiny is a lightweight state-of-the-art convolutional neural network (CNN) detector that can work on the on-board computer to provide detection. To utilize the detector to detect adjacent UAVs, an additional training set captured by the camera on the UAV, as well as containing the UAVs, is labeled and fed into the CNN. After training, the CNN is able to detect the UAVs efficiently.
For the tracking of the drone, a KF filter is adopted to predict the trajectory. Moreover, we use the Hungarian algorithm, which is a matching algorithm, to match the detected and predicted result. Blinking LED markers are used to identify unique IDs, preventing all UAVs in the swarm from having a similar appearance.
For the KF filter of the trajectory predicted, the state of the system is
where
and
are the center of the object in the image,
is the aspect ratio,
is the altitude and the observation equation can be expressed as
2.2.2. Calculating the Relative Position
The camera imaging plane is shown in
Figure 5.
is the origin located in the image coordinate system. The target’s coordinates in the image coordinate system are denoted by
. In the physical coordinate system of the camera, the corresponding coordinate origin is located at
, and the target coordinate system is denoted by
.
is the focal length of the camera, and
are the physical sizes of each pixel on the imaging plane along the axis of
.
where
refers to the line-of-sight angle,
refers to the line-of-sight azimuth angle and
refers to the line-of-sight elevation angle.
The position among neighboring drones in the swarms can be calculated as
where
.