StratoTrans: Unmanned Aerial System (UAS) 4G Communication Framework Applied on the Monitoring of Road Traffic and Linear Infrastructure

Abstract

This study provides an operational solution to directly connect drones to internet by means of 4G telecommunications and exploit drone acquired data, including telemetry and imagery but focusing on video transmission. The novelty of this work is the application of 4G connection to link the drone directly to a data server where video (in this case to monitor road traffic) and imagery (in the case of linear infrastructures) are processed. However, this framework is appliable to any other monitoring purpose where the goal is to send real-time video or imagery to the headquarters where the drone data is processed, analyzed, and exploited. We describe a general framework and analyze some key points, such as the hardware to use, the data stream, and the network coverage, but also the complete resulting implementation of the applied unmanned aerial system (UAS) communication system through a Virtual Private Network (VPN) featuring a long-range telemetry high-capacity video link (up to 15 Mbps, 720 p video at 30 fps with 250 ms of latency). The application results in the real-time exploitation of the video, obtaining key information for traffic managers such as vehicle tracking, vehicle classification, speed estimation, and roundabout in-out matrices. The imagery downloads and storage is also performed thorough internet, although the Structure from Motion postprocessing is not real-time due to photogrammetric workflows. In conclusion, we describe a real-case application of drone connection to internet thorough 4G network, but it can be adapted to other applications. Although 5G will -in time- surpass 4G capacities, the described framework can enhance drone performance and facilitate paths for upgrading the connection of on-board devices to the 5G network.

1. Introduction

Drones are one of the most fastest growing business of the decade 2010–2020, either for professional or personal uses, and scientific purposes [1,2,3]. New applications in unmanned aerial systems (UAS) are expected to become increasingly useful to accelerate productivity, save costs and risks, and to extend the current limitations of these systems, adapting them to the next decade. The technological challenge of this study is to develop a flexible solution to remotely and in real time control fixed-wing aircrafts by means of the 3G-4G LTE telecommunication network, that is ready to be adapted to 5G. The innovation aims at providing high control range Beyond the View Line of Sight of the pilot (BVLOS), accordingly with the aircraft flight autonomy. Furthermore, this will allow the integration of deep learning tools, such as Computer Vision (CV) technologies, and big data analysis, adapted to the new autonomous drone fleet paradigm.

In recent years, drone communications have been increasingly addressed by both the academia and the industry [4]. Unmanned vehicles and battery technologies have evolved to provide increasing endurance, currently allowing flight distances that exceed the range of the local radio link with the Ground Control Station (GCS)—where the pilot is usually located—especially when dealing with fixed wing drones. To fix this issue many researchers and companies have proposed alternatives to the local link that involve the connection of the UAS to the internet [5], thus being accessible from almost any connected device. For instance, flying ad hoc networks (FANETs) have become a hot topic since they represent a step further, enabling communications from single drones to multiple drones [6]. However, a challenge that is still pending to be solved is the BVLOS communication, which inherently implies long-range links. Moreover, if the desired solution requires high-capacity links, such as real-time video applications, the use of cellular network (i.e., 4G) is one of the best choices we can aim at. In this direction, [7] analyzed the main advantages and drawbacks of such schemes and provided some recommendations. [8] further discusses technological challenges including interferences and mobility issues and gives potential solutions. This also caused the appearance of dedicated protocols and communication network architecture solutions for a wide range of drone use cases. The use of cellular networks requirements for the connectivity of drones was already analyzed in 2015 [9] but it is fully in development since the implementation of 4G technology in most developed countries (around 2016), which has entailed further network design proposals [10] and even drone-to-drone communications [11]. This technology allows the transmission of relatively large data, exceeding mere telemetry packets, being able to handle video and images especially. Recent implementations like LARUS [12] have used 4G/LTE networks to connect various drones for long-range rescue missions [13]. However, some threats are still concerning the community, such as the on-board energy consumption, the handover rate between antennas, or the bandwidth [14]. Some alternatives to the use of the terrestrial antenna network as the main way to connect the drone to the internet are focusing upwards, to the High-Altitude Pseudo Satellites (HAPS) [15,16] or to satellite constellations [17], but these are in experimental stage and currently there are not available options.

The problem addressed in this work is to supplement the drone-to-local GCS data link with a drone-to-remote server data link. This new framework, based on 4G communications to directly link the drone to internet, allows the real-time processing of the drone acquired data in powerful servers located in the headquarters of the data user (Figure 1).

In this work, we provide application details of our own-operated UAS, while discussing implementation and component details in the system and sharing the resultant complete architectural scheme. The initial and main requirement was the unlimited range of communication between the drone and the ground segment (including the pilot GCS and the data processing center), accounting for telemetry and live video to monitor transportation infrastructures and traffic, namely StratoTrans project [18]. This requirement involves overpassing the local radio-link communications between the GCS and the drone Flight Controller (FC). The objective is tackled by using the 4G network, which allows to link the drone with the GCS and with any device connected to internet wherever it is located, being the latter the main interest of the presented framework. The purpose of the StratoTrans project is to run CV algorithms over the video acquired to extract object identification using convolutional networks [19,20], tracking trajectories [21,22], and velocity estimation [23,24]. Therefore, the drone acquired video is sent to a data server to be processed and finally it is visualized in the operator’s headquarters computer or in a user’s mobile device.

2. Materials and Methods

2.1. Methodological Framework for UAS—4G Communications

4G communications make sense when the aircraft cannot maintain the radio link with the GCS, typically when a limit distance is exceeded or when there are topographic occlusions between them. Therefore, we consider BVLOS conditions when it is necessary to use such cellular network. The presented solution can transmit imagery (both First Person View (FPV) orientation or zenithal captured) and telemetry data from the aircraft via the 4G network. The imagery, which can be real-time video, is intended to be viewed in the GCS o in any device with internet connection.

The hardware and system architecture inside the drone include a set of navigation sensors (GNSS, IMU, barometer, compass, and pitot tube) that feed data to the FC, and a telemetry module with an antenna to radio-link the FC with the GCS. In order to add 4G capabilities, an extra microcomputer, a modem and a SIM card are needed to connect the drone to Internet. Even though the extra hardware could be powered through the FC, it is worth noting that we power it with an independent power circuit using a power module and a regulator in between. This avoids the supply through the FC and helps to keep the independence between functions, for safety reasons. Regarding the video cameras, these must be compatible with the microcomputer ports. In the case of FPV video cameras, they must comply with specific regulation features such as the Field of View (FOV) and frames per second (fps), while for downwards Earth Observation (EO) cameras the fit for purpose spatial and spectral resolution and geometric quality are the critical issue.

The system network architecture is based on a Virtual Private Network (VPN) composed of three main IP address endpoints: 1/The on-board drone location. 2/The GCS pilot location. 3/The remote server location. Each of these three VPN endpoints have access to internet and are securely linked between them, with an internal fixed identifier, as shown (Figure 2).

A dedicated software is installed in an on-board microcomputer, which reads the telemetry from the Flight Controller (FC), the video from a camera, and is accessible to the VPN endpoints for its visualization in a web interface. In addition, it can stream both telemetry and video data to the VPN endpoints, which can then locally reproduce the data by reading directly from the network ports. The microcomputer acts both as a hub between the drone input data to be sent (FC and cameras), and as a bridge between the drone and the VPN.

Regarding the streaming protocols, TCP/IP is the recommended protocol to securely transmit the telemetry without losing data packages, while Real Time Streaming Protocol (RTSP) through UDP is the best option to transmit the video and assure a combined low latency. The telemetry data encoding and packaging is done by following the Micro Air Vehicle Link (MAVlink) protocol, which can be decoded and plotted in widely used GCS software.

2.2. Hardware, Software, and Methodology under Use Case

Many test flights have been performed at Barcelona Drone Center (BDC) [25] testing site facilities. BDC, located in Moià (Catalonia, Spain), manages a Temporary Segregated Area (TSA) of 2500 ha specifically conceived for BVLOS flights. Their facilities are located in a rural area, without the ubiquitous 4G antennas and signal overlapping of urban areas, making it an excellent site to test coverage and performance indicators. Also, other flights were carried out in different locations of Catalonia to apply the developments with different traffic and infrastructure scenarios, such as roundabouts and roads.

The implemented hardware and system architecture (namely EXO Data) have been integrated and developed by Exodronics, in collaboration with other institutions under the StratoTrans project. EACOM [26] has collaborated with telecommunications issues; UAVMatrix [27] provided a reference commercial product to develop 4G drone-to-ground services; the server to store the mission’s telemetry data and the video hosting, has been developed in collaboration with NEXIONA [28]; the CV algorithms specifically designed for tracking elements from the video imagery where developed by Eurecat Multimedia department [29]. The aircraft used to carry out the telecommunication experiments is an EXO C2-L+ fixed wing developed by Exodronics [30]. This lightweight platform (1.3 kg Maximum Take-off Weight (MTOW)) provides around 75 min flight autonomy, which is appropriate to test under BVLOS conditions. It is, then, a specific case example of where it is appropriate to use such cellular network, capable to transmit FPV video, EO imagery and telemetry data from the aircraft via the 4G network. However, for hovering traffic monitoring in roundabouts, we used multicopter platforms (Figure 3).

The specific hardware housed within the drone is a Pixhawk 2.1 Cube Black Flight Controller [31,32], and a telemetry module with an antenna to locally radio-link (Spread Spectrum at 868 Mhz) the FC with the GCS. In the EXO Data implementation, an extra microcomputer (Raspberry Pi Zero W [33]), a 4G/LTE USB-stick a modem with a data rate of 150 Mbps [34], and a multioperator SIM card [35,36] are used to connect the drone to Internet, with an aggregated extra weight of 40 g. The onboard power system is based on a Li-ion 4S 3.4 Ah battery and a power module that feeds independently the FC and the EXO Data hardware. The EXO Data system video cameras, compatible with the microcomputer ports, are a ZeroCam [37] for FPV video, and for downwards Earth Observation video cameras we use a PiCam V2 [38] or a remote sensing camera with video functionalities with a HDMI—CSI converter connected to the Raspberry video input port. The video encoding and packaging can be carried out with H264 compression (1:125 ratio), accounting for a video quality of 1280 × 720 pixels, 720 p and 30 fps, which can be decoded and reconstructed in widely used video software (e.g., VLC, Gstreamer). The complete hardware implementation in the EXO C2-L+ is shown as follows (Figure 4).

The implementation uses a secured VPN [39] to switch the 3 endpoints: The drone, the GCS and the EXO Data server. The telemetry link is based on a MAVLink decoders, with an open-source software in the GCS (Mission Planner [40]) that can be linked to the onboard FC thorough local radio-link or thorough TCP/IP protocol. Moreover, a specifically developed software [28] in the EXO Data server is used to store the telemetry and imagery data in separated dockers. The complete implementation is detailed as follows (Figure 5).

The Raspberry Pi, which runs a software based in GStreamer and ZeroTier, captures video from the Pi Camera through a CSI interface, and captures telemetry data from the PixHawk 2 Cube controller, through the GPIO interface. Additionally, a USB interface connects the Raspberry with the 4G modem. Finally, a PC acts as a GCS and receives the data through the VPN tunnel. For instance, it could play in real-time the video using GStreamer, VLC or MissionPlanner. In this practical case, the pilot is located near the ground control station, which is also acting as an EXO Data server, so there are only two VPN endpoints. As an extra interface, we enable a second gate to obtain access to the Raspberry through the ad-hoc WiFi network using the SSH protocol. This way, when it is necessary to setup the software before flight, the 4G connection does not have to be used.

Finally, video frames are processed in the EXO Data server, where Faster-RCNN algorithms are located to detect and classify vehicles [21], to track its trajectories applying the IOU-tracker algorithm and the intersection over union concept [22], and extracting the relative movement of the drone respect to the ground objects to determine its speed [23].

3. Results and Discussion

The implementation, namely EXO Data, is based on existing commercial products [27], but with important evolutions. The solution links the on-board drone hardware and the ground segment, composed by the GCS (link A) and the novelty of the data server with extra processing capacities for the traffic monitoring (link B), both with different requirements (Figure 6).

3.1. Telecommunications Analysis and Applied Results

The throughput of the 4G network is not symmetrical, having less available throughput in the upstream channel than in the downstream channel. In the case of streaming video transmission, the limiting factor is the throughput in the upstream channel, from the drone to the ground points. The standard specifies a maximum of 100 Mbps in the downstream channel and 50 Mbps in the upstream channel. However, since these transmission systems share the medium with several users, the actual throughput that we are going to obtain will depend on factors such as the load of the network at that time and the interference caused by such. In practice, we found a range from very low rates to peaks of 15 Mbps. Taking as an example a 720 p video at 30 fps, the calculation of the storage needs, in MB, assuming 3 colors per pixel and 8 bits per color and a 1:125 compression rate for one minute of video can be calculated as follows (Equation (1)):

C = ∆c · ∆r · fps · Nb · λ · t · CCR

(1)

where C is the storage need in MB, ∆c is the number of columns, ∆r is the number of rows, fps is the frames-per-second rate, Nb is the number of bits resolution of each color band, λ is the number of bands per pixel, and CCR is the Code Compression rate.

One minute of video will require approximately 40 MB of memory. Therefore, a bandwidth of at least 5 Mbps will be necessary to transmit this video in streaming with the appropriate quality. Taking as an example a video in 4 K quality and 30 fps, one minute of video will require approximately 170 MB of memory. We consider that a bandwidth of at least 25 Mbps will be necessary to transmit this video in streaming. This is a high consumption of data, which makes it unaffordable for common users, but with the upcoming 5G technology this scenario can change, as noticed in [41].

Latency can be the most critical point, especially in first person view (FPV) video, which must be below 250 ms following the current regulations in most European countries [42]. Depending on the position of the VPN root servers, the latency can reach 100 ms only due to the system architecture. This, added to the latency of the electronics (between 30 ms and 50 ms) and the one introduced by the 4G network, can in some cases cause problems for the video in real time (UDP protocol). In practice, our tests show that although there are some small micro-cuts, the quality of the FPV is good enough at a resolution of 720 p and 30 fps and the latency is under the 250 ms requirement. However, this value is very volatile and dependent on the signal quality. These results agree with the possibilities described in [9] and further developed in [5,10].

The major concern is the 4G network telecommunication feasibility for such applications, especially the link cuts due to sporadic coverage losses. We carried out theorical and practical experiments to demonstrate its feasibility. Typical antennas deployed for the 4G network have vertical radiation patterns with half-power beam widths that are around 7°. The antennas are subjected to both mechanical and electrical inclinations (tilt) to get them to focus on the coverage area and avoid overreaching with neighboring stations. We made a trigonometric approach, in order to estimate the gain of the antenna as a function of the horizontal distance from the drone to the base station. The vertical angles between drone and antenna, are set between 5° and 20° (taking into consideration the tilt, 5° is considered the maximum of the secondary lobe and 20° is considered out of the secondary lobe). At distances in the range of hundreds of meters in the horizontal plane, the gain of the antenna is expected to be more than 20 dB below its maximum gain. As the distance increases, the vertical angle decreases and the gain of the antenna gets closer to its maximum gain, compensating the loss due to free space propagation. Figure 7 illustrates the above described:

Taking a scenario considering the most restrictive frequency (2.6 GHz), a carrier power of 20 Watt, the gain of a typical base station antenna in the direction of maximum radiation (18.5), the free space loss the level of the main lobe to the secondary, and a distance of 1376 m, the resulting expected signal level at the drone antenna is −59.79 dBm. A good signal level in 4G is considered above −75 dBm so there are more than 15 dB of margin (75 dBm–59.79 dBm), that would increase the distance beyond the 2 km. The frequency of 2.6 Ghz is likely to be used in an urban area or an environment with small cells, but for rural environments it is much more likely to use lower frequencies (in the band of 800 Mhz) that would increase the coverage range even further.

Therefore, the drone can receive a good quality 4G signal at distances of around 2 km from the network antenna node, being the lack of coverage more probable in the GCS that in the drone, as explained below. In practice, we found that at flying heights under 120 m the coverage was better than at ground level. The explanation for such behavior is that the topography and the surface elements (vegetation, buildings) are not shadowing the signal when the drone is in the air, while it was compromised in ground before taking off. Moreover, if the 4G network antenna covering the flight area is located on a hill exceeding the height of the drone top altitude (as usual, for instance, in rural areas), the main antenna lobule is covering the drone 4G receiver and the signal is excellent. Nevertheless, if the 4G network antenna is located within a basin position (not common), or the drone if flying at higher altitudes than 120 m above ground level (commonly not allowed), the coverage quality drops. However, the movement of the drone will cause at some point to enter a zero radiation from the network antenna node. This will force the start of a handover protocol to transfer communication to another node on the network, which will interrupt for some seconds the communication. The antenna handovers and the performance indicators depending on the coverage have been analyzed in [7,14], with similar results, but they are dealing with copters and in urban areas with more 4G signal density. In other circumstances, such as in [13], the coverage is worse than in our study, but they are supported with mobile ground station to aid the drone.

There are some telecommunication companies that have agreements with the main telecommunication operators and provide multioperator SIM cards (e.g., Wirelesslogic [36]). Having a multioperator SIM card in the drone on-board modem enables us to connect the operator network with the best receiving signal at the drone antenna. In most developed countries, there are three or more physical network managers (e.g., Vodafone, Orange, Telefonica) and they share the infrastructure. Nevertheless, there will always be a signal better than the others, for instance due to network management parameters, so the multioperator feature adds a significant value. The multioperator SIM analyses the coverage when it becomes active and selects the operator with the best signal to perform the specific flight.

3.2. Traffic Monitoring Applied Results

The video acquired by the drone sensor is transmitted by the 4G network thorough the VPN and received in the EXO Data server endpoint. There, the video is stored in a docker, namely the video log repository. Tacking advantage of the higher computing capacities of the data server (compared with GCS and with the on-boards RPi), the video can be processed with the computer vision algorithms to detect traffic-flow events obtaining near-real time information.

In this work we have used the state-of-the-art algorithm for object detection, Faster R-CNN [21], trained on the Visdrone dataset (http://aiskyeye.com/) for the detection of six previously defined vehicle classes: Pedestrian, bicycle, car, van, bus, and truck. The algorithm processes an input video frame by frame. The output of the algorithm is a bounding box with the coordinates of each detected vehicle and the probability that the detected vehicle belongs to the corresponding vehicle class. The threshold for probability is set to 70%, meaning that only the vehicles that are detected with the probability higher than 70% are considered. After detecting the vehicles, we have applied the IOU-tracker algorithm [22] (https://github.com/bochinski/iou-tracker), assigning a unique ID to each detected vehicle. Finally, the algorithm implemented for speed estimation is based on Li et al. proposal [23]. The imagery analyzed was taken by a static drone where the surface of the road was flat and the camera was mounted with a zenithal orientation, so the distortion given by perspective is minimal.

The three algorithms streamline the process of manually annotating data by a traffic analyst, allowing, among others, the automatic computation of a roundabout matrix. The main applications of these CV algorithm are (Figure 8):

(a)

Generation of in-out matrices in roundabouts per vehicle type.

(b)

Estimation of vehicle velocities and trajectories.

(c)

Heatmap generation per each vehicle class.

3.3. Linear Infrastructure Monitoring Applied Results

The photographic imagery acquired by the drone sensor is transmitted by the 4G network thorough the VPN and received in the EXO Data server endpoint. There, the imagery is stored in a docker, namely the photo log repository.

The retrieval of information related to the road infrastructure and its affectation zone is not required in real-time. Moreover, mapping processing to obtain Digital Surface Models (DSM) or orthomosaic products is heavy time consuming and needs all the individual images to mosaic them. Also, it needs geometric [44], radiometric [45] and thematic [46] quality control. Commonly a Structure from Motion (SfM) processing of the images (alignment, dense point cloud generation, mosaicking of images, orthomosaic generation) is used with photogrammetric software, and Remote Sensing and Geographic Information System (RS&GIS) software to extract new information and to combine the drone acquired data with existing cartographic databases.

In our implementation, we used Mapir Survey 3 RGB sensor and Flir Vue Pro R 640 thermal sensor to acquire imagery over linear infrastructures. We used Metashape Photoscan photogrammetric software to perform the SfM processing, and for terrain modelling and land use land cover classification we used QGIS software.

The availability of linear infrastructure networks shapefiles in most of the official mapping agencies avoids the manual digitalization of a given road. Nevertheless, the operator must define the buffer distance from the road axis to delimit the affectation area to be monitored. Once the imagery is processed, georeferenced and clipped with the area of interest, the road manager can exploit and analyze the information (Figure 9).

The main applications of these mapping products, are:

(a)

Map update of constructed elements surrounding the linear infrastructure.

(b)

Detection of horizontal signing and concrete degradation.

(c)

Monitoring of conservation/maintenance works.

(d)

Locate wildlife paths or other thermal indicators of fauna activity.

4. Conclusions

This work provides a framework and system specification to use the currently deployed 4G telecommunication network to transmit real-time telemetry and video data from a UAS to a control ground station. Results are presented that show how data is being collected and exploited to improve traffic and road infrastructure monitoring. The on-board payload contains, apart from the drone, its flight controller (FC) and its positioning systems (GNSS), an extra microcomputer capable of reading telemetry data from the FC and video from a peripherical camera. A USB-modem and a SIM card connect the drone to the 4G network. The dedicated software allows the tunneling of such data though a Virtual Private Network to the public internet and towards the desired ground station. We have shown the logical schematic of such system and the main parameters needed for it to work, as well as experimental data carried out in a test site to prove the feasibility of the system. The on-board system combined with the 4G terrestrial antennas network have a limited capacity of performance, which was found to limit the upload stream to 9 Mbps and the download stream to 15 Mbps. Also, the 4G coverage is better when the drone is in the air than close to/on the ground, i.e., less than 400 ft (120 m) over the terrain, due to the 4G network antennas lobule shape and the absence of topographic shadows.

The complete 4G framework for drone telemetry and video is a solution for BVLOS operations that overpasses the classical radio-link range. Moreover, we foresee that this study that can be a base for an upcoming upgrade to 5G networks, since the challenges that cannot be reached with 4G technology in terms of data consumption and latency will be reached with the next communication paradigm. Data collection continues and further enhancements of the traffic monitoring tools are anticipated during the final year of the StratoTrans project [18].