**4. Discussion**

This chapter summarises and discusses the insights gained from the testing of the prototype system and proposes further development directions. There has been extensive testing involving the SITL simulator, and it was confirmed that the robustness of the system and the communications layer has been increased significantly. Real hardware tests also showed that the second version provides a superior performance. The two performance criteria used in the case of prototype testing were qualitative evaluations of system functionality (which is effectively safety for the vehicles) and the robustness of the system. Regarding robustness, the first implemented method required resetting the system and users when non-desirable behaviour occurred; this involves returning the vehicles to the starting positions, restarting the managemen<sup>t</sup> algorithm and disconnecting and reconnecting the communication systems (telemetry radios) of the vehicles. This can up to 1–5 min per reset. Note that the average endurance for drones is in the order of 30 min, so even a few resets take up significant portion of their available flight time. Recalling drones to replace batteries during the tests or demonstration would add an undesirable amount of waiting time, and in extreme cases could prevent the successful completion of all scenarios during a test day. Due to the persistent data storage and differing conflict resolution algorithm, the second iteration does not require resetting; furthermore, in the case of communication loss, it automatically attempts to reconnect to the users. In terms of the other criteria, the system functionality, the first implemented version of the stopand-go algorithm showed a number of mismanaged conflicts, e.g., the lower priority vehicle did not stop at a discovered conflict, or first stopped and continued its mission. Using the second algorithm eliminated these system errors, and in the scenarios tested, it provided appropriate resolutions for all conflicts. Quantitative evaluation of system safety, managemen<sup>t</sup> efficiency (time lost due to conflict management) and robustness is being carried out in the current, follow-up stage of the research project.

The current state of the research project was also presented at a live demo conducted in the ZalaZone Proving Grounds in Hungary. The demo was aimed at demonstrating the core principles of the conflict managemen<sup>t</sup> system. Unfortunately, during the demo, the allocated space was rather limited, and only a single intersection of roads was made available, so the scope of the demonstration was also limited. Refer to Figure 11 for photographs taken at the event. In the demo, the UAV has received a repeating mission to

fly between the two sides of the intersection. The ground vehicle had the PX4-in-a-box unit mounted to the roof and was operated by a driver. A manned ground vehicle seemed like the appropriate choice for such limited space, as the intention was to demonstrate that the system can react to a wide variety of situations that can arise. Relying on a driver means that the driver can also improvise in the situations and demonstrate the safe workings of the system. The demo was successfully presented, and while the system's performance was excellent, some areas for further improvement were identified.

**Figure 11.** ZalaZone demo setup: ground control station and system users (**left**), system in operation (**right**).

Among the identified areas to improve was the aforementioned case, when head-on conflicts occur. In this case, the stopped vehicle needs to perform an evasive manoeuvre before stopping to ensure that separation is maintained between the users. To do that safely, in addition to the system users' positions, information about the environment is also required. As such the next important step is the integration of a GIS system.

Another direction is the closer integration of DJI products. The Windows SDK was successfully tested, and basic automatic operation was demonstrated. There are significant limitations in this SDK version, and based on the lack of updates it seems that it might be on the road to obsolescence. As such the Mobile SDK (Android and iOS) is under consideration. Mobile development can also unlock new opportunities, as it would enable smartphones to join the system as users. However, mobile development is a separate niche of programming, and the costs and benefits need to be weighed before effort is invested in building up this technology skillset in the research group.

As the framework's performance is acceptable, the next primary direction of the research group is to develop the conflict managemen<sup>t</sup> algorithms. For the conflict detection algorithm, the most important thing is to include information from vehicle trajectories (mission plans), if available. Knowing the trajectories can potentially help to resolve some detected conflict situations more efficiently.

Regarding efficiency, another key development direction is a method to objectively evaluate the system's safety, performance and efficiency measures. While intuitively, a simple resolution algorithm such as stop-and-go seems to be less efficient than more advanced ones, this might be based on the inconvenience experienced as a human driver when the vehicle needs to be stopped or the route changed. For an autonomous system, however, convenience is not a factor that matters. Unless specific criteria are defined, it is impossible to compare methods objectively. Once the measurement method is developed, SITL simulations can be used to objectively compare detection and resolution methods. Furthermore, optimisation tools can be used to determine the optimum parameter settings for a specific type of algorithm.

In the follow-up studies since the completion of the first stage of the project the development of objective quantitative performance criteria is one of the key points of research. The following list highlights the criteria that are being used in the development for the system:

•Safety measures:

	- - Time and distance travelled by individual users without other users or conflicts;
	- - Time and distance travelled by individual users while other users are present, and the conflict is being managed by the system;
	- - Highest level of accelerations and decelerations commanded by the system;
	- - Total change in altitude commanded by the system (only applicable to UAVs);
	- - Time and distance spent off-road or parked (UGV);
	- - Time spent landed, if commanded (UAV).

When evaluating the performance criteria, the following approach is being developed:


The three-stage process described enables the objective evaluation of the performance measures, as it essentially compares the theoretical possible unrestricted operation of all users to the actual performance they can achieve while multiple users are around, and the conflict is managed. The best managemen<sup>t</sup> solutions provide the highest level of safety, so no impacts or near impacts and detect all conflicts, while at the same time being the least intrusive, so issuing a minimum number of resolution commands and imposing minimum additional time and distance (detours). Quantitatively evaluating these measures enables the objective comparison and development of different conflict managemen<sup>t</sup> methods and strategies, which is the ultimate aim of the research framework presented here.

## **5. Conclusions**

In conclusion, in this research a system framework was developed and its functionality successfully demonstrated. The framework implements short- and mid-term tactical conflict managemen<sup>t</sup> for a wide user base, including unmanned aerial and ground vehicles. The framework has gone through two iterations, the second of which significantly improved key issues that were identified during development and testing of the first. The final, second version of the framework achieves a technology and solution independent implementation of the stop-and-go conflict managemen<sup>t</sup> algorithm. This system is capable of handling a wide range of unmanned and aerial ground vehicles without introducing any modifications

to the communications or control systems of any particular vehicle. The framework is implemented as a modular suite of software algorithms, with simulation software integration. The framework is accessible as a web service, where an arbitrary number of users and supervisors can connect to the system using standard web browsers from various devices to monitor activities and perform administrative tasks. The system is designed at the moment to handle up to 250 users (vehicles). The web service framework relies natively on database persistent information storage, which improves the robustness and recovery characteristics of the system, including reconnecting communication systems when the connection is interrupted. The stop-and-go algorithm was also improved to consider system dynamics and head-on conflict cases.

The framework's performance is considered acceptable, and future research focus is based on conflict managemen<sup>t</sup> methods development and evaluation.

**Author Contributions:** Conceptualization, D.R. and D.S.; methodology, D.S.; software, D.S.; validation, D.R. and D.S.; formal analysis, D.S.; investigation, D.R. and D.S.; resources, D.R. and D.S.; data curation, D.S.; writing—original draft preparation, D.S.; writing—review and editing, D.S.; visualization, D.S.; supervision, D.S.; project administration, D.R. and D.S.; funding acquisition, D.R. and D.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** The research reported in this paper and carried out at the Budapest University Technology and Economics was supported by the National Research Development and Innovation Fund (TKP2020 National Challenges Subprogram, Grant No. BME-NC) based on the charter of bolster issued by the National Research Development and Innovation Office under the auspices of the Ministry for Innovation and Technology. In addition the research was supported by the National Research, Development and Innovation Office through the project "National Laboratory for Autonomous Systems" under Grant NKFIH-869/2020. Funder NKFIH Grant Nos.: TKP2020 BME-NC and NKFIH-869/2020.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Acknowledgments:** The authors would like to acknowledge Gábor Jandó and Károly Veres for their contribution to the extensive testing and operation of the system during demo events.

**Conflicts of Interest:** The authors declare no conflict of interest.
