A Framework for Intelligent Decision Making in Networks of Heterogeneous Systems (UAV and Ground Robots) for Civil Applications ^†

Cyber–physical systems (CPSs) are connected embedded devices with computing power, networking ability, control, and decision capability. The networks connecting these devices are different from the Internet because they can sense their environment, share information, make decisions, and act based on local and global information. These capabilities enable CPSs to improve processes in the transportation, agriculture, healthcare, and mining industries, and surveillance. Remarkable achievements in the development of cost effective, reliable, smaller, networked, and more powerful systems have allowed us to build new control and communication mechanisms, as well as cooperative and coordinated motion planning algorithms to enable these devices to assist humans to cope with real-time problems [1]. In this paper, we propose a learning-based distributed framework for intelligent decision making in networks of heterogeneous systems, to optimally plan their activities in highly dynamic environments. We utilized the multi-agent deep reinforcement learning (MADRL) technique to develop control and coordination strategies for teams of UAVs and group ground-moving robots. The developed framework enabled the team of unmanned aerial vehicles (UAVs) to observe the defined region above the ground correctly and efficiently, and to share information with ground robots, to perform robust actions. Our main objective was to maximize the utilization of the strong abilities of each CPS device. UAVs can observe the environment from above and rapidly gather reliable information to share with rescue robots working on the ground, but they cannot perform rescue tasks on the ground; in contrast, rescue robots cannot gather reliable information due to the lack of visual limitation. In this framework, we trained several DQN agents to learn optimal control policies for a team of cooperative heterogeneous robots in a centralized fashion, then perform actions in a decentralized way. These learned polices were further transferred in real time to the robots and evaluated against the real-time deployment of robots to perform tasks in the environment.