Rule-Based Verification of Autonomous Unmanned Aerial Vehicles

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article presented a rule-based verification of autonomous UAVs. First, a comprehensive background covering topics including safety constraints and representation of rule-based knowledge is presented. Next, the acquisition of knowledge and formulation of rules is presented. The rules are specifically written using the rule language SWRL. With the theory presented, a traffic monitoring use-case is illustrated to explain the function of the rule-based verification. The article concludes with a discussion on limitations such as large knowledge bases on low-end hardware and future work such as predictive components to provide warnings before a rule violation.

Overall, the article was organized well and it was clear that a lot of thought had gone into the discussions. The authors provided a clear and strong argument as to why rule-based verification is necessary and proceeded to present a sound method for accomplishing that goal.

In the future, it would be interesting to see a real-world example of how the rule-based verification system might be implemented, e.g., through the use of QGroundControl and/or ROS.

Additionally, the rule-based architecture seems like it may lend itself well to a behavior tree or state machine. A discussion on how this system might be integrated into one of those architectures would prove to be interesting and helpful for practitioners.

Finally, similar to the above comment, a discussion on how one might integrate rules into controllers would also prove fruitful for practitioners. For example, one may introduce a maximum flight altitude in the cost function for a trajectory generator. 

Comments on the Quality of English Language

There were some very minor grammatical errors, but not sufficient to detract from the message of the article.

Author Response

Dear Sir or Madam,

We sincerely appreciate your thorough review of our paper and the insightful comments you provided. Your feedback has significantly contributed to the refinement and enhancement of our work. We would like to express our gratitude for your time and valuable suggestions.

We are pleased to inform you that our rule-based verification component has already undergone both simulation and real-life implementation. To provide a more comprehensive overview of this aspect, we have expanded Section 4, detailing the used System Architecture and Development Environment. Additionally, a video demonstration of the simulation has been uploaded to the GitHub-Repository for reference.

Your observation regarding the potential integration of our rule-based approach into a behavior tree is inspiring. We agree that a behavior tree could enhance performance and address certain limitations of our approach. However, transforming rules into a behavior tree involves an additional compilation process, which comes with challenges in maintaining the verification component, especially when new constraints are introduced, or existing constraints are amended. Therefore, we have included a paragraph in Section 6 to address and elaborate on this point.

We appreciate your suggestion regarding the integration of rules into controllers, specifically mentioning the introduction of a maximum flight altitude in the cost function for a trajectory generator. However, we would like to clarify that direct interaction with the UAV is beyond the scope of our current work. Our focus lies in presenting a generalistic approach to monitor behavior, that is easy to integrate. Introducing direct interaction would require a deep understanding of the UAVs system architecture, and thus potentially merging controlling and monitoring functions. However, we prefer a strict separation of these aspects, as our aim is to extend the approach not only to UAVs, but also to other mobile robots on land and at sea, which have a different appearance and architecture. We hope that the clarification within the paper will provide you with comfort.

The changes in the paper have been highlighted in yellow at the editor's request to make them easier to follow. We are confident that the conducted changes, based on your feedback, will contribute to the overall impact of our work.

Thank you for your time and consideration!

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript describes the role of automation in enhancing the capabilities of Unmanned Aerial Vehicles (UAVs) by allowing self-determining behaviors while reducing the need for extensive human involvement. Future concepts envision a single human operator commanding multiple autonomous UAVs with minimal supervision.

Good attempt.

This may be improved by presenting the formulation rules more clearly.

Case study videos may be given.

 

Author Response

Dear Sir or Madam,

We sincerely appreciate the time and effort you invested in reviewing our paper. Your constructive comments have been valuable in refining our work and we would like to thank you for your insightful feedback.

We have taken your feedback into account and made substantial improvements to enhance the clarity of the formulation of rules. Specifically, we have expanded Section 3 and Section 4. To facilitate understanding, each stage in the formulation process has been assigned a number, to be again referenced in Section 4. We believe this modification will improve the transparency and accessibility of our rule formulation approach.

In response to your suggestion, we have uploaded a video demonstration of our simulation to the GitHub – Repository (https://github.com/hsu-aut/RIVA_Safety/tree/main/Videos). This video is offering a visual representation of our approach in action. We hope that this addition provides a valuable supplement to the text, offering further insights into the practical implementation of our rule-based verification component.

The revisions within the paper have been highlighted in yellow at the editor's request to make them easier to follow. We hope that these revisions align with your expectations and further strengthen the overall quality of our work.

Thank you for your time and valuable insights!

Reviewer 3 Report

Comments and Suggestions for Authors

It is not completely clear how this method for rule-based runtime verification was implemented for UAVs. The block-diagram of this method use in UAVs should be given,  explanations (diagrams) of what software and how was implemented for using this rule-based verification method in UAVs .

Author Response

Dear Sir or Madam,

We would like to express our gratitude for your review of our paper and the valuable feedback you provided. Your insights have been valuable in refining our work, and we appreciate the time and effort you dedicated to the evaluation.

We acknowledge your concern regarding the way our rule-based runtime verification was implemented. In response to your feedback, we have made additions to the paper. Specifically, we have included a comprehensive description of the System Architecture and Development Environment in Section 4. For a better understanding, the use of ROS2, px4 and Gazebo in combination with container technology is illustrated in a diagram. In addition, a video, uploaded into the Github - Repository is offering a visual representation of our approach in action. We hope that this addition provides a valuable supplement to the text, offering further insights into the practical implementation of our rule-based verification component.

We hope these additions contribute to a more transparent and comprehensible presentation of our implementation.

We would like to thank you for your feedback. We hope that the improvements we have made will address your concerns and improve the overall quality of our paper.

Thank you for your time and valuable insights!

Reviewer 4 Report

Comments and Suggestions for Authors

Despite advances in automation, there remains a demand for a "human in command"  to assume overall responsibility, driven by concerns about UAV safety and regulatory compliance.  In response to these challenges, a method for runtime verification of UAVs using a knowledge-based system is introduced.   The only issue from my perspective view is there is little mathematical background in this paper, so how to make sure the proposed framework has better performance in a theoretical background? Meanwhile, how to guarantee the proposed approach can be reused is still in dought.

Comments on the Quality of English Language

none

Author Response

Dear Sir or Madam,

We sincerely appreciate your thoughtful evaluation of our paper and the comments you provided. Your feedback has been valuable in the refinement of our work.

We acknowledge your concern regarding the limited mathematical background in our paper and the call for assurance of better performance in a theoretical context. To address this, we have revised Section 5 to emphasize that our approach does not aim to outperform other existing approaches in terms of processing time. Instead, we favor using a knowledge-based system, requiring minimal expertise to the individual UAV. This facilitates reusability and adaptability across various UAVs. As the approach is intended to be a monitoring add-on without executing automated countermeasures, it can integrate with pre-existing safety components without generating conflicts.

We have taken careful account of your comment regarding the reusability of our approach. In response, we have strengthened the emphasis on reusability within Section 6. Here we explained the simplicity of updating and amending rules, emphasizing the flexibility of adapting to rule changes. This adaptability appears promising in the dynamic world of UAVs. Furthermore, we emphasize that only minimal knowledge of the architecture of the UAVs is required, as the reasoning is performed in the knowledge base, as demonstrated in Section 4. While acknowledging the current method of "hard" data reading from UAVs, we have highlighted the future potential for obtaining information via communication, making our approach even more generalist and applicable to various UAV architectures.

Furthermore, we have underlined the potential for certification of the rule base by aviation supervisory authorities, providing an additional layer of credibility and trustworthiness to our approach.

The revisions within the paper have been highlighted in yellow at the editor's request to make them easier to follow. We extend our gratitude for your thorough review. We hope that these revisions align with your expectations and enhance the overall quality and clarity of our paper.

Thank you for your time and valuable insights!

Reviewer 5 Report

Comments and Suggestions for Authors

This paper has its basics on Autonomous UAV by proposing a Rule-based Verification idea. In general, the paper carries an important research part and well-designed. However, in terms of quality and novelty, the paper is not in a scientific scope, but it can be further improved. Introduction section has no sufficient background information and details for the proposed idea and its advantages. Authors discusses a paper for traffic scenario but they implement a different scenario? why? What about a performance comparison with state-of-the-art approaches? There is no numerical performance result to assess its applicability. So, authors must do another effort to make this paper a suitable one for a high-quality journal. 

Comments on the Quality of English Language

Minor issues

Author Response

Dear Sir or Madam,

Thank you for your valuable feedback. We appreciate the thoughtful insights you provided, have carefully considered your comments and made revisions to address the concerns raised.

We have incorporated additional background information and details in Section 1 to better articulate the proposed idea and its advantages. The new content focuses on related safety approaches with a specific emphasis on data-driven methodologies.

In response to your observation concerning the discrepancy between the discussed paper on traffic scenarios and the implemented scenario, we want to clarify that our use-case is intentionally not focused on new aspects such as computer vision or statistical calculations of traffic data. Instead, we highlight that our primary focus is on ensuring that UAVs tasked with Traffic Monitoring align with regulatory constraints. This is demonstrated and elaborated upon in the revised text.

Section 5 has been revised to explicitly state that our approach is not designed to outperform other existing approaches in terms of processing time. Rather, we emphasize the use of a knowledge-based system, requiring minimal expertise for individual UAVs. This approach facilitates reusability and adaptability across various UAVs. Furthermore, the revised section underscores the compatibility of our approach with pre-existing safety components without generating conflicts. In response to the absence of numerical performance results, Section 5 has been expanded to include a reference regarding computing time and ontology size. We have highlighted the importance of maintaining a small and simple knowledge base. Additionally, Section 6 now emphasizes the need to investigate ways to enhance the method's efficiency, which we will do.

We believe these updates strengthen the overall quality of the paper. We look forward to any further feedback and appreciate the opportunity to improve our work.

Thank you for your time and valuable input!

Round 2

Reviewer 5 Report

Comments and Suggestions for Authors

Thanks for your effort in addressing my previous concerns. I hope that the paper is now in better form.

Comments on the Quality of English Language

A final reading is required to correct typos.