Analysis of Distance and Environmental Impact on UAV Acoustic Detection

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The study investigates drone acoustic detection at various distances using machine learning models like YAMNet and others. It tests two methods: one with unattenuated signals, effectively detecting drones up to 200 meters, and another with attenuated signals, extending detection up to 500 meters. This research highlights the advancement in drone detection technology over different ranges.However, there are numerous areas in the article that require improvement：

1. The abstract of the article is too general and reads more like an overview than a summary of the paper. It is suggested to revise the abstract to highlight the main points of the article, focusing on its unique contributions and specific findings.

2. In the keywords section, is it necessary to include both "UAV" and "drone"? Do these terms represent different experimental characteristics in your study? If not, it is recommended to consolidate them into a single term for clarity and consistency.

3. In the Introduction, there appears to be a lack of a clear research gap. The current framing gives the impression that your study was conducted primarily because it is a popular area of inquiry among scholars. It would be beneficial to first summarize the limitations in existing research and then articulate the significance of your study, thereby establishing a more compelling rationale for your research.

4. Please review the formatting issue in the caption of part a of Figure 1 in your document.

5. In the database section, could you clarify how you define "sounds that could give rise to false alarms in real situations"? The data you have used appears to lack context-specific scenarios, which diminishes its persuasiveness. It would be beneficial to provide more detailed information on how these sounds were selected and the rationale behind their inclusion to strengthen the relevance and credibility of your study.

6.It is recommended to increase the number of distance nodes in Figure 8 and provide an analysis. Aligning the number of distance nodes with those in Table 2 could enhance the persuasiveness of your findings and add depth to this section of your study.

7.Regarding the conclusion of your paper, it's not sufficient to merely state the performance improvements of the two methods you proposed. You should also specify the specific working environments where each method is most applicable. Clarify the practical problems your proposed methods can solve, providing a clear context for their real-world relevance and utility.

Comments on the Quality of English Language

 Minor editing of English language required

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper is about a study that investigates the performance of drone acoustic detection systems in relation to distance. The study carried out by the researchers employs various learning machines with different complexities, ranging from simple linear discriminants to deep neural networks like YAMNet, which utilizes transfer learning. Other machine learning systems considered by researchers include multilayer perceptron, support vector machines, and random forest.

The evaluation is conducted using a carefully selected database comprising diverse drone and interference sounds. These sounds have been previously processed by array signal processing and are influenced by ambient noise. Two training strategies are explored: the first involves training with unattenuated signals, preserving information from the sound sources, and testing with attenuation at different distances. This approach achieves effective detection at distances up to 200 meters, particularly with linear discriminant methods. Interferences that impede detection are taken into account.

In the second strategy, the systems are trained and tested with signals attenuated as a function of distance. This results in an increased effective detection range, reaching up to 300 meters for most methods and up to 500 meters for the YAMNet-based detector. Additionally, this approach opens the possibility of developing specialized detectors for specific distance ranges, significantly extending the effective detection range. This work concludes by highlighting the promising potential of acoustic drone detection at various distances, encouraging further exploration in this research area.

The work seems relevant, especially in the context of emerging technologies and concerns related to drone detection and security. The study addresses the performance of drone acoustic detection systems, considering various learning machines and their effectiveness at different distances. The use of machine learning techniques, including deep neural networks like YAMNet, adds a contemporary and sophisticated dimension to the research.

Drone detection is a pertinent topic, given the increasing use of drones in various fields and the potential security and privacy implications associated with their deployment. Understanding the performance of detection systems at different distances, considering factors such as interference and ambient noise, is crucial for developing effective countermeasures.

The exploration of different training strategies and their impact on detection range is an interesting aspect of the study. The idea of developing specialized detectors for specific distance ranges to extend the effective detection range is also noteworthy.

 

Comments to improve the article:

1)  While the introduction discusses the necessity for drone detection systems, it might benefit from a more explicit statement regarding why acoustic detection, in particular, is a valuable avenue of research. Clearly stating the motivation can further emphasize the significance of the chosen approach.

2)  In section 2 Signal characterization, the explanation of the peaks and valleys in the spectrum envelope is somewhat concise. Consider providing a bit more detail or clarification, especially for readers less familiar with signal processing terminology. This can improve the overall understanding of the frequency domain characteristics.

3)  While an example is provided with the Hobbyking FPV250 drone, it could be beneficial to mention briefly that different drone types may exhibit variations in their sound characteristics. Acknowledging this adds completeness to the discussion.

4)  Consider concluding the section 2 with an anticipatory statement that sets the stage for the next section. This can help guide the reader and maintain a logical progression in the paper.

5)  In the third section, there's a slight inconsistency in terminology usage, such as "trainins set" instead of "training set." Ensure consistent and accurate terminology throughout the section for clarity.

6)  While the section 3 mentions the variation of drone detection probability as a function of distance, it could explicitly state how this variation is incorporated into the training and testing of the learning machines. A clearer connection between the distance factor and the learning process would enhance understanding.

7)  Also for the third section, consider including a visual representation of an ROC curve to supplement the explanation. This can aid readers in conceptualizing the relationship between PD and PFA.

8)  Section 3 could benefit from a clearer delineation of the different learning machines. Consider using subsections or a more structured format to enhance the organization and flow of information.

9)  For the section four, while the section mentions the total duration of audio files and the balance between drone and non-drone classes, providing specific statistics, such as the number of audio files, duration per class, and characteristics of false alarm sounds, is necesary and would enrich the reader's understanding.

10)  While the section 4 mentions the variation of performance measure using Million Floating Point Operations Per Second (MFLOPS), a more detailed explanation or a reference to the methodology behind this choice would add depth to the understanding.

11) Ensure consistency in the citation style (e.g., [29][31] in the "Preprocessing for Feature Extraction" subsection). Use a consistent format throughout the section for clarity.

12)  In section five, the discussion focuses on qualitative comparisons, such as which detectors "perform well" or are "sensitive to distance." Including quantitative metrics, such as the difference in AUC values or statistical significance, would add precision to the comparisons.

13)  The term "sensitivity to distance" is used, but the specific criteria or features contributing to this sensitivity are not detailed. Providing insights into why certain detectors are more sensitive to distance than others would enrich the analysis.

14) Continuing with section 5, while the performance of detectors is discussed in different distance scenarios, a more explicit discussion of the limitations of each detector and the specific challenges faced at longer distances would enhance the depth of the analysis.

15)  Ending with section 5, the impact of training with a dataset including sounds at all distances is discussed, but the size or specifics of this dataset are not provided. Including information on the diversity and size of the dataset would offer a clearer understanding of the training conditions.

16) Regarding section 6 Conclusion, while the conclusion provides a qualitative summary of the findings, incorporating some quantitative metrics or summary statistics could enhance the impact. For instance, highlighting the improvement percentages with the inclusion of distance-diverse training sets could add clarity.

17)  The acknowledgment of the need for enormous amounts of data for deep networks like YAMNet is insightful. However, a brief explanation or reference to potential strategies for addressing this data challenge would provide additional context.

18) The mention of developing models capable of handling distance variations with high precision is crucial. However, explicitly connecting this capability to real-world scenarios or applications could strengthen the argument and highlight the practical relevance.

 

19)  In summary, the conclusion effectively encapsulates the study's contributions and outlines promising avenues for future research. Adding some quantitative details, addressing potential biases, and connecting distance handling capabilities to real-world applications would further strengthen the conclusion.

 

Comments on the Quality of English Language

Regarding writing in the English language:

1)   Some sentences are quite complex, which might slightly hinder readability. Consider breaking down complex sentences into shorter ones to enhance clarity. This is especially important in academic writing, ensuring that readers can follow the logic easily.

Example:

• Complex: "To analyze the performance of the system as a function of distance, a model has been used to estimate the attenuation at a specific distance from the source, as a function of frequency."
• Simplified: "To analyze performance concerning distance, we used a model to estimate frequency-dependent attenuation at specific distances from the source."

2)  Ensure consistency in verb tenses, especially in the Materials and Methods section. For instance, there is a mix of past and present tenses in the description of the database.

3)  In some instances, there is a bit of wordiness that could be streamlined for conciseness. This involves removing redundant phrases or using more concise expressions without sacrificing clarity.

Example:

• Wordy: "To evaluate the detectors performance, ROC curves were obtained."
• Concise: "ROC curves were used to evaluate detector performance."

3)  Adding more transition phrases between sentences and paragraphs could improve the overall flow of the text. This helps guide the reader through the logical progression of ideas.

Example:

• Improved Transition: "Moreover, audio segments with a duration of 0.98 seconds and a sampling rate of 16 kHz were chosen, following the guidelines of the YAMNet documentation in Matlab."

4)  Ensure consistency in numerical representation. For instance, the use of both numerals (e.g., 1 meter) and spelled-out numbers (e.g., one meter) can be confusing. Stick to a consistent style, preferably using numerals for scientific writing.

5)  A final proofreading pass is recommended to catch any minor grammatical or typographical errors.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

I think it's a timely study because there are a lot of issues being raised about drones. The authors used machine learning techniques to predict the distance between the receiver and the drone, and it would be great to see more studies using this approach in the future. 

 

I have a few comments for the authors.

 

1. In chapter 4.1., you will need to provide more details about the data used in your experiment.

 

2. LD had the best results without attenuation, but YAMNet had the best results with attenuation. It would be nice to see a comment on why. 

 

3. Can this system also be used to predict the direction in which a drone is located (or traveling)? 

Author Response

"Please see the attachment."

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

This study investigates the performance of an unmanned aerial vehicle (UAV) sound detection system based on machine learning models of varying complexity, ranging from simple linear discriminant analysis to the deep neural network YAMNet, at different distances. The evaluation utilizes a diverse UAV and interference sound database processed through array signal processing. Two training strategies are employed: the first involves training with undecayed signals and testing considering distance decay, while the second uses distance-decayed signals for both training and testing. The results indicate an effective detection range of up to 500 meters, particularly for the YAMNet detector, showcasing the potential application of acoustic detection technology at varying distances. Overall, the paper demonstrates innovation and depth in methodology and data processing, especially in considering the impact of sound characteristics and environmental factors on UAV sound detection. However, there are areas for further improvement in the following aspects:

1. Figure 1(a) is not fully displayed, and the vertical axis is recommended to include units. Figure titles in Figures 1 and 2 follow one distance style, while the remaining figure titles use another; it is advisable to ensure consistency. Font sizes in Figure 5 should be standardized.

2. The description of the dataset seems indirect. It is suggested to present categories and quantities in tabular form. Additionally, the advantages and differences between datasets could be supplemented with textual explanations for better reader understanding of experimental data.

3. While the experimental aspects are well-executed, there may be room for improvement in theoretical and technical depth. Strengthening the discussion of existing research and theoretical foundations of UAV sound detection technology in the background section is recommended.

4. The discussion on the feasibility and potential value of experimental results in practical applications may be insufficient. It is suggested to enhance the discussion on the practical application prospects of these technologies in real-world scenarios.

5. Future work could be added to the conclusion section for a more comprehensive closing statement.

Comments on the Quality of English Language

To further enhance the quality of this paper, it is recommended to undertake minor revisions to improve the clarity and precision of the English language used.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 5 Report

Comments and Suggestions for Authors

This paper evaluates the acoustic detection performance of drones at different distances by extracting relevant features in the frequency domain and applying machine learning methods, together with the transfer learning technique using the YAMNet network. The simulation results demonstrate the distance variation has a significant impact on the detection results.

Overall, the ideas are innovative and the English is better expressed, but there are still a few minor aspects that need to be improved.

Firstly, the innovations of the paper would be better written in separate sections so that they can be found more clearly, similar to the format of line 65.

Secondly, in part 4.1, line 302,  the database used is updated, is it authoritative? Please give more detailed reasons for the update.

Finally, the parameters and the setup of the experiment could be more specific before the experiment starts, it contains the setup of the experimental parameters as well as the performance of the hardware used.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

In the first round of review, you addressed all seven of the concerns I raised with reasonable modifications or responses. I have no further suggestions or comments regarding your manuscript.