The Effect of Real-World Interference on CNN Feature Extraction and Machine Learning Classification of Unmanned Aerial Systems
Round 1
Reviewer 1 Report
This paper is good and acceptable quality.
Author Response
Thank you for your support.
Reviewer 2 Report
I have found the paper interesting and relatively well-written. However, the following comments need to be addressed before it will be ready for possible publication:
1) In the abstract, line 8, it should be "multiple UASs";
2) I think that the review of the existing literature about UASs detection and countermeasure techniques provided in the Introduction is quite succinct and misses important references. In the last years, there have been several important European projects that dealt with the problem of UAVs detection using different kind of sensors (RF, video-based, acoustics), prominent examples being Aladdin (https://aladdin2020.eu) and Safeshore (http://safeshore.eu), just to mention a few. Moreover, several novel RF-based techniques appeared in the literature to tackle the problem of UAS detection and classification. In this respect, the following (quite recent) references could be added to provide a potential reader with a pointer to such a literature, offering at the same time a more complete overview on the topic:
- A.; Schumann et al, Drone vs. Bird Detection: Deep Learning Algorithms and Results from a Grand Challenge. Sensors 2021, 21, 2824.
- Shi, Z. et al, An Acoustic-Based Surveillance System for Amateur Drones Detection and Localization. IEEE Trans. Veh. Technol. 2020, 69, 2731–2739.
- A. Coluccia et al, Detection and Classification of Multirotor Drones in Radar Sensor Networks: A Review. Sensors 2020, 20, 4172.
but authors are encouraged to add also other references they may be aware of.
3) Authors devoted almost all the space (actually, 12 figures) to show and briefly compare the time spectrogram and PSD contents of the different UASs under different flying conditions, while the far more interesting part related to the classification task is limited only to the very short Section 2.3. The paper is quite unbalanced in this respect. I could understand the focus on the preliminary part where the data collection including real-time interferences takes place, but also the algorithmic part is important as well and in its current form it is very underrated. Please add a critical mass of contents describing in more details the process of transfer learning and the classification approaches, highlighting the various choice of the design parameters.
4) As a final comment, it would be useful to explicitly report the expressions of the Accuracy and F1-score metrics. Although they are well-known in the object detection literature, they deserve to be introduced for the sake of completeness.
Author Response
Many thanks for thorough and helpful review comments. We have addressed the points as follows:
1) Line 8 - Changed to multiple UASs
2) Added in following information regarding European projects –
A number of different ways of detecting and classifying UASs are being researched and implemented today, including a number of significant European Projects. The European ALADDIN project integrates RADAR, acoustic and optical sensors to perform detection, classification and localisation with a command and control node. The project boasts an integrated system which counters malicious UAS \cite{aladdin2020}. Another project called SafeShore aims to detect UAS launching from boats, boats and people in the sea. The project seeks to achieve this using 3 dimensional LIDAR integrated with acoustic, radio and imagery data \cite{safeshore2016}. While current research and development programs such as SafeShore and ALADDIN look to combine and integrate various types of detection and classification methods, each individual method is still advancing with current research.
Added in following extra references –
A.; Schumann et al, Drone vs. Bird Detection: Deep Learning Algorithms and Results from a Grand Challenge. Sensors 2021, 21, 2824.
Coluccia et. Al \cite{coluccia2021} consider the results of the 2020 Drones v. Birds competition and in particular consider the top 3 performing algorithms. The observed the biggest challenges to be detection at a distance and moving cameras, recommending that training and test data be expanded to include this. Further, discrimination between birds and UAS gets considerably harder with distance.
Shi, Z. et al, An Acoustic-Based Surveillance System for Amateur Drones Detection and Localization. IEEE Trans. Veh. Technol. 2020, 69, 2731–2739.
Shi et al. \cite{shi2020ab} design a detection and localisation system using audio signals and an array of microphones. The system works in real time and used TDOA and a Bayesian filter for localisation.
- A. Coluccia et al, Detection and Classification of Multirotor Drones in Radar Sensor Networks: A Review. Sensors 2020, 20, 4172.
Coluccia et al. \cite{col2021} provide a comprehensive review of radar systems for UAS detection and classification. They consider the main challenges relating to detection, verification and classification. They also focus on the latest technology with respect to frequency modulated continuous wave (FMCW) radar and spatially distributed sensors.
Passafiume, N. Rojhani, G. Collodi, and A. Cidronali, “Modeling Small UAV Micro-Doppler Signature UsingMillimeter-Wave FMCW Radar,” MDPI Electron., 2021, doi: 10.3390/electronics10060747.
Passafiume et al. \cite{passa2021} investigate the reliability of FMCW radar images for classifying a UAS by the amount of motors it has and they rule out the rotation speed from effecting classification.
3) Updated subsection 2.3 to include information on VGG-16, LR and kNN. Alterations are as follows:
\subsection{Classification}
Once the datasets of spectrogram and PSD graph images were created and save as 224 x 224 pixels, a pre-trained VGG16 CNN on ImageNet \cite{Krizhevsky222} was used as a feature extractor. Through a process of transfer learning neural networks trained for one purpose can be used for an another purpose. ImageNet is an object detection database consisting of millions of images. Using a pre-trained VGG-16 we are able to utilise the process of transfer learning. This means we do not need to train our weights from scratch, a procedure which needs a massive amount of data and time. Table \ref{table:4} shows the structure of a VGG-16 CNN.
\begin{table}[H]
\caption{VGG-16 CNN} % title of Table
\centering % used for centering table
\begin{tabular*}{\linewidth}{@{\extracolsep{\fill}}p{0.3\linewidth}p{0.3\linewidth} p{0.2\linewidth}@{}}
\hline\hline %inserts double horizontal lines
Layer Type & Size & Feature Map \\ [0.5ex] % inserts table
%heading
\hline % inserts single horizontal line
Input Image & 224 x 224 x 3 & 1 \\
2x Convolutional & 224 x 224 x 64 & 64 \\
Max Pooling & 112 x 112 x 64 & 64 \\
2x Convolutional & 112 x 112 x 128 & 128 \\
Max Pooling & 56 x 56 x 128 & 128 \\
2x Convolutional & 56 x 56 x 256 & 256 \\
Max Pooling & 28 x 28 x 256 & 256 \\
3x Convolutional & 28 x 28 x 512 & 512 \\
Max Pooling & 14 x 14 x 512 & 512v\\
3x Convolutional & 14 x 14 x 512 & 512 \\
Max Pooling & 7 x 7 x 512 & 512v\\ [1ex] % [1ex] adds vertical space
\hline %inserts single line
\end{tabular*}
\label{table:4} % is used to refer this table in the text
\end{table}
At the end of the last pooling layer we are left with 25088 features. These features are then used to train machine learning classifiers Logistic Regression (LR) and k Nearest Neighbour (kNN). LR works but fitting an ‘S’ shaped curved, otherwise known as a Sigmoid function, to our features. The equation representing the sigmoid function can be seen in Equation (\ref{eq:1}).
\begin{equation}
\begin{split}
& sigmoid(x)=\ \ 1/(1+\ e^(-x)\ ) \\
\label{eq:1}
\end{split}
\end{equation}
LR estimates discrete value based on a set of independent variables. It estimates the probability of occurrence by measuring the relationships between one (or more) of those variables and a categorical dependant variable. Limited-memory Broyden–Fletcher–Goldfarb–Shanno was chosen as the solver for our experiments as it deals with multiclass due to its ability to handle multinomial loss \cite{hale2021}.
kNN on the other hand uses the dataset to find the closet point to the input point. The classifier then works by using a majority vote of neighbours. The Minkowski distance was utlisied as part of our experiments and can be calculated as show in Equation (\ref{eq:2}).
\begin{equation}
\begin{split}
& sum(|x - y|^p)^(1/p) \\
\label{eq:2}
\end{split}
\end{equation}
The k-nearest neighbour of the particular data point is then found and assigned to the class which has the highest probability.
\begin{equation}
\begin{split}
& P\left(y=j\middle|\ X=x\right)=\frac{1}{K}\ \sum_{i\epsilonA}{I(y^{\left(i\right)}=j)}\\
\label{eq:3}
\end{split}
\end{equation}
Equation (\ref{eq:3}) shows the probability of an input x being assigned to the class which has the highest probability \cite{pand2021}. For each of the models, 5-fold cross validation was used to indicate whether the model was overfitting. Hyper-parameters regularisation for LR and number of neighbours for kNN were optimised using 3 fold nested cross validation. The same process is followed for classification as described in previous work and more information can be found in the references \cite{swinney2020, swinneywoods2021}.
4) Updated subsection 2.3 to include information on the metrics accuracy and F1-score. Additional information included as follows:
Accuracy and F1-Score were used to evaluate the models performance. To understand both metrics we first must understand that a True Positive is when the algorithm predicted for example that a Phantom 4 was present and it was there in the data. A True Negative is where the prediction was no Phantom 4 and the prediction was correct or true. A False Positive is where the algorithm think the Phantom 4 was there but it wasn’t and lastly the False Negative predicts it wasn’t there but actually it was. Accuracy and F1-score can now be defined. Accuracy is show in Equation~(\ref{eq:4}).
\begin{equation}
\begin{aligned}
Accuracy\ =\ \frac{True Positive\ +\ True Negative}{True Positive\ +\ True Negative\ +\ False Positive\ +\ False Negative}
\label{eq:4}
\end{aligned}
\end{equation}
F1-Score includes the metrics precision and recall. Equation~(\ref{eq:5}) shows the formula for precision showing how many positive predictions were correct.
\begin{equation}
\begin{aligned}
Precision\ =\ \frac{True Positive}{True Positive\ +\ False Positive\}
\label{eq:5}
\end{aligned}
\end{equation}
Equation~(\ref{eq:6}) shows the equation for recall and considers the correctly predicted positives as a fraction.
\begin{equation}
\begin{aligned}
Recall\ =\ \frac{True Positive}{True Positive\ +\ False Negative\}
\label{eq:6}
\end{aligned}
\end{equation}
F1-Score can then be calculated as seen in Equation~(\ref{eq:7}).
\begin{equation}
\begin{aligned}
F1\ Score\ =\ \frac{2\ (Precision\ *\ Recall)}{Precision\ +\ Recall}
\label{eq:7}
\end{aligned}
\end{equation}
Round 2
Reviewer 2 Report
Authors correctly addressed all my comments.
