A Delay-Based Machine Learning Model for DMA Attack Mitigation^†

Round 1

Reviewer 1 Report

The proposed authentication technique looks promising to prevent DMA attacks even though the success rate is not high. I have several questions that need to be answered in the text regarding the usage of ML/DL methods: - How did you choose the DL parameters? Especially, 5 Convolutional layers look very high for the experiments. - Why do you think CNNs perform worse than other models? - Can you compare the inference timings of the models when one data sample is given to the model? As far as I understand, the training will be done offline and the training time is not a big issue. However, the authentication time will depend on the inference time of the models. - What are the contributions of this paper compared to [1]? This should be clarified in the text. - There are many spelling mistakes in the text. They should be corrected before publication.

Author Response

Dear Reviewer,

We appreciate your valuable reviews and acknowledge that the sections of the paper are updated accordingly to incorporate changes suggested and highlighted the changes within the manuscript using track changes.

- How did you choose the DL parameters? Especially, 5 Convolutional layers look very high for the experiments.

Response: Dear Reviewer, the DL parameters specifically for the convolutional neural network were chosen applying experiment data comparing different combinations using time and accuracy metrics. Starting with the shallowest configuration with 2 layers and going deeper to 7 layers and choosing the best fit model in comparison with time required to train as well as the accuracy. The analysis is included in the updated manuscript in section 5.2)

 - Why do you think CNNs perform worse than other models?

 Response: Dear Reviewer, The CNN performs worse than the other models tested because it is a deep layered network, where the scheme tries to model the differences or variations in the input data. In this case, as we know there is a big overlap in the input data for the classification, the model has a hard time modelling those variations and specifically taking only one feature into account. (The analysis is included in the updated manuscript in section 5.2)

 - Can you compare the inference timings of the models when one data sample is given to the model? As far as I understand, the training will be done offline, and the training time is not a big issue. However, the authentication time will depend on the inference time of the models.

Response: Dear Reviewer, yes, the training is done offline, and we have updated the paper to include the details of the inference timings when one data sample is given to the model. The authentication time will depend on the inference time of the models as the prediction from the model will then be used to further authenticate the device. (The analysis is included in the updated manuscript in section 6)

 - What are the contributions of this paper compared to [1]? This should be clarified in the text.

Response: The manuscript is updated with additional details on DMA attacks and ROI based Machine learning techniques are proposed and compared for the authentication process based on the neural network, decision trees, random forest, and multi-player perceptron. We got real-time inference times for all the models and the training time for the model is significantly less.

- There are many spelling mistakes in the text. They should be corrected before publication.

Response: Dear Reviewer the manuscript is proofread, the spelling and grammatical mistakes are fixed in the updated manuscript.

Reviewer 2 Report

The paper proposes a technique for registration and authentication of PCIe devices that have DMA based on the profiling time of the aforementioned devices. Different datasets with profiling times for 3 devices are collected and evaluated. When using DROI and CROI the devices are clearly distinguishable whereas when machine learning techniques are used, the accuracy of the classifier becomes approximately 40%.

First of all, I like the idea of using the time, which seems to be influenced by the manufacturing process, to identify devices. However, there are many things that are not completely clear to me and should be addressed to clarify the paper and the idea, and to make it consistent. For example, in the abstract it is mentioned that 3 machine learning models are used, but they are four. Later on, a detailed explanation (maybe even too deep for the purpose of this paper) is given for two of them (4.2) and the others are just mentioned. The introduction should be clearer, the most interesting contribution of the paper (in my opinion) i.e. the possibility to create an authentication scheme based on device profiling is not really introduced.

It is not clear to me how the profiling phase is carried out. How the collected samples are distributed is clear, but whether if the profile is carried out using certain memory locations of the host or if these are fixed (i.e. all the memory is probed or not), or if the host only collects timing measurements from the device. Note that how this profiled is executed, has a noticeable impact on the results. Also, authors mention that this timing could change with time and temperature, but this is not evaluated.

There are many details that are missing for the experiments using machine learning. What is the size of the convolutional network and the neural network inputs? One timing sample? Why these configurations for the networks have been chosen?  Moreover, and judging from the results obtained for the combination of DROI and CROI, it seems that the “order” or the location of the data used as input is really important, otherwise for me it is hard to understand why the samples that were clearly distinguishable are no longer distinguishable (40% of accuracy).

It is not fair to just use the construction and authentication time of the device to measure the speed of the proposal, when it requires several minutes to collect the samples needed for these processes.

Author Response

Dear Reviewer,

We appreciate your valuable reviews and acknowledge that the sections of the paper are updated accordingly to incorporate changes suggested and highlighted the changes within the manuscript using track changes.

In the abstract it is mentioned that 3 machine learning models are used, but they are four.

Dear Reviewer, we have updated the abstract to reflect that four ML models are studied.

A detailed explanation (maybe even too deep for the purpose of this paper) is given for two of them (4.2) and the others are just mentioned.

Response: Dear Reviewer, we have updated the Section 4.2 and revised the details of two techniques and have added the details of all four techniques.

The introduction should be clearer, the most interesting contribution of the paper (in my opinion) i.e. the possibility to create an authentication scheme based on device profiling is not really introduced.

Response: Dear reviewer, we updated the introduction of the paper and introduce the delay-based authentication scheme in the updated manuscript.

It is not clear to me how the profiling phase is carried out. How the collected samples are distributed is clear, but whether if the profile is carried out using certain memory locations of the host or if these are fixed (i.e. all the memory is probed or not), or if the host only collects timing measurements from the device. Note that how this profiled is executed, has a noticeable impact on the results.

Response: The profile time are the response time delays of the device connected to the host machine. The host machine reads the configuration parameters that are specific memory locations fixed to store device and other configuration information. The profile time does not probe all the memory for the profile time, as that will cause longer time to collect raw data and have additional overheads.

Also, authors mention that this timing could change with time and temperature, but this is not evaluated.

Response: Dear Reviewer, the aging is not studied in the paper, however the authors are studying the aging impact and plan a follow up research on the environmental variations and aging impacts on the authentication process. The conclusion is updated in the manuscript.

There are many details that are missing for the experiments using machine learning. What is the size of the convolutional network and the neural network inputs? One timing sample? Why these configurations for the networks have been chosen?  

Response: The size of the convolutional network and the neural network inputs of the best fit model is 5 convolutional layers, with each layer having 64 neurons as the neural network inputs. This configuration has been chosen based on experimental results comparing different combinations using time and accuracy metrics. Experiments included shallowest configuration with 2 layers and going deeper to 7 layers and choose the best fit model based on time required for training and model accuracy. (The analysis is included in the updated manuscript in section 5.2 and 6 to include details of inference times of models to predict device ID for one timing sample)

One timing sample can be reported using the testing samples however the accuracy is not accounted for just one timing sample, so the authors have removed the timing information for a single timing sample.

Moreover, and judging from the results obtained for the combination of DROI and CROI, it seems that the “order” or the location of the data used as input is really important, otherwise for me it is hard to understand why the samples that were clearly distinguishable are no longer distinguishable (40% of accuracy).

Response: Dear Reviewer, for the machine learning based scheme, the raw datasets are used that are not preprocessed unlike CROI and DROI. The dataset is split into training and testing sets randomly and does not maintain an order. The random assignment to the two datasets avoids the bias in the prediction model resulting in skewed outputs and preventing the model from overfitting.

It is not fair to just use the construction and authentication time of the device to measure the speed of the proposal, when it requires several minutes to collect the samples needed for these processes.

Response: Dear Reviewer, authors acknowledge that authentication process will include the data collection. The delay associated with the data collection is the same for the ML based scheme that is 10,000 samples per 3.4 mins, however, the measurement delay can be reduced for the ML based authentication by using fewer data samples for classification. The paper section 6 is updated in the manuscript to discuss the delays associated to the data collection are included in the authentication.

Reviewer 3 Report

The authors must cross check the similarity with the previous publication (DOI: 10.1109/ISQED51717.2021.9424262).

The authors should add more sentences about how the manuscript is closely related to "Cryptography".
(There is no other comment since it is well-written and well-organized.)

Author Response

Dear Reviewer,

We appreciate your valuable reviews and acknowledge that the sections of the paper are updated accordingly to incorporate changes suggested and highlighted the changes within the manuscript using track changes.

The authors must cross check the similarity with the previous publication (DOI: 10.1109/ISQED51717.2021.9424262).

Response: Dear reviewer, the sections that were similar to the existing publication are revised. we have also updated the manuscript and mention that the manuscript is an extended version of the conference paper.

The authors should add more sentences about how the manuscript is closely related to "Cryptography". (There is no other comment since it is well-written and well-organized.)

Response: The journal is submitted to the special issue on Special Issue "Cybersecurity, Cryptography, and Machine Learning", where we explore the machine learning based authentication scheme for the DMA resilient countermeasure.

Round 2

Reviewer 3 Report

The manuscript has been revised based on the previous reviewers' comments.

Thus, I recommend the manuscript for publication.