Uncovering the Limitations and Insights of Packet Status Prediction Models in IEEE 802.15.4-Based Wireless Networks and Insights from Data Science

Round 1

Reviewer 1 Report

--rewrite the abstract to focus more on the proposed model

--rewrite the related work, and add updated related literature because the works discussed in the related work table were done between 2009 and 2014. It is mandatory to add recent work in your analysis as well.

--add a table on data description for the used dataset, the table might include all features/ parameters 

Revise content to improve overall writing quality, but especially focus on the abstract and introduction sections.

Author Response

Dear

Editor

Informatics

We are submitting the paper:

“Uncovering the Limitations and Insights of Packet Status Prediction Models in IEEE 802.15.4-Based Wireless Networks and Insights from Data Science”

Authored by: Mariana Ávalos-Arce, Daniel Heráclito Pérez-Díaz, Carolina Del-Valle-Soto *, and Ramon A. Briseño.

We would like to thank the reviewers and editors for their detailed analysis of the manuscript; the comments are very valuable to us. In the revised version of the paper, we have incorporated the all changes recommended by the reviewers.

Comments to all observations and suggestions including point-by-point responses are addressed in the following text.

Reviewer 1 comments

Comment 1: rewrite the abstract to focus more on the proposed model.

Response: Many thanks to the Reviewer for his/her invaluable interest in the comments on this manuscript. We have added information about the proposed model in the Abstract that clarifies the questions that the Reviewer properly raises.

Wireless networks play a pivotal role in various domains, including industrial automation, autonomous vehicles, robotics, and mobile sensor networks. This research investigates the critical issue of packet loss in modern wireless networks and aims to identify the conditions within a network's environment that lead to such losses. We propose a packet status prediction model for data packets that travels through a wireless network based on the IEEE 802.15.4 standard and exposed to 5 different types of interference in a controlled experimentation environment. The proposed model focuses on the packetization process and its impact on network robustness. The study explores the challenges posed by packet loss, particularly in the context of interference, and puts forth the hypothesis that specific environmental conditions are linked to packet loss occurrences. The contribution of this work lies in advancing our understanding of the conditions leading to packet loss in wireless networks. Data is retrieved with a single CC2531 USB Dongle Packet Sniffer, whose information on packets becomes the features of each packet from which the classifier model will gather the training data aiming to predict whether a packet will unsuccessfully arrive at its destination. We found that interference causes more packet loss than that caused by various devices using a WiFi communication protocol simultaneously. Besides, we found that the most important predictors are network strength and packet size; low network strength tends to lead to more packet loss, especially for larger packets. The study contributes to the ongoing efforts to predict and mitigate packet loss, emphasizing the need for adaptive models in dynamic wireless environments.

Comment 2: rewrite the related work, and add updated related literature because the works discussed in the related work table were done between 2009 and 2014. It is mandatory to add recent work in your analysis as well.

Response: Many thanks to the Reviewer. The Reviewer is right, We have rewritten the Related Work section and significantly updated the references, as suggested by the Reviewer.

Comment 3: add a table on data description for the used dataset, the table might include all features/ parameters.

Response: Thanks to Reviewer. we have included the following table to describe all the features of the dataset.

Thank you very much.

Sincerely,

Author Response File: Author Response.docx

Reviewer 2 Report

The paper proposes a packet status prediction model for data packets in a wireless network based on the IEEE 802.15.4 standard, focusing on controlled experimentation in various types of interference.

The paper suggests that interference causes more packet loss than simultaneous communication by various devices using WiFi protocols. This finding, however, is presented without a clear indication of its significance or novelty.

The study identifies network strength and packet size as crucial predictors, indicating that low network strength tends to result in more packet loss, especially for larger packets. This observation, while potentially valuable, is somewhat expected and lacks depth in its analysis.

The findings are trivial, and the paper's contribution is not clearly articulated. The hypothesis is deemed straightforward — stating the presence of conditions causing packet loss in a network environment.

There is confusion in the terminology between the tested IEEE 802.15.4 standard and the reference to WiFi communication protocols, which are typically associated with the 802.11 standards. Clarification in this area is required for a more precise understanding.

Applying artificial intelligence to classification is not an apparent innovation; the authors should delve deeper into this. The suggestion is to dedicate a more substantial portion of the paper to feature engineering, shedding light on how this enhances the understanding of packet loss.

The experimental nature of the paper is acknowledged, but there is a lack of clear insights derived from the experiments. The paper needs to provide a robust analysis of its experimental results, leaving the reader questioning the significance of the presented findings.

Therefore, the paper is criticized for presenting findings perceived as expected or common rather than groundbreaking. There is a call for a more in-depth exploration of the role of AI in classification, more precise articulation of the paper's contribution, and a need for precise terminology to avoid confusion. The experimental approach is acknowledged but needs to deliver insightful outcomes.

The paper is well-written, but the authors should consider proofreading it to avoid minor typos.

Author Response

Dear

Editor

Informatics

We are submitting the paper:

“Uncovering the Limitations and Insights of Packet Status Prediction Models in IEEE 802.15.4-Based Wireless Networks and Insights from Data Science”

Authored by: Mariana Ávalos-Arce, Daniel Heráclito Pérez-Díaz, Carolina Del-Valle-Soto *, and Ramon A. Briseño.

We would like to thank the reviewers and editors for their detailed analysis of the manuscript; the comments are very valuable to us. In the revised version of the paper, we have incorporated the all changes recommended by the reviewers.

Comments to all observations and suggestions including point-by-point responses are addressed in the following text.

Reviewer 2 comments

Comment 1: The paper proposes a packet status prediction model for data packets in a wireless network based on the IEEE 802.15.4 standard, focusing on controlled experimentation in various types of interference.

The paper suggests that interference causes more packet loss than simultaneous communication by various devices using WiFi protocols. This finding, however, is presented without a clear indication of its significance or novelty.

The study identifies network strength and packet size as crucial predictors, indicating that low network strength tends to result in more packet loss, especially for larger packets. This observation, while potentially valuable, is somewhat expected and lacks depth in its analysis.

The findings are trivial, and the paper's contribution is not clearly articulated. The hypothesis is deemed straightforward — stating the presence of conditions causing packet loss in a network environment.

Response: The Reviewer is correct and we have strengthened the contribution to highlight our work in comparison to existing literature:

The experimentation involved various controlled-environment tests, each representing different interference scenarios, including a baseline test with no interference, interference from neighboring networks, radio-frequency interference, unrestricted device usage, wind interference, and wireless interference. The study captured and analyzed packet data, considering factors such as frame type, length, RSSI, and time.

To determine the dominant machine learning algorithm, the research applied several classification algorithms, including Naive Bayes, SVM, Neural Network, Logistic Regression, Random Forest, Gradient Boosting, and Decision Tree. The models were trained and evaluated based on precision, recall, F1 score, and confusion matrices, with a focus on predicting packet loss (Error class) in the presence of WiFi interference.

The results indicate that the Gradient Boosting and Random Forest models achieved the best performance, with precision rates of 97% and 96%, respectively. However, due to the dataset's imbalance, additional metrics such as F1 score and recall for the Error class were considered. The confusion matrices provided insights into the models' tendencies to predict both correct and incorrect outcomes.

Comment 2: There is confusion in the terminology between the tested IEEE 802.15.4 standard and the reference to WiFi communication protocols, which are typically associated with the 802.11 standards. Clarification in this area is required for a more precise understanding.

Response: Thanks to Reviewer.The Reviewer is correct, and the ideas were unclear. We have added a clarifying paragraph to the Introduction to better elucidate the relationship between concepts, accompanied by the appropriate reference.

IEEE 802.15.4 serves as a crucial standard delineating the physical layer and medium access control for low-rate wireless personal area networks (LR-WPANs). This standard plays a foundational role in shaping the ZigBee specification, aiming to provide a comprehensive solution for networks by constructing upper-level protocol stack layers not covered by the standard. The primary objective of IEEE 802.15.4 is to define basic network layers catering to a specific type of wireless personal area network (WPAN), facilitating communication among ubiquitous, cost-effective, and low-speed devices. Emphasizing economical communication with nearby nodes and minimal to no infrastructure, the standard prioritizes energy efficiency. In its fundamental form, it envisions a communication range of 10 meters with a transfer rate of 250 kbps. Multiple physical layers are defined to accommodate varying requirements, initially introducing alternative rates of 20 and 40 kbps, and the current version incorporates an additional rate of 100 kbps. Lower rates can be achieved, further reducing energy consumption. Notably, the key feature of 802.15.4 within WPANs is the attainment of exceptionally low manufacturing costs through technological simplicity without sacrificing generality or adaptability.

Comment 3: Applying artificial intelligence to classification is not an apparent innovation; the authors should delve deeper into this. The suggestion is to dedicate a more substantial portion of the paper to feature engineering, shedding light on how this enhances the understanding of packet loss.

Response: The Reviewer is correct. We have expanded our analysis by incorporating the calculation of important factors into the Random Forest and Gradient Boosting models to strengthen our results and discussions. The Figure 5 has been updated to reflect the new analysis, and the corresponding information has been added to the following paragraphs in the Results and Discussions section:

Finally, we observed the Random Forest trees formed in the training process looking for patterns, we calculated the correlation of the predictors with the target variable with 6 different coefficients (Info. gain, Gain ratio, Gini, chi-squirt, relief, and, FCBF) using the Rank orange widget and also we calculate and interpreted the important features for the Gradient Boosting and Random Forests models with the feature importance Orange widget. In the decision trees created by the Random Forest algorithm, as well as in the correlation coefficients and the important features of the Gradient Boosting model, the most important variable for predicting packet loss is RSSI, followed by LENGTH. Also, In decision trees formed by the Random Forest Model, it can be observed that packet loss decisions are oriented toward when the network is perceived as weak and the length of the packets is large. That can mean that the use of different devices using the same WiFi network tends to cause interference between them and in some occasions weaken the network

On the other hand, the time it takes for a packet to be perceived as lost or received, and the type of AKC frame, despite not being among the variables with the highest correlation with the target variable, are important variables for the prediction of Gradient Boosting and Random Forest models. It seems logical to think that the longer it takes for a packet to be perceived as received or lost, the more likely it is that no node will receive the packet. Similarly, the type of AKC frame has the highest presence in the network; it could be that the preparation interval for communication between devices is the period of loss of this type of frame.

Comment 4: The experimental nature of the paper is acknowledged, but there is a lack of clear insights derived from the experiments. The paper needs to provide a robust analysis of its experimental results, leaving the reader questioning the significance of the presented findings.

Therefore, the paper is criticized for presenting findings perceived as expected or common rather than groundbreaking. There is a call for a more in-depth exploration of the role of AI in classification, more precise articulation of the paper's contribution, and a need for precise terminology to avoid confusion. The experimental approach is acknowledged but needs to deliver insightful outcomes.

Response: The Reviewer is correct. Many thanks to the Reviewer.We understand the concern of the Reviewer, and it is very accurate. The article provides an essential overview in the analysis of interference in small-scale wireless networkWe have substantially enhanced the presentation of the results and the model to be more formal in its description throughout the manuscript. The Reviewer can now see a more organized model based on AI classification and comparative related works. The application of AI for packet interference detection and collision analysis in small-scale wireless networks holds significant importance in optimizing network performance and reliability. In the realm of modern wireless communication, where small-scale networks are prevalent, the accurate identification of interference and potential collisions is paramount for maintaining seamless and efficient data transmission. This AI-driven analysis contributes by providing a sophisticated and automated means to discern interference patterns, predict potential collisions, and enhance the overall robustness of wireless networks. By leveraging AI techniques, the study not only addresses the challenges posed by packet interference but also represents a progressive step towards developing intelligent solutions for improving the reliability and performance of small-scale wireless communication systems.

Thank you very much.

Sincerely,

Author Response File: Author Response.docx

Reviewer 3 Report

Good work with practical considerations. I would suggest that authors clarify better their differenciation from the other related works, and therefore highlight better their contribution. Moreover, if we scale the network from 5 nodes to 50 nodes and more, does the dominant ML algorithm remains the same with high prediction rate or not? 

Author Response

Dear

Editor

Informatics

We are submitting the paper:

“Uncovering the Limitations and Insights of Packet Status Prediction Models in IEEE 802.15.4-Based Wireless Networks and Insights from Data Science”

Authored by: Mariana Ávalos-Arce, Daniel Heráclito Pérez-Díaz, Carolina Del-Valle-Soto *, and Ramon A. Briseño.

We would like to thank the reviewers and editors for their detailed analysis of the manuscript; the comments are very valuable to us. In the revised version of the paper, we have incorporated the all changes recommended by the reviewers.

Comments to all observations and suggestions including point-by-point responses are addressed in the following text.

Reviewer 3 comments

Comment 1: Good work with practical considerations. I would suggest that authors clarify better their differenciation from the other related works, and therefore highlight better their contribution. Moreover, if we scale the network from 5 nodes to 50 nodes and more, does the dominant ML algorithm remains the same with high prediction rate or not?

Response: Thank you very much to the Reviewer for your kind words and insightful question. First and foremost, we have redesigned the Related Work section to provide greater clarity regarding the paper's contribution and to update the references more comprehensively. It is true that the model is designed for a small number of sensors, as the intention was to comprehend the device scheme in an average household. To address the question of whether the dominant machine learning algorithm remains the same with a high prediction rate when scaling the network from 5 nodes to 50 nodes and more, one must conduct further experiments and analysis. Similarly, for future work, we are considering extrapolating the current model with a larger number of sensors to analyze its effectiveness further. The provided research focuses on the critical role of packetization in wireless networks and the challenges posed by packet loss, particularly in the context of interference. The hypothesis of the study suggests that specific conditions within a network's environment are linked to packet loss occurrences.

In fact, while the research successfully addressed the challenges of packet loss in wireless networks, further experiments specifically scaling the network and evaluating the performance of different machine learning algorithms in larger setups would be necessary to answer the question about algorithm dominance in larger networks.

Additionally, we have strengthened the contribution to highlight our work in comparison to existing literature:

The experimentation involved various controlled-environment tests, each representing different interference scenarios, including a baseline test with no interference, interference from neighboring networks, radio-frequency interference, unrestricted device usage, wind interference, and wireless interference. The study captured and analyzed packet data, considering factors such as frame type, length, RSSI, and time.

To determine the dominant machine learning algorithm, the research applied several classification algorithms, including Naive Bayes, SVM, Neural Network, Logistic Regression, Random Forest, Gradient Boosting, and Decision Tree. The models were trained and evaluated based on precision, recall, F1 score, and confusion matrices, with a focus on predicting packet loss (Error class) in the presence of WiFi interference.

The results indicate that the Gradient Boosting and Random Forest models achieved the best performance, with precision rates of 97% and 96%, respectively. However, due to the dataset's imbalance, additional metrics such as F1 score and recall for the Error class were considered. The confusion matrices provided insights into the models' tendencies to predict both correct and incorrect outcomes.

Thank you very much.

Sincerely,

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

The authors adequately addressed my previous concerns in the review rounds, improving the paper significantly for publication. 

The only remaining issue is proofreading to enhance language quality.

Author Response

We have reviewed the English throughout the manuscript and have enhanced its quality.