Simulation of Scalability in Cloud-Based IoT Reactive Systems Leveraged on a WSAN Simulator and Cloud Computing Technologiesâ€
Round 1
Reviewer 1 Report
General Comments:
The study provides an approach for for the modeling and simulation of IoT systems and services in smart cities leveraged on a simulator of WSANs and cloud-based technologies. The objective of the approach is to facilitate the modeling of IoT systems for the design and experimental validation before system construction. The authors claim their approach is reusable. Two case studies are performed to demonstrate the approach.
Specific Comments:
(1) The paper would be improved if either the phrase "real cloud" was defined in the introduction or eschewed in favor of a different more descriptive term. I apologize for seeming pedantic but for most readers, even those with computing background, a "real cloud" will first be associated with a weather related "cloud in the sky". If the term is well established citing it and defining it will help avoid this issue. If it is not established a more descriptive term should be used instead.
(2) The paper would be improved if it clarified its use of the term "validation". The authors claim their "approach was validated through experiments with two use cases". From my reading it appears that the two use cases demonstrate how the approach is applied and can be used (in detail) to two relatively different use case problems. However, this is not a manifestation of "validation" when the term is used in the context of modeling and simulation. The standard definition of validation is concerned with determining whether
the conceptual simulation model (as opposed to the computer program) is an accurate representation of the system under study. Furthermore, it is usually discussed with respect to verification - determining that a simulation computer program performs as intended, i.e., debugging the computer program. Both of these definitions are established in several well-cited modeling and simulation sources.
Law, Averill M., W. David Kelton, and W. David Kelton. Simulation modeling and analysis. Vol. 3. New York: McGraw-Hill, 2000.
Balci, Osman. "Validation, verification, and testing techniques throughout the life cycle of a simulation study." Annals of operations research 53.1 (1994): 121-173.
Sargent, Robert G. "Verification and validation of simulation models." Journal of simulation 7.1 (2013): 12-24.
Gore, Ross. "Isolating the causes of emergent failures in computer software." Emergent Behavior in Complex Systems Engineering: A Modeling and Simulation Approach (2018): 263-284.
(3) In addition, the paper would be strengthened if in each of the use cases the authors were able to show which components of the simulation were most responsible for driving the behavior of the simulation. This could take the form of a sensitivity analysis or it could be even more robust and show to what extent conditions with respect to several variables are responsible.
Sensitivity analysis:
Kleijnen, Jack PC. "An overview of the design and analysis of simulation experiments for sensitivity analysis." European Journal of Operational Research 164.2 (2005): 287-300.
Technique for showing to what extent conditions with respect to several variables are responsible:
Gore, Ross J., Christopher J. Lynch, and Hamdi Kavak. "Applying statistical debugging for enhanced trace validation of agent-based models." Simulation 93.4 (2017): 273-284.
(4) The authors claim that their approach facilitates reuse of components. While this appears to have some merit based on their use cases, the paper would be improved if the claim was tempered. Researchers have shown how difficult it is to reuse components of a simulation due to the inability to document all assumptions that exist within a simulation related to how components are constructed and used. For example, Spiegel et al. showed that a group of 5 different PhDs and graduate students in computer science where unable to identify all the assumptions in an extremely simple falling body simulation. Including this limitation with respect to the claim of reuse would improve the paper by providing more context to the extent the claim is true.
(5) In the replication crisis error an anonymized version of the data provided in the paper, the scripts used for analysis, and the scripts used to create the figures for the paper need to be provided to both the reviewers and to the readership. This ensures completely transparent analysis and makes the authors paper significantly more impactful as other researchers can build off (and cite the paper).
(6) Figures 6 and 10 rely on the contrast of red vs. green. As many as 8% of men and 0.5% of women are affected with the common form of red-green color blindness that creates no contrast between the two colors. The Figure would be improved if a different color choice was made. There are numerous options that offer superior readability described here: https://www.somersault1824.com/tips-for-designing-scientific-figures-for-color-blind-readers/
Author Response
"Please see the attachment."
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper presents an interesting and promising approach to a contemporary topic. The introduction presents the issues raised in the paper and sets the expected and necessary connection of this paper with the previously published work of the authors. Furthermore, the prospects of new alternatives for simulating IoT systems are set.
The authors aim to achieve the research objectives through two use cases/case studies. However, in the first use case, it needs to be clarified what the homogeneous characteristics of the parks are and how the application is expected to change in the event that these features are modified, given that urban public open spaces present a wide range of features. Nevertheless, the analysis of the applications is considered satisfactory and well documented.
I would recommend that the authors further elaborate on the limitations of the research, as their presentation in the paper is very limited.
Author Response
"Please see the attachment."
Author Response File: Author Response.pdf
