Performance Testing of istSOS under High Load Scenarios
Round 1
Reviewer 1 Report
The paper discuses performance of istSOS software implementation of OGC open standard for management and querying spatio-temporal data. It states clearly the relevance of such implementation and the objectives of the work presented in this manuscript. Discussions and conclusions sections wrap up nicely the results presented in section 3, giving a complete picture for the context and aim of this work. Some of the discussion though feel like afterthoughts, e.g. performance metrics that are relevant for this study, INSIPRE requirements for network services chosen as reference for istSOS performance. Both, performance metrics and reference seem better placed at the Methodology section. This would pave the way to the Results section to benchmark the performance and give more prominence to the metrics used for evaluation.
The following has more detailed comments; line number, table of figure number are used for reference to text in the manuscript.
Fig.2-Fig.6: add accronyms of SOS requests in graphs title for easy connection/reference to the corresponding tables.
Fig.2 & 3: it is easier to compare response times in top and bottom graphs if y-axis range is kept the same.
If you use log-scale for the y-axis it will be easier to see the difference in response for the lower number of concurrent users 100 - 500.
Tables, 5 and further: add units of measure, e.g. req/sec, ...
Tables 6 - 9: good to have for reference but many numbers to absorb and conclude over. A visual presentation might help, e.g. a map with % completion in x-axis and number of concurrent users in y-axis, per request type IO, GO ... and interface m, g.
L365-379: The comparison of results from mod_wsgi and gevent in Fig.3 and related tables cannot be done from only response time. Seems the % failure affects the total response time from concurrent requests sent.
Fig.7: y-axis label should be changed for the graphs.
Author Response
We thanks the reviewer for the comments and we hope to have addressed all the inputs accordingly.
In the revised manuscript We have addressed the suggestions of reviewer 1 using the the red font color to better highlight the changes. Here below they are described point by point.
Reviewer comment 1:
Some of the discussion though feel like afterthoughts, e.g. performance metrics that are relevant for this study, INSIPRE requirements for network services chosen as reference for istSOS performance. Both, performance metrics and reference seem better placed at the Methodology section. This would pave the way to the Results section to benchmark the performance and give more prominence to the metrics used for evaluation.
Author's answer:
We added a new section (2.4.Reference for the evaluation of the results) in the Methodology chapter that presents the reference that we're going to use to evaluate the experiment results. In this way performance metrics and reference are introduced in the methodology and then used in the evaluation. (see L315-L338)
Reviewer comment 2:
Fig.2-Fig.6: add accronyms of SOS requests in graphs title for easy connection/reference to the corresponding tables.
Author's answer:
Acronyms of the requests has been added to the images title as suggested. It is easier now to connect images with data/tables.
Reviewer comment 3:
Fig.2 & 3: it is easier to compare response times in top and bottom graphs if y-axis range is kept the same.
Author's answer:
As suggested y-axis of all the graphs from the same case test are now kept consistent so that visual comparison among requests is simpler and clearer.
Reviewer comment 4:
If you use log-scale for the y-axis it will be easier to see the difference in response for the lower number of concurrent users 100 - 500.
Author's answer:
As suggested the bar plots are now represented in logaritmic scale in the y-axis. This permits to read the differences with low response time. On the contrary it is more difficult to quickly compare the difference in response time with increasing users, but the y-axis labels show this difference anyway. We agree that this solution leads to a better interpretation of results.
Reviewer comment 5:
Tables, 5 and further: add units of measure, e.g. req/sec, ...
Author's answer:
Figure 5 has been corrected introducing the right units of measure that is seconds vs number of concurrent users.
Reviewer comment 6:
Tables 6 - 9: good to have for reference but many numbers to absorb and conclude over. A visual presentation might help, e.g. a map with % completion in x-axis and number of concurrent users in y-axis, per request type IO, GO ... and interface m, g.
Author's answer:
In a first version we used heat maps to produce colored grid based on response time. Nevertheless, this make in our opinion more difficult to read actual vaules. if this is considered important we can certainly introduce additional images or replace the tables with heat maps.
Reviewer comment 7:
L365-379: The comparison of results from mod_wsgi and gevent in Fig.3 and related tables cannot be done from only response time. Seems the % failure affects the total response time from concurrent requests sent.
Author's answer:
This is very relevant and we gratefully thank the reviewer. We have now considered only tests with no failure to evaluate the two WSGI services. We have changed the descriprion of results accordingly in each of each case study results. (L378-383; 420-423; 439-443)
Reviewer comment 8:
Fig.7: y-axis label should be changed for the graphs.
Author's answer:
The y-axis label has been corrected in MB and %.
Reviewer 2 Report
A further description of the istSOS system is required, with proper additional figures, because the readers are new.
The last sentence in the abstract ' When the number of concurrent users increases the service response times quickly
degrade and became unsatisfactory in most modern usages.' -> I think it's too general a conclusion to be expected without experimentation.
It needs to justify if the specifications applied to the experiment among the numerous and complicated OGC standard specifications are reasonable.
The results of the experiment will be affected by the hardware configuration for testing experiments, but what is the author's opinion?
Two different WSGI server applications need more detail explanation and direct comparison.
In performance testing, inter-related parameter control is a very important factor to interpret the result. In other words, a variable can affect another variable. Authors' opinion in this study?
Figure 2, 3, 5, 6, 7, and 8 is not clear to read. And (a), (b), (c), and (d) numbering, and separate explanations or descriptions need.
Overall, the readability is too low.
Author Response
We thanks the reviewer for the interesting comments that we believe helped us in improving the paper.
In the revised document modifications that refer to reviewer 2 comments are in pink font color so that can be easily identified. Here below the comments are addressed point-by-point:
Reviewer comment 1:
A further description of the istSOS system is required, with proper additional figures, because the readers are new.
Authors response:
A new image composite has been added. Additionally the following sentences have been included in the istSOS section (1.4): "In particular using the Web interface, it is possible to create a new instance and configure it by: setting the database connection; defining the service provider and identification metadata; specifying the coordinate system used for data storage and those accepted for reprojection. It is also possible to define the maximum data-period length accepted with a single request; to create and organize the offerings (groups of sensors); to register new sensors to the network; and define virtual procedures as on-the-fly data processes. Finally, the administrator can define dictionaries of accepted observed properties, units of measures and data quality levels. From the same interface, it is possible to access the data management section that permits to manipulate the data by applying a time-series-calculator or manually adjust the values and the data quality indexes. A data viewer interface is available seamlessly to explore the data in space and time, analysing the different observed properties and plotting for the selected period one or two properties from more sensors." L146-157
Reviewer comment 2:
The last sentence in the abstract ' When the number of concurrent users increases the service response times quickly degrade and became unsatisfactory in most modern usages.' -> I think it's too general a conclusion to be expected without experimentation.
Authors response:
We understood and changed the sentece with "When the number of concurrent users increases to 1000 and 2000 only the 80% of the responses are below the limit the service response times of 30 seconds, performance unsatisfactory in most modern usages"
Reviewer comment 3:
It needs to justify if the specifications applied to the experiment among the numerous and complicated OGC standard specifications are reasonable.
Authors response:
We are sorry but we are not sure to fully understand this comment. We kindly the reviewer to eventually specify if there is a need to specify if the standard is reasonable to be used or if the parameters set for the experiment (hardware and configurations) are reasonable. However, as discussed in the introduction OGC has two "competing" standards to make sensor data interoperable: SOS and SensorThings API. In the section 1.3 Motivation of Research and Outline we clearly defined our target (istSOS) and motivations.
Reviewer comment 4:
The results of the experiment will be affected by the hardware configuration for testing experiments, but what is the author's opinion?
Authors response:
In the discussion we added the following sentence: "Since we recognize that, in general, hardware configuration affects the results in load testing experiments we tried to minimize this impact by avoiding the use of virtual systems, using a high-speed band and running the requests from a physically different machine. While we cannot state that the hardware did not affected the results we can safely assume that it doesn’t play a crucial role in this tests. In fact the recorded hardware usage showed that the system was adequately dimensioned with the RAM that were never fully used and the CPU that only in the case C with 1000 and 2000 users was close (but below) to 100%". See lines 609-615
Reviewer comment 5:
Two different WSGI server applications need more detail explanation and direct comparison.
Authors response:
We have added a new section "3.5 WSGI servers performance" (Lines 508-530) in the result section with two new figures. Figure 10 shows differences in response time between the two servers. In the discussion we have also added an additional section on this topic: "Analysing figure 10, we can see that in general gevent and mod_wsgi, the two WSGI server we have tested, are equivalent for low concurrency. However, for the getObservations request gevent is faster and presents the higher performance differences with respect to the mod_wsgi. On the contrary, in case B (larger number of observations) with 500 concurrent users the performance of mod_wsgi outperformed gevent in all the other requests (DS, GC and IO). A possible motivation for this behaviour could reside in a better capacity of gevent, that is based on co-routines, in handling I/O blocking operations like the GO request, while mod_wsgi using a multi-processes and multi-thread approach execute faster requests like GC, IO and DS without interferences of I/O blocking requests." Lines 616-623.
Reviewer comment 6:
In performance testing, inter-related parameter control is a very important factor to interpret the result. In other words, a variable can affect another variable. Authors' opinion in this study?
Authors response:
This is correct. To address this fact in the paper we have introduced a comment to the new figure 11 in the discussion: "The three varying parameters of tests are: number of sensors, number of stored observations and number of concurrent users. From the experiments it is clear that the increase in size of the observations and of the sensors, from case A (105 million of observations and 20 sensors) to case B (1.7 bilions of observations and 130 sensors), produce slower responses regardless of the WSGI server used. Similarly, the increase of size in the sensors and reduction of observations, from case A (20 sensors and 105 million of observations) and case C (5100 sensors and 122 thousand observations) produced slower response time. This indicates that both the increase of stored historical observations and the number of deployed sensors negatively affect the istSOS performance. While it is not possible to separate the negative contribution on performance of the two components it is clear that the number of stored data has a greater negative impact than the number of sensors. In fact, from case A to case C with respect to case A to case B sensors increase of 50 times while observations diminish of 500 time and performance degrades of 5 times." L624-635
Reviewer comment 7:
Figure 2, 3, 5, 6, 7, and 8 is not clear to read. And (a), (b), (c), and (d) numbering, and separate explanations or descriptions need.
Authors response:
The Figures has been adjusted to show logarithmic y-scale so that low response time differences can be appreciated to increase its readability. In the title of each plot we added the request acronym to better link the request it refers with tables. Letters of each plot have been added and referred in the description.
Round 2
Reviewer 2 Report
Properly revised.
