Cloud-Native Observability: The Many-Faceted Benefits of Structured and Unified Logging—A Multi-Case Study
Round 1
Reviewer 1 Report
The author presents a unified logging library for Python, a unified logging architecture, and their use in concrete (cloud-native) use cases. How to handle observability of cloud-native system is definitely a timely, relevant, and hot problem, as witnessed by the white/grey literature on the topic. Also, the author elaborates on the presented use cases, discussing the faced issues and the corresponding lessons learned.
The manuscript is very well-written and easy to follow. I personally enjoyed reading it, and I believe it deserves publication. At the same time, a few aspects should be addressed by the author before the manuscript is ready for publication. I am listing them hereafter.
* * Threats to Validity * *
The potential threats to the validity of the author’s study are presented at a quite high-level. I would recommend the author to enter more in the details of such potential threats, perhaps following Wohlin’s taxonomy of potential threats of validity. For instance, which threats affect the “internal”, “external”, “construct”, … validity of the study? This would help readers better understanding them and appreciating how the author applied countermeasure to nullify/mitigate their effects.
* * Relation with author’s previous work * *
The author cites various research works he formerly authored, which seem to be quite related to the present manuscript. I would recommend the author to better clarify the relation wrt such works, as well as to indicate whether the present work builds on top/extends any of them. If the latter is the case, it would be worthy to also indicate how the present work extends the author’s previous work, e.g., by explicitly indicating which is the novel contribution presented in this manuscript.
* * Related Work * *
The three pillars of metrics, traces, and logs have also been tackled in a recently published survey on anomaly detection and root cause analysis (http://dx.doi.org/10.1145/3501297), where they were used to distinguish among the existing research solutions themselves. Perhaps, it would be worthy that the author retakes such a distinction and the challenges discussed in the survey to better position/differentiate his solution from those already existing. Also, the above mentioned survey shows that there are recent (academic) studies dealing with observability in modern service-based applications. Therefore, the initial statement of the related work, and the related work itself, should be adapted to cover the survey and the studies classified therein.
* * Other minor comments * *
The author may consider consistently using the Oxford comma (https://www.grammarly.com/blog/what-is-the-oxford-comma-and-why-do-people-care-so-much-about-it), to further improve the readability of his manuscript
L101: “The often-used tool combination … is often used for” -> Too many “often used”
L138: “use cases insights” -> “use cases, insights” (missing comma?)
L341-342: “an institutes infrastructure” -> “an institute’s infrastructure” (?)
L498: “cause this” -> “because this” or “since this”
L504: “or above” -> “or below” (according to the syslog standard, the lower is the logging level, the higher is the criticality of the logged event)
L599: “straight-forward” -> “straightforward” (no dash?)
Author Response
I would like to thank for your time and valuable comments.
Changed or added sections are marked in blue.
Please find below my point-to-point answers and remarks.
---
Threats to Validity
The potential threats to the validity of the author’s study are presented at a quite high-level. I would recommend the author to enter more in the details of such potential threats, perhaps following Wohlin’s taxonomy of potential threats of validity. For instance, which threats affect the “internal”, “external”, “construct”, … validity of the study? This would help readers better understanding them and appreciating how the author applied countermeasure to nullify/mitigate their effects.
- The threat to validity section now addresses the "Wohlin" threats more explicitly. Large parts of this section have been rewritten according to the recommendations.
Relation with author’s previous work
The author cites various research works he formerly authored, which seem to be quite related to the present manuscript. I would recommend the author to better clarify the relation wrt such works, as well as to indicate whether the present work builds on top/extends any of them. If the latter is the case, it would be worthy to also indicate how the present work extends the author’s previous work, e.g., by explicitly indicating which is the novel contribution presented in this manuscript.
- An explanation has been added in Section 2 ("However, this previous research was not primarily concerned with observability or instrumentation per se. Especially, no instrumentation libraries have been developed, like it was done in this research.")
Related Work
The three pillars of metrics, traces, and logs have also been tackled in a recently published survey on anomaly detection and root cause analysis (http://dx.doi.org/10.1145/3501297), where they were used to distinguish among the existing research solutions themselves. Perhaps, it would be worthy that the author retakes such a distinction and the challenges discussed in the survey to better position/differentiate his solution from those already existing. Also, the above mentioned survey shows that there are recent (academic) studies dealing with observability in modern service-based applications. Therefore, the initial statement of the related work, and the related work itself, should be adapted to cover the survey and the studies classified therein.
- Thank you for this welcome and recent survey. The complete related work section has been rewritten, considering comments from your and further reviews.
The author may consider consistently using the Oxford comma (https://www.grammarly.com/blog/what-is-the-oxford-comma-and-why-do-people-care-so-much-about-it), to improve the readability of his manuscript further
- A thorough Grammarly check has been done to address this point. Grammarly considers the Oxford comma (as far as I know).
L101: “The often-used tool combination … is often used for” -> Too many “often used”
- Addressed, please have a look
L138: “use cases insights” -> “use cases, insights” (missing comma?)
- Addressed, please have a look
L341-342: “an institutes infrastructure” -> “an institute’s infrastructure” (?)
- Addressed, please have a look
L498: “cause this” -> “because this” or “since this”
- Addressed, please have a look
L504: “or above” -> “or below” (according to the syslog standard, the lower is the logging level, the higher is the criticality of the logged event)
- Addressed, please have a look
L599: “straight-forward” -> “straightforward” (no dash?)
- Addressed, please have a look
Reviewer 2 Report
The paper is aimed at the derivation of time-series from data collected in the cloud, showing that this allows the observation of the distributed system status. The title is very promising but little specific. In essence, it gives a number of examples of time series (“many faceted benefits”) in data processing based in time series (“structured and unified logging”) and demonstrate that existing tools can be used in all these different ways (“a case study”). Unfortunately, the discourse is not anchored on practical experimentation.
The manuscript presents a good story with high-educational value. The presentation is medium-sloppy. Abbreviations are not defined before the first use in the text (for instance ‘DevOps’ on line 40). An occasional illustration depicts the obvious case (for instance ‘Figure 14’). Though the topic is on ‘a case study’, there are 4 (line 596) cases mentioned.
The meaning of references in academic texts is to bring the reader to either related information or background material. If the links are carefully selected, each will broaden the scope of a single item in the discourse. The appearance of multiple links does not clarify what the issue is. A single link ought to be enough, or the items are not sufficient assembled. Furthermore, products do not give rise for discussion. Hence, information on the product as named can be given in a footnote.
The composition of the text lacks somewhat rigidity. It could outline strictly the structure of the concept (ingredients for a time series, composition of a time series, interpretation / prediction of a time series). Hence it misses the theoretical foundation, explaining why proper tooling can be used for any problem with the same foundation.
The presented text is not at all places equally smooth in writing, but overall it will be inspiring for a large audience. A last round of critical reading with the reader in mind will be helpful. I understand that the author has the urge to tell the readership that no new tooling is required. However, he should clearly state that the reason is the flexibility in the structured report writing for the internal logs. I.o.w. it is a typical effect of standardization.
Author Response
I would like to thank for your time and valuable comments.
Changed or added sections are marked in blue.
Please find below my point-to-point answers and remarks.
---
The manuscript presents a good story with high-educational value. The presentation is medium-sloppy. Abbreviations are not defined before the first use in the text (for instance ‘DevOps’ on line 40). Though the topic is on ‘a case study’, there are 4 (line 596) cases mentioned.
- Abbreviations are now defined before the first use in text (please have a look)
- The title has been slightly changed - "... a MULTI-case study"
The meaning of references in academic texts is to bring the reader to either related information or background material. If the links are carefully selected, each will broaden the scope of a single item in the discourse. The appearance of multiple links does not clarify what the issue is. A single link ought to be enough, or the items are not sufficient assembled.
- The use of multiple references especially in the Related Work Section has been reduced.
- In particular, the Related Work Section has undergone a complete revision.
The composition of the text lacks somewhat rigidity. It could outline strictly the structure of the concept (ingredients for a time series, composition of a time series, interpretation / prediction of a time series). Hence it misses the theoretical foundation, explaining why proper tooling can be used for any problem with the same foundation.
- This aspect is no addressed in Section 2. Please have a look.
A last round of critical reading with the reader in mind will be helpful.
- Some sections have been restructured especially 4, 5, and 6.
- The results section has been splitted up in a results and an evaluation section.
- The related work section addressed your comments.
I understand that the author has the urge to tell the readership that no new tooling is required. However, he should clearly state that the reason is the flexibility in the structured report writing for the internal logs. I.o.w. it is a typical effect of standardization.
- This has been explicitly mentioned and added in the conclusion section.
