A Scoping Literature Review of Natural Language Processing Application to Safety Occurrence Reports

Round 1

Reviewer 1 Report

This paper reviews the literature regarding the application of Natural Language Processing to safety occurrence reports. There are several concerns presented as below.

i) How to determine the nodes? Is the definition of nodes merely based on the key words of the literature?

ii) The number and legend in Figure 2 are quite small and vague.

iii) The authors could discuss the difference between Classification and Clustering when analyzing the paper aim. The two concepts are usually confused among the NLP papers.

iv) It could be better if the authors plot the relationship by bibliographic links between the different industries in addition to the Pie chart (Figure 3). In this case, the readers could find that how strong the influence of safety occurrence reports from a certain industry is, and how these reports could be referred by another industry.

v) I notice that the sum of the number of papers sorted by general aims is equal to 61. However, did the author considered the possibility that one paper may contain more than one general aim? For instance, a study could discuss the accident prediction and injury prediction simultaneously. Likewise, a study may use risk variables in the classification of hazards. As such, why the authors identify only one general aim for each study?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

This study aimed to present and discuss a literature review exploring how Natural Language Processing has been applied to occurrence reports within safety-critical industries, informing further research on the topic and highlighting common challenges. The topic is interesting. Some comments for the authors to improve the quality of the manuscript are below.

1.       In lines 32-34 and 40-43, please add some literature to support these statements.

2.       Is the selection of databases comprehensive? Why was Google Scholar not included in the database?

3.       There was a problem with the way the literature is cited in the text (e.g., line 136 and line 146); the author's name (and year) should be marked out, plus the serial number of the cited literature.

4.       Please add limitations to the discussion.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

Thanks for the nicely written article on this very interesting topic. Please consider my remarks below, most of which are in the direction of making your paper 'clearer' for the non-specialist reader.

General comments
-Easy to follow text
-Occasional grammar and spelling mistakes
-Revisit citations/reference style

Title: After reading the paper, I think the study is a 'scoping' review, not just a narrative one, as you performed a systematic review of literature to identify current knowledge, gaps, trends, etc.

Abstract: Although explained in the paper, here you could include some information about how NLP can assist (or already assists) safety professionals so that to attract attention and interest of the latter. Is NLP mainly for classifying text into descriptive categories which are part of a larger dataset for monitoring trends? What do you mean by 'new knowledge' when referring to Topic Modelling while considering that the reports also represent the knowledge we have about an occurrence?

Lines 33-34: What is 'previously unseen knowledge'? Please provide an example.

Lines 37-38: Please provide to the reader a bit more information regarding what deep learning and machine learning are about and their differences. You only have a footnote in the results section, but this is not sufficient.

Lines 40-44: Please add citations/sources of those arguments or remove them.

Line 60: Citation/reference style is wrong. Please check the whole paper; several 'issues' with that, especially in the results section.

Section 2:
-There is no information about any first- and second-level screening processes and results as well as inter-rater checks and resolutions of any disagreements. A graphical representation of the search flow and results from each phase (e.g., PRISMA) would also be appreciated.
-Figure 1 belongs to the results. Keep in this section only explanations about the use of VOS and some citations/examples of its application.

Section 3 introduces concepts not previously mentioned in the Introduction section (e.g., the various types of computational methods). Please enrich the Introduction section with references to anything necessary to make sense of the results (and their discussion). Please do not assume your readers are knowledgeable about everything you mention.

Lines 111-115: This is not results but discussion. Also, the argument is not clear. Your results do not indicate which industries use safety reporting systems, but which industries have been included in the sort of studies you were interested.

Figure 4/Table 2: It is not clear how those groups were created. Please offer explanations in the methods and/or this section, as applicable. Also, please provide some brief descriptions/explanations of what each group is about and what it targets. While reading the subsections following, it was not always crystal clear (e.g., difference between classification and entity extraction).

Line 146-149: It is not clear how this study ended up in your sample if it is not about safety reports. Please clarify.

Lines 163-164: Please either explain or remove technical jargon here (e.g., hamming loss and F1 scores) and elsewhere in the paper.

All Results subsections: In the first sentences defining each category, please give some practical examples from each application. Otherwise, it can be difficult to understand differences and aims of each (see also comment above). For instance, in section 3.3 you mention topics but it remains blurry what you really mean as there are no specific references to examples from the studies you cite.

Table 3: It is not clear whether your suggestion to treat any text as their 'own'/'separate' language in several instances would allow to combine reliably the results from different analyses to derive trends, etc. Also, some recommendations read intuitive and not very useful, like if A is necessary, then do A.

Lines 382-383: Considering the current limitations on NLP, I would see the role of the regulators as 'funding and encouraging' rather than endorsing at this stage.

References to the skills of safety professionals in the Discussion and Conclusion sections can be more specific. For instance, what do they need to 'study', how long this would take, etc.?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Author Response

Thank you for your time to review the manuscript.

The manuscript has only undergone English language editing.

Reviewer 2 Report

A minor comment is left. Please seek English editing service to improve the language of the manuscript. 

Author Response

Thank you for your time to review the manuscript.

The manuscript has undergone English language editing.

Reviewer 3 Report

I appreciate your efforts to address my comments. I feel comfortable to recommend acceptance of the paper.

Author Response

Thank you for your time to review the manuscript.

The manuscript has only undergone English language editing.

Round 3

Reviewer 2 Report

The authors did a good job in addressing my comments.