A Natural Language Processing Approach to Social License Management

Round 1

Reviewer 1 Report

In this paper, the authors have proposed a program (SLaCDA: social license and controversy detector and analyzer) that uses machine learning techniques to estimate the level of social license by analyzing the sentiment of statements expressed by stakeholders in both interview and online social media texts. The program uses natural language processing techniques to extract topics and discourses from the same data in order to detect, classify, and monitor stakeholders’ concerns and issues. The problem statement is well defined in the initial section of the paper. The methods and results sections are well written and very comprehensive. The validation of the SLaCDA program using data from a mining project in Bolivia, that the authors have presented here, upholds the relevance and novelty of this approach. As a future scope of work, the authors could use data from other sources and discuss how their program performs. 

Author Response

Thank you for your comments. We added several sentences to the end of the results section discussing our plans for further validation of the program across countries and sectors.

Reviewer 2 Report

From my point of view the paper needs to be improved in several aspects to be published:

1. Title: authors say that the tool will be applied for "social risk management". Risk management is not presented in the full paper. The title should be reviewed not to create false expectatives.
2. Introduction: there is a complete lack of references.
3. 1.1. Stakeholders: this a hot topic and really interesting. Authors don´t present it properly without considering relevant publications.
4. 1.3. no one reference is presented.
5. At the end of 1.3. authors afirm "Many methods for extracting themes and topics from texts already exist." Which ones?? at least they should be named.
6. Materials and methods: authors afirm that they will do sentiment analysis but this is not found in results. They analize words but not sentiments.
7. Figure 1. Panel A and B. The text bellow the graph can not be read.
8. Results should be improved to understand the work done.

Author Response

The original reviewer critiques are given, with our responses in blue below each:

1. Title: authors say that the tool will be applied for "social risk management". Risk management is not presented in the full paper. The title should be reviewed not to create false expectatives.
   1. Social risk is a broad area, and this paper focuses just on social license. The title was revised to match the focus of the paper: “A Natural Language Approach to Social License Management”.
2. Introduction: there is a complete lack of references.
   1. See revised Introduction and abstract with additional references
3. 1.1. Stakeholders: this a hot topic and really interesting. Authors don´t present it properly without considering relevant publications.
   1. We included references showing the history and relevant developments in the social license arena of stakeholder management
4. 1.3. no one reference is presented.
   1. Section 1.3 removed from the revised introduction, but relevant references are presented in other sections
5. At the end of 1.3. authors afirm "Many methods for extracting themes and topics from texts already exist." Which ones?? at least they should be named.
   1. This section and the above statement have been excised. We wanted to focus the paper on the difference between manual methods (see section 3.3 for and explanation of the manual method used) and the natural language processing method (LDA, see section 2.2)
6. Materials and methods: authors afirm that they will do sentiment analysis but this is not found in results. They analize words but not sentiments.
   1. The sentiments were also analyzed in section 3.2, but it was unclear that that particular analysis was related to the sentiment analysis. His section was revised for clarity.
7. Figure 1. Panel A and B. The text bellow the graph can not be read.
   1. See new version for corrections
8. Results should be improved to understand the work done.
   1. We revised our outline to more clearly present our results

Reviewer 3 Report

Thank you for giving me the opportunity to review the article „A Natural Language Processing Approach to Social Risk Management”. Below my remarks:

• “This paper describes how a blend of practical social science research and advanced language processing algorithms can provide the detection and monitoring needed for agile responses stakeholder concerns”. However, I think that the main aim of the work should be clearly formulated and included in the introduction and the abstract. It needs to be rewritten;
• The introduction to the article is too long and takes a lot of threads. It should be rewritten and an additional section should be created: a literature review;
• The Authors should carry out a solid review of the literature and present the results obtained. The current references are very poor;
• The statistical analyses carried out are very poor. They should be deepened (if possible);
• The discussion is not scientific in nature. It should be compared with the results obtained by other researchers;
• The article should be supplemented with parts: Conclusions, Further research and limitations
• In Figure 1 part of the legend was "cut off".
• In Table 1 a smaller font should be used.

Author Response

The original reviewer critiques are given, with our responses in blue below each:

• “This paper describes how a blend of practical social science research and advanced language processing algorithms can provide the detection and monitoring needed for agile responses stakeholder concerns”. However, I think that the main aim of the work should be clearly formulated and included in the introduction and the abstract. It needs to be rewritten;

• Please see the revised introduction and abstract. One of the things we focused on in the revision is the clear goal of reducing the time needed to perform these analyses, so that the insights can be used in real-time, and don’t become obsolete with rapidly shifting stakeholder opinions and perceptions.

• The introduction to the article is too long and takes a lot of threads. It should be rewritten and an additional section should be created: a literature review;

• We shortened the introduction, and took out section 1.3. We didn’t create a new section for the literature review, but we did update it to include a better examination of the literature.

• The Authors should carry out a solid review of the literature and present the results obtained. The current references are very poor;

• See above

• The statistical analyses carried out are very poor. They should be deepened (if possible);

• See the added significance testing in sections 3.1 and 3.3

• The discussion is not scientific in nature. It should be compared with the results obtained by other researchers;

• It is compared in the literature review. But it is also because we’re doing the first longitudinal comparison. While there are several other practitioners working with social license and public expectations in resource development projects, the focus of that research is too different to be compared. For example, Moffat and Zhang focus on multivariate analysis in order to determine correlations between project impacts and relationships, trust, and approval. The purpose of this paper is to explore the effectiveness of the ML tools in assessing stakeholder issues relative to the social license. The actual content of Boutilier’s stakeholder analysis, and its context within the framework of the SL literature is cited in our paper (Boutilier 2011, 2017, Cooney2017).
• Section 3.1 mentions comparison with other researchers for the sentiment analysis “This is typical performance for sentiment analysis algorithms [38], which range from the high 50’s to the low 90’s in terms of accuracy percent.”

• The article should be supplemented with parts: Conclusions, Further research and limitations

• We didn’t include extra sections for further research and limitations, but we did include plans for further research in the discussion (see second to last paragraph). Limitations are given in relevant sections throughout (see for example the discussion on domain specificity and sarcasm detection in section 2.1). We also included a “Conclusions” section.

• In Figure 1 part of the legend was "cut off".

• There must be some typesetting error. Nothing is cut off in the version that we have. We will talk to the journal editors to make sure it is typeset properly. Thanks for noticing that for us!

• In Table 1 a smaller font should be used.

• The formatting is dictated by the journal.

Round 2

Reviewer 2 Report

Authors have done a deep review improving significally the paper. I apreciate it.

Author Response

Thank you for your helpful comments. They vastly improved our paper.

Reviewer 3 Report

Dear Authors

thank you for including my comments in the improved article.

Author Response

Thank you for your helpful comments. They vastly improved our paper.