Local Event Detection Scheme by Analyzing Relevant Documents in Social Networks

Round 1

Reviewer 1 Report

Minor comments:

• it is not very clear on figure 1 what is the input/output data. Maybe some colors and a legend would help

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the atached "response-to reviewer's comments" file. Many thanks.

Author Response File: Author Response.docx

Reviewer 2 Report

The event detection scheme proposed in this white paper is new and practical, so it is worth publishing after minor revisions. The detailed comments are listed below.

1. 241: Please explain the “inverse document frequency” in more detail.
2. 319: There is no explanation about NM_G in Equation (5).
3. 351: The number of tweets used in the experiment (119,243) is very small compared to the number of tweets posted in a month. Please explain in more detail the criteria for collecting the tweets used in the experiment.
4. 355: What does K stand for?
5. 361: Please explain “Recall” and “F-measure” in more detail by citing related references.
6. 369: What does NMG stand for?

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the atached "response-to reviewer's comments" file. Many thanks.

Author Response File: Author Response.docx

Reviewer 3 Report

The authors propose a method based on the analysis of relevant documents (i.e. comments, post, threads), embedding with geographical data, in social networks for local event detection.

The proposed approach is interesting but there are some points that the authors have to better discuss.

The authors should be better described the novelties of their approach with respect to existing ones. In particular, the author should discuss limitation and cons that their approach aims to overcome at the end of the Related Works section. Furthermore, the authors should provide more details and discussion about the obtained results. The Discussion section also needs to be improved by analyzing the outcome of evaluation section.

I suggest to analyze also more recent approaches about the examined topics. In particular, I suggest to underline the deception activities of intruders by further investigating these approaches, :

1) Evolutionary game theoretical on-line event detection over tweet streams. Knowledge-Based Systems 211: 106563.

2) Extreme events management using multimedia social networks. Future Generation Computer Systems, 94, 444-452.

Finally, I suggest to perform a linguistic revision.

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the atached "response-to reviewer's comments" file. Many thanks.

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

I think that the authors have addressed all my concerns. 

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The authors utilise social network data and linked content for local event detection. Suck work has been attempted before and the related work provided in the manuscript could use additional analysis for the reader to understand the challenges.

The idea is sound, enriching data for analysis is expected to output improved results.

Items that need to be addressed:

Section 3.2 is titles "Data Collection".

It contains text on how data collection CAN be done, which is rather trivial. There is no novel information at the moment.

What the user is interested in is the dataset. What is the dataset or  the data used for this work?

Is it available and where?

Where had the data been collected from?

Was it from Twitter? Facebook? elsewhere?

Please provide the details for this.

Section 4 "Experiments".

The numbers seem incorrect. How were accuracy and recall computed? What was the baseline? How does 69% get computed?

Is there a manually annotated (human curator?) top-K reference corpus? It has to be, otherwise how is accuracy and recall compute?

And does such reference have a baseline for detection?

If so, what is the absolute number of events identified?

Top-K would have to be replaced with max (number of events in the dataset) and the trials with max/2, max/4 or similar. Otherwise top-10 could be already exceeding the above-threshold identifiable events...

 

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the attached file about the detailed revisions.

Author Response File: Author Response.docx

Reviewer 2 Report

The article proposes a local event detection scheme, that detects the local event based on a keyword graph that is constructed through the analysis of the relevant documents. The main contribution in the well-known and heavily used in the literature idea concerning local event analysis is to exploit and create a graph based on comments and threads in order to improve accuracy, the use of clustering and MAINLY the employment of geographical dictionaries in embedded with region-related information without requiring users to tag geographic data. The article also presents a set of experiments that depict the superiority of the proposed scheme in comparison to other schemes that were used in the literature.

The article is very well written and the various schemes presented of interest to the community and novel, as far as I was able to check, hence I vote for acceptance. On the other side, I would expect in the final version of the article a more detailed discussion concerning the novel point where the present article deviated from previous works plus a discussion concerning statistical tests (t-tests) on the experiments.

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the attached file about the detailed revisions.

Author Response File: Author Response.docx

Reviewer 3 Report

1. The related work section is too small for a journal paper and has to be significantly increased
2. Section 3.1 is supposed to talk about the system architecture, but in fact the section describes how the geo-tagging works and keyword extraction. A system architecture contains usually a general description of the data flow and main components used, without specific details. Figure 1 does not show how different component interact. Does this mean that the different processing pipelines are running independently without exchanging any data?
3. Data collection: What language/languages are targeted? Why are buzzwords and emoticons irrelevant? Considering that that users have only 140 characters (or a bit more with the latest expansion) to express themselves they may use an emoji / emoticon to spell a word. On top of that, how are the hastags processed?
4. Why are stop words removed? If attempted to do a simple keyword matching to build the graph, the stop words can have a very high impact:
   1. Emergency services are in London
   2. Police is looking for the suspect around London
   3. I’m glad I’m not in London for this incident
5. Section 3.3 Graph Modelling: it is not clear how this is applied on the data. Is there a graph computed for each tweet or for a collection of tweets? If the graph is computed for the whole dataset how do you distinguish between different events. This needs to be clarified.
6. I am wondering how the date of posting is integrated into this approach. In any event detection application, the date, location and event type are extremely important. This paper talks about the type and location, but the date does not seem to be the focus anywhere.
7. What is the Twitter Scrapper API? If you are using this https://pypi.org/project/twitter-scraper/ library, you should be careful that this may be against Twitter’s data policy. This raises some ethical questions about the aquisition of the dataset.
8. Why there is a difference between tweets and relevant documents? Especially that the number of relevant documents is higher.
9. On all the experiments, the authors are talking about precision and recall. What is the gold standard they are comparing against? Are the tweets/events manually annotated?

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the attached file about the detailed revisions.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors addressed the comments adequately.

Author Response

The reviewer mentioned that we addressed his/her comments adequately.

Thanks.

 

Reviewer 3 Report

I think this version of the article clarifies some things and some parts are moving into the right direction (i.e. clarifications regarding the language, timeframe for processing, etc).

On the other hand, the ethical concern is not solved: the authors point at a Twitter documentation page that shows the opposite of what they want to prove (Twitter strongly encourages, through the user license as well that developers use the API and not the scrapped website). They also promise to delete all their data after analysis, which they should have done a long time ago. I recommend restarting the analysis with a clean dataset.

 

Detailed answers:

[Comment 1] The related work section is too small for a journal paper and has to be significantly increased

Answer : As you pointed out, we further analyzed and described the existing schemes with high citations related to local event detection .

Review: The state of the art is still rather short considering how much work was done in this field.

 

[Comment 2] Section 3.1 is supposed to talk about the system architecture, but in fact the section describes how the geo-tagging works and keyword extraction. A system architecture contains usually a general description of the data flow and main components used, without specific details. Figure 1 does not show how different component interact. Does this mean that the different processing pipelines are running independently without exchanging any data?

Answer : As you pointed out, we described in detail the function of each component in the overall system structure and the interaction between them as follows.

Review: I think the edits are a step into the right direction, but Figure 1 still does not show any interaction between the larger boxes. I cannot tell by looking at Figure 1 if the algorithms are running in parallel or sequence and what is the output and input of each of these boxes. Moreover, some of the explanation offered privately to the reviewers should be reflected in the text as well.

 

[Comment 4] Why are stop words removed? If attempted to do a simple keyword matching to build the graph, the stop words can have a very high impact:

1. Emergency services are in London
2. Police is looking for the suspect around London
3. I’m glad I’m not in London for this incident

Answer : As you pointed out, stop words can be used to detect 'detailed and more accurate' events. However, since the proposed scheme is aimed at detecting local events, it is possible to detect local events that are meaningful just by extracting key keywords such as 'London', 'incident', 'police', and 'Suspect', so stop words were removed for quick analysis.

Review: I strongly believe that by removing stop words you are adding noise to the “disambiguation” algorithm. Plus, because you are aggregating within a 1h window, by just capturing the main words you probably could not distinguish between an ongoing incident and resolved incident with a suspect apprehended. The vocabularies will be very similar.  

 

[Comment 6] I am wondering how the date of posting is integrated into this approach. In any event detection application, the date, location and event type are extremely important. This paper talks about the type and location, but the date does not seem to be the focus anywhere.

Answer : As you pointed out, date is meaningful data. As mentioned in Section 3.3 Graph Modeling, TF-IDF is calculated based on a time window, and overall event extraction also performs a time window-based calculation. In this paper, time is reflected in local event extraction because time window is set to 1 hour and local event extraction is calculated.

Review: Is there an aggregation strategy for events spreading over multiple hours?

 

[Comment 7] What is the Twitter Scrapper API? If you are using this https://pypi.org/project/twitter-scraper/ library, you should be careful that this may be against Twitter’s data policy. This raises some ethical questions about the aquisition of the dataset.

Answer : We used the library you mentioned. According to https://developer.twitter.com/en/docs/basics/things-every-developer-should-know, we figured out that standard APIs are available to anyone. Of course, we agree with the ethical issues you mentioned, and we will never use them for any purpose other than research purposes, and we are discarding the data after generating the results through performance evaluations.

Review: The documentation you pointed out refers to the use of official Twitter APIs and not libraries that offer a way around their limitations. Twitter Scrapper states that it was built to avoid daily quotas and premium APIs. I would personally recommend restarting the analysis with a legitimate dataset.

 

[Comment 8] Why there is a difference between tweets and relevant documents? Especially that the number of relevant documents is higher.

Answer : As you mentioned, a relatively large number of relevant documents have been collected. There are no particular reasons, but the result shows that it is meaningful to consider relevant documents for detecting local events.

Review: I am still confused what the relevant documents are.

 

[Comment 9] On all the experiments, the authors are talking about precision and recall. What is the gold standard they are comparing against? Are the tweets/events manually annotated?

Answer : In general, event detection schemes have used accuracy and recall as the main metrics for evaluations. However, as you pointed out, evaluation methods such as the ROC and AUC can be meaningful as well. However, the revised period is short, so we will proceed with the performance evaluation for these measures in future research. As you pointed out, we have added some parts that are not clear to clarify how accuracy and recall are calculated. We used 50 local events directly tagged as a true set. The explanation is added in this paper as follows.

Review: I understand the metrics used, but what I’m asking is what you are comparing with: what is the ground truth for this task. This can be either annotated data by human users or the tagged events are reviewed by human users. A third option could be that the results are evaluated by another automated approach.

Author Response

We would like to sincerely thank you for your attentive indications and good comments. Our paper is partially rewritten in order to revise and complement your comments. Please refer to the attached file.

Thank you so much.

Author Response File: Author Response.docx