Sentence Graph Attention for Content-Aware Summarization

Round 1

Reviewer 1 Report

The authors have developed a sentence-level attention mechanism model r abstractive text summarization based on the well-known PageRank to find the relevant sentences, then propagating the resulting scores into a second word level attention layer. After reviewing the manuscript there are few concerns are addressed.

1. The authors need to check the references which are not in order, for instance the reference 24 is not cited properly. So cite it accordingly. I would suggest the authors to do a thorough check for all the references they cited.

2. Generally, related works section will come next to the introduction section to partly motivates further study of the field, i.e., by showing that previous experts have worked on similar problems, and that such problems are well established. But I found that the authors have kept the related works section after the results section which will reduce the readers interest. So I would suggest the authors to follow the format to insert the related works section after introduction.

3. After changing the related section, please change references numbers accordingly to make sure they are properly cited in an order.

4. I would suggest the authors to put all the parameter tuning in tabular format with all the parameters used for training purposes. and also they need to mention the ratio in that table as well.

Author Response

Point 1:  Thank you very much. We are checking the references and correcting the errors. If you noted something else, please provide it to us.

Point 2 and 3: We decided to move the related works after the results in order to do not break the pace and continuity of the article, making it easy to read. However, if it a problem, we are glad to move it after the introduction.

Point 4: We will add a table with the tuning parameters of the model.

Reviewer 2 Report

Authors have presented a defined sentence-level attention mechanism based on the well-known PageRank to find the relevant sentences. In order to improve the quality of the paper the following things can be done:

1. Authors should include a graphical representation of the process in a flow chart or any other convenient way.

2. Is it possible for authors to compare the Page Rank algorithms with other Natural Language Processing algorithms like KL algorithm, Latent Semantic analysis algorithm, etc., and further compare the performance on the basis of obtained ROUGE scores for each algorithm? 

Author Response

Thank you very much for your comments. Below you can find our replies to each point.

Point 1: It is an interesting comment. Could you please elaborate what you mean with "process"? How should this figure/flow chart be different from the network architecture (figure 1)?

Point. 2: The problem of comparing PageRank with other NLP algorithms (such as KL divergence or Latent Semantic Indexing) is that our version of PageRank is fully derivable, while some of the other methods are not; this means we have to use Reinforcement Learning (RL) to train the neural network. In my opinion, comparing the current model with one trained with RL is completely different from what you propose: in this case we are not comparing a single layer of the neural network, but a totally new neural network with the original one. However, the idea of comparing different NLP algorithms is a good future research point to improve our model. 

Reviewer 3 Report

This paper proposed a novel abstractive text summarization method by incorporating graph-based sentence scoring. The result in Rouge scores showed that the proposed method outperformed the traditional sentence scoring baseline (Nallapati's). However, the proposed method did not reach state-of-the-art performance. Additional analysis shows that the page rank losses mitigated the n-gram repetition problems and increased the abstractive power of the model.

 

While the paper has some merit in its quality of the presentation, other aspects can be improved:

1. The graph is constructed based on the sentence representation. It is misleading to call it a "knowledge graph", which usually associates with a large external knowledge base. It is also not obvious how this paper is distinct from Tan et al's. A discussion in the related work on how the authors extend from the previous work would clarify the originality of this work.

2. There is a slight disconnection between the main experiment results (Table 1) and the analysis part (Figure 5 and Figure 8). The authors used "pg xxxx" in the figures, but they did not seem to associate with any methods in the main experiment results. Keeping the names of the methods consistent would help audiences conclude, and increase the impact of the paper.

3. For the abstractiveness of the method, the authors claimed that their methods had higher power than See et al's (Conclusion), but there was no support or comparison in Section 3.3.

4. Since the results in Table 2 were from three human annotators, the authors should present the annotator agreement and whether the scores' differences were statistically significant. The authors also pointed out that Tan et al's method had a "very low" readability and relevance score, but Table 2 showed that the readability score was close to the proposed method.

 

 

Author Response

Thank you very much for your comments. Below you can find our replies to each point.

Point 1: tan et al.'s model only uses a double attention: one over the sentences and one over the words. However, both attention are not connected, i.e. the sentence attention has no impact on the word attention. Suppose that a sentence has a lower score, but a word in that sentence has a very high score. Since that the two attentions are separated, it is not possible to adjust the word score according to the sentence score. In our paper, we decided to calculate the word scores again combining the two scores together. In this way, a word will have an high score only when the sentence where that word appears has an high score.

Point 2: Thank you, we will improve the text and the images in order to match them and to make the paper more readable. 

Point 3: Thank you to notice it. It is a refuse from a previous version where we analyzed other aspects of our neural network. We decided to remove it in the final version. We will correct the conclusion

Point 4: we will add the inter annotator and if the differences are (or not) statistically significant.

Point 4 "Tan et al's method had a "very low" readability and relevance score, but Table 2 showed that the readability score was close to the proposed method.": we mean that Tan et al's method has a lower score with reference to our one (about 0.72 for readability and 1.89 for relevance). We will improve the text.

Round 2

Reviewer 1 Report

After reviewing the revised manuscript, it is observed that the authors have addressed the concerns which were raised earlier. 

I would recommend this manuscript towards acceptance if the editor has no concerns.