Abstractive Summarizers Become Emotional on News Summarization

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors evaluated some summarization models (Bart, Pegasus and T5) related to their capacities of emotional behavior (cohesion and bias), using the CNN/DAILYMAIL and XSUM corpora.

So, from a scientific and practical overview, the authors' study is interesting and has many contributions. With regard to the manuscript, it needs further revision and improvement, both textual and structural. Another critical point is the subtle and superficial description of the experimental aspects, preventing the reproducibility of the study performed.

In addition to these general comments, also contributing to an improvement of this study, the following suggestions are provided:

* Generally, the abstracts have the following structure: contextualization, problem, proposed solution, summary of results, and conclusion. Regarding the abstract of this work, only the last element was missing (conclusion). So I suggest readjusting the abstract, including such information.

* Regarding related works, it would be interesting to have a comparative table between related studies and the research carried out by the authors.

* How was the vocabulary size obtained? Was it from the NRC lexicon? It would be interesting to clarify this information for the purposes of reproducibility of the experiment carried out.

* Spelling or grammatical error in line 171 (impemented, instead of "implemented").

* Why were the BART, PEGASUS and T5 models chosen? Any special reason? What was the choice based on? For convenience? Based on some scientific study? Please, clarify this information in the text.

* Why were the CNN/DAILYMAIL and XSUM corpora chosen? Any special reason? What was the choice based on? For convenience? Based on some scientific study? Please, clarify this information in the text.

* Why did you choose such baselines? Any special reason? What was the choice based on? For convenience? Based on some scientific study? Please, clarify this information in the text.

* Why were Rouge and BertScore metrics chosen for the test scenarios? Any special reason? What was the choice based on? For convenience? Based on some scientific study? Please, clarify this information in the text.

* Figure 2 is relatively far from its first quote/citation.

* What would be the term "w/o" in Table 4 and on line 284? Please, clarify this information in the text.

* Apparently, the authors forgot to insert a period/dot after the term "XSUM" on line 294.

* In the article, a section on threats to validity should be included, highlighting the threats (internal, external, construction and conclusion) and their means of mitigation.

* In conclusion, the authors could improve the last paragraph about future works.

* The sections 4 to 7 could be improved/organized: (i) section 4 and some parts of sections 5 and 6 would correspond to "Experimental Evaluation" section, that it will follow the Wohlin's experimental guidelines; and (ii) the remain of sections 5 and 6, in addition to section 7, would be the Results and Discussion section. Related to the experimental process, the way in which it was explained, it is partially difficult to understand how the experiment was carried out, as well as it is impossible to replicate it for auditing and reproducibility of results and procedures in other contexts.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in software engineering. Springer Science & Business Media.

* Following Open Science principles, it is suggested to include the URL of a public Git repository containing the source code for pre-processing and analysis. In addition, sharing the data collected and processed.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper examined the emotional content of two summarization corpora, namely CNN/DAILYMAIL and XSUM, and explored the capability of existing generative summarization models in mining emotions in the generated summaries. In order to enhance the quality of the paper, here are some suggestions:

1. Briefly state the innovative points or contributions of the article in the last paragraph of the introduction.
2. Language needs to be polished.
3. Please add the overall architecture diagram of the model.
4. There is an issue of numerical values being obscured in Figure 1.
5. Please explain the meaning of the intersection of the emotion set in the references and the emotion set of generated summaries in formula (3) and formula (4).
6. What is the definition of "Emotions of novel words"? Is it referring to words that have not appeared in the corpus or emotions words?
7. It is suggested that the author propose a novel generative summary model based on the emotional needs of the generated summaries, as the current study only explores the emotional behavioral information of existing generative summary models, which may not have enough innovation and practical value.

Comments on the Quality of English Language

 Extensive editing of English language required.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

1. 1. Use consistent formatting for percentages, such as "0.9%" instead of "0,9%" for clarity and readability.

2. 2. Enhance the flow by grouping related findings together. For instance, consolidate observations about emotional content in articles and summaries to improve coherence.

3. 3. Provide more detailed insights into the analysis methodology, such as how emotion density and ratio are computed. This can enhance the reader's understanding and appreciation of the study.

4. 4. Minimize repetition of phrases, like "We observed," by varying language to maintain reader engagement. Consider using synonyms or restructuring sentences.

5. 5. Ensure consistency in terminology, such as using "emotion density" and "emotion ratio" consistently throughout the text.

6. 5. Discuss implications of the recall range (34% to 51%) for the emotions in novel words. Provide insights into the significance of this coverage in relation to the overall emotional content

Comments on the Quality of English Language

1. • 1. The language is generally clear and precise, conveying the key findings effectively.
   • Ensure consistency in representing percentages; use the standard format (e.g., 0.9%) for clarity.
2. • 2. Text exhibits good coherence, presenting findings in a logical order.
   • Consider grouping related information into cohesive paragraphs to enhance overall organization.
3. • 3. Maintain consistent verb tenses throughout. For instance, choose either "found" or "find" and use it consistently.
4. • 4. Introduce sentence variety to maintain reader engagement. Some sentences have a similar structure, which could be diversified.
5. • 5. Use a variety of words to avoid repetition. For example, instead of repeatedly using "emphasize," consider synonyms like "highlight" or "underscore."

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for your patience and active response to the review comments. I highly appreciate your thorough revision of the manuscript based on the feedback. Your efforts have resulted in significant improvements in the readability, quality, and scientific validity of the manuscript, which has now fully met the standards of our journal. Once again, I want to express my gratitude for your support and cooperation, and I look forward to working with you again in future manuscript submissions.

Comments on the Quality of English Language

The author has diligently addressed the review comments and made revisions to the manuscript, which now meets the requirements of the journal and is ready for acceptance.