Papers in and Papers out of the Spotlight: Comparative Bibliometric and Altmetrics Analysis of Biomedical Reports with and without News Media Stories

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for submission of this interesting manuscript. Overall, I found the topic, writing and methodology to be sound. Overall, I have only a few brief suggestions for improvements. I would have liked to have seen a look at other current literature in this area and how your study compared to that research. A quick look in PubMed shows some similar, though niche studies. How do their results compare to this study? This also could be addressed in the discussion. Right now the discussion is more a summary of the results. It also needs to tell me where this research sits with other research and how it can be used. Great limitations section, but also needs where you could go from here and what else could have been done to expand and improve the study.

Comments on the Quality of English Language

English language quality is excellent. Any issues I had were more with phrasing, not grammatical.

Author Response

For the research article publications-3177816, entitled: “Papers in and Papers out of the Spotlight: Comparative Bibliometric and Altmetrics Analysis of Biomedical Reports with and without News Media Stories”

Response to Reviewer 1 Comments

1. Summary

Thank you very much for taking the time to review this manuscript, for your kind words, appreciation, and for your valuable comments and suggestions on my manuscript. Please find the detailed responses below and the corresponding revisions highlighted in the re-submitted files.

2. Questions for General Evaluation Reviewer’s Evaluation 

Is the content succinctly described and

contextualized with respect to previous and

present theoretical background and empirical               Can be improved

research (if applicable) on the topic?

Are the research design, questions, hypotheses

and methods clearly stated?                                               Yes

Are the arguments and discussion of findings

coherent, balanced and compelling?                              Yes

For empirical research, are the results clearly

presented?                                                                               Yes

Is the article adequately referenced?                               Can be improved

Are the conclusions supported by the

results presented in the article or referenced in             Must be improved

secondary literature?

3. Point-by-point response to Comments and Suggestions for Authors

Comments 1: Thank you for submission of this interesting manuscript. Overall, I found the topic, writing and methodology to be sound. Overall, I have only a few brief suggestions for improvements.

Response 1: I would like to thank the reviewer for the kind words and appreciation.

Comments 2: I would have liked to have seen a look at other current literature in this area and how your study compared to that research. A quick look in PubMed shows some similar, though niche studies. How do their results compare to this study? This also could be addressed in the discussion. Right now the discussion is more a summary of the results. It also needs to tell me where this research sits with other research and how it can be used.

Response 2: Thank you for your comment. The Discussion was enriched with additional literature evidence that supports the superspreading capabilities of news media outlets on the internet and social media, as well as investigating the relationships between scientific papers and news media stories under different contexts and niches. Explanations were also added on where this investigation stands and how it can be used. The following paragraphs were added on page 12, second paragraph, lines348-389.

“In the literature, the interplay between scientific papers and news media stories has been considered under different contexts and niches, but in most cases, involving the internet and social media [26]. Contagion-like diffusion of information in social media could be attributed to network users or content derived from external sources like news outlets [27]. However, as superspreaders of information, penetrating and impactful newspapers or news channels may deliver erroneous messages due to citation bias, false consistency, lack of clarity and transparency, overgeneralization, exaggeration, insulation, or noncredible sources [28]. Recently, it was shown that there is a positive correlation between the external popularity of research outside the scientific community by noteworthy media coverage and the number of scientific reports published after media coverage [29]. This investigation aligned the Altmetric® Main-stream-Media-Scores (MMS) with thematically similar articles from the PubMed database to obtain increased scientific publications on the same topic as a research article that received widespread news media coverage after this incident. This report didn’t examine the citations, bibliometrics, or other altmetrics but showed that news stories may influence scientific authors’ research investigations. However, the trigger of scientific community attention was the extensive, exceeding a hundred news stories per paper, news media attention on a single paper. The data presented here for the biblio-metrics of scientific articles with and without news stories agrees with this observation, but by considering a single news story of a paper. The average number of citations received after a single news story mention was found to be increased by 60% compared to a matching paper without a news story. This finding is important because only a tiny fraction, in this investigation, 0.1%, of the published scientific literature achieves over a hundred news media stories.

In a recent bibliometric and altmetrics investigation, Digital Science Dimensions® was used to generate a bibliographic portfolio of COVID-19 reports derived from Jordan and published between 2019 and 2022, which were subgrouped according to Altmetric Explorer® or Semantic Scholar® Highly Influential Citations (HICs) and compared [30]. The Semantic Scholar® HICs represent an AI-generated classification of citations regarding the context of the citations in a paper. There was a lack of correlation between Semantic Scholar® HICs and Altmetric® Attention Scores. This finding agrees with the Pearson correlation accompanied by linear regression analyses presented here, as citations and Altmetric® Attention Score Pearson correlation coefficient value was 0.05. This is explained by the different motivations and criteria that define the citation, and the specific altmetrics mentioned behavior.

The data presented here concerning biomedicine suggests that news media are potent influencers of the scientific community and society and may generate trends in research investigations, funding flow, and public opinions and beliefs. This is important for all the stakeholders involved in scientific research, investigators, publishers, funders, policymakers, news media outlets, journalists, and the public, for introspection and careful consideration of these stimuli. News stories may deliver novel and groundbreaking science, but there is a risk of exaggeration, misinterpretation, or misleadingness. A critical perspective and cross-evaluation are necessary to extract conclusions and decide actions.”

Also, the following new references were added, pages 14-15, lines 478-490

“26.      Luc, J. G., Archer, M. A., Arora, R. C., Bender, E. M., Blitz, A., Cooke, D. T., Ni Hlci, T., Kidane, B., Ouzounian, M., Varghese Jr, T.K., & Antonoff, M. B. (2021). Does tweeting improve citations? One-year results from the TSSMN prospective randomized trial. The Annals of thoracic surgery, 111(1), 296-300. https://doi.org/10.1016/j.athoracsur.2020.04.065.

27. Pei, S., Muchnik, L., Andrade, Jr, J. S., Zheng, Z., & Makse, H. A. (2014). Searching for superspreaders of information in real-world social media. Scientific reports, 4(1), 5547. https://doi.org/10.1038/srep05547.
28. Elson, M., Ferguson, C. J., Gregerson, M., Hogg, J. L., Ivory, J., Klisanin, D., Markey, P.M., Nichols, D., Siddiqui, S., & Wilson, J. (2019). Do policy statements on media effects faithfully represent the science?. Advances in Methods and Practices in Psychological Science, 2(1), 12-25. ttps://doi.org/10.1177/251524591881130
29. Leidecker-Sandmann, M., Koppers, L., & Lehmkuhl, M. (2023). Correlations between the selection of topics by news media and scientific journals. Plos one, 18(1), e0280016. https://doi.org/10.1371/journal.pone.0280016
30. Dardas, L. A., Sallam, M., Woodward, A., Sweis, N., Sweis, N., & Sawair, F. A. (2023). Evaluating research impact based on Semantic Scholar highly influential citations, total citations, and altmetric attention scores: the quest for refined measures remains illusive. Publications, 11(1), 5. https://doi.org/10.3390/publications11010005.”

Comments 3: Great limitations section, but also needs where you could go from here and what else could have been done to expand and improve the study.

Response 3: Thank you very much for your kind words. I agree with your suggestion, and I’ve added future perspectives in page 13, second paragraph, lines 403-412.

“Future prospective studies would shed light on the timing of bibliometrics or altmetrics responses on a scientific paper upon news story release. Specific altmetrics indicators exhibit individual patterns of interest, converging or diverging from bibliometrics. Retrospective studies would demonstrate the socioeconomic determinants underlying these trends, expert opinions or stakeholders' perspectives against scientific investigations. These tendencies suggest social, economic, and political driving forces over science that may directly or indirectly influence scientific topic trends or research funding. Seeking specific thematic patterns or lemmas in scientific papers’ titles, abstracts, or keywords that attract news media interest would be important to be delivered.”

4. Response to Comments on the Quality of English Language

Point 1: English language quality is excellent. Any issues I had were more with phrasing, not grammatical.

Response 1: Thank you for your kind words. The phrasing was reexamined.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

It is an interesting and innovative study that analyses the influence of social media on the perception and citation of scientific articles.  The article is very clear and well organised.

Author Response

For the research article publications-3177816, entitled: “Papers in and Papers out of the Spotlight: Comparative Bibliometric and Altmetrics Analysis of Biomedical Reports with and without News Media Stories”

Response to Reviewer 2 Comments

1. Summary

Thank you very much for taking the time to review this manuscript and for your kind words and appreciation of the submitted work.

2. Questions for General Evaluation Reviewer’s Evaluation 

Is the content succinctly described and

contextualized with respect to previous and

present theoretical background and empirical               Can be improved

research (if applicable) on the topic?

Are the research design, questions, hypotheses

and methods clearly stated?                                               Yes

Are the arguments and discussion of findings

coherent, balanced and compelling?                               Yes

For empirical research, are the results clearly

presented?                                                                               Yes

Is the article adequately referenced?                                Yes

Are the conclusions supported by the

results presented in the article or referenced in             Yes

secondary literature?

3. Point-by-point response to Comments and Suggestions for Authors

N/A

4. Response to Comments on the Quality of English Language

Point 1: N/A

Response 1: N/A

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This is an excellent study that examines the relationship between media coverage and the dissemination of scientific research. The results show a more pronounced association between news coverage and altmetrics compared to traditional bibliometrics, suggesting that media dissemination contributes significantly to expanding the reach of scientific findings to a wider audience. Here are a few minor suggestions to improve the paper:

%ABSTRACT%

"This investigation, which explores the potential impact of news media stories on the citations and altmetrics of a paper, was conducted with thoroughness and rigour."

Such efforts are expected in good research and need not be mentioned, especially in an abstract. Conciseness and informativeness are key in an abstract. Instead, the objective of this paper should be explicitly stated.

%2.1. Journal and Articles Collection%

It might be better to mention Figure 1 in this section. It can also be referenced in later sections, but introducing it early could help orient readers.

%Results%

Cumulative totals could be misleading, especially if the number of papers in each group differs significantly. This is because the totals would be influenced by the number of papers rather than just the effect of media coverage. A better approach would be to compare averages or proportional metrics that account for the different group sizes.

Consider adding an explicit explanation of the color coding in the figure legend to ensure that readers understand what each color signifies.

%4. Discussion%

It’s important to mention that generalizability may be limited since the study only used high-impact journals.

Comments on the Quality of English Language

This paper contains several awkward expressions and areas where the English could be improved for clarity and readability. Some sentences are overly formal or wordy, making them harder to follow. Simplifying the language and rephrasing some of the more complex or convoluted sentences would help to make the paper more accessible and engaging for readers. Furthermore, ensuring consistency in tone and avoiding redundancy will enhance the overall flow of the paper.

For example, the sentence 'The public's increasing interest in scientific research, particularly in biomedicine, has led news media outlets to play a significant role in disseminating scientific advances' could be rephrased to something more direct and concise like, 'As public interest in biomedicine has grown, news media have taken on an important role in spreading scientific findings.' This revision reduces wordiness and improves clarity.

Author Response

For the research article publications-3177816, entitled: “Papers in and Papers out of the Spotlight: Comparative Bibliometric and Altmetrics Analysis of Biomedical Reports with and without News Media Stories”

Response to Reviewer 3 Comments

1. Summary

Thank you very much for taking the time to review this manuscript and for your valuable comments and suggestions on my manuscript. Please find the detailed responses below and the corresponding revisions highlighted in the re-submitted files.

2. Questions for General Evaluation Reviewer’s Evaluation 

Is the content succinctly described and

contextualized with respect to previous and

present theoretical background and empirical               Yes

research (if applicable) on the topic?

Are the research design, questions, hypotheses

and methods clearly stated?                                               Can be improved

Are the arguments and discussion of findings

coherent, balanced and compelling?                               Yes

For empirical research, are the results clearly

presented?                                                                               Can be improved

Is the article adequately referenced?                                Yes

Are the conclusions supported by the

results presented in the article or referenced in             Yes

secondary literature?

3. Point-by-point response to Comments and Suggestions for Authors

Comments 1: This is an excellent study that examines the relationship between media coverage and the dissemination of scientific research. The results show a more pronounced association between news coverage and altmetrics compared to traditional bibliometrics, suggesting that media dissemination contributes significantly to expanding the reach of scientific findings to a wider audience.

Response 1: I would like to thank the reviewer for the appreciation of the submitted work.

Comments 2: ABSTRACT. "This investigation, which explores the potential impact of news media stories on the citations and altmetrics of a paper, was conducted with thoroughness and rigour." Such efforts are expected in good research and need not be mentioned, especially in an abstract. Conciseness and informativeness are key in an abstract. Instead, the objective of this paper should be explicitly stated.

Response 2: Thank you for pointing this out. I agree with this comment and revise the abstract text accordingly. This change can be found on page 1, first paragraph, lines 9-14.

“As public interest in biomedicine has grown, news media have taken on an important role in spreading scientific findings. This investigation explores the potential impact of news media stories on the citations and altmetrics of a paper.”

Comments 3: 2.1. Journal and Articles Collection. It might be better to mention Figure 1 in this section. It can also be referenced in later sections, but introducing it early could help orient readers.

Response 3: Agree. I’ve, accordingly, introduced Figure 1 in this section, page 3, second paragraph, line 121-122

“…26 journal titles found within the pre-specified impact factor limits, as depicted in Figure 1.”

Comments 4: Results. Cumulative totals could be misleading, especially if the number of papers in each group differs significantly. This is because the totals would be influenced by the number of papers rather than just the effect of media coverage. A better approach would be to compare averages or proportional metrics that account for the different group sizes.

Response 4: Indeed. Cumulative totals may be reported in Table 2, along with averages and standard deviations, but not used for direct comparisons or statistics because of the different number of papers, as the reviewer commented. The comparisons were performed: (a) regarding the incidence of a paper, with or without news stories, to receive at least one of citations, FCR, RCR, Altmetric Score, blogs, policy sources, X posts, patent citations, peer review sites, Facebook, Weibo, Wikipedia, mentions in Q & A, Google+, Reddit, videos, faculty opinions, Mendeley, and CiteULike, data of Table 1 presented in page 5-6, first paragraph, lines 198-207, and odds ratios (OR) Figure 6, as described in page 9, first paragraph, lines 251-260; (b) regarding the averages of citations or mentions per parameter and paper, with or without news stories, presented in page 7, first paragraph, lines 213-228; (c) regarding the distributions of citations or altmetrics scores per journal and paper, with and without news stories, Figures 4 and 5, as described in page 7, second paragraph, lines 232-239; and (d) regarding the Pearson correlation of metrics with linear regression, Table 3 and Figure 7, as described in page 9-10, lines 266-276. The impact of the group size is evident in Figures 3-5, whereas the journals with the fewest papers in the study exhibited the most extensive heterogeneity. Only averages or proportional metrics were used in comparisons, except Pearson correlation and linear regression, as a different analytical path.

Comments 5: Results. Consider adding an explicit explanation of the color coding in the figure legend to ensure that readers understand what each color signifies.

Response 5: Thank you for pointing this out. I’ve, accordingly, added in the legend of Figure 6, page 9, line 265

“with black, p-value_{(likelihood ration chi-square)}>=0.05”

and of Figure 7, page 10-11, lines 283-294

“Figure 7. Pearson correlation and scatter plots with linear regression analysis of variables: (a) Pearson correlation analysis square of citations, FCR, altmetrics score, news stories, X posts, Facebook, Mendeley, and patents citations, with light green positive, almost linear correlation, dark green positive but less well-fitted to linear correlation, with black no correlation, with dark red negative but less well-fitted to linear correlation, and with red negative, almost linear correlation, according to the pallet of color coding and Pearson correlation coefficient values presented aside; (b) Scatter plots with linear regressions of the papers examined from top to bottom, citations as a function of FCR (left), Mendeley readers (middle) and news stories (right), news stories as a function of altmetrics score (left), X posts (middle) and Facebook (right), and X posts as a function of Facebook (left) as well as Mendeley as a function of FCR (right) (n=2020). The linear trendline is depicted in dark red along with the linear equation, and R-squared goodness-of-fit calculated by regression analysis are presented in the scatter plots.”

Comments 6: 4. Discussion. It’s important to mention that generalizability may be limited since the study only used high-impact journals.

Response 6: Thank you for this comment; the limitations paragraph in the Discussion was revised by adding the sentence, pages 12-13, last paragraph, lines 398-400

“Sixth, the generalizability of these findings may be limited because of the enrollment of highly-ranked biomedical scientific journals.”

4. Response to Comments on the Quality of English Language

Point 1: This paper contains several awkward expressions and areas where the English could be improved for clarity and readability. Some sentences are overly formal or wordy, making them harder to follow. Simplifying the language and rephrasing some of the more complex or convoluted sentences would help to make the paper more accessible and engaging for readers. Furthermore, ensuring consistency in tone and avoiding redundancy will enhance the overall flow of the paper.

 

For example, the sentence 'The public's increasing interest in scientific research, particularly in biomedicine, has led news media outlets to play a significant role in disseminating scientific advances' could be rephrased to something more direct and concise like, 'As public interest in biomedicine has grown, news media have taken on an important role in spreading scientific findings.' This revision reduces wordiness and improves clarity.

Response 1: The phrasing was reexamined.

Author Response File: Author Response.pdf