How Many Papers Should Scientists Be Reviewing? An Analysis Using Verified Peer Review Reports

Round 1

Reviewer 1 Report

Interesting viewpoint, but he makes several assertions and claims that are not supportable. His data also does not support some of these claims.

For example, in his first paragraph, he has the following sentence  “If the number of reviews required by the peer-review system increases faster than the number of reviews being conducted, one would expect a decline in the quality of reviews and/or a lengthening of the publication process.” This sentence contains vastly different arguments. A decline in the quality of reviews is very different than the length of time it takes to receive a review. Claiming both in the same sentence is not supported by his data or with experience. This sentence would be better if it just claimed one idea. If I were revising, I think the lengthening of the publication process is the most relevant of the two claims.

Making the claim “proliferation of open-access options are also believed to be responsible for a decline in the general quality of reviews” while submitting this article to one of those open-access journals seems to be a strange choice.  Why insult open-access options in an open-access journal?

There are also assumptions made about people reviewing the same standard amount of articles. I have reviewed 14 articles this year for several different journals. I think this number is high, although I haven’t asked my colleagues. I do know of one other Associate Professor in my department that reviewed 1 article this year as the person had to announce it at a departmental meeting. So, if I am reviewing 14 articles a year and a fellow professor is only reviewing 1, that would seem to me to be part of the issue. But, this article does not address the disparity between how many articles people are reviewing.

There are several professors in my department that have never reviewed articles and feel as if being a reviewer is beneath them. I have taught in many different universities and have encountered many that feel this same way. That once someone reaches Associate Professor rank, they do not have to be reviewing. Of course, this could be discipline-specific. I am in the Humanities. If this is discipline-specific, making that distinction in the article would be helpful.

There are many reasons the review process is slow and this article does not address these reasons. For example, the email asking me to review this article went to an email address of a former university that I no longer teach at. I had accessed that account looking for an old email and saw the request. If I had not accessed the email, I would not be writing this review, and the process would be slowed down as someone else would have been asked. For a different journal, I simply forgot that I had a review due and ended up submitting my review several months after the deadline. In my apologetic email to the editor, I found out that every reviewer had turned in their review late. This is probably more common than most people think and is a valid reason for why the review process is taking longer. A friend of mine submitted an article and after eight months of the article listed as “considering,” the friend emailed the editor. As she discovered, her article wasn’t being reviewed because the editors could not find someone with enough expertise to review her submission. Again, all of these are valid reasons why the review process is slowed down and none of these reasons are included in this article.

The results section felt very truncated and I would like to see more added to the results. I would encourage the author to include more explanation of what the charts are showing instead of just including the charts and not writing out the results. Some of us, including me, would prefer to see the researchers' interpretation of the results and not just muddle through reading a chart.

Why only include three boxplots? It would be nice to see the data presented in different ways.

Overall, I thought this article provided an interesting take on publications and review rates. With some revision, the article would be ready for publication.

Author Response

Thank you for the opportunity to modify our manuscript for possible publication in Publications.  We thank the reviewers for their very helpful comments and the we believe the manuscript has been greatly improved. As per their comments, we have completely re-analysed the data using a more appropriate approach and have also made other improvements to various aspects of the manuscript. We have adopted the reviewer comments and made corresponding changes in the manuscript. Our responses to the reviewer, along with changes made, are in red font below.

Reviewer #1:

Interesting viewpoint, but he makes several assertions and claims that are not supportable. His data also does not support some of these claims.

For example, in his first paragraph, he has the following sentence  “If the number of reviews required by the peer-review system increases faster than the number of reviews being conducted, one would expect a decline in the quality of reviews and/or a lengthening of the publication process.” This sentence contains vastly different arguments. A decline in the quality of reviews is very different than the length of time it takes to receive a review. Claiming both in the same sentence is not supported by his data or with experience. This sentence would be better if it just claimed one idea. If I were revising, I think the lengthening of the publication process is the most relevant of the two claims.

This is a valid point, and I have removed the statement on the quality of reviews.

Making the claim “proliferation of open-access options are also believed to be responsible for a decline in the general quality of reviews” while submitting this article to one of those open-access journals seems to be a strange choice.  Why insult open-access options in an open-access journal?

The intention was not to insult open-access, this is a statement from the cited paper. Since both other reviews identified this claim as an issue, I have removed it.

There are also assumptions made about people reviewing the same standard amount of articles. I have reviewed 14 articles this year for several different journals. I think this number is high, although I haven’t asked my colleagues. I do know of one other Associate Professor in my department that reviewed 1 article this year as the person had to announce it at a departmental meeting. So, if I am reviewing 14 articles a year and a fellow professor is only reviewing 1, that would seem to me to be part of the issue. But, this article does not address the disparity between how many articles people are reviewing.

This is a good point. I have added a statement in the introduction highlight the possible disparity of numbers of reviews conducted by various academics. See lines 35 – 38.

There are several professors in my department that have never reviewed articles and feel as if being a reviewer is beneath them. I have taught in many different universities and have encountered many that feel this same way. That once someone reaches Associate Professor rank, they do not have to be reviewing. Of course, this could be discipline-specific. I am in the Humanities. If this is discipline-specific, making that distinction in the article would be helpful.

While this study examines mainly scientific disciplines, I agree that introducing the notion of some academics thinking that others can simply pick up the slack is important. I have added a statement on this topic. See statement lines 167 – 168.

There are many reasons the review process is slow and this article does not address these reasons. For example, the email asking me to review this article went to an email address of a former university that I no longer teach at. I had accessed that account looking for an old email and saw the request. If I had not accessed the email, I would not be writing this review, and the process would be slowed down as someone else would have been asked. For a different journal, I simply forgot that I had a review due and ended up submitting my review several months after the deadline. In my apologetic email to the editor, I found out that every reviewer had turned in their review late. This is probably more common than most people think and is a valid reason for why the review process is taking longer. A friend of mine submitted an article and after eight months of the article listed as “considering,” the friend emailed the editor. As she discovered, her article wasn’t being reviewed because the editors could not find someone with enough expertise to review her submission. Again, all of these are valid reasons why the review process is slowed down and none of these reasons are included in this article.

I agree that the issues with publication wait times are not simply a product of too few reviewers. I have added in various parts of the manuscript that editorial processes are also to blame. See line 30, 50.

The results section felt very truncated and I would like to see more added to the results. I would encourage the author to include more explanation of what the charts are showing instead of just including the charts and not writing out the results. Some of us, including me, would prefer to see the researchers' interpretation of the results and not just muddle through reading a chart.

I have added additional detail here describing the results. See lines 130 – 141.

Why only include three boxplots? It would be nice to see the data presented in different ways.

I have added a figure on the relationship between author number and review ratio to try and address this comment. See lines 149 – 156.

Overall, I thought this article provided an interesting take on publications and review rates. With some revision, the article would be ready for publication.

I am happy the reviewer thought this was an interesting take.

Reviewer 2 Report

The author presents an empirical investigation of the peer review ratio's of authors. They present an interesting empirical approach that combines data sources for the investigation of the research question. I am particularly intrigued and excited by the point of using Publons data about the peer reviews and the total number of publications in a specific year. Of course there are issues with Publons, but those limitations are to a degree included in the discussions of the results. I think this is a superb way of investigating the research question.

From a methodological perspective beyond the data collection, I cannot vouch for the content. It seems like the author makes a core assumption about how many reviews occur (lines 91-93), which requires some more investigation. Either how that assumption influences the results or whether there's any indication that this is reasonable. Moreover, it seems like the whole point of the paper obfuscates the simple point that the author makes: For each paper you co-author, you need to review the same amount (1/4th of a review for a co-authorship on a 4 author paper). I do not see why all the information surrounding that parsimonious point, result in this paper. That point would need to be addressed in a revision. In the conclusion, the author jumps to increased publication times if reviewing rates are higher, but that conclusion is unwarranted given that reviewing times are not at all included in this empirical investigation.

My main non-methodological concern with the manuscript is the normative research question ("How much *should* authors review") and the lack of an evaluation of how the peer review system influences this normative evaluation. In the current evaluation, the autonomy for review is completely placed on authors (implicitly). Authors do not have complete control over the amount of reviews they do, and it would be responsible to take that into account in the paper's normative evaluation. If this is to be a normative claim, I recommend including a more complete analysis of how the individual's responsibility and the system's limitations interact.

All in all, it feels like this manuscript does not evaluate the question thoroughly enough.

Additional points (not minor per se)

"the proliferation of open-access options are also believed to be responsible for a decline in the general quality of reviews" this claim is unwarranted and the reference does not provide any support either. The exponential growth of publications and the lack of capacity increase in scholarly research is more likely to be a problem. The author misrepresents the current empirical investigation as a meta-analysis, which it is not (it does not provide the synthesis of multiple studies) The data are not available for verification; I recommend the author include those, given that it's archival data and would help reanalysis I am a bit concerned about order effects in the WoS investigation, due to just taking the first 500 items. I know about this limitation from personal experience, and I might recommend the author to use the CrossRef API in the future. the counts presented in the results are presented as a continuous variable, which provides a false sense of information fidelity.

Author Response

Thank you for the opportunity to modify our manuscript for possible publication in Publications.  We thank the reviewers for their very helpful comments and the we believe the manuscript has been greatly improved. As per their comments, we have completely re-analysed the data using a more appropriate approach and have also made other improvements to various aspects of the manuscript. We have adopted the reviewer comments and made corresponding changes in the manuscript. Our responses to the reviewers, along with changes made, are in red font below.

The author presents an empirical investigation of the peer review ratio's of authors. They present an interesting empirical approach that combines data sources for the investigation of the research question. I am particularly intrigued and excited by the point of using Publons data about the peer reviews and the total number of publications in a specific year. Of course there are issues with Publons, but those limitations are to a degree included in the discussions of the results. I think this is a superb way of investigating the research question.

I am happy to hear the reviewer finds this approach interesting.

From a methodological perspective beyond the data collection, I cannot vouch for the content. It seems like the author makes a core assumption about how many reviews occur (lines 91-93), which requires some more investigation. Either how that assumption influences the results or whether there's any indication that this is reasonable. Moreover, it seems like the whole point of the paper obfuscates the simple point that the author makes: For each paper you co-author, you need to review the same amount (1/4th of a review for a co-authorship on a 4 author paper). I do not see why all the information surrounding that parsimonious point, result in this paper. That point would need to be addressed in a revision. In the conclusion, the author jumps to increased publication times if reviewing rates are higher, but that conclusion is unwarranted given that reviewing times are not at all included in this empirical investigation.

I agree that this is an assumption, however, I do not see how the assumption that at least 2 reviews are necessary per publication is unreasonable. Peer-review by definition requires at least two viewpoints for the editor to make a decision on whether the work is appropriate or not. While revisions after initial review do not necessarily need two reviews, the initial stage certainly does.

Yes, the point that I’m trying to make is simple. However, there is currently no empirical data that supports this idea, and what I did here was provide that in a simple manner. I also believe I analysed these data in as simple a manner as possible, and have kept the discussion and introduction succinct for that reason.

The point re publication times has been made by other reviewers, and I agree the link needed to be made more explicit. Even if reviewing times are not explicitly included, I believe it’s a logical step to link more available reviewers willing to review to less publication time, since many publications (cited in this study) highlight that editors have difficulty finding reviews willing to review. See lines 40 – 41.

My main non-methodological concern with the manuscript is the normative research question ("How much *should* authors review") and the lack of an evaluation of how the peer review system influences this normative evaluation. In the current evaluation, the autonomy for review is completely placed on authors (implicitly). Authors do not have complete control over the amount of reviews they do, and it would be responsible to take that into account in the paper's normative evaluation. If this is to be a normative claim, I recommend including a more complete analysis of how the individual's responsibility and the system's limitations interact.

I agree with the reviewer’s comment that authors are not solely responsible for the number of reviews they conduct, and that this depends on their fields as well as the editors. I have made comments in the introduction that relate to the pressure on editors, and have also outlined that arts and humanities, which are likely to have very different reviewing standards, are not explicitly included in this study. See lines 30 and 167 – 170.

All in all, it feels like this manuscript does not evaluate the question thoroughly enough.

I feel this statement is in contradiction with the reviewer’s previous comment “I do not see why all the information surrounding that parsimonious point, result in this paper”. I cannot both be succinct to make a parsimonious point, and be more thorough and add more detail.

Additional points (not minor per se)

"the proliferation of open-access options are also believed to be responsible for a decline in the general quality of reviews" this claim is unwarranted and the reference does not provide any support either. The exponential growth of publications and the lack of capacity increase in scholarly research is more likely to be a problem.

Reviewer #3 makes a similar comment, as a result I have removed this statement.

The author misrepresents the current empirical investigation as a meta-analysis, which it is not (it does not provide the synthesis of multiple studies)

I agree this was inappropriate, and I have re-worded the title as a result to simply an analysis. It now states ‘An analysis using verified peer-review reports’ rather than ‘a meta-analysis’.

 The data are not available for verification; I recommend the author include those, given that it's archival data and would help reanalysis I am a bit concerned about order effects in the WoS investigation, due to just taking the first 500 items. I know about this limitation from personal experience, and I might recommend the author to use the CrossRef API in the future. the counts presented in the results are presented as a continuous variable, which provides a false sense of information fidelity.

I have now provided the summary data from which all data are extracted. I unfortunately cannot provide the 500 item summaries, as these are extracted from text files and interpreted by bibliometrix. Crossref API was not appropriate here, as it covers a smaller number of publications than both Web of Science and Scopus, and it was imperative to obtain as many publications as possible. I do agree that Crossref has many benefits, and it was in fact my first preference.

I understand the misgivings about the first 500 publications, however, these are ordered in date of publication. Therefore, it is unlikely that these would bias results. I have specified this in the methods lines 98 – 99.

 

Reviewer 3 Report

Your article gives evidence in confirmation of what many authors assume: that they should review a paper for each one they publish. Of course it's one thing to assume the need for such practice and another to see it empirically demonstrated. Even considering its limitations, I hope your article will, at a minimum, stimulate greater participation in Publons and the larger project of collating verified reviews from across all disciplines.

Below I list a few suggestions that I believe will improve your article.

 

Line 10: This is the first of two constructions in which "here," which I read in the present tense, conflicts with a verb in the past tense. Easily revised by deleting 'here.'

Lines 18, 159: Not 'we' but I. Why give credit to phantom co-authors?  :)

Lines 21, 132: I would soften this claim by narrowing the focus to time spent on the review task. In other words, there are many factors which contribute to lengthy publication processing times, such as instances when authors are slow to make recommended changes.

Line 34: It would be helpful if you added a sentence here to explain why readers "would expect" such consequences. Explain the relationship between increasing reviews and decreasing review quality.

Line 37: "other aspects of academia" ... such as? Make the consequences concrete.

Lines 42-43: This op-ed was vigorously challenged by Michael Eisen, Mike Taylor, and others (see: http://www.michaeleisen.org/blog/?p=1654). Eisen and Taylor are hardly disinterested observers; both are well-known OA advocates. Nevertheless, this is a shaky foundation on which to base so provocative a claim. I would like to see a more reliable source here.

Lines 49-50: subject-verb disagreement (appearance/have)

Line 58: Use i.e. here, or just substitute the word you want, 'publications', for 'contribution.'

Line 65: On line 132 you acknowledge that review ratios vary by discipline. Might it be worth noting that your study under-counts contributions from the Arts & Humanities owing to the many fewer verified reviews from those fields?

Line 85: Your description of Bibliometrix should follow the sentence which introduces the tool. In its present place, this sentence doesn't logically follow the one which precedes it.

Line 89: Here you describe two different kinds of reviews, verified and unverified, but the two aren't distinguished very well, which may confuse readers. I had to read this sentence two or three times.

Line 97: I think you should make this an explicit recommendation in the conclusion of your paper. Such review reports make your meta-analysis possible, and better data will improve confidence in your findings (not to mention the future studies such data might inspire). Verified review databases such as Publons are a good thing for scholarly communication; your paper can encourage wider participation by author and journal alike.

Line 136: What is the role of the journal editor in reducing review time? This is perhaps outside the scope of your paper, but it's demoralizing to review papers that should have been desk-rejected. As P.K. Friedman put it, "I’ve come to the conclusion that a lot of journal editors are shirking their responsibilities by sending papers out for peer review that should never have gotten that far." (https://savageminds.org/2012/11/08/desk-reject/) I suppose I would qualify your conclusion: authors should review at least 1 properly qualified publication per article they produce.

Line 145: offset the "coupled with" clause with commas

Line 158: subj/verb dis. (the number/are)

Line 165: As I mentioned above, I recommend strengthening your conclusion by making concrete recommendations. Is it true that you only hope your guidelines will be used to improve peer review, or do you believe they should be used for that purpose?

Author Response

Thank you for the opportunity to modify our manuscript for possible publication in Publications.  We thank the reviewers for their very helpful comments and the we believe the manuscript has been greatly improved. As per their comments, we have completely re-analysed the data using a more appropriate approach and have also made other improvements to various aspects of the manuscript. We have adopted the reviewer comments and made corresponding changes in the manuscript. Our responses to the reviewers, along with changes made, are in red font below.

Line 10: This is the first of two constructions in which "here," which I read in the present tense, conflicts with a verb in the past tense. Easily revised by deleting 'here.'

I deleted ‘here’.

Lines 18, 159: Not 'we' but I. Why give credit to phantom co-authors?  :)

It’s hard changing habits! I changed these to the first person.

Lines 21, 132: I would soften this claim by narrowing the focus to time spent on the review task. In other words, there are many factors which contribute to lengthy publication processing times, such as instances when authors are slow to make recommended changes.

I have removed the last sentence in the abstract to soften this claim, and specified in the discussion that this would lead to greater pools of reviewers willing to review, and as a result lower processing times.

Line 34: It would be helpful if you added a sentence here to explain why readers "would expect" such consequences. Explain the relationship between increasing reviews and decreasing review quality.

I have now specified that this would be because the reviewer pool is smaller than the required number of reviewers for the increase in publication count. See lines 40 – 41.

Line 37: "other aspects of academia" ... such as? Make the consequences concrete.

I have specified that this stifles creative thought and discourages long-term studies. See lines 40 – 45.

Lines 42-43: This op-ed was vigorously challenged by Michael Eisen, Mike Taylor, and others (see: http://www.michaeleisen.org/blog/?p=1654). Eisen and Taylor are hardly disinterested observers; both are well-known OA advocates. Nevertheless, this is a shaky foundation on which to base so provocative a claim. I would like to see a more reliable source here.

I agree this could be seen as provocative and agree with the reviewer’s comments re OA. I have deleted the portion of the sentence that highlights OA as a problem to remove the issue.

Lines 49-50: subject-verb disagreement (appearance/have)

I have reworded this sentence to correct this issue and improve word flow. See lines 59 – 60.

Line 58: Use i.e. here, or just substitute the word you want, 'publications', for 'contribution.'

I have substituted ‘publication’ for ‘contribution’.

Line 65: On line 132 you acknowledge that review ratios vary by discipline. Might it be worth noting that your study under-counts contributions from the Arts & Humanities owing to the many fewer verified reviews from those fields?

I have now specified that this study does not directly examine arts and humanities, partially due to the low number of verified reviews in those fields. See lines 168 – 170.

Line 85: Your description of Bibliometrix should follow the sentence which introduces the tool. In its present place, this sentence doesn't logically follow the one which precedes it.

I agree with this and have moved the sentence to after the package is first mentioned.

Line 89: Here you describe two different kinds of reviews, verified and unverified, but the two aren't distinguished very well, which may confuse readers. I had to read this sentence two or three times.

I have now reworked this paragraph with an explicit opening sentence to make this step clearer. See lines 86 – 87.

Line 97: I think you should make this an explicit recommendation in the conclusion of your paper. Such review reports make your meta-analysis possible, and better data will improve confidence in your findings (not to mention the future studies such data might inspire). Verified review databases such as Publons are a good thing for scholarly communication; your paper can encourage wider participation by author and journal alike.

This is a good point, and I have now highlighted that the benefits of verified reviews for studies like this one should encourage more researchers to verify their reviews. See lines 168 – 170.

Line 136: What is the role of the journal editor in reducing review time? This is perhaps outside the scope of your paper, but it's demoralizing to review papers that should have been desk-rejected. As P.K. Friedman put it, "I’ve come to the conclusion that a lot of journal editors are shirking their responsibilities by sending papers out for peer review that should never have gotten that far." (https://savageminds.org/2012/11/08/desk-reject/) I suppose I would qualify your conclusion: authors should review at least 1 properly qualified publication per article they produce.

While I agree with the reviewer’s comments, I feel it’s not part of this study to delve into discussing the editorial process as well. It is a good point, however, so I have added some points underlining that there are issues with the editorial process as well due to the high publication count. See lines 30 and 50 – 52.

Line 145: offset the "coupled with" clause with commas

I have added a comma here.

Line 158: subj/verb dis. (the number/are)

I have changed the statement to ‘numbers are fairly large’.

Line 165: As I mentioned above, I recommend strengthening your conclusion by making concrete recommendations. Is it true that you only hope your guidelines will be used to improve peer review, or do you believe they should be used for that purpose?

I have now tried to make the conclusion stronger. I want to avoid making too strong a statement, however, since the number of ‘criteria’ (h-index etc.) that academics must meet is already high, but the reviewer is correct in stating that I should be making a stronger statement. See lines 200 – 202.

Round 2

Reviewer 1 Report

Thank you for addressing all the comments and suggestions that I made on your first review. 

Author Response

I am happy to hear that I have addressed all the reviewer's comments.

Reviewer 2 Report

The author provides several decent rebuttals of my points. I have adjusted my ratings accordingly. I don't see major flaws in the work, but I also don't find it particularly strong. I adjusted my recommendation on the idea that any non-flawed work should be publishable.

I remain of the opinion that a normative evaluation of author's behavior's requires a more thorough evaluation of the other side of the equation too, and I sincerely recommend the author to provide a more balanced analysis if the question is how much authors should be reviewing.

Author Response

In an effort to appease the reviewer's concerns, I have added a paragraph in the introduction (lines 56 - 69) that outlines the other aspects of peer-review that may affect or determine the number of publications that researchers should review. I highlight some examples of what these may be, but also underline that understanding the quantitative requirement of the peer-review system (how many are needed for the system to function) may aid future research into these other aspects that may be more difficult to quantify.