Peer Review of Reviewers: The Author’s Perspective

Round  1

Reviewer 1 Report

English language and style: generally good, but the paper would benefit from another thorough proofread. I was occasionally distracted by incorrect or missing words (e.g., lines 39, 144 219)

Line 46 contains a generalization which strikes me as over-broad. Can we really say that dissatisfaction with reviewers' reports is considerable when the study adduced concerns only "journals dealing with chemistry and related subjects published in Serbia"? Consider qualifying this statement.

Lines 53-56: It's not clear to me why these journals and not others were included in your study. Is it because only these editors agreed to send your survey? Are there aspects of these journals that make them uniquely suited to the project? I'd like more explanation about these sites of study and why they were chosen.

Lines 102-105: These speculations may be correct but the data does not support them, as indicated by the uncertainty of "probably," which is used twice here. The data only show that authors of rejected papers have less confidence in their reviewers. To say more than this would require testimony from the survey respondents, i.e., answers to this question: why did you choose this answer and not another? This is perhaps a direction for future research.

Line 197: Why were participants limited to pre-defined answers? Why were they not given the option to supply their own reasons in free text? Without the additional explanation the authors may have provided, it's hard to draw any conclusions from an opaque category such as "personal reasons." What reasons? What evidence does an author have to suspect that personal reasons influenced the review process? For me, this is a limitation of the study and should be discussed as such.

Lines 216-235: This entire paragraph is out of place. It's important to contextualize your study, but I'm not sure why you've done so here. I expect to find information of this kind in an introduction or literature review. Here, it appears at the end of several pages of data analysis and feels incoherent as a result.   

Lines 252-254: You claim this study "can be used to target specific areas for improvement." Such as? What kind of improvements? How might they be implemented? I want more detail here, more explanation. A sense of direction for your future work on this subject.

Author Response

English language and style: generally good, but the paper would benefit from another thorough proofread. I was occasionally distracted by incorrect or missing words (e.g., lines 39, 144 219)

-We apologize for mistakes and we have corrected some. Unfortunately, we could not find a native English speaking colleague who could accept to edit and correct the manuscript in such a short time (10 days altogether for revision). Since in the e-mail which we received from the publisher it was said that “If the reviewers have suggested that your manuscript should undergo extensive English editing, please address this during revision.”, and the suggestion of the reviewer was “another thorough proofread”, we hope that we have addressed this point.

Line 46 contains a generalization which strikes me as over-broad. Can we really say that dissatisfaction with reviewers' reports is considerable when the study adduced concerns only "journals dealing with chemistry and related subjects published in Serbia"? Consider qualifying this statement.

-The statement was changed to avoid generalization.

Lines 53-56: It's not clear to me why these journals and not others were included in your study. Is it because only these editors agreed to send your survey? Are there aspects of these journals that make them uniquely suited to the project? I'd like more explanation about these sites of study and why they were chosen.

-The journals included in the study were those whose editors agreed to participate. We have sent invitation to more than 100 journals, trying to cover different scientific disciplines and journal origins, but positive responses were received from those listed in Table 1. This explanation is included in the revised manuscript.

Lines 102-105: These speculations may be correct but the data does not support them, as indicated by the uncertainty of "probably," which is used twice here. The data only show that authors of rejected papers have less confidence in their reviewers. To say more than this would require testimony from the survey respondents, i.e., answers to this question: why did you choose this answer and not another? This is perhaps a direction for future research.

-This part of the paragraph was rephrased to reduce speculations.

Line 197: Why were participants limited to pre-defined answers? Why were they not given the option to supply their own reasons in free text? Without the additional explanation the authors may have provided, it's hard to draw any conclusions from an opaque category such as "personal reasons." What reasons? What evidence does an author have to suspect that personal reasons influenced the review process? For me, this is a limitation of the study and should be discussed as such.

-Pre-defined answers were introduced in the survey to enable easier analysis of the data, but authors also had option to mark “other” as a choice of the influential factor and name it. At the same place in the survey, authors had opportunity to explain what they meant in a free form (as stated below Table 7). A category “personal reasons” was introduced, as editors occasionally receive comments from authors who suspect that negative comments/decisions originated from personal acquaintance between an author and a reviewer.  Limitation of the survey is, of course, inability to verify such statements. As suggested, this explanation is added in the revised manuscript.

Lines 216-235: This entire paragraph is out of place. It's important to contextualize your study, but I'm not sure why you've done so here. I expect to find information of this kind in an introduction or literature review. Here, it appears at the end of several pages of data analysis and feels incoherent as a result.

-Large part of this paragraph was moved to the Introduction section.

Lines 252-254: You claim this study "can be used to target specific areas for improvement." Such as? What kind of improvements? How might they be implemented? I want more detail here, more explanation. A sense of direction for your future work on this subject.

-The following was added at the end of the revised manuscript:

“Editors can pay more attention to reviewers and can introduce scoring of reviewers in order to confirm or eliminate reviewers for future tasks, or they can organize educational trainings, if necessary. Editors can improve instructions for reviewers or alter the reviewer invitation pattern in order to improve the efficiency (speed) and the effectiveness (acceptance of good quality papers, free of errors) of the scientific publishing process.”

Reviewer 2 Report

Most of the results seem obvious.

(lines 42-47) Having been an editor or AE for several journals, I agree that most reviews are poor.  Authors who have had their paper rejected always blame the reviewers.

  (61-63 versus 96-97) Is it 1 to 10 of 1-5?  If they are different, why were different scales used.

(75-77)  I would have liked to see a MUCH larger number of authors who had their papers rejected.  Such information could be helpful to journals and journal editors.

(table 1)  over 50% for one journal can bias the results--better to have a balance that reflects the number of papers.

It would be good to know how reviewers are selected for each journal; blindly by the editor or are they using author recommended reviewers?

(93-97) give a histogram of the results--not just the mean of 8.7.

(93-95) Not unexpected if more than 90% of the papers were accepted.

(table 3 and in text): a better description of the effects are needed--definitions.

(118-120)  Not surprising to me!  Given the sample size and degrees of freedom, theory would dictate this.

(table 3) include in the text a discussion of SE.

(table 3) Poorly structured table--do not need 6 digits.

(Fig. 1)  Needs a more descriptive title--what are the scores shown (eigenvector loadings?)

What were the values of the eigenvalues?

(138-141)  linear model I assume?  Is the criterion variable the 1 to 10?

(Table 4)  need definitions for the variables

(table 4)  should units of the variables be shown?  Are these the standardized coefficients?  To compare coefficients that have different units is troublesome.  Presentation here is not clear.

(table 7)  this appears to be back to the 1 to 5 scale???

(240-245)  seems obvious.

(243-244)  Do not understand.  I assume it is the authors, not the editor, who is rating competency?

Why was a more thorough verification performed?

Why was the distribution of journals better?

Author Response

Most of the results seem obvious.

(lines 42-47) Having been an editor or AE for several journals, I agree that most reviews are poor.  Authors who have had their paper rejected always blame the reviewers.

(61-63 versus 96-97) Is it 1 to 10 of 1-5?  If they are different, why were different scales used.

-Larger range of grades was offered to authors for the overall assessment of reviewers in order to enable their more precise evaluation. This sentence is included in the revised manuscript.

(75-77) I would have liked to see a MUCH larger number of authors who had their papers rejected.  Such information could be helpful to journals and journal editors.

-We entirely agree with the reviewer, it would be much better if we had more surveys completed by authors of rejected papers, but these authors did not seem to be willing to have anything more with the journal after receiving negative decision. We are aware that we could make firmer conclusions with larger number of such responses, but that did not happen.

(table 1) over 50% for one journal can bias the results--better to have a balance that reflects the number of papers.

-Again, we agree with the reviewer, but the number of responses from different journals was as given in Table 1.

It would be good to know how reviewers are selected for each journal; blindly by the editor or are they using author recommended reviewers?

-This information could possibly explain some of the findings, but taking into consideration our previous survey run among sub-editors (13) in the Journal of the Serbian Chemical Society (published in the Journal of the Serbian Chemical Society 2015, 80 (7) 959–969), it seems that each sub-editor develops his/her own reviewer invitation pattern, which is not directly correlated with the efficiency and the effectiveness of reviewing.

(93-97) give a histogram of the results--not just the mean of 8.7.

-A histogram is included in the revised manuscript, as Figure 1.

(93-95) Not unexpected if more than 90% of the papers were accepted.

(table 3 and in text): a better description of the effects are needed--definitions.

-Additional text is included in the revised manuscript, together with the reference which explains mixed-effects model in detail.

(118-120) Not surprising to me! Given the sample size and degrees of freedom, theory would dictate this.

(table 3) include in the text a discussion of SE.

-Standard error (SE) is a defined value (calculated from the data) and we do not see much space to discuss.

(table 3) Poorly structured table--do not need 6 digits.

-Table 3 is corrected and numbers contain 3 decimal places, as in other tables.

(Fig. 1)  Needs a more descriptive title--what are the scores shown (eigenvector loadings?)

-We changed the title to make it more informative.

What were the values of the eigenvalues?

-They are given in the text of the revised manuscript, when first introducing the factors.

(138-141) linear model I assume?  Is the criterion variable the 1 to 10?

-The model is explained in the added reference (26). We ran a factor analysis on seven questions detailing the authors’ opinion, each response was graded on the scale 1-5 (as explained in Table 2).

(Table 4) need definitions for the variables

-The variables are already defined in the text, so we did not do that again in the table.

(table 4) should units of the variables be shown?  Are these the standardized coefficients?  To compare coefficients that have different units is troublesome.  Presentation here is not clear.

-None of the variable we use has, rigorously speaking, units: the number of weeks and of reports are pure numbers; the final decision is binary (accepted/rejected); speed: on time and speed: fast also are binary (no/yes); the factors are pure numbers by design.

(table 7) this appears to be back to the 1 to 5 scale???

-Yes, only the overall assessment grade could be chosen on the scale 1-10; all others were on the scale 1-5 (as indicated in Tables 2 and 7, bellow questions).

(240-245) seems obvious.

(243-244) Do not understand. I assume it is the authors, not the editor, who is rating competency?

-One of the questions posed to authors in the survey was: “Do you think that reviewer was competent to review your paper?” (Table 2) – Yes, authors rated the competency. A revised text was corrected to avoid uncertainty who is rating competency.

Why was a more thorough verification performed?

-We are not certain what is the reviewer thinking about here.

Why was the distribution of journals better?

-We do not understand what the reviewer is referring to.

Reviewer 3 Report

The experimental results of this research agree with previous results in the literature. In the following,  two different examples.
Fist, [García, J.A., Rodriguez-Sánchez, R. & Fdez-Valdivia, J. Scientometrics (2016) 106: 967. https://doi.org/10.1007/s11192-015-1827-8] presented  a  Bayesian-based formal study on authors' overconfidence, self-serving bias, and lack of learning at the peer review stage. There, a  Bayesian author processes all information perfectly, and  arrives at the best conclusions that can be drawn from the data: (i) she assumes that any given editorial decision has different possible causes; (ii) how she interprets any editorial decision  depends on information she already had;  and  (iii) she uses the mathematical laws regarding probability to choosing between her subjective beliefs. Therefore, Bayesian-rational individuals maximize expected  profit based on their own belief.
For the  high-ability authors  in a very active scientific field, this  model provided rational explanations for a  self-serving bias: (i) Bayesian-rational authors are relatively overconfident about their likelihood of manuscript   acceptance, whereas authors who play the role of referees have less  confidence in manuscripts of other authors; (ii)   if the final disposition of his  manuscript  is acceptance, the Bayesian-rational author almost surely attributes this decision more to his  ability; (iii)   when the final disposition is rejection, the Bayesian-rational author almost surely attributes this decision more to negative bias in peer review; (iv)  some rational authors do  not learn as much from the critical  reviewers'  comments in case of rejection as they should from the journal editor's  perspective.
Second, because  rejection undermines self-confidence and self-worth, authors often seek an external factor that can be blamed for this negative decision. When  manuscripts are accepted for publication, however, authors can bolster their self-esteem by attributing acceptance  to the author's ability to write quality papers.
 In particular, when explaining  the causes of manuscript rejections, they stress external,situational factors such as different forms of bias in peer review.
[García, J.A., Rodriguez-Sánchez, R. & Fdez-Valdivia, J. Scientometrics (2016) 109: 1377. https://doi.org/10.1007/s11192-016-2079-y]  presented a formal model to study author's beliefs about the causes of manuscript rejection in peer review. There, a Bayesian author could have different beliefs: (1) the hypothesis that rejection was  caused by negative bias in peer review, x=Bias;  or (2) the hypothesis that rejection was caused by the lack of author's ability to write a quality manuscript, x=Ability.
Assume that the author initially believes that these hypotheses are  equally likely to be true  in peer review.
Suppose that the Bayesian author submits a series of papers to  peer-reviewed journals. As a result,  he or she then receives  review signals correlated with the true causes of manuscript rejection.
The problem is that there exists a positive probability that the author  misinterprets  review signals  that are counter to the hypothesis he  currently believes is more likely.
For instance,  authors could misread those  review signals   that conflict with the hypothesis that rejection was caused by negative bias, as supporting their current beliefs. Then, they suffer from confirmatory bias, and the Bayesian author is unaware that she is misreading editor's decisions.
  [García, J.A., Rodriguez-Sánchez, R. & Fdez-Valdivia, J. Scientometrics (2016) 109: 1377. https://doi.org/10.1007/s11192-016-2079-y] presented  a free online tool that  helps authors (who suffer from confirmatory bias) to experiment with their beliefs
 about the causes of rejection, given a series of review signals that  author perceived at different review processes. This web application predicts  whether the author is  overconfident or underconfident in  his belief  that manuscript rejection was caused by bias in peer review.

Some  issues:

- The introduction can be improved to provide further background and include all relevant references.

The estimation of the mixed-effect models should be clarify in the presentation.  More details should be given to improve the soundness of the data analysis.

Author Response

The experimental results of this research agree with previous results in the literature. In the following, two different examples.

Fist, [García, J.A., Rodriguez-Sánchez, R. & Fdez-Valdivia, J. Scientometrics (2016) 106: 967. https://doi.org/10.1007/s11192-015-1827-8] presented a Bayesian-based formal study on authors' overconfidence, self-serving bias, and lack of learning at the peer review stage. There, a Bayesian author processes all information perfectly, and arrives at the best conclusions that can be drawn from the data: (i) she assumes that any given editorial decision has different possible causes; (ii) how she interprets any editorial decision depends on information she already had; and (iii) she uses the mathematical laws regarding probability to choosing between her subjective beliefs. Therefore, Bayesian-rational individuals maximize expected profit based on their own belief.

For the  high-ability authors  in a very active scientific field, this  model provided rational explanations for a  self-serving bias: (i) Bayesian-rational authors are relatively overconfident about their likelihood of manuscript   acceptance, whereas authors who play the role of referees have less  confidence in manuscripts of other authors; (ii)   if the final disposition of his  manuscript  is acceptance, the Bayesian-rational author almost surely attributes this decision more to his  ability; (iii)   when the final disposition is rejection, the Bayesian-rational author almost surely attributes this decision more to negative bias in peer review; (iv)  some rational authors do  not learn as much from the critical  reviewers'  comments in case of rejection as they should from the journal editor's  perspective.

Second, because rejection undermines self-confidence and self-worth, authors often seek an external factor that can be blamed for this negative decision. When manuscripts are accepted for publication, however, authors can bolster their self-esteem by attributing acceptance to the author's ability to write quality papers.

 In particular, when explaining the causes of manuscript rejections, they stress external,situational factors such as different forms of bias in peer review.

[García, J.A., Rodriguez-Sánchez, R. & Fdez-Valdivia, J. Scientometrics (2016) 109: 1377. https://doi.org/10.1007/s11192-016-2079-y] presented a formal model to study author's beliefs about the causes of manuscript rejection in peer review. There, a Bayesian author could have different beliefs: (1) the hypothesis that rejection was caused by negative bias in peer review, x=Bias; or (2) the hypothesis that rejection was caused by the lack of author's ability to write a quality manuscript, x=Ability.

Assume that the author initially believes that these hypotheses are equally likely to be true in peer review.

Suppose that the Bayesian author submits a series of papers to peer-reviewed journals. As a result, he or she then receives review signals correlated with the true causes of manuscript rejection.

The problem is that there exists a positive probability that the author misinterprets review signals that are counter to the hypothesis he currently believes is more likely.

For instance, authors could misread those review signals   that conflict with the hypothesis that rejection was caused by negative bias, as supporting their current beliefs. Then, they suffer from confirmatory bias, and the Bayesian author is unaware that she is misreading editor's decisions.

  [García, J.A., Rodriguez-Sánchez, R. & Fdez-Valdivia, J. Scientometrics (2016) 109: 1377. https://doi.org/10.1007/s11192-016-2079-y] presented a free online tool that helps authors (who suffer from confirmatory bias) to experiment with their beliefs

About the causes of rejection, given a series of review signals that author perceived at different review processes. This web application predicts whether the author is overconfident or underconfident in his belief that manuscript rejection was caused by bias in peer review.

-Suggested articles and comments are included in our discussion of the results.

Some issues:

The introduction can be improved to provide further background and include all relevant references.

-Additional references are included in Introduction.

The estimation of the mixed-effect models should be clarify in the presentation.  More details should be given to improve the soundness of the data analysis.

-We hope that additions which we made in the revised manuscript respond to this request.

Reviewer 4 Report

The paper discusses an important problem of the quality of the peer review process. The authors use the questionnaire method to obtain the authors perspective.

In my opinion, two points should be clarified:

The description of the journals choice. The best methods would be a random choice. In the discussed case it seems that it is a regional based choice - it should be clearly stated. 

The second point is related to the main finding: that the authors of accepted papers are more likely to give a high score to the review while the rejected ones are of opposite opinion. I think that the phenomenon is quite obvious and it should be consulted to a psychologist to find the appropriate reference.

Author Response

The paper discusses an important problem of the quality of the peer review process. The authors use the questionnaire method to obtain the authors perspective.

In my opinion, two points should be clarified:

The description of the journals choice. The best methods would be a random choice. In the discussed case it seems that it is a regional based choice - it should be clearly stated.

-The journals included in the study were those whose editors agreed to participate. We have sent invitation to more than 100 journals, trying to cover different scientific disciplines and journal origins, but positive responses were received from those listed in Table 1. This explanation is included in the revised manuscript.

The second point is related to the main finding: that the authors of accepted papers are more likely to give a high score to the review while the rejected ones are of opposite opinion. I think that the phenomenon is quite obvious and it should be consulted to a psychologist to find the appropriate reference.

-Additional references and explanations are given in our discussion of the results.

Round  2

Reviewer 1 Report

I appreciate the difficulty of locating a native English speaker to correct the manuscript. Still, I worry that stilted language or the occasional typo may damage a reader's confidence in the work and/or the journal in which it appears. For example, the very first sentence ("Peer review is considered as a cornerstone of science...") would be greatly improved with a little concision ("Peer review IS a cornerstone of science). Also, it seems new errors were introduced during revision (e.g., line 269, "Despite of the limitations of the study..."). I don't think these errors are serious enough to bar acceptance, but I urge the authors to do their utmost to ensure the grammatical and stylistic soundness of their writing. 

Author Response

The manuscript was checked and corrected by a native English speaking colleague, as suggested by the Reviewer.