Papers

Do authors know who refereed their paper? A questionnaire survey

BMJ 1996; 313 doi: https://doi.org/10.1136/bmj.313.7066.1185 (Published 09 November 1996) Cite this as: BMJ 1996;313:1185
  1. S Wessely,
  2. T Brugha, associate editor, Psychological Medicinea,
  3. P Cowen, associate editor, Psychological Medicineb,
  4. L Smith, editorial assistant, Psychological Medicinec,
  5. E Paykel, editor, Psychological Medicinec
  1. a Department of Psychiatry, University of Leicester, Leicester LE5 4PW
  2. b Littlemore Hospital, Oxford OX4 4XN
  3. c Department of Psychiatry, Addenbrooke's Hospital, Cambridge CB2 2QQ
  1. Department of Psychological Medicine, King's College School of Medicine, London SE5 8AZ Simon Wessely, associate editor, Psychological Medicine. Correspondence to: Dr Wessely.
  • Accepted 23 March 1996

The process of peer review of medical research before publication has come under considerable scrutiny,1 although it would be fair to say that no better system has yet been devised. Much attention has been given to the question of whether or not referees produce better quality reports when blinded to the identity of the authors of the papers they are asked to review—the answer being a qualified yes.2 3 Another frequently asked question is whether or not referees should sign their opinions.4 However, to our knowledge no one has asked a simpler question: can authors guess the identity of the reviewer anyway?

Methods and results

Psychological Medicine is a leading international academic journal of psychiatry. For a five month period all those who submitted a manuscript to the journal were asked if they could guess the identity of the referees assigned to their paper (usually two or three), drawn from the pool of 580 available to the editors. All authors were sent a simple form asking them to write down the presumed identity of each referee and to indicate their degree of certainty on a four point scale, ranging from very uncertain (1) to certain (4). Alternatively the author could say that he or she had no idea of each referee's identity. The single page questionnaire was sent at the same time as the author was given the final decision about acceptance or rejection of the manuscript. Proportions were compared using the χ2 test without Yates's correction.

A total of 135 forms were sent out and 94 received back (70%). As expected,5 non-responders were more likely than responders to have had their paper rejected (44.0% (upsilon) 7.8%, χ2 = 19.9, df = 1, P<0.001). The total number of referees' reports for the 94 papers for which we received responses was 252. Of these 252 referees 15 were correctly identified (5.9%), 36 were incorrectly identified (14.3%), and in 201 (79.7%) the author had no idea of the referee's identity. Nearly all papers were reviewed by more than one referee (usually three) In four instances the author indicated the correct referee but against the wrong report. In two instances there were reasons to believe this was because of a misreading of the reference number and that the identity had been correctly guessed. If all those who had identified a referee of their paper but for the wrong report were given the benefit of the doubt then the correct number of guesses rose to 19 (7.5%).

The mean level of certainty for those who correctly identified the referee was 2.5 (lying between uncertain and fairly certain), compared with 1.8 for inaccurate guesses (between very uncertain and uncertain) (t = 2.55, df = 46, P = 0.014).

Using authors rather than referees as the denominator we found that those who correctly identified one or more referee were more likely to have had their paper accepted (χ2 = 4.61, df = 1, P = 0.03).

Comment

Anyone who has ever submitted a scientific paper will no doubt be familiar with the elaborate process of intuition and detection that goes into attempting to deduce the identity of the anonymous referee who has praised or damned the paper. This study suggests that even for a specialty journal such efforts are largely unrewarding and that most referees remain anonymous.

Footnotes

  • Funding None.

  • Conflict of interest None

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
View Abstract