Papers

Do authors know who refereed their paper? A questionnaire survey

^a Department of Psychiatry, University of Leicester, Leicester LE5 4PW, ^b Littlemore Hospital, Oxford OX4 4XN, ^c Department of Psychiatry, Addenbrooke's Hospital, Cambridge CB2 2QQ

Department of Psychological Medicine, King's College School of Medicine, London SE5 8AZ Simon Wessely, associate editor, Psychological Medicine. Correspondence to: Dr Wessely.

The process of peer review of medical research before publication has come under considerable scrutiny,¹ although it would be fair to say that no better system has yet been devised. Much attention has been given to the question of whether or not referees produce better quality reports when blinded to the identity of the authors of the papers they are asked to review--the answer being a qualified yes.^{2 3} Another frequently asked question is whether or not referees should sign their opinions.⁴ However, to our knowledge no one has asked a simpler question: can authors guess the identity of the reviewer anyway?

Methods

and results

Psychological Medicine is a leading international academic journal of psychiatry. For a five month period all those who submitted a manuscript to the journal were asked if they could guess the identity of the referees assigned to their paper (usually two or three), drawn from the pool of 580 available to the editors. All authors were sent a simple form asking them to write down the presumed identity of each referee and to indicate their degree of certainty on a four point scale, ranging from very uncertain (1) to certain (4). Alternatively the author could say that he or she had no idea of each referee's identity. The single page questionnaire was sent at the same time as the author was given the final decision about acceptance or rejection of the manuscript. Proportions were compared using the ² test without Yates's correction.

A total of 135 forms were sent out and 94 received back (70%). As expected,⁵ non-responders were more likely than responders to have had their paper rejected (44.0% (upsilon) 7.8%, ² = 19.9, df = 1, P<0.001). The total number of referees' reports for the 94 papers for which we received responses was 252. Of these 252 referees 15 were correctly identified (5.9%), 36 were incorrectly identified (14.3%), and in 201 (79.7%) the author had no idea of the referee's identity. Nearly all papers were reviewed by more than one referee (usually three) In four instances the author indicated the correct referee but against the wrong report. In two instances there were reasons to believe this was because of a misreading of the reference number and that the identity had been correctly guessed. If all those who had identified a referee of their paper but for the wrong report were given the benefit of the doubt then the correct number of guesses rose to 19 (7.5%).

The mean level of certainty for those who correctly identified the referee was 2.5 (lying between uncertain and fairly certain), compared with 1.8 for inaccurate guesses (between very uncertain and uncertain) (t = 2.55, df = 46, P = 0.014).

Using authors rather than referees as the denominator we found that those who correctly identified one or more referee were more likely to have had their paper accepted (² = 4.61, df = 1, P = 0.03).

Comment

Anyone who has ever submitted a scientific paper will no doubt be familiar with the elaborate process of intuition and detection that goes into attempting to deduce the identity of the anonymous referee who has praised or damned the paper. This study suggests that even for a specialty journal such efforts are largely unrewarding and that most referees remain anonymous.

Funding: None.

Conflict of interest: None

1. Lock S. A difficult balance: editorial peer review in medicine. London: BMJ Publishing, 1991.
2. McNutt R, Evan A, Fletcher R, Fletcher S. The effects of blinding on the quality of peer review; a randomized trial. JAMA 1990;263:1371-6. [Abstract/Free Full Text]
3. Fisher M, Friedman S, Strauss B. The effects of blinding on acceptance of research papers by peer review. JAMA 1994;272:143-6. [Abstract/Free Full Text]
4. Cicchetti D. The reliability of peer review for manuscript and grant submission: a cross disciplinary investigation. Behavioral and Brain Sciences 1991;14:119-86.
5. Sweitzer B, Cullen D. How well does a journal's peer review process function? A survey of authors' opinions. JAMA1994;272:152-3.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?