Opening up BMJ peer review
BMJ 1999; 318 doi: https://doi.org/10.1136/bmj.318.7175.4 (Published 02 January 1999) Cite this as: BMJ 1999;318:4All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Raj
Though this is a late response to your note, I feel it is worth
getting it 'on air'.
It is well-known that junior doctors often do not complain over
working conditions for fear of a bad reference. Sure, some things are
worth complaining about - MTAS in 2007 - but there is no discernible
disadvantage to complaining about this, as senior doctors hate it too!
At many ivory tower teaching hospitals across the land, I am sure a
generation of junior doctors worked in exploitative conditions just to get
the 'golden circuit' reference.
Analagous to this, junior researchers will NOT give open-minded
reviews on the peer-review process if identified.
I would argue for a DOUBLE-BLIND process - with the authors of the
report being anonymous too. This would prevent over-zealous critiscm, but
equally would prevent reviewers from assuming that the work must be good
because the author/institution is of a high status.
Your story was thus highly pertinent!
Competing interests:
None declared
Competing interests: No competing interests
Dear Sir,
Thank you very much for a very interesting article on open peer review
system in BMJ by Richard Smith 1999;318:4-5.With progressive globalisation
of the search of truth in the medical arena,we should rise progressively
above the institutional ,regional and other forms of human biases that may
often gag the expression of medical science,in the present days of e-media
.For example,the wisdom of treatment of non eltor mutagenic strain of
vibrio cholera in Calcutta is
possibly more than the clinical counterpart of Bayler college of
Washington ,similarly teloisomerase and cellular senescence and its
implications would possibly be better understood by the scientists of the
Bayler College.Hence I have a humble suggestion ;Is it possible to strike
out the name and the place of work of the authors before we send the
papers for peer review in BMJ?
Sincerely Yours
Dr Niranjan Bhattacharya
MBBS,MD,MS,FACS(USA),Dr Sanjukta Bhattacharya PhD,Dr Mahua
BhattacharyaMBBS,DA,DGO,Dr Kanailal Mukherjee MBBS,Phd,PhD.
Competing interests: No competing interests
The BMJ does not attach any conditions to the reviewers' comments
that we send to authors. Perhaps we should, but we have not so far
experienced any sort of problem.
We do not publish case reports without the consent of the patient.
Any patient who is making a claim of clinical negligence would thus know
that a case report might be published. If he or she consented, then there
would be no problem from our point of view. If he or she did not consent,
then we wouldn't publish the paper. It's hard to imagine that a patient
would consent to the publication of a report when a case of negligence is
pending. Probably the patient's lawyers would advise against publication.
Richard Smith
Editor, BMJ
Competing interests: No competing interests
On what conditions of confidentiality are authors sent the comments
of reviewers?
Are there any conditions or guidance about the submitting of a case
report which is also the subject of claim for clinical negligence?
Should a journal object to lawyers seeing the comments?
There are legal arguements that a patient claiming clinical
negligence can obtain disclosure of all documents created for the purposes
of publication of their case as their dominant purpose is not for
litigation.
Reviewers comments cannot be a substitute for a medicolegal opinion
but may be evidence of a responsible body of opinion relevant to the
issues in both the case report & the litigation.
What is the BMJ's position?
Competing interests: No competing interests
EDITOR - The BMJ plans to let the authors know the identity of the
reviewers (1). The decision is supported by a randomized trial on 125
manuscripts, and a few other scientific studies in which the main outcome
measure was the quality of the review (2,3). There are clear arguments
against close peer review ; I am not sure that all the arguments against
open peer review have been considered, especially concerning the
relationships within the scientific community, at a national and
international level. Why not tackle this question with an evaluative
epidemiology approach, as if it was, let us say, evaluation of an
ergonomic or psychosocial change in a working environment ?
It seems that it is too late for performing a before-after comparison
based on the relevant dimensions of the possible effects of the change.
For some of the dimensions ( for example, recommandations concerning
publication ) a retrospective evaluation would be possible, but we know
that retrospective evaluation is a source of bias. It seems also difficult
(and it woud be ineffective) to randomise the scientific community to two
groups, open and close peer review. Randomisation through randomisation of
scientific journals is also unrealistic. At least it woud be possible to
have comparisons based on a natural experiment ; this remains feasible as
far as some scientific journals keep closed peer review, that these
journals are comparable to those which have changed to open peer review,
and that the choice of a journal for submission by the authors is
independant of whether the peer review will be open or not. All the
relevant dimensions of the possible effects of the change have to be
defined ; the range is large, since it includes effects on the manuscript,
on the authors and reviewers, and on the scientific community as a whole.
Quality is probably the most important point for the journals ; for
authors and reviewers, open peer review can be seen as a threat to
satisfactory relationships with their peers. Whether the threat is real or
not remains to be evaluated.
Evaluating an intervention is most often a complicated task, which
may need the help of specialists from several disciplines ; the situation
here is not much more difficult than evaluating other kinds of
intervention. One may consider that the treatment of manuscripts is
unimportant : after all, if it is suboptimal, nobody will die ; but if it
deserves some interest, then widening the debate in the scientific
community, beyond specialists of peer-review, might be beneficial. It woud
also alleviate the task of peer review researchers.
Annette Leclerc, epidemiologist
researcher at INSERM Unit 88
HNSM
14 rue du val d'Osne
94410 St-Maurice
France
(1) Smith R. Opening up BMJ peer review. BMJ 1999 ; 318 :4-5.
(2) Van Rooyen S, Goodlee F, Evans S, Black N, Smith R. Effect of
open peer review on quality of reviews and on reviewers'recommandations :
a randomised trial. BMJ 1999, 318 :23-27.
(3) Goldbeck-Wood S. Evidence on peer review - scientific quality
control or smokescreen? BMJ 1999, 318 :44-45.
Competing interests: No competing interests
Rejected Letters to the Editor and Independent Audit Committees:
Dear Sir:
Mr. Gbolade [1] states that an open peer review system exists after
publication. There is a potential problem in the case of letters to the
editor that identify failures of the editorial process and raise issues
about managerial competence. Such letters present conflicts of interest
to the journal management. Currently, there is no recourse for the writer
whose rejected letter raised fundamental concerns about the validity of
published articles.
A standard practice of both public and private organizations is to
appoint independent audit committees to review the disputed decisions of
management. I propose that medical journals have a similar mechanism for
writers of rejected letters that have questioned the editorial process by
identifying gross defects within articles. Even if, as I expect, the
majority of the letters have been appropriately rejected, the independent
review is needed.
Yours sincerely,
Vincent V. Richman MBA PhD
Research Associate, AlgoPlus Consulting Limited, Suite 502,
5675 Spring Garden Road, Halifax, Canada B3J 1H1
Email: vrichman@compuserve.com
References:
[1] Gbolade BA, Open peer review system already exists, albeit, post
publication, eBMJ, 17 Jan 1999.
Competing interests: No competing interests
review system, with reviewers known to the authors, has existed since the
advent of paper-based publications. Although this system occurs post
publication, it is in no way inferior to the pre-publication closed system
and results from wider exposure to the scientific world.
I am referring to correspondences to editors, which in the form of
critical comments on specific articles form important and integral parts
of most journals. Those that get published (at least in the BMJ) are
usually original, and contain assertions supported by data or citation.[2]
Collective appraisal of articles in the process of acquiring critical
appraisal skills can be developed and enhanced in the forum of a journal
club.[3] The importance of the journal club in the unofficial peer review
process is highlighted by Sandifer et al.[4] In their evaluation of their
journal club's activities in the first six months of its existence, the
impact on commissioning policy and the publication of letters to the
editor of the journal from which the articles were selected were used as
proxy outcomes. Six out of ten letters generated after collective
appraisal were published.
One of the cardinal rules for successful critical appraisal of
articles is to avoid prejudging the articles on account of reputation,
source, authors or preconceptions or bias. In the ambience of a journal
club, this condition is easily met because of the variation in the
composition of its membership. Criticism of any article is therefore more
likely to be based purely on scientific principles. A letter to the editor
in response to an article is therefore less prone to the criticisms of the
closed peer review system outlined in Smith’s editorial. Such letters at
times identify gross defects in studies that have been missed by a
journal’s peer reviewers and may help in identifying instances of
scientific fraud. Such letters tend to be courteous because the writers
know that the authors of the original articles have the last word.
The writers (junior or senior) of these letters put their names and
addresses at the end of the correspondences. If these “unofficial
reviewers” are not afraid of identification and therefore “reprisals”, why
should the “official reviewers”? My premise is that an open peer review
system already exists, albeit, as a post publication exercise. All that
remains is for it to be used more widely pre-publication.
Babatunde A Gbolade
Consultant Gynaecologist &
Director of Fertility Control Unit
St James’s University Hospital
Beckett Street, Leeds LS9 7TF
E-mail B.A.Gbolade@leed.ac.uk
References
1 Smith R. Opening up BMJ peer review. A beginning
that should lead to complete transparency. BMJ
1999; 318:4-5.
2 BMJ. Advice to authors. BMJ 1997; 314:370.
3 Sackett DL, Haynes RB, Guyatt GH, Tugwell P.
Clinical epidemiology: a basic science for clinical
medicine. 2nd edition. Boston: Little, Brown and
Company, 1991:398-418
4 Sandifer QD, Lo S, Crompton G. Evaluation of a
journal club as a forum to practice critical
appraisal skills. J R Coll Physic Lond 1996; 30:
520-22.
Competing interests: No competing interests
Writers and reviewers will be watching your experiment carefully.
You write, "the main argument against open peer review - a sad one - is
that junior reviewers will be reluctant to criticize the work of senior
researchers for fear of reprisals." The short story by Montague James
"Casting the Runes" is required reading for your prospective reviewers,
but let us trust that it will not implant ideas
in the minds of those whose papers are rejected. In Montague's short
story, based on the authority of peer review the Council of a Learned
Association rejects Mr Karswell's paper "The Truth of Alchemy". On appeal
the writer was assured by the secretary of the Association that
the fullest consideration was given to the submitted draft, and that it
was declined on the judgement of a most competent authority, whose name it
was impossible for the Association to reveal. On second appeal, which
requested the name of the reviewer, the secretary refused the request and
closed the correspondence. Sadly, as you know writers can guess or
discover the identity of reviewers.
Mr Karswell's earlier book "A history of witchcraft" had been
rejected by Mr John Harrington FRS, a scholar found dead three months
after his review, having fallen off a tree in mysterious circumstances.
Mr Karswell quickly identified Mr Dunning as the reviewer of "The Truth
of Alchemy". Strange events soon overtook Mr Dunning. Black magic was
at work. Within days he was a nervous wreck, who dared not go home or his
place of study, the museum, for fear that Karswell might turn up there.
His appearance became forlorn, his conversation empty. Clearly,
Harrington's fate awaited Dunning. Clearly, he had misjudged the truth of
the content of the paper!
This story of inadvertent `open' scientific peer review had a
murderous end. I shall not spoil the story by telling you and BMJ readers
who murdered who. I hope for a less macabre outcome of your bold
experiment.
Yours sincerely
Prof Raj Bhopal
Department of Epidemiology and Public Health,
School of Health Sciences,
The Medical School,
University of Newcastle upon Tyne,
Framlington Place,
Newcastle NE2 4HH
References
1.Smith R, Opening up BMJ peer review, BMJ, 1999: (318); 4-5
2.James MR, Casting the runes (from, Ghost stories of an antiquary,
1911). Reprinted in, The Oxford Library of Classic English Short Stories
Volume 1. Guild Publishing, London, 1989.
Competing interests: No competing interests
Dear Sir
On reading your editorial announcing the BMJ policy of identifying
reviewers to authors of papers [1], I was struck by the absence of
evidence supporting this change - notably in the accompanying paper by van
Rooyen and co-authors (of which you were one)[2]. Although as you suggest,
abusive
or destructive reviews in a cloak of anonymity are unacceptable, is an
experienced editor not likely to detect these? Nor is another abuse of
anonymity which you identify, theft of ideas, enough to justify opening up
peer review on a large scale. This abuse could be detected or prevented by
a number of controls.
The trial by van Rooyen and co-authors found that open peer review
had no effect on the quality of the review or recommendation regarding
publication. However it was found that it increased the likelihood of
chosen reviewers declining to review. As you speculate, this consequence
of
open peer review may mean that junior researchers, the reviewers that
consistently are found to give the 'best' opinions [3,4], will be
reluctant to referee papers by senior researchers or, in my view worse
still, review
but not criticise them. Your change to open peer review is not an evidence
-based change in practice. Editors, authors and readers benefit from
rigorous and fair reviews, but efforts to obtain them may be frustrated
and not served by the transparency that you champion.
Yours sincerely
Frank Larkin
Consultant Ophthalmic Surgeon
Moorfields Eye Hospital
City Road
London EC1V 2PD
References
1 Smith R. Opening up BMJ peer review. BMJ 1999;318:4-5
2 van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of
opening up peer review on quality of reviews and on reviewers'
recommendations: a randomised trial. BMJ 1999;318:23-7
3 Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a
good reviewer and a good review for a general medical journal? JAMA
1998;280:231-3
4 Goldbeck-Wood S. Evidence on peer review - scientific quality
control or smokescreen? BMJ 1999;318:44-5
Competing interests: No competing interests
Re:Open peer review system: advantages and disadvantages
Open peer review system has both advantages and disadvantages. The
most important advantage is that a reviewer will hesitate to make comments
using the terms that would hurt the authors. He/she will make constructive
comments more politely so that the authors will find it easy to accept.
The disadvantage is that the comments on the papers by more senior or
reknowned authors may be biased.
The open peer review system, however, is worth trying to evalutae its
results in the future. Perhaps, an option could be given to the reviewer
whether he/she would like to be identified.
Competing interests: No competing interests