Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study
BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g4145 (Published 01 July 2014) Cite this as: BMJ 2014;349:g4145
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Dear Editor,
Hopewell et al.’s paper [1] and empirical research are welcome contributions to peer review scholarship. If our goal as researchers is to investigate journal peer review as a scientific object of study and not as self-evident based on common experience process [2,3,4], however, the study appears to hold a few misguided assumptions.
First, Hopewell et al. state that “[t]he practise of peer reviewing manuscripts has been around for more than 200 years” [1]. Dating manuscript review practices only to ‘more than 200 years’ is a preconceived notion that disregards how journal peer review appears to have been shaped [3,4,5,6,7,8], including obligations by early journals (since 1665) to perform the censorship function of inquisition [3,4,5]. Socio-historical research frames review of manuscripts within a wider historical lens dating back to the beginning of universities in the twelfth century in precursor forms of censorship and inquisition [3,4,6].
Second, Hopewell et al. propose that “[o]ne of the main aims of peer review is to improve the quality and transparency of a publication by checking that the reported research has been carried out correctly and that the results presented have been interpreted appropriately” [1]. Yet, the authors advance that “[g]iven the wealth of evidence on the poor and inadequate reporting in published research activities, particularly the reporting of randomised trials6, peer reviewers may be failing to detect important deficiencies in the scientific literature” [1]. An assumption that a main goal of journal peer review is to improve quality and transparency, is not supported empirically in the authors’ review of literature (and more generally [3,9]) or in the authors’ findings (i.e., where some changes requested by referees purportedly do not contribute to these goals [1]). Rather, I propose that to understand journal peer review as scientific object of study means understanding it as a form of boundary judgement where editorial decisions delineate if a manuscript can be considered as scientific, or not. How editorial readers go about determining if a manuscript is scientific, or not, has not been subject to much empirical research [10].
Third, there appears to be a competing boundary based on a relation of accountability to readers where some journals evaluate if a manuscript is newsworthy, or not [10]. Hopewell et al. did not consider potentially competing boundary dynamics in their research on the ‘reporting of methodological aspects of randomised trial’. They could not investigate boundary intersections because they only looked at one of the dichotomous outcomes of journal peer review: published papers. What of the ‘reporting of methodological aspects of randomised trial’ in scientifically sound papers that were rejected? These remain invisible.
Fourth, Hopewell et al. appear to assume that editorial readers do not hold competing interests and accountabilities. This rests on a deeper-seated assumption that journal peer review is a purely rational process based on rational decision-making [3,5]. Evidence that referees might be requesting changes based on their personal interests [1] renders competing interests and accountabilities visible. In contrast, when looking at journal peer review as multiple and intersecting accountabilities for all actors, divergent goals are expected [10]. Linked with this are authors who might not have been willing to accommodate requested change. What of authors who withdrew manuscripts from peer review following what they might have deemed as unreasonable requests for change? These remain invisible.
Fifth, Hopewell et al. propose that open review with public access to original and subsequent manuscripts, author response, editorial judgements and decisions “…enable[s] readers to have a clear and transparent account of the peer review process. We would strongly recommend this model should be followed by other leading journals” [1]. The researchers propose what appears on the surface as a reasonable recommendation, but it does not reflect an understanding of peer review as a scientific object of study, it is based on peer review as common experience. To understand journal peer review as a scientific object of study is also to look at how it intersects with the exchange of scientific knowledge. Missing in a recommended form of peer review, therefore, is a need to maximize scientific exchange. Maximal exchange includes unpublished papers and their accompanying editorial documents and access to the totality of editorial decisions and judgements including those from external academics. In addition, maximizing potential for rational decision-making means adopting forms with wider scientific and public scrutiny using public review, in perpetuity [10,11]. The Copernicus suite of journals offers an example of a profitable business model with post-publication peer review and integrated public review [12,13].
Finally, Hopewell et al. state caution with generalizability of their findings because of variations of peer review ‘processes’. They nevertheless assert that “…the problem of poor reporting is something that is seen consistently across the medical literature,12 suggesting that the findings seen in this sample of journals might be representative of other journals” [1]. Missing from the comment is an understanding of the object of study as scientific with different forms of journal peer review and of structural properties that contribute to different forms [3,4]. Relations of transparency with open access and non-anonymity for referees and all editorial readers, for example, can contribute to increased potential for rational decision-making by editorial readers [10,14] and public review create conditions for self-correction [10]. This means that generalizability might best be restricted to similar forms, with compatible structural properties.
References
1. Hopewell S, Collins GS, Boutron I, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ 2014;349:g4145.
2. Bacon, F. 1620. The New Organon of True Directions Concerning the Interpretation of Nature.
3. Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part I. uO Research. Pp. 1-24. http://www.ruor.uottawa.ca/bitstream/10393/31319/1/journal.peer.review.a...
4. Gaudet, J. 2014. Investigating journal peer review as scientific object of study: unabridged version – Part II. uO Research. Pp. 1-20. http://www.ruor.uottawa.ca/bitstream/10393/31320/1/journal.peer.review.a...
5. Biagioli, M. 2002. "From Book Censorship to Academic Peer Review." Emergences: Journal for the Study of Media & Composite Cultures 12:11-45.
6. Gould, T.H.P. 2013. Do We Still Need Peer Review? Toronto: The Scarecrow Press, Inc.
7. Rip, A. 1985. "Commentary: Peer Review is Alive and Well in the United States." Science, Technology & Human Values 10:82-86.
8. Spier, R. 2002. "The history of the peer-review process." TRENDS in Biotechnology Vol.20 No.8 20:357-358.
9. Smith, R. 2011. The Trouble with Medical Journals. London:The Royal Society of Medicine Press.
10. Gaudet, J. 2014. How pre-publication journal peer review (re)produces ignorance at scientific and medical journals: a case study. uO Research. Pp. 1-67. http://www.ruor.uottawa.ca/bitstream/10393/31198/5/how_peer_review_repro...
11. Gaudet, J. 2014. All that glitters is not gold: The shaping of contemporary journal peer review at scientific and medical journals. uO Research. Pp. 1-23. http://www.ruor.uottawa.ca/bitstream/10393/31238/1/contemporary_shaping_...
12. Pöschl, U. 2010. “Interactive open access publishing and public peer review: The
effectiveness of transparency and self-regulation in scientific quality assurance.”
International Federation of Library Associations and Institutions 36:40-46.
13. Pöschl, U. 2012. “Multi-stage open peer review: scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation.”
Frontiers in Computational Neuroscience. 6:1-16.
14. Gaudet, J. 2014. An end to ‘God-like’ scientific knowledge? How non-anonymous referees and open review alter meanings for scientific knowledge. uO Research. Pp. 1-12. http://www.ruor.uottawa.ca/bitstream/10393/31415/1/anonymity.god.like.sc...
Competing interests: No competing interests
The study performed by Hopewell et al is really important for our daily basis research and publication processes. Peer review process has helped us to improve the quality of some of our published papers in an empirical way. However, the process to improve the quality and transparency of our papers is not focused either just on the peer reviewer or on reporting system (i.e. CONSORT statement).
The managing editor, the editor, the editorial board, the writer and the reviewer, among others, are all involved in the process and each one of them gives some details to improve the written paper. That´s why I think the quality of a paper goes beyond.
Scientific reporting must agree with international guidelines (ICMJE – equator network) along with the relevance, pertinence and the accordance with the journal´s focus and vision. These are issues that should be covered by the editor or the managing editor in the first place before the paper goes to peer reviewing process.
Peer reviewers must be able to recognize these key points regarding: reporting, methods and the scientific relevance of the paper if the editorial group has missed some of them.
Checklists are important to standardize processes, however I believe in the importance of some knowledge on epidemiology and methods for these reviewers.
Most of the time, scientists are expert in their field, perhaps in biology, internal medicine, urology, among others, perhaps they do not know about scientific method in depth since in many universities or centers around the world, clinicians and researchers have a place where there are expert people in statistics, methodology and other important areas such as scientific writers, administrators, among others. Sometimes reviewers downgrade or upgrade papers regarding the methods used but without any expertise on epidemiology or a specific type of method.
Finally I would like to say that this paper makes an excellent point to start changing the peer review process, however I believe the effort must be educational. Reviewers (excellent scientists) must know about their field and also about methodology, epidemiology and statistics and if this could not be reached, perhaps the editorial process must need expert reviewers on each one of the different issues stated before it can improve the quality of our papers. I do believe in this, as a way to improve paper´s quality and patient´s health as well.
References:
1. ICJME. Uniform Requirements for Manuscripts Submitted to Biomedical Journals [Internet]. International Committee of Medical Journal Editors; 2010. Available from: www.icmje.org
2. Dickersin K, Ssemanda E, Mansell C, Rennie D. What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Med Res Methodol [Internet]. 2007 Jan [cited 2014 Jul 14];7:44. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2121101&tool=p...
3. Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O’Callaghan A, Kostov B, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ [Internet]. 2011 Jan 22 [cited 2014 Jul 14];343(nov22_2):d6783. Available from: http://www.bmj.com/content/343/bmj.d6783?ijkey=397f961192e369dd969d47669...
4. Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of peer review: randomised controlled trial. BMJ [Internet]. 2004 Mar 20 [cited 2014 Jul 14];328(7441):673. Available from: http://www.bmj.com/content/328/7441/673?ijkey=dbec8f9986c821189caeb6bb45...
5. Hopewell S, Collins GS, Boutron I, Yu L-M, Cook J, Shanyinde M, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ [Internet]. 2014 Jan 1 [cited 2014 Jul 14];349(jul01_8):g4145. Available from: http://www.bmj.com/content/349/bmj.g4145
Competing interests: No competing interests
Hopewell S et al. [1] doubted about the effectiveness of peer review as there are still some important deficiencies in reporting of the methods and results of randomised controlled trials (RCTs) after peer review in a series of medical journals with open access, and some comments of reviewer could have a negative impact on reporting. A similar finding has been pointed out in a recent cochrane review [2], inadequate reporting of essential elements in RCTs is common and journals often fail to send a clear message about endorsement of the Consolidated Standards of Reporting Trials (CONSORT) agreement to authors, and fidelity of endorsement as an ‘intervention’ is weak to date. We argue that the editors rather than peer reviewers have more advantages and should be more responsible for solving this problem, since there are 37 items in the CONSORT checklist [3], it is hard for the reviewers to keep all items in mind when they are reviewing paper. Editors could contribute more for improving peer review.
First of all, it would be very helpful if editors could develop very comprehensive “information to authors”, “information to reviewers” and detailed checklists according to different types of manuscript, and make these documents be accessible for authors before submission and for reviewers before peer review. In addition, editors could conduct a careful screening of newly submitted manuscript by authors and also the reviewer's comments to check if there are essential elements are missing, the manuscripts that do not meet the requirements should be returned to authors for improvement, and the inappropriate comments from reviewers should be corrected. Even though some extra burdens to editors may be generated, everything will soon be put in order once these documents are well-developed, and it would make the peer review process quite effective in the long run.
Moreover, editors are encouraged to join in researches, especially prospective studies, on how to make peer review more effective if it is convenient. So far, many published papers on this topic are editorials, while original investigations are still limited. More focus is required to standardize peer review systems, and the proven inefficient system should be abandoned. For example, studies may be useful on following topics: comparing the effectiveness among single-blinded system, double-blinded system, open system, peer agreement system, and author suggestion-based system; comparing the effectiveness between one stage and two stage peer review systems; assessing the value of increasing reviewers, training for peer reviewers and editors, use a comprehensive information (checklist) to authors/reviewers, et al.
At last, it is strongly recommended to make peer reviewers' comments accessible online, in order to not only benefit reviewers to learn from each other to improve skills, but also allow researchers to conduct retrospective studies. Many journals already have their websites, so it is practicable to list the reviewers’ comments alongside the paper and put a selection button as “read the reviews”, but the exposure of peer reviewer’s name should get consent from reviewers.
References
1. Hopewell S, Collins GS, Boutron I, Yu LM, Cook J, Shanyinde M, Wharton R, Shamseer L, Altman DG. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014; 349:g4145.
2. Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012;1:60.
3. Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869.
Competing interests: No competing interests
This article highlights what many people working in medical publishing have known for years. The crucial but generally unregulated peer review process is far from perfect.
As a result, clinical trial reports are too often flawed and evidence-based medicine has failed to deliver clear indisputable guidance.
The situation has been compounded by unavailability of trial data for further scrutiny, lack of transparency, and failure to publish negative results.
But the future is bright.
Open-access models that embrace and exploit the latest communication technologies to promote and support dialogue can now involve all stakeholders (scientists, clinicians, patients) worldwide as peer reviewers, who could be invited to contribute to moderated SWOT—strengths, weaknesses, opportunities, and threats—analyses for all published research articles.
The same technology also provides the potential for stakeholders to get to know each other by adding a photo and a paragraph summarising their stakeholder status, qualifications, affiliations, job title, and any disclosures.
Its time for journals to add real value to the original research by creating constructive, respected, and reliable online stakeholder communities who actively contribute to further article review and analysis. Such interaction will facilitate collaborations and ensure timely and effective realisation of true translational potential to improve the quality of lives worldwide.
Competing interests: No competing interests
To the editor:
Hopewell and colleagues’ (1) study investigating the effectiveness of the open peer review process is both timely and needed. As scientists, we are collectively consumed with testing hypotheses regarding our research interests, but few give equal attention to the processes by which our research is communicated in scholarly journals and the media. The unfortunate misconstrued interpretation of scientific findings by the media is better discussed at a later date, but I would like to draw attention to a very important finding in this manuscript. In their study, Hopewell and colleagues report findings that suggest that peer reviewers often fail to detect deficiencies in the reporting of methods and results among randomized trails. The inexcusable inability of multiple peer reviewers to detect deficiencies in the methods of a submitted trial has a direct effect on the progress of science. This is not only because science is partly define by the process by which one acquires knowledge (2) , but also because deficiencies in the reported methods of existing literature increases the odds of irreproducible results. The inability to reproduce findings in research is a detriment to the progress of science (3).
There may be many reasons for these findings, some of which Hopewell and colleagues highlight in their study. I agree with the authors and their conclusions that journal editors should have a protocol in place to ensure that reviewers of manuscripts make appropriate comments. I also agree that the use of checklists and guidelines might be beneficial in the review process, but I would like to make another suggestion. It has been documented for at least as long as 30 years that that the direction of a study’s findings has an influence on the level of scrutiny that a reviewer gives towards a manuscript (4). Therefore, I suggest that BMJ (and perhaps other journals) consider adopting an internal 2-stage peer-review and submission process. For the first stage, corresponding authors are required to submit only of the manuscript’s introduction, methods, and additional comments on the proposed novelty in the research question of interest. Hence, for the first stage of the peer-review process, the study’s results are blinded to reviewers. If a group of peer reviewers consider the research question a priority, and the methodology is deemed appropriate, the manuscript should proceed to the second stage with a contingent “acceptance for publication.” For the second stage, the corresponding author submits the full manuscript to be peer-reviewed, and as long as the abstract, results, and discussion sections are appropriately written in the context of the approved methodology, the manuscript is published regardless of whether findings produce null or significant results. I strongly believe that having a contingent acceptance for publication in the second stage will prevent the abundance of submitted manuscripts that over-state their findings. I am aware that there are various journals that implement a two-stage review process, but none that I am aware of blind reviewers to the study’s results in the initial stage. I believe blinding the study’s results in the early stages of peer-review process reassures the journal’s editor that manuscripts are judged consistently by the quality of the scientific methodology and not overly judged by minuscule issues because findings are reported in a particular direction.
I know that this is wishful thinking, but as a junior scientist/researcher who will soon go through the rigors of the tenure process of professorship, it would be nice to know that my peers will judge my work on the quality of my methods and novel contributions to science, as oppose to solely on the results produced by the “process.”
Best regards,
Dwight William Lewis Jr., PhD
Postdoctoral Trainee
The University of Alabama at Birmingham
References
1. Hopewell S, Collins GS, Boutron I, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ 2014;349:g4145.
2. Jaus HH. Editorial: Science Is Process, Product, and …. Science Activities: Classroom Projects and Curriculum Ideas. 2002/01/01 2002;39(3):3-4.
3. Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible Research: Moving toward Research the Public Can Really Trust. Annals of Internal Medicine. 2007;146(6):450-453.
4. Mahoney M. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cogn Ther Res. 1977/06/01 1977;1(2):161-175.
Competing interests: No competing interests
Re: Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study
To the Editor:
The purpose of peer review is to appraise the degree to which methods used in a study are suited to its stated aims and whether the study was carried out as planned. Peer review is intended to reward well constructed arguments of whatever political or moral persuasion – preferably based on unbiased data, observation and reasoning. Several reviewers are assigned to maintain levity based on a range of feedback. Until a better method is devised, peer review remains the current standard to help us from drowning in information and opinion. Although it aspires to the best of intentions, peer review is not perfect and could still be subverted by those to whom peer review is entrusted.
Improving the peer-review process relies on understanding its context and culture, involving complex interactions between authors, editors, reviewers, readers and academic publishers. The challenge remains that of finding expert reviewers who are willing to spend adequate [often pro bono] time compiling constructive unbiased commentary within a reasonable turn-around time. As a senior reviewer for several high impact medical journals over ten years (Annals Emergency Medicine, Cochrane Collaboration, Medical Journal of Australia), I have spent hundreds of non-remunerated hours assessing original manuscripts in the public spirited endeavour to enhance the quality of research that achieves publication. There is frequently no recognition of my contribution to improving quality of submitted articles other than a brief mention of thanks on a yearly list. Other than reasonable payment for reviews conducted in one’s own time, the motivation of reviewers to do a better job could be boosted by better recognition from journal editors for a job well done. For instance, Annals of Emergency Medicine lists its Top 50 reviewers on each monthly issue’s prestigious masthead over 12 months.
Transparency on the contribution of reviewers, including their names and academic affiliation, reduces susceptibility of peer review to personally motivated agendas such as mutual rivalry, reprisals or the return of favour between author and appraiser. Non-anonymous reviewers are more likely to offer fairer and more objective reviews to submitting authors. Communal reviewing and editing models such as that used for Wikipedia are more efficient, evolve organically to accommodate changes, and foster cooperative authorship. Peer reviewers, time-pressured in their usual work and personal lives, could benefit from a collaborative model of web-based group peer review. Existing web-based journal platforms could potentially facilitate efficient and dynamic reviewer interactions online. As collaborative peer review potentially reduces time and intellectual effort required from each reviewer, approached reviewers may be more willing to accept assignments. As science moves towards an interdisciplinary and collaborative model which works better in the contemporary scientific ecosystem in the web-connected world, peer review and other journal quality control mechanisms could also be enhanced by open source collaboration.
Mutual non-anonymity of author and peer reviewers has largely been demonstrated to not improve quality of feedback on submitted manuscripts. It is about time that performance-graded reviewer remuneration, improved journal recognition of high quality assessments and reduced individual work burden using a collaborative model payment be explored as factors that could improve quality of peer review.
Competing interests: No competing interests