Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study
BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g4145 (Published 01 July 2014) Cite this as: BMJ 2014;349:g4145- Sally Hopewell, senior research fellow12,
- Gary S Collins, senior research fellow1,
- Isabelle Boutron, professor2,
- Ly-Mee Yu, senior medical statistician1,
- Jonathan Cook, associate professor13,
- Milensu Shanyinde, medical statistician1,
- Rose Wharton, medical statistician1,
- Larissa Shamseer, PhD student4,
- Douglas G Altman, director1
- 1Centre for Statistics in Medicine, University of Oxford, UK
- 2Centre d’Epidémiologie Clinique, Université Paris Descartes, INSERM U1153, France
- 3Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, UK
- 4Ottawa Hospital Research Institute, Canada
- Correspondence to: S Hopewell Centre for Statistics in Medicine, University of Oxford, Oxford OX3 7LD, UK sally.hopewell{at}csm.ox.ac.uk
- Accepted 12 June 2014
Abstract
Objective To investigate the effectiveness of open peer review as a mechanism to improve the reporting of randomised trials published in biomedical journals.
Design Retrospective before and after study.
Setting BioMed Central series medical journals.
Sample 93 primary reports of randomised trials published in BMC-series medical journals in 2012.
Main outcome measures Changes to the reporting of methodological aspects of randomised trials in manuscripts after peer review, based on the CONSORT checklist, corresponding peer reviewer reports, the type of changes requested, and the extent to which authors adhered to these requests.
Results Of the 93 trial reports, 38% (n=35) did not describe the method of random sequence generation, 54% (n=50) concealment of allocation sequence, 50% (n=46) whether the study was blinded, 34% (n=32) the sample size calculation, 35% (n=33) specification of primary and secondary outcomes, 55% (n=51) results for the primary outcome, and 90% (n=84) details of the trial protocol. The number of changes between manuscript versions was relatively small; most involved adding new information or altering existing information. Most changes requested by peer reviewers had a positive impact on the reporting of the final manuscript—for example, adding or clarifying randomisation and blinding (n=27), sample size (n=15), primary and secondary outcomes (n=16), results for primary or secondary outcomes (n=14), and toning down conclusions to reflect the results (n=27). Some changes requested by peer reviewers, however, had a negative impact, such as adding additional unplanned analyses (n=15).
Conclusion Peer reviewers fail to detect important deficiencies in reporting of the methods and results of randomised trials. The number of these changes requested by peer reviewers was relatively small. Although most had a positive impact, some were inappropriate and could have a negative impact on reporting in the final publication.
Introduction
The International Committee of Medical Journal Editors defines peer review as the “critical assessment of manuscripts submitted to journals by experts who are not part of the editorial staff.”1 The practise of peer reviewing manuscripts has been around for more than 200 years and today editorial peer review is used almost universally by scientific journals as a tool to assess and improve the quality of submissions to biomedical journals.2 One of the main aims of peer review is to improve the quality and transparency of a publication by checking that the reported research has been carried out correctly and that the results presented have been interpreted appropriately.3 Clinical decisions are then made on the basis of this research evidence, so if it is misleading, by being incomplete, inaccurate, or poorly reported this has a direct impact on patient care.
Traditionally, the process of peer review has been carried out “blinded” so that the identity of the peer reviewers is concealed from the authors and the comments made by the peer reviewers are never available in the public domain. Some journals have since switched to a process of open peer review,4 whereby the identity of the peer reviewers is disclosed and, in some instances, the peer reviewer’s comments are included alongside the published article. However, despite several studies assessing the effectiveness of the peer review process, and its wide acceptance by the scientific community, little is known about its impact on the quality of reporting of the published research.5 Given the wealth of evidence on the poor and inadequate reporting in published research activities, particularly the reporting of randomised trials,6 peer reviewers may be failing to detect important deficiencies in the scientific literature.
We investigated the effectiveness of open peer review as a mechanism to improve the reporting of randomised trials published in biomedical journals. In particular, we examined the extent of changes made to the reporting of methodological aspects of manuscripts of randomised trials after peer review, the type of changes requested by peer reviewers, and the extent to which authors adhere to these requests.
Methods
Sample selection
We searched the US National Library of Medicine’s PubMed database to identify all primary reports of randomised trials published in the BioMed Central series medical journals (see supplementary appendix 1 for search strategy) in 2012 and indexed with the publication type “Randomized Controlled Trial” (search as of 28 May 2013). One reviewer (SH) screened the titles and abstracts of all retrieved reports to exclude any obvious reports of non-trials. A copy of the full article was then obtained for all non-excluded reports, and two reviewers assessed each article for eligibility. We chose to use the BMC-series medical journals as they operate a system of open peer review whereby all submitted versions of a manuscript are published along with the corresponding peer reviewers’ comments and authors’ responses.7 Manuscripts submitted to BMC-series medical journals are typically handled by external academic editors, in house editors, or a combination of both at different stages in the manuscript submission process. Comments made by the external academic editors are not usually published alongside the submitted manuscript. Journals included in the BMC-series medical journals also endorse the CONSORT (Consolidated Standards of Reporting Trials) statement.8
Eligibility criteria
We included all primary reports (that is, those reporting the main study outcome) of randomised trials, defined as a prospective study assessing healthcare interventions in human participants who were randomly allocated to study groups. We included all studies of parallel group, crossover, cluster, factorial, and split body design. We excluded protocols of randomised trials, secondary analyses, systematic reviews, methodological studies, pilot studies, and early phase trials (for example, phase 1).
Data extraction
Eight reviewers working in pairs extracted data. Each pair independently extracted data from eligible reports; any differences between reviewers were resolved by discussion, with the involvement of an arbitrator if necessary. To ensure consistency between reviewers we first piloted the data extraction form. We discussed any disparities in the interpretation and modified the data extraction form accordingly. By accessing the prepublication history from each report (using the URL cited at the end of each published article) we recorded the number of submitted versions of the manuscript and the date of the original submission, final submission, and published article. For each eligible report we used the “compare documents” function in PDF Converter (Enterprise 7) to determine differences in reporting between the original submitted version of the manuscript and the final submitted version.
For each trial report we first examined whether specific items related to the methodological aspects of randomised trials were reported; based on items in the CONSORT checklist8; on average this step took between 15 and 20 minutes per trial report. We then examined any changes in reporting of these items between the original and final version of the manuscript. We classified changes as being items added (item new to final version, not in original version of manuscript), subtracted (item removed from final version, was in original version of manuscript), or altered (item in both original and final version of manuscript, but wording changed). By accessing the prepublication history from each report we then assessed each of the peer reviewer’s comments about reporting of methodological items and how authors responded to these requests for each round of the peer review process. In particular we examined comments relating to the reporting of trial design (randomisation and blinding), sample size, details of primary and secondary outcomes, requests for additional analyses, and the trial conclusion. We classified the nature of the comments as having either a positive impact on reporting (comment by peer reviewer judged to have beneficial effect on reporting) or negative impact (comment by peer reviewer judged to have harmful effect on reporting). We were unable to assess any additional comments made by the academic editors as these reports were typically not published alongside the peer reviewers’ comments.
Data analysis
We summarised the characteristics of the studies and details of peer review. The primary analysis focused on the nature and extent of changes made to manuscripts after peer review, in relation to the reporting of methodological aspects of randomised trials, measured as the number and type of items added, subtracted, or altered in the final submitted version of the manuscript. We also carried out a descriptive analysis of the type of methodological changes requested in the peer reviewers’ comments and the extent to which authors responded to these requests in their manuscript revision.
Results
The PubMed publication type search term “Randomized Controlled Trial” identified 513 possible reports of such trials published in the BMC-series medical journals in the specified period. After screening the titles and abstracts of all retrieved citations, we reviewed 140 full text articles, resulting in 103 primary reports of randomised trials (figure⇓); we excluded 10 as no prepublication history was available electronically (for technical reasons) at the time of performing this study. This resulted in 93 primary reports of randomised trials (see supplementary appendix 2 for list of included BMC-series journals, and appendix 3 for list of included trials) where all submitted versions of the manuscript and the corresponding peer review comments and author responses were available online from the respective BMC-series medical journal websites.
Characteristics of primary reports of randomised trials
Table 1⇓ provides information on the general characteristics of the primary reports of randomised trials as described in the published article. The majority of reports were single centre (n=47; 51%), parallel group trials (n=75; 81%), with two study groups (n=75; 81%) and a median sample size of 132 participants in each trial (10th to 90th centile, 30 to 527). Around a third (n=35; 38%) of reports assessed counselling or lifestyle interventions, 27% (n=25) drug interventions, and 14% (n=13) surgical interventions. The remaining studies (n=20, 21%) assessed other types of interventions, such as educational strategies or use of equipment.
Peer reviewers and numerical changes to manuscripts
By accessing the prepublication history for each published article we were able to record information on the life cycle of each manuscript (table 2⇓). The median time interval between the original and final submitted version of a manuscript was 148 days (range 29-240) with 36 days (range 4-142) from final submission to online publication. The number of submitted versions of a manuscript varied (median 3, range 2-8), with a median of two peer reviewers (range 1-5) and two review rounds of peer review for each manuscript (range 1-4). Overall, the median proportion of words deleted from the original manuscript was 11% (range 1-60%) and for words added was 20% (range 2-88%).
Comparison between the original and final submitted manuscript
Table 3⇓ shows the extent of changes in the reporting of methodological details made to manuscripts after peer review. More than half of the articles did not adequately report the results for the primary or secondary outcomes (for example, the estimated effect size and confidence interval) (n=51; 55%), whether the study was blinded (n=46; 50%), concealment of the random allocation sequence (n=50; 54%), important harms (n=60; 65%), or details of the trial protocol (n=84; 90%), in either the original or the final version of the manuscript. More than one third of articles also failed to describe the method of random sequence generation (n=35; 38%), report a sample size calculation (n=32; 34%), or clearly specify the primary and secondary outcomes (n=33; 35%).
Where there were changes between manuscript versions these usually involved “adding” new information that had been missing from the original submission, or “altering” existing information. Changes included adding information about the random sequence generation (n=11; 12%) and concealment of allocation sequence (n=9; 10%), blinding (n=11; 12%), sample size (n=9; 10%), a CONSORT flow diagram (n=9; 10%), changes to tables (n=17; 18%) or figures (n=14; 15%), details of trial registration (n=22; 24%), or adding additional analyses (n=19; 20%) recommended or requested by the peer reviewer. Other changes included altering information about the trial primary (n=13; 14%) or secondary (n=11; 12%) outcomes in the methods section, altering the presentation of the results for the primary (n=9; 10%) or secondary (n=8; 9%) outcomes, or altering the trial conclusions (n=30; 32%). In addition, several items were omitted from the final manuscript, these included “randomised” in the manuscript title (n=3), one or more study authors (n=3), sample size calculations (n=2), tables (n=3), and figures (n=4). Few changes were made to the abstract, those that were made concerned altering of the abstract conclusion (n=15; 16%).
Nature of changes requested by peer reviewers and impact on reporting
In the final part of this study we looked at the nature of changes requested by peer reviewers and the extent to which authors responded to these requests (table 4⇓). In around a third of manuscripts peer reviewers commented on the trial design (for example, the method of randomisation, blinding) (n=29; 31%), sample size (n=30; 32%), and trial conclusions (n=30; 32%); 24% (n=22) commented on the primary or secondary outcomes and 16% (n=15) on the outcome results. In 22% (n=20) of the manuscripts, peer reviewers recommended or requested additional analyses, 22% (n=20) mentioned the CONSORT statement, 5% (n=5) mentioned the trial protocol, and 12% (n=11) mentioned trial registration.
The number of changes requested by peer reviewers was relatively small and most were classified as having a positive impact on the reporting of the final manuscript (table 5⇓). For example, where peer reviewers commented on the method of randomisation or blinding, the authors either added the item (n=19) or gave more details on how it was done (n=8). Other positive changes included the adding (n=7) or justification (n=8) of the sample size, clarification of the primary and secondary outcomes (n=5) and how they were measured (n=11), clarification (n=12) or the addition (n=2) of the outcomes results, and toning down the conclusion to reflect the trial results (n=27). In some instances peer reviewers’ comments had no impact on reporting, in that the authors ignored the comments or responded in the “response to reviewers” without making a change in the manuscript (n=16). For example, the peer reviewer commented on the lack of or small sample size (n=11), clarification of how the primary outcome was measured (n=3), or details on randomisation (n=2), to which the authors either did not respond to the peer reviewers request or responded but said that they did not do it, which was not reflected in the final manuscript.
However, some of the changes requested by peer reviewers were classified as having a negative impact on reporting. This was particularly apparent in the analysis where authors added new subgroup or sensitivity analyses (n=15) requested by the peer reviewer (that is, were not part of the original study and were not reported as such in the final version of the manuscript). Other negative changes included deleting (n=2) or adding retrospective (n=2) sample size calculations, adding a new secondary outcome (n=2), and over-inflation of the conclusion that did not reflect the trial results (n=3). In addition there were two instances where the peer reviewers had requested that the authors change the primary outcome (n=1) or add additional analyses (n=1), and the authors responded to say that was not the purpose of their study. Supplementary appendix 4 provides examples of the type of changes requested by peer reviewers and the authors’ response to these requests.
Discussion
This sample of BMC-series medical journals offered a unique opportunity to investigate the effects of peer review on the reporting of published manuscripts, something which for most journals remains hidden within the editorial decision making process.9 In our study, the number of changes between the original and final submitted version of a manuscript was relatively small. Most involved adding new information (where this had been missing from the original submission) or altering existing information to give more detail about a specific item. The majority of changes recommended or requested by peer reviewers were classified as having a positive impact on reporting, such as adding information about randomisation, blinding, and sample size, altering the trial outcomes (for example, by specifying how the outcome was measured or at what time point), the presentation of the trial results, and the toning down of the conclusion to reflect the trial results.10 Some of the changes, however, were classified as having a negative impact on reporting, such as requests by peer reviewers for unplanned additional analyses that were not prespecified in the original trial protocol and that were not reported as such in the final manuscript. There is a concern that these additional analyses could be driven by an existing knowledge of the data, or the interests of the reader, rather than the primary focus of the study.11 Importantly, despite some improvements in reporting after peer review, it is clear that peer reviewers often fail to detect several important deficiencies in the reporting of the methods and results of randomised trials. The extent to which these findings are generalisable to other journals with different editorial and peer review processes is unclear. However, the problem of poor reporting is something that is seen consistently across the medical literature,12 suggesting that the findings seen in this sample of journals might be representative of other journals.
Comparison with other studies
We are not aware of other studies specifically dealing with the type and nature of changes made to manuscripts after peer review in the reporting of methodological aspects of randomised trials. Other studies on the impact of peer review have predominantly looked at the use of checklists to improve the quality of peer review, the effects of blinding authors and peer reviewers,13 14 and the implementation of training strategies for peer reviewers,15 16 with little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.5 More recently, two randomised controlled trials have shown that adding a statistical reviewer as part of the peer review process has a positive effect on the quality of the final manuscript,17 18 and that additional review based on reporting guidelines such as STROBE (strengthening the reporting of observational studies in epidemiology) and CONSORT can also improve the quality of submitted manuscripts.19
Limitations of this study
Our study has several limitations. Firstly, our sample was limited to BMC-series medical journals where peer reviews are published and available in the public domain. It is unclear the extent to which open peer review hinders reviewers from being overly critical or demanding and whether journals with a closed system might see more changes. Secondly, it is possible that manuscripts published in journals with different editorial and peer review processes, such as journals with a higher impact, greater editorial control, and more resources, may show a greater number of changes after peer review.9 Thirdly, we do not know the level of methodological expertise of the peer reviewers and whether they were predominately from a clinical, trial methodology, or statistical background. Finally, we only assessed the effects of peer review on the reporting of methodological aspects of randomised trials. We did not look at the clinical aspects of peer review as this would have required content expertise in each of the specific disease areas under investigation. We are therefore unable to comment on the effect of peer review on improving the reporting of clinical aspects of randomised trials.
Implications for peer review
This sample of BMC-series medical journals, whereby all peer reviewers’ comments, authors’ responses, and submitted versions of the manuscript are included alongside the published article provides a unique insight into the editorial process. This information is essential to enable readers to have a clear and transparent account of the peer review process. We would strongly recommend this model should be followed by other leading journals.
In our study it is clear that not all changes requested by peer reviewers are appropriate, as authors sometimes acceded to requests for inappropriate revisions to their manuscript; presumably in the hope that it would increase the likelihood of the manuscript being accepted for publication if they made the changes. Getting good peer reviewers for manuscripts is a common problem, particularly for smaller “lower impact” journals. Peer reviewers who do not engage with the manuscript and do it merely to say they review for a particular journal is equally as harmful as an author who fails to provide full details in how the study was carried out. Journal editors need to have better mechanisms to ensure that comments by peer reviewers are appropriate20 and that authors respond appropriately. Editors should also consider peer reviewers who are dedicated to reviewing different elements of the manuscript.18 The use of checklists and reporting guidelines (such as the CONSORT statement which aims to improve the completeness and transparent reporting of randomised trials) has been shown to help peer reviewers to identify problems in the reporting of manuscripts more effectively.19 In our study it took on average between 15 and 20 minutes to review each manuscript against the CONSORT checklist items; a task that could be done internally by a trained editorial assistant (internal peer review) before sending a manuscript out for external peer review. It is extremely unlikely that the “average” peer reviewer has the relevant clinical and methodological expertise to perform each task effectively.21
Conclusion
Based on this sample of BMC-series medical journals, we found that peer reviewers failed to detect important deficiencies in the reporting of the methods and results in randomised trials. The number of changes requested by peer reviewers was relatively small; however, most authors did comply with recommendations or requests by peer reviewers in their revised manuscript. The majority of requests could be seen to have a positive impact on reporting, although there were instances of suggestions that were inappropriate and had a negative impact on the reporting of the final manuscript. Better use and adherence of reporting checklists (www.equator-network.org) by journal editors, peer reviewers, and authors could be one important step towards improving the reporting of published articles.
What is already known on this topic
Despite the widespread use of peer review little is known about its impact on the quality of reporting of published research articles
Inadequacies in the methodology and reporting of research is widely recognised
Substantial uncertainty exists about the peer review process as a mechanism to improve reporting of the scientific literature
What this study adds
Peer reviewers often fail to detect important deficiencies in the reporting of the methods and results of randomised trials
Peer reviewers requested relatively few changes for reporting of trial methods and results
Most requests had a positive impact on reporting but in some instances the requested changes could have a negative impact
Notes
Cite this as: BMJ 2014;348:g4145
Footnotes
Contributors: SH and DA were involved in the design, implementation, and analysis of the study, and in writing the final manuscript. GC, LY, JC, MS, RW, IB, and LS were involved in the implementation of the study and in commenting on drafts of the final manuscript. SH is responsible for the overall content as guarantor.
Funding: This study received no external funding.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Data sharing: No additional data available.
Transparency: The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.