Research on peer review and biomedical publication

At The BMJ we try to practise what we preach—just as in healthcare, our decisions should be based on solid evidence. Of course, solid evidence can only be generated by solid research that is well conceived, executed, and disseminated. Paradoxically, the processes underlying the generation, dissemination, and implementation of research have not been well studied: we need more and better research on research.

Journals amass huge amounts of data from authors, reviewers, and editors. Each year The BMJ alone receives around 3500 research submissions, which annually generate over 2000 reviews. And as the publisher of a broad portfolio of over 70 general and specialty journals, BMJ Publishing Group’s manuscript tracking systems hold considerably more data. Journal publishers and editors hold this wealth of data but have limited time or expertise to analyse them, whereas researchers have the skills but no access. Some publishers are concerned about data protection and privacy issues, others are just secretive. Attracting funding for conducting research on research is challenging, yet evidence of serious problems with the quality of research is stacking up.12

The BMJ is committed to improving the evidence base around the publishing process. We have published many studies on peer review and the functioning of scientific and scholarly journals, and with JAMA we co-organise the quadrennial Peer Review Congress (https://peerreviewcongress.org), the main forum for this type of research. The BMJ has had its own research programme (https://www.bmj.com/about-bmj/evidence-based-publishing) for over 20 years and has taken a lead in carrying out research into all aspects of the publishing process.3

Our research helps inform our editorial policies and practice: randomised trials of opening up the peer review process4567 showed no detrimental effect on review quality, so we now operate fully open peer review (which identifies both authors and reviewers) and publish signed reviews alongside research articles. In another randomised trial, peer reviewer training had no sustainable effect on review quality,8 so we did not invest in training packages, despite their popularity with reviewers.

Our strict policy of competing interest disclosure for research articles and a zero tolerance policy for financial competing interests in education articles and clinical editorials9 are informed by trials reporting the effects of competing interests on readers’ confidence in articles.101112 Finally, The BMJ has implemented patient and public review of research and other content and mandates the reporting of patient and public involvement in research and other articles; robust evaluation of both these initiatives is under way.1314

Some of The BMJ’s research projects answer questions that arise in the course of our work, such as whether reviewers suggested by authors write better or worse (or comparable) reviews than those chosen by editors15 or what happens to clinical trials rejected by The BMJ because they are not prospectively registered.16 We also participate in external research, both as a study site and a partner1718192021; for example, through our partnership with the EU funded doctoral training programme on Methods in Research on Research (http://miror-ejd.eu).

Partnerships with academic institutions are particularly valuable for sharing ideas and developing research talent in this field. We are launching a new collaboration with Maastricht University on the responsible conduct of publishing scientific research. PhD students will use BMJ Publishing Group as a study site for research into a wide range of issues relating to the publication of research, including peer review, authorship, conflicts of interest, publication ethics, handling retractions, data sharing, patient and public involvement in research, bias, preprints, and protocol registration. Full details on how to apply for the BMJ and Maastricht University PhD Programme can be found at https://www.bmj.com/about-bmj/evidence-based-publishing.

We are also embarking on a new programme of research into conflicts of interest with the Evidence, Policy and Influence Collaborative at the University of Sydney in Australia and an evaluation of the quality of a peer review training programme (PEERSPECTIVES), in conjunction with the Institute of Public Health at the Charité-Universitätsmedizin in Berlin. We plan to present findings from these projects at the Ninth International Congress on Peer Review in September 2021, and we encourage others to submit too. Other key upcoming conferences for presenting research on research include the Second PEERE International Conference on Peer Review in March 2020 and the Seventh World Conference on Research Integrity in May 2021.

Doing research into research publication is challenging. There is little consensus across journals on research priorities, and there are no agreed definitions of fundamental practices such as peer review. Every journal is different, so results are hard to generalise. Most research is currently done with a single journal or publisher.

Future research should be a more collaborative effort, including a broad range of journals and publishers to capture the culture and practices of authors, editors, and reviewers from different disciplines. Studies need to be larger, more rigorous, multisite, more interventional, and less descriptive.

Researching research is not academic navel gazing but an essential contribution to the quality, integrity, and safety of the evidence base on which all healthcare rests. We urge funders to support these initiatives and urge journals and publishers to open their doors and partner us in this important work.